Skip to content

feat: add APIs for append-table updates with deletion vectors - #558

Open
lxy-9602 wants to merge 2 commits into
apache:mainfrom
lxy-9602:dv-expose
Open

lxy-9602 wants to merge 2 commits into
apache:mainfrom
lxy-9602:dv-expose

Conversation

@lxy-9602

Copy link
Copy Markdown
Member

Purpose

Linked issue: close #557

Allow external engines to update append tables by locating live physical rows, appending replacement rows, and committing deletion vectors together with the appended files.

  • Add split-bound FileIndexRead queries that reuse per-file readers and historical DV across predicates. Reuse metadata statistics and the existing precise read path.
  • Add DeletionVectorWrite::PrepareCommit to merge new deletion positions with historical DV at one base snapshot and return commit messages.
  • Support fixed-bucket and unaware-bucket managed append tables with bitmap32 DV. Callers must serialize updates within each partition/bucket through commit.

Tests

  • Add parameterized coverage for fixed/unaware buckets, embedded/external indexes, repeated queries, schema evolution, historical DV merging, and combined append/DV commits.
  • Cover empty results, invalid targets, unsupported configurations, and partition/bucket isolation.
  • Extend regression coverage for predicate-state isolation, batch DV merging, and snapshot deletion counts.

API and Format

Add public FileIndexRead, FileRowSelection, and DeletionVectorWrite APIs without exposing BinaryRow.

DataSplit::GetFileList reports snapshot DV cardinality when available.

No storage format or protocol changes.

Documentation

Document the new APIs and the append/DV update workflow, including supported table modes and serialization requirements.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-6)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Expose APIs for append-table updates with deletion vectors

1 participant