Skip to content

feat: support custom compaction for bucketed append tables - #572

Merged
SteNicholas merged 3 commits into
apache:mainfrom
lxy-9602:customize-compact
Nov 19, 2026
Merged

SteNicholas merged 3 commits into
apache:mainfrom
lxy-9602:customize-compact

Conversation

@lxy-9602

@lxy-9602 lxy-9602 commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Purpose

Linked issue: close #571

The existing Compact API selects input files through built-in strategies. This change allows callers to compact an explicit file set using their own selection policy.

  • Add asynchronous FileStoreWrite::CompactFiles.
  • Support non-contiguous input files for fixed-bucket append tables with or without deletion vectors.
  • Reuse the existing compact manager and collect results and background errors through PrepareCommit.
  • Align bucket-append-ordered defaults and writer restoration behavior with Java.

Custom compaction requires bucket-append-ordered=false, write-only=false, and a dedicated writer with the full schema. Real-time mode and data evolution are unsupported. Callers must serialize deletion-vector updates and compaction for the same partition-bucket until commit or discard.

Tests

Unit tests cover input validation, writer restoration, asynchronous task handling, and repeated file selection.

Integration tests cover non-contiguous inputs with and without deletion vectors, a fully deleted input file, commit-message serialization, and post-commit reads.

API and Format

Adds the public FileStoreWrite::CompactFiles API.

No storage format or commit-message protocol changes.

Documentation

Documents custom compaction and bucket-append-ordered behavior in the user guide.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-6)

@lxy-9602
lxy-9602 requested a review from lucasfang November 17, 2026 11:29

@lucasfang lucasfang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@SteNicholas SteNicholas left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@SteNicholas
SteNicholas merged commit d0b5f60 into apache:main Nov 19, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Support custom compaction for bucketed append tables

3 participants