Repository navigation
Description
Search before asking
- I searched in the issues and found no existing issue tracking the full dynamic-bucket primary-key table lifecycle.
Related work: #493 tracks batch reads; #494 introduced resolved-bucket-mode handling for index manifests; PR #499 implements batch reads with Java compatibility coverage. PR #499 is still open at the time of filing, so its actual landing should be tracked as part of this issue.
Motivation
Sub-issue of #519: Core Feature Alignment — Support dynamic-bucket primary-key tables.
Paimon Java primary-key tables can use bucket = -1 to grow the number of buckets as keys arrive. Paimon C++ recognizes BucketMode::HASH_DYNAMIC, but the table write path still rejects non-positive bucket counts outside postpone mode, and BucketIdCalculator rejects dynamic primary-key bucket calculation. The existing write restore path also supplies no dynamic-bucket index.
Batch reading can reuse bucket IDs recorded in manifests, which is the scope of PR #499. Full support additionally needs stateful key routing, Java-compatible bucket-index persistence and recovery, and streaming reads that account for newly created buckets.
This issue targets HASH_DYNAMIC primary-key tables, including partitioned tables whose primary keys contain all partition fields. Cross-partition upserts remain a separate roadmap item in #519.
Solution
Implement this in focused stages while preserving the existing fixed-bucket and append-table behavior:
- Complete batch-read support. Reuse [Feature] Support batch reads of primary-key tables with dynamic buckets #493 / refactor(index): pass resolved bucket mode to index manifests #494 / PR feat(scan): batch read primary-key tables with dynamic buckets #499, track the batch-read PR through landing, and verify batch planning, predicate filtering, snapshot selection, and deletion-vector lookup across multiple partitions and buckets. Keep
HASH_DYNAMICdistinct from append-tableBUCKET_UNAWAREeven though both configurebucket = -1. - Add a stateful dynamic bucket assigner. Restore the Java-compatible primary-key-hash-to-bucket mapping, reuse a key's assigned bucket for subsequent upserts and deletes, and allocate buckets for new keys according to the supported Java dynamic-bucket options, including
dynamic-bucket.target-row-num. Define assignment ownership and supported writer topology consistently with Java. - Persist and restore bucket indexes. Implement compatible hash-index file reading/writing and manifest handling. Connect them to
RestoreFiles::DynamicBucketIndex()andFileSystemWriteRestore, restoring only mappings visible in the selected committed snapshot. Keep dynamic-bucket indexes and deletion-vector indexes correctly scoped by partition and bucket. - Integrate dynamic routing with writes and commits. Add the stateful routing path to table writes and support insert, upsert, delete, flush, prepare-commit, commit, abort, and restart. Publish data, bucket-index changes, and applicable deletion-vector changes consistently in snapshots; a failed or retried commit must not lose or reassign committed keys.
- Preserve correctness through compaction and cleanup. Retain valid key routing as data files are compacted or retired, and retain index files needed by supported snapshots and tags. Apply existing writer/concurrency constraints and explicitly validate unsupported combinations.
- Support streaming reads. Discover newly appearing buckets while planning successive snapshots and preserve the existing supported incremental/changelog and deletion-vector semantics.
- Document the support matrix. Describe supported reads and writes, Java-compatible options, assignment ownership, recovery behavior, and remaining restrictions. Update the current statement that primary-key dynamic-bucket writes are unsupported.
Useful implementation entry points:
- Bucket mode definition and resolution.
- Table read/write construction and bucket calculation.
- RestoreFiles and filesystem write restore.
- Index manifest handling and the existing primary-key write/commit and deletion-vector paths.
Anything else?
Acceptance and validation:
- Add focused assigner, index-file, restore, and commit unit tests, plus table-level integration tests.
- Cover unpartitioned and partitioned tables, multiple buckets, bucket growth, repeated keys, hash collisions, upserts, deletes, and writer restart from committed snapshots.
- Cover commit/abort/retry failures and compaction, ensuring that committed keys retain consistent routing and no duplicate live primary keys appear across buckets.
- Cover deletion vectors enabled and disabled and the existing applicable primary-key merge modes.
- Verify batch and streaming reads across snapshots that introduce new buckets, and confirm that fixed-bucket and append-unaware tables retain their existing behavior.
- Cross-read Java-written tables from C++ and C++-written tables from Java, including bucket-index files, restored assignments, updates/deletes, and deletion vectors.
- Update documentation and keep the parent roadmap item unchecked until the agreed full lifecycle is supported and validated.
The existing batch-read PR is an initial delivery of this roadmap item. Its completion alone does not complete dynamic-bucket writes and streaming reads.
Are you willing to submit a PR?
- I'm willing to submit a PR!
Activity