Skip to content

[Feature] Support dynamic-bucket primary-key tables #541

Open

Description

@SteNicholas

Search before asking

  • I searched in the issues and found no existing issue tracking the full dynamic-bucket primary-key table lifecycle.

Related work: #493 tracks batch reads; #494 introduced resolved-bucket-mode handling for index manifests; PR #499 implements batch reads with Java compatibility coverage. PR #499 is still open at the time of filing, so its actual landing should be tracked as part of this issue.

Motivation

Sub-issue of #519: Core Feature Alignment — Support dynamic-bucket primary-key tables.

Paimon Java primary-key tables can use bucket = -1 to grow the number of buckets as keys arrive. Paimon C++ recognizes BucketMode::HASH_DYNAMIC, but the table write path still rejects non-positive bucket counts outside postpone mode, and BucketIdCalculator rejects dynamic primary-key bucket calculation. The existing write restore path also supplies no dynamic-bucket index.

Batch reading can reuse bucket IDs recorded in manifests, which is the scope of PR #499. Full support additionally needs stateful key routing, Java-compatible bucket-index persistence and recovery, and streaming reads that account for newly created buckets.

This issue targets HASH_DYNAMIC primary-key tables, including partitioned tables whose primary keys contain all partition fields. Cross-partition upserts remain a separate roadmap item in #519.

Solution

Implement this in focused stages while preserving the existing fixed-bucket and append-table behavior:

  • Complete batch-read support. Reuse [Feature] Support batch reads of primary-key tables with dynamic buckets #493 / refactor(index): pass resolved bucket mode to index manifests #494 / PR feat(scan): batch read primary-key tables with dynamic buckets #499, track the batch-read PR through landing, and verify batch planning, predicate filtering, snapshot selection, and deletion-vector lookup across multiple partitions and buckets. Keep HASH_DYNAMIC distinct from append-table BUCKET_UNAWARE even though both configure bucket = -1.
  • Add a stateful dynamic bucket assigner. Restore the Java-compatible primary-key-hash-to-bucket mapping, reuse a key's assigned bucket for subsequent upserts and deletes, and allocate buckets for new keys according to the supported Java dynamic-bucket options, including dynamic-bucket.target-row-num. Define assignment ownership and supported writer topology consistently with Java.
  • Persist and restore bucket indexes. Implement compatible hash-index file reading/writing and manifest handling. Connect them to RestoreFiles::DynamicBucketIndex() and FileSystemWriteRestore, restoring only mappings visible in the selected committed snapshot. Keep dynamic-bucket indexes and deletion-vector indexes correctly scoped by partition and bucket.
  • Integrate dynamic routing with writes and commits. Add the stateful routing path to table writes and support insert, upsert, delete, flush, prepare-commit, commit, abort, and restart. Publish data, bucket-index changes, and applicable deletion-vector changes consistently in snapshots; a failed or retried commit must not lose or reassign committed keys.
  • Preserve correctness through compaction and cleanup. Retain valid key routing as data files are compacted or retired, and retain index files needed by supported snapshots and tags. Apply existing writer/concurrency constraints and explicitly validate unsupported combinations.
  • Support streaming reads. Discover newly appearing buckets while planning successive snapshots and preserve the existing supported incremental/changelog and deletion-vector semantics.
  • Document the support matrix. Describe supported reads and writes, Java-compatible options, assignment ownership, recovery behavior, and remaining restrictions. Update the current statement that primary-key dynamic-bucket writes are unsupported.

Useful implementation entry points:

Anything else?

Acceptance and validation:

  • Add focused assigner, index-file, restore, and commit unit tests, plus table-level integration tests.
  • Cover unpartitioned and partitioned tables, multiple buckets, bucket growth, repeated keys, hash collisions, upserts, deletes, and writer restart from committed snapshots.
  • Cover commit/abort/retry failures and compaction, ensuring that committed keys retain consistent routing and no duplicate live primary keys appear across buckets.
  • Cover deletion vectors enabled and disabled and the existing applicable primary-key merge modes.
  • Verify batch and streaming reads across snapshots that introduce new buckets, and confirm that fixed-bucket and append-unaware tables retain their existing behavior.
  • Cross-read Java-written tables from C++ and C++-written tables from Java, including bucket-index files, restored assignments, updates/deletes, and deletion vectors.
  • Update documentation and keep the parent roadmap item unchecked until the agreed full lifecycle is supported and validated.

The existing batch-read PR is an initial delivery of this roadmap item. Its completion alone does not complete dynamic-bucket writes and streaming reads.

Are you willing to submit a PR?

  • I'm willing to submit a PR!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions