Skip to content

[Feature] Support partial-update in realtime writes. #524

Closed

Description

@HaHaJeff

Motivation and scope

Support PARTIAL_UPDATE alongside the existing DEDUPLICATE engine for primary-key realtime writes by reusing the ordinary Writer's merge functions and options. Implementation: #525.

  • Reuse existing sequence groups, field aggregation, and configured delete handling. The Store retains sorted batches and handles storage and spilling; sorting and merging remain the framework's responsibility. This extends realtime access to the existing merge engine.
  • Cover real memory/disk union reads, prepare, commit, reopen, and replay.
  • Preserve non-realtime production behavior and the realtime Noop compaction policy. Standalone AGGREGATE and FIRST_ROW engines are outside this change.
  • Realtime partial-update currently requires deletion-vectors.enabled=false. Existing ordinary Writer option constraints continue to apply; this feature does not add support for whole-row sequence.field.

Known limitations

The feature inherits the ordinary Writer's existing partial-update merge semantics and their merge-boundary limitations. #525 does not fix the observations below, and realtime reads and writes may inherit them. The C++ observations and Java author reports have different triggering boundaries; they do not establish an identical root cause.

  • C++ Case 1 — delete/reinsert in one batch and commit: the ordinary Writer on baseline becc48a51c8a740bf36f201df5fb23fbb2d2b0f1 retains old b=20 instead of b=null. DELETE and UPDATE_AFTER are in the same batch after a committed seed; no C++ compaction was triggered. The separate-commit control passes before any subsequent partial compaction.
  • Java Case 1b — partial compaction after separate commits: lxy's report, with partial-update.remove-record-on-delete=true and DVs disabled, first commits and compacts the seed to a high-level file, then commits DELETE and reinsertion separately. Reads initially return b=null; partial compaction of only the two newer L0 files, leaving the old high-level file untouched, then causes incorrect b=20, including after reopening. This is the author's report, not an independent Java run here.
  • C++ Case 2 — two later batches before the same commit: after the committed two-row seed, both later batches below precede commit 1. Reads before and after reopening return seq=30, total=47, last="middle". The original last="thirty" expectation is retained below as a continuous-merge reference, not an established guarantee for staged, order-sensitive aggregation.
  • Java Case 2 — separate commits for the two later batches: lxy's current report says that committing those batches separately also produces total=47, last="middle" rather than the reported expectation of "thirty". This differs from the C++ commit boundary and was not independently run here. The intended guarantee for order-sensitive sequence-group aggregation across merge stages still needs clarification.

Case 1 documents incorrect old-field resurrection after a delete. Case 2 documents observed stage-sensitive sequence-group/aggregation behavior and an expectation requiring clarification. These observations are retained as known limitations of the feature's inherited semantics, without promising an independent fix for every case.

Reproduction details

Both C++ cases below were reproduced with the ordinary Writer on baseline becc48a51c8a740bf36f201df5fb23fbb2d2b0f1, without production-code changes. Only diagnostic tests were added. Java results are attributed to lxy's reports; no new C++ or Java tests were run for this description update.

Reproduction setup

The C++ examples below are pseudocode, not runnable Paimon APIs. Each write_batch represents one input batch; record order, batch boundaries, and commit order are intentional. commit(writer, n) represents ordinary Write → PrepareCommit(true, n) → Commit. read_all() performs a fresh scan/plan/read of committed data; reopen_and_read_all() destroys and recreates the writer/commit context before reading again. No realtime context, transport offsets, or progress APIs are involved.

Common options for both C++ cases:

file.format=parquet
file-system=local
bucket=1
bucket-key=id
target-file-size=1048576
merge-engine=partial-update
read.batch-size=1
num-sorted-run.compaction-trigger=100
num-sorted-run.stop-trigger=200

The ordinary Writer retained its normal MergeTreeCompactManager; compaction was not triggered in these C++ cases. This was not a Noop-compaction control, and write-only was not explicitly set.

Case 1: delete followed by reinsertion in the same batch retains an old field

Schema: id BIGINT, a BIGINT, b BIGINT, primary key id.

Additional option:

partial-update.remove-record-on-delete=true
writer.write_batch([INSERT(id=1, a=10, b=20)])
commit(writer, 0)

writer.write_batch([
    DELETE(id=1, a=null, b=null),
    UPDATE_AFTER(id=1, a=30, b=null)
])
commit(writer, 1)

read_all()
# expected: [(id=1, a=30, b=null)]
# actual:   [(id=1, a=30, b=20)]

reopen_and_read_all()
# actual:   [(id=1, a=30, b=20)]

The delete should reset the record. The subsequent partial update must not resurrect b=20 from before the delete.

Passing control: with the same schema, options, seed, and operation order, committing the DELETE separately as commit 1 and then the reinsertion separately as commit 2 passes. Commit 1 and its reopened read are empty; commit 2 and its reopened read return [(id=1, a=30, b=null)]. This control does not cover subsequent partial compaction; Case 1b shows why separate commits alone do not establish correctness across intermediate merges.

Source analysis suggests that an intermediate summary loses the preceding delete boundary, allowing a later null field to retain an older value. This is a hypothesis from source analysis, not an instrumented observation of intermediate records.

Case 1b: Java partial compaction loses the delete boundary

Reported by lxy; not independently rerun here. Schema: id BIGINT, a BIGINT, b BIGINT, primary key id. The following is pseudocode paraphrasing that report:

merge-engine=partial-update
partial-update.remove-record-on-delete=true
deletion-vectors.enabled=false

write(INSERT(id=1, a=10, b=20))
commit()
compact_to_high_level()

write(DELETE(id=1))
commit()  # separate commit

write(UPDATE_AFTER(id=1, a=30, b=null))
commit()  # separate commit

read_all()
# expected and actual: [(id=1, a=30, b=null)]

partial_compact(only_the_two_newer_L0_files)
# Leave the older high-level file unchanged.

read_all()
# expected: [(id=1, a=30, b=null)]
# actual:   [(id=1, a=30, b=20)]

reopen_and_read_all()
# actual:   [(id=1, a=30, b=20)]

As lxy explains, the intermediate merge loses the delete boundary. When its result is later merged with the untouched older high-level file, the pre-delete b=20 can reappear.

Case 2: sequence-group aggregation across merge stages

Schema: id BIGINT, seq BIGINT, total BIGINT, last STRING, primary key id.

Additional options:

fields.seq.sequence-group=total,last
fields.total.aggregate-function=sum
fields.last.aggregate-function=last_non_null_value
writer.write_batch([
    INSERT(id=1, seq=9, total=6, last="nine"),
    INSERT(id=1, seq=10, total=4, last="ten")
])
commit(writer, 0)
# committed seed: [(id=1, seq=10, total=10, last="ten")]

writer.write_batch([
    INSERT(id=1, seq=30, total=30, last="thirty"),
    INSERT(id=1, seq=20, total=20, last="twenty")
])
writer.write_batch([
    UPDATE_AFTER(id=1, seq=25, total=5, last="middle"),
    UPDATE_BEFORE(id=1, seq=20, total=20, last=null),
    UPDATE_AFTER(id=1, seq=30, total=2, last=null)
])
commit(writer, 1)

read_all()
# continuous-merge reference (not an established staged-merge guarantee): [(id=1, seq=30, total=47, last="thirty")]
# actual:   [(id=1, seq=30, total=47, last="middle")]

reopen_and_read_all()
# actual:   [(id=1, seq=30, total=47, last="middle")]

There is no commit or explicit ordinary Seal/spill between the two batches preceding commit 1; the ordinary reproduction uses one flush per commit. The sum is 47, while last="middle" differs from the original continuous-merge reference of "thirty". Clarification: only the expectation label and its interpretation have changed here; the original inputs, batch/commit order, and observed outputs are unchanged. Whether this order-sensitive aggregation must preserve that reference across merge stages is not established by this reproduction.

Source analysis suggests that staged merging loses the association between a field value and its originating sequence: a summary may associate "middle" with sequence 30 despite its origin at sequence 25. The original run did not instrument intermediate stages, so the precise mechanism still needs confirmation.

Evidence and scope

The original C++ test patches, exact inputs, logs, JUnit results, and passing separate-commit control are preserved locally. They have not been uploaded as public attachments; no external evidence links are claimed here. The original test assertions remain part of that evidence; this description clarifies Case 2's expectation as a continuous-merge reference without changing its observed output or claiming a staged-merge contract.

The realtime feature reuses the same merge implementation and may inherit these limitations. The original C++ examples were not independently reproduced on realtime feature revision 2e2f194ccdab76831b2988a9b749f5d8c299d8f7; this issue does not claim that realtime introduced these limitations or that the feature fixes them. The value-stats pruning problem in wangyong’s PR #525 comment is separate from these merge-boundary observations. Separate hidden sequence-group projection/deletion dependency problems remain outside this issue’s documented reproductions.

Further investigation should distinguish delete-boundary correctness from the intended guarantees of order-sensitive aggregation across merge stages, retaining the exact inputs, boundaries, observed results, and passing separate-commit control above. Realtime coverage of these observations should be established separately against an explicitly identified source revision and its observable stages.

Activity

  1. changed the title [-][Bug] Partial-update staged merging can lose delete boundaries and field-version information[/-] [+][Feature] Support partial-update in realtime writes.[/+] on Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions