Repository navigation
Description
Motivation and scope
Support PARTIAL_UPDATE alongside the existing DEDUPLICATE engine for primary-key realtime writes by reusing the ordinary Writer's merge functions and options. Implementation: #525.
- Reuse existing sequence groups, field aggregation, and configured delete handling. The Store retains sorted batches and handles storage and spilling; sorting and merging remain the framework's responsibility. This extends realtime access to the existing merge engine.
- Cover real memory/disk union reads, prepare, commit, reopen, and replay.
- Preserve non-realtime production behavior and the realtime Noop compaction policy. Standalone
AGGREGATEandFIRST_ROWengines are outside this change. - Realtime partial-update currently requires
deletion-vectors.enabled=false. Existing ordinary Writer option constraints continue to apply; this feature does not add support for whole-rowsequence.field.
Known limitations
The feature inherits the ordinary Writer's existing partial-update merge semantics and their merge-boundary limitations. #525 does not fix the observations below, and realtime reads and writes may inherit them. The C++ observations and Java author reports have different triggering boundaries; they do not establish an identical root cause.
- C++ Case 1 — delete/reinsert in one batch and commit: the ordinary Writer on baseline
becc48a51c8a740bf36f201df5fb23fbb2d2b0f1retains oldb=20instead ofb=null. DELETE and UPDATE_AFTER are in the same batch after a committed seed; no C++ compaction was triggered. The separate-commit control passes before any subsequent partial compaction. - Java Case 1b — partial compaction after separate commits: lxy's report, with
partial-update.remove-record-on-delete=trueand DVs disabled, first commits and compacts the seed to a high-level file, then commits DELETE and reinsertion separately. Reads initially returnb=null; partial compaction of only the two newer L0 files, leaving the old high-level file untouched, then causes incorrectb=20, including after reopening. This is the author's report, not an independent Java run here. - C++ Case 2 — two later batches before the same commit: after the committed two-row seed, both later batches below precede commit 1. Reads before and after reopening return
seq=30, total=47, last="middle". The originallast="thirty"expectation is retained below as a continuous-merge reference, not an established guarantee for staged, order-sensitive aggregation. - Java Case 2 — separate commits for the two later batches: lxy's current report says that committing those batches separately also produces
total=47, last="middle"rather than the reported expectation of"thirty". This differs from the C++ commit boundary and was not independently run here. The intended guarantee for order-sensitive sequence-group aggregation across merge stages still needs clarification.
Case 1 documents incorrect old-field resurrection after a delete. Case 2 documents observed stage-sensitive sequence-group/aggregation behavior and an expectation requiring clarification. These observations are retained as known limitations of the feature's inherited semantics, without promising an independent fix for every case.
Reproduction details
Both C++ cases below were reproduced with the ordinary Writer on baseline becc48a51c8a740bf36f201df5fb23fbb2d2b0f1, without production-code changes. Only diagnostic tests were added. Java results are attributed to lxy's reports; no new C++ or Java tests were run for this description update.
Reproduction setup
The C++ examples below are pseudocode, not runnable Paimon APIs. Each write_batch represents one input batch; record order, batch boundaries, and commit order are intentional. commit(writer, n) represents ordinary Write → PrepareCommit(true, n) → Commit. read_all() performs a fresh scan/plan/read of committed data; reopen_and_read_all() destroys and recreates the writer/commit context before reading again. No realtime context, transport offsets, or progress APIs are involved.
Common options for both C++ cases:
file.format=parquet
file-system=local
bucket=1
bucket-key=id
target-file-size=1048576
merge-engine=partial-update
read.batch-size=1
num-sorted-run.compaction-trigger=100
num-sorted-run.stop-trigger=200
The ordinary Writer retained its normal MergeTreeCompactManager; compaction was not triggered in these C++ cases. This was not a Noop-compaction control, and write-only was not explicitly set.
Case 1: delete followed by reinsertion in the same batch retains an old field
Schema: id BIGINT, a BIGINT, b BIGINT, primary key id.
Additional option:
partial-update.remove-record-on-delete=true
writer.write_batch([INSERT(id=1, a=10, b=20)])
commit(writer, 0)
writer.write_batch([
DELETE(id=1, a=null, b=null),
UPDATE_AFTER(id=1, a=30, b=null)
])
commit(writer, 1)
read_all()
# expected: [(id=1, a=30, b=null)]
# actual: [(id=1, a=30, b=20)]
reopen_and_read_all()
# actual: [(id=1, a=30, b=20)]
The delete should reset the record. The subsequent partial update must not resurrect b=20 from before the delete.
Passing control: with the same schema, options, seed, and operation order, committing the DELETE separately as commit 1 and then the reinsertion separately as commit 2 passes. Commit 1 and its reopened read are empty; commit 2 and its reopened read return [(id=1, a=30, b=null)]. This control does not cover subsequent partial compaction; Case 1b shows why separate commits alone do not establish correctness across intermediate merges.
Source analysis suggests that an intermediate summary loses the preceding delete boundary, allowing a later null field to retain an older value. This is a hypothesis from source analysis, not an instrumented observation of intermediate records.
Case 1b: Java partial compaction loses the delete boundary
Reported by lxy; not independently rerun here. Schema: id BIGINT, a BIGINT, b BIGINT, primary key id. The following is pseudocode paraphrasing that report:
merge-engine=partial-update
partial-update.remove-record-on-delete=true
deletion-vectors.enabled=false
write(INSERT(id=1, a=10, b=20))
commit()
compact_to_high_level()
write(DELETE(id=1))
commit() # separate commit
write(UPDATE_AFTER(id=1, a=30, b=null))
commit() # separate commit
read_all()
# expected and actual: [(id=1, a=30, b=null)]
partial_compact(only_the_two_newer_L0_files)
# Leave the older high-level file unchanged.
read_all()
# expected: [(id=1, a=30, b=null)]
# actual: [(id=1, a=30, b=20)]
reopen_and_read_all()
# actual: [(id=1, a=30, b=20)]
As lxy explains, the intermediate merge loses the delete boundary. When its result is later merged with the untouched older high-level file, the pre-delete b=20 can reappear.
Case 2: sequence-group aggregation across merge stages
Schema: id BIGINT, seq BIGINT, total BIGINT, last STRING, primary key id.
Additional options:
fields.seq.sequence-group=total,last
fields.total.aggregate-function=sum
fields.last.aggregate-function=last_non_null_value
writer.write_batch([
INSERT(id=1, seq=9, total=6, last="nine"),
INSERT(id=1, seq=10, total=4, last="ten")
])
commit(writer, 0)
# committed seed: [(id=1, seq=10, total=10, last="ten")]
writer.write_batch([
INSERT(id=1, seq=30, total=30, last="thirty"),
INSERT(id=1, seq=20, total=20, last="twenty")
])
writer.write_batch([
UPDATE_AFTER(id=1, seq=25, total=5, last="middle"),
UPDATE_BEFORE(id=1, seq=20, total=20, last=null),
UPDATE_AFTER(id=1, seq=30, total=2, last=null)
])
commit(writer, 1)
read_all()
# continuous-merge reference (not an established staged-merge guarantee): [(id=1, seq=30, total=47, last="thirty")]
# actual: [(id=1, seq=30, total=47, last="middle")]
reopen_and_read_all()
# actual: [(id=1, seq=30, total=47, last="middle")]
There is no commit or explicit ordinary Seal/spill between the two batches preceding commit 1; the ordinary reproduction uses one flush per commit. The sum is 47, while last="middle" differs from the original continuous-merge reference of "thirty". Clarification: only the expectation label and its interpretation have changed here; the original inputs, batch/commit order, and observed outputs are unchanged. Whether this order-sensitive aggregation must preserve that reference across merge stages is not established by this reproduction.
Source analysis suggests that staged merging loses the association between a field value and its originating sequence: a summary may associate "middle" with sequence 30 despite its origin at sequence 25. The original run did not instrument intermediate stages, so the precise mechanism still needs confirmation.
Evidence and scope
The original C++ test patches, exact inputs, logs, JUnit results, and passing separate-commit control are preserved locally. They have not been uploaded as public attachments; no external evidence links are claimed here. The original test assertions remain part of that evidence; this description clarifies Case 2's expectation as a continuous-merge reference without changing its observed output or claiming a staged-merge contract.
The realtime feature reuses the same merge implementation and may inherit these limitations. The original C++ examples were not independently reproduced on realtime feature revision 2e2f194ccdab76831b2988a9b749f5d8c299d8f7; this issue does not claim that realtime introduced these limitations or that the feature fixes them. The value-stats pruning problem in wangyong’s PR #525 comment is separate from these merge-boundary observations. Separate hidden sequence-group projection/deletion dependency problems remain outside this issue’s documented reproductions.
Further investigation should distinguish delete-boundary correctness from the intended guarantees of order-sensitive aggregation across merge stages, retaining the exact inputs, boundaries, observed results, and passing separate-commit control above. Realtime coverage of these observations should be established separately against an explicitly identified source revision and its observable stages.
Activity
[-][Bug] Partial-update staged merging can lose delete boundaries and field-version information[/-][+][Feature] Support partial-update in realtime writes.[/+]