Repository navigation
feat(core): support partial updates in PK realtime writes - #525
Conversation
|
The code LGTM. I also reproduced a similar issue in Java partial-update with The intermediate merge loses the delete boundary, allowing the old Given that Java also has this issue, perhaps we could adjust the issue and PR descriptions to clarify that it is a shared defect in the existing merge semantics. A reference to #524 may be sufficient, without presenting it as an additional correctness boundary for this realtime feature. |
|
@HaHaJeff Hi, could you please have this issue’s classification and scope reviewed by human. This needs a deliberate assessment of the behavior and the feature’s intended scope. I’m not sure a separate bug report is the most useful framing for this work. The delete-boundary and field-version problems appear to be existing merge-boundary limitations shared with Java, rather than something introduced by realtime support. For example, the issue says “it does not establish Java reproduction of Case 2.” However, committing Case 2’s two batches separately in Java also produces My expectation was a feature issue for supporting realtime writes with partial-update, with the known merge-boundary limitations documented alongside the comparable Java behavior. |
|
@lxy-9602 Would it be enough to document the feature scope and known merge-boundary limitations in #525, and then close #524? That would avoid maintaining the same information in both places. Context: original comment. |
|
@lxy-9602 I’ve reframed #524 as a feature issue for realtime partial-update, keeping the original reproductions and adding the known merge-boundary limitations alongside your Java observations. The C++ and Java commit/compaction boundaries are documented separately. I’ve also updated #525’s description to match, making clear that this PR reuses the ordinary Writer’s semantics without changing those existing merge behaviors. Context: original comment. |
Retain both Arrow utility headers required by the partial-update tests and the upstream refresh regression test.
Purpose
Enable partial-update for primary-key realtime writes by reusing the existing merge functions and options used by the ordinary Writer.
Feature tracking issue: #524
PARTIAL_UPDATEalongsideDEDUPLICATE, preserving the existing deduplicate ignore-delete restrictions. Standalone aggregation and first-row engines remain unsupported.Realtime partial-update currently requires deletion vectors to be disabled (deletion-vectors.enabled=false).
Known Limitations
This PR reuses the ordinary Writer's existing partial-update merge semantics and inherits their merge-boundary limitations. It does not fix delete/reinsert old-field resurrection or stage-sensitive sequence-group/field-aggregation behavior. Existing C++ ordinary Writer observations and lxy's Java reports use different triggering boundaries: C++ delete/reinsert is in one batch/commit, while the Java report becomes incorrect after partial compaction of separately committed delete/reinsert files; C++ Case 2 writes both later batches before one commit, while Java commits them separately. Both Case 2 reports produce
total=47, last="middle";"thirty"is a continuous-merge reference, and the intended staged-aggregation guarantee needs clarification. See #524 for the full reproductions, original results, and attributed Java reports. Realtime may inherit these limitations; this feature does not establish correctness for all partial-update combinations.Tests
Previously completed validation for feature commit
2e2f194ccdab76831b2988a9b749f5d8c299d8f7: 9 realtime integration tests and 20 core tests passed (29 total). These tests were not rerun when publishing this PR.The four new integration tests are:
RealtimeWriteInteTest.TestPkPartialUpdateDeleteAcrossCommitsRealtimeWriteInteTest.TestPkPartialUpdateSequenceGroupsRealtimeWriteInteTest.TestPkPartialUpdateAggregationSpillAndRecoveryRealtimeWriteInteTest.TestPkPartialUpdatePartitionedDefaultAggregationRetractThe other five integration tests cover existing primary-key reads, recovery, writer handoff, and multiple batches. Core coverage includes
KeyValueFileStoreWriteTest.*andFileStoreWriteTest.*.Coverage exercises real memory queries, prepare/commit and normal disk reads, spill, reopen, and replay. This is focused validation, not exhaustive partial-update semantics, full CI, Java interoperability, or all repository tests.
API and Format
No public
include/paimon/API, storage format, or protocol changes.Documentation
Updated the realtime user guide to document partial-update support and its scope.
Generative AI tooling
Generated-by: OpenAI Codex (CLI 0.160.0)