Skip to content

feat(core): support partial updates in PK realtime writes - #525

Merged
lxy-9602 merged 9 commits into
apache:mainfrom
HaHaJeff:codex/pk-realtime-partial-update
Nov 10, 2026
Merged

lxy-9602 merged 9 commits into
apache:mainfrom
HaHaJeff:codex/pk-realtime-partial-update

Conversation

@HaHaJeff

@HaHaJeff HaHaJeff commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Enable partial-update for primary-key realtime writes by reusing the existing merge functions and options used by the ordinary Writer.

Feature tracking issue: #524

  • Allow PARTIAL_UPDATE alongside DEDUPLICATE, preserving the existing deduplicate ignore-delete restrictions. Standalone aggregation and first-row engines remain unsupported.
    Realtime partial-update currently requires deletion vectors to be disabled (deletion-vectors.enabled=false).
  • Reuse framework merge logic for prepared readers, sequence groups, field aggregation, and configured delete handling. The Store retains sorted batches and handles storage and spilling; sorting and merging remain the framework's responsibility.
  • Pass full table primary keys to the merge factory; retain trimmed keys for physical keys and sorting.
  • Keep realtime Noop compaction unchanged.
  • Preserve non-realtime merge/read behavior; the merge-factory change is limited to the realtime creation call.

Known Limitations

This PR reuses the ordinary Writer's existing partial-update merge semantics and inherits their merge-boundary limitations. It does not fix delete/reinsert old-field resurrection or stage-sensitive sequence-group/field-aggregation behavior. Existing C++ ordinary Writer observations and lxy's Java reports use different triggering boundaries: C++ delete/reinsert is in one batch/commit, while the Java report becomes incorrect after partial compaction of separately committed delete/reinsert files; C++ Case 2 writes both later batches before one commit, while Java commits them separately. Both Case 2 reports produce total=47, last="middle"; "thirty" is a continuous-merge reference, and the intended staged-aggregation guarantee needs clarification. See #524 for the full reproductions, original results, and attributed Java reports. Realtime may inherit these limitations; this feature does not establish correctness for all partial-update combinations.

Tests

Previously completed validation for feature commit 2e2f194ccdab76831b2988a9b749f5d8c299d8f7: 9 realtime integration tests and 20 core tests passed (29 total). These tests were not rerun when publishing this PR.

The four new integration tests are:

  • RealtimeWriteInteTest.TestPkPartialUpdateDeleteAcrossCommits
  • RealtimeWriteInteTest.TestPkPartialUpdateSequenceGroups
  • RealtimeWriteInteTest.TestPkPartialUpdateAggregationSpillAndRecovery
  • RealtimeWriteInteTest.TestPkPartialUpdatePartitionedDefaultAggregationRetract

The other five integration tests cover existing primary-key reads, recovery, writer handoff, and multiple batches. Core coverage includes KeyValueFileStoreWriteTest.* and FileStoreWriteTest.*.

Coverage exercises real memory queries, prepare/commit and normal disk reads, spill, reopen, and replay. This is focused validation, not exhaustive partial-update semantics, full CI, Java interoperability, or all repository tests.

API and Format

No public include/paimon/ API, storage format, or protocol changes.

Documentation

Updated the realtime user guide to document partial-update support and its scope.

Generative AI tooling

Generated-by: OpenAI Codex (CLI 0.160.0)

Comment thread test/inte/realtime_write_inte_test.cpp Resolved
@lxy-9602

lxy-9602 commented Nov 8, 2026

Copy link
Copy Markdown
Member

The code LGTM. I also reproduced a similar issue in Java partial-update with partial-update.remove-record-on-delete=true and DVs disabled:

1. Commit (id=1, a=10, b=20) and compact it into a high-level file.
2. Commit DELETE(id=1).
3. Commit UPDATE_AFTER(id=1, a=30, b=null).

Reading at this point correctly returns (id=1, a=30, b=null).

4. Run a partial compaction that merges only the two newer L0 files,
   leaving the older high-level file untouched.

Reading now incorrectly returns (id=1, a=30, b=20).
Reopening the table gives the same incorrect result.

The intermediate merge loses the delete boundary, allowing the old b=20 to reappear during subsequent merging.

Given that Java also has this issue, perhaps we could adjust the issue and PR descriptions to clarify that it is a shared defect in the existing merge semantics. A reference to #524 may be sufficient, without presenting it as an additional correctness boundary for this realtime feature.

@lxy-9602

lxy-9602 commented Nov 9, 2026

Copy link
Copy Markdown
Member

@HaHaJeff Hi, could you please have this issue’s classification and scope reviewed by human. This needs a deliberate assessment of the behavior and the feature’s intended scope.

I’m not sure a separate bug report is the most useful framing for this work. The delete-boundary and field-version problems appear to be existing merge-boundary limitations shared with Java, rather than something introduced by realtime support.

For example, the issue says “it does not establish Java reproduction of Case 2.” However, committing Case 2’s two batches separately in Java also produces total=47, last="middle" instead of the expected "thirty". The precise triggering boundary differs, but the underlying limitation appears similar.

My expectation was a feature issue for supporting realtime writes with partial-update, with the known merge-boundary limitations documented alongside the comparable Java behavior.

@HaHaJeff

HaHaJeff commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

@lxy-9602 Would it be enough to document the feature scope and known merge-boundary limitations in #525, and then close #524? That would avoid maintaining the same information in both places.

Context: original comment.

@HaHaJeff

HaHaJeff commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

@lxy-9602 I’ve reframed #524 as a feature issue for realtime partial-update, keeping the original reproductions and adding the known merge-boundary limitations alongside your Java observations. The C++ and Java commit/compaction boundaries are documented separately. I’ve also updated #525’s description to match, making clear that this PR reuses the ordinary Writer’s semantics without changing those existing merge behaviors.

Context: original comment.

lxy-9602
lxy-9602 previously approved these changes Nov 9, 2026

@lxy-9602 lxy-9602 left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. Thank you for your update! Could you please resolve the conflicts so CI can run.

Retain both Arrow utility headers required by the partial-update tests and the upstream refresh regression test.

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lxy-9602
lxy-9602 merged commit 0c91f02 into apache:main Nov 10, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants