Skip to content

Support pinned-base assertions and staged (not-yet-submitted) commits - #83

Open
gbrgr wants to merge 2 commits into
mainfrom
gb/base-snapshot-assertion-and-stage-commit
Open

gbrgr wants to merge 2 commits into
mainfrom
gb/base-snapshot-assertion-and-stage-commit

Conversation

@gbrgr

@gbrgr gbrgr commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

Two related additions to Transaction, needed for a caller (raicode) that must assert a commit against an externally-pinned base snapshot -- possibly one that no longer matches whatever the in-memory Table currently reports -- and, separately, that wants to compute a commit's requirements/updates without submitting them to a catalog at all (e.g. to submit later as one element of a multi-table transactions/commit).

  • OverwriteAction::assert_requirements(requirements) -- supply arbitrary TableRequirements alongside the ones this action derives automatically (currently UuidMatch and a RefSnapshotIdMatch for the target branch). A supplied requirement of an auto-derived kind replaces the derived one; any other kind is added alongside; repeated calls accumulate.

  • Transaction::stage_commit() -- runs every action's commit logic (including storage side effects, e.g. writing manifest files) and returns a new StagedCommit { identifier, requirements, updates } value instead of submitting anything to a catalog. StagedCommit serializes to exactly the JSON body the Iceberg REST catalog protocol expects for a table commit, or for one element of a multi-table transactions/commit call.

Why the ActionCommit change

Transaction::apply's existing local requirement check (requirement.check(Some(table.metadata()))?) validates every requirement against the in-memory Table it's being applied to. For auto-derived requirements this is a tautology (they're read straight off that same table), but for a caller-supplied override it's actively wrong -- the whole reason to override is to assert something that may not match what's currently loaded (e.g. a retry that must keep asserting an old base even though a fresher load of the table has since moved past it, so the catalog -- not this local check -- can correctly reject it as a genuine conflict). ActionCommit now carries caller-supplied requirements separately via with_unchecked_requirements/take_unchecked_requirements, so only auto-derived requirements go through the local check; both still end up in the final commit.

Test plan

  • cargo test -p iceberg --lib -- 1798 passed, 0 failed
  • cargo fmt -p iceberg -- --check
  • cargo clippy -p iceberg --lib --all-targets -- clean
  • New unit tests cover: assert_requirements overriding an auto-derived requirement, adding a new requirement kind, accumulating across multiple calls; stage_commit never touching the catalog, its output being independently submittable as a valid commit via TableCommit::builder() + catalog.update_table(), respecting an assert_requirements override end-to-end, and serializing to the exact REST catalog JSON shape.

🤖 Generated with Claude Code

gbrgr and others added 2 commits September 17, 2026 10:31
OverwriteAction::assert_requirements lets a caller supply arbitrary
TableRequirements alongside the ones an action derives automatically:
a supplied requirement of an auto-derived kind (UuidMatch,
RefSnapshotIdMatch for the target ref) replaces the derived one, any
other kind is added, and repeated calls accumulate. This makes it
possible to assert a pinned base snapshot id that differs from
whatever the in-memory Table currently reports -- needed so a retried
commit can keep asserting the same base every attempt rather than
silently rebasing onto whatever is current.

Transaction::stage_commit runs every action's commit logic (including
its storage side effects, e.g. writing manifest files) and returns the
resulting identifier/requirements/updates as a new StagedCommit value,
without ever touching the catalog. StagedCommit serializes to exactly
the JSON body the Iceberg REST catalog protocol expects for a table or
transaction commit, so it can be submitted later, elsewhere, including
as one element of a multi-table transactions/commit call.

Since Transaction::apply's local requirement-check is a sanity check
against the in-memory Table's own metadata, it's a tautology for
auto-derived requirements but actively wrong for caller-supplied ones
that are meant to be validated by the catalog instead (they may
deliberately diverge from what's currently loaded). ActionCommit now
carries those separately via with_unchecked_requirements/
take_unchecked_requirements so only auto-derived requirements go
through the local check; both still end up in the final commit.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CI's check-public-api job flags any undeclared public API surface
change; this snapshot was missed when the feature landed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant