Add ManageSnapshots.cherry_pick_snapshot - #3769
Conversation
Publishing a Write-Audit-Publish staged write means replaying a snapshot's changes onto the current table state. pyiceberg can stage the write on a branch and validate it there, but has no way to publish it, so the last step of the loop has to happen outside Python. Add cherry_pick_snapshot(), mirroring Java's CherryPickOperation: an append snapshot is replayed as a new snapshot on top of current carrying its added data files, recording source-snapshot-id and published-wap-id. A snapshot with another operation is fast-forwarded to when its parent is already current, and rejected otherwise. Picking an existing ancestor is a no-op, and a wap.id may only be published once. Dynamic overwrite is not covered; it raises rather than silently doing nothing. Signed-off-by: 1fanwang <1fannnw@gmail.com>
There was a problem hiding this comment.
Pull request overview
Adds a ManageSnapshots.cherry_pick_snapshot(snapshot_id) API to support Write-Audit-Publish (WAP) flows in PyIceberg by publishing a staged branch snapshot onto the current table state, mirroring Iceberg Java’s cherry-pick behavior.
Changes:
- Added
ManageSnapshots.cherry_pick_snapshot()implementation, including WAP summary properties (source-snapshot-id,published-wap-id) and duplicate WAP publish protection. - Added unit and integration tests covering replay vs fast-forward behavior, ancestor no-op, and error paths.
- Documented the WAP workflow in the public API docs.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
pyiceberg/table/update/snapshot.py |
Implements ManageSnapshots.cherry_pick_snapshot() plus helpers for replaying appended files and validating WAP publish semantics. |
tests/table/test_manage_snapshots.py |
Adds unit tests for replay, fast-forward, ancestor no-op, unknown snapshot, non-append rejection, and WAP id duplication. |
tests/integration/test_snapshot_operations.py |
Adds an integration test validating WAP publish behavior against REST/Hive catalogs. |
mkdocs/docs/api.md |
Adds a Write-Audit-Publish usage section for cherry_pick_snapshot. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| If the table has not changed since the branch was cut, the branch is fast-forwarded rather than | ||
| replayed. Picking a snapshot that is already an ancestor of the current state does nothing. Only | ||
| append snapshots can be replayed; anything else raises. |
There was a problem hiding this comment.
Good catch, fixed in 8abb5b5. That paragraph described an earlier draft where a pick was fast-forwarded whenever the table had not moved. The implementation follows Java: appends are always replayed, so the wap trail is recorded either way, and fast-forward is the fallback for a non-append whose parent is current.
The Write-Audit-Publish section described the earlier draft, where a pick was fast-forwarded when the table had not moved. Appends are always replayed, so the wap trail is recorded either way; fast-forward is the fallback for non-appends whose parent is current. Signed-off-by: 1fanwang <1fannnw@gmail.com>
Rationale for this change
pyiceberg can stage a write on a branch and validate it there, but cannot publish it. Write and Audit work; Publish does not, so the last step happens outside Python — typically a Spark session calling
system.cherrypick_snapshot.This adds
ManageSnapshots.cherry_pick_snapshot(), mirroring Java'sCherryPickOperation:source-snapshot-id, pluspublished-wap-idwhen staged with awap.id.wap.idmay only be published once.Not covered: dynamic overwrite (
OVERWRITEwithreplace-partitions), which Java replays. It raises here rather than silently doing nothing. Worth a follow-up.Prior art
#750 by @chinmay-bhat implemented this in 2024 and was closed by the stale bot with no review on merit; credit for getting there first is theirs. I rewrote rather than rebased, since
ManageSnapshotshas moved topyiceberg/table/update/snapshot.pyand the producer API changed underneath. This one is narrower: a single method, nopublish_changes(wap_id)and no producer class, sincefast_append()already does the work.@chinmay-bhat — glad to hand this back if you would rather carry it.
Touches the same class as #3649 and the same test file as #3760; whichever merges second needs a trivial additive resolution, keeping both sides. Happy to rebase.
Are these changes tested?
Integration, against the REST catalog and Hive metastore from
dev/docker-compose-integration.yml. It stages a branch write with awap.id, advancesmainso the pick is a replay not a fast-forward, publishes, then checks the summary properties and that a second publish is refused.Red, with
pyiceberg/table/update/snapshot.pyat upstream/mainGreen, with the change restored
The table goes
[1, 9]->[1, 2, 3, 9], the published snapshot carriessource-snapshot-idandpublished-wap-id=etl-001, and re-publishing raisesDuplicate request to cherry pick wap id that was published already: etl-001.Ran the docs example verbatim too.
Unit, across the
memory,sql, andsql_without_rowcountcatalogs: replay onto a moved-onmain, wap id recorded, duplicate publish rejected, append replayed when its parent is current, non-append fast-forwarded, ancestor no-op, and both rejection paths. Same red on unpatched source.Full unit and integration suites pass;
prek run -aclean.Are there any user-facing changes?
ManageSnapshots.cherry_pick_snapshot(snapshot_id), plus a Write-Audit-Publish section inmkdocs/docs/api.md.