[core] Prevent branch commits from overwriting Iceberg metadata - #9476
[core] Prevent branch commits from overwriting Iceberg metadata#9476ArnavBalyan wants to merge 2 commits into
Conversation
|
cc @JingsongLi thanks! |
| private boolean icebergCompatibilityEnabled(FileStoreTable table) { | ||
| return options.toConfiguration().get(IcebergOptions.METADATA_ICEBERG_STORAGE) | ||
| != IcebergOptions.StorageType.DISABLED | ||
| && BranchManager.isMainBranch( |
There was a problem hiding this comment.
[P1] Refresh Iceberg metadata when a branch is fast-forwarded to main
Disabling these callbacks on non-main branches prevents the premature overwrite, but it also leaves the normal promotion path without any Iceberg refresh. FileSystemBranchManager.fastForward only copies the branch snapshot/schema/tag files into the main directories and invalidates the Paimon snapshot cache; it never runs a main-branch IcebergCommitCallback. Therefore, after table.fastForward("b1"), Paimon main points at the promoted snapshot while the external Iceberg catalog and version-hint.text still point at the old main snapshot indefinitely. The new test stops before promotion, so it does not expose this stale-reader state.
Please make the fast-forward path rebuild/commit Iceberg metadata for the newly promoted main snapshot (while keeping branch commits callback-free), and add a regression that verifies the external pointer advances only after fast-forward.
Purpose
Tests