Skip to content

[core] Prevent branch commits from overwriting Iceberg metadata - #9476

Open
ArnavBalyan wants to merge 2 commits into
apache:masterfrom
ArnavBalyan:arnavb/fix-icbrg
Open

[core] Prevent branch commits from overwriting Iceberg metadata#9476
ArnavBalyan wants to merge 2 commits into
apache:masterfrom
ArnavBalyan:arnavb/fix-icbrg

Conversation

@ArnavBalyan

Copy link
Copy Markdown
Member

Purpose

  • Paimon runs Iceberg callbacks for commits on every branch.
  • Non main branch operations use the main table’s Iceberg metadata location, causing main metadata and the external catalog pointer to be overwritten.
  • Run Iceberg validation, commit and tag callbacks only for the main branch.

Tests

  • UT

@ArnavBalyan

Copy link
Copy Markdown
Member Author

cc @JingsongLi thanks!

private boolean icebergCompatibilityEnabled(FileStoreTable table) {
return options.toConfiguration().get(IcebergOptions.METADATA_ICEBERG_STORAGE)
!= IcebergOptions.StorageType.DISABLED
&& BranchManager.isMainBranch(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Refresh Iceberg metadata when a branch is fast-forwarded to main

Disabling these callbacks on non-main branches prevents the premature overwrite, but it also leaves the normal promotion path without any Iceberg refresh. FileSystemBranchManager.fastForward only copies the branch snapshot/schema/tag files into the main directories and invalidates the Paimon snapshot cache; it never runs a main-branch IcebergCommitCallback. Therefore, after table.fastForward("b1"), Paimon main points at the promoted snapshot while the external Iceberg catalog and version-hint.text still point at the old main snapshot indefinitely. The new test stops before promotion, so it does not expose this stale-reader state.

Please make the fast-forward path rebuild/commit Iceberg metadata for the newly promoted main snapshot (while keeping branch commits callback-free), and add a regression that verifies the external pointer advances only after fast-forward.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants