Skip to content

Glue: Prune removed table properties from Glue Parameters - #4133

Open
Ankitraj-builds wants to merge 1 commit into
apache:mainfrom
Ankitraj-builds:fix/glue-prune-removed-properties
Open

Ankitraj-builds wants to merge 1 commit into
apache:mainfrom
Ankitraj-builds:fix/glue-prune-removed-properties

Conversation

@Ankitraj-builds

Copy link
Copy Markdown

Closes #3977

Rationale for this change

_construct_parameters in pyiceberg/catalog/glue.py starts from the existing Glue Parameters and only adds or overwrites keys. When a property is removed from the table, it stays in Glue forever, so Glue and the table properties drift apart.

This change passes the table's previous properties into _construct_parameters during commit_table. Any key that was in the previous properties but is not in the new ones is removed from Parameters. Glue keys that never came from Iceberg table properties are kept, which matches the Java Glue catalog behavior that @David-Banquet pointed out in the issue.

The new argument is optional with a default of None, so create_table, register_table and other callers are unchanged.

Are these changes tested?

Yes. Added test_commit_table_prunes_removed_properties in tests/catalog/test_glue.py. It creates a table with two properties, removes one, commits, and checks that the removed key is gone from Glue Parameters while the other one stays. The test fails on main and passes with this change. All 53 tests in tests/catalog/test_glue.py pass locally.

Are there any user-facing changes?

Yes. After this change, properties removed from a Glue-backed table are also removed from the Glue table Parameters on the next commit. Glue keys not managed by Iceberg are not touched.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Glue table Parameters are never pruned: removed table properties persist indefinitely

1 participant