Skip to content

fix: arrow schema to schema supports uuids - #2930

Open
JosephLenton wants to merge 7 commits into
apache:mainfrom
JosephLenton:fix-arrow-schema-to-schema-supports-uuids
Open

fix: arrow schema to schema supports uuids#2930
JosephLenton wants to merge 7 commits into
apache:mainfrom
JosephLenton:fix-arrow-schema-to-schema-supports-uuids

Conversation

@JosephLenton

@JosephLenton JosephLenton commented Jul 30, 2026

Copy link
Copy Markdown

Which issue does this PR close?

  • This adds UUID support to arrow_schema_to_schema and arrow_schema_to_schema_auto_assign_ids.
  • When they see a field with the extension ARROW:extension:metadata=arrow.uuid, then the field is converted to an Iceberg UUID type.

What changes are included in this PR?

  • The visitor used for converting from Arrow to Iceberg has a UUID section added. This is similar to how it handles variant.
  • UUID support is added to the ToArrowSchemaConverter as well for converting from iceberg to arrow.

Are these changes tested?

  • A new test is added to capture UUID conversion on its own.
  • UUID is added to large arrow_schema_to_schema_auto_assign_ids test.
  • UUID tests are added for the Iceberg to arrow conversions.
  • Tests are added that do a round trip from arrow to iceberg to arrow, and iceberg to arrow to iceberg.

AI Disclosure

AI was used to apply the fix to the visitor, and to add a UUID case to the test test_arrow_schema_to_schema_with_field_id. I built a failing test first and reviewed it's code to ensure it's output worked.

The test_arrow_schema_to_schema_should_convert_uuids test, commit message, and the bug report, were all written without any AI at all.

@JosephLenton
JosephLenton force-pushed the fix-arrow-schema-to-schema-supports-uuids branch from 30d1bc2 to 2fd4148 Compare July 31, 2026 12:57
@JosephLenton
JosephLenton marked this pull request as ready for review July 31, 2026 15:10
@mbutrovich
mbutrovich self-requested a review August 6, 2026 14:34

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First pass, thanks for looking at this @JosephLenton!

Comment thread crates/iceberg/src/arrow/schema.rs Outdated
Comment thread crates/iceberg/src/arrow/schema.rs Outdated
@JosephLenton
JosephLenton force-pushed the fix-arrow-schema-to-schema-supports-uuids branch from 2fd4148 to dc6f224 Compare August 6, 2026 19:52
@JosephLenton
JosephLenton requested a review from mbutrovich August 6, 2026 19:56
@JosephLenton

Copy link
Copy Markdown
Author

Thank you for the review @mbutrovich . You should find all of your comments addressed. I made all of the changes you suggested, and brought in UUID support on ToArrowSchemaConverter.

When you have time if I could get another review please, that would be grand!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

arrow_schema_to_schema fails to convert Arrow fields with UUID extension to Iceberg UUID

2 participants