Skip to content

[format] Follow Parquet list backward-compatibility rules in schema clipping - #9550

Open
juntaozhang wants to merge 1 commit into
apache:masterfrom
juntaozhang:pr-list-backward-compatibility
Open

[format] Follow Parquet list backward-compatibility rules in schema clipping#9550
juntaozhang wants to merge 1 commit into
apache:masterfrom
juntaozhang:pr-list-backward-compatibility

Conversation

@juntaozhang

Copy link
Copy Markdown
Contributor

Purpose

Follow Parquet's backward-compatibility rules for the LIST logical type when clipping Parquet schemas, so that nested variant column pruning and schema conversion correctly distinguish wrapper groups from element types across all standard and legacy list encodings:

  • Rule 1: repeated primitive field is the element type.
  • Rule 2: repeated group with multiple fields is the element type.
  • Rule 3: repeated group whose single child is also repeated is the element type.
  • Rule 4: repeated group named array or <list>_tuple is the element type.
  • Rule 5: any other repeated group with exactly one non-repeated child is a wrapper; its child is the element type.

Previously the list-structure check did not fully apply these rules, which could misidentify legacy two-level encodings as three-level lists and return the wrong element type during nested variant pruning.

Tests

mvn -pl paimon-format -DwildcardSuites=none -Dtest=ParquetSchemaConverterTest test

@juntaozhang juntaozhang closed this Sep 2, 2026
@juntaozhang juntaozhang reopened this Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant