Repository navigation
test: cover a native Iceberg read of a data file that omits an optional nested field - #7359
yunchipang wants to merge 2 commits into
Conversation
github-actions
Bot
added
enhancement
sunchao
left a comment
sunchao
left a comment
There was a problem hiding this comment.
The new regression passed on Spark 4.1.3 with Iceberg 1.11. I inspected the Parquet footers produced by the same fixture and confirmed that the narrow files omit both nested leaves. One assertion would preserve that coverage as the writer changes.
| SELECT 3, CAST(NULL AS STRUCT<a: INT>), CAST(NULL AS ARRAY<STRUCT<x: INT>>) | ||
| """) | ||
| .writeTo(table) | ||
| .append() |
There was a problem hiding this comment.
Could the test check that at least one appended Parquet file lacks both s.b and the list element’s y leaf? If a writer upgrade materializes them as all-null columns, both Spark comparisons and the native-plan assertion will still pass, and the test will stop exercising #7033. This suite already reads file_path from the Iceberg files table and opens its footer with ParquetFileReader.
There was a problem hiding this comment.
thanks! added a footer check.
Which issue does this PR close?
Closes #7033.
Rationale for this change
The native Iceberg scan failed with
Incorrect number of arrays for StructArray fields, expected 2 got 1when a data file omitted an optional nested field without a schema change. The iceberg-rust bump in #7028 fixed this, since iceberg-rust now matches nested fields by field id and fills in missing ones with NULLs. Nothing tested this case.What changes are included in this PR?
A test in
CometIcebergNativeSuitethat writes a data file withouts.bandl.element.y, usingwrite.spark.accept-any-schema. It then checks that the native scan matches Spark, with nested schema pruning on and off.How are these changes tested?
The new test passes on main. On the parent of #7028, it fails with the error from the issue.