Describe the bug
is_pure_structural_narrowing (native/core/src/parquet/schema_adapter.rs) decides whether Comet can leave DataFusion's CastExpr in place so apache/datafusion#24090's nested leaf pruning fires. It requires an exact, case-sensitive name match for every target field:
source_fields
.iter()
.find(|f| f.name() == target_field.name())
.is_some_and(|source_field| ...)An exact match is not the same as an unambiguous one. With spark.sql.caseSensitive=false, a Parquet column s: struct<ID: bigint, id: bigint> read through the explicit schema s: struct<id: bigint> passes this check, so the generic cast is retained and returns id. Spark, and Comet's own spark_parquet_convert, instead reject the ambiguity.
So a query Spark errors on can now silently return one of the two candidate fields.
Steps to reproduce
Read a Parquet file whose struct has two fields differing only in case, requesting one of them by its exact name, with spark.sql.caseSensitive=false.
Expected behavior
Uniqueness is required under Spark's configured name resolver at each nested struct level before the cast is retained; otherwise fall back to CometCastColumnExpr and let the existing duplicate-field check fire.
Additional context
The existing CAFÉ/café native-reader regression asks for Café, so it does not cover the case where one sibling matches exactly. A regression requesting café (or id) exactly should be added alongside the fix.
Raised during review of #5262; @comphead agreed it is a follow-up rather than a blocker.
Describe the bug
is_pure_structural_narrowing(native/core/src/parquet/schema_adapter.rs) decides whether Comet can leave DataFusion'sCastExprin place so apache/datafusion#24090's nested leaf pruning fires. It requires an exact, case-sensitive name match for every target field:source_fields .iter() .find(|f| f.name() == target_field.name()) .is_some_and(|source_field| ...)An exact match is not the same as an unambiguous one. With
spark.sql.caseSensitive=false, a Parquet columns: struct<ID: bigint, id: bigint>read through the explicit schemas: struct<id: bigint>passes this check, so the generic cast is retained and returnsid. Spark, and Comet's ownspark_parquet_convert, instead reject the ambiguity.So a query Spark errors on can now silently return one of the two candidate fields.
Steps to reproduce
Read a Parquet file whose struct has two fields differing only in case, requesting one of them by its exact name, with
spark.sql.caseSensitive=false.Expected behavior
Uniqueness is required under Spark's configured name resolver at each nested struct level before the cast is retained; otherwise fall back to
CometCastColumnExprand let the existing duplicate-field check fire.Additional context
The existing
CAFÉ/cafénative-reader regression asks forCafé, so it does not cover the case where one sibling matches exactly. A regression requestingcafé(orid) exactly should be added alongside the fix.Raised during review of #5262; @comphead agreed it is a follow-up rather than a blocker.