Skip to content

[spark] Stop accepting variant extraction pushdown on Spark 4.1 and 4.2 - #11292

Open
LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:spark4-disable-variant-pushdown
Open

LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:spark4-disable-variant-pushdown

Conversation

@LuciferYang

Copy link
Copy Markdown
Contributor

Purpose

Since #7913, the Spark 4.1 connector accepts variant_get extractions through SupportsPushDownVariantExtractions. On Spark 4.1 and 4.2, Spark then builds the scan without column pruning and without its metadata columns (SPARK-57499 fixes this in 4.3.0). spark.sql.variant.pushVariantIntoScan is on by default, so ordinary queries break:

  • With two variant columns, SELECT variant_get(v1, '$.a', 'string') FROM t fails at planning with UnresolvedException.
  • A metadata column next to an extraction fails with INTERNAL_ERROR_ATTRIBUTE_NOT_FOUND, and so do UPDATE and DELETE with a variant condition under the default v2 write.
  • The scan reads every column.

4.2 never had the trait. This change removes it from 4.1 too, so Spark evaluates variant_get on the full value on every version, and shredded tables lose the leaf-only read on 4.1. VariantPushDownUtils is kept for enabling pushdown on 4.3, where a scan must accept all extractions in a call or none.

Tests

The 4.2 module now depends on the paimon-spark-ut test jar and runs the four shared Variant suites, as 4.1 does. Spark 4.2 adds a field to DataSourceV2ScanRelation, so two 5-field pattern matches in VariantTestBase now match the type instead.

New VariantTestBase cases run the queries above: an extraction next to a second variant column, which also checks that the scan reads only the extracted column; an extraction next to a metadata column; and UPDATE and DELETE with a variant condition, with and without deletion vectors. They fail on 4.1 with the removed trait put back.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant