Repository navigation
[spark] Stop accepting variant extraction pushdown on Spark 4.1 and 4.2 - #11292
Open
LuciferYang wants to merge 1 commit into
Open
LuciferYang wants to merge 1 commit into
LuciferYang wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Since #7913, the Spark 4.1 connector accepts
variant_getextractions throughSupportsPushDownVariantExtractions. On Spark 4.1 and 4.2, Spark then builds the scan without column pruning and without its metadata columns (SPARK-57499 fixes this in 4.3.0).spark.sql.variant.pushVariantIntoScanis on by default, so ordinary queries break:SELECT variant_get(v1, '$.a', 'string') FROM tfails at planning withUnresolvedException.INTERNAL_ERROR_ATTRIBUTE_NOT_FOUND, and so do UPDATE and DELETE with a variant condition under the default v2 write.4.2 never had the trait. This change removes it from 4.1 too, so Spark evaluates
variant_geton the full value on every version, and shredded tables lose the leaf-only read on 4.1.VariantPushDownUtilsis kept for enabling pushdown on 4.3, where a scan must accept all extractions in a call or none.Tests
The 4.2 module now depends on the
paimon-spark-uttest jar and runs the four shared Variant suites, as 4.1 does. Spark 4.2 adds a field toDataSourceV2ScanRelation, so two 5-field pattern matches inVariantTestBasenow match the type instead.New
VariantTestBasecases run the queries above: an extraction next to a second variant column, which also checks that the scan reads only the extracted column; an extraction next to a metadata column; and UPDATE and DELETE with a variant condition, with and without deletion vectors. They fail on 4.1 with the removed trait put back.