Repository navigation
[spark] Add an isolated Spark 4.3 connector build - #11246
LuciferYang wants to merge 4 commits into
Conversation
JingsongLi
left a comment
There was a problem hiding this comment.
The Spark 4.3 build has end-to-end value. The 41 existing local Spark 4.3 compatibility, Ossie, model and variant tests passed on JDK 17, along with 26 CI planning/command checks. An additional SQL regression on the merge with current master exposed a replacement behavior that can remove existing data on a statement Spark would normally reject.
| val columns = schemaOrColumns.asInstanceOf[Seq[ColumnDefinition]] | ||
| ResolveDefaultColumns.validateCatalogForDefaultValue(columns, catalog, ident) | ||
| GeneratedColumn.validateGeneratedColumns(tableSchema, catalog, ident, statementType) | ||
| validateGeneratedColumns(tableSchema, columns, catalog, ident, statementType) |
There was a problem hiding this comment.
[P1] Reject unsupported CHECK constraints before replacing the table
Spark 4.3's native DataSourceV2Strategy validates TableSpec.constraints against catalog capabilities before planning a replacement. Paimon's custom PaimonReplaceTableStrategy takes this path and checks generated, default and identity columns only. On the merged tree with Spark 4.3.0 I created a Paimon REST table t(id INT) and inserted 7. Plain CREATE TABLE other (id INT, CONSTRAINT positive CHECK (id > 0)) failed with UNSUPPORTED_FEATURE.TABLE_OPERATION as expected. CREATE OR REPLACE TABLE t (id INT, CONSTRAINT positive CHECK (id > 0)) succeeded and SELECT * FROM t was empty. Please run Spark's table-constraint capability check before creating the replacement exec (or reject constraints explicitly) and add a case asserting that the old row remains. Spark 4.3.0 native check
There was a problem hiding this comment.
Confirmed. This is not specific to 4.3: on master, Spark 4.1 and 4.2 plan a column-list REPLACE TABLE through the same PaimonReplaceTableStrategy and ignore the constraint the same way. I opened #11267 to fix it there, with your repro as a test on 4.1 and 4.2. Once it is merged I'll rebase this PR and add the same case to Spark43CompatibilityTest.
|
could you take another look If you have time? @JingsongLi Thanks ~ |
| </executions> | ||
| </plugin> | ||
| </plugins> | ||
| </build> |
There was a problem hiding this comment.
All 4 production sources (OssieModel.scala, OssieSql.scala, OssieToSparkMetricView.scala, PaimonSemanticViewResolver.scala), ossie-0.3.0.dev0.schema.json, Spark4xRowTest.scala, and OssieModelTest.scala in paimon-spark-4.3 are byte-for-byte identical copies of paimon-spark-4.2, and OssieSemanticViewTest.scala / Spark43CompatibilityTest.scala only differ by spark.sql.analyzer.singlePassResolver.dualRunWithLegacy=false (plus two 4.3-specific tests).
As seen in commit b66bcaab, duplicating ~2,265 lines across both modules means every change to Ossie semantic views or compatibility tests in paimon-spark-4.2 (like #11237, #11238, and #11267) has to be manually copy-pasted into paimon-spark-4.3.
Could we share the sources/resources/tests from ../paimon-spark-4.2 via build-helper-maven-plugin (add-source / add-test-source) and <resources> (similar to paimon-spark-ut-4.0/pom.xml, or via a shared source directory in paimon-spark4-common), and keep only the Spark 4.3-specific test additions in paimon-spark-4.3?
There was a problem hiding this comment.
After reconsidering, I think this is a good suggestion. Thank you @Akash3121
There was a problem hiding this comment.
Done in 28a7ef8. paimon-spark-4.3 no longer has its own copy: build-helper adds the main and test sources of paimon-spark-4.2, and <resources> takes its resources except META-INF/NOTICE. The 4.3 module keeps only its NOTICE and a Spark43CompatibilityTest for splitStatements, which needs a 4.3 API. The two REPLACE TABLE cases moved into Spark42CompatibilityTest, so both connectors run them. The dualRunWithLegacy=false setting moved into the 4.2 tests too; Spark 4.2 already defaults it to false.
Purpose
Add an experimental
paimon-spark-4.3_2.13connector through a separatespark43profile, as #11127 did for Spark 4.2. Thespark4andspark42connectors are unchanged.Spark 4.3 keeps the 4.2 APIs Paimon adapts to, so the five
SPARK_VERSION.startsWith("4.2.")gates now read>= "4.2". The rest covers what 4.3 changed:SpecializedGettersgains two nanosecond timestamp getters andParserInterfacegainssplitStatements. Their types do not exist in 4.2, sopaimon-spark4-commongets 4.3 copies of the row, array and parser classes undersrc/main/spark43. No Paimon type maps to nanosecond timestamps, so the getters throw.GeneratedColumn.validateGeneratedColumnsis gone. REPLACE TABLE calls it or 4.3'svalidateCatalogForGeneratedColumn, whichever the runtime has, as Spark does.paimon-spark-4.3compiles the main and test sources ofpaimon-spark-4.2, Ossie semantic views included, against Spark 4.3. It adds only its NOTICE and the tests that need 4.3 APIs. Like 4.2, it gets a CI lane and is not published.Tests
paimon-spark-4.3runs everypaimon-spark-4.2test on Spark 4.3.Spark42CompatibilityTestgains two cases, so both connectors cover them:CREATE OR REPLACE TABLE, which goes through the generated-column check above;CREATE OR REPLACE TABLE ... AS SELECT, which takes the separate RTAS strategy.Spark43CompatibilityTestcallssplitStatements, a parser API that 4.2 does not have, through Paimon's parser.Spark42CompatibilityTestandOssieSemanticViewTestsetspark.sql.analyzer.singlePassResolver.dualRunWithLegacy=false. Spark 4.3 (SPARK-56596) turns that dual run on by default in tests, and the single-pass resolver skips injected resolution rules, so it cannot resolve Paimon views. Spark 4.2 already defaults it tofalse.