Skip to content

[spark] Add an isolated Spark 4.3 connector build - #11246

Open
LuciferYang wants to merge 4 commits into
apache:masterfrom
LuciferYang:spark43-connector
Open

LuciferYang wants to merge 4 commits into
apache:masterfrom
LuciferYang:spark43-connector

Conversation

@LuciferYang

@LuciferYang LuciferYang commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Add an experimental paimon-spark-4.3_2.13 connector through a separate spark43 profile, as #11127 did for Spark 4.2. The spark4 and spark42 connectors are unchanged.

Spark 4.3 keeps the 4.2 APIs Paimon adapts to, so the five SPARK_VERSION.startsWith("4.2.") gates now read >= "4.2". The rest covers what 4.3 changed:

  • SpecializedGetters gains two nanosecond timestamp getters and ParserInterface gains splitStatements. Their types do not exist in 4.2, so paimon-spark4-common gets 4.3 copies of the row, array and parser classes under src/main/spark43. No Paimon type maps to nanosecond timestamps, so the getters throw.
  • GeneratedColumn.validateGeneratedColumns is gone. REPLACE TABLE calls it or 4.3's validateCatalogForGeneratedColumn, whichever the runtime has, as Spark does.

paimon-spark-4.3 compiles the main and test sources of paimon-spark-4.2, Ossie semantic views included, against Spark 4.3. It adds only its NOTICE and the tests that need 4.3 APIs. Like 4.2, it gets a CI lane and is not published.

Tests

paimon-spark-4.3 runs every paimon-spark-4.2 test on Spark 4.3. Spark42CompatibilityTest gains two cases, so both connectors cover them:

  • a column-list CREATE OR REPLACE TABLE, which goes through the generated-column check above;
  • CREATE OR REPLACE TABLE ... AS SELECT, which takes the separate RTAS strategy.

Spark43CompatibilityTest calls splitStatements, a parser API that 4.2 does not have, through Paimon's parser.

Spark42CompatibilityTest and OssieSemanticViewTest set spark.sql.analyzer.singlePassResolver.dualRunWithLegacy=false. Spark 4.3 (SPARK-56596) turns that dual run on by default in tests, and the single-pass resolver skips injected resolution rules, so it cannot resolve Paimon views. Spark 4.2 already defaults it to false.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Spark 4.3 build has end-to-end value. The 41 existing local Spark 4.3 compatibility, Ossie, model and variant tests passed on JDK 17, along with 26 CI planning/command checks. An additional SQL regression on the merge with current master exposed a replacement behavior that can remove existing data on a statement Spark would normally reject.

val columns = schemaOrColumns.asInstanceOf[Seq[ColumnDefinition]]
ResolveDefaultColumns.validateCatalogForDefaultValue(columns, catalog, ident)
GeneratedColumn.validateGeneratedColumns(tableSchema, catalog, ident, statementType)
validateGeneratedColumns(tableSchema, columns, catalog, ident, statementType)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Reject unsupported CHECK constraints before replacing the table

Spark 4.3's native DataSourceV2Strategy validates TableSpec.constraints against catalog capabilities before planning a replacement. Paimon's custom PaimonReplaceTableStrategy takes this path and checks generated, default and identity columns only. On the merged tree with Spark 4.3.0 I created a Paimon REST table t(id INT) and inserted 7. Plain CREATE TABLE other (id INT, CONSTRAINT positive CHECK (id > 0)) failed with UNSUPPORTED_FEATURE.TABLE_OPERATION as expected. CREATE OR REPLACE TABLE t (id INT, CONSTRAINT positive CHECK (id > 0)) succeeded and SELECT * FROM t was empty. Please run Spark's table-constraint capability check before creating the replacement exec (or reject constraints explicitly) and add a case asserting that the old row remains. Spark 4.3.0 native check

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed. This is not specific to 4.3: on master, Spark 4.1 and 4.2 plan a column-list REPLACE TABLE through the same PaimonReplaceTableStrategy and ignore the constraint the same way. I opened #11267 to fix it there, with your repro as a test on 4.1 and 4.2. Once it is merged I'll rebase this PR and add the same case to Spark43CompatibilityTest.

@LuciferYang

Copy link
Copy Markdown
Contributor Author

could you take another look If you have time? @JingsongLi Thanks ~

</executions>
</plugin>
</plugins>
</build>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All 4 production sources (OssieModel.scala, OssieSql.scala, OssieToSparkMetricView.scala, PaimonSemanticViewResolver.scala), ossie-0.3.0.dev0.schema.json, Spark4xRowTest.scala, and OssieModelTest.scala in paimon-spark-4.3 are byte-for-byte identical copies of paimon-spark-4.2, and OssieSemanticViewTest.scala / Spark43CompatibilityTest.scala only differ by spark.sql.analyzer.singlePassResolver.dualRunWithLegacy=false (plus two 4.3-specific tests).

As seen in commit b66bcaab, duplicating ~2,265 lines across both modules means every change to Ossie semantic views or compatibility tests in paimon-spark-4.2 (like #11237, #11238, and #11267) has to be manually copy-pasted into paimon-spark-4.3.

Could we share the sources/resources/tests from ../paimon-spark-4.2 via build-helper-maven-plugin (add-source / add-test-source) and <resources> (similar to paimon-spark-ut-4.0/pom.xml, or via a shared source directory in paimon-spark4-common), and keep only the Spark 4.3-specific test additions in paimon-spark-4.3?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After reconsidering, I think this is a good suggestion. Thank you @Akash3121

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 28a7ef8. paimon-spark-4.3 no longer has its own copy: build-helper adds the main and test sources of paimon-spark-4.2, and <resources> takes its resources except META-INF/NOTICE. The 4.3 module keeps only its NOTICE and a Spark43CompatibilityTest for splitStatements, which needs a 4.3 API. The two REPLACE TABLE cases moved into Spark42CompatibilityTest, so both connectors run them. The dualRunWithLegacy=false setting moved into the 4.2 tests too; Spark 4.2 already defaults it to false.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants