Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 10 additions & 8 deletions docs/docs/concepts/rest/semantic-views.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ preserves the original document text. Servers validate the model formats, syntax
and features they support; consumers provide query execution.

Registration alone does not grant access to source data or guarantee engine support.
The Spark 4.2 connector supports the [Ossie query profile](#spark-42-ossie-queries) below.
The Spark 4.2 and 4.3 connectors support the [Ossie query profile](#spark-42-ossie-queries) below.
This implementation supplies Java clients, wire models, and the
[Catalog OpenAPI contract](/rest-catalog-open-api.yaml). REST server providers must implement
persistence, validation, authorization, and the atomic operations described below.
Expand Down Expand Up @@ -168,12 +168,13 @@ Client-side invalid input throws `IllegalArgumentException` before HTTP. An unsu
never falls back to SQL views, labels, or table options. Materialization, rename, and per-measure
or per-field REST resources are outside this version.

## Spark 4.2 Ossie queries
## Spark 4.2 and 4.3 Ossie queries {#spark-42-ossie-queries}

The `paimon-spark-4.2_2.13` connector can query an `ossie-yaml` definition through the configured
Paimon REST Catalog. Enable `PaimonSparkSessionExtensions` as described in the
[Spark installation guide](../../spark/installation.mdx). The REST server must implement semantic
view management; the connector does not provide production REST persistence or validation.
The `paimon-spark-4.2_2.13` and `paimon-spark-4.3_2.13` connectors can query an `ossie-yaml`
definition through the configured Paimon REST Catalog. Enable `PaimonSparkSessionExtensions` as
described in the [Spark installation guide](../../spark/installation.mdx). The REST server must
implement semantic view management; the connector does not provide production REST persistence or
validation.

Register a definition whose content is a flat Ossie document, for example:

Expand Down Expand Up @@ -237,8 +238,9 @@ register and list definitions. Ordinary tables and SQL views retain their existi
Source table authorization is required independently of permission to read the semantic definition.
When using a CTE, evaluate measures inside its definition (for example,
`WITH m AS (SELECT MEASURE(revenue) AS revenue FROM paimon.sales.order_metrics) SELECT revenue FROM m`).
Forwarding raw measure columns through a CTE or temporary view is not supported by Spark 4.2's
Metric View resolver; evaluate measures inside the defining query block before reusing its results.
Forwarding raw measure columns through a CTE or temporary view is not supported by the Metric View
resolver of Spark 4.2 and 4.3; evaluate measures inside the defining query block before reusing its
results.

The bundled schema is pinned to Ossie commit
[`2692b1cc4708fdcf6a91e597ca928a9bc9d0da60`](https://github.com/apache/ossie/tree/2692b1cc4708fdcf6a91e597ca928a9bc9d0da60).
Expand Down
14 changes: 9 additions & 5 deletions docs/docs/spark/installation.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,9 @@ may require a newer Java runtime. Check its runtime requirements before starting

- Spark 3.x (including 3.5, 3.4, 3.3, 3.2) : Pre-built with Java 8 and Scala 2.12/2.13

Spark 4.2 is available as an experimental source build with Java 17 and Scala 2.13.
Its independent `spark42` profile recompiles the shared connector modules against Spark 4.2;
the `spark4` profile continues to build the Spark 4.0 and 4.1 connectors.
Spark 4.2 and 4.3 are available as experimental source builds with Java 17 and Scala 2.13.
Their independent `spark42` and `spark43` profiles recompile the shared connector modules against
Spark 4.2 and 4.3; the `spark4` profile continues to build the Spark 4.0 and 4.1 connectors.

## Download the Connector

Expand Down Expand Up @@ -89,14 +89,18 @@ mvn clean package -DskipTests -pl paimon-spark/paimon-spark-4.1 -am -Pspark4

# build paimon spark 4.2
mvn clean package -DskipTests -pl paimon-spark/paimon-spark-4.2 -am -Pspark42

# build paimon spark 4.3
mvn clean package -DskipTests -pl paimon-spark/paimon-spark-4.3 -am -Pspark43
```

Use `clean` when switching between Spark profiles: the shared modules produce artifacts with
the same names, but Spark 4.2 changes internal binary interfaces such as `CatalogManager`.
the same names, but Spark 4.2 and 4.3 change internal binary interfaces such as `CatalogManager`
and `ParserInterface`.

For Spark 3.5, you can find the bundled jar in `./paimon-spark/paimon-spark-3.5/target/paimon-spark-3.5_2.12-@@VERSION@@.jar`.

The Spark 4.2 connector can also query [Ossie Semantic Views](../concepts/rest/semantic-views.md#spark-42-ossie-queries)
The Spark 4.2 and 4.3 connectors can also query [Ossie Semantic Views](../concepts/rest/semantic-views.md#spark-42-ossie-queries)
registered in a Paimon REST Catalog.

## Add the Connector to Spark
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,10 @@ import org.apache.spark.sql.test.SharedSparkSession
import java.nio.file.Files
import java.util.Collections

/** Exercise the APIs whose signatures changed in Spark 4.2 using real Paimon files. */
/**
* Exercise the APIs whose signatures changed in Spark 4.2 using real Paimon files. The Spark 4.3
* connector compiles and runs this suite too.
*/
class Spark42CompatibilityTest extends QueryTest with SharedSparkSession {
private val warehouse = Files.createTempDirectory("paimon-spark42-").toFile
private lazy val server = {
Expand All @@ -54,6 +57,8 @@ class Spark42CompatibilityTest extends QueryTest with SharedSparkSession {
.set("spark.sql.catalog.paimon.token", "test-token")
.set("spark.sql.defaultCatalog", "paimon")
.set("spark.sql.extensions", classOf[PaimonSparkSessionExtensions].getName)
// Spark 4.3 dual-runs the single-pass resolver in tests, and it cannot resolve Paimon views.
.set("spark.sql.analyzer.singlePassResolver.dualRunWithLegacy", "false")

override protected def beforeAll(): Unit = {
super.beforeAll()
Expand Down Expand Up @@ -86,6 +91,27 @@ class Spark42CompatibilityTest extends QueryTest with SharedSparkSession {
}
}

test("create or replace table as select") {
withTable("t") {
sql("CREATE TABLE t (id INT, label STRING) TBLPROPERTIES ('file.format'='avro')")
sql("INSERT INTO t VALUES (1, 'old')")
sql(
"CREATE OR REPLACE TABLE t TBLPROPERTIES ('file.format'='avro') " +
"AS SELECT 2 AS id, 'new' AS label")
checkAnswer(sql("SELECT * FROM t"), Seq(Row(2, "new")))
}
}

test("replace table with a column list") {
withTable("t") {
sql("CREATE TABLE t (id INT, label STRING) TBLPROPERTIES ('file.format'='avro')")
sql("INSERT INTO t VALUES (1, 'old')")
sql("CREATE OR REPLACE TABLE t (id INT, score DOUBLE) TBLPROPERTIES ('file.format'='avro')")
sql("INSERT INTO t VALUES (2, 0.5)")
checkAnswer(sql("SELECT * FROM t"), Seq(Row(2, 0.5)))
}
}

test("replace table rejects table constraints and keeps the old table") {
withTable("t") {
sql("CREATE TABLE t (id INT) TBLPROPERTIES ('file.format'='avro')")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,8 @@ class OssieSemanticViewTest extends QueryTest with SharedSparkSession {
.set("spark.sql.catalog.paimon.token", "test-token")
.set("spark.sql.catalog.paimon.cache-enabled", "false")
.set("spark.sql.extensions", classOf[PaimonSparkSessionExtensions].getName)
// Spark 4.3 dual-runs the single-pass resolver in tests, and it cannot resolve Paimon views.
.set("spark.sql.analyzer.singlePassResolver.dualRunWithLegacy", "false")

override protected def beforeAll(): Unit = {
super.beforeAll()
Expand Down
219 changes: 219 additions & 0 deletions paimon-spark/paimon-spark-4.3/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,219 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<parent>
<groupId>org.apache.paimon</groupId>
<artifactId>paimon-spark</artifactId>
<version>2.2-SNAPSHOT</version>
</parent>

<artifactId>paimon-spark-4.3_2.13</artifactId>
<name>Paimon : Spark : 4.3 : 2.13</name>

<!--
Compiles `paimon-spark-4.2`'s main and test sources against Spark 4.3, so both connectors share
one copy. This module owns only its NOTICE and the tests that need Spark 4.3 APIs.
-->
<properties>
<spark.version>4.3.0</spark.version>
<scala.binary.version>2.13</scala.binary.version>
<scala.version>2.13.18</scala.version>
<spark42.module.dir>${project.basedir}/../paimon-spark-4.2</spark42.module.dir>
</properties>

<dependencies>
<dependency>
<groupId>com.networknt</groupId>
<artifactId>json-schema-validator</artifactId>
<version>2.0.0</version>
<exclusions>
<exclusion>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</exclusion>
<exclusion>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-yaml</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.yaml</groupId>
<artifactId>snakeyaml</artifactId>
<version>2.5</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.paimon</groupId>
<artifactId>paimon-format</artifactId>
</dependency>
<dependency>
<groupId>org.apache.paimon</groupId>
<artifactId>paimon-spark4-common_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql-api_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.spark</groupId>
<artifactId>spark-connect-shims_${scala.binary.version}</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j2-impl</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-catalyst_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
<classifier>tests</classifier>
<scope>test</scope>
<exclusions>
<exclusion>
<groupId>org.apache.spark</groupId>
<artifactId>spark-connect-shims_${scala.binary.version}</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-catalyst_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
<classifier>tests</classifier>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
<classifier>tests</classifier>
<scope>test</scope>
</dependency>
</dependencies>

<build>
<resources>
<resource>
<directory>src/main/resources</directory>
</resource>
<resource>
<directory>${spark42.module.dir}/src/main/resources</directory>
<excludes>
<exclude>META-INF/NOTICE</exclude>
</excludes>
</resource>
</resources>

<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<executions>
<execution>
<id>add-spark42-sources</id>
<phase>generate-sources</phase>
<goals>
<goal>add-source</goal>
</goals>
<configuration>
<sources>
<source>${spark42.module.dir}/src/main/scala</source>
</sources>
</configuration>
</execution>
<execution>
<id>add-spark42-test-sources</id>
<phase>generate-test-sources</phase>
<goals>
<goal>add-test-source</goal>
</goals>
<configuration>
<sources>
<source>${spark42.module.dir}/src/test/scala</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
<id>shade-paimon</id>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<artifactSet>
<includes combine.children="append">
<include>org.apache.paimon:paimon-spark4-common_${scala.binary.version}</include>
<include>com.networknt:json-schema-validator</include>
<include>com.ethlo.time:itu</include>
</includes>
</artifactSet>
<relocations>
<relocation>
<pattern>com.networknt</pattern>
<shadedPattern>org.apache.paimon.spark.shade.networknt</shadedPattern>
</relocation>
<relocation>
<pattern>com.ethlo.time</pattern>
<shadedPattern>org.apache.paimon.spark.shade.time</shadedPattern>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All 4 production sources (OssieModel.scala, OssieSql.scala, OssieToSparkMetricView.scala, PaimonSemanticViewResolver.scala), ossie-0.3.0.dev0.schema.json, Spark4xRowTest.scala, and OssieModelTest.scala in paimon-spark-4.3 are byte-for-byte identical copies of paimon-spark-4.2, and OssieSemanticViewTest.scala / Spark43CompatibilityTest.scala only differ by spark.sql.analyzer.singlePassResolver.dualRunWithLegacy=false (plus two 4.3-specific tests).

As seen in commit b66bcaab, duplicating ~2,265 lines across both modules means every change to Ossie semantic views or compatibility tests in paimon-spark-4.2 (like #11237, #11238, and #11267) has to be manually copy-pasted into paimon-spark-4.3.

Could we share the sources/resources/tests from ../paimon-spark-4.2 via build-helper-maven-plugin (add-source / add-test-source) and <resources> (similar to paimon-spark-ut-4.0/pom.xml, or via a shared source directory in paimon-spark4-common), and keep only the Spark 4.3-specific test additions in paimon-spark-4.3?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After reconsidering, I think this is a good suggestion. Thank you @Akash3121

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 28a7ef8. paimon-spark-4.3 no longer has its own copy: build-helper adds the main and test sources of paimon-spark-4.2, and <resources> takes its resources except META-INF/NOTICE. The 4.3 module keeps only its NOTICE and a Spark43CompatibilityTest for splitStatements, which needs a 4.3 API. The two REPLACE TABLE cases moved into Spark42CompatibilityTest, so both connectors run them. The dualRunWithLegacy=false setting moved into the 4.2 tests too; Spark 4.2 already defaults it to false.

</project>
12 changes: 12 additions & 0 deletions paimon-spark/paimon-spark-4.3/src/main/resources/META-INF/NOTICE
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
paimon-spark-4.3
Copyright 2023-2026 The Apache Software Foundation

This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).

This project bundles the following dependencies under the Apache Software License 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt)
- com.networknt:json-schema-validator:2.0.0
- com.ethlo.time:itu:1.14.0

The bundled Ossie JSON Schema is from Apache Ossie (incubating), commit
2692b1cc4708fdcf6a91e597ca928a9bc9d0da60, under the Apache Software License 2.0.
Loading
Loading