Vortex spark integration is a fileformat not a tableprovider

.github/workflows/ci.yml

@@ -549,9 +549,11 @@ jobs:
       - uses: ./.github/actions/setup-prebuild
         with:
           enable-sccache: "true"
-      - run: ./gradlew javadoc
+      - name: Generate Java API documentation
+        run: ./gradlew javadoc
         working-directory: ./java
-      - run: ./gradlew check
+      - name: Check all Spark and Scala variants
+        run: ./gradlew check
         working-directory: ./java
 
   license-check-and-audit-check:

.github/workflows/docs.yml

@@ -46,7 +46,7 @@ jobs:
           mkdir -p docs/_static/vortex-jni
           mkdir -p docs/_static/vortex-spark
           cp -r java/vortex-jni/build/docs/javadoc/* docs/_static/vortex-jni/
-          cp -r java/vortex-spark/build/vortex-spark_2.13/docs/javadoc/* docs/_static/vortex-spark/
+          cp -r java/vortex-spark/v4.0/build/vortex-spark-4.0_2.13/docs/javadoc/* docs/_static/vortex-spark/
       - name: build Python and Rust docs
         run: |
           uv run --all-packages make -C docs html

.github/workflows/publish.yml

@@ -104,7 +104,7 @@ jobs:
           cp ../libvortex_jni_aarch64-apple-darwin.zip/libvortex_jni.dylib ./vortex-jni/src/main/resources/native/darwin-aarch64
           cp ../libvortex_jni_aarch64-unknown-linux-gnu.zip/libvortex_jni.so ./vortex-jni/src/main/resources/native/linux-aarch64
           cp ../libvortex_jni_x86_64-unknown-linux-gnu.zip/libvortex_jni.so ./vortex-jni/src/main/resources/native/linux-amd64
-      - name: Build Java
+      - name: Build all Java release artifacts
         run: ./gradlew shadowJar
       - name: Publish to Maven Central
         run: ./gradlew -i publishAndReleaseToMavenCentral --no-configuration-cache

README.md

@@ -6,7 +6,7 @@
 [![CodSpeed Badge](https://img.shields.io/endpoint?url=https://codspeed.io/badge.json)](https://codspeed.io/vortex-data/vortex)
 [![Crates.io](https://img.shields.io/crates/v/vortex.svg)](https://crates.io/crates/vortex)
 [![PyPI - Version](https://img.shields.io/pypi/v/vortex-data)](https://pypi.org/project/vortex-data/)
-[![Maven - Version](https://img.shields.io/maven-central/v/dev.vortex/vortex-spark_2.13)](https://central.sonatype.com/artifact/dev.vortex/vortex-spark_2.13)
+[![Maven - Version](https://img.shields.io/maven-central/v/dev.vortex/vortex-spark-4.0_2.13)](https://central.sonatype.com/artifact/dev.vortex/vortex-spark-4.0_2.13)
 [![codecov](https://codecov.io/github/vortex-data/vortex/graph/badge.svg)](https://codecov.io/github/vortex-data/vortex)
 [![Cite](https://img.shields.io/badge/cite-CITATION.cff-blue)](CITATION.cff)
 

docs/developer-guide/integrations/spark.md

@@ -1,57 +1,52 @@
 # Spark
 
-The `vortex-spark` connector implements Apache Spark's DataSource V2 API, allowing Spark to read
-and write Vortex files as a native data source registered under the format name `vortex`.
+The `vortex-spark` connector is built on Spark's file-source DataSource V2 framework. Shared
+sources compile into Spark 3.5/Scala 2.12, Spark 3.5/Scala 2.13, and Spark 4.0/Scala 2.13
+artifacts; the Spark 4 artifact is also tested on Spark 4.1.
 
-## Registration
+## File-source integration
 
-The connector implements Spark's `TableProvider` and `DataSourceRegister` interfaces. When a
-query references the `vortex` format, Spark creates a `VortexTable` that supports both batch
-reads (`SupportsRead`) and writes (`SupportsWrite`). Schema inference reads the footer of a
-discovered file to extract the Arrow schema and map it to Spark's schema representation.
+`VortexDataSourceV2` extends `FileDataSourceV2`, while `VortexTable`, `VortexScan`, and
+`VortexWrite` use Spark's `FileTable`, `FileScan`, and `FileWrite` abstractions. Spark therefore
+owns file listing, partition discovery and pruning, input bin-packing, output commit, and
+overwrite behavior. `VortexFileFormat` supplies the functional V1 fallback used by catalog
+tables and direct path queries.
 
-## Multiple Files
+Vortex files are not internally split. A Spark file partition may contain several files, and the
+reader factory opens each `PartitionedFile` in turn. Hive partition values are appended with
+constant column vectors.
 
-Spark's scan builder enumerates Vortex files by scanning the provided path. If the path is a
-directory, native code lists all `.vortex` files within it. Each file becomes an independent
-input partition, and Spark's task scheduler distributes partitions across executors in the
-cluster.
+## I/O and JNI
 
-Each partition creates its own file handle and scan state, so there is no shared mutable state
-between partitions. This maps naturally to Spark's execution model where each task runs
-independently on a separate JVM thread.
+Spark lists paths through Hadoop. Content reads go through pooled Hadoop input streams exposed to
+native Vortex through the JNI `NativeReadable` interface. Writes expose the committer's Hadoop
+task path through `NativeWritable`. Vortex's own object-store clients are not used, so the
+connector sees the same schemes and credentials as Spark's file index and commit protocol.
 
-## Threading Model
+Native arrays cross into Spark through the Arrow C Data Interface. Each partition reader owns its
+native scan, Arrow allocator, and exported batches and closes them at task completion.
 
-The Spark integration crosses the JNI boundary between Java and Rust. Each Spark partition
-reader opens a Vortex file and creates a native scan via JNI. The native side manages its own
-async runtime and drives I/O internally, returning results to Java as Arrow-compatible columnar
-batches.
+## Pushdown and statistics
 
-Because each partition reader owns its native resources exclusively, there is no contention
-across Spark threads. The JNI boundary is crossed once per batch rather than once per row,
-keeping overhead low. A prefetching iterator on the Java side buffers upcoming batches to
-overlap I/O with Spark's processing.
+Schema inference merges every footer, so the dataset schema is the union of the top-level fields its
+files carry. The partition reader projects only the fields the file it opened actually holds and
+fills the rest with constant null vectors, and it converts filters against those same fields, so a
+filter on a column the file lacks stays a Spark residual rather than reaching the native scan.
 
-## Filter and Projection Pushdown
+Spark's required schema becomes the Vortex scan projection. Convertible V1 filters become Vortex
+expressions; filters the converter rejects remain Spark residuals.
 
-Projection pushdown is supported through Spark's `SupportsPushDownRequiredColumns` interface.
-The scan builder prunes the column list to only those referenced by the query, and the pruned
-column set is passed to the native scan via `ScanOptions`.
+The scan accepts `COUNT(*)` aggregation when there are no data filters and grouping uses only
+partition columns, and only from footers that state their row count exactly. Readers return one
+footer count per file and Spark performs the final merge; the scan then reports one row per file as
+its statistics rather than reading those footers again on the driver. Footer row counts from files
+left after partition pruning are reported through Spark scan statistics when no aggregate is
+pushed. MIN/MAX and `COUNT(column)` need read-side column statistics in the JNI API and are
+not pushed down.
 
-Filter pushdown is supported through `SupportsPushDownV2Filters`. The scan builder converts each
-predicate it recognizes into a Vortex expression and keeps the rest for Spark to evaluate after the
-scan; the converted expression reaches native code as the `ScanOptions` filter.
+## Source layout
 
-## Data Export
-
-Native Vortex arrays are exported to Arrow via the C Data Interface, then wrapped in Spark's
-columnar batch format using custom `ArrowColumnVector` wrappers. This avoids a copy between
-the Rust and JVM heaps -- the Arrow buffers remain in native memory and are accessed from Java
-through direct byte buffers.
-
-## Future Work
-
-The current integration builds directly on the native file and scan APIs via JNI. Future work
-will migrate it to use the [Scan API](/concepts/scanning) `Source` trait, which will provide a
-standard interface for file discovery, partitioning, and pushdown.
+Shared Java and Scala live in `java/vortex-spark/common`. Thin projects in `v3.5` and `v4.0`
+select the corresponding Spark and Scala dependencies. The Scala shims isolate the binary
+differences in Spark's Scala APIs; nothing in Java depends on them, so javac compiles the Java
+sources and ErrorProne and Nopen keep covering them.

docs/developer-guide/internals/architecture.md

@@ -80,7 +80,7 @@ Query engine integrations allow Vortex files to be queried through existing anal
 |----------------------------------| ---------- |----------------------------------------------|
 | `vortex-datafusion/`             | DataFusion | `TableProvider` and `FileFormat` integration |
 | `vortex-duckdb/`                 | DuckDB     | Table function integration                   |
-| `java/vortex-spark_{2.12,2.13}/` | Spark      | Spark DataSource V2 connector via JNI        |
+| `java/vortex-spark/`             | Spark      | Versioned Spark file-source connector via JNI |
 | `java/vortex-trino/`             | Trino      | Trino connector (in development)             |
 
 ## Other Crates