Pluggable extension types on the read and write registries (#530)

CHANGELOG.md

@@ -7,6 +7,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+- `vortex.json` and the `vortex.st.*` geometry columns open without a decoder and read as their storage, as vortex-jni reads them. ([#547](https://github.com/dfa1/vortex-java/pull/547))
+- The CLI takes `--allow-unknown` to open files with extension types it has no decoder for, and CSV `ExportOptions.withReadRegistry` picks the registry an export opens the file with. ([#547](https://github.com/dfa1/vortex-java/pull/547))
+- Custom extension types: register an `ExtensionDecoder` on `ReadRegistry` and an `ExtensionEncoder` on `WriteRegistry`, with a `validate` hook on both sides, and `Chunk.as` decodes any registered type. ([#530](https://github.com/dfa1/vortex-java/issues/530))
+
+### Changed
+- **Breaking:** opening a file with an extension id that has no registered decoder fails unless `ReadRegistry.Builder#allowUnknown()` is set, as in Rust; the column then reads as its storage array. ([#530](https://github.com/dfa1/vortex-java/issues/530))
+- **Breaking:** `vortex.date`, `vortex.time`, `vortex.timestamp` and `vortex.uuid` dtypes are checked as Rust checks them (time unit, storage type, metadata) when writing and reading, and a file whose extension dtype has no metadata field is refused, as Rust refuses it. ([#547](https://github.com/dfa1/vortex-java/pull/547))
+- **Breaking:** `DateExtensionDecoder.decode` and `decodeAll` take the column's `DType.Extension`, which carries the time unit. ([#547](https://github.com/dfa1/vortex-java/pull/547))
+- **Breaking:** `ExtensionDecoder` is generic in its domain type and declares `domainType()` and `decodeAll(DType.Extension, Array)`. ([#530](https://github.com/dfa1/vortex-java/issues/530))
+- **Breaking:** `ExtensionId` is a sealed `WellKnown`/`Custom` type like `EncodingId`, so third-party extension ids can be named, and `ExtensionId.parse` returns the id instead of an `Optional`. ([#530](https://github.com/dfa1/vortex-java/issues/530))
+
+### Fixed
+- `vortex count`, `schema`, `stats` and `export` report a malformed file as an error instead of crashing with a stack trace. ([#547](https://github.com/dfa1/vortex-java/pull/547))
+- `vortex.date` columns in milliseconds, as Rust writes an Arrow `Date64`, decode to the right dates instead of failing or reading the count as days. ([#547](https://github.com/dfa1/vortex-java/pull/547))
+- An extension column with no metadata no longer writes a file Rust refuses to read. ([#547](https://github.com/dfa1/vortex-java/pull/547))
+- `VortexWriter.create(…, WriteRegistry)` uses the registry's extension encoders instead of the built-in ones, and a `Collection` for an extension column with no encoder fails with a `VortexException`. ([#530](https://github.com/dfa1/vortex-java/issues/530))
+
 ## [0.17.2] — 2026-10-17
 
 ### Changed

CLAUDE.md

@@ -192,11 +192,13 @@ Add an `EncodingId.WellKnown` constant `VORTEX_FOO("vortex.foo")` (re-exported o
 
 ### Adding an extension type
 
-Add `ExtensionId` constant, then per side:
-- **Decode:** singleton `FooExtensionDecoder implements ExtensionDecoder` in `reader.extension` +
-  a `case VORTEX_FOO` in `Chunk.as()` — not registry-managed.
-- **Encode:** `FooExtensionEncoder implements ExtensionEncoder` in `writer` + a
-  `.register(FooExtensionEncoder.INSTANCE)` call in `WriteRegistry.Builder#registerDefaults()`
+Add an `ExtensionId.WellKnown` constant `VORTEX_FOO("vortex.foo")` (re-exported on the interface), then per side:
+- **Decode:** singleton `FooExtensionDecoder implements ExtensionDecoder<Domain>` in `reader.extension` +
+  a `.register(FooExtensionDecoder.INSTANCE)` call in `ReadRegistry.Builder#registerDefaults()`.
+- **Encode:** `FooExtensionEncoder implements ExtensionEncoder` in `writer.encode` + a
+  `.register(FooExtensionEncoder.INSTANCE)` call in `WriteRegistry.Builder#registerDefaultExtensions()`.
+- Both may override `validate(DType.Extension)` (Rust's `validate_dtype`). Third-party types skip the
+  constant: `ExtensionId.parse("acme.foo")` gives a `Custom` id, registered the same way.
 
 ## Memory model
 
@@ -267,8 +269,9 @@ When stuck on encode/decode behavior, consult **in this order**:
 
 - **DType is pluggable only via `Extension`.** `DType` is a sealed interface; downstream code must
   not add variants. Use `new DType.Extension("ip.address", new DType.Primitive(PType.I32, false),
-  null, false)` and register decoders/encoders on the registries.
-  Mirrors Rust (`vortex.date`, `vortex.uuid`, …). No SPI for DType variants planned.
+  null, false)` and register an `ExtensionDecoder`/`ExtensionEncoder` on `ReadRegistry`/`WriteRegistry`.
+  Mirrors Rust's `ExtVTable` (`vortex.date`, `vortex.uuid`, …): an unregistered id fails the read
+  unless `allowUnknown()`, then reads as storage. No SPI for DType variants planned.
 - **Encoding and layout decode are both pluggable via builder-only registries** — no
   `ServiceLoader`. Encodings register on `ReadRegistry`/`WriteRegistry`
   (`ReadRegistry.builder().registerDefaults().register(custom).build()`); layouts register on
@@ -418,6 +421,11 @@ Generated proto records live in `io.github.dfa1.vortex.core.proto`; the runtime
   one for every encoding round-trip and file-format boundary. Include **filtered and row-range**
   vortex-jni reads, not just full scans: a full scan never prunes or slices, which hid a JVM abort
   (buffer alignment) and silently empty filtered results (zone stride) until 0.15.x.
+- **Every fix and feature gets a cross-check test against vortex-jni**, not only Java unit tests:
+  vortex-jni writes and Java reads, and/or Java writes and vortex-jni reads, asserting both agree.
+  Write it first, so it fails without the change. Java-only tests share Java's assumptions;
+  cross-checks are what caught unknown extension ids that Rust refuses, absent extension metadata
+  that made files unreadable, and `vortex.date` milliseconds read as days (#547).
 - JUnit 5 + Mockito (BDDMockito) + AssertJ. Class under test named `sut`. Every test has
   `// Given` / `// When` / `// Then`. BDDMockito only: `given(mock.m()).willReturn(v)` /
   `then(...)` (static-import only `given`/`then`, never `willReturn`/`willThrow`).

cli/src/main/java/io/github/dfa1/vortex/cli/CliHandles.java

@@ -1,6 +1,7 @@
 package io.github.dfa1.vortex.cli;
 
 import io.github.dfa1.vortex.cli.tui.IoWorker;
+import io.github.dfa1.vortex.reader.ReadRegistry;
 import io.github.dfa1.vortex.reader.VortexHandle;
 import io.github.dfa1.vortex.reader.VortexHttpReader;
 import io.github.dfa1.vortex.reader.VortexReader;
@@ -88,9 +89,10 @@ static String describe(Throwable t) {
     /// @return the opened handle, or `null` if the target is missing or malformed
     /// @throws IOException if opening the file or URL fails
     static VortexHandle openTarget(String target) throws IOException {
+        ReadRegistry registry = readRegistry();
         if (target.startsWith("http://") || target.startsWith("https://")) {
             try {
-                return VortexHttpReader.open(new URI(target));
+                return VortexHttpReader.open(new URI(target), registry);
             } catch (URISyntaxException _) {
                 System.err.println("invalid URL: " + target);
                 return null;
@@ -101,6 +103,25 @@ static VortexHandle openTarget(String target) throws IOException {
             System.err.println("file not found: " + path);
             return null;
         }
-        return VortexReader.open(path);
+        return VortexReader.open(path, registry);
+    }
+
+    // One CLI invocation per process; VortexCli.execute sets it before every dispatch.
+    private static ReadRegistry readRegistry = ReadRegistry.loadAll();
+
+    /// Sets whether files open with unknown extension ids allowed (the `--allow-unknown` flag).
+    ///
+    /// @param allowUnknown `true` to read unregistered extension types as their storage
+    static void allowUnknown(boolean allowUnknown) {
+        readRegistry = allowUnknown
+                ? ReadRegistry.builder().registerDefaults().allowUnknown().build()
+                : ReadRegistry.loadAll();
+    }
+
+    /// The registry every subcommand opens Vortex files with.
+    ///
+    /// @return the read registry for this invocation
+    static ReadRegistry readRegistry() {
+        return readRegistry;
     }
 }

cli/src/main/java/io/github/dfa1/vortex/cli/CountCommand.java

@@ -22,10 +22,10 @@ static int run(String[] args) {
             System.err.println("file not found: " + path);
             return ExitStatus.FILE_NOT_FOUND;
         }
-        try (VortexReader reader = VortexReader.open(path)) {
+        try (VortexReader reader = VortexReader.open(path, CliHandles.readRegistry())) {
             System.out.println(reader.layout().rowCount());
             return ExitStatus.OK;
-        } catch (IOException e) {
+        } catch (IOException | io.github.dfa1.vortex.core.error.VortexException e) {
             System.err.println("error: " + e.getMessage());
             return ExitStatus.ERROR;
         }

cli/src/main/java/io/github/dfa1/vortex/cli/ExportCommand.java

@@ -5,6 +5,7 @@
 import io.github.dfa1.vortex.csv.ExportOptions;
 import io.github.dfa1.vortex.parquet.ParquetExporter;
 import io.github.dfa1.vortex.reader.VortexHandle;
+import io.github.dfa1.vortex.reader.VortexReader;
 
 import java.io.IOException;
 import java.io.OutputStreamWriter;
@@ -43,7 +44,7 @@ static int run(String[] args) {
                 return runParquet(inputPath, outputPath);
             }
             return runCsv(inputPath, outputPath, toStdout);
-        } catch (IOException e) {
+        } catch (IOException | io.github.dfa1.vortex.core.error.VortexException e) {
             ProgressBar.clear();
             System.err.println("error: " + e.getMessage());
             return ExitStatus.ERROR;
@@ -69,7 +70,7 @@ private static int runRemote(String target, String[] args, boolean toStdout) {
             ProgressBar.clear();
             System.out.printf("written: %s  (%s)%n", outputPath, ByteSize.format(Files.size(outputPath)));
             return ExitStatus.OK;
-        } catch (IOException e) {
+        } catch (IOException | io.github.dfa1.vortex.core.error.VortexException e) {
             ProgressBar.clear();
             System.err.println("error: " + e.getMessage());
             return ExitStatus.ERROR;
@@ -78,7 +79,8 @@ private static int runRemote(String target, String[] args, boolean toStdout) {
 
     private static int runCsv(Path inputPath, Path outputPath, boolean toStdout) throws IOException {
         ExportOptions options = ExportOptions.defaults()
-                .withProgressListener(ProgressBar::render);
+                .withProgressListener(ProgressBar::render)
+                .withReadRegistry(CliHandles.readRegistry());
         if (toStdout) {
             Writer stdout = new OutputStreamWriter(System.out, StandardCharsets.UTF_8);
             CsvExporter.exportCsv(inputPath, stdout, options);
@@ -96,7 +98,9 @@ private static int runParquet(Path inputPath, Path outputPath) throws IOExceptio
         io.github.dfa1.vortex.parquet.ExportOptions options =
                 io.github.dfa1.vortex.parquet.ExportOptions.defaults()
                         .withProgressListener(ProgressBar::render);
-        ParquetExporter.exportParquet(inputPath, outputPath, options);
+        try (VortexReader reader = VortexReader.open(inputPath, CliHandles.readRegistry())) {
+            ParquetExporter.exportParquet(reader, outputPath, options);
+        }
         ProgressBar.clear();
         printResult(inputPath, outputPath);
         return ExitStatus.OK;

cli/src/main/java/io/github/dfa1/vortex/cli/FilterCommand.java

@@ -58,7 +58,8 @@ static int run(String[] args) {
         RowPredicate rowPred = toRowPredicate(filter);
         try {
             Writer stdout = new OutputStreamWriter(System.out, StandardCharsets.UTF_8);
-            CsvExporter.exportCsvFiltered(path, stdout, ExportOptions.defaults(), scanOptions, rowPred);
+            CsvExporter.exportCsvFiltered(path, stdout,
+                    ExportOptions.defaults().withReadRegistry(CliHandles.readRegistry()), scanOptions, rowPred);
             stdout.flush();
             return ExitStatus.OK;
         } catch (IOException | io.github.dfa1.vortex.core.error.VortexException e) {

cli/src/main/java/io/github/dfa1/vortex/cli/SchemaCommand.java

@@ -27,10 +27,10 @@ static int run(String[] args) {
             System.err.println("file not found: " + path);
             return ExitStatus.FILE_NOT_FOUND;
         }
-        try (VortexReader reader = VortexReader.open(path)) {
+        try (VortexReader reader = VortexReader.open(path, CliHandles.readRegistry())) {
             printSchema(System.out, path, reader.dtype(), reader.layout().rowCount());
             return ExitStatus.OK;
-        } catch (IOException e) {
+        } catch (IOException | io.github.dfa1.vortex.core.error.VortexException e) {
             System.err.println("error: " + e.getMessage());
             return ExitStatus.ERROR;
         }

cli/src/main/java/io/github/dfa1/vortex/cli/SelectCommand.java

@@ -48,7 +48,8 @@ static int run(String[] args) {
             System.err.println("error: " + e.getMessage());
             return ExitStatus.USAGE_ERROR;
         }
-        ExportOptions options = ExportOptions.defaults().withColumns(parsed.columns());
+        ExportOptions options = ExportOptions.defaults().withColumns(parsed.columns())
+                .withReadRegistry(CliHandles.readRegistry());
         try {
             Writer stdout = new OutputStreamWriter(System.out, StandardCharsets.UTF_8);
             if (parsed.conditions().isEmpty()) {

cli/src/main/java/io/github/dfa1/vortex/cli/StatsCommand.java

@@ -27,7 +27,7 @@ static int run(String[] args) {
             System.err.println("file not found: " + path);
             return ExitStatus.FILE_NOT_FOUND;
         }
-        try (VortexReader reader = VortexReader.open(path)) {
+        try (VortexReader reader = VortexReader.open(path, CliHandles.readRegistry())) {
             if (!(reader.dtype() instanceof DType.Struct schema)) {
                 System.err.println("error: root dtype is not a struct");
                 return ExitStatus.ERROR;
@@ -50,7 +50,7 @@ static int run(String[] args) {
                 System.out.printf("%-20s  %-12s  %15s  %15s%n", name.value(), type, min, max);
             }
             return ExitStatus.OK;
-        } catch (IOException e) {
+        } catch (IOException | io.github.dfa1.vortex.core.error.VortexException e) {
             System.err.println("error: " + e.getMessage());
             return ExitStatus.ERROR;
         }

cli/src/main/java/io/github/dfa1/vortex/cli/VortexCli.java

@@ -22,6 +22,9 @@ private VortexCli() {
 
     /// Global flag, accepted anywhere on the command line: prints how long the operation took.
     private static final String TIMING_FLAG = "--timing";
+    /// Global flag, accepted anywhere on the command line: opens files whose extension types have no
+    /// decoder, reading those columns as their storage (`ReadRegistry.Builder#allowUnknown()`).
+    private static final String ALLOW_UNKNOWN_FLAG = "--allow-unknown";
 
     public static void main(String[] args) {
         System.exit(execute(args));
@@ -33,7 +36,10 @@ public static void main(String[] args) {
     static int execute(String[] args) {
         long start = System.nanoTime();
         boolean timing = Arrays.asList(args).contains(TIMING_FLAG);
-        String[] commandArgs = Arrays.stream(args).filter(arg -> !TIMING_FLAG.equals(arg)).toArray(String[]::new);
+        CliHandles.allowUnknown(Arrays.asList(args).contains(ALLOW_UNKNOWN_FLAG));
+        String[] commandArgs = Arrays.stream(args)
+                .filter(arg -> !TIMING_FLAG.equals(arg) && !ALLOW_UNKNOWN_FLAG.equals(arg))
+                .toArray(String[]::new);
         int exit = dispatch(commandArgs);
         if (timing) {
             Duration elapsed = Duration.ofNanos(System.nanoTime() - start);
@@ -67,7 +73,7 @@ private static int dispatch(String[] args) {
     }
 
     static void printUsage(PrintStream out) {
-        out.println("Usage: java -jar vortex-cli-<version>-all.jar <subcommand> [args] [--timing]");
+        out.println("Usage: java -jar vortex-cli-<version>-all.jar <subcommand> [args] [--timing] [--allow-unknown]");
         out.println("  inspect [--html] <file|url>         print file structure; --html writes an HTML report");
         out.println("  tui     <file|url>                  open interactive inspector; url is http(s)://");
         out.println("  view    <file|url>                  open scrollable data grid; url is http(s)://");
@@ -80,5 +86,7 @@ static void printUsage(PrintStream out) {
         out.println("  stats   <file.vortex>               print per-column min/max statistics");
         out.println("  filter  <file.vortex> <expr>        filter rows to CSV (e.g. \"price >= 100\")");
         out.println("Any subcommand also takes --timing: print the elapsed time to stderr, JVM start-up excluded.");
+        out.println("Any subcommand also takes --allow-unknown: open a file whose extension types have no decoder,");
+        out.println("  reading those columns as their storage.");
     }
 }

cli/src/main/java/io/github/dfa1/vortex/cli/tui/GridRender.java

@@ -99,16 +99,17 @@ private static boolean isUnsigned(Array arr) {
     }
 
     private static Optional<String> formatExtension(DType.Extension ext, Array storage, long i) {
-        Optional<ExtensionId> idOpt = ExtensionId.parse(ext.extensionId());
-        if (idOpt.isEmpty()) {
+        if (!(ExtensionId.parse(ext.extensionId()) instanceof ExtensionId.WellKnown id)) {
             return Optional.empty();
         }
-        return Optional.of(switch (idOpt.get()) {
-            case VORTEX_DATE -> DateExtensionDecoder.INSTANCE.decode(storage, i).toString();
-            case VORTEX_TIME -> TimeExtensionDecoder.INSTANCE.decode(ext, storage, i).toString();
-            case VORTEX_TIMESTAMP -> TimestampExtensionDecoder.INSTANCE.instant(ext, storage, i).toString();
-            case VORTEX_UUID -> UuidExtensionDecoder.INSTANCE.decode(storage, i).toString();
-        });
+        return switch (id) {
+            case VORTEX_DATE -> Optional.of(DateExtensionDecoder.INSTANCE.decode(ext, storage, i).toString());
+            case VORTEX_TIME -> Optional.of(TimeExtensionDecoder.INSTANCE.decode(ext, storage, i).toString());
+            case VORTEX_TIMESTAMP -> Optional.of(TimestampExtensionDecoder.INSTANCE.instant(ext, storage, i).toString());
+            case VORTEX_UUID -> Optional.of(UuidExtensionDecoder.INSTANCE.decode(storage, i).toString());
+            // JSON text and geometries have no Java domain type yet: render the storage
+            default -> Optional.empty();
+        };
     }
 
     private static String bytesToHex(byte[] bytes) {

cli/src/main/java/io/github/dfa1/vortex/cli/tui/InspectorRender.java

@@ -39,11 +39,9 @@ private InspectorRender() {
     /// @return formatted value, or an angle-bracketed fallback for unknown shapes
     static String formatValue(Array array, int i, DType declared) {
         if (declared instanceof DType.Extension ext
-                && ExtensionId.parse(ext.extensionId())
-                        .filter(id -> id == ExtensionId.VORTEX_DATE)
-                        .isPresent()) {
+                && ExtensionId.parse(ext.extensionId()) == ExtensionId.VORTEX_DATE) {
             try {
-                return DateExtensionDecoder.INSTANCE.decode(array, i).toString();
+                return DateExtensionDecoder.INSTANCE.decode(ext, array, i).toString();
             } catch (RuntimeException _) {
                 // fall through to generic rendering on shape mismatch
             }