feat: vortex-mapper module (parked)

CHANGELOG.md

@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Added
+- New `vortex-mapper` module reads and writes your own objects as rows: `ColumnMapping` declares each column with a getter, and `Rows.stream` maps a projected scan through a `Row` callback. ([#528](https://github.com/dfa1/vortex-java/issues/528))
 - `vortex.json` and the `vortex.st.*` geometry columns open without a decoder and read as their storage, as vortex-jni reads them. ([#547](https://github.com/dfa1/vortex-java/pull/547))
 - The CLI takes `--allow-unknown` to open files with extension types it has no decoder for, and CSV `ExportOptions.withReadRegistry` picks the registry an export opens the file with. ([#547](https://github.com/dfa1/vortex-java/pull/547))
 - Custom extension types: register an `ExtensionDecoder` on `ReadRegistry` and an `ExtensionEncoder` on `WriteRegistry`, with a `validate` hook on both sides, and `Chunk.as` decodes any registered type. ([#530](https://github.com/dfa1/vortex-java/issues/530))
@@ -20,6 +21,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Breaking:** `ExtensionId` is a sealed `WellKnown`/`Custom` type like `EncodingId`, so third-party extension ids can be named, and `ExtensionId.parse` returns the id instead of an `Optional`. ([#530](https://github.com/dfa1/vortex-java/issues/530))
 
 ### Fixed
+- Unsigned integer columns that step evenly only past their type's range, or above the signed range of `U8`/`U16`/`U32`, no longer write a `vortex.sequence` that Rust refuses to read. ([#528](https://github.com/dfa1/vortex-java/issues/528))
 - `vortex count`, `schema`, `stats` and `export` report a malformed file as an error instead of crashing with a stack trace. ([#547](https://github.com/dfa1/vortex-java/pull/547))
 - `vortex.date` columns in milliseconds, as Rust writes an Arrow `Date64`, decode to the right dates instead of failing or reading the count as days. ([#547](https://github.com/dfa1/vortex-java/pull/547))
 - An extension column with no metadata no longer writes a file Rust refuses to read. ([#547](https://github.com/dfa1/vortex-java/pull/547))

CLAUDE.md

@@ -55,9 +55,13 @@ writer  — VortexWriter, WriteRegistry, WriteOptions, ColumnEncoding, Extension
           writer.encode — EncodingEncoder, EncodeContext, EncodeResult, EncodedBuffer, NullableData
           + *EncodingEncoder impls,
           extension encoders
+mapper  — io.github.dfa1.vortex.mapper: objects in and out as rows through explicit callbacks
+          (no reflection/annotations) — ColumnMapping (write: name + type + getter per column),
+          Rows.stream + Row (read: projection-first, cells by projection position)
 ```
 
 Dependency rule: `writer → core`, `reader → core`. **Writer never depends on reader.**
+`mapper` is the only library module that depends on both.
 `Array` and subtypes are decode outputs — they live in `reader.array`, not `core`.
 
 ## Branching

bom/pom.xml

@@ -55,6 +55,11 @@
                 <artifactId>vortex-csv</artifactId>
                 <version>${project.version}</version>
             </dependency>
+            <dependency>
+                <groupId>io.github.dfa1.vortex</groupId>
+                <artifactId>vortex-mapper</artifactId>
+                <version>${project.version}</version>
+            </dependency>
             <dependency>
                 <groupId>io.github.dfa1.vortex</groupId>
                 <artifactId>vortex-parquet</artifactId>

docs/how-to.md

@@ -628,6 +628,83 @@ java -jar cli/target/vortex-cli-*-all.jar filter data.vortex "price >= 100" > ou
 
 ---
 
+## Write and read Java objects
+
+The `vortex-mapper` module maps objects to rows through explicit callbacks: no reflection, no
+annotations, any class. On the write side, `ColumnMapping` declares each column once, with its
+name, its Vortex type (the method name) and a getter. The file schema is that list:
+
+```java
+import io.github.dfa1.vortex.core.model.ColumnName;
+import io.github.dfa1.vortex.mapper.ColumnMapping;
+import io.github.dfa1.vortex.mapper.Rows;
+import java.nio.channels.FileChannel;
+import java.nio.file.StandardOpenOption;
+import java.util.stream.Stream;
+
+record Trade(long ts, String symbol, double price, int volume, boolean buy, Double fee, byte[] orderId) {}
+
+ColumnMapping<Trade> trades = ColumnMapping.<Trade>builder()
+        .i64("ts", Trade::ts)
+        .utf8("symbol", Trade::symbol)
+        .f64("price", Trade::price)
+        .u32("volume", Trade::volume)            // unsigned: the raw bits of an int
+        .bool("buy", Trade::buy)
+        .nullableF64("fee", Trade::fee)          // null is a null cell
+        .binary("order_id", Trade::orderId)
+        .build();
+
+Path path = Path.of("trades.vortex");
+List<Trade> rows = List.of(
+        new Trade(1_700_000_000_000L, "AAPL", 189.95, 100, true, 0.5, new byte[]{1}),
+        new Trade(1_700_000_001_000L, "MSFT", 410.20, -1, false, null, new byte[]{2}));
+try (var channel = FileChannel.open(path, StandardOpenOption.CREATE, StandardOpenOption.WRITE,
+             StandardOpenOption.TRUNCATE_EXISTING);
+     var writer = VortexWriter.create(channel, trades.schema(), WriteOptions.defaults())) {
+    trades.write(writer, rows);          // call again to stream more rows
+}
+```
+
+On the read side, **projection is the model**: list only the columns you need in `ScanOptions`,
+and only those are decoded, however wide the file. `Rows.stream` hands each row to a mapper,
+which reads the cells by their position in that list:
+
+```java
+record PriceOnly(double price) {}
+
+try (var reader = VortexReader.open(Path.of("trades.vortex"));
+     Stream<PriceOnly> prices = Rows.stream(reader,
+             ScanOptions.columns(ColumnName.of("price")),
+             row -> new PriceOnly(row.getDouble(0)))) {
+    prices.forEach(System.out::println);
+}
+```
+
+Nulls and unsigned values are explicit: check `isNull` before a primitive getter on a nullable
+column, and widen unsigned raw bits yourself. Limits and filters on the `ScanOptions` apply as in
+a plain scan:
+
+```java
+record Fill(String symbol, long volume, Double fee) {}
+
+try (var reader = VortexReader.open(Path.of("trades.vortex"));
+     Stream<Fill> fills = Rows.stream(reader,
+             ScanOptions.columns(ColumnName.of("symbol"), ColumnName.of("volume"), ColumnName.of("fee"))
+                     .withLimit(10),
+             row -> new Fill(
+                     row.getString(0),
+                     Integer.toUnsignedLong(row.getInt(1)),
+                     row.isNull(2) ? null : row.getDouble(2)))) {
+    fills.forEach(System.out::println);
+}
+```
+
+A getter of the wrong type for its column (`getLong` on an `F64`) and a projected column the file
+does not have both fail with `IllegalArgumentException`. Encoding overrides stay on
+`WriteOptions#withColumnEncoding`.
+
+---
+
 ## Write and read a Map column
 
 `DType.Map` has no dedicated write-side value type. Physically, a map column's `vortex.map`

docs/reference.md

@@ -10,6 +10,7 @@ For task-oriented usage see [how-to.md](how-to.md); for design rationale see [ex
 - [Encoding registry](#encoding-registry)
 - [FSST (`io.github.dfa1.vortex.fsst`)](#fsst-iogithubdfa1vortexfsst)
 - [Parquet / CSV import and Parquet export](#parquet--csv-import-and-parquet-export)
+- [Object mapping](#object-mapping)
 - [JDBC import](#jdbc-import)
 - [Calcite SQL adapter](#calcite-sql-adapter)
 - [CLI](#cli)
@@ -556,6 +557,65 @@ Parquet's random-access footer-first format, CSV needs no Range requests and no
 
 ---
 
+## Object mapping
+
+Module `vortex-mapper`: objects in and out as rows, through explicit callbacks (no reflection, no
+annotations). See [Write and read Java objects](how-to.md#write-and-read-java-objects).
+
+### `ColumnMapping` (`io.github.dfa1.vortex.mapper.ColumnMapping`)
+
+| Method                                     | Description                                                              |
+|--------------------------------------------|--------------------------------------------------------------------------|
+| `ColumnMapping.<T>builder()`               | Starts a mapping; columns are declared in file order                     |
+| `schema()`                                 | The `DType.Struct` to pass to `VortexWriter.create`                      |
+| `write(VortexWriter, List<? extends T>)`   | Gathers each column from the rows and writes one `writeChunk`; empty list writes nothing |
+
+Builder methods, each `(String name, getter)`; every one has a `nullableXxx` twin taking a
+`Function<? super T, Box>` whose `null` is a null cell:
+
+| Method        | Column  | Getter                         |
+|---------------|---------|--------------------------------|
+| `bool`        | `Bool`  | `Predicate<? super T>`         |
+| `i8`, `u8`    | `I8`, `U8` | `Function<? super T, Byte>`  |
+| `i16`, `u16`  | `I16`, `U16` | `Function<? super T, Short>` |
+| `i32`, `u32`  | `I32`, `U32` | `ToIntFunction<? super T>`  |
+| `i64`, `u64`  | `I64`, `U64` | `ToLongFunction<? super T>` |
+| `f32`         | `F32`   | `Function<? super T, Float>`   |
+| `f64`         | `F64`   | `ToDoubleFunction<? super T>`  |
+| `utf8`        | `Utf8`  | `Function<? super T, String>`  |
+| `binary`      | `Binary`| `Function<? super T, byte[]>`  |
+
+Unsigned columns take the raw bits of the same-width Java type. A non-nullable column rejects a
+`null` with `IllegalArgumentException`; a duplicate or invalid column name fails at declaration.
+
+### `Rows` (`io.github.dfa1.vortex.mapper.Rows`)
+
+| Method                                                                        | Description |
+|-------------------------------------------------------------------------------|-------------|
+| `stream(VortexReader, ScanOptions, Function<? super Row, ? extends T>)`       | Lazily maps every row the scan selects; close the stream to release the scan. Fails with `IllegalArgumentException` if a projected column is not in the file |
+
+### `Row` (`io.github.dfa1.vortex.mapper.Row`)
+
+One row of a `Rows.stream` scan; cells are addressed by their position in the projection (file
+order with no projection). The instance moves along the scan, so do not keep it.
+
+| Method                    | Reads                     | Null cell          |
+|---------------------------|---------------------------|--------------------|
+| `isNull(int)`             | any column                | `true`             |
+| `getBoolean(int)`         | `Bool`                    | unspecified        |
+| `getByte(int)`            | `I8`, `U8` (raw bits)     | unspecified        |
+| `getShort(int)`           | `I16`, `U16` (raw bits)   | unspecified        |
+| `getInt(int)`             | `I32`, `U32` (raw bits)   | unspecified        |
+| `getLong(int)`            | `I64`, `U64` (raw bits)   | unspecified        |
+| `getFloat(int)`           | `F32`                     | unspecified        |
+| `getDouble(int)`          | `F64`                     | unspecified        |
+| `getString(int)`          | `Utf8`                    | `null`             |
+| `getBytes(int)`           | `Binary`, `Utf8` (a copy) | `null`             |
+
+A getter on a column of another type throws `IllegalArgumentException`.
+
+---
+
 ## JDBC import
 
 ### `JdbcImporter` (`io.github.dfa1.vortex.jdbc.JdbcImporter`)

integration/pom.xml

@@ -51,6 +51,11 @@
             <artifactId>vortex-csv</artifactId>
             <scope>test</scope>
         </dependency>
+        <dependency>
+            <groupId>io.github.dfa1.vortex</groupId>
+            <artifactId>vortex-mapper</artifactId>
+            <scope>test</scope>
+        </dependency>
         <dependency>
             <groupId>io.github.dfa1.vortex</groupId>
             <artifactId>vortex-jdbc</artifactId>
@@ -172,6 +177,11 @@
                     <artifactId>vortex-csv</artifactId>
                     <scope>compile</scope>
                 </dependency>
+                <dependency>
+                    <groupId>io.github.dfa1.vortex</groupId>
+                    <artifactId>vortex-mapper</artifactId>
+                    <scope>compile</scope>
+                </dependency>
             </dependencies>
             <build>
                 <plugins>

integration/src/test/java/io/github/dfa1/vortex/integration/JavaWritesRustReadsIntegrationTest.java

@@ -577,6 +577,60 @@ void javaWriter_jniReader_sequence_fullFilteredAndRowRange(@TempDir Path tmp) th
         assertThat(range).containsExactlyElementsOf(slice(Arrays.stream(times).boxed(), 1_501, 70_333));
     }
 
+    /// Columns that are arithmetic sequences only in wrapping 64-bit arithmetic, or only once a
+    /// narrow unsigned value is sign-extended. Rust builds a sequence with checked arithmetic in the
+    /// column's own type, so it refuses to read these as one; the unsigned ones that do step evenly
+    /// within their type must still be written as a sequence, with an unsigned base.
+    static Stream<Arguments> sequenceEdges() {
+        return Stream.of(
+                // a U8 base above 127, sign-extended, is a base Rust cannot narrow to u8
+                Arguments.of(DType.U8, new byte[]{(byte) 200, (byte) 201, (byte) 202}, true),
+                Arguments.of(DType.U16, new short[]{(short) 60_000, (short) 60_001, (short) 60_002}, true),
+                Arguments.of(DType.U32, new int[]{(int) 3_000_000_000L, (int) 3_000_000_001L, (int) 3_000_000_002L},
+                        true),
+                // descending: the step is negative, which a u64 multiplier cannot hold
+                Arguments.of(DType.U64, new long[]{10, 7, 4}, false),
+                // ascending past 2^64: each step overflows u64
+                Arguments.of(DType.U64, new long[]{0, -3_000_000_019L, -6_000_000_038L}, false),
+                // a step of 255 does not fit an i8
+                Arguments.of(DType.I8, new byte[]{-128, 127}, false),
+                // MAX - MIN wraps to -1
+                Arguments.of(DType.I64, new long[]{Long.MIN_VALUE, Long.MAX_VALUE}, false));
+    }
+
+    @ParameterizedTest(name = "{0} {1}")
+    @MethodSource("sequenceEdges")
+    void javaWriter_jniReader_sequenceInColumnTypeArithmetic(DType dtype, Object values, boolean sequence,
+            @TempDir Path tmp) throws IOException {
+        // Given
+        ColumnName column = ColumnName.of("v");
+        Path file = tmp.resolve("java_sequence_edge.vtx");
+        WriteOptions options = WriteOptions.defaults().withColumnEncoding(column,
+                io.github.dfa1.vortex.writer.ColumnEncoding.candidates(
+                        io.github.dfa1.vortex.core.model.EncodingId.VORTEX_SEQUENCE));
+        try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE);
+             var sut = VortexWriter.create(ch, new DType.Struct(List.of(column), List.of(dtype), false), options)) {
+            sut.writeChunk(Map.of(column, values));
+        }
+        int n = java.lang.reflect.Array.getLength(values);
+
+        // When
+        List<Object> result = readRowRange(file, "v", 0, n);
+
+        // Then
+        List<Object> expected = new ArrayList<>();
+        for (int i = 0; i < n; i++) {
+            expected.add(java.lang.reflect.Array.get(values, i));
+        }
+        // Arrow hands a UInt16 back as a char
+        assertThat(result).map(v -> v instanceof Character c ? (Object) (short) c.charValue() : v)
+                .containsExactlyElementsOf(expected);
+        try (var reader = io.github.dfa1.vortex.reader.VortexReader.open(file)) {
+            assertThat(reader.footer().arraySpecs().contains(io.github.dfa1.vortex.core.model.EncodingId.VORTEX_SEQUENCE))
+                    .isEqualTo(sequence);
+        }
+    }
+
     @Test
     void javaWriter_jniReader_floatDictWithCompressedValues_fullFilteredAndRowRange(@TempDir Path tmp) throws IOException {
         // Given — 500 distinct full-precision prices over 100,000 rows: ALP cannot encode them, so