Repository navigation
New module mapper: read and write Java records as typed rows #528
Description
A small module, mapper, that maps a Java record to a Vortex struct schema, for both the reader and the writer. Same idea as the typed-record module we did for typesafe-java; inspired by Vorticity v0.1.0, whose [VortexRecord] declares the columns of a file.
Minimum requirements
- A Java
recordis the schema: its components are the columns, in order, under their names (ColumnName). - Write: records in, a
.vortexfile out, through the existingVortexWriter. Taking aList<Record>is fine and convenient: the usual source is row oriented, so the writer pivots rows into column batches. - Read: a
.vortexfile in, records out, through the existingVortexReader. - A new
mappermodule that depends onreaderandwriter; neither of them depends on it (the writer never depends on the reader today, and this module is the only place that sees both). - An unsupported component type fails fast with a clear error.
Projection is the reading model
A columnar file is read by column, never select *. The read side must make that the obvious path and say so in the docs:
- A record passed to read is a projection: only its components are decoded, so a record with 1 or 2 components reads 1 or 2 columns of a file that has many more (it maps to
ScanOptionscolumns). - The record need not match the file's full schema: any subset of the file's columns, by name, is valid. A component with no matching column, or with a different type, fails fast.
- Document it with an example: a wide file written from a full
Traderecord, read back through a smallPriceOnly(double price)record. Do not show reading the whole schema as the default.
First round: basic types only
Mapped from the Rust DType variants, only the basic ones:
DType |
Java component |
|---|---|
Bool |
boolean / Boolean |
Primitive |
byte, short, int, long, float, double (boxed = nullable); unsigned via @Unsigned (see below); F16 is deferred |
Utf8 |
String |
Binary |
byte[] |
Nullability follows the component type: a boxed type or a reference type is nullable, a Java primitive is not.
F16 is deferred until Valhalla. A short of raw half bits needs an annotation to tell it from I16, and that API would be replaced once the JDK has a half-float value type. Until then a record component cannot map to an F16 column: it fails fast with a clear error, and the array API still reads and writes F16 as short[].
Later rounds, not in this ticket: Decimal, List, FixedSizeList, Map, nested Struct (a record inside a record), Union, Variant, and Extension (dates, times, timestamps, UUID), plus Null.
Exploratory: annotations
The module is exploratory, so these are proposals to try, not commitments. Principle: a schema annotation (changes the DType) must agree on the read and write side and is checked fail-fast; a write-time annotation is ignored when reading.
- Unsigned (schema): Java has no unsigned types, and the existing array API already stores
U8..U64in the samebyte[]/short[]/int[]/long[]asI8..I64. Try@Unsignedon a same-width Java primitive (@Unsigned intisU32), bit-identical with no conversion; the user reads it withInteger.toUnsignedLongand friends. A lossless widening on read (U32aslong) is a possible later addition. - Forcing encodings (write-time):
@Encoding("vortex.zstd", ...)on a component maps toWriteOptions#withColumnEncoding(ColumnName, ColumnEncoding.candidates(...)), i.e. the whole candidate set for that column, as in Rust. Strings are parsed once at the boundary intoEncodingId; an unknown id fails fast. - Column name (schema):
@Column("price")when the component name is not the column name; useful for projection records over files written by someone else. - Later, once the nested and extension types land:
@Decimal(precision, scale)onBigDecimal, and the zone-map/stats toggles.
Out of scope
Filters and aggregates as lambdas, async APIs, annotation processing or code generation (the first version can use java.lang.reflect.RecordComponent).
Activity
dfa1 commentedon Oct 17, 2026
Status: parked. Implementation pushed on
feat/mapper-module(draft PR #549), built with explicit callbacks instead of annotations:ColumnMapping.<Trade>builder().i64("ts", Trade::ts).u32("volume", Trade::volume).nullableF64("fee", Trade::fee)...build(), thenVortexWriter.create(ch, mapping.schema(), opts)andmapping.write(writer, rows). No reflection, any class, the column type is the builder method (so no@Unsigned/@Column/@Encoding).Rows.stream(reader, ScanOptions.columns(...), row -> new PriceOnly(row.getDouble(0))), aRowcursor addressed by projection position.Cross-checked against vortex-jni both ways (57 cases). It also uncovered an unrelated writer bug:
vortex.sequencewrote U8/U16/U32/U64 and wrapping I64 sequences Rust refuses to read (fixed in the same branch,de90624a).Why parked: the read side competes with what we already have
Yes on the read side; the write side has no counterpart (the Calcite adapter is read-only).
Rows/Row(mapper)VortexCalcite.connect)ResultSetvia SQLScanOptions(columns,RowFilter, limit)RowFilterObject[]per row, every cell boxed (VortexEnumerator)So the mapper read side is a lighter, typed
ResultSet: genuinely cheaper than Calcite, but it serves the same "projected scan → rows" job. It also sits awkwardly in the middle:Chunk/Array(row-at-a-time callbacks defeat auto-vectorization, and a record per row means heap copies of every string, the GC pressure the zero-copy design avoids);ResultSetmapper (SpringRowMapper, Jdbi);Recommendation if resumed: ship write-only.
ColumnMappingis the unique part (pivoting row-shaped objects into columns), the module then depends only onwriter, and the how-to points toChunkfor speed and Calcite for rows/SQL. Add reading later if asked, with a benchmark (Rows.streamvs aChunkloop) showing "rows without Calcite" is worth a third API.