Skip to content

New module mapper: read and write Java records as typed rows #528

Open

Description

@dfa1

A small module, mapper, that maps a Java record to a Vortex struct schema, for both the reader and the writer. Same idea as the typed-record module we did for typesafe-java; inspired by Vorticity v0.1.0, whose [VortexRecord] declares the columns of a file.

Minimum requirements

  • A Java record is the schema: its components are the columns, in order, under their names (ColumnName).
  • Write: records in, a .vortex file out, through the existing VortexWriter. Taking a List<Record> is fine and convenient: the usual source is row oriented, so the writer pivots rows into column batches.
  • Read: a .vortex file in, records out, through the existing VortexReader.
  • A new mapper module that depends on reader and writer; neither of them depends on it (the writer never depends on the reader today, and this module is the only place that sees both).
  • An unsupported component type fails fast with a clear error.

Projection is the reading model

A columnar file is read by column, never select *. The read side must make that the obvious path and say so in the docs:

  • A record passed to read is a projection: only its components are decoded, so a record with 1 or 2 components reads 1 or 2 columns of a file that has many more (it maps to ScanOptions columns).
  • The record need not match the file's full schema: any subset of the file's columns, by name, is valid. A component with no matching column, or with a different type, fails fast.
  • Document it with an example: a wide file written from a full Trade record, read back through a small PriceOnly(double price) record. Do not show reading the whole schema as the default.

First round: basic types only

Mapped from the Rust DType variants, only the basic ones:

DType Java component
Bool boolean / Boolean
Primitive byte, short, int, long, float, double (boxed = nullable); unsigned via @Unsigned (see below); F16 is deferred
Utf8 String
Binary byte[]

Nullability follows the component type: a boxed type or a reference type is nullable, a Java primitive is not.

F16 is deferred until Valhalla. A short of raw half bits needs an annotation to tell it from I16, and that API would be replaced once the JDK has a half-float value type. Until then a record component cannot map to an F16 column: it fails fast with a clear error, and the array API still reads and writes F16 as short[].

Later rounds, not in this ticket: Decimal, List, FixedSizeList, Map, nested Struct (a record inside a record), Union, Variant, and Extension (dates, times, timestamps, UUID), plus Null.

Exploratory: annotations

The module is exploratory, so these are proposals to try, not commitments. Principle: a schema annotation (changes the DType) must agree on the read and write side and is checked fail-fast; a write-time annotation is ignored when reading.

  • Unsigned (schema): Java has no unsigned types, and the existing array API already stores U8..U64 in the same byte[]/short[]/int[]/long[] as I8..I64. Try @Unsigned on a same-width Java primitive (@Unsigned int is U32), bit-identical with no conversion; the user reads it with Integer.toUnsignedLong and friends. A lossless widening on read (U32 as long) is a possible later addition.
  • Forcing encodings (write-time): @Encoding("vortex.zstd", ...) on a component maps to WriteOptions#withColumnEncoding(ColumnName, ColumnEncoding.candidates(...)), i.e. the whole candidate set for that column, as in Rust. Strings are parsed once at the boundary into EncodingId; an unknown id fails fast.
  • Column name (schema): @Column("price") when the component name is not the column name; useful for projection records over files written by someone else.
  • Later, once the nested and extension types land: @Decimal(precision, scale) on BigDecimal, and the zone-map/stats toggles.

Out of scope

Filters and aggregates as lambdas, async APIs, annotation processing or code generation (the first version can use java.lang.reflect.RecordComponent).

Activity

  1. dfa1 commented on Oct 17, 2026

    @dfa1
    OwnerAuthor

    Status: parked. Implementation pushed on feat/mapper-module (draft PR #549), built with explicit callbacks instead of annotations:

    • write: ColumnMapping.<Trade>builder().i64("ts", Trade::ts).u32("volume", Trade::volume).nullableF64("fee", Trade::fee)...build(), then VortexWriter.create(ch, mapping.schema(), opts) and mapping.write(writer, rows). No reflection, any class, the column type is the builder method (so no @Unsigned/@Column/@Encoding).
    • read: Rows.stream(reader, ScanOptions.columns(...), row -> new PriceOnly(row.getDouble(0))), a Row cursor addressed by projection position.

    Cross-checked against vortex-jni both ways (57 cases). It also uncovered an unrelated writer bug: vortex.sequence wrote U8/U16/U32/U64 and wrapping I64 sequences Rust refuses to read (fixed in the same branch, de90624a).

    Why parked: the read side competes with what we already have

    Yes on the read side; the write side has no counterpart (the Calcite adapter is read-only).

    Rows / Row (mapper) Calcite (VortexCalcite.connect)
    Surface typed getters by index → your class JDBC ResultSet via SQL
    Projection / filter ScanOptions (columns, RowFilter, limit) SQL → translated to the same projection + RowFilter
    Aggregates, group by, joins no yes (whole-table min/max/sum even from zone stats)
    Per-row cost cursor, no boxing Object[] per row, every cell boxed (VortexEnumerator)
    Dependencies reader only calcite-core + babel (Avatica, Guava, Janino, …)

    So the mapper read side is a lighter, typed ResultSet: genuinely cheaper than Calcite, but it serves the same "projected scan → rows" job. It also sits awkwardly in the middle:

    • people who want speed should use Chunk/Array (row-at-a-time callbacks defeat auto-vectorization, and a record per row means heap copies of every string, the GC pressure the zero-copy design avoids);
    • people who want rows plus expressiveness should use Calcite plus a stock ResultSet mapper (Spring RowMapper, Jdbi);
    • the mapper read side only wins "rows, but without the Calcite dependency": real, but narrow, and it would be a third read API to document and maintain.

    Recommendation if resumed: ship write-only. ColumnMapping is the unique part (pivoting row-shaped objects into columns), the module then depends only on writer, and the how-to points to Chunk for speed and Calcite for rows/SQL. Add reading later if asked, with a benchmark (Rows.stream vs a Chunk loop) showing "rows without Calcite" is worth a third API.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    PARKEDdo not yet mergeenhancementNew feature or requesthelp wantedExtra attention is needed

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions