Repository navigation
Pluggable extension types on the read and write registries (#530) - #547
Merged
Merged
Conversation
VortexWriter rebuilt its extension set from the four built-ins, so an ExtensionEncoder registered on the WriteRegistry was silently dropped. A Collection for an extension column with no encoder now fails fast instead of reaching the encoders unconverted. Refs #530 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Mirrors EncodingId/LayoutId so a third-party extension id can be named and registered. parse() is total (returns the id, not an Optional). The file parser rejects a missing, blank or control-character extension id with a VortexException, so later parses of a read dtype cannot throw IllegalArgumentException. WriteRegistry.Builder orders extensions by id string: a Custom record has no natural ordering and broke the TreeMap. Refs #530 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ExtensionDecoder<T> registers on ReadRegistry (built-ins via registerDefaults()) and Chunk.as dispatches through it instead of a switch over the four spec ids. Both ExtensionDecoder and ExtensionEncoder gain validate(DType.Extension), Rust's validate_dtype, run on every extension dtype in the schema at VortexWriter.create and at open. An extension id with no registered decoder now fails the open unless ReadRegistry.Builder#allowUnknown() is set, then reads as its storage array: Rust errors with "No such DType extension ID" unless the session allows unknown ids (ForeignExtDType). vortex-jni 0.88.0 exposes no allow_unknown, so it refuses such files; CustomExtensionIntegrationTest pins that. CustomExtensionIntegrationTest defines an example.vector type (float[] over FixedSizeList<F32>, dimension in metadata) and covers write/read through full, filtered and limited scans, nulls, raw storage writes, validation on both sides, and unregistered ids. Closes #530 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This was referenced Oct 17, 2026
DTypeFbsSerializer omitted the FbsExtension metadata vector when
DType.Extension#metadata() was null. Rust requires the field ("failed to
parse extension metadata from flatbuffer") and refuses the whole file, even
when only a sibling column is projected. Rust writes an empty vector when
there is no metadata; so does Java now.
Found by the #547 parity review.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- A file with no dtype blob parses with a null schema; validateExtensions switched on it and threw NullPointerException at open. - A third-party ExtensionDecoder#validate throwing anything but VortexException on malformed metadata is wrapped, so reading a file only fails with VortexException. - The built-in extension encoder list lives once, in WriteRegistry.Builder#registerDefaultExtensions(). - Javadoc of ExtensionDecoder, ReadRegistry.loadAll and isAllowUnknown matches the strict default; ReadRegistryTest helpers follow the tests. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…data Closes the parity gaps the #547 review found against vortex-jni 0.88.0: - ExtensionId.WellKnown#validate ports Rust's deserialize_metadata + validate_dtype for vortex.date/time/timestamp/uuid (time unit to storage ptype, timestamp metadata layout and I64 storage, uuid version byte and FixedSizeList<U8, 16>). The built-in decoders and encoders use it, so Java no longer writes or reads e.g. vortex.timestamp over I32, which Rust refuses ("Timestamp storage dtype must be i64"). - Extension dtypes embedded in Variant/ParquetVariant/constant-variant metadata go through ReadRegistry#validateExtensions, as Rust's DType::from_proto resolves them on the session; their nullability follows the storage instead of a hardcoded false. - The file parser refuses an extension dtype without a metadata field, as Rust does ("failed to parse extension metadata from flatbuffer"). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Rust stores an Arrow Date64 as vortex.date(Milliseconds) over I64, but DateExtensionDecoder read every storage value as days: a Date64 column failed with 'Invalid value for EpochDay' or came back as wrong dates. The decoder now reads the unit from the dtype and floors milliseconds to the day (so instants before the epoch land on the earlier date). decode and decodeAll take the DType.Extension, as TimeExtensionDecoder already does; the CLI renderers and the HTML report go through it. DateInteropIntegrationTest cross-checks it: vortex-jni writes Date32 and Date64 columns (nulls, pre-epoch dates, three batches) and Java reads them through a full and a filtered scan. It failed before this change. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
They caught only IOException, so a VortexException from opening or reading a file (a malformed file, or an extension type the reader refuses) escaped as a stack trace instead of 'error: ...' and an error exit status, unlike select and filter. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The vortex-jni session registers vortex.json and the vortex-spatial geometries (vortex.st.wkb/point/linestring/polygon/multipoint/ multilinestring/multipolygon/box) by default, so the strict reader refused files the reference reads. They are now ExtensionId.WellKnown: a file opens without a decoder and the column reads as its storage, held to Rust's storage rules (Utf8 with empty metadata, Binary, GeoArrow coordinate structs nested in lists; the spatial CRS protobuf is not decoded). The writer applies the same rules to a well-known id with no encoder. Java has no JSON or geometry domain type, so Chunk.as refuses them. Any other unregistered id stays strict. The CLI takes --allow-unknown anywhere on the command line to open such files, reaching the CSV export paths through the new csv ExportOptions#withReadRegistry. WellKnownExtensionsInteropIntegrationTest cross-checks: Java writes json, wkb and point columns, and Java's default reader and vortex-jni see the same values. The invalid-timestamp parity test now builds its file by renaming a same-length custom id, since no Java write path emits it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dfa1
added a commit
that referenced
this pull request
Oct 17, 2026
DTypeFbsSerializer omitted the FbsExtension metadata vector when
DType.Extension#metadata() was null. Rust requires the field ("failed to
parse extension metadata from flatbuffer") and refuses the whole file, even
when only a sibling column is projected. Rust writes an empty vector when
there is no metadata; so does Java now.
Found by the #547 parity review.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dfa1
added a commit
that referenced
this pull request
Oct 17, 2026
…data Closes the parity gaps the #547 review found against vortex-jni 0.88.0: - ExtensionId.WellKnown#validate ports Rust's deserialize_metadata + validate_dtype for vortex.date/time/timestamp/uuid (time unit to storage ptype, timestamp metadata layout and I64 storage, uuid version byte and FixedSizeList<U8, 16>). The built-in decoders and encoders use it, so Java no longer writes or reads e.g. vortex.timestamp over I32, which Rust refuses ("Timestamp storage dtype must be i64"). - Extension dtypes embedded in Variant/ParquetVariant/constant-variant metadata go through ReadRegistry#validateExtensions, as Rust's DType::from_proto resolves them on the session; their nullability follows the storage instead of a hardcoded false. - The file parser refuses an extension dtype without a metadata field, as Rust does ("failed to parse extension metadata from flatbuffer"). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #530. Commits are meant to land separately (rebase merge).
Commits
VortexWriter.create(…, WriteRegistry)honors the registry's extension encoders (they were silently dropped); aCollectionwith no encoder fails fast.ExtensionIdis a sealedWellKnown/Customtype; the file parser rejects missing/blank/control-character ids withVortexException.ExtensionDecoder<T>registers onReadRegistry;Chunk.asdispatches through it;validate(DType.Extension)(Rust'svalidate_dtype) on both sides, run atcreateand at open.nullmetadata).validateexceptions, one built-in encoder list, javadoc.ExtensionId.WellKnown#validate); Variant-embedded extension dtypes go through the registry; a missing metadata field is refused on read.vortex.datein milliseconds (ArrowDate64) decodes to the right dates.count/schema/stats/exportreport malformed files instead of crashing.vortex.jsonandvortex.st.*(the ids the vortex-jni session registers by default) open without a decoder and read as storage, held to Rust's storage rules; CLI--allow-unknown; CSVExportOptions#withReadRegistry.Parity (vs vortex-jni 0.88.0)
ReadRegistry.Builder#allowUnknown()(Rust: "No such DType extension ID"), except the well-known ids above. vortex-jni has noallow_unknown, so it refuses files with custom ids.Tests
Cross-checks with vortex-jni:
CustomExtensionIntegrationTest,DateInteropIntegrationTest,WellKnownExtensionsInteropIntegrationTest, plusJavaWritesRustReadsIntegrationTestcases for null metadata and invalid built-in dtypes../mvnw verifygreen locally (2910 integration tests).Follow-ups
jsonedition family; JSON/geometry domain types.🤖 Generated with Claude Code