Pluggable extension types: register custom logical types on the read/write registries #530

Activity

1. dfa1

   added

   enhancementNew feature or requestNew feature or request

   on Oct 11, 2026
2. dfa1 commented on Oct 11, 2026

   

   dfa1

   on Oct 11, 2026

   OwnerAuthor

   More actions

   Write-side details

   WriteRegistry is mostly ready: it keeps a Map<ExtensionId, ExtensionEncoder> with register(ExtensionEncoder) / lookup(ExtensionId), so it works unchanged once ExtensionId is sealed with a Custom variant. What needs to change:

   • VortexWriter (~line 554) resolves the encoder with ExtensionId.parse(extDtype.extensionId()).map(defaultRegistry::lookup).orElse(null). parse only knows the four built-in ids, so an unknown id yields empty and the lookup is skipped. It must resolve unknown ids to ExtensionId.Custom.
   • When no encoder is found, impl is null and a Collection is passed through unconverted, which fails later with a confusing error. Fail fast with a VortexException naming the id, or accept only raw storage arrays.
   • Add the storage-dtype validation hook (Rust's validate_dtype) to ExtensionEncoder, checked at VortexWriter.create time.
   • ExtensionEncoder.extensionId(), dtype(boolean) and encodeAll are generic enough for custom types as they are.
   • DateTimePartsEncodingEncoder compares the timestamp id as a string and is unaffected.
3. dfa1 commented on Oct 17, 2026

   

   dfa1

   on Oct 17, 2026

   OwnerAuthor

   More actions

   Implemented in #547. Follow-ups found along the way:

   • Unknown ids are strict, not passthrough. Rust errors with No such DType extension ID unless the session allows unknown ids (then it reads ForeignExtDType, i.e. storage). Pluggable extension types on the read and write registries (#530) #547 matches this: ReadRegistry.Builder#allowUnknown() is required. vortex-jni 0.88.0 exposes no allow_unknown, so it cannot read a file with a custom extension id.
   • Editions don't gate extension dtypes, and the json family is missing. Tracked in Editions: gate extension dtypes and support crate-declared families (json) #548.
   • Built-in validate checks are not ported. Rust's date/time/timestamp validate_dtype check storage ptypes (e.g. timestamp storage must be I64); the Java built-ins accept every dtype.
   • No Rust-writes/Java-reads test for a custom id. vortex-jni cannot write one, so this needs a cargo harness or a checked-in fixture.
   • Auto-routing a Collection applies to top-level columns only. An extension nested in a list or struct is written from its storage array.
4. dfa1

   added 2 commits that reference this issue on Oct 17, 2026

   fix: create(…, WriteRegistry) honors the registry's extension encoders

   ...

   7ae2906

   feat!: ExtensionId is a sealed WellKnown/Custom type

   ...

   5d8708e
5. dfa1

   closed this as completedin #547on Oct 17, 2026