Skip to content

Editions: gate extension dtypes and support crate-declared families (json) #548

Open

Description

@dfa1

Rust editions gate extension dtypes as well as encodings, and edition families are no longer a fixed set: optional crates declare their own. vortex-java models neither.

Rust behavior

  • An edition lists its members by component kind, and an extension dtype is one of those kinds (EditionMember::dtype(..)). A writer with an edition policy may emit only members of enabled families, so a schema with an extension dtype outside them is refused.
  • Families are declared per crate. vortex-json/src/editions.rs declares the json family (origin vortex-json) with one draft edition, json2026.08.0, adding the vortex.json dtype. vortex_json::initialize(session) registers the dtype and enables the family together. Being outside core reflects that a reader built without vortex-json cannot resolve vortex.json.

vortex-java today

  1. The edition guard (WriteOptions#editions(), ADR 0023) checks array encodings only, so withEdition(..) lets a schema carry vortex.json or any custom extension id (now easy to register, Pluggable extension types: register custom logical types on the read/write registries #530/Pluggable extension types on the read and write registries (#530) #547). docs/compatibility.md already notes the catalog models encodings only.
  2. EditionFamily is a closed enum (CORE/PREVIEW/ZSTD), and CLAUDE.md justifies this because a private family carries no real guarantee. Rust now has crate-declared families (json), so the enum is missing one, and the closed-world rationale no longer matches the reference.

To decide

  • Extend the Editions catalog with extension-dtype members and enforce them in the writer's edition guard at VortexWriter.create.
  • Model json (and decide between keeping EditionFamily closed with known Rust families added, and opening it the way Rust does). Update ADR 0023 / CLAUDE.md accordingly.
  • Related: a built-in vortex.json ExtensionDecoder<String> (UTF-8 storage, validate rejecting non-Utf8), so JSON files open without allowUnknown() after Pluggable extension types on the read and write registries (#530) #547.

Refs #530

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions