Skip to content

feat(ffi): add vx_scalar_new_extension for extension-typed literals - #9691

Open
balicat wants to merge 1 commit into
vortex-data:developfrom
balicat:ffi-extension-scalars
Open

feat(ffi): add vx_scalar_new_extension for extension-typed literals#9691
balicat wants to merge 1 commit into
vortex-data:developfrom
balicat:ffi-extension-scalars

Conversation

@balicat

@balicat balicat commented Aug 28, 2026

Copy link
Copy Markdown

Summary

The C API can build scan predicates against primitive, utf8, binary and
decimal columns, but not against extension columns: none of the scalar
constructors can express a value of an extension type, and the compare
kernel deliberately refuses to compare an extension column with a bare
storage-typed literal (Extension's CompareKernel requires both sides
to share the extension dtype — correctly, since e.g. timestamps in
different units must not compare raw storage).

For date and timestamp columns this leaves FFI consumers unable to push a
time window into a scan. In a .NET binding, a "series between two dates"
query over a date32 column currently has to decode each selected series'
full history and trim client-side; on a 219k-row query over an 18 MB file
we measured 521 ms for that pattern against 473 ms for an equivalent
Parquet reader pruning by row-group statistics. Everything else in the
chain already works — the engine compares extension constants efficiently
by extracting the storage scalar. The missing piece is only a way to
construct the literal through the C API.

Changes

Adds vx_scalar_new_extension(dtype, storage, err): builds an
extension-typed scalar from an extension dtype and a scalar of its
storage dtype (compared ignoring nullability; the value is copied and
neither argument is consumed). Validation is delegated to
Scalar::try_new, with explicit errors for a non-extension dtype and a
storage-dtype mismatch.

Callers obtain the extension dtype from the data source's schema or via
vx_dtype_from_arrow_schema, build the storage value with an existing
primitive constructor (e.g. vx_scalar_new_i32 for a day-precision
date), wrap it, and pass the result to vx_expression_literal in a scan
predicate.

Unit tests cover the happy path (day-precision date over i32 storage,
dtype round-trip through vx_scalar_dtype) and both rejection paths
(non-extension dtype; mismatched storage dtype). Header regenerated via
the nightly cbindgen path in build.rs.

The C API can build scan predicates against primitive, utf8, binary and
decimal columns but not extension columns: no scalar constructor can
express a value of an extension type, and the compare kernel (correctly)
requires both sides of a comparison to share the extension dtype, so a
bare storage-typed literal cannot stand in. For date/timestamp columns
this leaves FFI consumers unable to push a time window into a scan.

Adds vx_scalar_new_extension(dtype, storage, err): builds an
extension-typed scalar from an extension dtype and a scalar of its
storage dtype (compared ignoring nullability; value copied, arguments
not consumed). Validation is delegated to Scalar::try_new, with explicit
errors for a non-extension dtype and a storage-dtype mismatch. Header
regenerated. Tests cover the date-over-i32 happy path and both
rejection paths.

Signed-off-by: David Linton <e.david.linton@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant