This is a tracking issue for constructing extension-typed scalars through the C API (vortex-ffi), so extension columns — dates and timestamps foremost — can be used in scan predicates by FFI consumers.
Motivation
The C API's scalar constructors cover primitives, utf8, binary and decimal, but none can express a value of an extension type. The compare kernel deliberately requires both sides of a comparison to share the extension dtype (raw-storage comparison across, e.g., timestamp units would be wrong), so a bare storage-typed literal is correctly rejected — which leaves FFI consumers with no way to push a time window into a scan over a date32 or timestamp column.
Who hits it: a .NET binding doing "series between two dates" over date32 columns must decode each selected series' full history and trim client-side. Measured on a 219k-row query over an 18 MB file: 521 ms this way vs 473 ms for an equivalent Parquet reader that prunes via row-group statistics — a race Vortex should win.
The rest of the chain already works: Extension's CompareKernel compares an extension constant efficiently by extracting the storage scalar. Only the literal's construction is missing from the C surface.
Design
vx_scalar *vx_scalar_new_extension(const vx_dtype *dtype,
const vx_scalar *storage,
vx_error **err);dtype must be an extension dtype (taken from the data source's schema, or built via vx_dtype_from_arrow_schema); storage a scalar of its storage dtype, compared ignoring nullability. The value is copied, neither argument is consumed, and validation is delegated to Scalar::try_new.
Steps
Unresolved questions
Implementation history
This is a tracking issue for constructing extension-typed scalars through the C API (
vortex-ffi), so extension columns — dates and timestamps foremost — can be used in scan predicates by FFI consumers.Motivation
The C API's scalar constructors cover primitives, utf8, binary and decimal, but none can express a value of an extension type. The compare kernel deliberately requires both sides of a comparison to share the extension dtype (raw-storage comparison across, e.g., timestamp units would be wrong), so a bare storage-typed literal is correctly rejected — which leaves FFI consumers with no way to push a time window into a scan over a date32 or timestamp column.
Who hits it: a .NET binding doing "series between two dates" over date32 columns must decode each selected series' full history and trim client-side. Measured on a 219k-row query over an 18 MB file: 521 ms this way vs 473 ms for an equivalent Parquet reader that prunes via row-group statistics — a race Vortex should win.
The rest of the chain already works:
Extension'sCompareKernelcompares an extension constant efficiently by extracting the storage scalar. Only the literal's construction is missing from the C surface.Design
dtypemust be an extension dtype (taken from the data source's schema, or built viavx_dtype_from_arrow_schema);storagea scalar of its storage dtype, compared ignoring nullability. The value is copied, neither argument is consumed, and validation is delegated toScalar::try_new.Steps
Unresolved questions
vx_scalar_new_date32) exist alongside the general form?Implementation history