feat(ffi): add vx_scalar_new_extension for extension-typed literals - #9691
Open
balicat wants to merge 1 commit into
Open
feat(ffi): add vx_scalar_new_extension for extension-typed literals#9691balicat wants to merge 1 commit into
balicat wants to merge 1 commit into
Conversation
The C API can build scan predicates against primitive, utf8, binary and decimal columns but not extension columns: no scalar constructor can express a value of an extension type, and the compare kernel (correctly) requires both sides of a comparison to share the extension dtype, so a bare storage-typed literal cannot stand in. For date/timestamp columns this leaves FFI consumers unable to push a time window into a scan. Adds vx_scalar_new_extension(dtype, storage, err): builds an extension-typed scalar from an extension dtype and a scalar of its storage dtype (compared ignoring nullability; value copied, arguments not consumed). Validation is delegated to Scalar::try_new, with explicit errors for a non-extension dtype and a storage-dtype mismatch. Header regenerated. Tests cover the date-over-i32 happy path and both rejection paths. Signed-off-by: David Linton <e.david.linton@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The C API can build scan predicates against primitive, utf8, binary and
decimal columns, but not against extension columns: none of the scalar
constructors can express a value of an extension type, and the compare
kernel deliberately refuses to compare an extension column with a bare
storage-typed literal (
Extension'sCompareKernelrequires both sidesto share the extension dtype — correctly, since e.g. timestamps in
different units must not compare raw storage).
For date and timestamp columns this leaves FFI consumers unable to push a
time window into a scan. In a .NET binding, a "series between two dates"
query over a date32 column currently has to decode each selected series'
full history and trim client-side; on a 219k-row query over an 18 MB file
we measured 521 ms for that pattern against 473 ms for an equivalent
Parquet reader pruning by row-group statistics. Everything else in the
chain already works — the engine compares extension constants efficiently
by extracting the storage scalar. The missing piece is only a way to
construct the literal through the C API.
Changes
Adds
vx_scalar_new_extension(dtype, storage, err): builds anextension-typed scalar from an extension dtype and a scalar of its
storage dtype (compared ignoring nullability; the value is copied and
neither argument is consumed). Validation is delegated to
Scalar::try_new, with explicit errors for a non-extension dtype and astorage-dtype mismatch.
Callers obtain the extension dtype from the data source's schema or via
vx_dtype_from_arrow_schema, build the storage value with an existingprimitive constructor (e.g.
vx_scalar_new_i32for a day-precisiondate), wrap it, and pass the result to
vx_expression_literalin a scanpredicate.
Unit tests cover the happy path (day-precision date over i32 storage,
dtype round-trip through
vx_scalar_dtype) and both rejection paths(non-extension dtype; mismatched storage dtype). Header regenerated via
the nightly cbindgen path in build.rs.