Repository navigation
Conversation
Retain the smaller side of bitmap selections and preserve physical row mapping. Add regression coverage for dense, sparse, and reconfigured selections. Use callback I/O in native-scan test fixtures to avoid the Rust async filesystem path reported by TSAN, without adding suppressions. Add Apache license headers to both Vortex patch files.
Make membership and I/O limits configurable, simplify reader state, and bound native Arrow materialization, including nested arrays. Preserve safe temporal and decimal fallbacks, propagate type errors, and extend native and SDK regressions without changing shared readers.
Release selection and row-index state on close, build dense selections from compressed complements, and push empty projections into native scans. Preserve Arrow schema error details, clarify pushdown eligibility and document fallback and LTZ metadata rules. Add focused regressions and fix cast declarations reported by clang-tidy.
Preserve configurable read coalescing and both integration-test includes while adopting upstream comments and the shared timezone utility.
Continue apache#471. Consolidate the native patch, simplify reader setup and predicate conversion, and trim redundant test coverage. Add ASF headers to dependency patches and document Vortex read options and timestamp storage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Continue the Vortex predicate pushdown and selective-read work started by @zhouyc-ali. This branch preserves the original commits through
a2631bband adds the follow-up changes in a separate commit. Original PR #471Supersedes #553 with identical source contents. Related issue #319.
TIMESTAMP_LTZcolumns to UTC while preserving epoch values. Include the dependency patches in Apache RAT checks.Tests
INandNOT IN, nestedANDandOR, and fallback for unsupported values and oversized lists._row_idx, I/O coalescing, and callback and Arrow batch lifetimes.NOT INlist falls back. Java fixture tests cover cross-language reads andTIMESTAMP_LTZacross timezones.API and Format
Vortex readers accept
vortex.read.in-pushdown.max-values,vortex.read.io-coalesce.max-gap, andvortex.read.io-coalesce.max-size. The private Vortex FFI adds scan and scalar capabilities. New Vortex files storeTIMESTAMP_LTZtimezone metadata in UTC and preserve the stored epoch values. Public C++ signatures and the table protocol remain compatible.Documentation
Document the Vortex reader options and defaults, predicate fallback and residual filtering, and
TIMESTAMP_LTZstorage indocs/source/api/file_format.rst.Generative AI tooling
Generated-by: OpenAI Codex 0.167.0