Skip to content

feat(vortex): add predicate pushdown and optimize selective reads - #554

Closed
zjw1111 wants to merge 7 commits into
apache:mainfrom
zjw1111:codex/vortex-pushdown-471-continuation
Closed

zjw1111 wants to merge 7 commits into
apache:mainfrom
zjw1111:codex/vortex-pushdown-471-continuation

Conversation

@zjw1111

@zjw1111 zjw1111 commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator

Purpose

Continue the Vortex predicate pushdown and selective-read work started by @zhouyc-ali. This branch preserves the original commits through a2631bb and adds the follow-up changes in a separate commit. Original PR #471

Supersedes #553 with identical source contents. Related issue #319.

  • Push supported predicates, projections, and bitmap selections into Vortex scans while preserving physical file row IDs. Fall back to candidate rows when a predicate cannot be pushed safely.
  • Make membership limits and I/O coalescing configurable per reader, and bound Arrow batch materialization before export, including nested values.
  • Normalize timezone metadata for new TIMESTAMP_LTZ columns to UTC while preserving epoch values. Include the dependency patches in Apache RAT checks.

Tests

  • Vortex predicate converter tests cover comparisons, null checks, IN and NOT IN, nested AND and OR, and fallback for unsupported values and oversized lists.
  • Vortex file-format tests cover bitmap selections, physical row IDs, projections, reader reconfiguration, a user column named _row_idx, I/O coalescing, and callback and Arrow batch lifetimes.
  • Vortex table-read integration tests cover late materialization and the exact candidate and filtered results when an oversized NOT IN list falls back. Java fixture tests cover cross-language reads and TIMESTAMP_LTZ across timezones.

API and Format

Vortex readers accept vortex.read.in-pushdown.max-values, vortex.read.io-coalesce.max-gap, and vortex.read.io-coalesce.max-size. The private Vortex FFI adds scan and scalar capabilities. New Vortex files store TIMESTAMP_LTZ timezone metadata in UTC and preserve the stored epoch values. Public C++ signatures and the table protocol remain compatible.

Documentation

Document the Vortex reader options and defaults, predicate fallback and residual filtering, and TIMESTAMP_LTZ storage in docs/source/api/file_format.rst.

Generative AI tooling

Generated-by: OpenAI Codex 0.167.0

zhouyc-ali and others added 7 commits October 27, 2026 10:01
Retain the smaller side of bitmap selections and preserve physical row mapping.
Add regression coverage for dense, sparse, and reconfigured selections.

Use callback I/O in native-scan test fixtures to avoid the Rust async
filesystem path reported by TSAN, without adding suppressions.
Add Apache license headers to both Vortex patch files.
Make membership and I/O limits configurable, simplify reader state,
and bound native Arrow materialization, including nested arrays.

Preserve safe temporal and decimal fallbacks, propagate type errors,
and extend native and SDK regressions without changing shared readers.
Release selection and row-index state on close, build dense selections
from compressed complements, and push empty projections into native scans.

Preserve Arrow schema error details, clarify pushdown eligibility and
document fallback and LTZ metadata rules. Add focused regressions and
fix cast declarations reported by clang-tidy.
Preserve configurable read coalescing and both integration-test includes
while adopting upstream comments and the shared timezone utility.
Continue apache#471. Consolidate the native patch, simplify reader setup and predicate conversion, and trim redundant test coverage.

Add ASF headers to dependency patches and document Vortex read options and timestamp storage.
@zjw1111 zjw1111 closed this Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants