Skip to content

feat(vortex): add predicate pushdown and optimize selective reads - #553

Merged
lxy-9602 merged 7 commits into
apache:mainfrom
zjw1111:codex/vortex-pushdown-471
Nov 16, 2026
Merged

lxy-9602 merged 7 commits into
apache:mainfrom
zjw1111:codex/vortex-pushdown-471

Conversation

@zjw1111

@zjw1111 zjw1111 commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Purpose

Continue the Vortex predicate pushdown and selective-read work started by @zhouyc-ali. The branch preserves the original commits through a2631bb and adds the follow-up changes in a separate commit. Original pull request #471

Closes #567. Parent tracking issue #319.

  • Push supported predicates, projections, and bitmap selections into Vortex scans while preserving physical file row IDs. Fall back to candidate rows when a predicate cannot be pushed safely.
  • Make membership limits and I/O coalescing configurable per reader. Bound Arrow batch materialization before export, including nested values.
  • Normalize timezone metadata for newly written TIMESTAMP_LTZ columns to UTC while preserving their epoch values. Add ASF headers to the dependency patches and include them in RAT checks.

Tests

  • Vortex predicate converter tests cover comparisons, null checks, IN and NOT IN, nested AND/OR expressions, and safe fallback for unsupported values and oversized lists.
  • Vortex file reader tests cover sparse and dense selections, physical row IDs, projection and reader reconfiguration, user columns named _row_idx, and callback I/O and Arrow batch ownership.
  • Vortex table-read integration tests cover late materialization with native filtering and the complete candidate and exact results when an oversized NOT IN list falls back.

API and Format

Vortex readers accept vortex.read.in-pushdown.max-values, vortex.read.io-coalesce.max-gap, and vortex.read.io-coalesce.max-size. The private Vortex FFI adds scan and scalar capabilities. New TIMESTAMP_LTZ files record UTC timezone metadata and preserve the stored epoch values.

Documentation

Document the Vortex reader options and defaults, predicate fallback rules, residual filtering, and TIMESTAMP_LTZ storage in docs/source/api/file_format.rst.

Generative AI tooling

Generated-by: OpenAI Codex 0.167.0

Retain the smaller side of bitmap selections and preserve physical row mapping.
Add regression coverage for dense, sparse, and reconfigured selections.

Use callback I/O in native-scan test fixtures to avoid the Rust async
filesystem path reported by TSAN, without adding suppressions.
Add Apache license headers to both Vortex patch files.
Make membership and I/O limits configurable, simplify reader state,
and bound native Arrow materialization, including nested arrays.

Preserve safe temporal and decimal fallbacks, propagate type errors,
and extend native and SDK regressions without changing shared readers.
Release selection and row-index state on close, build dense selections
from compressed complements, and push empty projections into native scans.

Preserve Arrow schema error details, clarify pushdown eligibility and
document fallback and LTZ metadata rules. Add focused regressions and
fix cast declarations reported by clang-tidy.
Preserve configurable read coalescing and both integration-test includes
while adopting upstream comments and the shared timezone utility.
@zjw1111
zjw1111 requested a review from SteNicholas November 13, 2026 06:39
Continue apache#471. Consolidate the native patch, simplify reader setup and predicate conversion, and trim redundant test coverage.

Add ASF headers to dependency patches and document Vortex read options and timestamp storage.

@SteNicholas SteNicholas left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lxy-9602
lxy-9602 merged commit aab33b6 into apache:main Nov 16, 2026
39 of 40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Support predicate pushdown and optimize selective reads for Vortex

4 participants