Skip to content

[Feature] Support predicate pushdown and optimize selective reads for Vortex #567

Closed

Description

@zjw1111

Search before asking

  • I searched the issue tracker. The parent issue tracks overall Vortex format support.

Motivation

Track the Vortex read optimizations proposed in #319. Native scans can apply supported filters, projections, and bitmap selections before materializing data for the C++ reader.

Solution

  • Push supported comparisons, null checks, and IN / NOT IN predicates into Vortex scans. Preserve physical file row IDs and retain candidate rows for residual filtering when pushdown is unsafe.
  • Support per-reader limits for membership pushdown and I/O coalescing. Bound Arrow batch materialization for selective reads.
  • Cover predicate conversion, projections, bitmap selections, row IDs, and residual filtering with Vortex file and table read tests.

Anything else?

Implementation is in #553, continuing the work from #471.

Are you willing to submit a PR?

  • I'm willing to submit a PR!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions