Skip to content

[Spark] COUNT(*) scans all rows instead of using footer row counts #9655

Description

@AadhiKat

Spark's df.count() / SELECT COUNT(*) against the vortex datasource currently plans a full
scan: VortexScanBuilder implements SupportsPushDownRequiredColumns and
SupportsPushDownV2Filters but not SupportsPushDownAggregates, so Spark prunes to an empty
projection and iterates every row through the JNI scan pipeline just to count them.

The row count is already available without touching data: every file footer records it, and
VortexScan already surfaces it through SupportsReportStatistics (used by the CBO). The scan
path just never uses it to answer the query.

Measured on a real 2M-row × 804-column table on GCS (Spark 3.5.9, local[16]):

  • COUNT(*) via vortex-spark: 10.5s
  • same query, parquet: ~0.9s (Spark's built-in metadata path)
  • same query, lance-spark: ~0.45s (implements the aggregate pushdown)

Proposal: implement SupportsPushDownAggregates on VortexScanBuilder for the narrow,
always-correct case — a single global CountStar, no grouping, no pushed predicates — answered
by summing per-file footer counts (partial pushdown; Spark does the final sum). Analogous to the
ungrouped aggregate pushdown already done for DuckDB (#8645), and a first step toward MIN/MAX
from zone maps on the Spark side.

Happy to contribute the implementation — PR to follow.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions