Spark's df.count() / SELECT COUNT(*) against the vortex datasource currently plans a full
scan: VortexScanBuilder implements SupportsPushDownRequiredColumns and
SupportsPushDownV2Filters but not SupportsPushDownAggregates, so Spark prunes to an empty
projection and iterates every row through the JNI scan pipeline just to count them.
The row count is already available without touching data: every file footer records it, and
VortexScan already surfaces it through SupportsReportStatistics (used by the CBO). The scan
path just never uses it to answer the query.
Measured on a real 2M-row × 804-column table on GCS (Spark 3.5.9, local[16]):
COUNT(*) via vortex-spark: 10.5s
- same query, parquet: ~0.9s (Spark's built-in metadata path)
- same query, lance-spark: ~0.45s (implements the aggregate pushdown)
Proposal: implement SupportsPushDownAggregates on VortexScanBuilder for the narrow,
always-correct case — a single global CountStar, no grouping, no pushed predicates — answered
by summing per-file footer counts (partial pushdown; Spark does the final sum). Analogous to the
ungrouped aggregate pushdown already done for DuckDB (#8645), and a first step toward MIN/MAX
from zone maps on the Spark side.
Happy to contribute the implementation — PR to follow.
Spark's
df.count()/SELECT COUNT(*)against the vortex datasource currently plans a fullscan:
VortexScanBuilderimplementsSupportsPushDownRequiredColumnsandSupportsPushDownV2Filtersbut notSupportsPushDownAggregates, so Spark prunes to an emptyprojection and iterates every row through the JNI scan pipeline just to count them.
The row count is already available without touching data: every file footer records it, and
VortexScanalready surfaces it throughSupportsReportStatistics(used by the CBO). The scanpath just never uses it to answer the query.
Measured on a real 2M-row × 804-column table on GCS (Spark 3.5.9, local[16]):
COUNT(*)via vortex-spark: 10.5sProposal: implement
SupportsPushDownAggregatesonVortexScanBuilderfor the narrow,always-correct case — a single global
CountStar, no grouping, no pushed predicates — answeredby summing per-file footer counts (partial pushdown; Spark does the final sum). Analogous to the
ungrouped aggregate pushdown already done for DuckDB (#8645), and a first step toward MIN/MAX
from zone maps on the Spark side.
Happy to contribute the implementation — PR to follow.