Skip to content

perf(calcite): fold GROUP BY on a dictionary string column per code - #555

Merged
dfa1 merged 2 commits into
mainfrom
feat/groupby-dict-pushdown
Oct 18, 2026
Merged

dfa1 merged 2 commits into
mainfrom
feat/groupby-dict-pushdown

Conversation

@dfa1

@dfa1 dfa1 commented Oct 18, 2026

Copy link
Copy Markdown
Owner

Two commits.

1. GROUP BY on a dictionary string column folds in the scan

Calcite ran GROUP BY by building a boxed row per record, copying it through the interpreter and hashing the key into a HashMap per row (25% of the query in the profile; the reader was under 5%).

A GROUP BY on one string column now folds per dictionary code:

  • Reader: Compute.groupedAggregate(chunk, groupColumn, filter, aggColumn) accumulates count, non-null count, sum, min and max per code in primitive arrays (sliced shared dictionaries included) and turns each code into its string once per chunk. Empty when the shapes do not qualify.
  • Calcite: VortexTable.groupedFold merges the chunks; VortexAggregatePushDownRule replaces the subtree with one Values row per group, with the same WHERE support as the whole-table rewrite.

Falls back to the scan when a chunk is not dictionary-encoded, an aggregate is not COUNT/SUM/MIN/MAX of a plain numeric column (u64, f16 excluded), there are more than 10,000 groups, or a group value is outside Latin-1.

Calcite fires the rule 6 to 9 times per query, so folds are memoized per planner run (a weak map keyed by the planner): one fold per query, nothing cached across queries.

DictStringBenchmark.sqlGroupBy, single fork, ms/op:

Before After
4M synthetic rows, 8 symbols 136.8 4.5 (30×)
sf-salaries (Rust-written, 1.18M rows) 24.4 2.5 (9.6×)

2. Filtered aggregates fold once per query

The same memo now covers VortexTable.filteredFold: a filtered count/min/max over sf-salaries decoded its boundary zones 6 times while planning one query; now once.

Tests

  • Unit: Compute.groupedAggregate (null group, long/double lanes with nulls, Rust float order, filter, sliced dictionary, duplicate dictionary strings, malformed and truncated codes, declined shapes); rule rewrites GROUP BY on a dictionary column and leaves numeric, multi-key and AVG groupings alone.
  • vortex-jni cross-check: Rust writes a nullable dictionary string column; GROUP BY with COUNT(*), COUNT(col), SUM, MIN, MAX (long and double) under three WHERE clauses equals the aggregates computed from the rows written, and the plan is a Values; a non-Latin-1 key falls back to the scan with the same answer.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WtuVGQ5h5Lk6Zn4JnkxF1K

dfa1 and others added 2 commits October 18, 2026 09:31
Calcite ran GROUP BY by building one boxed row per record, copying it
through the interpreter and hashing the key into a HashMap per row: on
the profile, HashMap lookups alone were 25% of the query and the reader
under 5%.

A GROUP BY on one string column now folds in the scan when the column
is dictionary-encoded. Compute.groupedAggregate accumulates count,
non-null count, sum, min and max per dictionary code in primitive
arrays (a slice of a shared dictionary included) and turns each code
into its string once per chunk; VortexTable.groupedFold merges the
chunks. The aggregate rule then replaces the subtree with one Values
row per group, as it already does for whole-table aggregates, and
supports the same WHERE.

It falls back to the scan when a chunk is not dictionary-encoded, an
aggregate is not COUNT/SUM/MIN/MAX of a plain numeric column (u64 and
f16 excluded), there are more than 10,000 groups, or a group value is
outside Latin-1 (no VARCHAR literal can hold it).

Calcite fires the rule 6 to 9 times per query, once per matching
operand and subset, so the folds are memoized per planner run (a weak
map keyed by the planner): each query folds once, and no result
outlives its planning.

DictStringBenchmark.sqlGroupBy, single fork, ms/op:

  4M synthetic rows, 8 symbols   136.8 -> 4.5  (30x)
  sf-salaries (Rust, 1.18M rows)  24.4 -> 2.5  (9.6x)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WtuVGQ5h5Lk6Zn4JnkxF1K
Calcite fires the aggregate rule once per matching operand and subset,
and every firing called VortexTable.filteredFold for each aggregate. A
filtered count/min/max over sf-salaries decoded its boundary zones six
times while planning one query. The fold now goes through the same
per-planner memo as the grouped fold: once per distinct (filter, column)
per query, nothing kept past the planning.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WtuVGQ5h5Lk6Zn4JnkxF1K
@dfa1
dfa1 merged commit c7ec637 into main Oct 18, 2026
9 checks passed
@dfa1
dfa1 deleted the feat/groupby-dict-pushdown branch October 18, 2026 08:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant