Repository navigation
perf(calcite): fold GROUP BY on a dictionary string column per code - #555
Merged
Merged
Conversation
Calcite ran GROUP BY by building one boxed row per record, copying it through the interpreter and hashing the key into a HashMap per row: on the profile, HashMap lookups alone were 25% of the query and the reader under 5%. A GROUP BY on one string column now folds in the scan when the column is dictionary-encoded. Compute.groupedAggregate accumulates count, non-null count, sum, min and max per dictionary code in primitive arrays (a slice of a shared dictionary included) and turns each code into its string once per chunk; VortexTable.groupedFold merges the chunks. The aggregate rule then replaces the subtree with one Values row per group, as it already does for whole-table aggregates, and supports the same WHERE. It falls back to the scan when a chunk is not dictionary-encoded, an aggregate is not COUNT/SUM/MIN/MAX of a plain numeric column (u64 and f16 excluded), there are more than 10,000 groups, or a group value is outside Latin-1 (no VARCHAR literal can hold it). Calcite fires the rule 6 to 9 times per query, once per matching operand and subset, so the folds are memoized per planner run (a weak map keyed by the planner): each query folds once, and no result outlives its planning. DictStringBenchmark.sqlGroupBy, single fork, ms/op: 4M synthetic rows, 8 symbols 136.8 -> 4.5 (30x) sf-salaries (Rust, 1.18M rows) 24.4 -> 2.5 (9.6x) Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WtuVGQ5h5Lk6Zn4JnkxF1K
Calcite fires the aggregate rule once per matching operand and subset, and every firing called VortexTable.filteredFold for each aggregate. A filtered count/min/max over sf-salaries decoded its boundary zones six times while planning one query. The fold now goes through the same per-planner memo as the grouped fold: once per distinct (filter, column) per query, nothing kept past the planning. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WtuVGQ5h5Lk6Zn4JnkxF1K
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two commits.
1. GROUP BY on a dictionary string column folds in the scan
Calcite ran
GROUP BYby building a boxed row per record, copying it through the interpreter and hashing the key into aHashMapper row (25% of the query in the profile; the reader was under 5%).A
GROUP BYon one string column now folds per dictionary code:Compute.groupedAggregate(chunk, groupColumn, filter, aggColumn)accumulates count, non-null count, sum, min and max per code in primitive arrays (sliced shared dictionaries included) and turns each code into its string once per chunk. Empty when the shapes do not qualify.VortexTable.groupedFoldmerges the chunks;VortexAggregatePushDownRulereplaces the subtree with oneValuesrow per group, with the sameWHEREsupport as the whole-table rewrite.Falls back to the scan when a chunk is not dictionary-encoded, an aggregate is not
COUNT/SUM/MIN/MAXof a plain numeric column (u64,f16excluded), there are more than 10,000 groups, or a group value is outside Latin-1.Calcite fires the rule 6 to 9 times per query, so folds are memoized per planner run (a weak map keyed by the planner): one fold per query, nothing cached across queries.
DictStringBenchmark.sqlGroupBy, single fork, ms/op:2. Filtered aggregates fold once per query
The same memo now covers
VortexTable.filteredFold: a filteredcount/min/maxover sf-salaries decoded its boundary zones 6 times while planning one query; now once.Tests
Compute.groupedAggregate(null group, long/double lanes with nulls, Rust float order, filter, sliced dictionary, duplicate dictionary strings, malformed and truncated codes, declined shapes); rule rewritesGROUP BYon a dictionary column and leaves numeric, multi-key andAVGgroupings alone.GROUP BYwithCOUNT(*),COUNT(col),SUM,MIN,MAX(long and double) under threeWHEREclauses equals the aggregates computed from the rows written, and the plan is aValues; a non-Latin-1 key falls back to the scan with the same answer.🤖 Generated with Claude Code
https://claude.ai/code/session_01WtuVGQ5h5Lk6Zn4JnkxF1K