Repository navigation
fix(calcite): aggregate push-down wrong results (DISTINCT, FILTER) and Latin-1 failure - #554
Merged
Merged
Conversation
VortexAggregatePushDownRule answered every COUNT/MIN/MAX/SUM call from zone statistics without looking at the call's modifiers. Statistics describe every row of a zone, so they cannot answer an aggregate over a subset of its rows: count(distinct g) 3000 rows, 7 values -> 3000 count(*) filter (where id < 10) -> 3000, not 10 min(id) filter (where id > 2000) -> 0, not 2001 On Raincloud sf-salaries, count(distinct `Organization Group`) returned 1,180,053 instead of 7. The rule now leaves the plan to the scan when any call is DISTINCT or carries a FILTER clause. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WtuVGQ5h5Lk6Zn4JnkxF1K
Calcite types VARCHAR as ISO-8859-1, and the statistics push-down built a literal from a string MIN/MAX stat without checking it fits. A column whose max holds a character Latin-1 cannot encode failed the whole query: select min(s), max(s) from t -- s holds "Ωmega" CalciteException: Failed to encode 'Ωmega' in character set 'ISO-8859-1' The scan returns the same strings without building a literal, so the rule now abandons when the type's charset cannot encode the value. Rust records string statistics as bounds the rule does not use, so this hit files with exact string stats, as the Java writer emits. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WtuVGQ5h5Lk6Zn4JnkxF1K
dfa1
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two bugs in
VortexAggregatePushDownRule, one commit each.1. DISTINCT and FILTER were ignored
The rule answered
COUNT/MIN/MAX/SUMfrom zone statistics without checkingAggregateCall.isDistinct()orhasFilter(). Statistics cover every row of a zone, so both returned wrong results:count(distinct g)(7 values, 3000 rows)count(*) filter (where id < 10)min(id) filter (where id > 2000)count(distinct "Organization Group")The rule now leaves the plan to the scan when any call is
DISTINCTor has aFILTERclause.2. String MIN/MAX failed outside Latin-1
Calcite types
VARCHARas ISO-8859-1, and the rule built a literal from a string stat without checking it fits:select min(s), max(s)over a column holdingΩmegafailed withFailed to encode 'Ωmega' in character set 'ISO-8859-1'. The rule now abandons when the charset cannot encode the value; the scan answers.Tests
DISTINCT/FILTERqueries over a Rust-written file (5 failed before), andMIN/MAXover Rust-written non-Latin-1 strings.AggregateRuleBranchTest:DISTINCT/FILTERaggregates stay in the plan (3 failed before);AggregatePushDownTest:MIN/MAXoverΩmegaanswers (failed before).🤖 Generated with Claude Code
https://claude.ai/code/session_01WtuVGQ5h5Lk6Zn4JnkxF1K