fix: hash join dynamic filter with dictionary-encoded join keys - #24587
Open
jayzhan211 wants to merge 1 commit into
Open
fix: hash join dynamic filter with dictionary-encoded join keys#24587jayzhan211 wants to merge 1 commit into
jayzhan211 wants to merge 1 commit into
Conversation
- Use logical_null_count() for keys_have_null so a null-equal join's pushed filter keeps probe NULL rows when the build NULL lives in a dictionary value - Gate ArrayMap (perfect hash join) creation on the key type instead of the presence of bounds, which are also collected for dynamic filters on any type - Rename should_collect_min_max_for_perfect_hash -> is_perfect_hash_join_candidate
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #24587 +/- ##
==========================================
- Coverage 81.38% 81.38% -0.01%
==========================================
Files 1116 1116
Lines 397960 398019 +59
Branches 397960 398019 +59
==========================================
+ Hits 323880 323922 +42
- Misses 55120 55127 +7
- Partials 18960 18970 +10 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Hash joins on a dictionary-encoded integer join key misbehave when dynamic filter pushdown is active. A dictionary array can carry NULL in its values while its key bitmap has no physical nulls (
null_count() == 0butlogical_null_count() > 0), and two places in the hash join only looked at the physical count:NullEqualsNulljoin,keys_have_nullwas computed withnull_count(), so the pushed dynamic filter was not widened withkey IS NULLand pruned the probe-side NULL row that should have null-matched the build-side NULL.try_create_array_mapinferred "the key type is supported byArrayMap" from the presence of min/max bounds. But bounds are also collected for dynamic filters on any key type, so with a dictionary key and a filter consumer the build side failed withUnsupported type for ArrayMap: Dictionary(Int32, Int32)instead of falling back to the regular hash map.What changes are included in this PR?
stream.rs: computekeys_have_nullwithlogical_null_count().exec.rs: gateArrayMap(perfect hash join) creation on the key type viais_perfect_hash_join_candidaterather than on the presence of bounds; also uselogical_null_count()for theNullEqualsNullcheck in the same function.should_collect_min_max_for_perfect_hash→is_perfect_hash_join_candidatewith a doc comment, since it describes what is checked rather than a side effect.Are these changes tested?
Yes. A new test,
test_null_equal_dynamic_filter_keeps_probe_nulls_for_build_logical_null, builds aNullEqualsNullinner join over dictionary keys with a dynamic filter consumer on the probe side and asserts both that the pushed filter contains theIS NULLdisjunct and that the NULL–NULL row is produced. Before this PR it fails with theArrayMaperror; with only theArrayMapgate it fails on the missingIS NULL. Existing hash join unit tests and thejoins,push_down_filter_parquet, andexplain_analyzesqllogictests pass.Are there any user-facing changes?
No API changes. Joins on dictionary-encoded integer keys with dynamic filter pushdown now return correct results instead of erroring / dropping rows.