Skip to content

fix: hash join dynamic filter with dictionary-encoded join keys - #24587

Open
jayzhan211 wants to merge 1 commit into
apache:mainfrom
jayzhan211:fix/hash-join-dict-key-dynamic-filter
Open

fix: hash join dynamic filter with dictionary-encoded join keys#24587
jayzhan211 wants to merge 1 commit into
apache:mainfrom
jayzhan211:fix/hash-join-dict-key-dynamic-filter

Conversation

@jayzhan211

Copy link
Copy Markdown
Contributor

Rationale for this change

Hash joins on a dictionary-encoded integer join key misbehave when dynamic filter pushdown is active. A dictionary array can carry NULL in its values while its key bitmap has no physical nulls (null_count() == 0 but logical_null_count() > 0), and two places in the hash join only looked at the physical count:

  1. Wrong results: for a NullEqualsNull join, keys_have_null was computed with null_count(), so the pushed dynamic filter was not widened with key IS NULL and pruned the probe-side NULL row that should have null-matched the build-side NULL.
  2. Query error: try_create_array_map inferred "the key type is supported by ArrayMap" from the presence of min/max bounds. But bounds are also collected for dynamic filters on any key type, so with a dictionary key and a filter consumer the build side failed with Unsupported type for ArrayMap: Dictionary(Int32, Int32) instead of falling back to the regular hash map.

What changes are included in this PR?

  • stream.rs: compute keys_have_null with logical_null_count().
  • exec.rs: gate ArrayMap (perfect hash join) creation on the key type via is_perfect_hash_join_candidate rather than on the presence of bounds; also use logical_null_count() for the NullEqualsNull check in the same function.
  • Rename should_collect_min_max_for_perfect_hashis_perfect_hash_join_candidate with a doc comment, since it describes what is checked rather than a side effect.

Are these changes tested?

Yes. A new test, test_null_equal_dynamic_filter_keeps_probe_nulls_for_build_logical_null, builds a NullEqualsNull inner join over dictionary keys with a dynamic filter consumer on the probe side and asserts both that the pushed filter contains the IS NULL disjunct and that the NULL–NULL row is produced. Before this PR it fails with the ArrayMap error; with only the ArrayMap gate it fails on the missing IS NULL. Existing hash join unit tests and the joins, push_down_filter_parquet, and explain_analyze sqllogictests pass.

Are there any user-facing changes?

No API changes. Joins on dictionary-encoded integer keys with dynamic filter pushdown now return correct results instead of erroring / dropping rows.

- Use logical_null_count() for keys_have_null so a null-equal join's pushed
  filter keeps probe NULL rows when the build NULL lives in a dictionary value
- Gate ArrayMap (perfect hash join) creation on the key type instead of the
  presence of bounds, which are also collected for dynamic filters on any type
- Rename should_collect_min_max_for_perfect_hash -> is_perfect_hash_join_candidate
@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label Aug 23, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.33333% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.38%. Comparing base (5134a1a) to head (71456db).

Files with missing lines Patch % Lines
...tafusion/physical-plan/src/joins/hash_join/exec.rs 83.07% 1 Missing and 10 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24587      +/-   ##
==========================================
- Coverage   81.38%   81.38%   -0.01%     
==========================================
  Files        1116     1116              
  Lines      397960   398019      +59     
  Branches   397960   398019      +59     
==========================================
+ Hits       323880   323922      +42     
- Misses      55120    55127       +7     
- Partials    18960    18970      +10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants