feat: add projection support to SortMergeJoinExec - #24517
Conversation
|
run benchmarks run tpch tpcds tpch10 |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/sort-merge-join-projection (65415ac) to 5091b42 (merge-base) diff Run configurationrun benchmark run
env:
PREFER_HASH_JOIN: "false"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/sort-merge-join-projection (65415ac) to 5091b42 (merge-base) diff Run configurationrun benchmark tpch
env:
PREFER_HASH_JOIN: "false"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/sort-merge-join-projection (65415ac) to 5091b42 (merge-base) diff Run configurationrun benchmark tpch10
env:
PREFER_HASH_JOIN: "false"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/sort-merge-join-projection (65415ac) to 5091b42 (merge-base) diff Run configurationrun benchmark tpcds
env:
PREFER_HASH_JOIN: "false"Results will be posted here when complete File an issue against this benchmark runner |
A sort merge join emitted every column it joined, so a query selecting a subset needed a ProjectionExec above it, and one joining on an expression kept that expression in the output. Projection pushdown could only push into the children when each side's columns stayed together, and gave up otherwise. SortMergeJoinExec now carries an optional projection, as HashJoinExec does, and projection pushdown embeds into the join whatever it cannot push into the children. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012YiAABcW4WSqij31zz2P6c
65415ac to
2d55b49
Compare
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
|
Benchmark for this request failed before finishing (Kubernetes reason: Benchmarks requested: Kubernetes messageFile an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing feat/sort-merge-join-projection (65415ac) to 5091b42 (merge-base) diff Run configurationrun benchmark tpch
env:
PREFER_HASH_JOIN: "false"CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing feat/sort-merge-join-projection (65415ac) to 5091b42 (merge-base) diff Run configurationrun benchmark tpch10
env:
PREFER_HASH_JOIN: "false"CPU Details (lscpu)Details
Resource Usagetpch10 — base (merge-base)
tpch10 — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing feat/sort-merge-join-projection (65415ac) to 5091b42 (merge-base) diff Run configurationrun benchmark tpcds
env:
PREFER_HASH_JOIN: "false"CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
run benchmarks run tpch tpcds tpch10 |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/sort-merge-join-projection (2d55b49) to 40c208e (merge-base) diff Run configurationrun benchmark tpcds
env:
PREFER_HASH_JOIN: "false"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/sort-merge-join-projection (2d55b49) to 40c208e (merge-base) diff Run configurationrun benchmark tpch10
env:
PREFER_HASH_JOIN: "false"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/sort-merge-join-projection (2d55b49) to 40c208e (merge-base) diff Run configurationrun benchmark tpch
env:
PREFER_HASH_JOIN: "false"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/sort-merge-join-projection (2d55b49) to 40c208e (merge-base) diff Run configurationrun benchmark run
env:
PREFER_HASH_JOIN: "false"Results will be posted here when complete File an issue against this benchmark runner |
|
Benchmark for this request failed before finishing (Kubernetes reason: Benchmarks requested: Kubernetes messageFile an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing feat/sort-merge-join-projection (2d55b49) to 40c208e (merge-base) diff Run configurationrun benchmark tpch
env:
PREFER_HASH_JOIN: "false"CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #24517 +/- ##
==========================================
+ Coverage 81.29% 81.30% +0.01%
==========================================
Files 1116 1117 +1
Lines 395460 395961 +501
Branches 395460 395961 +501
==========================================
+ Hits 321486 321949 +463
- Misses 55167 55185 +18
- Partials 18807 18827 +20 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing feat/sort-merge-join-projection (2d55b49) to 40c208e (merge-base) diff Run configurationrun benchmark tpch10
env:
PREFER_HASH_JOIN: "false"CPU Details (lscpu)Details
Resource Usagetpch10 — base (merge-base)
tpch10 — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing feat/sort-merge-join-projection (2d55b49) to 40c208e (merge-base) diff Run configurationrun benchmark tpcds
env:
PREFER_HASH_JOIN: "false"CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
jayzhan211
left a comment
There was a problem hiding this comment.
Thanks @Dandandan , a few comments are all somethings we have in HashJoin but not there for SMJ
Swapping a projected join left the indices pointing at the columns the other side now occupies, so it emitted the wrong columns. An empty projection also decoded as an absent one, which is a different output schema. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012YiAABcW4WSqij31zz2P6c
|
run benchmarks run tpch tpcds tpch10 |
|
Benchmark for this request failed before finishing (Kubernetes reason: Benchmarks requested: Kubernetes messageFile an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/sort-merge-join-projection (dc0e106) to 40c208e (merge-base) diff Run configurationrun benchmark tpch10
env:
PREFER_HASH_JOIN: "false"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/sort-merge-join-projection (dc0e106) to 40c208e (merge-base) diff Run configurationrun benchmark tpch
env:
PREFER_HASH_JOIN: "false"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing feat/sort-merge-join-projection (dc0e106) to 40c208e (merge-base) diff Run configurationrun benchmark tpcds
env:
PREFER_HASH_JOIN: "false"Results will be posted here when complete File an issue against this benchmark runner |
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012YiAABcW4WSqij31zz2P6c
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing feat/sort-merge-join-projection (dc0e106) to 40c208e (merge-base) diff Run configurationrun benchmark tpch
env:
PREFER_HASH_JOIN: "false"CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing feat/sort-merge-join-projection (dc0e106) to 40c208e (merge-base) diff Run configurationrun benchmark tpch10
env:
PREFER_HASH_JOIN: "false"CPU Details (lscpu)Details
Resource Usagetpch10 — base (merge-base)
tpch10 — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing feat/sort-merge-join-projection (dc0e106) to 40c208e (merge-base) diff Run configurationrun benchmark tpcds
env:
PREFER_HASH_JOIN: "false"CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
Which issue does this PR close?
SortMergeJoinExec#24518Rationale for this change
SortMergeJoinExecemits every column of both inputs.We can add projection support to only produce the output columns (saving some compute / copy).
The cast column is only there to be joined on, and every operator above the join
carries it until the projection removes it.
What changes are included in this PR?
SortMergeJoinExectakes an optional projection, set withwith_projection, the sameshape as
HashJoinExec's.try_swapping_with_projectionembeds the projection into the join when it cannot pushit into the children, so the query above becomes:
The change doesn't bring a big speedup but helps aligning with other join types and helping simplify join optimization in other areas (e.g. join enumeration).
Are these changes tested?
Yes:
the existing pushdown cannot handle
ProjectionExecin four placesAre there any user-facing changes?
Explain / proto changes.