Derive total_byte_size for join statistics - #24536
Conversation
A join knew how many rows it would emit but not how wide they were: every join type but semi and anti reported no total byte size at all. An operator above it reading that size finds nothing, so `hash_join_single_partition_threshold`, a byte threshold, falls back to counting rows. The width of an output row follows from the sides the join emits, which the join type already says: both sides for an inner or outer join, the preserved side for a semi or anti join, and one side plus a boolean for a mark join. Multiplying it by the estimated cardinality gives the size. Semi and anti joins keep the size they derive from their column statistics, which knows which columns survive; the width fills a gap rather than replacing a better answer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012YiAABcW4WSqij31zz2P6c
github-actions
Bot
added
core
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #24536 +/- ##
==========================================
+ Coverage 80.87% 81.32% +0.44%
==========================================
Files 1101 1117 +16
Lines 375765 396322 +20557
Branches 375765 396322 +20557
==========================================
+ Hits 303915 322299 +18384
- Misses 53747 55191 +1444
- Partials 18103 18832 +729 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
run benchmarks |
|
Benchmark for this request failed before finishing (Kubernetes reason: Benchmarks requested: Kubernetes messageFile an issue against this benchmark runner |
|
Benchmark for this request failed before finishing (Kubernetes reason: Benchmarks requested: Kubernetes messageFile an issue against this benchmark runner |
|
Benchmark for this request failed before finishing (Kubernetes reason: Benchmarks requested: Kubernetes messageFile an issue against this benchmark runner |
|
run benchmarks |
|
run benchmark tpc-ds |
|
run benchmark tpcds |
|
Benchmark for this request failed before finishing (Kubernetes reason: Benchmarks requested: Kubernetes messageFile an issue against this benchmark runner |
|
Benchmark for this request failed before finishing (Kubernetes reason: Benchmarks requested: Kubernetes messageFile an issue against this benchmark runner |
|
Benchmark for this request failed before finishing (Kubernetes reason: Benchmarks requested: Kubernetes messageFile an issue against this benchmark runner |
Dandandan
changed the title
Dandandan
changed the title
|
Benchmark for this request failed before finishing (Kubernetes reason: Benchmarks requested: Kubernetes messageFile an issue against this benchmark runner |
|
Benchmark for this request failed before finishing (Kubernetes reason: Benchmarks requested: Kubernetes messageFile an issue against this benchmark runner |
3 participants
Which issue does this PR close?
Rationale for this change
Currently it returns total_byte_size: Precision::Absent` for most joins - we can derive it to produce better estimates for nested joins.
What changes are included in this PR?
The width of one output row follows from the sides a join emits, which the join type
already says: both sides for an inner or outer join, the preserved side for a semi or
anti join, and one side plus a boolean for a mark join. Multiplied by the estimated
cardinality, that gives the output size.
Are these changes tested?
Yes. A new unit test covers an inner join summing both sides, a mark join adding one
bit per row, and a semi join keeping its column-derived size.
Benchmarks
TPC-DS SF1: three plans change, each because a join that
now reports a size clears
hash_join_single_partition_thresholdand becomesCollectLeft.No query regressed.
Are there any user-facing changes?
No API change. Joins now report a
total_byte_sizewhere they previously reportednone, which
EXPLAIN ANALYZEand statistics-driven decisions can read.