fix(planner): fix range-pair NULL adjustment under outer joins - #1981
fix(planner): fix range-pair NULL adjustment under outer joins #1981yjhjstz wants to merge 1 commit into
Conversation
…amp join selectivities
On an assert-enabled build the following query crashed the QD with
FailedAssertion("pselec >= 0.0 && pselec <= 1.0", costsize.c:5506):
CREATE TABLE m1(c0 inet);
CREATE TABLE m2(c0 inet);
INSERT INTO m2 VALUES ('88.147.138.141'), ('76.163.212.11'), ('214.10.65.144');
ANALYZE m1, m2;
SELECT COUNT(*) FROM ONLY m1 LEFT OUTER JOIN m2 ON true
WHERE (m1.c0 IS NOT NULL)
OR (m2.c0 BETWEEN SYMMETRIC '75.175.243.19' AND '230.9.216.68');
The pushed-down WHERE clause is an OR of an IS NOT NULL test on the
outer side (selectivity 0.999, no stats on the empty table) and a range
pair on the nullable side. When clauselist_selectivity() merges the
range pair (hibound + lobound - 1) it adds back the column's null
fraction to undo the double exclusion of NULLs, and it does so by
calling nulltestsel(IS_NULL, ..., jointype). GPDB's nulltestsel()
special-cases IS NULL under an outer join and returns 0.5 (a guess for
the anti-join "WHERE inner.col IS NULL" pattern, removed upstream in
e006a24), so the range pair comes out as 0.99 + 0.5 = 1.49 instead of
0.99 + 0.0. Combining that with the IS NOT NULL arm via
s1 + s2 - s1*s2 gives 1.00049, which adjust_selectivity_for_nulltest()
asserts on.
Two changes:
1. In clauselist_selectivity_ext(), ask nulltestsel() for the column's
real null fraction by passing JOIN_INNER. The range-pair correction
is a statistical adjustment, not an IS NULL predicate evaluated at
the join level, so the outer-join guess never belonged here. Before
this, every range pair evaluated under an outer join was inflated by
an absolute 0.5; after it, a LEFT JOIN ON clause with a range
condition gets the same estimate as the equivalent inner join. This
matches upstream behaviour, where nulltestsel() ignores jointype.
2. Selectivities are probabilities, so also clamp jselec and pselec to
[0, 1] in calc_joinrel_size_estimate() before handing them to
adjust_selectivity_for_nulltest(), so round-off in the OR combination
can never trip the assertion again.
Add the SQLancer query to bfv_planner as a regression test.
Fixes apache#1950
leborchuk
left a comment
leborchuk
left a comment
There was a problem hiding this comment.
LGTM
It's amazing.
e006a24 (Tom Lane, 2008-08-14, PG 8.4, "Implement SEMI and ANTI joins") does contain exactly this fix in src/backend/utils/adt/selfuncs.c:
-
if (IS_OUTER_JOIN(jointype) && nulltesttype == IS_NULL) -
return (Selectivity) 0.5;
Tom added it in 0ee5a39 (2007-08-31, PG 8.3) as an explicit band-aid for t1 LEFT JOIN t2 USING(key) WHERE t2.key IS NULL, saying "I went with 0.5 for lack of a better idea." He removed it in e006a24 because real ANTI-join detection made it unnecessary.
Cloudberry's behaviour was not changed 16 years ago. GPDB kept the hack when it cherry-picked that same commit in fe2eb2c (2017-05-09), adding the note that survives verbatim in our tree at src/backend/utils/adt/selfuncs.c:1719:
▎ GPDB_84_MERGE_NOTE: Following hack is removed in the upstream commit e006a24. However, removing this causes cost differences for some ICG queries. Hence, keeping the hack in GPDB
Time is come. The bug lived with us more then 18 years. Now he's an adult, and we can finally get rid of him.
2 participants
On an assert-enabled build the following query crashed the QD with FailedAssertion("pselec >= 0.0 && pselec <= 1.0", costsize.c:5506):
CREATE TABLE m1(c0 inet);
CREATE TABLE m2(c0 inet);
INSERT INTO m2 VALUES ('88.147.138.141'), ('76.163.212.11'), ('214.10.65.144');
ANALYZE m1, m2;
SELECT COUNT(*) FROM ONLY m1 LEFT OUTER JOIN m2 ON true
WHERE (m1.c0 IS NOT NULL)
OR (m2.c0 BETWEEN SYMMETRIC '75.175.243.19' AND '230.9.216.68');
The pushed-down WHERE clause is an OR of an IS NOT NULL test on the outer side (selectivity 0.999, no stats on the empty table) and a range pair on the nullable side. When clauselist_selectivity() merges the range pair (hibound + lobound - 1) it adds back the column's null fraction to undo the double exclusion of NULLs, and it does so by calling nulltestsel(IS_NULL, ..., jointype). GPDB's nulltestsel() special-cases IS NULL under an outer join and returns 0.5 (a guess for the anti-join "WHERE inner.col IS NULL" pattern, removed upstream in e006a24), so the range pair comes out as 0.99 + 0.5 = 1.49 instead of 0.99 + 0.0. Combining that with the IS NOT NULL arm via s1 + s2 - s1*s2 gives 1.00049, which adjust_selectivity_for_nulltest() asserts on.
Two changes:
In clauselist_selectivity_ext(), ask nulltestsel() for the column's real null fraction by passing JOIN_INNER. The range-pair correction is a statistical adjustment, not an IS NULL predicate evaluated at the join level, so the outer-join guess never belonged here. Before this, every range pair evaluated under an outer join was inflated by an absolute 0.5; after it, a LEFT JOIN ON clause with a range condition gets the same estimate as the equivalent inner join. This matches upstream behaviour, where nulltestsel() ignores jointype.
Selectivities are probabilities, so also clamp jselec and pselec to [0, 1] in calc_joinrel_size_estimate() before handing them to adjust_selectivity_for_nulltest(), so round-off in the OR combination can never trip the assertion again.
Add the SQLancer query to bfv_planner as a regression test.
Fixes #1950
What does this PR do?
Type of Change
Breaking Changes
Test Plan
make installcheckmake -C src/test installcheck-cbdb-parallelImpact
Performance:
User-facing changes:
Dependencies:
Checklist
Additional Context
CI Skip Instructions