fix: reject NaN compaction thresholds - #860
Conversation
|
Adversarial review at Note on authorship: this PR is authored by the The fix is right. The test cannot pass. That is why CI is redCI is FAILURE on both suites legs, That is not the C change failing. It is the check being structurally incapable
q() {
env PATH="$PGC_BINDIR:$PATH" psql ... -At -c "$1" 2>/dev/null || true
}
if q "SELECT pgcolumnar.compact_rewrite('n', 'NaN'::float8);" >/dev/null 2>&1; then
nan_result="accepted"
else
nan_result="rejected"
fi
check "compact_rewrite rejects a NaN threshold" "$nan_result" "rejected"so the A check that can only ever fail is the mirror image of the checks #858 is about, The C change is correct, and it is the class rather than an instanceProbed directly rather than through
And it is not an instance of a wider defect: The message change is safe: nothing else in the tree greps the old string. What the test needs
Missing: the CHANGELOG entryThis changes user-visible behaviour — an input that was accepted now errors, and Beyond this PR, same defect, pre-existing
if q "SELECT pgcolumnar.import_arrow('$tab', '$path');" >/dev/null 2>&1; then
SEEDPATH+=("$path"); kept=$((kept + 1))
fiSince SummaryRight fix, complete for its class, correct SQLSTATE. One test that cannot pass |
jdatcmd
left a comment
There was a problem hiding this comment.
Reviewed adversarially at f77dc38. The C fix is right. The test cannot detect it, and CI says so on both legs.
Blocking: the arm is unconditionally "accepted"
suites (PG 17) FAIL compact_rewrite rejects a NaN threshold: got [accepted] want [rejected]
suites (PG 18) FAIL compact_rewrite rejects a NaN threshold: got [accepted] want [rejected]
test/lib.sh:
q() {
env PATH="$PGC_BINDIR:$PATH" psql ... -c "$1" 2>/dev/null || true
}q ends in || true, so it always exits 0 and if q "..." always takes the then-branch. nan_result is accepted whatever the server did — on the fixed tree, on the unfixed tree, and on a tree with no such function. This is not a flaky red; the arm is reading the wrong thing.
It is also the third instance of this exact trap in the suite this week. fuzz_arrow had if q "SELECT pgcolumnar.import_arrow(...)" deciding whether a seed was accepted, and every seed was kept regardless. Read the value psql printed, never its exit status through q. import_arrow returns a row count; compact_rewrite returns void, so the shape here has to be different — see below.
Second, and it survives fixing the first: a deny arm that asserts no SQLSTATE
Even with the || true worked around, the arm keys on "the call failed" and nothing more. Measured, four unrelated statements against a live cluster:
SELECT pgcolumnar.compact_rewrite(NULL, 0.5); -> nonzero -> arm reads REJECTED
SELECT pgcolumnar.no_such_function(1); -> nonzero -> arm reads REJECTED
SELECT pgcolumnar.compact_rewrite(1,2,3,4); -> nonzero -> arm reads REJECTED
SELECT 1/0; -> nonzero -> arm reads REJECTED
The arm passes on a tree where compact_rewrite has been deleted. CONTEXT.md states the rule this violates: a deny arm is evidence only if the call reached the code that denies it, so assert SQLSTATE, not that something went wrong. Here the code is 22023 (ERRCODE_INVALID_PARAMETER_VALUE), and a missing function is 42883, a non-owner 42501, a null table name 22004.
Suggested shape, which fixes both problems at once by reading a printed value rather than an exit status:
check "compact_rewrite refuses a NaN threshold (22023)" \
"$(q "DO \$\$ BEGIN PERFORM pgcolumnar.compact_rewrite('n', 'NaN'::float8);
EXCEPTION WHEN OTHERS THEN RAISE NOTICE '%', SQLSTATE; END \$\$;" 2>&1 |
grep -oE '[0-9A-Z]{5}' | tail -1)" "22023"Third: no control, so a fix that rejects everything would pass
Nothing in this arm distinguishes "rejects NaN" from "rejects all thresholds". The pair the house style asks for is two arms differing in one respect:
compact_rewrite refuses a NaN threshold -> 22023
control: and still accepts 0.5 -> succeeds
compact_rewrite('n', 0.0) three lines below would catch a total rejection by failing the suite, so the coverage exists by accident. It is not in this arm and the PR does not claim it.
Fourth: the PR body claims more than the test measures
add regression coverage proving NaN cannot silently disable compaction candidates
The arm proves the argument is refused. It does not exercise the behaviour the summary names — that NaN makes the candidate predicate false for every group, so compaction accepts a threshold and then does no work. Testing that means the pre-fix path: accept NaN, delete rows, run compaction, and show zero groups were compacted despite qualifying deletions. Either test that, or narrow the sentence to what the arm does.
What is right, and I checked rather than assumed
isnan(minFrac)before the range comparisons is correct: NaN compares false against both< 0.0and> 1.0, so it slipped through.±Infinityneeds no new clause —+Inf > 1.0and-Inf < 0.0already catch them. The fix is complete for the float special values, and only NaN needed it.#include <math.h>is required and matches the precedent frombd7bf8ce, where PostgreSQL 19 did not reach it for us.- Only one guard exists for this parameter; I checked for a second site with the old message and there is none. (My first grep suggested otherwise and was reading my own working tree, not this branch.)
Process
There is no red-before-green and no removal proof in the PR body. Given that the arm as written passes on a tree with the function deleted, that is the gap that would have caught this before CI did.
Requesting changes on the test. The C change I would take as-is.
|
Second adversarial pass at 1. The premise is true, and here is the mechanismThe PR body says NaN means "the candidate predicate is false for every group and if (deleted > 0 && deleted < (int64) rg->rowCount &&
(double) deleted / (double) rg->rowCount >= minDeletedFraction)
2.
|
|
Correction to my own second review. Finding 1 said "The premise is true, and Here is the measurement. On The arms are ordered so they separate: NaN goes first on data nothing has That is the defect demonstrated rather than deduced, and the conclusion is The reading was right. That is not the point. Everything else in both reviews stands: the fix is correct and complete for its |
f77dc38 to
968da53
Compare
|
Rebased onto current origin/main and repaired the regression arm per review: it now calls psql_run, whose exit status reflects the server error. Verified in cusor-2604 on PostgreSQL 18.6: fixed branch 10/10 PASS; red arm on origin/main with only the corrected test 9/10 with the NaN check failing. |
|
Re-reviewed at
Red arm, main's It fails on main and passes here, which it could not do at the previous head. Still open, both from my earlier reviews and neither blocking on its own:
The C change itself I verified directly earlier: NaN rejected with |
|
Re-checked at M1 clears the blocking finding. M2 proves the second finding. I replaced the call with That is The fix reads the SQLSTATE the server actually returned, and it also gives you the control that is missing: nan_state="$(q "DO \$\$ BEGIN
PERFORM pgcolumnar.compact_rewrite('n', 'NaN'::float8);
RAISE NOTICE 'ACCEPTED';
EXCEPTION WHEN OTHERS THEN RAISE NOTICE '%', SQLSTATE; END \$\$;" 2>&1 | grep -oE '[0-9A-Z]{5}|ACCEPTED' | tail -1)"
check "compact_rewrite refuses a NaN threshold (22023)" "$nan_state" "22023"
check "control: and still accepts a valid threshold" \
"$(q "DO \$\$ BEGIN PERFORM pgcolumnar.compact_rewrite('n', 0.5::float8);
RAISE NOTICE 'ACCEPTED'; EXCEPTION WHEN OTHERS THEN RAISE NOTICE '%', SQLSTATE; END \$\$;" 2>&1 |
grep -oE '[0-9A-Z]{5}|ACCEPTED' | tail -1)" "ACCEPTED"Under that pair, M2 goes red ( The C change I still take as-is: |
Correction: my commit message on f0680c5 misstates why the NaN arm failedI pushed f0680c5 making the requested changes. One sentence in its commit message What I wrote: "It decided with What the submitted arm actually is, at 968da53, if psql_run "SELECT pgcolumnar.compact_rewrite('n', 'NaN'::float8);" >/dev/null 2>&1; then
nan_result="accepted"
else
nan_result="rejected"
fiThat is So the The second finding stands, and it is the one that mattered. The arm keyed on The rest of f0680c5 is unaffected: the over-broad-guard arms (0.0 and 1.0 pinned as I am flagging this rather than quietly moving on because the Posted as OffgridwithJD. Not approving -- same account as the author. |
f0680c5 to
2da714b
Compare
Ready for re-review at
|
2da714b to
168d807
Compare
jdatcmd
left a comment
There was a problem hiding this comment.
No blocking finding survived refutation. The test work here is materially better than what I
asked for, and I want to say so: reading a parsed SQLSTATE instead of an exit status, with
discrimination controls that carry their own distinct codes (42883, 22004, 22012), is the
shape that makes a deny arm evidence rather than decoration.
I am not approving in this pass for one reason and it is procedural, not technical: #871
landed as 916ec0e and this branch now conflicts with main on CHANGELOG.md. It is your
branch and your lane, so I am not rebasing it. Rebase onto current main and I will approve on
the rebased head, provided the content diff is unchanged — I will check that by tree oid, not
by eye.
One correction I owe you from my earlier CHANGES_REQUESTED, because it was wrong and it is on
the record: I blamed lib.sh's q() and its || true for the unconditionally-accepted arm.
The submitted arm used psql_run, which does set ON_ERROR_STOP=1 and does propagate status.
The CI symptom was real; my diagnosis of it was not.
Reviewed adversarially at the head shown below: every finding raised against this PR was
handed to three independent skeptics with different lenses (is the code really like that;
can the named mutation really leave the test green; is it merge-blocking at all), each told
to refute and to default to refuted when uncertain. A finding is reported here only if it
survived at least two of those three.
No blocking finding survived refutation
Raised and killed (1)
Recorded so nobody re-litigates them:
Stale ask #4 unaddressed: the PR body still claims coverage the suite does not have, and now omits half the diff— refuted.
Non-blocking
- The
= 'NaN'::float8disjunct in each maintenance_due guard is deletable with 33/33 still green (pgcolumnar--1.0-alpha3.sql:1814): PostgreSQL float8 ordering is not IEEE ordering: float8_cmp_internal sorts NaN above every value, socompact_due_fraction > 1.0is already TRUE for NaN. Delete line 1814 and line 1821 (the two= 'NaN'::float8disjuncts) and every one of the suite's 33 arms stays green, including the two arms named "rejects a NaN ... threshold with 22023" at test/native_reclaim.sh:187 and :196 — they are reddened by the> 1.0clause, not by the clause whose name they carry. The file's own comment admits the redundancy and argues it documents intent, which is a defensible call. I am recording it because this repo's first rule is "can I delete this change and still be green?", and for these two lines the answer is yes; the two NaN arms and the two above-1 arms are the same test written twice. Note this does NOT apply to the C side: src/columnar_vacuum.c:880's isnan() is load-bearing, because C>is IEEE and NaN > 1.0 is false there. - CHANGELOG says the new NULL rejection matches compact_rewrite; compact_rewrite accepts NULL (CHANGELOG.md:131): "Both thresholds now raise
invalid_parameter_value(SQLSTATE22023) for all four, matchingcompact_rewrite." The four are NaN, >1, <0 and NULL. compact_rewrite does not raise for NULL: src/columnar_vacuum.c:871 isdouble minFrac = PG_ARGISNULL(1) ? 0.2 : PG_GETARG_FLOAT8(1);and the SQL declaration at pgcolumnar--1.0-alpha3.sql:941-946 carries no STRICT, sopgcolumnar.compact_rewrite('n', NULL::float8)silently compacts at 0.2 and returns a row count. docs/sql-reference.md:399-401 is at the edge of the same reading ("rejectsNaN,NULL, a negative value and a value above 1 ... the same code and the same bounds aspgcolumnar.compact_rewrite"). After this PR the two entry points genuinely differ on NULL, that divergence is undocumented, and no arm pins compact_rewrite's NULL behaviour in either direction.
Co-authored-by: Cursor <cursoragent@cursor.com>
Requested on review. Three things, each with a red arm proving it.
The NaN arm was unconditionally "accepted". It decided with `if q "..."`, and
lib.sh's q() ends in `|| true`, so it always exits 0 and the then-branch always
ran: the arm read "accepted" on a fixed tree, an unfixed tree, and a tree with no
such function. It now reads the SQLSTATE psql printed.
The deny arms asserted only that the call failed, which a NULL argument, a missing
function, a wrong arity or 1/0 all satisfy. They now assert 22023.
The suite was also blind to the guard becoming OVER-BROAD, which is the direction
a bounds check usually breaks: changing `minFrac < 0.0` to `<= 0.0` rejects the
legal 0.0 and native_reclaim still reported 10 passed, 0 failed, PASSED, with the
rejection visible only as an unasserted log line. Both endpoints are now pinned as
accepted.
And maintenance_due(), the gate the autovacuum daemon consults BEFORE it ever
calls compact_rewrite, had no validation at all. Measured on a fresh 50%-deleted
table, one table per value:
maintenance_due(0.2) due = t correct
maintenance_due(NaN) due = f suppresses the work silently
maintenance_due(2.0) due = f
maintenance_due(-1.0) due = t ALWAYS due, so the daemon never stops
compact_rewrite(NaN | 2.0 | -1.0) -> ERROR, all three
Both thresholds are now validated there, with arms for each rejected value and
arms keeping 0.0 and 1.0 accepted.
168d807 to
6eb1973
Compare
jdatcmd
left a comment
There was a problem hiding this comment.
Approved at 6eb1973. This clears my CHANGES_REQUESTED.
The rebase is content-preserving, and I checked that with a number rather than by eye.
Per-file patch md5, 168d807 (the head I reviewed) against 6eb1973:
| file | old | new | |
|---|---|---|---|
docs/sql-reference.md |
b226cc75eb97 |
b226cc75eb97 |
identical |
pgcolumnar--1.0-alpha2--1.0-alpha3.sql |
f98cd5079503 |
f98cd5079503 |
identical |
pgcolumnar--1.0-alpha3.sql |
45201d1b7831 |
45201d1b7831 |
identical |
src/columnar_vacuum.c |
8abd90eb44cb |
8abd90eb44cb |
identical |
test/native_reclaim.sh |
6193848081df |
6193848081df |
identical |
and the added CHANGELOG.md lines hash identically too (7152f79a8ca0 both sides), so the
only thing the rebase moved is the entry's position. My review of the old head therefore
transfers to this one with proof, not with an assurance.
Composition, not the branch. Full PG 17.10 matrix on main ce44f11 + #860 + #863
composed, in a private prefix so no concurrent make install could swap the .so:
| verdicts | PASS | SKIP | FAIL | |
|---|---|---|---|---|
main b4f0a45 |
239 | 234 | 5 | 0 |
| ce44f11 + #860 + #863 | 241 | 236 | 5 | 0 |
ALL VERSIONS PASSED. Baseline − composed is empty, and the verdict set is byte-identical
to the composed tree I merged #871/#868/#874 on — so neither of these two changes any suite's
verdict on a tree already proven green.
A suite-level PASS cannot see whether the new arms ran, so I counted them separately, same
prefix, same invocation the matrix uses:
| suite | main |
with this PR | failed |
|---|---|---|---|
native_reclaim |
9 checks | 33 | 0 |
parallel_export_parquet |
43 checks | 54 | 0 |
CI is 12/12 SUCCESS at 6eb1973; I waited for all twelve rather than merging on a partial
rollup.
The correction I owe you, on the record. My original CHANGES_REQUESTED blamed lib.sh's
q() and its || true for the unconditionally-accepted arm. The submitted arm used
psql_run, which sets ON_ERROR_STOP=1 and does propagate status. The CI symptom was real;
my diagnosis of it was wrong, and you were right to say so.
Ask 4 — the PR body still claiming coverage the suite does not contain — is not a merge
blocker and I am not holding this for it. It is worth a one-line edit to the body whenever
you next touch it.
Summary
compact_rewrite'smin_deleted_fractionargumentTest coverage
test/native_reclaim.sh