From 1dc632fb86a87119bd86bb8375e1ff4e33ac2819 Mon Sep 17 00:00:00 2001 From: jnasbyupgrade Date: Fri, 31 Jul 2026 18:38:55 -0500 Subject: [PATCH] Update docs: make test now returns non-zero on regressions (pgxntool 2.3.0) CLAUDE.md, RELEASE.md, and a ci.yml comment all claimed 'make test' never exits non-zero on failures (pgxntool marked installcheck .IGNORE). pgxntool 2.3.0 fixed this: test's own recipe now checks regression.diffs and exits non-zero when it's non-empty. verify-results remains the stricter, documented gate (it also confirms test actually ran), but the old blanket claim about make test is no longer accurate. Verified locally: make test exits 0 when clean and non-zero (with the diff printed) when a test is made to fail. --- .github/workflows/ci.yml | 7 ++++--- CLAUDE.md | 8 +++++--- RELEASE.md | 6 +++--- 3 files changed, 12 insertions(+), 9 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 11b225c..f3b6712 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -310,9 +310,10 @@ jobs: make check-relkind-source # verify-results is the real gate: base.mk declares `verify-results: # test`, so this runs the suite via its `test` prerequisite and then - # checks the pgtap/regression.diffs. A bare `make test` is redundant here - # and would not gate anyway -- it never exits non-zero on regressions - # (pgxntool marks installcheck `.IGNORE`), so failures would pass silently. + # checks the pgtap/regression.diffs. A bare `make test` is redundant here: + # it now also exits non-zero on regressions (pgxntool 2.3.0+), but + # verify-results is still the stricter, documented check, and confirms + # test actually ran. make verify-results # Style linter (https://github.com/Postgres-Extensions/linter, vendored at diff --git a/CLAUDE.md b/CLAUDE.md index 0acd2ad..69234ba 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -14,9 +14,11 @@ and `pgxntool/CLAUDE.md` first. High-value gotchas that those docs explain: - `DATA`, `MODULES`, `DOCS`, and `installcheck` are PGXS variables/targets, not pgxntool's; pgxntool only wraps/seeds them. Don't assume they belong to pgxntool. -- `make test` does **not** return non-zero on test regressions — pgxntool marks - `installcheck` as `.IGNORE`. To actually detect failures, use `make verify-results` - (or inspect `test/regression.diffs`). This is why CI must gate on `verify-results`. +- `make test` returns non-zero on test regressions (pgxntool 2.3.0+): `installcheck` + itself is still marked `.IGNORE`, but `test`'s own recipe now checks + `test/regression.diffs` and exits non-zero if it's non-empty. `make verify-results` + remains the stricter, documented CI gate (it also confirms `test` actually ran), but + a bare `make test` failing locally is now a real signal too. - `test/install/*.sql` runs once, committed, before the suite in the same `pg_regress` invocation, so its state persists into every test. `test/build/*.sql` are separate build sanity checks run before the suite. diff --git a/RELEASE.md b/RELEASE.md index 8a0cb4b..6ee6d87 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -38,9 +38,9 @@ what that version actually shipped. - [ ] Open issues/PRs for this release reviewed, merged or deferred. - [ ] CI green on all supported PostgreSQL versions (the `all-checks-passed` job on master). - [ ] Locally: `make verify-results` passes. It depends on `test` (so it runs the suite - first, then gates on the results). `make test` alone is non-gating — pgxntool marks - `installcheck` `.IGNORE`, so it never returns non-zero on a regression; only - `verify-results` (which inspects `test/regression.diffs`) is a real gate. + first, then gates on the results). `make test` itself also returns non-zero on a + regression as of pgxntool 2.3.0, but `verify-results` remains the stricter, + documented gate (it also confirms `test` actually ran). ## 3. Decide the version and what to track - [ ] Pick the new version (semantic versioning).