From f6d8235a823c08933ed5eda2759fa90015e6d8ea Mon Sep 17 00:00:00 2001 From: Mark Kreyman Date: Wed, 12 Aug 2026 11:32:32 -0600 Subject: [PATCH] Make the advisory smoke job legible as advisory From handoff loopctl#675, which is a correction of something I got wrong earlier today and verified against the run data before acting on it. Post-deploy Smoke gates nothing. It is needs: [deploy], appears in no other job's needs, and is not among master's eight required checks. But when it fails it renders the same red X as a run where a GATING job failed and Deploy was skipped, because red is a property of the RUN while "did we ship" is a property of a JOB inside it, and gh run list shows only the former. That produced a real misdiagnosis. I reviewed the day's CI and concluded "every deploy blocked today was blocked by the gate misfiring", filing a three-for-three scoreboard. Checking the runs: 31571624060 and 31572377582 both had Deploy success with this job red - releases that were already live and serving. Exactly one deploy was blocked, by the toolchain fetch. Two of my three were this ambiguity. The cost is not bookkeeping. "The gate misfires" argues for LOOSENING gates: raise budgets, add retries, downgrade to advisory. That was right for the cold-start latency defect and exactly backwards for the toolchain fetch, where the build genuinely could not produce assets and the fix was to remove the network dependency. Conflate the two shapes a few more times and you get a gate that passes everything. Two changes, both presentation only. Nothing about what the check measures or when it runs is touched. The job name now carries its own status, so the summary line says it without anyone opening the run. Safe to rename, verified: it is not one of the eight required status checks, so branch protection is unaffected. On failure the script says which shape of failure it is. The same script runs in two jobs with opposite consequences - pre-deploy GATES, post-deploy does not - so the message is keyed on a SMOKE_CONTEXT the post-deploy job sets and the pre-deploy job deliberately does not. Telling an operator "nothing was blocked" on the gating job would be false. Verified both ways: the note renders under post-deploy and is absent otherwise, and both still exit 1. --- .github/workflows/ci.yml | 24 +++++++++++++++++++++++- docs/runbooks/post-deploy-smoke.md | 12 +++++++++++- scripts/smoke.sh | 12 ++++++++++++ 3 files changed, 46 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b12848f2..44ab073f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1060,7 +1060,24 @@ jobs: FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }} smoke: - name: Post-deploy Smoke + # The name carries the job's own status, because the run summary cannot (#675). This job + # gates NOTHING — it is `needs: [deploy]` and appears in no other job's `needs`, and it + # is not among master's required checks — but when it fails, `gh run list` shows the same + # red X as a run where a GATING job failed and `Deploy` was skipped. Red is a property of + # the RUN; "did we ship" is a property of a JOB inside it. + # + # That cost a real misreading on 2026-08-12: a session reviewing the day's CI concluded + # "every deploy blocked today was blocked by the gate misfiring" and filed a + # three-for-three scoreboard. Two of those three were THIS job failing on releases that + # had already deployed and were serving; exactly one deploy was blocked. + # + # The cost is not bookkeeping. "The gate misfires" argues for LOOSENING gates — raise + # budgets, add retries, downgrade to advisory — which was right for the cold-start + # latency defect (#664) and exactly backwards for the toolchain fetch (#674), where the + # build genuinely could not produce assets and the fix was to remove the network + # dependency. Conflate the two shapes a few more times and you get a gate that passes + # everything. + name: Post-deploy Smoke (advisory — release already shipped) runs-on: ubuntu-latest # Runs AFTER the Fly deploy so it validates the freshly-deployed release, not # the old one. It does NOT gate anything (deploy already happened) — it is a @@ -1083,4 +1100,9 @@ jobs: env: BASE_URL: https://loopctl.com LOOPCTL_SMOKE_KEY: ${{ secrets.LOOPCTL_SMOKE_KEY }} + # Tells smoke.sh which of its two jobs this is, so a failure can say whether it + # BLOCKED the release (pre-deploy) or merely reports on one already serving + # (here). The pre-deploy job deliberately does NOT set this — its failures gate, + # and telling an operator "nothing was blocked" there would be false. + SMOKE_CONTEXT: post-deploy run: bash scripts/smoke.sh diff --git a/docs/runbooks/post-deploy-smoke.md b/docs/runbooks/post-deploy-smoke.md index 0d418a72..48f1a043 100644 --- a/docs/runbooks/post-deploy-smoke.md +++ b/docs/runbooks/post-deploy-smoke.md @@ -235,7 +235,17 @@ There are now TWO smoke jobs running the same `scripts/smoke.sh`: | Job | Runs on | Probes | Can be a required check? | |---|---|---|---| | `Pre-deploy Smoke (current release)` | every PR + every master push | the release already live | **yes** | -| `Post-deploy Smoke` | master push, after deploy | the release just shipped | **no** | +| `Post-deploy Smoke (advisory — release already shipped)` | master push, after deploy | the release just shipped | **no** | + +The job carries "advisory" in its NAME on purpose (#675). It gates nothing, but a failure +renders the same red X as a run where a gating job failed and `Deploy` was skipped — and +`gh run list` shows only the run's colour, never which job inside it went red. That +ambiguity produced a real misdiagnosis on 2026-08-12: two advisory smoke failures on +already-serving releases were counted as blocked deploys, supporting a conclusion that "the +gates are misfiring". Loosening gates is the right response to a genuine false positive and +the wrong response to a build that truly could not produce assets, so the two shapes must +stay tellable apart. On failure the script now says so directly, and names rollback rather +than re-run as the operator's decision. The post-deploy job cannot be made a required branch-protection check. It only runs on a master push, i.e. *after* the merge, so it never reports on a PR's head SHA — adding it to diff --git a/scripts/smoke.sh b/scripts/smoke.sh index 44c43c0c..01bb4531 100755 --- a/scripts/smoke.sh +++ b/scripts/smoke.sh @@ -515,5 +515,17 @@ if [ "$FAILURES" -eq 0 ]; then exit 0 else echo "$NO $FAILURES smoke check(s) failed" + # Say which SHAPE of failure this is, because the run summary cannot (#675). The same + # script runs in two jobs with opposite consequences: pre-deploy GATES (deploy needs it, so + # a red one stopped the release), post-deploy does NOT (the release already shipped and is + # serving). Both render as an identical red X, and reading one as the other led to a + # documented misdiagnosis — "the deploy was blocked" about a release that was live. + if [ "${SMOKE_CONTEXT:-}" = "post-deploy" ]; then + echo + echo " NOTE: this check is ADVISORY. The deploy already succeeded and the release at" + echo " ${BASE_URL} is LIVE and serving traffic right now — nothing was blocked." + echo " Your decision is whether to ROLL BACK, not whether to re-run this job." + echo " See docs/runbooks/post-deploy-smoke.md." + fi exit 1 fi