diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b12848f2..44ab073f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1060,7 +1060,24 @@ jobs: FLY_API_TOKEN: ${{ secrets.FLY_API_TOKEN }} smoke: - name: Post-deploy Smoke + # The name carries the job's own status, because the run summary cannot (#675). This job + # gates NOTHING — it is `needs: [deploy]` and appears in no other job's `needs`, and it + # is not among master's required checks — but when it fails, `gh run list` shows the same + # red X as a run where a GATING job failed and `Deploy` was skipped. Red is a property of + # the RUN; "did we ship" is a property of a JOB inside it. + # + # That cost a real misreading on 2026-08-12: a session reviewing the day's CI concluded + # "every deploy blocked today was blocked by the gate misfiring" and filed a + # three-for-three scoreboard. Two of those three were THIS job failing on releases that + # had already deployed and were serving; exactly one deploy was blocked. + # + # The cost is not bookkeeping. "The gate misfires" argues for LOOSENING gates — raise + # budgets, add retries, downgrade to advisory — which was right for the cold-start + # latency defect (#664) and exactly backwards for the toolchain fetch (#674), where the + # build genuinely could not produce assets and the fix was to remove the network + # dependency. Conflate the two shapes a few more times and you get a gate that passes + # everything. + name: Post-deploy Smoke (advisory — release already shipped) runs-on: ubuntu-latest # Runs AFTER the Fly deploy so it validates the freshly-deployed release, not # the old one. It does NOT gate anything (deploy already happened) — it is a @@ -1083,4 +1100,9 @@ jobs: env: BASE_URL: https://loopctl.com LOOPCTL_SMOKE_KEY: ${{ secrets.LOOPCTL_SMOKE_KEY }} + # Tells smoke.sh which of its two jobs this is, so a failure can say whether it + # BLOCKED the release (pre-deploy) or merely reports on one already serving + # (here). The pre-deploy job deliberately does NOT set this — its failures gate, + # and telling an operator "nothing was blocked" there would be false. + SMOKE_CONTEXT: post-deploy run: bash scripts/smoke.sh diff --git a/docs/runbooks/post-deploy-smoke.md b/docs/runbooks/post-deploy-smoke.md index 0d418a72..48f1a043 100644 --- a/docs/runbooks/post-deploy-smoke.md +++ b/docs/runbooks/post-deploy-smoke.md @@ -235,7 +235,17 @@ There are now TWO smoke jobs running the same `scripts/smoke.sh`: | Job | Runs on | Probes | Can be a required check? | |---|---|---|---| | `Pre-deploy Smoke (current release)` | every PR + every master push | the release already live | **yes** | -| `Post-deploy Smoke` | master push, after deploy | the release just shipped | **no** | +| `Post-deploy Smoke (advisory — release already shipped)` | master push, after deploy | the release just shipped | **no** | + +The job carries "advisory" in its NAME on purpose (#675). It gates nothing, but a failure +renders the same red X as a run where a gating job failed and `Deploy` was skipped — and +`gh run list` shows only the run's colour, never which job inside it went red. That +ambiguity produced a real misdiagnosis on 2026-08-12: two advisory smoke failures on +already-serving releases were counted as blocked deploys, supporting a conclusion that "the +gates are misfiring". Loosening gates is the right response to a genuine false positive and +the wrong response to a build that truly could not produce assets, so the two shapes must +stay tellable apart. On failure the script now says so directly, and names rollback rather +than re-run as the operator's decision. The post-deploy job cannot be made a required branch-protection check. It only runs on a master push, i.e. *after* the merge, so it never reports on a PR's head SHA — adding it to diff --git a/scripts/smoke.sh b/scripts/smoke.sh index 44c43c0c..01bb4531 100755 --- a/scripts/smoke.sh +++ b/scripts/smoke.sh @@ -515,5 +515,17 @@ if [ "$FAILURES" -eq 0 ]; then exit 0 else echo "$NO $FAILURES smoke check(s) failed" + # Say which SHAPE of failure this is, because the run summary cannot (#675). The same + # script runs in two jobs with opposite consequences: pre-deploy GATES (deploy needs it, so + # a red one stopped the release), post-deploy does NOT (the release already shipped and is + # serving). Both render as an identical red X, and reading one as the other led to a + # documented misdiagnosis — "the deploy was blocked" about a release that was live. + if [ "${SMOKE_CONTEXT:-}" = "post-deploy" ]; then + echo + echo " NOTE: this check is ADVISORY. The deploy already succeeded and the release at" + echo " ${BASE_URL} is LIVE and serving traffic right now — nothing was blocked." + echo " Your decision is whether to ROLL BACK, not whether to re-run this job." + echo " See docs/runbooks/post-deploy-smoke.md." + fi exit 1 fi