Skip to content

Make the advisory smoke job legible as advisory - #677

Merged
mkreyman merged 1 commit into
masterfrom
ci/make-post-deploy-smoke-legibly-advisory
Aug 12, 2026
Merged

Make the advisory smoke job legible as advisory#677
mkreyman merged 1 commit into
masterfrom
ci/make-post-deploy-smoke-legibly-advisory

Conversation

@mkreyman

Copy link
Copy Markdown
Owner

Addresses #675, which is a correction of something I got wrong earlier today. I verified
it against the run data before acting on it.

The ambiguity

Post-deploy Smoke gates nothing — needs: [deploy], in no other job's needs, and not
among master's eight required checks
(verified via the branch-protection API). But when it
fails it renders the same red X as a run where a gating job failed and Deploy was
skipped. Red is a property of the RUN; "did we ship" is a property of a JOB inside it, and
gh run list shows only the former.

It already cost a misdiagnosis — mine

I reviewed the day's CI and concluded "every deploy blocked today was blocked by the gate
misfiring"
, filing a three-for-three scoreboard. The runs say otherwise:

Run Red job Deploy
31571624060 Post-deploy Smoke (keyword floor 7704ms) success
31572377582 Post-deploy Smoke (knowledge count 6806ms) success
31618467843 attempt 1 Lint → toolchain fetch skipped

Exactly one deploy was blocked. Two of my three were this ambiguity — both smoke reds
landed on releases already live and serving.

The cost isn't bookkeeping. "The gate misfires" argues for loosening gates — raise
budgets, add retries, downgrade to advisory. That was right for the cold-start latency defect
(#664) and exactly backwards for the toolchain fetch (#674), where the build genuinely
could not produce assets and the correct fix removed the network dependency. Conflate the two
shapes a few more times and you get a gate that passes everything.

Two changes, presentation only

Nothing about what the check measures or when it runs is touched — per #675, the check is
correct and only its presentation is at fault.

1. The job name carries its own status: Post-deploy Smoke (advisory — release already shipped). Safe to rename — confirmed it is not one of the eight required checks, so branch
protection is unaffected.

2. The failure says which shape it is. The same script runs in two jobs with opposite
consequences — pre-deploy gates (deploy needs it), post-deploy does not. So the message is
keyed on a SMOKE_CONTEXT the post-deploy job sets and the pre-deploy job deliberately does
not: telling an operator "nothing was blocked" on the gating job would be false.

x 2 smoke check(s) failed

  NOTE: this check is ADVISORY. The deploy already succeeded and the release at
  https://loopctl.com is LIVE and serving traffic right now — nothing was blocked.
  Your decision is whether to ROLL BACK, not whether to re-run this job.

Verified both ways: the note renders under post-deploy, is absent otherwise, and both paths
still exit 1.

I did not take #675's option 3 (open an issue instead of reddening the run) — it trades away
the at-a-glance red that makes anyone look at all, which #675 itself flags.

mix precommit green (7,506 tests). Runbook updated to match. Reviewed inline (this session's
system prompt forbids dispatching review agents; per CLAUDE.md the gate is satisfied by the
best available reviewer with that stated).

From handoff loopctl#675, which is a correction of something I got wrong earlier
today and verified against the run data before acting on it.

Post-deploy Smoke gates nothing. It is needs: [deploy], appears in no other
job's needs, and is not among master's eight required checks. But when it fails
it renders the same red X as a run where a GATING job failed and Deploy was
skipped, because red is a property of the RUN while "did we ship" is a property
of a JOB inside it, and gh run list shows only the former.

That produced a real misdiagnosis. I reviewed the day's CI and concluded "every
deploy blocked today was blocked by the gate misfiring", filing a three-for-three
scoreboard. Checking the runs: 31571624060 and 31572377582 both had Deploy
success with this job red - releases that were already live and serving. Exactly
one deploy was blocked, by the toolchain fetch. Two of my three were this
ambiguity.

The cost is not bookkeeping. "The gate misfires" argues for LOOSENING gates:
raise budgets, add retries, downgrade to advisory. That was right for the
cold-start latency defect and exactly backwards for the toolchain fetch, where
the build genuinely could not produce assets and the fix was to remove the
network dependency. Conflate the two shapes a few more times and you get a gate
that passes everything.

Two changes, both presentation only. Nothing about what the check measures or
when it runs is touched.

The job name now carries its own status, so the summary line says it without
anyone opening the run. Safe to rename, verified: it is not one of the eight
required status checks, so branch protection is unaffected.

On failure the script says which shape of failure it is. The same script runs in
two jobs with opposite consequences - pre-deploy GATES, post-deploy does not - so
the message is keyed on a SMOKE_CONTEXT the post-deploy job sets and the
pre-deploy job deliberately does not. Telling an operator "nothing was blocked"
on the gating job would be false. Verified both ways: the note renders under
post-deploy and is absent otherwise, and both still exit 1.
@mkreyman
mkreyman enabled auto-merge (squash) August 12, 2026 17:34
@mkreyman
mkreyman merged commit 720da0d into master Aug 12, 2026
15 checks passed
@mkreyman
mkreyman deleted the ci/make-post-deploy-smoke-legibly-advisory branch August 12, 2026 17:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant