Make the advisory smoke job legible as advisory - #677
Merged
Conversation
From handoff loopctl#675, which is a correction of something I got wrong earlier today and verified against the run data before acting on it. Post-deploy Smoke gates nothing. It is needs: [deploy], appears in no other job's needs, and is not among master's eight required checks. But when it fails it renders the same red X as a run where a GATING job failed and Deploy was skipped, because red is a property of the RUN while "did we ship" is a property of a JOB inside it, and gh run list shows only the former. That produced a real misdiagnosis. I reviewed the day's CI and concluded "every deploy blocked today was blocked by the gate misfiring", filing a three-for-three scoreboard. Checking the runs: 31571624060 and 31572377582 both had Deploy success with this job red - releases that were already live and serving. Exactly one deploy was blocked, by the toolchain fetch. Two of my three were this ambiguity. The cost is not bookkeeping. "The gate misfires" argues for LOOSENING gates: raise budgets, add retries, downgrade to advisory. That was right for the cold-start latency defect and exactly backwards for the toolchain fetch, where the build genuinely could not produce assets and the fix was to remove the network dependency. Conflate the two shapes a few more times and you get a gate that passes everything. Two changes, both presentation only. Nothing about what the check measures or when it runs is touched. The job name now carries its own status, so the summary line says it without anyone opening the run. Safe to rename, verified: it is not one of the eight required status checks, so branch protection is unaffected. On failure the script says which shape of failure it is. The same script runs in two jobs with opposite consequences - pre-deploy GATES, post-deploy does not - so the message is keyed on a SMOKE_CONTEXT the post-deploy job sets and the pre-deploy job deliberately does not. Telling an operator "nothing was blocked" on the gating job would be false. Verified both ways: the note renders under post-deploy and is absent otherwise, and both still exit 1.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #675, which is a correction of something I got wrong earlier today. I verified
it against the run data before acting on it.
The ambiguity
Post-deploy Smokegates nothing —needs: [deploy], in no other job'sneeds, and notamong master's eight required checks (verified via the branch-protection API). But when it
fails it renders the same red X as a run where a gating job failed and
Deploywasskipped. Red is a property of the RUN; "did we ship" is a property of a JOB inside it, and
gh run listshows only the former.It already cost a misdiagnosis — mine
I reviewed the day's CI and concluded "every deploy blocked today was blocked by the gate
misfiring", filing a three-for-three scoreboard. The runs say otherwise:
Exactly one deploy was blocked. Two of my three were this ambiguity — both smoke reds
landed on releases already live and serving.
The cost isn't bookkeeping. "The gate misfires" argues for loosening gates — raise
budgets, add retries, downgrade to advisory. That was right for the cold-start latency defect
(#664) and exactly backwards for the toolchain fetch (#674), where the build genuinely
could not produce assets and the correct fix removed the network dependency. Conflate the two
shapes a few more times and you get a gate that passes everything.
Two changes, presentation only
Nothing about what the check measures or when it runs is touched — per #675, the check is
correct and only its presentation is at fault.
1. The job name carries its own status:
Post-deploy Smoke (advisory — release already shipped). Safe to rename — confirmed it is not one of the eight required checks, so branchprotection is unaffected.
2. The failure says which shape it is. The same script runs in two jobs with opposite
consequences — pre-deploy gates (
deployneeds it), post-deploy does not. So the message iskeyed on a
SMOKE_CONTEXTthe post-deploy job sets and the pre-deploy job deliberately doesnot: telling an operator "nothing was blocked" on the gating job would be false.
Verified both ways: the note renders under
post-deploy, is absent otherwise, and both pathsstill
exit 1.I did not take #675's option 3 (open an issue instead of reddening the run) — it trades away
the at-a-glance red that makes anyone look at all, which #675 itself flags.
mix precommitgreen (7,506 tests). Runbook updated to match. Reviewed inline (this session'ssystem prompt forbids dispatching review agents; per CLAUDE.md the gate is satisfied by the
best available reviewer with that stated).