diff --git a/SPEC.md b/SPEC.md index 2fe0ec5..ca69686 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1212,8 +1212,27 @@ e2e had passed and the result was discarded and re-queued for nothing. The same `existing` was always empty and every run created a brand-new PR instead of updating one. A force-push alone is sufficient and non-destructive: an open PR follows its head ref and picks up the new commit. +**DEC-080 — Caller-template drift is detected, because nothing else can see it.** +`make link-workflows` *copies* the caller templates into a consumer's `.github/workflows/`, and +`devkit-sync` deliberately never writes that directory (its `GITHUB_TOKEN` may not push workflow +files). So the vendored devkit advances while the copied caller stands still, and **nothing at +runtime notices** — both known instances were silent by construction: share-chat could not reach the +`capture_all_traces` input for weeks because its caller predated it, and `actions: write` (DEC-074) +had to be hand-propagated to four repos in four PRs, with the prune degrading to a *warning* in the +meantime rather than failing. +`e2e/ci/check-caller-drift.sh` compares the **executable contract** — triggers, dispatch inputs, +permissions, job ids, `uses` refs, `with` inputs, and the secrets shape — and ignores comments, +whitespace, quoting, key order and prose descriptions. That line is deliberate: a byte-diff would +have fired on all four consumers the moment the devkit rewrote a header comment, and a checker that +cries wolf gets `|| true`'d. Templates resolve to the consumer's own pinned `tests/_testkit` when +available, so each repo is judged against the devkit ref *it* pinned rather than whatever `main` +happens to be. +The self-test is not decoration. It replays both real incidents as positive controls and pins the +cosmetic tolerances as negative controls, so the tempting "fix" for a noisy checker — loosening it +until it stops complaining — fails the suite instead of passing quietly. + > **Numbering note:** DEC-070–072 were originally introduced as code comments only. They are now -> written up above, so this section documents an unbroken 070–079. The lesson is worth keeping: a +> written up above, so this section documents an unbroken 070–080. The lesson is worth keeping: a > decision that exists only as a code comment is invisible to anyone reading the SPEC, and all three > of these were load-bearing (a gate's timeout budget, the image the whole fleet pulls, and a wait > that was flaking half the runs). diff --git a/e2e/ci/check-caller-drift.sh b/e2e/ci/check-caller-drift.sh new file mode 100755 index 0000000..ea801e8 --- /dev/null +++ b/e2e/ci/check-caller-drift.sh @@ -0,0 +1,216 @@ +#!/usr/bin/env bash +# +# Caller-workflow drift check — does a consumer's copied caller still match the +# devkit template it was copied from? +# +# WHY THIS EXISTS +# `make link-workflows` COPIES templates/{plugin-e2e,devkit-sync}.caller.yml +# into a consumer's .github/workflows/. `devkit-sync` then keeps the SUBMODULE +# pin current but deliberately never writes .github/workflows/ (its +# GITHUB_TOKEN may not push workflow files). So the vendored devkit advances +# while the copied caller stands still, and nothing anywhere notices. +# +# That is not hypothetical. Two live examples: +# * share-chat's caller predated the `capture_all_traces` dispatch input by +# weeks — the input existed in the reusable workflow, the repo just could +# not reach it, and no run ever failed to say so. +# * `actions: write` (count-based artifact retention, DEC-074) had to be +# hand-propagated to four repos in four separate PRs, because a reusable +# workflow's permissions are CAPPED BY THE CALLER's grant. The prune step +# degraded to a warning instead of failing — silent, again. +# Both failure modes are invisible at runtime. A checker is the only detector. +# +# WHAT COUNTS AS DRIFT (and why not a byte diff) +# Byte-for-byte would be trivially correct and useless: every comment reflow in +# the devkit — e.g. the DEC-079 rewrite of the devkit-sync header — would fire +# on all consumers at once, and a checker that cries wolf gets `|| true`'d +# within a week. So this compares the EXECUTABLE CONTRACT of the YAML: +# +# compared name, triggers (`on`), permissions, job ids, `uses` refs, +# `with` inputs, `secrets` (inherit vs the explicit key set), +# dispatch input names/types/defaults +# ignored comments, whitespace, quoting, key order, scalar-list order +# (`branches: [main, master]` is a set), and the human-prose +# `description:` of a dispatch input +# +# Rationale for that line: everything on the left changes what CI DOES, and +# both real incidents above are on the left. Everything on the right changes +# only how the file reads. Ignoring `description` is the one judgement call — +# it is UI prose in the Actions run dialog, and letting a wording fix fail four +# repos is exactly how the signal gets ignored. +# +# Drift is reported in three flavours, all fatal, because they rot differently: +# missing template has it, consumer does not — the silent-rot case above +# changed both have it, values differ — a local edit or a stale value +# extra consumer has it, template does not — a local edit that the NEXT +# `make link-workflows` will silently delete +# +# USAGE +# check-caller-drift.sh [consumer-repo-root ...] # default: . +# +# In consumer CI (the vendored copy, no args) it checks the repo it runs in +# against its own pinned devkit templates — so the check advances exactly when +# the submodule does. Locally, pass repo roots to sweep the fleet: +# check-caller-drift.sh ~/src/.../agent-zero-plugin-* +# +# Env: +# TEMPLATES_DIR devkit templates/ dir. Auto-detected when unset, in order: +#