From 923bbefd907231dcdd6cfefbb0141b94da0fbe4c Mon Sep 17 00:00:00 2001 From: omar-nahhas-agent Date: Wed, 5 Aug 2026 01:05:06 +0100 Subject: [PATCH] feat(ci): detect caller-template drift (DEC-080) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit make link-workflows COPIES the caller templates into a consumer, and devkit-sync deliberately never writes .github/workflows/. So the vendored devkit advances while the copied caller stands still and nothing at runtime notices. Both known instances were silent by construction: - share-chat could not reach the capture_all_traces input for weeks, because its caller predated it; no run ever failed to say so - actions: write (DEC-074) needed hand-propagating to 4 repos in 4 PRs, and until then the prune degraded to a WARNING rather than failing The check compares the EXECUTABLE CONTRACT — triggers, dispatch inputs, permissions, job ids, uses refs, with inputs, secrets shape — and ignores comments, whitespace, quoting, key order and prose descriptions. That line is deliberate: a byte-diff would have fired on all four consumers the moment the devkit rewrote a header comment for DEC-079, and a checker that cries wolf gets || true-d. Templates resolve to the consumers own pinned tests/_testkit when available, so each repo is judged against the devkit ref IT pinned, not whatever main is. The self-test replays both real incidents as positive controls and pins the cosmetic tolerances as negative controls — so the tempting fix for a noisy checker (loosen until quiet) fails the suite instead of passing silently. 10/10 green. Verified against live main: 3 of 4 consumers in sync, share-chat drifted on 4 paths (the capture_all_traces input and its wiring) — the one genuine gap. --- SPEC.md | 21 ++- e2e/ci/check-caller-drift.sh | 216 +++++++++++++++++++++++++++ e2e/ci/test-check-caller-drift.sh | 233 ++++++++++++++++++++++++++++++ 3 files changed, 469 insertions(+), 1 deletion(-) create mode 100755 e2e/ci/check-caller-drift.sh create mode 100755 e2e/ci/test-check-caller-drift.sh diff --git a/SPEC.md b/SPEC.md index 2fe0ec5..ca69686 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1212,8 +1212,27 @@ e2e had passed and the result was discarded and re-queued for nothing. The same `existing` was always empty and every run created a brand-new PR instead of updating one. A force-push alone is sufficient and non-destructive: an open PR follows its head ref and picks up the new commit. +**DEC-080 — Caller-template drift is detected, because nothing else can see it.** +`make link-workflows` *copies* the caller templates into a consumer's `.github/workflows/`, and +`devkit-sync` deliberately never writes that directory (its `GITHUB_TOKEN` may not push workflow +files). So the vendored devkit advances while the copied caller stands still, and **nothing at +runtime notices** — both known instances were silent by construction: share-chat could not reach the +`capture_all_traces` input for weeks because its caller predated it, and `actions: write` (DEC-074) +had to be hand-propagated to four repos in four PRs, with the prune degrading to a *warning* in the +meantime rather than failing. +`e2e/ci/check-caller-drift.sh` compares the **executable contract** — triggers, dispatch inputs, +permissions, job ids, `uses` refs, `with` inputs, and the secrets shape — and ignores comments, +whitespace, quoting, key order and prose descriptions. That line is deliberate: a byte-diff would +have fired on all four consumers the moment the devkit rewrote a header comment, and a checker that +cries wolf gets `|| true`'d. Templates resolve to the consumer's own pinned `tests/_testkit` when +available, so each repo is judged against the devkit ref *it* pinned rather than whatever `main` +happens to be. +The self-test is not decoration. It replays both real incidents as positive controls and pins the +cosmetic tolerances as negative controls, so the tempting "fix" for a noisy checker — loosening it +until it stops complaining — fails the suite instead of passing quietly. + > **Numbering note:** DEC-070–072 were originally introduced as code comments only. They are now -> written up above, so this section documents an unbroken 070–079. The lesson is worth keeping: a +> written up above, so this section documents an unbroken 070–080. The lesson is worth keeping: a > decision that exists only as a code comment is invisible to anyone reading the SPEC, and all three > of these were load-bearing (a gate's timeout budget, the image the whole fleet pulls, and a wait > that was flaking half the runs). diff --git a/e2e/ci/check-caller-drift.sh b/e2e/ci/check-caller-drift.sh new file mode 100755 index 0000000..ea801e8 --- /dev/null +++ b/e2e/ci/check-caller-drift.sh @@ -0,0 +1,216 @@ +#!/usr/bin/env bash +# +# Caller-workflow drift check — does a consumer's copied caller still match the +# devkit template it was copied from? +# +# WHY THIS EXISTS +# `make link-workflows` COPIES templates/{plugin-e2e,devkit-sync}.caller.yml +# into a consumer's .github/workflows/. `devkit-sync` then keeps the SUBMODULE +# pin current but deliberately never writes .github/workflows/ (its +# GITHUB_TOKEN may not push workflow files). So the vendored devkit advances +# while the copied caller stands still, and nothing anywhere notices. +# +# That is not hypothetical. Two live examples: +# * share-chat's caller predated the `capture_all_traces` dispatch input by +# weeks — the input existed in the reusable workflow, the repo just could +# not reach it, and no run ever failed to say so. +# * `actions: write` (count-based artifact retention, DEC-074) had to be +# hand-propagated to four repos in four separate PRs, because a reusable +# workflow's permissions are CAPPED BY THE CALLER's grant. The prune step +# degraded to a warning instead of failing — silent, again. +# Both failure modes are invisible at runtime. A checker is the only detector. +# +# WHAT COUNTS AS DRIFT (and why not a byte diff) +# Byte-for-byte would be trivially correct and useless: every comment reflow in +# the devkit — e.g. the DEC-079 rewrite of the devkit-sync header — would fire +# on all consumers at once, and a checker that cries wolf gets `|| true`'d +# within a week. So this compares the EXECUTABLE CONTRACT of the YAML: +# +# compared name, triggers (`on`), permissions, job ids, `uses` refs, +# `with` inputs, `secrets` (inherit vs the explicit key set), +# dispatch input names/types/defaults +# ignored comments, whitespace, quoting, key order, scalar-list order +# (`branches: [main, master]` is a set), and the human-prose +# `description:` of a dispatch input +# +# Rationale for that line: everything on the left changes what CI DOES, and +# both real incidents above are on the left. Everything on the right changes +# only how the file reads. Ignoring `description` is the one judgement call — +# it is UI prose in the Actions run dialog, and letting a wording fix fail four +# repos is exactly how the signal gets ignored. +# +# Drift is reported in three flavours, all fatal, because they rot differently: +# missing template has it, consumer does not — the silent-rot case above +# changed both have it, values differ — a local edit or a stale value +# extra consumer has it, template does not — a local edit that the NEXT +# `make link-workflows` will silently delete +# +# USAGE +# check-caller-drift.sh [consumer-repo-root ...] # default: . +# +# In consumer CI (the vendored copy, no args) it checks the repo it runs in +# against its own pinned devkit templates — so the check advances exactly when +# the submodule does. Locally, pass repo roots to sweep the fleet: +# check-caller-drift.sh ~/src/.../agent-zero-plugin-* +# +# Env: +# TEMPLATES_DIR devkit templates/ dir. Auto-detected when unset, in order: +#