feat(corpus): fetch the API documents the tests read, rather than committing them

[ShocOne]

First step of removing per-API-product artefacts from this repo. Touches no existing test — nothing reads this package yet, so main cannot move.

Why

95% of this repository's tracked files are artefacts of specific API products (pilot/ 4,673 files, recordings/ 1,340, blueprints/ 203, openapi/ 7). The toolkit itself is 280. Those artefacts belong in the provider repos the pipeline generates.

But the tests still need real documents. Every inference defect this generator has had was found by a second vendor's specification disagreeing with the first — not by a synthetic fixture that agrees with the code. So documents are fetched at test time and pinned, rather than vendored.

What

internal/corpus wraps the existing internal/snapshot rather than reimplementing pinning. Snapshot already stores documents immutably with a checksum and self-describing metadata; the only thing it lacked as a cache was a directory name derived from the pin rather than the wall clock, so a pinned document can be found again instead of merely being present.

internal/corpus/corpus.lock.json is the reviewable artefact — version, SHA-256, path and operation counts per document. A fetch matching none of them fails rather than proceeding.

That is the point rather than a detail. Tests assert facts about these documents — enum members, candidate counts, inferred bindings — so a vendor editing a specification changes what those tests mean. The lock converts an inexplicable failure weeks later into a deliberate review.

New CLI: corpus sync | check | refresh, documented in docs/cli.md.

Three decisions, each with a failure mode behind it

Staged then renamed. go test ./... runs packages as separate processes, so several race to fill a cold cache, and snapshot.Pin refuses to overwrite — immutability is the property it defends. Without atomic publication the first run after any lock change fails spuriously in whichever packages lost the race. That would present as flakiness and be diagnosed as anything but this.

Skip locally, fail in CI. Failing everywhere turns a developer's flaky connection into a red suite, which teaches people to ignore red. Skipping everywhere lets CI skip its way to green, which is a standing lie about coverage. A document that was fetched but did not match its pin is never skipped anywhere.

Mirror preferred over upstream — and this is the part that stopped being hypothetical while I was writing it.

ThousandEyes had already moved

The first cold sync failed:

the pinned thousandeyes document is not what https://pubhub.devnetcloud.com/.../api.yaml served.
  pinned  sha256:969c3dfceafe  version 7.0.98  201 path(s) / 315 operation(s)
  fetched sha256:664dffebf5db  version 7.0.99  205 path(s) / 323 operation(s)

That URL serves current, not a version. 7.0.98 — the snapshot committed in this repo — is no longer fetchable from anywhere. GitHub's description has the same shape of problem, published from main.

So the pin here is 7.0.99, taken deliberately through corpus refresh, and refresh warns on every pin lacking a mirror. A mirror is a prerequisite for this scheme, not a nicety — without one, refreshes are forced on the vendor's schedule. A mirror cannot weaken anything, since the digest is checked identically whichever source answered; it is a CDN, not a source of truth. None is configured yet — that needs a decision on where it lives.

Verification

$ tfpfgen corpus check
✅ github         unchanged: version 1.1.4 (sha256:80850db290cd, 808 paths, 1220 operations)
✅ jamfpro        unchanged: version production (sha256:e9a22a2d4af4, 547 paths, 817 operations)
📌 thousandeyes   upstream has moved

• Cold sync materialises all three in 3.7s; warm sync is 0.44s and makes no request.
• Seven hermetic tests (httptest, no network): lock integrity, cold-fetches-warm-does-not with a request counter, the mismatch message, recovery from a corrupted cache, mirror-preferred and mirror-cannot-change-meaning, offline-is-not-corruption, cache redirection.
• go build ./..., go vet ./..., go test ./..., gofumpt all clean. .tfpfgen/cache/ is already gitignored — verified with git check-ignore.

🤖 Generated with Claude Code