Skip to content

Commit 92447bd

Browse files
dmealingclaude
andcommitted
docs(fr-041): two validity threats found by running the thing, not reasoning about it
A pilot run on 2026-09-12 (first execution of the revived harness against 1.0.3) surfaced two §8 entries that no amount of reading the design would have produced. The agent inherits the HOST's MCP servers and user-level agent instructions, because it is launched headless with the sandbox as its working directory and nothing isolates its configuration. Part of each arm's context therefore comes from whichever workstation runs it: not a fixed treatment, and not reproducible by a stranger, which is the whole claim a published benchmark makes. Every archived run shares the exposure. And the per-task budget cap has to be re-derived whenever the agent model changes — the same run took roughly twice the turns and cost of its archived predecessor on the previous model generation, so a cap carried over unchanged binds at a different rate than the one §8 reasons about. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011cQxyVuTkdPmAduLiedNEg
1 parent 49f1c47 commit 92447bd

1 file changed

Lines changed: 11 additions & 0 deletions

File tree

docs/superpowers/specs/2026-09-11-fr-041-drift-ab-benchmark-design.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -317,6 +317,17 @@ unflattering headline is the specific thing pre-registration exists to stop.
317317
flatters it on a measure that requires completion. The cap is set from the pilot so that it
318318
binds at the same *rate* in both arms rather than at the same number, every truncation is
319319
recorded, and truncated tasks count as failures (§5a).
320+
- **Host contamination of the agent's context — OBSERVED, not hypothetical.** A 2026-09-12 pilot
321+
run launched the agent headless with the sandbox as its working directory, and it inherited the
322+
**host machine's** MCP servers and user-level agent instructions. Part of the arm's context
323+
therefore came from whichever workstation ran it, so the treatment was neither fixed across runs
324+
nor reproducible by anyone else — and every archived run shares the exposure. Mitigation: run
325+
each arm in a clean room with its settings and MCP configuration supplied explicitly, and record
326+
in the pre-registration exactly what context each arm was given.
327+
- **A per-task budget sized on the wrong model.** The cap must be re-derived whenever the agent
328+
model changes: the same pilot run took ~2x the turns and cost of its archived predecessor on the
329+
previous model generation, so a cap carried over unchanged would bind at a different rate than
330+
intended.
320331
- **Oracle bugs.** Every oracle is run against a known-good and a known-drifted reference solution
321332
in both arms before any scored run (§4b).
322333
- **Model drift.** Arms are interleaved; the model version is recorded per run.

0 commit comments

Comments
 (0)