[MaxGhenis]

Stacked on #240 (v2-microcosm-batches). The calculator stage is reviewed (rounds 6 and 7 agreed at 9b30a77 and 4002f8e). Commit 047c868 adds the no-tools stage's script. Rounds 8 to 10 asked for changes, made in 08d7b18 and 5a9735b, then in c60086b, 248e777 and 4ce7f8e, and then in 2f795ea, 6597adb and 9acc920, which round 11 reviewed. Round 11 asked for four P3 changes (watcher-startup and descriptor-order tests, three wordings), made in a1114fe (tests only) and 89dd946 (docstrings and comments only), which round 12 reviews. The PR stays a draft until the pre-registration it implements is frozen.

scripts/confirmatory_run.py runs the v2 calculator-interface study's confirmatory calculator stage as its draft pre-registration specifies, and writes the run records the confirmatory report checks before it scores anything. It runs the treatment harness unchanged: tool_pilot_argon.py and policybench here are 6b00a6f's, and the runner refuses to start if they differ.

Nothing here runs a model. The confirmatory run waits on the frozen pre-registration and Max's go.

What it does

• Lanes. Two per model. Households alternate between them in draw order. A lane runs a household's arms in the order drawn for it (stream order/<model>), then its reruns in passes, and only then decides its next household. The six models run at once.
• One engine pool per model. The pool starts once, with no features of its own. Each call carries its arm's features in the harness's call frame. In development, starting a pool per arm cost 2.8 to 3.6 minutes each time.
• Admission. Each model has a cap and a reserve per household under way. After one refusal, every later decision is a refusal. A decision waits while a pass's reruns are undecided, so the runner always knows whether a household has finished.
• The hard ceiling. Overall cap plus $25, counted over every model through the harness's Spend. When spend crosses it, no attempt starts, conversations under way end budget_stop, and every model's log gets a ceiling event. Admission checks the ceiling and takes its time under the ceiling's lock, so no household is decided after the crossing.
• The attempt deadline. A conversation still running an hour after it started (the longest development attempt took 714 s) is abandoned. Its attempt gets a harness-error record, AttemptDeadline, with the spend so far, and the model's lanes stop with cause deadline. The conversation ends budget_stop at its next turn; its spend after the deadline still counts toward the ceiling and goes to the run log as late-cost events. An attempt whose conversation raises keeps the spend it counted.
• Failures. Each attempt gets a class from its record and the host evidence, in the report's precedence, and an action from its class. The class lists hold every litellm subclass the harness groups by isinstance, explain traces count as calls, and a Timeout on the escalated request after a truncated turn is class model. An operator failure pauses the model's lanes until confirmatory_run.py resume, accepted only if written after the pause began. A harness failure stops them, and so does one inconclusive error that ends runs in more than five households. Replaced attempts are saved, and in-flight records fsynced, before their reruns.
• Stops name their cause: harness, inconclusive, deadline, pool, runner-error, interrupt or sampler (ceiling events have cause ceiling). An exception in the runner itself stops every model's lanes after settling and writing what had ended. The signal handler only notes the signal, and the main loop stops the lanes.
• Host log. A sample every 30 seconds, with the monotonic clock beside the wall clock, then the guard's and the power log's lines for the booking. A failed memory-pressure reading is no sample. A read of the guard's or the power log is tried five times before it is a runner error. A pass's classes wait until its host evidence is complete, which takes at most about 90 seconds. A sampler whose samples fail for five minutes, or whose thread ends, stops every model.
• Chains and anchors. Every line of the run, admission and host logs carries prev, the sha256 of the line before. Every 15 minutes and at the end, anchors.jsonl gets every log's line count and head and every manifest's sha256, and the design's anchor_command runs on its own snapshot of the file (so no push sends a line half written) in the background. scripts/confirmatory_anchor.py is that command: it commits the snapshot to a clone and pushes it without forcing, so a remote outside the run holds each anchor's time. It holds an exclusive lock in the clone and passes the locked descriptor to every Git command it runs, so a helper, hook or remote helper that outlives a push makes later pushes fail as busy, never writing beside them. It refuses a snapshot that does not extend the clone's file byte for byte. A push still running after 5 minutes is killed with its process group; the final heads are pushed up to three times, a minute apart. Each push's anchor, times, return code and outcome go to anchor-pushes.jsonl.
• The run's end. The runner waits up to 60 seconds for abandoned conversations, closes the pools and the host log, then the spend ledger. Late costs, a crossing of the ceiling and any signal by then are logged before the final anchor; a request still running when the ledger closes is logged as late-unresolved with the spend it had counted. After its last check the runner ignores signals, checks once more, and anchors again if a first interrupt arrived in between.
• Setup spans. Each pool start and restart is logged as a span, which the report's idle check sets aside.
• Workers. --workers must equal the design's engine workers per model.
• Logs from the start. Each model's run and admission logs exist from its manifest on, so a model that never decides (the ceiling crossed first) still has both.
• A clock that never goes back. The host's wall clock steps back by up to about 0.1 ms, even within a thread. The study's clock probe saw 29 backward steps in 177 million readings by twelve threads, the largest 99 µs, at a load of about 150. The run records order attempts, stops, admission decisions and the ceiling's crossing by their times, and an attempt's end and the next start can be microseconds apart. So clock() returns time.time(), never earlier than a time it returned before, and each such time is taken under the lock that orders it. The harness's request times stay raw. The report allows them 0.1 s of slack, widened by the largest backward step the host log shows.

The no-tools stage

scripts/confirmatory_no_tools.py (047c868) runs the stage before the calculator stage: the three cheap models answer every drawn household without tools, and their answers only flag references for audit. It calls the harness's run_household directly with each model's none profile and a tool function that raises, so the harness's own resume, which folds earlier attempts into their status and totals, is never used.

• Threads. Each model runs in its own thread, one household at a time in draw order.
• Classes and actions. The calculator runner's classify and action, unchanged, with no host evidence. A provider or none attempt reruns at once; an operator attempt pauses its model until the script is run again; a harness attempt stops its model; a ceiling attempt stops every model. At most two reruns.
• Attempts. Before a rerun, the final record is moved byte for byte to households/attempts/<household>-<n>.json, and the new final lists every replaced attempt (number, file, sha256, status, class, reason). The script resumes from these files alone, refuses prior_attempts, numbering gaps, links that disagree with the files and archives no rule reruns, and resumes an archive left without a final as the next attempt.
• Spend. Every saved record's cost, final and replaced, of every model folder, counts at startup. The $5 cap is shared by all threads, checked before each attempt and, through the harness, before each turn. A saved ceiling attempt stops later invocations. An attempt that raises keeps the spend it counted. Concurrent requests and requests within a turn, among them a request's retries and the harness's unchecked resend at a larger output budget after a reply cut short for length, can take the total past $5.
• Baseline exclusions. The design names baseline_exclusions ({path, sha256}): the board's exclusions as they stood before the stage's first call. Before any provider call, the script refuses to start unless that file hashes as declared and has the form the report reads: an object whose exclusions list holds one object per cell, with nonempty scenario_id and variable strings and no cell twice ({"exclusions": []} when empty).
• One invocation at a time. An exclusive, non-blocking flock on .confirmatory-no-tools.lock in the stage's runs folder, held for the whole invocation, dry runs included. A second invocation refuses.
• Interrupts. For the run's duration the main thread handles SIGINT and SIGTERM and restores the previous handlers afterwards. The first signal's handler takes no lock and raises nothing: it records the signal and writes a byte to a nonblocking pipe. A watcher thread reads it and sets the interrupt event under a short admission lock, which archive moves and attempt admissions also take for the interrupt check and the move or the admission alone. A move or attempt admitted before the watcher sets the interrupt goes ahead; every later one is refused. Provider calls run outside the lock. Attempts already admitted finish and save, and the script prints each model's state and the spend. A signal received before the handlers are restored makes the exit 1, even when every model finished. The interrupt is separate from the budget stop. A second signal takes no lock: it writes to stderr without blocking and exits with 128 plus the signal number. Every unsaved attempt then under way is lost, with its spend; saved finals, archives and the spend they record survive.
• Failures. An exception escaping execute, a thread that fails to start included, sets the interrupt under the admission lock and waits for every started thread before the handlers are restored and the invocation lock is released. Each thread's start and finish are tracked with the script's own events, a wait cut short by another exception is resumed, and a thread whose start() raised gets 10 seconds to begin; one that begins later returns before touching a file. The watcher thread is tracked the same way. The handlers are restored before the watcher's pipe is closed. The original exception propagates. An exception caught in a model's thread marks that model error and does not set the interrupt; the other models go on, and the exit is nonzero.
• Exit. 0 only when every selected model has a standing final for every drawn household. --dry-run lists the standing households and the next attempt owed, with no provider call.

The calculator runner changes only in its preflight (5a9735b). It calls stage_problems, a file-only check that loads no board and calls no provider. It applies the script's own history checks, which classify every saved final (since 248e777, an optional model's and one outside the draw too), and refuses to start while any no-tools history fails them or any cheap model lacks a standing final for a drawn household. It also refuses no-tools and calculator runs roots that are the same folder or where one contains the other.

Invariants

The tests' oracle (run_problems in tests/test_confirmatory_run.py) checks these on every fake run, and the property test checks them on random mixes of planted failures:

• Every attempt has one run-log line, and every record and replaced-attempt file hashes as its line logs it.
• Each logged class and action equals the one recomputed from the record and the host evidence.
• Each lane runs its households in draw order. Each household runs its drawn arm order, then its reruns in passes. No two attempts on a lane overlap. No arm has more than two reruns.
• Each admission decision equals the rule applied to the finished households' spend and the households under way. A household a stop cut short is under way until the stop. After one refusal, every decision refuses.
• clock() never returns a time earlier than one it has returned, including under a wall clock that steps back (test_the_clock_never_goes_back and test_a_wall_clock_that_steps_back_keeps_the_run_procedure). The run log itself is not in time order: an attempt's line is written when its household ends, and a ceiling event carries the crossing's time. Every check that orders events reads their times, never their line order.
• Every record's request times lie within its attempt, to within 1 s.
• No attempt starts during a pause, after a stop, or after the ceiling's crossing.
• Conservation: the ceiling's total equals the spend before the stage plus every record's cost plus every late-cost event.
• Each log is an unbroken chain from its seed; each anchor matches the logs' prefixes at its line counts; the last anchor matches every log whole, and its push returned 0.
• Every stop names a cause. Each harness, deadline and inconclusive trigger is followed by a stop of that cause within 5 s, no attempt starts after a harness or deadline trigger, and every stop of those causes has its trigger.

Tests

• tests/test_confirmatory_no_tools.py (60 tests, fake providers through the real run_household): draw order and one thread per model; a rerun links the unchanged archived bytes; a third provider failure stands with two archives; operator pause and resume; harness and budget stops that persist; a crash after the archive move; spend carried across invocations and optional models; refusals of foreign, gapped and corrupted histories; an attempt that raises keeps its spend; the cap agrees with the report's. Commit 08d7b18 adds a real os.kill(os.getpid(), signal.SIGINT) from inside a fake provider call (the attempt saves, no later household starts, the exit is 1, the handlers are restored), a second Stage refused while the first holds the lock, both baseline refusals, archive identity and rerun eligibility, a harness stop at attempt 3, and a conversation through the real run_household that ends at its next turn after another model's spend crosses the cap. c60086b to 4ce7f8e add real signals between the interrupt check and an archive or an attempt (both refused); a second thread that fails to start while the first holds a fake provider (an independent flock probe shows the invocation lock held until the first finishes); a second signal while a worker holds the output lock (exit 128 plus SIGTERM within the deadline); optional and off-draw finals that cannot be classified, refused at startup and by the preflight; and baseline files that are a hashed bare list, invalid JSON or missing a field. Commit 2f795ea replaces the handler's lock with the watcher and adds tests that run each signal path in a bounded subprocess: a real SIGINT inside the cleanup's own Event condition (the worker saves, the handlers stay installed and the invocation lock held until it does); exceptions injected once and twice into the cleanup's waits; a start() that raises before or after the native thread exists, within and past the start grace; a signal while a model's thread holds its admission (with the lock removed from the worker or from the watcher, all four fail); a second signal while the watcher waits for admission or while a worker holds the output lock; and a signal after every model finished (exit 1). The tests that slept 50 ms now wait for the interrupt event, and the forced-exit deadline is 30 s. Commit 9acc920 fixes a race in one new probe: its control thread could exit the process before the main thread's handler did (at 6597adb it failed that way in two of three runs: once in run 2, twice in run 3). The two forced-exit probes then pass five more runs in a row. a1114fe adds six cases in bounded subprocesses: the handlers must be restored before either of the watcher's pipe descriptors closes, and a real SIGINT after a descriptor's number is reused must write nothing to its new owner (with the teardown order reversed, both fail); a watcher whose start() raises before or after the native thread exists, or whose target begins after the start grace (the original error propagates, the handlers and descriptors are released, and a late target returns without reading a closed descriptor); and a first SIGINT after every model finished (exit 1, every final saved, no interrupt set and no announcement). All six pass on the script at 9acc920, which needed no change. Commit 89dd946 qualifies the docstrings and comments on the handler and the cleanup's waits; its executable code is identical to 9acc920's. With the existing files, 315 pass at 89dd946, one worker, in each of three runs in a row (309 at 9acc920, 297 at 4ce7f8e), and the two forced-exit probes pass five more runs. The fix build also started a run of the whole tests/ directory; it reached about 57% before the build's 6-hour limit stopped it, with three or four failures that its -q output does not name. This PR changes only scripts/confirmatory_*.py, tests/test_confirmatory_*.py and tests/fake_anchor.py, and the whole directory has not been run to completion at this head. Commit 060b287 makes two host-log tests (test_the_host_log_samples_and_closes and test_pressure_near_an_attempt_is_host_trouble_once_settled) wait for the samples they need instead of sleeping a fixed time; on a loaded host the 0.02 s sampler had missed both windows.
• tests/test_confirmatory_run.py and tests/test_confirmatory_anchor.py: 248 pass at 4002f8e, one worker. The anchor tests run the helper against a real temporary clone and a local bare remote. They kill the helper inside Git's pre-commit, pre-push and pre-receive hooks, inside a clean filter during git add and during git status, and with a detached descendant holding the lock. Each checks that later helpers fail as busy and leave the clone's file, index, HEAD and the remote unchanged, and the clean-filter and descendant tests each fail when inheritance is removed from only the command they target. Fake OpenAI Responses and Gemini JSON conversations are compared with the unchanged treatment harness's, complete requests and calculator inputs included.
• Earlier, at f5099db: 189 passed (load about 90). f5099db makes the interrupt test independent of timing: it now interrupts from inside a provider call and waits for the lanes to stop, and it passed 5 of 5 at load 91. Each of 19 mutants of the rules added in fdbd411 fails a named test; the one that first survived (a resume kept until a pause) is now caught by a resume with a misdated time. Until fdbd411, 121 tests passed in 8 runs of the full suite in a row at loads from 12 to 146. They cover unit tests (streams, arms, classes, host evidence, the ceiling), fake runs with planted failures, the harness's own conversation under the runner, and the property test. Stacked PRs into v2-microcosm-batches get no CI, since ci.yml runs on pull requests to main.
  • Three oracle faults, fixed in 808ccc8. First, the oracle took a household that a stop cut short to have finished at its last attempt. The runner counts it as under way until the stop, and a new test makes that case deterministic. Second, under load the sampler stalled past the 90 ms scaled host-log gap, so an untouched run got host reruns. The gap is now 0.5 s in the tests. Third, the oracle now allows the harness's raw request times 1 s of slack.
  • The clock. A test reads the wall clock up to 0.2 ms early everywhere. Without clock() it fails every time: attempts overlap, and a household runs before its admission.
• Differential test, in the study's scripts repo (test_confirmatory_differential.py). It runs this runner's Run on a copy of the development board. The harness's run_household is real, and so are this runner's host log and lanes. Only three things are fakes: the model (replying in each API's shape: chat tool, chat JSON, Responses), the engine pool and the host's readers. The confirmatory report's analyse then checks every record and log. At the report's 7cf22e2 against 4002f8e, 27 tests pass, and a 60-example run of its property test passed. At the report's ef297fa against 047c868, 34 pass, among them the no-tools script's own output checked by the report, and a 60-example run of the property test passed. At the report's 0b8e275 against 5a9735b, 39 pass, among them a harness stop and a ceiling stop on the last household, each of which the report refuses, and a 60-example run of the property test passed. At the report's 35005db against 89dd946, 64 pass, among them 15 baseline files that this runner and the report must both accept or both refuse, and a 60-example run of the property test passed. (At eda54e1 against f5099db, 22 passed, among them the same jittered clock.)
  • It found one defect here: the missing admission log, fixed in ec31e83.
  • It found four rules the report applied more strictly than the procedure. All four are fixed in the report:
    • an arm left empty by a stop;
    • a rerun cut off by a stop;
    • a household cut short by a stop, which counts as under way until the stop;
    • a truncated, retried request at the ceiling.
  • Six planted runner faults each fail the report, so the test can fail:
    • a skipped provider rerun;
    • admission past the cap;
    • a runner blind to memory pressure;
    • a runner that ignores a harness failure;
    • the ceiling's crossing logged an hour late;
    • a lane that stops deciding.

Plan

• Runner
• Tests
• Differential test (in the study's scripts repo): run this runner on fakes and check its records pass the confirmatory report
• Independent review, round 1 (Opus 5.5): REQUEST_CHANGES, nine findings, all answered in fdbd411
• Differential test updated for fdbd411's API (in the study's scripts repo)
• Round 2 (GPT-6.1 Sol): REQUEST_CHANGES, six findings, answered in f210b6e and 36d553a (a ledger that closes, the sampler's failure timer, ambiguous local hours in the guard log, stop states before event I/O, the run's end, setup spans, push retries)
• Round 3 (Opus 5.5): REQUEST_CHANGES, six findings; the runner's four answered in 00d267a (push snapshots, three final tries, bounded joins, signals at the end, push times on the runner's clock)
• Round 4 (GPT-6.1 Sol): REQUEST_CHANGES; its runner finding, a retired push's command that could still write, answered in 019f842 (the clone lock, the byte-prefix check, no force)
• Round 5 (GPT-6.1 Sol): REQUEST_CHANGES, one finding, Git children that outlived the lock, answered in 9b30a77 (the descriptor goes to every Git command)
• Round 6 (GPT-6.1 Sol): AGREE at 9b30a77, two minor items answered in 4002f8e (docstring wording and three more regressions)
• Round 7 (GPT-6.1 Sol): AGREE at 4002f8e
• The no-tools stage's script (047c868), for the pre-registration review's finding that the stage had no checked rerun procedure
• Round 8 (GPT-6.1 Sol): REQUEST_CHANGES, four findings; the runner's answered in 08d7b18 and 5a9735b (a coordinated interrupt, the invocation lock, the baseline exclusions, the calculator preflight's check on the no-tools stage, more tests), the report's in its own repo
• Round 9 (GPT-6.1 Sol): REQUEST_CHANGES, six findings (an archive or attempt admitted after a signal, a failure releasing the lock under a live thread, the preflight skipping optional and off-draw finals, a second signal blocking on output, the baseline's form, the ceiling wording); answered in c60086b, 248e777 and 4ce7f8e
• Round 10 (GPT-6.1 Sol): REQUEST_CHANGES, four findings (a signal inside the cleanup's own lock could deadlock, a cut-short wait could release the invocation lock under a live thread, admission tests that passed without the lock, two wordings); answered in 2f795ea, 6597adb and 9acc920
• Round 11 (GPT-6.1 Sol): REQUEST_CHANGES at 9acc920, four P3 findings (no regression for the handlers' restoration before the pipe closes or for a watcher that fails or starts late; three wordings: a first signal after the watcher shuts down, unguarded cleanup instructions, the 6597adb failure count); answered in a1114fe and 89dd946
• Round 12 (GPT-6.1 Sol): a1114fe and 89dd946
• The pre-registration frozen (the draft's round-10 review agreed with two wordings; the report's round-10 review and its full mutation sweep asked for changes, made in the report's f37f6cf to 35005db; the draft's round-11 and the report's round-11 reviews run with this one)
• Merge, then the run on Max's go

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policybench-site	Ready	Preview	Oct 28, 2026 1:17pm UTC