Skip to content

Prepare the d1386 Route A rerun: CT-ladder base plan, fresh lineage evidence, launch-day sequence - #1289

Merged
MaxGhenis merged 10 commits into
mainfrom
route-a-rerun-prep
Oct 20, 2026
Merged

MaxGhenis merged 10 commits into
mainfrom
route-a-rerun-prep

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Launch-day prep for d1386, the next certified US build (a Route A rerun on main). Max ruled on 2026-10-15 to start it with whatever had merged. The 10-18 start didn't fire. On 2026-10-19 Max confirmed it launches that day on current main, without #1275 (d1556 moves to the following build).

Launched 2026-10-19 from this branch on main 8b0c1950b, which carries the #1186 Social Security fix and the #1227 EITC fix:

  • The base on Modal, run route-a-rerun-base-20261018-8b0c1950b65f, after check mode passed.

    • The first attempt was cancelled after 822 s, when the launching Mac slept. That led to --run --spawn, below.
    • The relaunch COMPLETED in 5,988 s. Both attempts together listed at $6.18.
  • Local stage 000/001 lineage evidence, under supervision.

    • compare-lineage matched everything except the stage-001 digest.
    • The d1712 check found only person.pre_subsidy_rent differing: 313 rows, 6.08e-15 relative at most, with identical sums. That is the 9/30 class.
    • The base was accepted on 2026-10-19.
  • The guard sweep passed: 107 of 107 reform passes.

  • The gate dry run ended AT-RISK only (exit 2). At base weights, the QRF register and export input mass both depend on the solve.

  • The release started 2026-10-19 22:21Z from a read-only export of this branch's driver. After its solve, 8.3 h in, it failed four terminal gates, and the next build needs Max's rulings on them:

    • the Social Security total came out 6.4% under SSA's: retirement benefits were 8.5% under, against 3.9% in the last certified release;
    • three new QRF tail columns came out over 0.75;
    • bond_assets is now stale.

    None of these is in what this PR changes, and the release's checkpoints are kept for a rerun.

Commits after the first review (of 4010c4b84):

  • f053b3eaa and 54f6dbc34 answer it;

  • 0779d4280 adds --run --spawn;

  • c0fb05094 clones the fetched base instead of copying it;

  • fd557a50d gives the gate dry run a populace-us- release id;

  • f3341e507 answers an Opus 5.5 review of those five, which asked for changes:

    • Spawn: the "receipt either way" claim was false. Step 5 now names all four end states, and base.fc blocks a second paid launch.
    • Step 6 reuses a recorded check only when its config matches, including W's HEAD and every file argument's sha256.
    • tools/route_a/lineage_class_check.py is the d1712 check that ran. The provenance copy has the same syntax tree before formatting; it has example and Hypothesis tests.
    • Step 6b renews the QRF register under d1711.
    • Step 7 starts the release from a read-only export of the driver.
  • 4e1392430 answers the GPT-6.1 Sol review of f3341e507, which asked for changes:

    • Step 6 hashed output files as inputs, so the second pass refused a finished sweep. It now hashes named inputs only.

    • Step 5 matches the attempt by call id and the receipt by attempt_id.

    • Step 3 gives each launch its own log and waits a bounded time.

    • Four more lineage-check tests; deleting the total-tolerance check now makes one of them fail.

    • Left open: the step 6 checks run without the driver preflight's HF_HUB_OFFLINE=1. They ran this way, and changing their environment would invalidate their recorded results.

    • b11330202 and 60a75bf88 answer two more Sol rounds. The last one approved with nits, and the nits are fixed in 60a75bf88.

What

  • Base plan. docs/us-modal-stage-route-a-rerun-base-plan.json is a Modal base plan for a Route A rerun on main:

    • the Connecticut-fixed block ladder (d713: 6840b990…, 18,991,218 bytes), now on the inputs volume;
    • main's corrected congressional-district crosswalk (a347303f…), as a separate us-puf-support-base-rerun registration, so the historical us-puf-support-base and its local reference stay pinned to 4b57d15;
    • the commit, PICK_MAIN_COMMIT_ON_2026_10_18, a placeholder that validate refuses.

    The list-price ceiling is unchanged at $14.90.

  • Lineage evidence. tools/route_a/make_local_reference.py produces fresh local stage 000/001 digests for the chosen commit, so compare-lineage can run strictly.

    • The reference's argv and environment are written from the plan's registry. The masked run_config, source and dependency identity, hash seed, pipeline and checkpoint digests carry the evidence.
    • The only local measurement is a whole --stage all attempt that the disk floor killed: 2,055 s and 11.66 GiB peak RSS. It likely ran past stage 001, and the supervisor misses processes that had already exited, so it is a rough upper estimate for stages 000/001, not their measured cost.
  • Driver. tools/route_a/route_a.sh keeps its published feed and crosswalk pins as defaults, but takes FEED_SHA, FEED_BYTES and CROSSWALK_SHA from settings when the chosen commit changed them. Main's release feed is now 965a29ac…, and the base feed is kept separately.

  • Docs. The launch-day sequence is docs/us-route-a-rerun-20261018.md, with a settings template, a runbook delta and the prep report. The sequence:

    • records Max's two rulings of 2026-10-19:
      • d1711: d490 covers this build's QRF tail register, although the d177/d450 waivers expire at it;
      • d1712: a repeat of the 9/30 stage-001 float-rounding difference may be accepted, and anything else stops for him;
    • runs the release guard checks under supervise.py.

Inputs checked against main

Known gaps

  • validate omits measured_locally for the rerun tool because MEASURED is keyed by tool name. This is left as is: those measurements are of the 4b57d15 base, and the cost-cap figures don't use them.

Tests

377 isolated tests pass. Four environment checks were deferred to the launch-day env. Lint, formatting of the changed files, Bash syntax of every launch-doc block and tools/ci_test_plan.py verify pass.

🤖 Generated with Claude Code

MaxGhenis and others added 10 commits October 15, 2026 07:43
Pin CT ladder and current crosswalk, separate base and release feeds, and prepare fail-closed lineage evidence plus launchd operator sequence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The QRF tail register: d490 covers this build (d1711), since the d177/d450
  waivers expire at the next certified build.
- Stage-001 lineage: a repeat of the 9/30 float-rounding class may be accepted
  (d1712); anything else still stops for Max.
- Step 6 runs the guard fixture, guard sweep and gate dry run under
  supervise.py, with RSS caps, admission and the disk floor.
- Step 4's limits are sized to stages 000/001 rather than the whole base.
- PREP points at the real checkout; the stale sandbox hand-off text is gone.
- The reference's argv and environment are registry-derived; say so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The guard fixture and sweep need no base, so GUARDS_ONLY=1 runs them while
the Modal base builds, as its own job and script. The full run then reuses
their recorded results and requires ACCEPTED before the gate dry run. A check
with an existing supervisor directory is never rerun; a supervisor refusal
reports 99.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The d1386 base's first attempt was cancelled after 822 s, 99 s after the
launching Mac went to sleep on battery: `modal run --detach ... --run` waits
on the call, and a client that goes away cancels it. `--run --spawn` spawns
the stage and returns its function call id. A probe on 2026-10-19 showed a
call spawned under --detach running to completion after its client exited.
The receipt and attempt record land on the runs volume either way, and the
launch doc now waits for the receipt there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 5 copied the 2.4 GB base-out from the fetch directory into the run, which
doubled its disk use on a host held near the release's 45 GiB admission.
`cp -pc` makes an APFS clone that shares the fetched blocks. A 64 MB test
moved free space by 0.7 MB instead of 64.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The release refuses any id that doesn't start with populace-us-, and the dry
run stops where the release would. So the launch doc's
route-a-rerun-20261018-gate-check failed after 119 s on 2026-10-19 without
grading a gate. Use populace-us-2024-gate-check-<commit12>.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Spawn: the docstring, launch doc and runbook no longer claim a receipt
  "either way". A refusal or error leaves only an attempt record, a rejected
  plan leaves nothing, and a killed container's record goes stale. Step 5
  names all four end states; step 3 records the call id in base.fc and
  refuses a second paid launch while it exists. Every runbook launch now
  uses --run --spawn.
- Step 6: a check's config records W's HEAD and every file argument's
  sha256. A recorded result is reused only for an identical config (logged);
  a changed register refuses. Scripts are written aside and moved in, never
  over a running job's.
- d1712: tools/route_a/lineage_class_check.py, the check run on
  2026-10-19 (same syntax tree; formatted), with example and Hypothesis tests;
  step 5b runs it.
- d1711: step 6b renews the QRF register before the release.
- Step 7 starts the release from a read-only export of the driver and checks
  AC power. RUN is defined in step 4, and BLOCKER checks use grep.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Step 6 hashes only the inputs each check names, in 1 MiB chunks. Hashing
  every existing argument also hashed the sweep's and dry run's own outputs,
  so the second pass refused a finished sweep. A failed config render now
  returns 99 and leaves nothing behind. The running-job probe reads
  launchctl print into a variable instead of piping it into grep -q, which
  can SIGPIPE under pipefail.
- Step 5 reads this call's attempt record (by modal.function_call_id) and
  its receipt (by attempt_id), since every attempt shares the run id.
- Step 3 gives each launch its own log and waits at most 30 minutes for its
  call id. launchd appended to one log, so a relaunch would have read the old
  call id.
- Step 5b checks the fetched checkpoint against the receipt's digest and
  removes the scratch copy on any exit.
- Tests now cover a total beyond tolerance with every value within it, a
  dtype change, a renamed column and NaNs at equal positions. Deleting the
  total check makes the first fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The gate dry run's reuse identity includes EXPORT_MASS_REF_H5, which the
  dry run reads through RELEASE_EXTRA_ARGS.
- Step 5b keeps its scratch copy and EXIT trap in a subshell.
- The local reference runs with HF_HUB_OFFLINE=1, as the Modal base does.
- The AT-RISK arm says its exit-code check shows agreement, not freshness.
- FunctionCall.get() returns COMPLETED and FAILED receipts; it re-raises only
  for an attempt that raised.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The approving review's nits:
- under bash 3.2 with set -u, an unset or empty RELEASE_EXTRA_ARGS failed
  the gate as "dry run failed ()";
- a file passed through those args must be named as an input to be hashed;
- one line was overlong.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis marked this pull request as ready for review October 20, 2026 03:58
@MaxGhenis
MaxGhenis merged commit 046bf90 into main Oct 20, 2026
10 checks passed
anth-volk added a commit that referenced this pull request Oct 20, 2026
GitHub builds each stacked pull request's test merge on top of main, so
once main moved on (#1289, #1305), the stack's test merges had to include
it. main's new commits do not touch this pull request's files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk added a commit that referenced this pull request Oct 20, 2026
…tore

Brings in main (#1289, #1305) through the base branch. No conflicts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk added a commit that referenced this pull request Oct 20, 2026
Brings in main (#1289, #1305) through the base branch. No conflicts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk added a commit that referenced this pull request Oct 20, 2026
Brings in main (#1289, #1305) through the base branch. GitHub builds each
stacked pull request's test merge on top of main, and #1305's commit
7965d69 added a withhold_loss keyword to thinned_epochs, whose every
default this branch moved to the shared CALIBRATION_EPOCH_EVERY. The
signature conflict is resolved by keeping both: every defaults to
CALIBRATION_EPOCH_EVERY, and withhold_loss stays keyword-only with its
default of False. The callers #1305 added already pass
staging_epoch_every(args), which this branch defines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk added a commit that referenced this pull request Oct 20, 2026
…-schema

Brings in main (#1289, #1305) through the base branch. GitHub builds each
stacked pull request's test merge on top of main.

test_uk_graph_terminal.py conflicted: this branch and #1305 each pulled the
terminal gate kernel run out of a test into a helper. #1305's helper,
_run_final_gate_kernel, also takes brma_row and solved, so it is kept, with
this branch's docstring sentence on the forbidden diagnostics rebuild. The
schema test this branch added calls it, and #1305's two withheld-row tests
follow it.

main's #1305 also changed microcosm.diagnostics.schema, so the committed
JSON Schema is stale after this merge; the next commit regenerates it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant