Repository navigation
Prepare the d1386 Route A rerun: CT-ladder base plan, fresh lineage evidence, launch-day sequence - #1289
Merged
Conversation
Pin CT ladder and current crosswalk, separate base and release feeds, and prepare fail-closed lineage evidence plus launchd operator sequence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The QRF tail register: d490 covers this build (d1711), since the d177/d450 waivers expire at the next certified build. - Stage-001 lineage: a repeat of the 9/30 float-rounding class may be accepted (d1712); anything else still stops for Max. - Step 6 runs the guard fixture, guard sweep and gate dry run under supervise.py, with RSS caps, admission and the disk floor. - Step 4's limits are sized to stages 000/001 rather than the whole base. - PREP points at the real checkout; the stale sandbox hand-off text is gone. - The reference's argv and environment are registry-derived; say so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The guard fixture and sweep need no base, so GUARDS_ONLY=1 runs them while the Modal base builds, as its own job and script. The full run then reuses their recorded results and requires ACCEPTED before the gate dry run. A check with an existing supervisor directory is never rerun; a supervisor refusal reports 99. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The d1386 base's first attempt was cancelled after 822 s, 99 s after the launching Mac went to sleep on battery: `modal run --detach ... --run` waits on the call, and a client that goes away cancels it. `--run --spawn` spawns the stage and returns its function call id. A probe on 2026-10-19 showed a call spawned under --detach running to completion after its client exited. The receipt and attempt record land on the runs volume either way, and the launch doc now waits for the receipt there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step 5 copied the 2.4 GB base-out from the fetch directory into the run, which doubled its disk use on a host held near the release's 45 GiB admission. `cp -pc` makes an APFS clone that shares the fetched blocks. A 64 MB test moved free space by 0.7 MB instead of 64. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The release refuses any id that doesn't start with populace-us-, and the dry run stops where the release would. So the launch doc's route-a-rerun-20261018-gate-check failed after 119 s on 2026-10-19 without grading a gate. Use populace-us-2024-gate-check-<commit12>. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Spawn: the docstring, launch doc and runbook no longer claim a receipt "either way". A refusal or error leaves only an attempt record, a rejected plan leaves nothing, and a killed container's record goes stale. Step 5 names all four end states; step 3 records the call id in base.fc and refuses a second paid launch while it exists. Every runbook launch now uses --run --spawn. - Step 6: a check's config records W's HEAD and every file argument's sha256. A recorded result is reused only for an identical config (logged); a changed register refuses. Scripts are written aside and moved in, never over a running job's. - d1712: tools/route_a/lineage_class_check.py, the check run on 2026-10-19 (same syntax tree; formatted), with example and Hypothesis tests; step 5b runs it. - d1711: step 6b renews the QRF register before the release. - Step 7 starts the release from a read-only export of the driver and checks AC power. RUN is defined in step 4, and BLOCKER checks use grep. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Step 6 hashes only the inputs each check names, in 1 MiB chunks. Hashing every existing argument also hashed the sweep's and dry run's own outputs, so the second pass refused a finished sweep. A failed config render now returns 99 and leaves nothing behind. The running-job probe reads launchctl print into a variable instead of piping it into grep -q, which can SIGPIPE under pipefail. - Step 5 reads this call's attempt record (by modal.function_call_id) and its receipt (by attempt_id), since every attempt shares the run id. - Step 3 gives each launch its own log and waits at most 30 minutes for its call id. launchd appended to one log, so a relaunch would have read the old call id. - Step 5b checks the fetched checkpoint against the receipt's digest and removes the scratch copy on any exit. - Tests now cover a total beyond tolerance with every value within it, a dtype change, a renamed column and NaNs at equal positions. Deleting the total check makes the first fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The gate dry run's reuse identity includes EXPORT_MASS_REF_H5, which the dry run reads through RELEASE_EXTRA_ARGS. - Step 5b keeps its scratch copy and EXIT trap in a subshell. - The local reference runs with HF_HUB_OFFLINE=1, as the Modal base does. - The AT-RISK arm says its exit-code check shows agreement, not freshness. - FunctionCall.get() returns COMPLETED and FAILED receipts; it re-raises only for an attempt that raised. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The approving review's nits: - under bash 3.2 with set -u, an unset or empty RELEASE_EXTRA_ARGS failed the gate as "dry run failed ()"; - a file passed through those args must be named as an input to be hashed; - one line was overlong. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis
marked this pull request as ready for review
October 20, 2026 03:58
anth-volk
added a commit
that referenced
this pull request
Oct 20, 2026
Brings in main (#1289, #1305) through the base branch. GitHub builds each stacked pull request's test merge on top of main, and #1305's commit 7965d69 added a withhold_loss keyword to thinned_epochs, whose every default this branch moved to the shared CALIBRATION_EPOCH_EVERY. The signature conflict is resolved by keeping both: every defaults to CALIBRATION_EPOCH_EVERY, and withhold_loss stays keyword-only with its default of False. The callers #1305 added already pass staging_epoch_every(args), which this branch defines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk
added a commit
that referenced
this pull request
Oct 20, 2026
…-schema Brings in main (#1289, #1305) through the base branch. GitHub builds each stacked pull request's test merge on top of main. test_uk_graph_terminal.py conflicted: this branch and #1305 each pulled the terminal gate kernel run out of a test into a helper. #1305's helper, _run_final_gate_kernel, also takes brma_row and solved, so it is kept, with this branch's docstring sentence on the forbidden diagnostics rebuild. The schema test this branch added calls it, and #1305's two withheld-row tests follow it. main's #1305 also changed microcosm.diagnostics.schema, so the committed JSON Schema is stale after this merge; the next commit regenerates it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Launch-day prep for d1386, the next certified US build (a Route A rerun on main). Max ruled on 2026-10-15 to start it with whatever had merged. The 10-18 start didn't fire. On 2026-10-19 Max confirmed it launches that day on current main, without #1275 (d1556 moves to the following build).
Launched 2026-10-19 from this branch on main
8b0c1950b, which carries the #1186 Social Security fix and the #1227 EITC fix:The base on Modal, run
route-a-rerun-base-20261018-8b0c1950b65f, after check mode passed.--run --spawn, below.Local stage 000/001 lineage evidence, under supervision.
compare-lineagematched everything except the stage-001 digest.person.pre_subsidy_rentdiffering: 313 rows, 6.08e-15 relative at most, with identical sums. That is the 9/30 class.The guard sweep passed: 107 of 107 reform passes.
The gate dry run ended AT-RISK only (exit 2). At base weights, the QRF register and export input mass both depend on the solve.
The release started 2026-10-19 22:21Z from a read-only export of this branch's driver. After its solve, 8.3 h in, it failed four terminal gates, and the next build needs Max's rulings on them:
None of these is in what this PR changes, and the release's checkpoints are kept for a rerun.
Commits after the first review (of
4010c4b84):f053b3eaaand54f6dbc34answer it;0779d4280adds--run --spawn;c0fb05094clones the fetched base instead of copying it;fd557a50dgives the gate dry run apopulace-us-release id;f3341e507answers an Opus 5.5 review of those five, which asked for changes:base.fcblocks a second paid launch.tools/route_a/lineage_class_check.pyis the d1712 check that ran. The provenance copy has the same syntax tree before formatting; it has example and Hypothesis tests.4e1392430answers the GPT-6.1 Sol review off3341e507, which asked for changes:Step 6 hashed output files as inputs, so the second pass refused a finished sweep. It now hashes named inputs only.
Step 5 matches the attempt by call id and the receipt by
attempt_id.Step 3 gives each launch its own log and waits a bounded time.
Four more lineage-check tests; deleting the total-tolerance check now makes one of them fail.
Left open: the step 6 checks run without the driver preflight's
HF_HUB_OFFLINE=1. They ran this way, and changing their environment would invalidate their recorded results.b11330202and60a75bf88answer two more Sol rounds. The last one approved with nits, and the nits are fixed in60a75bf88.What
Base plan.
docs/us-modal-stage-route-a-rerun-base-plan.jsonis a Modal base plan for a Route A rerun on main:6840b990…, 18,991,218 bytes), now on the inputs volume;a347303f…), as a separateus-puf-support-base-rerunregistration, so the historicalus-puf-support-baseand its local reference stay pinned to 4b57d15;PICK_MAIN_COMMIT_ON_2026_10_18, a placeholder thatvalidaterefuses.The list-price ceiling is unchanged at $14.90.
Lineage evidence.
tools/route_a/make_local_reference.pyproduces fresh local stage 000/001 digests for the chosen commit, socompare-lineagecan run strictly.run_config, source and dependency identity, hash seed, pipeline and checkpoint digests carry the evidence.--stage allattempt that the disk floor killed: 2,055 s and 11.66 GiB peak RSS. It likely ran past stage 001, and the supervisor misses processes that had already exited, so it is a rough upper estimate for stages 000/001, not their measured cost.Driver.
tools/route_a/route_a.shkeeps its published feed and crosswalk pins as defaults, but takesFEED_SHA,FEED_BYTESandCROSSWALK_SHAfrom settings when the chosen commit changed them. Main's release feed is now965a29ac…, and the base feed is kept separately.Docs. The launch-day sequence is
docs/us-route-a-rerun-20261018.md, with a settings template, a runbook delta and the prep report. The sequence:supervise.py.Inputs checked against main
412f9165):microcosm-data(contract.py,release.py,publish_cli.py) changed with the UK UK local-area release: publish a K=15, 150,000-household dense dataset in at most two builds #1301 disclosure work. The release's own gates and the offline publisher preflight are the check on the US side.8b0c1950b.Known gaps
validateomitsmeasured_locallyfor the rerun tool becauseMEASUREDis keyed by tool name. This is left as is: those measurements are of the 4b57d15 base, and the cost-cap figures don't use them.Tests
377 isolated tests pass. Four environment checks were deferred to the launch-day env. Lint, formatting of the changed files, Bash syntax of every launch-doc block and
tools/ci_test_plan.py verifypass.🤖 Generated with Claude Code