diff --git a/docs/measurements/ddp-runtime-state-pilot-20261102.md b/docs/measurements/ddp-runtime-state-pilot-20261102.md
new file mode 100644
index 00000000..af12da6d
--- /dev/null
+++ b/docs/measurements/ddp-runtime-state-pilot-20261102.md
@@ -0,0 +1,388 @@
+# Data-dependent DDP training: optional TrainCheck pilot
+
+**Status: one matched pair and independent grading completed, 2026-11-02.**
+Both agents localized the cause from source before their first requested run.
+Both final submissions pass all seven independent scenarios. No TrainCheck
+instrumentation activation was observed; the treatment agent read its manual
+and used ordinary PyTorch/Trainer diagnostics. This case did not meet the intended
+difficulty of requiring runtime observation for localization, and provides no
+evidence of a TrainCheck efficiency benefit.
+
+Native qualification, held-out positive controls, optional-tool preflight and
+packet review passed before launch. The pair started after the 2026-11-01
+21:30:49 UTC protocol freeze. This is private operator documentation, excluded
+from task packets. Results from the earlier
+[tensor-parallel clipping pilot](real-rank-runtime-pilot-20261102.md) remain:
+its four agents also localized the cause from source before their first GPU
+request; no TrainCheck instrumentation activation was observed there either.
+
+## Why this case
+
+The next case reproduces the native Hugging Face Trainer behavior discussed in
+[Transformers issue #34242](https://github.com/huggingface/transformers/issues/34242)
+and addressed by [PR #34373](https://github.com/huggingface/transformers/pull/34373).
+When ranks have different numbers of valid target tokens, averaging their local
+mean-loss gradients does not yield the gradient of the global token-mean loss.
+DDP can keep model replicas bitwise identical while training the wrong objective.
+Replica agreement alone therefore cannot diagnose this problem.
+
+The relevant runtime evidence is the actual example assignment and valid target
+count on each rank, the denominator reaching the model's native loss function,
+and the resulting gradient before the optimizer update. The qualification uses
+counts of 30 and 78 per rank versus a global denominator of 108. Equal-count
+controls distinguish this condition from a general distributed-training failure.
+This makes the case sensitive to cross-rank execution state; it does not prove
+that an agent cannot derive the problem by reading the source and data. Such
+source-based successes remain valid results.
+
+## Exact workload and provenance
+
+- Vulnerable native Transformers revision:
+  `a17f287ac039f92835b5cd9bd8ee28b584c9f65e`.
+- Corrected native revision: `d21dbd1520937c993de1409215b1418bd6be74a1`.
+  The correction requires `average_tokens_across_devices=True`; the disabled
+  flag is a separate negative control.
+- Native `Trainer` and `Qwen2ForCausalLM`, with 115,520 parameters, two layers,
+  hidden width 64, four attention/KV heads and sequence length 64. The model's
+  literal `**loss_kwargs` activates the historical Trainer path. Llama at this
+  revision uses a different signature and would not qualify the same path.
+- Float32, AdamW, learning rate 0.001, zero weight decay, gradient clipping disabled,
+  constant learning rate, seed 1234, deterministic settings and TF32 disabled.
+- Four examples per optimizer update, one equivalent single-GPU batch versus
+  two-rank DDP, with gradient accumulation of one or two microbatches.
+- A controlled local text fixture derived from repository README bytes, with
+  fixed tokenization and equal/unequal valid-length variants. No pretrained
+  weights or external dataset downloads. This reproduces a real framework
+  defect with a small fixture; it is not a production-quality training study.
+- Ring18 NVIDIA A16 GPUs 2 and 3, Torch 2.2.2+cu121, Python 3.10.14 and
+  Accelerate 1.0.1. Additional dependencies are installed in an isolated
+  experiment environment. The shared environment remains unchanged.
+
+All 4,715 files in each native source tree match its downloaded upstream archive.
+The archive SHA-256 values are
+`8977b0f525446b5ea80db0444a48b4ee5dac30373a13c64e8d979e4281638c65`
+and `2d12575c66970d5f04a8fa9814ca1136f73de3d54759af20b1ea9df43fb85274`.
+The shared environment package-list hash is
+`70eda80894c761ff4c75e673681e86749b74158756cd8792d39027d923829fb4`.
+
+## Qualification and numerical error
+
+Sixteen completed native runs cover equal/unequal token counts, accumulation
+factors one/two, vulnerable/corrected/flag-disabled code, the equivalent
+single-GPU reference, repetition and observer removal. Two earlier setup failures
+are retained. These 18 attempts consumed 402.53 allocated GPU-seconds, separately
+from later optional-tool preflight and solver work.
+
+| Comparison against the equivalent single-GPU objective | Measured result |
+| --- | --- |
+| Vulnerable, unequal counts, accumulation 1 | Gradient relative L2 error up to 38.27%; model maximum absolute difference about 0.00582 |
+| Vulnerable, unequal counts, accumulation 2 | Gradient relative L2 error up to 37.21%; model maximum absolute difference about 0.00594 |
+| Corrected with global averaging | Gradient maximum absolute difference at most 8.57e-8 and relative L2 at most 3.29e-7; optimizer-state maximum absolute difference at most 1.12e-8 |
+| Equal-count controls | Gradient and optimizer-state comparisons pass |
+| Rank agreement within every two-rank run | Model replicas match bitwise, including in the vulnerable runs |
+| Repetition and forward/loss observer removal | Bitwise equality of captured model, optimizer and CPU/CUDA/Python/NumPy RNG; 718 tensor and 10,448 scalar comparisons per comparison |
+
+The initial model comparison used absolute tolerance 1e-6 and relative tolerance
+1e-4. It rejected some corrected results: model differences reached about
+4.49e-6, while equal-count accumulation controls also reached about 3.21e-6.
+Small floating-point gradient differences can be amplified by Adam near zero.
+The original strict report is retained. A separate calibrated report uses model
+absolute tolerance 1e-5. Gradient/optimizer absolute tolerance remains 1e-6;
+relative tolerance remains 1e-4 for all three components. This is
+chosen during qualification, before any agent output, with a margin of over 500
+between the model tolerance and the observed faulty-model deviation. It is not
+bitwise equivalence between single-process and distributed arithmetic.
+
+All observed ranks, updates, examples and native calls must be present. Missing
+activation, malformed snapshots and failed workers are errors, not successful
+zero-difference results. No parameter, optimizer array or RNG contents are dropped
+to make comparisons pass. Observer parity covers the recorded state, not
+unobserved application state or live object aliasing.
+
+## Additional prelaunch checks
+
+The scalar-argument TrainCheck preflight observes native
+`torch.nn.functional.dropout` calls: 12 entry/return pairs per rank, with no
+unmatched records. Recorded model, optimizer and RNG state is bitwise identical
+to the ordinary matched run (718 tensor and 10,448 scalar checks), as are all
+162 gradient comparisons. This verifies the tested direct wrapper interface;
+it does not qualify arbitrary tensor metadata, model tracking or the inference
+engine. Its measured cost is 34.444 allocated GPU-seconds. A preceding import-only
+attempt failed because the CUDA library path was missing; its GPU cost was not
+measured and is retained as unknown, not zero.
+
+The independent grader attaches observers to the actual submitted Trainer,
+model and optimizer. It does not replace the submitted training pipeline with
+the qualification worker. An actual-entry replay matches every recorded state,
+RNG and gradient value from the archived native parent execution bitwise.
+
+Fresh seed 5678 uses a separate fixture whose global valid-target counts vary
+across three updates: 124, 140 and 152. Corrected actual-entry DDP runs with
+accumulation 1 and 2 pass all 58 declared comparisons against the equivalent
+single-GPU reference. Maximum held-out differences are 1.121e-6 for gradients,
+1.0548e-5 for model parameters and 1.576e-7 for optimizer state. These pass the
+unchanged **combined** absolute-plus-relative tolerance; a maximum difference
+can exceed the absolute term alone. The actual-entry replay plus three held-out
+controls cost 87.914 GPU-seconds, separately from solver requests.
+
+The frozen independent grading contract has seven scenarios: equal, unequal and
+held-out data, each with DDP accumulation 1/2, plus single-GPU compatibility on
+the original unequal-length data. It requires actual sample coverage, full
+model/gradient/optimizer comparisons and exact recorded RNG against a matched
+same-world corrected execution. It does not compare single-GPU RNG with two-GPU
+RNG. A valid repartition may change rank-local sample assignments while preserving
+each global four-example window; no particular normalization implementation is
+required.
+
+Native ordinary eight-update service runs verify both backends. A first pair of
+service runs revealed that native serialized training arguments contained an
+operator path whose experiment name could reveal the case. The service now
+passes a relative output path to the same destination. Two fresh ordinary runs
+confirm neutral serialized paths and byte-identical final model files relative
+to the earlier runs. Both sets of receipts are retained. This service qualification
+costs 74.882 GPU-seconds in total; no binary checkpoint is edited or normalized.
+
+## Frozen agent comparison
+
+The matched pair receives the same native source, local fixture, ordinary
+backend logs, model and resource limits. The task asks whether the extension from
+single-process to DDP training is ready for a longer run and what to do next.
+It neither asserts a bug nor asks for any particular diagnostic. Logged backend
+training losses are not presented as directly comparable held-out quality curves.
+
+The treatment additionally receives the unchanged generic TrainCheck observation
+manual and implementation. It may choose custom logging or no instrumentation.
+The control has the same ability to author arbitrary local diagnostics and use
+the remote execution service. Neither arm receives issue IDs, this report,
+private observations, a targeted invariant or grading feedback.
+
+It uses the same native CLI model/settings and accounting protocol as the previous
+pilot: gpt-6-astra/high, priority service, CLI 0.160.0, six requests, 1–16 global
+updates per request, 240 seconds per worker phase, 900 aggregate CLI seconds and
+eight decision turns. Both the two-GPU `current` and one-GPU `reference` execute
+the agent's submitted source; the latter is not a hidden corrected source.
+Allocated GPU cost uses two or one devices accordingly and includes startup and
+failed requests. Qualification and independent grading costs remain separate.
+
+Record source-supported localization, runtime-supported localization, earliest
+submitted source that later passes independent grading, agent-visible repair
+verification and final completion separately. Preserve native token counters,
+cache subsets, measured service waiting, every submitted source and raw events.
+Within-turn token costs may only be intervals. Availability or manual reading
+does not count as actual TrainCheck use.
+
+An independent packet comparison confirms that all 1,850 exported native runtime
+source/README/license files match the upstream parent archive, and all 12 actual
+training examples retain their qualified arrays. The control has 1,862 files;
+the treatment has 1,894. Only the initial prompt differs among shared files, by
+one sentence offering optional access. The 32 treatment-only files are the
+generic manual and 31 frozen TrainCheck source files. The source and data remain
+visible, so this review does not establish that runtime observation is necessary.
+
+## Completed pair and independent results
+
+The control produced its first passing implementation in turn 2; the treatment
+in turn 3. Independent replay covered every one of the 13 frozen submissions,
+using eight distinct complete source maps. Exact duplicate maps reuse the same
+seven-scenario result; changes to diagnostic helpers still count as distinct
+maps even when the ordinary training code is unchanged. The original control
+source and treatment turns 1/2 fail the unequal-count and held-out comparisons,
+with complete observation coverage. No earlier unknown grade blocks either
+first-valid result. Both final complete source maps pass all seven scenarios.
+
+The grader runs the actual immutable entry point against each submitted
+pipeline. It checks full model, gradient and optimizer state, native sample/mask
+coverage and matched same-world RNG, under the frozen tolerances. It does not
+reward a particular repair pattern or a training exit code alone. A first-valid
+time is the **original submission boundary**, established retrospectively by
+these grades; it excludes independent grading latency and GPU cost.
+
+| Milestone, elapsed seconds from trial start | Control | TrainCheck available |
+| --- | ---: | ---: |
+| First source-supported diagnosis | 47.928 | 38.785 |
+| First runtime-supported diagnosis | 149.654 | 102.714 |
+| First visible implementation edit | 157.952 | 170.017 |
+| Earliest submission passing independent grading | 181.380 | 204.074 |
+| First agent-visible repaired-run comparison | 287.709 | 259.778 |
+| Final finish decision | 449.068 | 469.626 |
+| Terminal trial boundary | 453.645 | 474.343 |
+
+Event times are receipt-time upper bounds, not inferred internal reasoning times.
+The runtime-supported diagnosis is the first explicit statement connecting
+observed local denominators of 30/78 and synchronized rank gradients to the
+source hypothesis. Later gradient comparisons quantify the initial error at
+about 0.332294 relative L2. The treatment's first local repaired-run comparison
+failed because `safetensors` was absent; the table counts its subsequent
+successful comparison, not that failed command. Agent-visible verification
+remains distinct from the stronger independent grade.
+
+| Milestone resource cost | Control | TrainCheck available |
+| --- | ---: | ---: |
+| First-valid wall minus queue, seconds | 166.428 | 204.071 |
+| First-valid recorded CLI time, seconds | 138.627 | 149.629 |
+| First-valid allocated GPU-seconds | 25.043 | 37.548 |
+| First-valid cumulative input + output tokens | 595,207 | 845,768 |
+| First-valid input / cached input tokens | 586,784 / 536,192 | 835,843 / 772,480 |
+| First-valid output / reasoning output tokens | 8,423 / 2,093 | 9,925 / 1,859 |
+| Runtime-diagnosis allocated GPU-seconds | 25.043 | 25.025 |
+| Repaired-run-comparison allocated GPU-seconds | 62.597 | 62.588 |
+
+Within-turn token costs are intervals, bounded by the previous and current
+completed native turn. They are not interpolated from elapsed time:
+
+| Visible milestone, cumulative input + output interval | Control | TrainCheck available |
+| --- | ---: | ---: |
+| Source-supported diagnosis | 0–276,705 | 0–298,887 |
+| Runtime-supported diagnosis | 276,705–595,207 | 298,887–514,729 |
+| First implementation edit | 276,705–595,207 | 514,729–845,768 |
+| Repaired-run comparison | 778,431–982,728 | 845,768–1,142,119 |
+| Final finish decision | 1,207,772–1,471,632 | 1,558,352–1,744,783 |
+
+| Complete trial cost | Control | TrainCheck available |
+| --- | ---: | ---: |
+| Raw elapsed wall, seconds | 453.645 | 474.343 |
+| Recorded resource queue, seconds | 17.777 | 5.362 |
+| Wall minus recorded queue, seconds | 435.868 | 468.981 |
+| Recorded CLI lifetime, seconds | 286.349 | 309.522 |
+| Completed CLI invocations | 6 | 7 |
+| Requested executions | 5 | 6 |
+| Allocated GPU-seconds | 100.203 | 124.993 |
+| Input tokens | 1,455,546 | 1,724,856 |
+| Cached input tokens, subset of input | 1,373,184 | 1,641,472 |
+| Output tokens | 16,086 | 19,927 |
+| Reasoning output tokens, subset of output | 4,236 | 4,579 |
+| Input + output tokens | 1,471,632 | 1,744,783 |
+
+Input counts include reused context; they are processed tokens, not unique prompt
+text. Cached input is already included in input, and reasoning output in output.
+The native CLI does not expose underlying model API-request counts; six/seven CLI
+invocations must not be reported as six/seven model calls. The captured model
+flags record requested configuration, not an independent attestation of weights.
+
+The paired solver total is 225.196 allocated GPU-seconds. Independent grading
+added **1,416.962 operator GPU-seconds**, over 62 new supervised phases, with
+three earlier references reused. This cost is separate from solver costs and
+all qualification/preflight costs above. Cleanup and device assignments are
+verified for known allocation cost. These short runs include process startup
+and cleanup; GPU-seconds do not measure active utilization or pure training.
+No failed CLI attempt or transport recovery occurred in this pair. Queue-adjusted
+wall removes only recorded shared-resource waits, not all host variability.
+
+The treatment used one extra ordinary DDP request, costing 25.086 GPU-seconds,
+which completed 16 updates on both ranks without diagnostic hooks. Its preceding
+single-GPU request ran two separate eight-update configurations within a
+16-update allowance, not one uninterrupted 16-update run. The control's five
+requests completed 2/8/8/8/8 updates. Both agents checked accumulation 1 and 2;
+their terminal validation scopes were not identical. Exported checkpoints do
+not establish checkpoint-restore testing.
+
+The control's reference requests already executed its submitted correction;
+the treatment first captured an unchanged single-GPU reference before editing.
+Both follow the documented service contract. Neither reference backend is a
+hidden immutable corrected pipeline.
+
+## Interpretation and limits
+
+The treatment read the generic manual at turn 1, event 13. Review of all original
+completed item streams, changed source occurrences and returned public
+artifacts found no TrainCheck activation. Its diagnostics instead wrapped native
+Trainer calls and installed ordinary optimizer hooks. This is an observation
+about the captured evidence, not proof that an unobserved import never occurred.
+Tool availability and manual reading are not instrumentation activation.
+
+Both agents found a real data-dependent distributed-training problem, but both
+localized it before any requested execution. The observed case therefore fails
+the intended runtime-required localization criterion. Runtime state still
+confirmed the cause and quantified the gradient error. One optional-tool trial
+per arm, with no observed activation and different chosen validation scopes,
+cannot establish either a causal efficiency benefit or a slowdown from TrainCheck
+availability. The timing table is a descriptive result, not an improvement claim.
+This does not imply that agents can solve all bugs in the TrainCheck paper or
+that agent-written training pipelines cannot contain silent errors.
+
+## Evidence and reproduction
+
+Local evidence root:
+`/Users/matt/gitrepos/traincheck/outputs/ring18-token-normalization-20261102`.
+Remote evidence root:
+`/home/yuxuan/experiments/traincheck-token-normalization-20261102` on Ring18.
+
+- `provenance.json`: native archive/tree, environment, corpus and operator hashes.
+- `qualification-strict.json` and `qualification-model-calibrated.json`: separate
+  full comparisons with original and calibrated model tolerances.
+- `execution-ledger.jsonl`: successful runs and retained setup failures, commands,
+  elapsed time, devices and allocated GPU costs.
+- `operator/`: versioned qualification workers, analysis, regression tests and
+  commands; snapshots and native outputs remain in the remote experiment tree.
+- `TASK.md`, `pilot.py`, `protocol-reuse.json`: neutral task and isolated service
+  configuration. Final packets are under
+  `/private/tmp/traincheck-ddp-solvers-20261102-v2`.
+- `launch-plan.json`, SHA-256
+  `1046652737f8fe2a446f81d5c864a499a64ac9cdf9506ee6a157640f9cace98e`:
+  exact model/settings/budgets, execution order, code, packet and five gate hashes.
+- `actual-entry-positive-gate.json`, SHA-256
+  `d66be5d0c9b0acb7f61885df30e24a19d6786ead61f5d6a708a459e4f637991d`:
+  actual-entry parity and fresh-seed positive controls.
+- `independent-grading-contract.json`, SHA-256
+  `4ea06f939a254117cfb8fbb10c7b21e9f000b16f15d6e28ce520fe53da5977d5`:
+  the seven scenarios and unchanged numerical/coverage criteria.
+- `prelaunch-neutrality-review.json`, SHA-256
+  `a79346c2664a1e63796b1557bbec76906d1c08784b69fcd6274515bbc692888f`:
+  independent packet comparison and evidence review.
+
+
+- `solver-completion.json`, SHA-256
+  `c733a9fab181e100e1a19fbcaa3e7c119eaa4c2f561d6e60d91e5a5c6e5d10c6`:
+  both terminal records; all raw turns, source snapshots and service receipts
+  remain under `trials/`.
+- `milestone-annotations-visible.json` and `milestone-summary-visible.json`:
+  preserved adjudication before private grades, with valid-patch status pending.
+- `milestone-annotations-final.json`, SHA-256
+  `d4359e925180995d496f604155d054be1dbac97e8c864bee4979293cabc4c744`,
+  and `milestone-summary-final.json`, SHA-256
+  `e38bd48dd379db775572423503517af64183a4a6153f147f92e329ffec8d04d2`:
+  final source/runtime/verification/finish events, within-turn token intervals,
+  exact submitted-turn totals and backend-specific GPU accounting.
+- `visible-activation-audit.json`, SHA-256
+  `cba7ba4be95832c70f033612e22e84ceefd3d6f44540bb8978e6b695a9bcda6f`:
+  changed-source hashes and returned public artifact inventories/text checks.
+- `independent-grade-v1/summary.json`, SHA-256
+  `a63142a6c82cfaff98457e213a896b454a43d16eedd291c5978345526f327afa`:
+  all independent outcomes, exact-map deduplicates and separate operator cost.
+  `independent-grade-export-manifest.json` binds 89 downloaded report files.
+- `private-grading-join.json`, SHA-256
+  `6ef62ad56251cec1c09fb86cbbb05138cf2c6b29dcc12c623fc70d0371d8d8b1`:
+  verifies all 13 original complete source maps and their turn/submission hashes
+  against the grading plan and per-candidate manifests, including all seven
+  scenarios and both final identities.
+- `accounting/`: the preserved typed extractor and its provenance. Forty-three
+  focused synthetic tests and pinned strict mypy/format/lint checks passed during
+  preparation; the final extraction also revalidated actual receipt chronology,
+  native usage, backend allocation and evidence hashes. These checks do not
+  independently prove every scientific annotation or eliminate all harness bugs.
+
+Reproduce the final accounting from the local evidence root with a new output:
+
+```sh
+PYTHONDONTWRITEBYTECODE=1 /Users/matt/miniconda3/envs/traincheck/bin/python \
+  accounting/summarize_hf.py --study-root . \
+  --annotations milestone-annotations-final.json \
+  --output milestone-summary-reproduced.json
+```
+
+The local operator bundle `frozen-operator-20261102.tar.gz` contains the 35 files
+listed in `frozen-operator-manifest.json`; SHA-256 is
+`6dc24b4b154a4aedecf974d539cf17fd80fa0291bb8adf0b6a9c9eaadb45937d`.
+It is not a claim that every checkpoint or later report is inside that archive.
+Ring18 separately retains the existing runtime/source bundle
+`runtime-reproduction-20261102.tar.gz`, SHA-256
+`2f7f4c5643b919951f5f2f4bbb32455ab1b4275364e04a8046c0bb5a19f113fb`,
+created in place from material already on that machine. Automatic approval review
+rejected uploading the full private operator bundle because export authorization
+was not established. That upload was not retried; the local bundle and existing
+remote runtime archive are the retained alternative.
+
+The operator request service and qualification code do not modify the TrainCheck
+package. Large native sources, data, checkpoints and traces remain outside Git.
+The PR records reproducible evidence and limitations; it does not claim a general
+debugging speedup or silently promote a source-readable case to a runtime-only one.
diff --git a/docs/measurements/fsdp-source-consistency-pilot-20261102.md b/docs/measurements/fsdp-source-consistency-pilot-20261102.md
new file mode 100644
index 00000000..3f12f5b2
--- /dev/null
+++ b/docs/measurements/fsdp-source-consistency-pilot-20261102.md
@@ -0,0 +1,418 @@
+# Native FSDP initialization: optional TrainCheck pilot
+
+**Status: completed; both final repairs pass all six independent scenarios.**
+Both agents used cross-rank runtime diagnostics, but the optional-tool agent did
+not activate TrainCheck. This single pair does not establish a tool efficiency
+benefit. The earlier
+[tensor-parallel clipping](real-rank-runtime-pilot-20261102.md) and
+[DDP token-normalization](ddp-runtime-state-pilot-20261102.md) comparisons both
+produced source-first diagnoses and no observed TrainCheck activation. They are
+retained negative results. This candidate tests an external runtime condition:
+different workers can resolve the same model identifier to different cached
+weights before native FSDP partitions the parameters.
+
+## Provenance and reproduction boundary
+
+[Halo release PR #103](https://github.com/whitecircle/halo/pull/103) adds agreement
+on model source and commit across ranks. It is a production fix with regression
+coverage, contained in a larger release; the PR does not link a specific incident
+report. The parent is `8bc800971e739777311862544e5f1a321e12e4ba`, and the merged
+release is `5abddcb7ee12b46481f6d0facb00046c358724fb`.
+
+The reproduction executes byte-identical selected parent definitions, including
+the native cache-loading helpers and c10d-store coordination, bound to real
+installed Hugging Face Hub, Accelerate and PyTorch APIs. Native GPTNeoX loading
+and Torch FSDP perform the training. It does not run the complete current Halo
+package: that package requires a newer Python/framework/CUDA stack than the
+shared Ring18 installation. The original source and extraction hashes remain
+available for review; no fake Hub API or replacement collective implements the
+failure.
+
+The [official Pythia model card](https://huggingface.co/EleutherAI/pythia-14m#model-details)
+documents replacement of the model previously served as `EleutherAI/pythia-14m`,
+moving that earlier model to `EleutherAI/pythia-14m-deduped`. The experiment uses
+unchanged official artifacts:
+
+| Artifact | Revision | Safetensors SHA-256 |
+| --- | --- | --- |
+| Earlier model, now `pythia-14m-deduped` | `f33025648652797a390d8c54835273845b437161` | `821667edaab2cb38aac634fd1bbda547ecf451f4ea514f47b5a0080be24154fb` |
+| Replacement `pythia-14m` | `94f7c35d5e9f2e9bac8ca839329f505b4d007d5d` | `116a02532db461f91386a5b20f942ff2c8d4de7341e21b55caafc3d7b25f49a1` |
+
+These are separate repository histories, not ancestor/descendant commits. The
+historical cache namespace and `refs/main` mapping are reconstructed around those
+official blobs. This is not a preserved incident cache, and there is no claim
+that the Halo failure mode involved Pythia. The model migration establishes an
+authentic source of different published weights, not a record of that particular
+deployment.
+
+Two worker-local cache domains emulate logical nodes on one physical Ring18 host,
+using GPUs 4/5, both NVIDIA A16. Each worker has its own local rank zero and cache
+metadata; global ranks and the actual NCCL process group remain distinct. This
+does not measure multi-host networking or a filesystem race. Requests use
+independent copies of the official caches, not writable links to shared source
+blobs. The shared repositories, environment and driver are unchanged.
+
+The execution environment uses Python 3.10.14, PyTorch `2.2.2+cu121` and compiled
+CUDA 12.1. The loaded Transformers source is pinned to
+`a17f287ac039f92835b5cd9bd8ee28b584c9f65e`; the full environment and executed
+source-file hashes are saved with the launch evidence.
+
+## Workload and qualification
+
+The workload uses the official 14,067,712-parameter GPTNeoX model, explicit native FP32
+loading, eager attention, no TF32, deterministic algorithms, AdamW with learning
+rate `1e-4` and weight decay zero, sequence length 64 and two global examples per
+update. The qualification uses two optimizer updates. Identical pretokenized
+README-derived inputs isolate the model-loading question from the official
+tokenizers' different postprocessing configurations.
+
+The raw checkpoints have different schemas: the replacement has 76 FP16 tensors;
+the earlier artifact has 88 FP16 tensors and six Boolean tensors. The additional
+18 entries include legacy attention buffers. Native loading reports no missing
+or unexpected keys, but that alone is insufficient: the pinned loader can filter
+buffer keys from its unexpected-key report, and `state_dict()` omits nonpersistent
+buffers. The qualification therefore retains all raw keys and loading messages,
+and separately captures complete named runtime buffers. Raw artifact equality
+and runtime equality are different claims.
+
+The first complete source-state check confirms that all 76 learned tensors load
+exactly as native FP16-to-FP32 conversion, and both models have 19 bitwise-equal
+runtime buffers. Native loading recreates some values: the earlier raw
+`masked_bias` is negative infinity while the runtime uses finite `-1e9`, and raw
+FP16 rotary frequencies differ from their recreated FP32 values. These are
+recorded loader behaviors, not values discarded to manufacture equality.
+
+The bounded matrix includes uniform earlier/replacement sources, mixed and
+swapped sources, their one-GPU references, and same-world runs with additional
+intermediate observations removed. A missing-cache control must produce the
+native offline failure. Every run retains worker lifetime, allocation, cleanup,
+source/cache integrity and raw failures. Native creation of `hub/version.txt`
+initially caused an overly strict cache-integrity receipt to fail; that attempt
+and its cost are retained, with immutable blob/ref checks separated from the
+recorded metadata initialization.
+
+The 18 recorded qualification attempts through the full-buffer matrix consumed
+435.376 allocated GPU-seconds, including the missing-cache control and failed
+integrity receipt. This is operator qualification cost, not agent cost; later
+side probes, service preflight and grading must be accounted for separately.
+
+Independent checks of the initial parameter-focused matrix established:
+
+- Each uniform initial model exactly matches its corresponding one-GPU source.
+- Native FSDP constructs the mixed model from 38 parameter tensors belonging to
+  one source and 38 belonging to the other. The result differs from each complete
+  authentic source in 38 parameter tensors. The swapped case follows the actual
+  native shard ownership too; this is checked from values and padding metadata,
+  not inferred from filenames or a model hash alone.
+- All ranks can reconstruct the same hybrid. Post-wrap replica equality alone
+  therefore does not establish coherent initialization.
+- Same-world observed/unobserved final parameter, gradient, optimizer, buffer and
+  recorded RNG values agree exactly for the checked pairs. The follow-up checked
+  104 overlapping artifacts against earlier runs, 30 buffer comparisons, 34
+  replica checks and 20 terminal observer comparisons, all exactly equal.
+
+The original cross-world model tolerance **failed** for healthy uniform controls.
+With model `atol=1e-5` and `rtol=1e-4`, maximum two-update differences were about
+`3.758e-4` for the earlier model and `3.328e-4` for the replacement. Initial states
+were exact. First-gradient relative L2 differences are approximately 0.5275% and
+0.04506%, respectively. A key-bias-only explanation was not supported: most
+first-step elementwise model violations occur elsewhere, including weights.
+Measured gradient differences and Adam's update rule explain update sensitivity,
+but do not identify its kernel-level numerical cause. It is not reported as
+ordinary rounding proved harmless.
+
+The failed criterion remains failed; no threshold was silently relaxed. The
+operator explicitly selected a separate pre-agent scope: exact initial source
+coherence and full same-topology healthy trajectories, without a claim of narrow
+single-versus-two-GPU numerical parity. Its gate must bind both the successful
+state checks and the retained failed original criterion. Two additional bounded
+single-GPU controls used one sample per microbatch and two accumulation steps,
+matching each distributed worker's forward batch shape. Both models then matched
+the two-GPU parameters bitwise at steps 0, 1 and 2. All buffers and recorded RNG
+values matched. The replacement model's gradients and optimizer state matched
+exactly; the earlier model retained one subnormal gradient/moment discrepancy,
+at most about `1.5e-43`, without a parameter difference. These controls consumed
+24.690 allocated GPU-seconds. They localize the discrepancy to batch shape and
+accumulation order, without proving which numerical kernel causes it. Neither
+the submitted workload nor the original failed criterion changed.
+
+## Matched comparison and acceptance
+
+The neutral task extends a pretrained fine-tuning pipeline to two-GPU
+FSDP, supplies ordinary short-run outputs, and requests a readiness decision and
+justified revision. It does not assert a bug, prescribe a diagnosis or require
+TrainCheck. This is pretrained fine-tuning, not checkpoint restoration with
+optimizer and step state.
+
+Both arms receive identical editable runtime source, data, ordinary diagnostics
+and execution budgets. Treatment adds the same generic optional TrainCheck
+manual and implementation used in the earlier pair. Both current and reference
+backends execute the submitted source. Both ordinary eight-update service runs
+passed, using 46.958 allocated GPU-seconds in total. Their genuine logs and all
+16 curve rows are provided. These are cyclic-sample short-run training losses,
+not held-out model-quality measurements.
+
+The private actual-entry observer and six-scenario evaluation passed before
+launch: six baseline runs produced the expected four healthy and two incoherent
+outcomes, and six positive controls passed after changing only
+`sync_module_states=True`. Mixed and swapped positives selected different
+authentic sources, so the evaluator does not require the latest version. These
+12 qualification jobs consumed 328.727 allocated GPU-seconds, including the
+previously qualified actual-entry mixed run once.
+
+The observer captures complete named parameters and buffers, optimizer and
+recorded RNG values, local gradients, actual forward inputs and native shard
+metadata. Step 0 is explicitly after the first backward and before the first
+optimizer step. It supports multiple preliminary loads and unused optimizers,
+binding the trained model through actual FSDP or stepped-optimizer identity.
+Multiple root FSDP constructions or multiple actually stepped optimizers are
+unsupported and must produce an unknown grade. The observer does not establish
+live alias/storage identity beyond its recorded native layout.
+
+The optional-tool preflight compared two-update runs with and without selective
+TrainCheck observation of scalar `dropout` arguments. Each rank executed 38
+native calls and emitted exactly 38 entry and 38 return records. All 42 saved
+state comparisons matched, covering 2,050 tensor and 5,742 scalar checks, with
+no unmatched, duplicate or variable-tracker records. An independent counter-only
+run also matched the saved no-counter reference. The two new jobs used 69.336
+allocated GPU-seconds. This establishes compatibility for that interface; it is
+not a repeated overhead benchmark or validation of arbitrary APIs, tensor
+metadata, model tracking or the inference engine.
+
+Uniform-source controls must retain their respective complete initial models.
+Mixed/swapped inputs may be repaired to either authentic coherent source; there
+is no hidden requirement to select the newest one. The subsequent full runtime
+model, gradients, optimizer, input coverage and recorded RNG must follow that
+source's independently qualified same-topology trajectory. Detecting a mismatch
+and aborting is recorded as detection/containment, not as a completed working
+repair under the public contract.
+
+Keep the previous diagnosis definitions. A specific pre-run explanation of
+per-worker source disagreement and FSDP assembly counts as a source-supported
+diagnosis even if the agent calls it a hypothesis. Later runtime confirmation
+does not make that diagnosis runtime-first. If agents again solve the problem
+from source, report it; do not force execution or tool use to make the case look
+harder.
+
+The pair retains six requests of 1–16 global updates, 240 seconds per
+worker request, 900 seconds aggregate CLI execution and eight decision turns.
+Measure source diagnosis, runtime-supported diagnosis, first independently valid
+submitted repair, visible verification and final completion separately. Within-
+turn tokens remain intervals; GPU cost is supervised allocated-device lifetime,
+including startup and cleanup. Qualification and private grading costs are
+separate from solver costs. No tool efficiency benefit is established.
+
+An independent review of the exported packets found 3,790 shared files: all
+except the optional-offer prompt are byte-identical. The treatment adds 32 pinned
+tool/manual files. Both arms can edit the same 3,346 Python source files and have
+the same fixed entry point, data and contracts. Original source, native names,
+warnings and genuine tokenizer-revision provenance remain visible; they could
+enable a source-first diagnosis. The review establishes no deliberately added
+cause or issue hint, not that the case requires runtime observation. Both fresh
+CLI agents use `gpt-6-astra`, high reasoning and priority service. GPU requests
+are serialized, with queue time reported separately.
+
+## Completed comparison and independent repair validation
+
+Both trials finished without CLI or transport failures. Control submitted five
+decision turns and used four GPU requests; optional TrainCheck submitted four
+turns and used three requests. Every submitted source is retained. Nine ordered
+source occurrences reduce to five exact complete source maps. All 30 independent
+scenario replays finished: 26 passed, four failed as expected for initial
+unrepaired sources, and none were unknown. Both earliest valid submissions are
+turn 2, with no earlier unresolved source. Both final submissions pass all six.
+
+| Submitted source occurrences | Independent outcome |
+| --- | --- |
+| Control turn 1 | Four healthy cases pass; mixed and swapped sources fail |
+| Control turns 2–3 | Six of six pass; earliest valid repair is turn 2 |
+| Control turns 4–5, including final | Six of six pass |
+| Optional TrainCheck turn 1 | Four healthy cases pass; mixed and swapped sources fail |
+| Optional TrainCheck turns 2–4, including final | Six of six pass; earliest valid repair is turn 2 |
+
+The replay executes the immutable ordinary training entry, with complete source
+integrity and state/input coverage checks. Both repairs change the training
+pipeline only by enabling `sync_module_states` and adding comments. Standalone
+diagnostic/comparison helpers differ across snapshots and were retained in their
+complete source identities, even when not imported by that entry.
+
+| Milestone, seconds | Control raw | Control queue-adjusted | Optional TrainCheck raw | Optional TrainCheck queue-adjusted |
+| --- | ---: | ---: | ---: | ---: |
+| Pre-run mechanism hypothesis | 30.251 | 30.251 | 33.934 | 33.934 |
+| Runtime-confirmed state inconsistency | 131.170 | 131.168 | 184.793 | 134.135 |
+| First independently valid submitted repair | 148.365 | 148.362 | 204.705 | 154.047 |
+| First agent-visible repair verification | 228.722 | 224.517 | 271.333 | 218.093 |
+| Terminal trial boundary | 432.754 | 408.810 | 414.381 | 333.094 |
+
+Queue-adjusted time subtracts only measured shared-service waiting. Visible
+statements use received-event timestamps, an upper bound on creation time.
+First-valid submission time is retrospective; it excludes later private grading
+latency and is distinct from the agent's own verification. Both agents' upstream
+cache-origin diagnosis remains unresolved, as detailed below.
+
+| Cost through first valid submission | Control | Optional TrainCheck |
+| --- | ---: | ---: |
+| Native CLI elapsed seconds | 93.470 | 99.253 |
+| Input tokens, including cached input | 714,921 | 683,299 |
+| Cached input subset | 661,504 | 628,352 |
+| Output tokens | 4,994 | 5,031 |
+| Input plus output tokens | 719,915 | 688,330 |
+| Allocated solver GPU-seconds | 35.563 | 34.992 |
+
+Native counters do not identify tokens at an intermediate statement. The
+runtime-confirmation milestones therefore have cumulative input-plus-output
+brackets of 293,535–719,915 for control and 304,371–688,330 for treatment. The
+pre-run hypotheses occur within the first turn, giving only 0–293,535 and
+0–304,371. No interpolation is used.
+
+| Full trial | Control | Optional TrainCheck |
+| --- | ---: | ---: |
+| Raw elapsed seconds | 432.754 | 414.381 |
+| Measured GPU-service queue seconds | 23.944 | 81.287 |
+| Input tokens, including cached input | 1,358,537 | 1,127,302 |
+| Cached input subset | 1,298,944 | 1,066,496 |
+| Output tokens | 11,210 | 9,983 |
+| Reasoning output subset | 1,910 | 2,931 |
+| Input plus output tokens | 1,369,747 | 1,137,285 |
+| Allocated solver GPU-seconds | 118.764 | 82.527 |
+
+Cached input is already included in input tokens, and reasoning is already
+included in output tokens. Neither subset is added again. The seven solver
+requests consumed 201.291 GPU-seconds. The five operator qualification groups
+above total 905.087 GPU-seconds, counting reused actual-entry/no-counter evidence
+only once. Private post-trial grading used another 815.932 GPU-seconds across
+30 newly executed phases; reused frozen oracles added zero GPU work. These three
+categories total 1,922.310 allocated GPU-seconds for this case. GPU reservations
+were released after completion. Oracle self-checks ran on the CPU controller;
+their preflight elapsed time was about 52.47 seconds, including imports and hash
+validation, and is not reported as pure oracle-comparison time.
+
+The optional-tool agent read the TrainCheck manual but neither prepared nor
+activated TrainCheck instrumentation. Both arms used ordinary custom diagnostic
+wrappers, and all returned artifacts contained zero native TrainCheck API or
+variable-trace files. Their validation choices differed: control compared eight
+updates across backends and then ran 16 ordinary distributed updates; treatment
+compared 16 diagnostic updates across backends. Control checked the 76 saved
+parameter tensors, while treatment also checked nonpersistent named buffers,
+covering 95 records. Full-trial cost differences therefore reflect different
+chosen validation work, not matched measurements of a TrainCheck repair-speed
+benefit.
+
+Both agents proposed an unsafe-initialization mechanism before the first run:
+unsynchronized FSDP could assemble inconsistent rank-local tensors. However,
+their specific proposed origin was RAM-efficient loading skipping nonzero-rank
+weights. Actual runtime diagnostics found the loading flags absent and rejected
+that explanation, while confirming cross-rank initial-state differences and
+38 changed tensors after sharding. Both submissions enabled state synchronization
+in turn 2. Neither final answer identified the different cached model sources.
+The result must therefore distinguish a source-supported mechanism hypothesis,
+runtime-confirmed state inconsistency, and the upstream cause unresolved by
+either agent. Operator qualification did establish the cache-version divergence.
+This is not evidence of full root-cause localization by either agent from source
+or from runtime.
+
+## Saved evidence and current validation
+
+Local evidence root:
+`/Users/matt/gitrepos/traincheck/outputs/ring18-source-consistency-20261102`.
+Remote root:
+`/home/yuxuan/experiments/traincheck-runtime-study-c-20261102`.
+
+The roots retain the official source/artifact provenance, original and revised
+qualification specifications, all attempted native runs, and typed operator
+helpers. `evaluation/source-proof-v1.json` records raw-to-loaded tensor and buffer
+checks plus exact native shard reconstruction, SHA-256
+`380d01eef23cb4e0ecc589bcb4db9c1b42d2beb120c2786cf04eb29ca174dd38`.
+`evaluation/native-review-v1.json`, SHA-256
+`15eaee07020af3c9a4a7472e4976aa4690cdecbc9d552649d1cc040f06e5e856`,
+records complete observer/replica overlap and numerical differences.
+`evaluation/original-crossworld-gate-v1.json`, SHA-256
+`8d7b0be8c6a16504b474eac002c649b2b0e22b416752f4e8700c173a86c92dc1`,
+preserves the original failed cross-world criterion with unchanged thresholds.
+`evaluation/accumulation-diagnostic-v2.json`, SHA-256
+`7a170941297b2a8a7bc2a394d21cb6dace3d0a1b616a8a5e7565d515a721c126`,
+records the additional batch-shape controls.
+
+The frozen launch plan has SHA-256
+`c02a5cb735b52a00b918c5dd456415eadf7d97b4ee05f61f509da4dfc20fc4b2`.
+It binds the packets, environment, operator code, native qualification, service,
+actual-entry positive controls, optional-tool preflight and independent grading
+contract. The independent neutrality review is
+`prelaunch-neutrality-review.json`, SHA-256
+`ffe22a0d859f9e571fbd5e5ff1e53a605db4a87e6e643dbdf8204564788a3b36`.
+The six-scenario grading contract has SHA-256
+`6abfeddd99d16aa109424f4cdec899708c3dfa78aa847efad785487fa517a43c`.
+
+The completed grading summary is
+`grading-results-v1/independent-grade-v1/summary.json`, SHA-256
+`9b2d524617f61d549aa59a948cf7767deb5736637e5f9c5815c91bc93f2b13c3`.
+Its replay plan has SHA-256
+`2890a5e00cabbdd934c4f327e9c695c5e577a40fdcad3fa4c51c162f133eb5bf`.
+All 165 exported reports/receipts were downloaded and hash-verified against
+`grading-result-export-manifest.json`, SHA-256
+`3e67d014a07063c258e9a7205c1e0513da2661095d137195f7ac7997c383f060`.
+Full tensor snapshots remain in the isolated Ring18 evidence directory.
+`grading-reproduction-and-results.json` records exact source-to-occurrence
+mapping, replay commands, per-scenario results and cost scope.
+Independent export verification is saved in
+`evaluation/grading/export-review-v1.json`, SHA-256
+`09d00a2e71f3b1515b3ebb169223fc9ba3610ed35f6b0bfd0b82bdc90720e493`.
+The final visible-milestone/grade join is `private-grading-join.json`, SHA-256
+`07caa05c568cf069157cc973fc8dd56d106dd80fbc83aaa4c1507e0b5bf63add`.
+`milestone-summary-final.json`, SHA-256
+`4729939543dbec89cb0feed9d0f98cbee0040bf4437494ba422f5becca1521ec`,
+records the complete timing, token and GPU accounting. Earlier visible-only
+annotations remain unchanged; private grades do not retroactively create
+agent-visible milestones.
+
+A local operator checkpoint contains 158 code, contract, review and accounting
+files in `operator-evidence-final-v1.tar.gz`, SHA-256
+`d47f9017f81bbd2da95cf78e2933ea30ba9e115623f23d52408886103f3a2f8f`.
+Its verified file inventory is `operator-evidence-final-v1.json`. Large native
+states and original solver source snapshots remain at their recorded locations.
+
+Both agents' completion and every submitted source occurrence were frozen before
+grading. Only identical complete source maps shared a result. Evaluation
+compares ordered keys, shapes, dtypes and full tensor values against independent
+same-topology oracles; it does not infer correctness from the patch syntax.
+Missing evidence, malformed snapshots and unsupported observer coverage remain
+unknown. The bounded validation covers two updates, one seed and one small
+corpus, not generalization to other workloads.
+
+The new request service and pilot adapter passed 35 focused local tests, strict
+mypy for their changed helpers, and pinned formatting/lint checks. Packet
+preparation passed nine additional tests and strict checks. The observer passed
+six regression tests, the optional-tool preflight helpers passed 20, and the
+reused accounting helpers passed 50, each with scoped strict typing and pinned
+format/lint checks. These counts describe separate validation scopes and should
+not be summed as unique tests. The native gates also passed under the explicit
+same-topology scope above. The post-trial grader passed 14 focused tests plus
+pinned strict mypy, Ruff, Black and isort checks, independently rerun by a second
+reviewer. Its negative-path tests cover incomplete cost receipts, duplicate
+finishes, malformed snapshots and a confirmed incorrect scenario alongside an
+unknown one. The TrainCheck package is unchanged.
+
+The recorded solver launch, from the local evidence root after qualification
+and packet export, was:
+
+```sh
+PYTHONDONTWRITEBYTECODE=1 /Users/matt/miniconda3/envs/traincheck/bin/python pilot.py freeze
+PYTHONDONTWRITEBYTECODE=1 /Users/matt/miniconda3/envs/traincheck/bin/python pilot.py check-launch
+PYTHONDONTWRITEBYTECODE=1 /Users/matt/miniconda3/envs/traincheck/bin/python pilot.py run
+```
+
+The independent replay uses the frozen plan on Ring18:
+
+```sh
+/home/yuxuan/experiments/traincheck-token-normalization-20261102/venv/bin/python \
+  /home/yuxuan/experiments/traincheck-runtime-study-c-20261102/evaluation/grading/grade.py \
+  --plan /home/yuxuan/experiments/traincheck-runtime-study-c-20261102/grading-input-v1/plan.json \
+  --execute-after-both-freezes
+```
+
+`grading-payload-allowlist.json` binds the staged execution inputs;
+`grading-source-inventory.json` retains every original submission. The payload
+contains public submitted source and minimal grading inputs, excluding raw CLI
+transcripts and private model snapshots. Preserve the frozen directories; a new
+repetition needs fresh isolated output roots and regenerated qualification and
+launch manifests. Per-request receipts retain the exact native training commands.
diff --git a/docs/measurements/real-rank-runtime-pilot-20261102.md b/docs/measurements/real-rank-runtime-pilot-20261102.md
new file mode 100644
index 00000000..8921b60b
--- /dev/null
+++ b/docs/measurements/real-rank-runtime-pilot-20261102.md
@@ -0,0 +1,485 @@
+# Real cross-rank runtime case: agent-efficiency protocol
+
+**Status: completed four-trial pilot, 2026-11-02.** All four agents identified the
+clipping cause from source before their first GPU request. Each first submitted
+a valid repair in turn 2, later passing all four private scenarios. No TrainCheck
+instrumentation activation was observed in either treatment. This is a source-localizable
+case and does not establish a TrainCheck efficiency benefit. Transport failures
+and same-thread recoveries remain in the raw evidence; timing below reports both
+raw and adjusted values. This is operator documentation, excluded from solver
+packets.
+
+The first two-update run had finite losses of 10.39899 and 10.62776. Among 27
+replicated trainable parameters, cross-rank mismatches increased from zero before
+the first update to 6 after update 1 and 18 after update 2. The corrected control
+passed 630 model, 774 optimizer, 42 RNG and 162 replicated-parameter checks against
+itself. Comparing the vulnerable run against that control failed 140 checks,
+with maximum absolute model difference 0.00067138671875. These are qualification
+results, not measurements of agent performance.
+
+Removing inherited research annotations and isolating ordinary imports preserved
+all archived state exactly: each vulnerable/corrected comparison covered six
+rank/step snapshots, 384 tensors, 144,752,448 tensor bytes, 378 scalar values and
+606 ordered dictionary keys, with no differences. Recorded CPU/CUDA/Megatron RNG
+and optimizer values are included; unrecorded state and live object aliasing are
+outside that comparison. Evidence: `private-state-parity.json` in the local
+experiment directory below.
+
+Both ordinary eight-update runs complete with finite losses and zero skipped or
+NaN iterations. Final FP16/BF16 losses are 10.25196/10.25426. This short difference
+does not establish a quality regression. The neutral task presented measured
+curves without asserting one. An issue-named installation-path prefix in exported
+text is replaced with `/runtime/python-env`, identically for both arms; raw logs,
+exact mappings and original/public hashes remain operator evidence. Error/warning
+wording, source filenames, line numbers and numeric values are preserved.
+
+The private clipping observer records actual FP32 gradient selection and values
+around native clipping, using the pipeline grid's slice-parallel rank. It is never
+automatically provided to solvers. Its 26 CPU regressions cover call/exception
+preservation, RNG and model state, flattened gradient views, real grid rank APIs,
+missing/malformed metadata and duplicate evidence. Together with evaluator and
+control-import tests, **91 tests pass**. Pinned hooks and explicit strict mypy
+pass for the observer and tests; the repository mypy hook alone skips tools.
+
+Setup failures are retained, including one incorrect private grid API and native
+shutdown crashes after successful updates. The worker now explicitly releases
+distributed process groups after application exit. A successful rerun alone does
+not prove elimination of an intermittent shutdown failure; later receipts and
+failure counts remain part of the results.
+
+The development base is `codex/checkpoint-continuation-costs`,
+[PR #170](https://github.com/OrderLab/TrainCheck/pull/170), at
+`c79218c960a823f954bf49dad354975da39e6f24`. Subsequent source revisions, workload
+configuration, packet manifests and commands were frozen before solver exposure.
+The final evidence references are recorded below.
+
+## Question and case selection
+
+The question is whether optional access to TrainCheck helps an agent investigate
+and repair a real distributed training problem more efficiently under a neutral
+engineering task. The comparison measures causal localization, independently
+valid code, agent-visible runtime verification and final completion separately.
+
+The selected candidate is the historical DeepSpeed BF16 optimizer failure commonly
+identified as **DS-1801** in the TrainCheck paper. Tensor-parallel training includes
+both partitioned and replicated parameters. The affected implementation excludes
+replicated parameters on nonzero TP ranks from clipping, although excluding their
+duplicate contributions is appropriate when calculating the global norm. Thus
+replicated parameters can receive different updates across TP ranks while the
+training process continues.
+
+| Candidate considered | Reason to consider it | Selection limitation |
+| --- | --- | --- |
+| DS-1801, BF16 clipping under tensor parallelism | Real silent training defect; combines rank, replication metadata, clipping activation and optimizer state; an existing Megatron–DeepSpeed environment and CodeParrot data are available on Ring18. | The repair is small once the relevant framework path is identified. Source inspection or prior knowledge can still succeed. |
+| [PyTorch #104336](https://github.com/pytorch/pytorch/issues/104336), DDP device transfers | Real failure of gradient synchronization after moving an already wrapped model between CPU and GPU. | The device-transfer sequence is a recognizable source-level clue; it may repeat the weakness of the earlier pilot. |
+| [DeepSpeed #5794](https://github.com/deepspeedai/DeepSpeed/issues/5794), MoE communication | Rank-local routing and collective arguments provide relevant runtime evidence. | The available artifact uses an inference workload with additional dependencies and a hang; it is less direct for this silent-training study. |
+
+DS-1801 is selected for its real provenance, training relevance and available
+environment, not because a preliminary agent failed to solve it. All subsequent
+qualification attempts and failures must be retained. This study does not claim
+that cross-rank observation is logically necessary for every successful diagnosis.
+Source-based successes remain valid results.
+
+## Historical and current source provenance
+
+The upstream repository is now `deepspeedai/DeepSpeed`.
+[PR #1801](https://github.com/deepspeedai/DeepSpeed/pull/1801), titled
+“bf16+pipeline parallelism,” is merged. The following sources were inspected:
+
+- The [vulnerable parent
+  `a4cbf0c1d300d78c6124656458115d1050869142`](https://github.com/deepspeedai/DeepSpeed/blob/a4cbf0c1d300d78c6124656458115d1050869142/deepspeed/runtime/bf16_optimizer.py#L298)
+  applies the nonzero-TP-rank filtering outside `if not for_clipping`.
+- The [fix commit
+  `e24814a10de04ce280efe2adb027b023e3336493`](https://github.com/deepspeedai/DeepSpeed/commit/e24814a10de04ce280efe2adb027b023e3336493)
+  moves the condition inside that branch. Its literal intermediate patch uses
+  `p` instead of `lp`; this commit alone is not the executable positive control.
+- The [final merged source
+  `56c522386878df2513ccb548de76b50191ae3af7`](https://github.com/deepspeedai/DeepSpeed/blob/56c522386878df2513ccb548de76b50191ae3af7/deepspeed/runtime/bf16_optimizer.py#L406)
+  has the corrected branch and `lp` reference.
+
+**Correction to an older discovery note:** the live Ring18 source inspection for
+this study found that DeepSpeed HEAD
+`32d97976ce08f00228ceb55a2d93005bd8b2769a` has the correct nested condition in
+`bf16_optimizer.py`. The working-tree diff moves that condition outside
+`if not for_clipping`, making the inspected dirty working tree vulnerable. The
+older `outputs/trace-bug-impact/checkpoint-discovery.md` described this edit as a
+repair; that characterization must not be used as provenance for this study.
+
+The saved live diff is
+`/Users/matt/gitrepos/traincheck/outputs/ring18-runtime-20261102/original-bf16-working.diff`,
+SHA-256 `c04efe0f7f59bbcd973a6289a57b40fa4f832a7855b1b4ada9ebb73e3fcde8ad`.
+Its two changed lines move the identical condition and `continue` out by one
+indentation level. This artifact was read and hashed locally; the complete
+source snapshots were subsequently frozen in the qualification and packet
+manifests.
+
+The candidate is an isolated snapshot of the inspected working-tree source.
+The positive control uses the corresponding correct HEAD version of
+`bf16_optimizer.py` while preserving the same remaining compatibility changes.
+The evidence bundle retains the exact source diff and per-file hashes,
+including the check that the control differs only in the intended logic. This is a
+historical semantic reproduction on a compatible environment, not execution of
+an untouched historical checkout. Shared repositories and installed environments
+are inputs and must remain unchanged.
+
+## Workload and qualification
+
+The frozen native Megatron–DeepSpeed GPT workload uses CodeParrot, four layers,
+hidden width 64, eight attention heads, sequence length 128, TP2, PP1, DP1,
+BF16 and ZeRO stage 0. Micro/global batch sizes are both 1. Apex FusedAdam uses
+learning rate 0.0005, beta2 0.999, weight decay 0.1 and clipping threshold 1.0.
+Seed 1234 is public; 5678 is the additional validation seed. Ordinary public runs
+contain eight updates. Private grading uses two updates, capturing steps 0–2.
+
+Named parameters contain 1,722,880 elements per TP rank, including 9,856 replicated
+elements. Counting each partition once and replicated parameters once gives
+**3,435,904 logical parameter elements**. This is the GPT case used here, not
+the earlier TorchTitan tiny-Llama trace. Ring18 uses NVIDIA A16 GPUs, Python
+3.10.14 and Torch 2.2.2+cu121. GPU idleness is checked before each phase.
+
+Qualification confirms the cause: clipping selects 27/27 replicated gradients on
+rank 0 and 0/27 on rank 1 in the vulnerable implementation. The first-step maximum
+cross-rank gradient gap grows from exactly zero before clipping to
+0.7917701378 after clipping; the corrected control retains zero gaps. Clipping-
+disabled and TP1 candidate/control states match exactly. The fresh seed reproduces
+the discrepancy. Selective TrainCheck preflight records two real clipping calls
+per rank with complete entry/return pairs, no exceptions and exact recorded
+model/optimizer/RNG parity. It observes scalar norm arguments, not tensor/model
+tracking. Full evidence is in `qualification-summary.json`.
+
+Source-integrity checks cover all 271 framework/application Python files, all
+139 cached native sources and four binaries. All remain unchanged. Private
+snapshots stay on Ring18; compact reports, hashes and ordinary outputs are local.
+
+The prespecified qualification gates were:
+
+1. The candidate initializes successfully and executes finite training updates.
+   The measured gradient norm actually activates clipping; configured clipping
+   without an active event does not qualify.
+2. Replicated trainable parameters agree at initialization and diverge after an
+   affected update. Compare logical parameter identity within the relevant TP
+   group, not the complete partitioned `state_dict` across all ranks.
+3. The isolated corrected control preserves replicated-parameter agreement and
+   completes the same training computation. Capture full values, shapes and
+   dtypes, with any numerical tolerance declared before grading.
+4. Pre/post-clipping FP32 gradient observations explain the differing updates.
+   BF16 parameter rounding can mask an early difference; missing or unchanged
+   low-precision observations are not proof that clipping was equivalent.
+5. Clipping-disabled and TP1 controls separate the triggering conditions from
+   unrelated startup or precision problems. Fresh seeds, and an additional
+   TP2×DP2 configuration if feasible, provide independent validation.
+6. An instrumentation preflight preserves the relevant model, optimizer and RNG
+   behavior and establishes actual observation coverage. This preflight is
+   operator work, not solver use of TrainCheck.
+
+State observations must identify the run, global update, rank/group membership
+and logical parameter. Wall-clock timestamp ordering across processes alone is
+not sufficient alignment. Setup errors, nonactivating conditions, absent
+observations, failed comparisons and successful comparisons remain separate
+outcomes. Qualification and preflight GPU costs are recorded separately from
+solver costs.
+
+## Neutral solver task and matched access
+
+The intended engineering context is an FP16-to-BF16 extension of the existing
+tensor-parallel pretraining pipeline. The core prompt is:
+
+> We are extending this language-model pretraining pipeline to BF16 while
+> retaining tensor parallelism and the existing training configuration. The
+> supplied outputs show the current short training run. Assess whether this
+> implementation is ready for a longer run, decide what to investigate next,
+> and implement any justified revision. Explain your conclusion using the
+> evidence you collected.
+
+Supply measured ordinary training outputs, configuration and relevant full
+application/framework source. If curves show no material quality difference,
+do not describe a quality regression that was not observed. Do not include this
+protocol, issue identities, Git history, the private corrected source, a targeted
+checker, or bug/fix annotations in solver packets. Preserve ordinary source names
+and behavior; do not obscure code to make static reasoning artificially harder.
+The exact allowed revision scope and preserved training contracts must be stated
+and frozen before launch.
+
+Two fresh matched pairs ran, with launch order reversed in the second pair.
+Both used gpt-6-astra/high, the same tool interface and runtime, six remote
+requests, 1–16 updates per request, 240 seconds per worker phase, 900 aggregate
+CLI seconds and eight decision turns. These limits were frozen before exposure.
+Controls may use ordinary logging, assertions and custom Python diagnostics.
+Treatments additionally receive the generic TrainCheck observation documentation
+and implementation; no rule targets this case and no task step requires its use.
+
+Standard responses return native logs, training metrics and ordinary artifacts.
+Do not automatically supply private grader state exports as a diagnostic shortcut.
+Either arm may request and capture equivalent runtime state explicitly under the
+same resource budget. Private grading began only after all final submissions
+froze; grader feedback was not supplied during the task.
+
+## Milestones, costs and evidence
+
+| Milestone | Definition |
+| --- | --- |
+| First supported localization | First received visible statement containing a sufficient causal explanation supported by evidence available to the agent at that time. Source evidence qualifies; later traces cannot be credited retroactively. |
+| First valid submitted implementation | Earliest captured submission later passing all prespecified independent contracts. This is a retrospective correctness measure, not proof the agent had already verified it. |
+| First agent-visible runtime verification | First returned runtime evidence that supports the submitted repair under the declared checks. State exactly which conditions were checked; this can precede or follow independent qualification of that submission. |
+| Final completion | Agent's final frozen implementation and report, including all accepted diagnostic requests and their outcomes. |
+
+Times begin at each trial's launch. Retain raw wall time, shared-service waiting,
+wall time excluding only that measured wait, and CLI execution time. Source-copy,
+transport, startup, diagnostics and each agent's own execution remain charged as
+appropriate; shared service waiting is not cluster scheduler time. Message receipt
+and submission completion are observable upper bounds, not the instant of a
+private reasoning step or edit.
+
+Native input and output token counters are recorded at completed-turn boundaries.
+For resumed sessions with cumulative counters, use validated deltas. Cached input
+is a subset of input and reasoning output a subset of output; do not add either
+twice. A within-turn localization can have a token interval rather than a precise
+count. Missing usage is unknown, not zero. Report cache distribution and avoid
+translating total-token differences into monetary savings without actual pricing.
+
+GPU cost is supervised worker lifetime multiplied by allocated devices, including
+initialization, diagnostics and failed requests. This measures allocated
+GPU-seconds, not utilization or pure training compute. Keep solver, qualification,
+instrumentation-preflight and private-grading costs in distinct ledgers. Record
+process termination and ownership so failed distributed workers cannot silently
+continue consuming resources.
+
+Preserve prompts, raw CLI events with receive times, every source submission,
+request/worker receipts, output manifests, environment versions and grading
+artifacts. Record actual TrainCheck activation, trace coverage, failures and
+produced records. Reading its manual or source is not instrumentation use; unused
+availability cannot establish the effect or overhead of active use. Large data,
+checkpoints and traces stay outside Git, with hashes and retrieval locations in
+the final report. Source and protocol progress use signed commits and a focused
+PR; no merge is part of this protocol.
+
+## Interpretation limits and prior result
+
+The [previous synthetic checkpoint pilot](checkpoint-continuation-costs-20261101.md)
+had four first-turn repairs that later passed independent validation. All four
+agents localized the ordering problem from source before requesting a GPU run;
+neither treatment activated TrainCheck. Supported localization took
+26.79–38.01 seconds, first valid submission took 99.39–111.12 seconds, and GPU
+cost before those submissions was zero. These are measured results for that
+particular case, not a general one-shot success rate for the paper's bugs.
+
+This new case has public historical provenance and may be recognized from model
+training data. A small, purposefully selected pairwise pilot cannot estimate the
+incidence of silent errors in agent-authored ML systems or establish a general
+TrainCheck advantage. It also does not evaluate the offline inference engine:
+agents may author their own checks around observed state. If agents solve the
+case from source, choose ordinary diagnostics, fail to use TrainCheck, or obtain
+no efficiency benefit, retain and report those outcomes without changing the
+task or success criteria after seeing them.
+
+## Interrupted transport and recovery protocol
+
+Both matched pairs used the frozen limits above. All four agents first proposed
+the correct clipping hypothesis from source and requested runtime diagnostics
+before editing the implementation. This case therefore must not be described
+as requiring runtime observation for successful localization.
+
+An SSH connection closed after control request 3 finished training but before
+artifact download. Treatment request 2 failed while creating its remote directory,
+before training launched. A bounded subsequent SSH check succeeded. The completed
+control phase has exit code 0, unchanged source/native binaries, empty worker/GPU
+process lists, and 17.31105645187199 seconds of two-GPU worker lifetime; it must
+be recovered rather than rerun. The treatment request directory was absent.
+
+Recovery preserved all original trial and error receipts. Pending results were
+retrieved, the unlaunched request executed once, and the same CLI thread IDs
+resumed with their original remaining request/turn/time budgets. New receipts
+are append-only under `recovery-1/`. The second treatment's request 1 also
+completed GPU work before an artifact-download failure; `recovery-2/` retrieved
+that existing result and continued its original thread. Composed records
+explicitly reference every original and continuation artifact. Report raw wall time including interruption and measured interruption
+intervals separately. Select one effective receipt per logical request so recovery
+does not double-count GPU cost. No private scoring or operator diagnosis is sent
+to solvers. Independent grading began only after all four trials froze and is
+now complete.
+
+## Completed solver and independent-validation results
+
+All four agents first localized the same clipping-selection cause from source,
+then wrote ordinary cross-rank diagnostics to test that hypothesis. No TrainCheck
+instrumentation activation was observed in either treatment. Both controls and both treatments
+submitted their first independently valid implementation in **turn 2**, after
+one diagnostic GPU request. All later submitted implementations also passed.
+
+Independent grading covered every frozen submission, deduplicated by all Python
+source bytes: eight distinct candidate sources and one known-correct reference,
+each executed in four scenarios, for **36 GPU phases**. Each scenario compared
+recorded model parameters, the complete exported optimizer state and recorded
+Torch/Megatron RNG at steps 0, 1 and 2 against its fresh reference. Step 0 is
+after the first backward pass. Model tolerances were `atol=0`, `rtol=0`;
+optimizer container types/order, scalars and tensor contents remained significant.
+Replicated parameters were also checked within their tensor-parallel group.
+
+| Private scenario | Seed | TP / allocated GPUs | Clipping threshold | Turn-1 sources | Turn-2 and later sources |
+| --- | ---: | ---: | ---: | --- | --- |
+| Base | 1234 | 2 / 2 | 1.0 | All four fail | All four pass |
+| Held-out seed | 5678 | 2 / 2 | 1.0 | All four fail | All four pass |
+| TP1 control | 1234 | 1 / 1 | 1.0 | All four pass | All four pass |
+| Clipping disabled | 1234 | 2 / 2 | 0.0 | All four pass | All four pass |
+
+Thus the earliest valid-submission milestone is justified by all four gates,
+not by the agent's own assertions or a successful training exit. These short
+checks do not establish long-horizon convergence, checkpoint-resume correctness,
+or coverage of other distributed configurations. No TP2×DP2 validation is claimed.
+
+### Milestones and resource costs
+
+All times are seconds from the trial's original start. In a `raw / adjusted`
+cell, adjusted time subtracts the measured union of resource-lock waits and
+operator suspension intervals. It is an accounting view, not a reconstruction
+of uninterrupted execution. Source diagnosis has no prior queue/interruption.
+“First valid” is the captured submission boundary, retrospectively graded;
+“finish” is the final visible decision, not when private grading completed.
+
+| Trial | Source diagnosis | Runtime diagnosis, raw / adjusted | First valid submission, raw / adjusted | Agent-visible repair verification, raw / adjusted | Finish, raw / adjusted |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Control 1 | 36.118 | 134.876 / 134.875 | 162.692 / 162.691 | 232.732 / 232.729 | 1,033.415 / 310.957 |
+| TrainCheck 1 | 18.708 | 201.057 / 163.998 | 286.436 / 249.376 | 1,018.755 / 316.404 | 1,121.005 / 418.653 |
+| TrainCheck 2 | 35.336 | 776.050 / 125.544 | 807.169 / 156.663 | 890.986 / 240.479 | 920.339 / 269.831 |
+| Control 2 | 16.462 | 127.342 / 127.341 | 387.317 / 387.316 | 451.859 / 451.857 | 539.296 / 539.293 |
+
+Agent-visible verification is the first completed inspection/check command that
+exposes both ranks' corrected-run coverage, separate from a later prose claim.
+Every corrected run actually contains all **16 updates on both ranks**, 52
+clipped gradients per rank, and zero errors for the scaling/replica fields that
+its agent chose to observe. Initial diagnostics used 3, 8, 4 and 4 updates in the
+table's order. Focused tests executed on both workers: respectively 5, 7, 4 and
+3 tests per worker. The public diagnostics have different coverage and are not
+the independent uniform oracle. Control 1, TrainCheck 1 and Control 2 also ran
+eight FP16 updates; TrainCheck 2 did not, and acknowledged that limitation.
+
+The full-trial boundary occurs slightly after the visible finish statement.
+Allocated GPU-seconds count devices × supervised worker lifetime, including
+startup; they do not measure device utilization.
+
+| Trial | Requests | Full raw wall s | Queue s | Operator suspension s | Full adjusted wall s | CLI s | GPU-s before first valid | Full solver GPU-s |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| Control 1 | 3 | 1,037.150 | 0.003 | 722.455 | 314.692 | 173.658 | 34.731 | 104.607 |
+| TrainCheck 1 | 3 | 1,124.788 | 37.062 | 665.290 | 422.436 | 271.649 | 35.239 | 94.831 |
+| TrainCheck 2 | 2 | 924.628 | 42.890 | 607.618 | 274.121 | 159.723 | 34.730 | 69.355 |
+| Control 2 | 3 | 543.570 | 0.002 | 0.000 | 543.568 | 397.481 | 34.720 | 94.298 |
+
+The four solver trials used **363.091 GPU-s total**. The separate 36-phase private
+grading used **1,069.616 GPU-s**; none is charged to solver milestones.
+Qualification, instrumentation preflight and earlier setup work remain separate
+operator costs in their own receipts; the two totals here are not a total
+campaign-cost claim. Recollecting grades from existing snapshots added no GPU
+training phases. Recovered completed requests are counted once through their
+effective receipt; their original failures and unknown original measurements
+remain preserved.
+
+### Native token accounting
+
+The following counts are native cumulative input-plus-output tokens, including
+cached input. Intervals are inclusive previous/current completed-turn bounds;
+no within-turn interpolation is used. First-valid and final completed-turn
+counts are exact as reported. All source diagnoses occur in turn 1 before a GPU
+request. Full per-category bounds are retained in the final JSON.
+
+| Trial | Source-diagnosis interval | Runtime-diagnosis interval | First-valid tokens | Final completed-turn tokens |
+| --- | ---: | ---: | ---: | ---: |
+| Control 1 | 0–393,071 | 393,071–693,107 | 693,107 | 1,236,382 |
+| TrainCheck 1 | 0–423,568 | 423,568–979,286 | 979,286 | 1,455,127 |
+| TrainCheck 2 | 0–334,917 | 334,917–590,405 | 590,405 | 805,536 |
+| Control 2 | 0–480,682 | 480,682–835,766 | 835,766 | 1,336,513 |
+
+| Trial | Final input | Of input, cached | Final output | Of output, reasoning |
+| --- | ---: | ---: | ---: | ---: |
+| Control 1 | 1,226,306 | 1,137,280 | 10,076 | 1,975 |
+| TrainCheck 1 | 1,440,861 | 1,350,784 | 14,266 | 4,260 |
+| TrainCheck 2 | 796,875 | 727,936 | 8,661 | 2,396 |
+| Control 2 | 1,327,625 | 1,243,776 | 8,888 | 1,431 |
+
+Cached input and reasoning output are subsets, not extra tokens. These counts
+are not a dollar-cost estimate. The final visible statement occurs within the
+last turn; its own within-turn token interval is preserved separately from the
+exact final completed-turn count.
+
+### TrainCheck usage and the negative finding
+
+TrainCheck 2 visibly read the generic manual. TrainCheck 1 only visibly listed
+its filename; a listing is not counted as reading. Neither treatment emitted
+observed TrainCheck trace files or added an explicit TrainCheck import in its
+changed submitted Python. Both used ordinary custom state diagnostics. The
+launcher option permitting TrainCheck access is not activation. No exhaustive
+runtime module/dynamic-import audit was collected, so the precise conclusion
+is **no instrumentation activation observed**, not proof that no incidental
+import could ever have occurred.
+
+This real distributed case still did not meet the intended difficulty criterion:
+all four agents identified the cause from source in 16–36 seconds, before runtime
+observations. Runtime evidence confirmed the consequence and supported the
+repair, but was not necessary for their observed localization. These results
+therefore do not establish a TrainCheck debugging-efficiency benefit or an
+overhead for active use. They describe optional availability that went unused.
+
+Two runs per arm on one selected case, unequal diagnostic lengths, one omitted
+FP16 rerun, normal agent variation and asymmetric transport interruptions prevent
+a causal speedup interpretation. Raw and adjusted values are both retained; no
+benefit ratio is claimed. This finding also does not show that agents can solve
+all TrainCheck paper cases in one shot. A subsequent case should first qualify a
+failure whose cause remains ambiguous after source inspection and whose rank
+state supplies the discriminating evidence, without hiding code or forcing use.
+The next candidate is documented in the [DDP runtime-state
+pilot](ddp-runtime-state-pilot-20261102.md).
+
+### Evidence, replay and collection failure
+
+Local operator evidence root:
+`/Users/matt/gitrepos/traincheck/outputs/ring18-runtime-20261102/`.
+The table below uses paths relative to that root. Private state snapshots remain
+under `/home/yuxuan/experiments/traincheck-runtime-20261102/private-grading/` on
+Ring18. Large snapshots, solver transcripts and datasets are not committed.
+
+| Evidence | SHA-256 |
+| --- | --- |
+| `solver-completion-composed.json` | `c7c654a72841b75f66b0ed5de0b2af7668940c77295e58a467369b4d203c8c79` |
+| `grading-plan.json` | `b2dd407117d09165b93d7596ef13ef13c4dc80055b6371b9f86ab3bff002e18c` |
+| `grading/result-20261101T211500631757Z.json` | `6a9810b43694cb8b13c06da6edc8feb7ab489c59fc6357ca4342d6d6864307a6` |
+| `milestone-annotations-final.json` | `fe1048a72fd9a77bef24f84f2ab110e2bb4bf6265792ac2fa021b65365dce509` |
+| `milestone-summary-final.json` | `a858a1003cd14c6e179efdf410723e25caa99c0ddb5eaa4af2b12d6c0dfbaeea` |
+| `private-grading-milestone-join.json` | `792219847ed15062d5b8b0b29090e52068c60fcc860d3a69df8077a9b6201c77` |
+
+The join records every turn's submission path, source ID, Python-manifest hash,
+optimizer-file hash and four scenario outcomes. Earliest valid submissions are:
+
+| Trial | Submission path relative to the evidence root | Python source ID / SHA-256 |
+| --- | --- | --- |
+| Control 1 | `trials/control_1/submissions/turn-2` | `4a6f8230a5967b40a93d1ee792427048d72405ece18abd5944d5b1caf3510b11` |
+| TrainCheck 1 | `trials/traincheck_1/submissions/turn-2` | `13db280c9f102dab5b50b8d663e4f6c4637bea6a5535bc811f16e2d350fbcc51` |
+| TrainCheck 2 | `recovery-2/trials/traincheck_2/submissions/turn-2` | `e51f85808dc26b77adc7f13eca38e2aa82f1b26db69114b9a12c4805f745487a` |
+| Control 2 | `trials/control_2/submissions/turn-2` | `d972171d51412890b226fab07f616ad1d2819a403e23fdc83a16b67a914faab6` |
+
+`milestone-annotations-visible.json` and `milestone-summary-visible.json` remain
+unchanged as the pre-grade analysis. Final artifacts are new files. Exact event
+lines and hashes are bound in the annotations; raw transcripts are under
+`trials/`, `recovery-1/trials/` and `recovery-2/trials/`. Recompute the final
+accounting without training or solver interaction using:
+
+```sh
+python summarize_milestones.py --study-root "$PWD" --annotations milestone-annotations-final.json --output milestone-summary-recomputed.json
+```
+
+The output path must be new. `join_private_grading.py` validates the frozen plan,
+all source/occurrence mappings, original phase receipts and all four scenario
+outcomes before producing final annotations; it never launches remote work.
+
+The initial collection produced only **unknown** outcomes because its caller
+passed a tuple to an evaluator requiring a nonempty list of steps. It is retained
+as `grading/result-20261101T211129986599Z.json`, SHA-256
+`2b56e92008716bf7a7bc6d6642d7cb67fab6478f5460628ae5fc5b70feedf296`.
+The collector's generated argument changed from `steps=(0,1,2)` to
+`steps=[0,1,2]`; the old helper is archived. A focused integration regression
+exercised the actual evaluator contract. Read-only recollection of the same
+snapshots produced the final result above: no evaluator contract, plan, solver
+feedback or GPU phase changed. Unknown evaluation was never treated as success.
+
+The accounting/extraction and private-grade-join helpers pass 42 focused local
+tests, explicit strict mypy and pinned formatting/lint checks. This validates
+the checked contracts, not absence of every possible harness defect. Independent
+state comparisons and retained failures are the basis of the reported outcomes.
diff --git a/tests/test_runtime_pilot_access.py b/tests/test_runtime_pilot_access.py
new file mode 100644
index 00000000..f5a465e7
--- /dev/null
+++ b/tests/test_runtime_pilot_access.py
@@ -0,0 +1,160 @@
+"""Exercise import availability in fresh processes without changing pytest state."""
+
+from __future__ import annotations
+
+import json
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+REPO = Path(__file__).resolve().parents[1]
+
+
+def worker(tmp_path: Path, statements: str) -> dict[str, object]:
+    package = tmp_path / "traincheck"
+    package.mkdir()
+    (package / "__init__.py").write_text("IMPORTED = True\n")
+    (package / "child.py").write_text("IMPORTED = True\n")
+    (tmp_path / "trainchecker.py").write_text("VALUE = 17\n")
+    source = (
+        "import importlib, json, sys\n"
+        "from types import ModuleType\n"
+        "sys.path.insert(0, sys.argv[1])\n"
+        "from tools.runtime_pilot.access import install_traincheck_blocker\n"
+        "sys.path.insert(0, sys.argv[2])\n" + statements
+    )
+    completed = subprocess.run(
+        [sys.executable, "-I", "-c", source, str(REPO), str(tmp_path)],
+        check=True,
+        capture_output=True,
+        text=True,
+        timeout=20,
+    )
+    raw: object = json.loads(completed.stdout)
+    assert isinstance(raw, dict)
+    result: dict[str, object] = {}
+    for name, value in raw.items():
+        assert isinstance(name, str)
+        result[name] = value
+    return result
+
+
+def test_installed_package_blocked_then_owned_finder_cleanup(tmp_path: Path) -> None:
+    result = worker(
+        tmp_path,
+        "original = tuple(sys.meta_path)\n"
+        "finder = install_traincheck_blocker()\n"
+        "try:\n"
+        "    importlib.import_module('traincheck')\n"
+        "except ModuleNotFoundError as error:\n"
+        "    missing = error.name\n"
+        "else:\n"
+        "    raise AssertionError('Installed TrainCheck was not blocked')\n"
+        "assert 'traincheck' not in sys.modules\n"
+        "sys.meta_path.remove(finder)\n"
+        "assert tuple(sys.meta_path) == original\n"
+        "package = importlib.import_module('traincheck')\n"
+        "print(json.dumps({'missing': missing, 'imported_after_cleanup': package.IMPORTED}))\n",
+    )
+    assert result == {"missing": "traincheck", "imported_after_cleanup": True}
+
+
+@pytest.mark.parametrize("name", ["traincheck.child", "traincheck.instrumentor.tracer"])
+def test_submodule_namespace_blocked(tmp_path: Path, name: str) -> None:
+    result = worker(
+        tmp_path,
+        "finder = install_traincheck_blocker()\n"
+        f"name = {name!r}\n"
+        "try:\n"
+        "    finder.find_spec(name, [])\n"
+        "except ModuleNotFoundError as error:\n"
+        "    assert error.name == name\n"
+        "else:\n"
+        "    raise AssertionError('Submodule was not blocked')\n"
+        "try:\n"
+        "    importlib.import_module(name)\n"
+        "except ModuleNotFoundError as error:\n"
+        "    assert error.name == 'traincheck'\n"
+        "else:\n"
+        "    raise AssertionError('Normal submodule import was not blocked')\n"
+        "sys.meta_path.remove(finder)\n"
+        "print(json.dumps({'blocked': name}))\n",
+    )
+    assert result == {"blocked": name}
+
+
+def test_unrelated_imports_and_similar_prefix_unchanged(tmp_path: Path) -> None:
+    result = worker(
+        tmp_path,
+        "finder = install_traincheck_blocker()\n"
+        "module = importlib.import_module('trainchecker')\n"
+        "decimal = importlib.import_module('decimal')\n"
+        "assert finder.find_spec('other.traincheck', None) is None\n"
+        "sys.meta_path.remove(finder)\n"
+        "print(json.dumps({'prefix': module.VALUE, 'decimal': str(decimal.Decimal('1.25'))}))\n",
+    )
+    assert result == {"prefix": 17, "decimal": "1.25"}
+
+
+@pytest.mark.parametrize("name", ["traincheck", "traincheck.child"])
+@pytest.mark.parametrize("placeholder", [False, True])
+def test_preloaded_namespace_rejected_without_removal(
+    tmp_path: Path, name: str, placeholder: bool
+) -> None:
+    result = worker(
+        tmp_path,
+        f"name = {name!r}\n"
+        f"existing = None if {placeholder!r} else ModuleType(name)\n"
+        "sys.modules[name] = existing\n"
+        "original = tuple(sys.meta_path)\n"
+        "try:\n"
+        "    install_traincheck_blocker()\n"
+        "except RuntimeError as error:\n"
+        "    assert name in str(error)\n"
+        "else:\n"
+        "    raise AssertionError('Preloaded namespace was accepted')\n"
+        "assert name in sys.modules and sys.modules[name] is existing\n"
+        "assert tuple(sys.meta_path) == original\n"
+        "print(json.dumps({'preserved': name}))\n",
+    )
+    assert result == {"preserved": name}
+
+
+def test_repeat_install_restores_precedence_without_duplicate(tmp_path: Path) -> None:
+    result = worker(
+        tmp_path,
+        "original = tuple(sys.meta_path)\n"
+        "finder = install_traincheck_blocker()\n"
+        "assert install_traincheck_blocker() is finder\n"
+        "sys.meta_path.remove(finder)\n"
+        "sys.meta_path.append(finder)\n"
+        "assert install_traincheck_blocker() is finder\n"
+        "assert sys.meta_path[0] is finder\n"
+        "assert sum(item is finder for item in sys.meta_path) == 1\n"
+        "sys.meta_path.remove(finder)\n"
+        "assert tuple(sys.meta_path) == original\n"
+        "print(json.dumps({'idempotent': True}))\n",
+    )
+    assert result == {"idempotent": True}
+
+
+def test_repeat_install_does_not_hide_later_namespace_injection(tmp_path: Path) -> None:
+    result = worker(
+        tmp_path,
+        "finder = install_traincheck_blocker()\n"
+        "existing = ModuleType('traincheck')\n"
+        "sys.modules['traincheck'] = existing\n"
+        "try:\n"
+        "    install_traincheck_blocker()\n"
+        "except RuntimeError:\n"
+        "    pass\n"
+        "else:\n"
+        "    raise AssertionError('Later namespace injection was hidden')\n"
+        "assert sys.modules['traincheck'] is existing\n"
+        "assert sum(item is finder for item in sys.meta_path) == 1\n"
+        "sys.meta_path.remove(finder)\n"
+        "print(json.dumps({'rejected': True}))\n",
+    )
+    assert result == {"rejected": True}
diff --git a/tests/test_runtime_pilot_clipping.py b/tests/test_runtime_pilot_clipping.py
new file mode 100644
index 00000000..3c24bd4e
--- /dev/null
+++ b/tests/test_runtime_pilot_clipping.py
@@ -0,0 +1,337 @@
+"""CPU-only contracts for the operator's private clipping qualification probe."""
+
+from __future__ import annotations
+
+import sys
+from dataclasses import dataclass
+from pathlib import Path
+from types import ModuleType
+
+import pytest
+import torch
+import torch.nn as nn
+
+from tools.runtime_pilot import clipping as audit
+
+
+class Model(nn.Module):
+    def __init__(self) -> None:
+        super().__init__()
+        self.rep_a = nn.Parameter(torch.ones(2, 2))
+        self.rep_b = nn.Parameter(torch.ones(2, 2))
+        self.shard = nn.Parameter(torch.ones(2, 2))
+        self.frozen = nn.Parameter(torch.ones(2, 2), requires_grad=False)
+        setattr(self.shard, "tensor_model_parallel", True)
+
+
+@dataclass
+class Optimizer:
+    bf16_groups: list[list[torch.Tensor]]
+    fp32_groups_gradients: list[list[torch.Tensor]]
+
+
+class MPU:
+    @staticmethod
+    def get_slice_parallel_rank() -> int:
+        return 1
+
+    @staticmethod
+    def get_model_parallel_rank() -> int:
+        # The native grid combines PP and TP here. Do not label this as TP.
+        return 5
+
+    @staticmethod
+    def get_data_parallel_rank() -> int:
+        return 0
+
+
+class Engine:
+    def __init__(self, native: ModuleType) -> None:
+        self.module = Model()
+        self.optimizer = Optimizer(
+            [
+                [
+                    self.module.rep_a,
+                    self.module.rep_b,
+                    self.module.shard,
+                    self.module.frozen,
+                ]
+            ],
+            [[torch.full((4,), 2.0) for _ in range(4)]],
+        )
+        self.global_steps = 0
+        self.native = native
+        self.mpu = MPU()
+        gradients = self.optimizer.fp32_groups_gradients[0]
+        self.clip_kwargs: dict[str, object] = {
+            "input_tensors": [gradients[0], gradients[2]],
+            "max_norm": 1.0,
+            "global_norm": 4.0,
+            "mpu": self.mpu,
+            "eps": 0.5,
+        }
+        self.step_calls: list[tuple[tuple[object, ...], dict[str, object]]] = []
+        self.clip_return = object()
+        self.step_return = object()
+        self.observed_clip_return: object = None
+
+    def _take_model_step(self, *args: object, **kwargs: object) -> object:
+        # This method stands in for an already installed state observer.
+        self.step_calls.append((args, kwargs))
+        clip: object = getattr(self.native, "clip_tensors_by_global_norm")
+        assert callable(clip)
+        self.observed_clip_return = clip(**self.clip_kwargs)
+        self.global_steps += 1
+        return self.step_return
+
+
+@dataclass
+class Harness:
+    engine: Engine
+    native: ModuleType
+    clip_calls: list[dict[str, object]]
+    output: Path
+
+    def snapshot(self, step: int = 1) -> dict[object, object]:
+        value: object = torch.load(
+            self.output / "rank1" / f"step{step}.pt", weights_only=True
+        )
+        assert isinstance(value, dict)
+        return value
+
+
+@pytest.fixture
+def harness(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> Harness:
+    native = ModuleType("deepspeed.runtime.bf16_optimizer")
+    engine = Engine(native)
+    clip_calls: list[dict[str, object]] = []
+
+    def native_clip(*args: object, **kwargs: object) -> object:
+        assert not args
+        clip_calls.append(kwargs)
+        tensors = kwargs["input_tensors"]
+        assert isinstance(tensors, list)
+        for tensor in tensors:
+            assert isinstance(tensor, torch.Tensor)
+            tensor.mul_(1.0 / 4.5)
+        return engine.clip_return
+
+    engine_module = ModuleType("deepspeed.runtime.engine")
+    setattr(engine_module, "DeepSpeedEngine", Engine)
+    setattr(native, "clip_tensors_by_global_norm", native_clip)
+    monkeypatch.setitem(sys.modules, native.__name__, native)
+    monkeypatch.setitem(sys.modules, engine_module.__name__, engine_module)
+    monkeypatch.setattr(audit, "_installed", False)
+    monkeypatch.setattr(torch.distributed, "get_rank", lambda: 1)
+    monkeypatch.setenv("RUNTIME_CLIPPING_OUTPUT", str(tmp_path))
+    # The fixture restores the class method after install replaces it.
+    monkeypatch.setattr(Engine, "_take_model_step", Engine._take_model_step)
+    return Harness(engine, native, clip_calls, tmp_path)
+
+
+def tensor_map(value: object) -> dict[str, torch.Tensor]:
+    assert isinstance(value, dict)
+    result: dict[str, torch.Tensor] = {}
+    for name, tensor in value.items():
+        assert isinstance(name, str)
+        assert isinstance(tensor, torch.Tensor)
+        result[name] = tensor
+    return result
+
+
+def test_native_call_once_and_correct_independent_snapshots(harness: Harness) -> None:
+    audit.install()
+    marker = object()
+    rng_before = torch.get_rng_state().clone()
+    parameters_before = {
+        name: parameter.detach().clone()
+        for name, parameter in harness.engine.module.named_parameters()
+    }
+    result = harness.engine._take_model_step(marker, future_option=marker)
+    assert torch.equal(torch.get_rng_state(), rng_before)
+    assert all(
+        torch.equal(parameter, parameters_before[name])
+        for name, parameter in harness.engine.module.named_parameters()
+    )
+    assert result is harness.engine.step_return
+    assert harness.engine.observed_clip_return is harness.engine.clip_return
+    assert len(harness.engine.step_calls) == len(harness.clip_calls) == 1
+    args, kwargs = harness.engine.step_calls[0]
+    assert len(args) == 1 and args[0] is marker
+    assert kwargs["future_option"] is marker
+    for name, value in harness.engine.clip_kwargs.items():
+        assert harness.clip_calls[0][name] is value
+    snapshot = harness.snapshot()
+    assert snapshot["rank"] == snapshot["tp_rank"] == snapshot["step"] == 1
+    assert snapshot["dp_rank"] == 0
+    assert snapshot["max_norm"] == 1.0
+    assert snapshot["global_norm"] == 4.0
+    assert snapshot["selected"] == {"rep_a": True, "rep_b": False}
+    before = tensor_map(snapshot["before"])
+    after = tensor_map(snapshot["after"])
+    assert list(before) == list(after) == ["rep_a", "rep_b"]
+    assert torch.equal(before["rep_a"], torch.full((4,), 2.0))
+    assert torch.equal(before["rep_b"], before["rep_a"])
+    assert torch.equal(after["rep_a"], before["rep_a"] * (1.0 / 4.5))
+    assert torch.equal(after["rep_b"], before["rep_b"])
+    for gradient in harness.engine.optimizer.fp32_groups_gradients[0]:
+        gradient.zero_()
+    persisted = tensor_map(harness.snapshot()["after"])
+    assert torch.equal(persisted["rep_a"], after["rep_a"])
+    assert audit._current_engine.get() is None
+
+
+def test_successive_steps_have_distinct_artifacts(harness: Harness) -> None:
+    audit.install()
+    harness.engine._take_model_step()
+    harness.engine._take_model_step()
+    assert harness.snapshot(1)["step"] == 1
+    assert harness.snapshot(2)["step"] == 2
+    assert len(harness.clip_calls) == 2
+
+
+def test_native_grid_tensor_slice_rank_is_used(harness: Harness) -> None:
+    assert not hasattr(harness.engine.mpu, "get_tensor_model_parallel_rank")
+    assert harness.engine.mpu.get_model_parallel_rank() != 1
+    audit.install()
+    harness.engine._take_model_step()
+    assert harness.snapshot()["tp_rank"] == 1
+
+
+def test_original_error_is_unchanged_and_context_restored(harness: Harness) -> None:
+    error = RuntimeError("native clipping failed")
+    calls: list[None] = []
+
+    def failing_clip(*args: object, **kwargs: object) -> object:
+        calls.append(None)
+        raise error
+
+    setattr(harness.native, "clip_tensors_by_global_norm", failing_clip)
+    audit.install()
+    outer = object()
+    token = audit._current_engine.set(outer)
+    try:
+        with pytest.raises(RuntimeError) as caught:
+            harness.engine._take_model_step()
+        assert caught.value is error
+        assert calls == [None]
+        assert audit._current_engine.get() is outer
+        assert not list(harness.output.rglob("*.pt"))
+    finally:
+        audit._current_engine.reset(token)
+
+
+@pytest.mark.parametrize("field", ["input_tensors", "max_norm", "global_norm", "mpu"])
+def test_missing_required_argument_is_not_defaulted(
+    harness: Harness, field: str
+) -> None:
+    del harness.engine.clip_kwargs[field]
+    audit.install()
+    with pytest.raises(TypeError, match="Missing required clipping argument"):
+        harness.engine._take_model_step()
+    assert not harness.clip_calls
+    assert audit._current_engine.get() is None
+
+
+@pytest.mark.parametrize("flag", [1, None, "false"])
+def test_malformed_partition_flag_rejected(harness: Harness, flag: object) -> None:
+    setattr(harness.engine.module.rep_a, "tensor_model_parallel", flag)
+    audit.install()
+    with pytest.raises(TypeError, match="tensor_model_parallel"):
+        harness.engine._take_model_step()
+    assert not harness.clip_calls
+
+
+def test_missing_replicated_gradient_rejected(harness: Harness) -> None:
+    harness.engine.optimizer.bf16_groups[0].pop(1)
+    harness.engine.optimizer.fp32_groups_gradients[0].pop(1)
+    audit.install()
+    with pytest.raises(ValueError, match="Missing replicated gradients"):
+        harness.engine._take_model_step()
+    assert not harness.clip_calls
+
+
+def test_missing_optimizer_field_rejected(harness: Harness) -> None:
+    delattr(harness.engine.optimizer, "fp32_groups_gradients")
+    audit.install()
+    with pytest.raises(AttributeError, match="fp32_groups_gradients"):
+        harness.engine._take_model_step()
+    assert not harness.clip_calls
+
+
+def test_mismatched_gradient_group_rejected(harness: Harness) -> None:
+    harness.engine.optimizer.fp32_groups_gradients[0].pop()
+    audit.install()
+    with pytest.raises(ValueError, match="lengths differ"):
+        harness.engine._take_model_step()
+    assert not harness.clip_calls
+
+
+@pytest.mark.parametrize("value", [True, None, float("inf"), float("nan"), 0.0])
+def test_malformed_norm_rejected(harness: Harness, value: object) -> None:
+    harness.engine.clip_kwargs["global_norm"] = value
+    audit.install()
+    with pytest.raises((TypeError, ValueError), match="global_norm"):
+        harness.engine._take_model_step()
+    assert not harness.clip_calls
+
+
+def test_duplicate_identity_rejected(harness: Harness) -> None:
+    optimizer = harness.engine.optimizer
+    optimizer.bf16_groups[0].append(optimizer.bf16_groups[0][0])
+    optimizer.fp32_groups_gradients[0].append(torch.ones(4))
+    audit.install()
+    with pytest.raises(ValueError, match="Duplicate optimizer"):
+        harness.engine._take_model_step()
+    assert not harness.clip_calls
+
+
+@pytest.mark.parametrize("dtype", [torch.float16, torch.float64])
+def test_non_fp32_gradient_rejected(harness: Harness, dtype: torch.dtype) -> None:
+    harness.engine.optimizer.fp32_groups_gradients[0][1] = torch.ones(4, dtype=dtype)
+    audit.install()
+    with pytest.raises(TypeError, match="FP32"):
+        harness.engine._take_model_step()
+    assert not harness.clip_calls
+
+
+def test_selected_view_is_not_confused_with_selected_object(harness: Harness) -> None:
+    gradient = harness.engine.optimizer.fp32_groups_gradients[0][0]
+    harness.engine.clip_kwargs["input_tensors"] = [gradient.view_as(gradient)]
+    audit.install()
+    with pytest.raises(ValueError, match="Selected gradient object"):
+        harness.engine._take_model_step()
+    assert not harness.clip_calls
+
+
+def test_empty_selection_records_omissions(harness: Harness) -> None:
+    harness.engine.clip_kwargs["input_tensors"] = []
+    audit.install()
+    harness.engine._take_model_step()
+    snapshot = harness.snapshot()
+    assert snapshot["selected"] == {"rep_a": False, "rep_b": False}
+    before = tensor_map(snapshot["before"])
+    after = tensor_map(snapshot["after"])
+    assert all(torch.equal(before[name], after[name]) for name in before)
+
+
+def test_duplicate_artifact_rejected_before_clipping(harness: Harness) -> None:
+    audit.install()
+    harness.engine._take_model_step()
+    original = (harness.output / "rank1" / "step1.pt").read_bytes()
+    harness.engine.global_steps = 0
+    with pytest.raises(FileExistsError):
+        harness.engine._take_model_step()
+    assert len(harness.clip_calls) == 1
+    assert (harness.output / "rank1" / "step1.pt").read_bytes() == original
+
+
+def test_outside_step_and_repeat_install_rejected(harness: Harness) -> None:
+    audit.install()
+    clip: object = getattr(harness.native, "clip_tensors_by_global_norm")
+    assert callable(clip)
+    with pytest.raises(RuntimeError, match="outside a model step"):
+        clip(**harness.engine.clip_kwargs)
+    with pytest.raises(RuntimeError, match="already installed"):
+        audit.install()
+    assert not harness.clip_calls
diff --git a/tests/test_runtime_pilot_evaluator.py b/tests/test_runtime_pilot_evaluator.py
new file mode 100644
index 00000000..2919913c
--- /dev/null
+++ b/tests/test_runtime_pilot_evaluator.py
@@ -0,0 +1,483 @@
+"""Operator-only runtime evidence tests; do not export to solver packets."""
+
+from __future__ import annotations
+
+import shutil
+from collections import OrderedDict
+from collections.abc import Callable
+from pathlib import Path
+
+import pytest
+import torch
+
+from tools.runtime_pilot.evaluate import Evaluation, evaluate
+
+
+def artifact(rank: int, step: int, tp_size: int = 2) -> dict[str, object]:
+    return {
+        "rank": rank,
+        "tp_rank": rank % tp_size,
+        "dp_rank": rank // tp_size,
+        "step": step,
+        "parameters": {
+            "norm": torch.tensor([1.0, 2.0]) + step,
+            "partition": torch.tensor([float(rank), 3.0]) + step,
+        },
+        "replicated_names": ["norm"],
+        "optimizer": {
+            "state": {0: {"momentum": torch.tensor([rank + 0.25, 2.0])}},
+            "groups": [{"lr": 0.125, "params": (0,)}],
+            "active": True,
+            "optional": None,
+        },
+        "rng": {"cpu": torch.tensor([1, rank + 3, step], dtype=torch.uint8)},
+    }
+
+
+def save(root: Path, rank: int, step: int, state: dict[str, object]) -> Path:
+    path = root / f"rank{rank}" / f"step{step}.pt"
+    path.parent.mkdir(parents=True, exist_ok=True)
+    torch.save(state, path)
+    return path
+
+
+def runs(
+    root: Path, world_size: int = 2, tp_size: int = 2, steps: tuple[int, ...] = (0, 1)
+) -> tuple[Path, Path]:
+    candidate, reference = root / "candidate", root / "reference"
+    for directory in (candidate, reference):
+        for rank in range(world_size):
+            for step in steps:
+                save(directory, rank, step, artifact(rank, step, tp_size))
+    return candidate, reference
+
+
+def change(
+    root: Path,
+    update: Callable[[dict[str, object]], None],
+    rank: int = 0,
+    step: int = 1,
+) -> None:
+    path = root / f"rank{rank}" / f"step{step}.pt"
+    raw: object = torch.load(path, map_location="cpu", weights_only=True)
+    assert isinstance(raw, dict)
+    state: dict[str, object] = {}
+    for key, value in raw.items():
+        assert isinstance(key, str)
+        state[key] = value
+    update(state)
+    torch.save(state, path)
+
+
+def change_field(
+    root: Path, name: str, value: object, rank: int = 0, step: int = 1
+) -> None:
+    def update(state: dict[str, object]) -> None:
+        state[name] = value
+
+    change(root, update, rank, step)
+
+
+def check(
+    paths: tuple[Path, Path],
+    world_size: int = 2,
+    steps: list[int] | None = None,
+    atol: float = 0.0,
+    rtol: float = 0.0,
+) -> Evaluation:
+    return evaluate(*paths, world_size, [0, 1] if steps is None else steps, atol, rtol)
+
+
+def test_complete_partitioned_state_passes(tmp_path: Path) -> None:
+    report = check(runs(tmp_path))
+    assert report.passed
+    assert report.candidate_artifacts == report.reference_artifacts == 4
+    assert report.tensor_parallel_size == 2 and report.data_parallel_size == 1
+    assert report.replicated_status == "passed" and report.replicated_checks > 0
+    assert report.model_comparable_tensors == 8
+    assert report.maximum_absolute_model_difference == 0
+    assert report.failed_checks == 0 and report.failed_paths == ()
+
+
+def test_partitioned_parameter_is_still_compared_to_reference(tmp_path: Path) -> None:
+    paths = runs(tmp_path)
+    change_field(
+        paths[0],
+        "parameters",
+        {"norm": torch.tensor([2.0, 3.0]), "partition": torch.tensor([9.0, 4.0])},
+        rank=1,
+    )
+    report = check(paths)
+    assert not report.passed
+    assert report.replicated_status == "passed"
+    assert report.maximum_absolute_model_difference == 7.0
+    assert any("partition" in path for path in report.failed_paths)
+
+
+def test_replicated_parameter_failure(tmp_path: Path) -> None:
+    paths = runs(tmp_path)
+    change_field(
+        paths[0],
+        "parameters",
+        {"norm": torch.tensor([2.5, 3.0]), "partition": torch.tensor([2.0, 4.0])},
+        rank=1,
+    )
+    report = check(paths)
+    assert not report.passed and report.replicated_status == "violated"
+    assert report.maximum_absolute_model_difference == 0.5
+
+
+@pytest.mark.parametrize(
+    "field,value",
+    [
+        (
+            "optimizer",
+            {
+                "state": {0: {"momentum": torch.tensor([9.0, 2.0])}},
+                "groups": [{"lr": 0.125, "params": (0,)}],
+                "active": True,
+                "optional": None,
+            },
+        ),
+        ("rng", {"cpu": torch.tensor([1, 99, 1], dtype=torch.uint8)}),
+    ],
+)
+def test_nonmodel_state_failure(tmp_path: Path, field: str, value: object) -> None:
+    paths = runs(tmp_path)
+    change_field(paths[0], field, value)
+    report = check(paths, atol=100.0, rtol=100.0)
+    assert not report.passed
+    assert report.maximum_absolute_model_difference == 0
+    assert report.replicated_status == "passed"
+    assert all(field in path for path in report.failed_paths)
+
+
+@pytest.mark.parametrize("remove_rank", [False, True])
+def test_missing_rank_or_step_is_error(tmp_path: Path, remove_rank: bool) -> None:
+    paths = runs(tmp_path)
+    if remove_rank:
+        shutil.rmtree(paths[0] / "rank1")
+    else:
+        (paths[0] / "rank1/step1.pt").unlink()
+    with pytest.raises(ValueError, match="coverage"):
+        check(paths)
+
+
+def test_duplicate_rank_step_is_error(tmp_path: Path) -> None:
+    paths = runs(tmp_path)
+    shutil.copyfile(paths[0] / "rank0/step1.pt", paths[0] / "rank0/step01.pt")
+    with pytest.raises(ValueError, match="duplicate rank/step"):
+        check(paths)
+
+
+def test_filename_identity_must_match(tmp_path: Path) -> None:
+    paths = runs(tmp_path)
+    change_field(paths[0], "rank", 1)
+    with pytest.raises(ValueError, match="path disagrees"):
+        check(paths)
+
+
+@pytest.mark.parametrize(
+    "field,value,error",
+    [
+        ("rank", True, TypeError),
+        ("step", "1", TypeError),
+        ("tp_rank", -1, ValueError),
+        ("dp_rank", 0.0, TypeError),
+        ("parameters", {}, ValueError),
+        ("parameters", {"norm": [1, 2]}, TypeError),
+        ("parameters", {"norm": torch.tensor([float("nan")])}, ValueError),
+        ("parameters", {"norm": torch.tensor([float("inf")])}, ValueError),
+        (
+            "parameters",
+            {
+                "norm": torch.sparse_coo_tensor(
+                    torch.tensor([[0]]), torch.tensor([1.0]), (2,)
+                )
+            },
+            TypeError,
+        ),
+        ("replicated_names", "norm", TypeError),
+        ("replicated_names", ["missing"], ValueError),
+        ("replicated_names", ["norm", "norm"], ValueError),
+        ("optimizer", {True: 1}, TypeError),
+        ("optimizer", {"x": float("nan")}, ValueError),
+        ("optimizer", OrderedDict([("x", 1)]), TypeError),
+        ("rng", {}, ValueError),
+        ("rng", {"cpu": torch.tensor([1.0])}, ValueError),
+        ("rng", {"cpu": torch.zeros((1, 2), dtype=torch.uint8)}, ValueError),
+        ("rng", {"cpu": torch.empty(0, dtype=torch.uint8)}, ValueError),
+    ],
+)
+def test_malformed_state_rejected(
+    tmp_path: Path, field: str, value: object, error: type[Exception]
+) -> None:
+    paths = runs(tmp_path)
+    change_field(paths[0], field, value)
+    with pytest.raises(error):
+        check(paths)
+
+
+@pytest.mark.parametrize("extra", [False, True])
+def test_exact_required_fields(tmp_path: Path, extra: bool) -> None:
+    paths = runs(tmp_path)
+
+    def update(state: dict[str, object]) -> None:
+        if extra:
+            state["unexpected"] = 1
+        else:
+            del state["optimizer"]
+
+    change(paths[0], update)
+    with pytest.raises(ValueError, match="snapshot fields differ"):
+        check(paths)
+
+
+@pytest.mark.parametrize(
+    "tensor",
+    [torch.tensor([2.0, 3.0], dtype=torch.float64), torch.tensor([[2.0, 3.0]])],
+)
+def test_shape_dtype_failure_is_not_value_comparison(
+    tmp_path: Path, tensor: torch.Tensor
+) -> None:
+    paths = runs(tmp_path)
+    change_field(
+        paths[0], "parameters", {"norm": tensor, "partition": torch.tensor([1.0, 4.0])}
+    )
+    report = check(paths)
+    assert not report.passed
+    assert report.model_comparable_tensors == 7
+    assert any("dtype_shape" in path for path in report.failed_paths)
+
+
+def test_parameter_tolerances_do_not_change_exact_state_checks(tmp_path: Path) -> None:
+    paths = runs(tmp_path)
+    change_field(
+        paths[0],
+        "parameters",
+        {"norm": torch.tensor([2.001, 3.0]), "partition": torch.tensor([1.0, 4.0])},
+    )
+    assert not check(paths).passed
+    tolerant = check(paths, atol=0.002)
+    assert tolerant.passed and tolerant.replicated_status == "passed"
+    assert tolerant.maximum_absolute_model_difference is not None
+    assert tolerant.maximum_absolute_model_difference > 0
+
+
+def test_empty_replicated_coverage_is_error(tmp_path: Path) -> None:
+    paths = runs(tmp_path)
+    for directory in paths:
+        for rank in range(2):
+            for step in (0, 1):
+                change_field(directory, "replicated_names", [], rank, step)
+    with pytest.raises(ValueError, match="Empty replicated"):
+        check(paths)
+
+
+def test_tp1_explicitly_not_applicable_with_multiple_dp_ranks(tmp_path: Path) -> None:
+    paths = runs(tmp_path, world_size=2, tp_size=1)
+    for directory in paths:
+        for rank in range(2):
+            for step in (0, 1):
+                change_field(directory, "replicated_names", [], rank, step)
+    report = check(paths)
+    assert report.passed and report.tensor_parallel_size == 1
+    assert report.data_parallel_size == 2
+    assert (
+        report.replicated_status == "not_applicable" and report.replicated_checks == 0
+    )
+
+
+def test_replication_compares_only_within_same_dp_group(tmp_path: Path) -> None:
+    paths = runs(tmp_path, world_size=4, tp_size=2)
+    for directory in paths:
+        for rank in (2, 3):
+            for step in (0, 1):
+                change_field(
+                    directory,
+                    "parameters",
+                    {
+                        "norm": torch.tensor([10.0, 20.0]),
+                        "partition": torch.tensor([float(rank), float(step)]),
+                    },
+                    rank,
+                    step,
+                )
+    assert check(paths, world_size=4).passed
+
+
+def test_reference_must_satisfy_replication(tmp_path: Path) -> None:
+    paths = runs(tmp_path)
+    change_field(
+        paths[1],
+        "parameters",
+        {"norm": torch.tensor([99.0, 3.0]), "partition": torch.tensor([2.0, 4.0])},
+        rank=1,
+    )
+    with pytest.raises(ValueError, match="reference violates replication"):
+        check(paths)
+
+
+@pytest.mark.parametrize(
+    "optimizer",
+    [
+        {
+            "active": True,
+            "groups": [{"lr": 0.125, "params": (0,)}],
+            "state": {0: {"momentum": torch.tensor([0.25, 2.0])}},
+            "optional": None,
+        },
+        {
+            "state": {0: {"momentum": torch.tensor([0.25, 2.0])}},
+            "groups": [{"lr": 0.125, "params": [0]}],
+            "active": True,
+            "optional": None,
+        },
+        {
+            "state": {"0": {"momentum": torch.tensor([0.25, 2.0])}},
+            "groups": [{"lr": 0.125, "params": (0,)}],
+            "active": True,
+            "optional": None,
+        },
+        {
+            "state": {0: {"momentum": torch.tensor([0.25, 2.0])}},
+            "groups": [{"lr": 0.125, "params": (0,)}],
+            "active": 1,
+            "optional": None,
+        },
+    ],
+)
+def test_optimizer_order_key_and_value_types_are_significant(
+    tmp_path: Path, optimizer: object
+) -> None:
+    paths = runs(tmp_path)
+    change_field(paths[0], "optimizer", optimizer)
+    report = check(paths)
+    assert not report.passed
+    assert all("optimizer" in path for path in report.failed_paths)
+
+
+def test_optimizer_signed_zero_tensor_and_scalar_are_exact(tmp_path: Path) -> None:
+    paths = runs(tmp_path)
+    for directory in paths:
+        change_field(directory, "optimizer", [0.0, torch.tensor([0.0])])
+    change_field(paths[0], "optimizer", [-0.0, torch.tensor([-0.0])])
+    report = check(paths, atol=1.0)
+    assert not report.passed and report.failed_checks == 2
+
+
+def test_cyclic_optimizer_rejected(tmp_path: Path) -> None:
+    paths = runs(tmp_path)
+    cycle: list[object] = []
+    cycle.append(cycle)
+    change_field(paths[0], "optimizer", cycle)
+    with pytest.raises(ValueError, match="cyclic"):
+        check(paths)
+
+
+def test_model_integer_difference_is_not_rounded_to_zero(tmp_path: Path) -> None:
+    paths = runs(tmp_path)
+    for directory in paths:
+        change_field(
+            directory,
+            "parameters",
+            {
+                "norm": torch.tensor([2.0, 3.0]),
+                "partition": torch.tensor([2**62], dtype=torch.int64),
+            },
+        )
+    change_field(
+        paths[0],
+        "parameters",
+        {
+            "norm": torch.tensor([2.0, 3.0]),
+            "partition": torch.tensor([2**62 + 1], dtype=torch.int64),
+        },
+    )
+    report = check(paths, atol=10.0)
+    assert not report.passed and report.maximum_absolute_model_difference == 1.0
+
+
+def test_total_failures_are_not_truncated(tmp_path: Path) -> None:
+    paths = runs(tmp_path, steps=(0,))
+    for rank in range(2):
+        change_field(paths[0], "optimizer", [1] * 120, rank, 0)
+        change_field(paths[1], "optimizer", [0] * 120, rank, 0)
+    report = check(paths, steps=[0])
+    assert report.failed_checks == 240
+    assert len(report.failed_paths) == 100 and report.failed_paths_truncated
+
+
+@pytest.mark.parametrize("steps", [[], [1, 0], [0, 0], [True], [-1]])
+def test_invalid_expected_steps(tmp_path: Path, steps: list[int]) -> None:
+    paths = runs(tmp_path)
+    with pytest.raises((ValueError, TypeError)):
+        check(paths, steps=steps)
+
+
+def test_unexpected_step_and_symlink_are_rejected(tmp_path: Path) -> None:
+    paths = runs(tmp_path)
+    path = save(paths[0], 0, 9, artifact(0, 9))
+    with pytest.raises(ValueError, match="coverage"):
+        check(paths)
+    path.unlink()
+    path = paths[0] / "rank0/step1.pt"
+    path.unlink()
+    path.symlink_to(paths[1] / "rank0/step1.pt")
+    with pytest.raises(ValueError, match="nonsymlink"):
+        check(paths)
+
+
+def test_topology_changes_and_duplicate_coordinates_are_errors(tmp_path: Path) -> None:
+    paths = runs(tmp_path)
+    change_field(paths[0], "tp_rank", 0, rank=1, step=0)
+    with pytest.raises(ValueError, match="topology"):
+        check(paths)
+    change_field(paths[0], "tp_rank", 1, rank=1, step=0)
+    change_field(paths[0], "dp_rank", 1, rank=1, step=1)
+    with pytest.raises(ValueError, match="topology"):
+        check(paths)
+
+
+def test_extreme_topology_rejected_before_constructing_rank_grid(
+    tmp_path: Path,
+) -> None:
+    paths = runs(tmp_path)
+    change_field(paths[0], "tp_rank", 10**20, rank=1, step=0)
+    with pytest.raises(ValueError, match="topology"):
+        check(paths)
+
+
+def test_empty_replicated_tensors_are_not_exercised_checks(tmp_path: Path) -> None:
+    paths = runs(tmp_path)
+    for directory in paths:
+        for rank in range(2):
+            for step in (0, 1):
+                change_field(
+                    directory,
+                    "parameters",
+                    {"norm": torch.empty(0), "partition": torch.tensor([1.0])},
+                    rank,
+                    step,
+                )
+    with pytest.raises(ValueError, match="Empty replicated tensor"):
+        check(paths)
+
+
+def test_finite_values_with_unrepresentable_difference_are_explicit(
+    tmp_path: Path,
+) -> None:
+    paths = runs(tmp_path)
+    for directory, value in zip(paths, (1.7e308, -1.7e308)):
+        change_field(
+            directory,
+            "parameters",
+            {
+                "norm": torch.tensor([2.0, 3.0]),
+                "partition": torch.tensor([value], dtype=torch.float64),
+            },
+        )
+    report = check(paths)
+    assert not report.passed
+    assert report.model_absolute_difference_overflow
+    assert report.maximum_absolute_model_difference is None
diff --git a/tools/runtime_pilot/__init__.py b/tools/runtime_pilot/__init__.py
new file mode 100644
index 00000000..1bc76f00
--- /dev/null
+++ b/tools/runtime_pilot/__init__.py
@@ -0,0 +1 @@
+"""Operator-owned real distributed-runtime experiment utilities."""
diff --git a/tools/runtime_pilot/access.py b/tools/runtime_pilot/access.py
new file mode 100644
index 00000000..90e9115b
--- /dev/null
+++ b/tools/runtime_pilot/access.py
@@ -0,0 +1,64 @@
+"""Process-local TrainCheck import boundary for a control worker.
+
+Install before dependency bootstrap, while the worker is still single-threaded.
+This finder blocks normal imports, including an otherwise installed TrainCheck;
+it is not an OS sandbox against code that edits import machinery, injects
+``sys.modules`` entries, or executes source through a custom loader.
+"""
+
+from __future__ import annotations
+
+import sys
+from collections.abc import Sequence
+from importlib.abc import MetaPathFinder
+from importlib.machinery import ModuleSpec
+from types import ModuleType
+
+
+def _is_traincheck(fullname: str) -> bool:
+    return fullname == "traincheck" or fullname.startswith("traincheck.")
+
+
+class TrainCheckImportBlocker(MetaPathFinder):
+    """Reject only TrainCheck's exact package namespace through normal imports."""
+
+    def find_spec(
+        self,
+        fullname: str,
+        path: Sequence[str] | None,
+        target: ModuleType | None = None,
+    ) -> ModuleSpec | None:
+        if _is_traincheck(fullname):
+            raise ModuleNotFoundError(
+                f"TrainCheck is unavailable in this control worker: {fullname}",
+                name=fullname,
+            )
+        return None
+
+
+def install_traincheck_blocker() -> TrainCheckImportBlocker:
+    """Install first in the import search order, or reuse the owned finder.
+
+    Existing TrainCheck namespace entries, even ``None`` placeholders, are an
+    error: removing them would conceal earlier imports and their side effects.
+    Reinstallation validates that invariant again and restores the same finder
+    to first position if another component has moved it.
+    """
+    loaded = sorted(name for name in tuple(sys.modules) if _is_traincheck(name))
+    if loaded:
+        raise RuntimeError(
+            "Cannot install TrainCheck import blocker after namespace entries "
+            f"already exist: {', '.join(loaded)}"
+        )
+    for index, finder in enumerate(sys.meta_path):
+        # Only reuse this exact implementation, not a subclass with different
+        # behavior. Do not remove or replace unrelated import finders.
+        if type(finder) is TrainCheckImportBlocker:
+            assert isinstance(finder, TrainCheckImportBlocker)
+            if index:
+                del sys.meta_path[index]
+                sys.meta_path.insert(0, finder)
+            return finder
+    blocker = TrainCheckImportBlocker()
+    sys.meta_path.insert(0, blocker)
+    return blocker
diff --git a/tools/runtime_pilot/clipping.py b/tools/runtime_pilot/clipping.py
new file mode 100644
index 00000000..bc1409ae
--- /dev/null
+++ b/tools/runtime_pilot/clipping.py
@@ -0,0 +1,236 @@
+"""Private synchronous qualification of native BF16 gradient clipping.
+
+Install after audit_runtime so its model-step observer stays in the call chain.
+This probe is intentionally limited to the native keyword-based clipping call;
+it is not part of either solver arm or its performance measurements.
+"""
+
+from __future__ import annotations
+
+import math
+import os
+from contextvars import ContextVar
+from importlib import import_module
+from pathlib import Path
+from typing import TypedDict
+
+import torch
+import torch.nn as nn
+
+
+class ClippingSnapshot(TypedDict):
+    rank: int
+    tp_rank: int
+    dp_rank: int
+    step: int
+    max_norm: int | float
+    global_norm: int | float
+    selected: dict[str, bool]
+    before: dict[str, torch.Tensor]
+    after: dict[str, torch.Tensor]
+
+
+_current_engine: ContextVar[object | None] = ContextVar(
+    "clipping_audit_engine", default=None
+)
+_installed = False
+
+
+def _nonnegative_integer(value: object, field: str) -> int:
+    if not isinstance(value, int) or isinstance(value, bool):
+        raise TypeError(f"{field} must be an integer")
+    if value < 0:
+        raise ValueError(f"{field} must be nonnegative")
+    return value
+
+
+def _positive_number(value: object, field: str) -> int | float:
+    if type(value) not in (int, float):
+        raise TypeError(f"{field} must be an int or float")
+    assert isinstance(value, (int, float))
+    if not math.isfinite(value) or value <= 0:
+        raise ValueError(f"{field} must be finite and positive")
+    return value
+
+
+def _tensor(value: object, field: str) -> torch.Tensor:
+    if not isinstance(value, torch.Tensor):
+        raise TypeError(f"{field} must be a tensor")
+    if value.layout != torch.strided or value.dtype != torch.float32:
+        raise TypeError(f"{field} must be a dense FP32 gradient")
+    if value.numel() == 0 or not bool(torch.isfinite(value).all()):
+        raise ValueError(f"{field} must be nonempty and finite")
+    return value
+
+
+def _groups(value: object, field: str) -> list[list[torch.Tensor]]:
+    if not isinstance(value, (list, tuple)):
+        raise TypeError(f"{field} must contain tensor groups")
+    groups: list[list[torch.Tensor]] = []
+    for index, group in enumerate(value):
+        if not isinstance(group, (list, tuple)):
+            raise TypeError(f"{field}[{index}] must be a tensor group")
+        tensors: list[torch.Tensor] = []
+        for tensor in group:
+            if not isinstance(tensor, torch.Tensor):
+                raise TypeError(f"{field}[{index}] contains a non-tensor")
+            tensors.append(tensor)
+        groups.append(tensors)
+    return groups
+
+
+def _replicated_gradients(
+    engine: object,
+) -> tuple[dict[str, torch.Tensor], set[int]]:
+    module: object = getattr(engine, "module")
+    optimizer: object = getattr(engine, "optimizer")
+    if not isinstance(module, nn.Module):
+        raise TypeError("Engine module must be a torch.nn.Module")
+    names: dict[int, str] = {}
+    required: set[str] = set()
+    for name, parameter in module.named_parameters():
+        if not isinstance(name, str) or not name:
+            raise TypeError("Parameter names must be nonempty strings")
+        if id(parameter) in names or name in names.values():
+            raise ValueError("Duplicate named parameter identity or name")
+        names[id(parameter)] = name
+        # Megatron explicitly treats an absent flag as nonpartitioned.
+        partitioned: object = getattr(parameter, "tensor_model_parallel", False)
+        if not isinstance(partitioned, bool):
+            raise TypeError(f"{name}.tensor_model_parallel must be a bool")
+        if parameter.requires_grad and not partitioned:
+            required.add(name)
+    if not required:
+        raise ValueError("No replicated trainable parameters were observed")
+
+    low_groups = _groups(getattr(optimizer, "bf16_groups"), "bf16_groups")
+    grad_groups = _groups(
+        getattr(optimizer, "fp32_groups_gradients"), "fp32_groups_gradients"
+    )
+    if len(low_groups) != len(grad_groups):
+        raise ValueError("BF16 parameter and FP32 gradient group counts differ")
+    gradients: dict[str, torch.Tensor] = {}
+    parameter_ids: set[int] = set()
+    gradient_ids: set[int] = set()
+    for group_index, (parameters, group_gradients) in enumerate(
+        zip(low_groups, grad_groups)
+    ):
+        if len(parameters) != len(group_gradients):
+            raise ValueError(
+                f"Parameter/gradient lengths differ in group {group_index}"
+            )
+        for low_parameter, gradient in zip(parameters, group_gradients):
+            parameter_id = id(low_parameter)
+            if parameter_id not in names:
+                raise ValueError("Optimizer parameter is absent from named_parameters")
+            if parameter_id in parameter_ids or id(gradient) in gradient_ids:
+                raise ValueError("Duplicate optimizer parameter or gradient identity")
+            parameter_ids.add(parameter_id)
+            gradient_ids.add(id(gradient))
+            name = names[parameter_id]
+            gradient = _tensor(gradient, f"{name}.gradient")
+            # Native FP32 gradient views are flat, even for matrix parameters.
+            if low_parameter.numel() != gradient.numel():
+                raise ValueError(f"Parameter/gradient element counts differ for {name}")
+            if name in required:
+                gradients[name] = gradient
+    missing = required.difference(gradients)
+    if missing:
+        raise ValueError(f"Missing replicated gradients: {sorted(missing)}")
+    return gradients, gradient_ids
+
+
+def _mpu_rank(mpu: object, method_name: str) -> int:
+    method: object = getattr(mpu, method_name)
+    if not callable(method):
+        raise TypeError(f"mpu.{method_name} must be callable")
+    value: object = method()
+    return _nonnegative_integer(value, method_name)
+
+
+def _snapshot(tensors: dict[str, torch.Tensor]) -> dict[str, torch.Tensor]:
+    return {
+        name: _tensor(tensor, f"{name}.gradient").detach().cpu().clone()
+        for name, tensor in tensors.items()
+    }
+
+
+def install() -> None:
+    """Wrap the current native hooks once, without replacing prior observers."""
+    global _installed
+    if _installed:
+        raise RuntimeError("Clipping audit is already installed")
+    optimizer_module = import_module("deepspeed.runtime.bf16_optimizer")
+    engine_module = import_module("deepspeed.runtime.engine")
+    engine_class: object = getattr(engine_module, "DeepSpeedEngine")
+    if not isinstance(engine_class, type):
+        raise TypeError("DeepSpeedEngine must be a class")
+    original_clip: object = getattr(optimizer_module, "clip_tensors_by_global_norm")
+    original_step: object = getattr(engine_class, "_take_model_step")
+    if not callable(original_clip) or not callable(original_step):
+        raise TypeError("Native clipping and model-step hooks must be callable")
+
+    def observed_clip(*args: object, **kwargs: object) -> object:
+        if args:
+            raise TypeError("Clipping qualification requires native keyword arguments")
+        for field in ("input_tensors", "max_norm", "global_norm", "mpu"):
+            if field not in kwargs:
+                raise TypeError(f"Missing required clipping argument: {field}")
+        engine = _current_engine.get()
+        if engine is None:
+            raise RuntimeError("Clipping was called outside a model step")
+        input_tensors = kwargs["input_tensors"]
+        if not isinstance(input_tensors, (list, tuple)):
+            raise TypeError("input_tensors must be a list or tuple of tensors")
+        selected_ids: set[int] = set()
+        for tensor in input_tensors:
+            tensor = _tensor(tensor, "input_tensors entry")
+            if id(tensor) in selected_ids:
+                raise ValueError("input_tensors contains a duplicate gradient object")
+            selected_ids.add(id(tensor))
+        gradients, known_gradient_ids = _replicated_gradients(engine)
+        if not selected_ids.issubset(known_gradient_ids):
+            raise ValueError("Selected gradient object is absent from optimizer state")
+        step = _nonnegative_integer(getattr(engine, "global_steps"), "global_steps")
+        rank = _nonnegative_integer(torch.distributed.get_rank(), "rank")
+        output = os.environ["RUNTIME_CLIPPING_OUTPUT"]
+        if not output:
+            raise ValueError("RUNTIME_CLIPPING_OUTPUT must not be empty")
+        destination = Path(output) / f"rank{rank}" / f"step{step + 1}.pt"
+        if destination.exists():
+            raise FileExistsError(destination)
+        snapshot: ClippingSnapshot = {
+            "rank": rank,
+            # PipelineParallelGrid separates tensor slicing from its combined
+            # pipeline/model rank. The latter is not a TP rank when PP > 1.
+            "tp_rank": _mpu_rank(kwargs["mpu"], "get_slice_parallel_rank"),
+            "dp_rank": _mpu_rank(kwargs["mpu"], "get_data_parallel_rank"),
+            "step": step + 1,
+            "max_norm": _positive_number(kwargs["max_norm"], "max_norm"),
+            "global_norm": _positive_number(kwargs["global_norm"], "global_norm"),
+            "selected": {
+                name: id(gradient) in selected_ids
+                for name, gradient in gradients.items()
+            },
+            "before": _snapshot(gradients),
+            "after": {},
+        }
+        result: object = original_clip(*args, **kwargs)
+        snapshot["after"] = _snapshot(gradients)
+        destination.parent.mkdir(parents=True, exist_ok=True)
+        # Exclusive creation refuses duplicate evidence rather than overwriting it.
+        with destination.open("xb") as stream:
+            torch.save(snapshot, stream)
+        return result
+
+    def observed_step(engine: object, *args: object, **kwargs: object) -> object:
+        token = _current_engine.set(engine)
+        try:
+            result: object = original_step(engine, *args, **kwargs)
+            return result
+        finally:
+            _current_engine.reset(token)
+
+    setattr(optimizer_module, "clip_tensors_by_global_norm", observed_clip)
+    setattr(engine_class, "_take_model_step", observed_step)
+    _installed = True
diff --git a/tools/runtime_pilot/evaluate.py b/tools/runtime_pilot/evaluate.py
new file mode 100644
index 00000000..569f8c48
--- /dev/null
+++ b/tools/runtime_pilot/evaluate.py
@@ -0,0 +1,527 @@
+"""Validate independently captured rank/step state without importing candidates.
+
+The operator owns these artifacts and must freeze source and artifact manifests
+outside this evaluator. ``weights_only=True`` limits deserialization; it does
+not establish artifact provenance or defend against a candidate rewriting files.
+Optimizer containers are native dict/list/tuple values. Their key/element order,
+scalar types and tensor bytes remain significant. Model tolerances apply only
+to floating/complex parameter comparisons, including replicated parameters.
+"""
+
+from __future__ import annotations
+
+import math
+import re
+import struct
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Literal, TypeAlias
+
+import torch
+
+Optimizer: TypeAlias = (
+    None
+    | bool
+    | int
+    | float
+    | str
+    | torch.Tensor
+    | list["Optimizer"]
+    | tuple["Optimizer", ...]
+    | dict[int | str, "Optimizer"]
+)
+Identity: TypeAlias = tuple[int, int]
+Category: TypeAlias = Literal["model", "optimizer", "rng", "replicated"]
+
+
+@dataclass(frozen=True)
+class Snapshot:
+    rank: int
+    tp_rank: int
+    dp_rank: int
+    step: int
+    parameters: dict[str, torch.Tensor]
+    replicated_names: tuple[str, ...]
+    optimizer: Optimizer
+    rng: dict[str, torch.Tensor]
+
+
+@dataclass(frozen=True)
+class Evaluation:
+    passed: bool
+    world_size: int
+    steps: tuple[int, ...]
+    candidate_artifacts: int
+    reference_artifacts: int
+    tensor_parallel_size: int
+    data_parallel_size: int
+    model_checks: int
+    optimizer_checks: int
+    rng_checks: int
+    replicated_checks: int
+    replicated_status: Literal["passed", "violated", "not_applicable"]
+    failed_checks: int
+    failed_paths: tuple[str, ...]
+    failed_paths_truncated: bool
+    maximum_absolute_model_difference: float | None
+    model_comparable_tensors: int
+    model_absolute_difference_overflow: bool
+    atol: float
+    rtol: float
+
+
+@dataclass
+class _Checks:
+    counts: dict[Category, int] = field(
+        default_factory=lambda: {"model": 0, "optimizer": 0, "rng": 0, "replicated": 0}
+    )
+    failures: int = 0
+    replicated_failures: int = 0
+    paths: list[str] = field(default_factory=list)
+    maximum_model_difference: float | None = None
+    comparable_model_tensors: int = 0
+    model_difference_overflow: bool = False
+
+    def record(self, category: Category, path: str, passed: bool) -> None:
+        self.counts[category] += 1
+        if not passed:
+            self.failures += 1
+            if category == "replicated":
+                self.replicated_failures += 1
+            if len(self.paths) < 100:
+                self.paths.append(path)
+
+
+def _integer(value: object, path: str, *, positive: bool = False) -> int:
+    if isinstance(value, bool) or not isinstance(value, int):
+        raise TypeError(f"{path}: expected integer, excluding bool")
+    if value < (1 if positive else 0):
+        raise ValueError(f"{path}: integer is out of range")
+    return value
+
+
+def _tolerance(value: object, path: str) -> float:
+    if isinstance(value, bool) or not isinstance(value, (int, float)):
+        raise TypeError(f"{path}: expected finite nonnegative number")
+    if not math.isfinite(value) or value < 0:
+        raise ValueError(f"{path}: expected finite nonnegative number")
+    return float(value)
+
+
+def _mapping(value: object, path: str) -> dict[str, object]:
+    if not isinstance(value, dict):
+        raise TypeError(f"{path}: expected dict")
+    result: dict[str, object] = {}
+    for key, child in value.items():
+        if not isinstance(key, str) or not key:
+            raise TypeError(f"{path}: keys must be nonempty strings")
+        result[key] = child
+    return result
+
+
+def _tensor(value: object, path: str) -> torch.Tensor:
+    if not isinstance(value, torch.Tensor):
+        raise TypeError(f"{path}: expected tensor")
+    if value.layout != torch.strided or value.is_quantized:
+        raise TypeError(f"{path}: expected dense nonquantized tensor")
+    if value.device.type != "cpu":
+        raise ValueError(f"{path}: deserialized tensor is not on CPU")
+    if not bool(torch.isfinite(value).all()):
+        raise ValueError(f"{path}: tensor contains nonfinite values")
+    return value
+
+
+def _tensor_mapping(value: object, path: str) -> dict[str, torch.Tensor]:
+    mapping = _mapping(value, path)
+    if not mapping:
+        raise ValueError(f"{path}: empty mapping does not establish coverage")
+    return {name: _tensor(item, f"{path}[{name!r}]") for name, item in mapping.items()}
+
+
+def _optimizer(value: object, path: str, active: set[int]) -> Optimizer:
+    if isinstance(value, torch.Tensor):
+        return _tensor(value, path)
+    if value is None or isinstance(value, (bool, int, str)):
+        return value
+    if isinstance(value, float):
+        if not math.isfinite(value):
+            raise ValueError(f"{path}: nonfinite scalar")
+        return value
+    # Native container types are the artifact contract. Reject subclasses rather
+    # than silently normalizing their type (e.g. OrderedDict) into a plain dict.
+    if type(value) not in (dict, list, tuple):
+        raise TypeError(f"{path}: unsupported optimizer value {type(value).__name__}")
+    identity = id(value)
+    if identity in active:
+        raise ValueError(f"{path}: cyclic optimizer containers are unsupported")
+    active.add(identity)
+    try:
+        if isinstance(value, list):
+            return [
+                _optimizer(child, f"{path}[{i}]", active)
+                for i, child in enumerate(value)
+            ]
+        if isinstance(value, tuple):
+            return tuple(
+                _optimizer(child, f"{path}[{i}]", active)
+                for i, child in enumerate(value)
+            )
+        assert isinstance(value, dict)
+        result: dict[int | str, Optimizer] = {}
+        for key, child in value.items():
+            if isinstance(key, bool) or not isinstance(key, (int, str)):
+                raise TypeError(
+                    f"{path}: optimizer dict keys must be int or str, excluding bool"
+                )
+            result[key] = _optimizer(child, f"{path}[{key!r}]", active)
+        return result
+    finally:
+        active.remove(identity)
+
+
+def _snapshot(value: object, path: Path) -> Snapshot:
+    record = _mapping(value, str(path))
+    required = {
+        "rank",
+        "tp_rank",
+        "dp_rank",
+        "step",
+        "parameters",
+        "replicated_names",
+        "optimizer",
+        "rng",
+    }
+    if set(record) != required:
+        raise ValueError(
+            f"{path}: snapshot fields differ; missing={sorted(required - set(record))}, extra={sorted(set(record) - required)}"
+        )
+    parameters = _tensor_mapping(record["parameters"], f"{path}.parameters")
+    names = record["replicated_names"]
+    if not isinstance(names, list):
+        raise TypeError(f"{path}.replicated_names: expected list")
+    replicated: list[str] = []
+    for name in names:
+        if not isinstance(name, str) or name not in parameters:
+            raise ValueError(
+                f"{path}.replicated_names: unknown parameter name {name!r}"
+            )
+        if name in replicated:
+            raise ValueError(f"{path}.replicated_names: duplicate parameter {name!r}")
+        replicated.append(name)
+    rng = _tensor_mapping(record["rng"], f"{path}.rng")
+    for name, tensor in rng.items():
+        if tensor.dtype != torch.uint8 or tensor.ndim != 1 or tensor.numel() == 0:
+            raise ValueError(f"{path}.rng[{name!r}]: expected nonempty 1-D byte tensor")
+    return Snapshot(
+        rank=_integer(record["rank"], f"{path}.rank"),
+        tp_rank=_integer(record["tp_rank"], f"{path}.tp_rank"),
+        dp_rank=_integer(record["dp_rank"], f"{path}.dp_rank"),
+        step=_integer(record["step"], f"{path}.step"),
+        parameters=parameters,
+        replicated_names=tuple(replicated),
+        optimizer=_optimizer(record["optimizer"], f"{path}.optimizer", set()),
+        rng=rng,
+    )
+
+
+def _load(
+    directory: Path, world_size: int, steps: tuple[int, ...]
+) -> dict[Identity, Snapshot]:
+    if not directory.is_dir() or directory.is_symlink():
+        raise ValueError(f"{directory}: expected existing nonsymlink directory")
+    snapshots: dict[Identity, Snapshot] = {}
+    for path in sorted(directory.glob("rank*/step*.pt")):
+        if path.is_symlink() or path.parent.is_symlink() or not path.is_file():
+            raise ValueError(f"{path}: expected regular nonsymlink artifact")
+        rank_match = re.fullmatch(r"rank([0-9]+)", path.parent.name)
+        step_match = re.fullmatch(r"step([0-9]+)\.pt", path.name)
+        if rank_match is None or step_match is None:
+            raise ValueError(f"{path}: malformed rank/step filename")
+        raw: object = torch.load(path, map_location="cpu", weights_only=True)
+        state = _snapshot(raw, path)
+        identity = (state.rank, state.step)
+        if identity in snapshots:
+            raise ValueError(f"{path}: duplicate rank/step identity {identity}")
+        if identity != (int(rank_match[1]), int(step_match[1])):
+            raise ValueError(f"{path}: path disagrees with recorded rank/step")
+        snapshots[identity] = state
+    expected = {(rank, step) for rank in range(world_size) for step in steps}
+    if set(snapshots) != expected:
+        raise ValueError(
+            f"{directory}: incomplete or unexpected rank/step coverage; missing={sorted(expected - set(snapshots))}, extra={sorted(set(snapshots) - expected)}"
+        )
+    return snapshots
+
+
+def _topology(
+    states: dict[Identity, Snapshot], world_size: int, steps: tuple[int, ...]
+) -> tuple[int, int]:
+    first = {
+        rank: (states[rank, steps[0]].dp_rank, states[rank, steps[0]].tp_rank)
+        for rank in range(world_size)
+    }
+    dp_size = max(dp for dp, _ in first.values()) + 1
+    tp_size = max(tp for _, tp in first.values()) + 1
+    if dp_size * tp_size != world_size:
+        raise ValueError("Incomplete or duplicated data/tensor parallel topology")
+    expected = {(dp, tp) for dp in range(dp_size) for tp in range(tp_size)}
+    if len(expected) != world_size or set(first.values()) != expected:
+        raise ValueError("Incomplete or duplicated data/tensor parallel topology")
+    for (rank, _), state in states.items():
+        if (state.dp_rank, state.tp_rank) != first[rank]:
+            raise ValueError("Rank topology changed across observed steps")
+    for step in steps:
+        for dp in range(dp_size):
+            group = [
+                state
+                for state in states.values()
+                if state.step == step and state.dp_rank == dp
+            ]
+            names = group[0].replicated_names
+            if any(state.replicated_names != names for state in group):
+                raise ValueError("Replicated parameter names differ within a TP group")
+            if tp_size > 1 and not names:
+                raise ValueError(
+                    "Empty replicated parameter coverage for a multi-rank TP group"
+                )
+            if tp_size > 1 and any(
+                state.parameters[name].numel() == 0 for state in group for name in names
+            ):
+                raise ValueError("Empty replicated tensor does not establish coverage")
+    return tp_size, dp_size
+
+
+def _same_tensor_bytes(left: torch.Tensor, right: torch.Tensor) -> bool:
+    return torch.equal(
+        left.contiguous().reshape(-1).view(torch.uint8),
+        right.contiguous().reshape(-1).view(torch.uint8),
+    )
+
+
+def _maximum_difference(left: torch.Tensor, right: torch.Tensor) -> float:
+    if left.numel() == 0:
+        return 0.0
+    if left.is_floating_point() or left.is_complex():
+        dtype = torch.complex128 if left.is_complex() else torch.float64
+        return float((left.to(dtype) - right.to(dtype)).abs().max().item())
+    # Python integer subtraction avoids int64 overflow and float64 rounding
+    # that could otherwise report zero for unequal large integer parameters.
+    left_values: object = left.reshape(-1).tolist()
+    right_values: object = right.reshape(-1).tolist()
+    assert isinstance(left_values, list) and isinstance(right_values, list)
+    maximum = 0
+    for left_item, right_item in zip(left_values, right_values):
+        assert isinstance(left_item, (bool, int)) and isinstance(
+            right_item, (bool, int)
+        )
+        maximum = max(maximum, abs(int(left_item) - int(right_item)))
+    return float(maximum)
+
+
+def _compare_tensor(
+    candidate: torch.Tensor,
+    reference: torch.Tensor,
+    path: str,
+    category: Category,
+    checks: _Checks,
+    atol: float,
+    rtol: float,
+) -> None:
+    comparable = (
+        candidate.dtype == reference.dtype and candidate.shape == reference.shape
+    )
+    checks.record(category, path + ".dtype_shape", comparable)
+    if not comparable:
+        return
+    if category == "model":
+        checks.comparable_model_tensors += 1
+        difference = _maximum_difference(candidate, reference)
+        if math.isfinite(difference):
+            previous = checks.maximum_model_difference
+            checks.maximum_model_difference = (
+                difference if previous is None else max(previous, difference)
+            )
+        else:
+            checks.model_difference_overflow = True
+    if (atol or rtol) and (candidate.is_floating_point() or candidate.is_complex()):
+        same = bool(torch.allclose(candidate, reference, atol=atol, rtol=rtol))
+    else:
+        same = _same_tensor_bytes(candidate, reference)
+    checks.record(category, path + ".value", same)
+
+
+def _compare_optimizer(
+    candidate: Optimizer, reference: Optimizer, path: str, checks: _Checks
+) -> None:
+    checks.record("optimizer", path + ".type", type(candidate) is type(reference))
+    if type(candidate) is not type(reference):
+        return
+    if isinstance(candidate, torch.Tensor):
+        assert isinstance(reference, torch.Tensor)
+        _compare_tensor(candidate, reference, path, "optimizer", checks, 0.0, 0.0)
+    elif isinstance(candidate, dict):
+        assert isinstance(reference, dict)
+        candidate_keys = [(type(key), key) for key in candidate]
+        reference_keys = [(type(key), key) for key in reference]
+        same = candidate_keys == reference_keys
+        checks.record("optimizer", path + ".ordered_keys", same)
+        if same:
+            for key in reference:
+                _compare_optimizer(
+                    candidate[key], reference[key], f"{path}[{key!r}]", checks
+                )
+    elif isinstance(candidate, (list, tuple)):
+        assert isinstance(reference, (list, tuple))
+        checks.record("optimizer", path + ".length", len(candidate) == len(reference))
+        for index, (left, right) in enumerate(zip(candidate, reference)):
+            _compare_optimizer(left, right, f"{path}[{index}]", checks)
+    elif isinstance(candidate, float):
+        assert isinstance(reference, float)
+        checks.record(
+            "optimizer",
+            path + ".value",
+            struct.pack("!d", candidate) == struct.pack("!d", reference),
+        )
+    else:
+        checks.record("optimizer", path + ".value", candidate == reference)
+
+
+def evaluate(
+    candidate_directory: Path,
+    reference_directory: Path,
+    world_size: int,
+    steps: list[int],
+    atol: float = 0.0,
+    rtol: float = 0.0,
+) -> Evaluation:
+    """Compare complete coverage; malformed/missing evidence raises an error.
+
+    Replication checks use each step's TP ranks within one DP group. A TP1
+    topology reports ``not_applicable`` even if several DP replicas exist.
+    References must independently satisfy the same replication invariant.
+    """
+    world_size = _integer(world_size, "world_size", positive=True)
+    if not isinstance(steps, list) or not steps:
+        raise ValueError("steps must be a nonempty list")
+    expected_steps = tuple(_integer(step, "step") for step in steps)
+    if tuple(sorted(set(expected_steps))) != expected_steps:
+        raise ValueError("steps must be strictly increasing and unique")
+    atol = _tolerance(atol, "atol")
+    rtol = _tolerance(rtol, "rtol")
+    candidate = _load(candidate_directory, world_size, expected_steps)
+    reference = _load(reference_directory, world_size, expected_steps)
+    topology = _topology(candidate, world_size, expected_steps)
+    if topology != _topology(reference, world_size, expected_steps):
+        raise ValueError("Candidate and reference parallel topology differ")
+    checks = _Checks()
+    for identity in sorted(reference):
+        left, right = candidate[identity], reference[identity]
+        path = f"rank{identity[0]}/step{identity[1]}"
+        if (left.dp_rank, left.tp_rank) != (right.dp_rank, right.tp_rank):
+            raise ValueError(f"{path}: candidate/reference rank topology differs")
+        if left.replicated_names != right.replicated_names:
+            raise ValueError(f"{path}: candidate/reference replicated metadata differs")
+        checks.record(
+            "model",
+            path + ".parameter_names",
+            list(left.parameters) == list(right.parameters),
+        )
+        for name in right.parameters:
+            if name in left.parameters:
+                _compare_tensor(
+                    left.parameters[name],
+                    right.parameters[name],
+                    f"{path}.parameters[{name!r}]",
+                    "model",
+                    checks,
+                    atol,
+                    rtol,
+                )
+        _compare_optimizer(left.optimizer, right.optimizer, path + ".optimizer", checks)
+        checks.record("rng", path + ".rng_names", list(left.rng) == list(right.rng))
+        for name in right.rng:
+            if name in left.rng:
+                _compare_tensor(
+                    left.rng[name],
+                    right.rng[name],
+                    f"{path}.rng[{name!r}]",
+                    "rng",
+                    checks,
+                    0.0,
+                    0.0,
+                )
+    tp_size, dp_size = topology
+    if tp_size > 1:
+        for step in expected_steps:
+            for dp in range(dp_size):
+                group = sorted(
+                    (
+                        state
+                        for state in candidate.values()
+                        if state.step == step and state.dp_rank == dp
+                    ),
+                    key=lambda state: state.tp_rank,
+                )
+                base = group[0]
+                for state in group[1:]:
+                    for name in base.replicated_names:
+                        path = (
+                            f"step{step}/dp{dp}/tp{state.tp_rank}.replicated[{name!r}]"
+                        )
+                        _compare_tensor(
+                            state.parameters[name],
+                            base.parameters[name],
+                            path,
+                            "replicated",
+                            checks,
+                            atol,
+                            rtol,
+                        )
+                        reference_base = reference[base.rank, step].parameters[name]
+                        reference_other = reference[state.rank, step].parameters[name]
+                        reference_checks = _Checks()
+                        _compare_tensor(
+                            reference_other,
+                            reference_base,
+                            path,
+                            "replicated",
+                            reference_checks,
+                            atol,
+                            rtol,
+                        )
+                        if reference_checks.failures:
+                            raise ValueError(
+                                f"{path}: known-correct reference violates replication"
+                            )
+        assert checks.counts["replicated"] > 0
+    status: Literal["passed", "violated", "not_applicable"] = (
+        "not_applicable"
+        if tp_size == 1
+        else "violated" if checks.replicated_failures else "passed"
+    )
+    return Evaluation(
+        passed=checks.failures == 0,
+        world_size=world_size,
+        steps=expected_steps,
+        candidate_artifacts=len(candidate),
+        reference_artifacts=len(reference),
+        tensor_parallel_size=tp_size,
+        data_parallel_size=dp_size,
+        model_checks=checks.counts["model"],
+        optimizer_checks=checks.counts["optimizer"],
+        rng_checks=checks.counts["rng"],
+        replicated_checks=checks.counts["replicated"],
+        replicated_status=status,
+        failed_checks=checks.failures,
+        failed_paths=tuple(checks.paths),
+        failed_paths_truncated=checks.failures > len(checks.paths),
+        maximum_absolute_model_difference=(
+            None
+            if checks.model_difference_overflow
+            else checks.maximum_model_difference
+        ),
+        model_comparable_tensors=checks.comparable_model_tensors,
+        model_absolute_difference_overflow=checks.model_difference_overflow,
+        atol=atol,
+        rtol=rtol,
+    )