[Essoz]

This measures whether optional TrainCheck access helps agents diagnose and repair real distributed training failures under neutral tasks. No task says there is a bug, and treatments may choose ordinary custom diagnostics. The TrainCheck package itself is unchanged.

Three native case comparisons are complete. The first two produced source-first diagnoses. In the third, agents used cross-rank runtime evidence to confirm a state inconsistency and reject an incorrect loading explanation, while leaving the cache-version origin unresolved. No optional-tool arm activated TrainCheck. These results establish bounded working repairs and measured costs, not a TrainCheck efficiency benefit.

The DeepSpeed #1801 pilot ran four fresh agents on native tensor-parallel Megatron training. First source diagnoses took 16–36 seconds, before requesting GPUs. All first valid repairs were submitted in turn 2 and passed the base, fresh-seed, TP1 and clipping-disabled scenarios.

Trial	First valid repair, adjusted seconds	Final input + output tokens	Solver GPU-seconds
Control 1	162.691	1,236,382	104.607
Optional TrainCheck 1	249.376	1,455,127	94.831
Optional TrainCheck 2	156.663	805,536	69.355
Control 2	387.316	1,336,513	94.298

Adjusted time subtracts measured service waiting and operator suspension intervals; raw times are retained. Three trials needed same-thread transport recovery under the original budgets, without rerunning completed GPU work. Independent grading cost 1,069.616 GPU-seconds separately from 363.091 solver GPU-seconds.

The second case uses native Transformers #34242/#34373: unequal per-rank valid-token counts change the effective objective despite bitwise-identical DDP replicas. Qualification measured gradient relative L2 errors of 0.3827/0.3721 for accumulation 1/2, versus at most 3.29e-7 with global-token averaging. Both final agent submissions passed all seven frozen independent scenarios. Eight complete source maps cover all thirteen submitted revisions; no earlier unknown result blocks first-valid adjudication.

DDP milestone or cost	Control	Optional TrainCheck
Source-supported diagnosis, seconds	47.928	38.785
Runtime-supported diagnosis, seconds	149.654	102.714
First independently valid submission, raw seconds	181.380	204.074
First independently valid submission, queue-adjusted seconds	166.428	204.071
Input + output tokens through first valid submission	595,207	845,768
GPU-seconds through first valid submission	25.043	37.548
Full trial raw wall, seconds	453.645	474.343
Final input + output tokens	1,471,632	1,744,783
Full trial allocated GPU-seconds	100.203	124.993

The DDP pair had no transport or CLI failures. Independent grading added 1,416.962 operator GPU-seconds separately from 225.196 solver GPU-seconds and earlier qualification costs. The treatment read the manual and used ordinary Trainer/optimizer diagnostics. Different chosen validation scopes and one trial per arm preclude a causal efficiency ratio. Tokens include cached input once; GPU cost is allocated devices times worker lifetime, including startup and cleanup, not active utilization. First-valid submission time is retrospective and excludes private grading latency.

The third case reproduces the production source-consistency failure mode fixed in Halo release #103. Unchanged cache helpers and official Pythia weights produce a 14,067,712-parameter native FSDP model containing 38 parameter tensors from each of two independently cached sources. All ranks reconstruct the same hybrid, so replica equality alone misses it. This uses logical nodes on one Ring18 host and reconstructed historical cache namespaces, with the full original relevant source visible; it is a component reproduction, not a preserved incident.

The fresh pair completed without CLI/transport failures. Both agents proposed unsafe FSDP assembly from source, then used cross-rank runtime diagnostics to reject their specific RAM-efficient-loading explanation and confirm inconsistent initial state. Neither identified the actual cache-version origin. The optional-tool agent read the manual but neither prepared nor activated TrainCheck. Both changed the training pipeline only to enable initial-state synchronization, plus comments.

Every submitted revision was evaluated after both solvers finished: nine ordered occurrences, five exact complete source maps, and 30 scenario replays. Both final submissions pass all six scenarios; both earliest valid repairs are turn 2, with no earlier unknown source. The two initial snapshots each pass four healthy controls and fail mixed/swapped-source coherence. Overall: 26 correct scenario outcomes, four expected initial-source failures, zero unknowns or cost gaps. Mixed/swapped repairs may select either authentic coherent generation; full model/buffer/gradient/optimizer/RNG/input state must match that source's same-topology oracle.

FSDP milestone or cost	Control	Optional TrainCheck
Pre-run mechanism hypothesis, seconds	30.251	33.934
Runtime-confirmed state inconsistency, queue-adjusted seconds	131.168	134.135
First independently valid submission, raw seconds	148.365	204.705
First independently valid submission, queue-adjusted seconds	148.362	154.047
Input + output tokens through first valid submission	719,915	688,330
GPU-seconds through first valid submission	35.563	34.992
Full trial raw wall, seconds	432.754	414.381
Final input + output tokens	1,369,747	1,137,285
Full trial allocated GPU-seconds	118.764	82.527

Control chose eight matched updates plus 16 ordinary distributed updates; treatment chose 16 matched diagnostic updates and covered nonpersistent buffers as well as saved parameters. Different validation work and one pair preclude a causal efficiency ratio. Runtime-diagnosis token counts are bounded by completed-turn counters (293,535–719,915 vs 304,371–688,330), not interpolated. All token totals include cached input once. The four optional-tool arms across these three pilots show no observed TrainCheck activation; these results do not establish a TrainCheck speedup or measure its benefit when actually used.

The original strict single-vs-two-GPU model check failed and remains failed. Separately labeled batch1×2 controls match FSDP model values exactly at steps 0/1/2 for both sources, isolating batch-shape/accumulation effects without a kernel-level explanation. The pre-agent acceptance scope is source coherence and exact same-topology trajectories over two updates, one seed and one small corpus. Qualification cost 905.087 GPU-seconds; solver requests 201.291; private grading 815.932, all separate. GPUs were released. The selective scalar-API TrainCheck preflight preserved complete observed state and record multiplicity, but is not an overhead benchmark. The post-trial grader passed 14 focused regressions, pinned strict mypy and formatting/lint checks, independently rerun by another reviewer.

Reusable validation checks complete rank/step coverage and full model, optimizer and recorded RNG state. Private observers never go to solver packets; control workers reject ordinary TrainCheck imports. Validation includes 91 repository tests and pinned hooks/strict mypy for new helpers/tests. Machine-specific checks include 33 DeepSpeed grading tests, 42 accounting/join tests, 27 DDP service tests, 43 DDP accounting tests and 18 DDP grading tests. The original strict DDP numerical-tolerance report and separately labeled pre-agent calibration are retained. A DeepSpeed collection caller tuple/list error has a failing-before/passing-after integration regression; recollection used unchanged snapshots/evaluator/plan without GPU retraining.

See docs/measurements/real-rank-runtime-pilot-20261102.md, docs/measurements/ddp-runtime-state-pilot-20261102.md and docs/measurements/fsdp-source-consistency-pilot-20261102.md for protocols, completed outcomes, qualification evidence and limits. Large data, raw transcripts and snapshots remain in isolated local/Ring18 evidence directories. The full private DDP operator archive remains local after automatic review rejected its upload; an existing runtime/source archive was made in place on Ring18. No rejected upload was retried.

Depends on PR #170 (base: codex/checkpoint-continuation-costs). All three bounded pilot comparisons and independent grading are complete. Progress is saved in signed commits, with final results in 5530554. Ready for review; no merge requested.