From 5051d5f074832aec54e9abb4e495c7f37fa305ff Mon Sep 17 00:00:00 2001 From: Yuxuan Jiang Date: Mon, 2 Nov 2026 03:39:51 +0800 Subject: [PATCH 01/13] Define real tensor-parallel runtime validation protocol --- .../real-rank-runtime-pilot-20261102.md | 211 ++++++++++++++++++ 1 file changed, 211 insertions(+) create mode 100644 docs/measurements/real-rank-runtime-pilot-20261102.md diff --git a/docs/measurements/real-rank-runtime-pilot-20261102.md b/docs/measurements/real-rank-runtime-pilot-20261102.md new file mode 100644 index 00000000..bccf2b49 --- /dev/null +++ b/docs/measurements/real-rank-runtime-pilot-20261102.md @@ -0,0 +1,211 @@ +# Real cross-rank runtime case: agent-efficiency protocol + +**Status: in progress, 2026-11-02.** Case provenance and the experiment design +are being established. Ring18 qualification and agent trials are pending. This +document reports no new training result, repair-success rate, or TrainCheck +efficiency improvement. It is operator documentation and must not be included +in solver packets. + +The development base is `codex/checkpoint-continuation-costs`, +[PR #170](https://github.com/OrderLab/TrainCheck/pull/170), at +`c79218c960a823f954bf49dad354975da39e6f24`. Subsequent source revisions, workload +configuration, packet manifests, commands and raw evidence must be frozen before +the first solver sees the task. + +## Question and case selection + +The question is whether optional access to TrainCheck helps an agent investigate +and repair a real distributed training problem more efficiently under a neutral +engineering task. The comparison will measure causal localization, independently +valid code, agent-visible runtime verification and final completion separately. + +The selected candidate is the historical DeepSpeed BF16 optimizer failure commonly +identified as **DS-1801** in the TrainCheck paper. Tensor-parallel training includes +both partitioned and replicated parameters. The affected implementation excludes +replicated parameters on nonzero TP ranks from clipping, although excluding their +duplicate contributions is appropriate when calculating the global norm. Thus +replicated parameters can receive different updates across TP ranks while the +training process continues. + +| Candidate considered | Reason to consider it | Selection limitation | +| --- | --- | --- | +| DS-1801, BF16 clipping under tensor parallelism | Real silent training defect; combines rank, replication metadata, clipping activation and optimizer state; an existing Megatron–DeepSpeed environment and CodeParrot data are available on Ring18. | The repair is small once the relevant framework path is identified. Source inspection or prior knowledge can still succeed. | +| [PyTorch #104336](https://github.com/pytorch/pytorch/issues/104336), DDP device transfers | Real failure of gradient synchronization after moving an already wrapped model between CPU and GPU. | The device-transfer sequence is a recognizable source-level clue; it may repeat the weakness of the earlier pilot. | +| [DeepSpeed #5794](https://github.com/deepspeedai/DeepSpeed/issues/5794), MoE communication | Rank-local routing and collective arguments provide relevant runtime evidence. | The available artifact uses an inference workload with additional dependencies and a hang; it is less direct for this silent-training study. | + +DS-1801 is selected for its real provenance, training relevance and available +environment, not because a preliminary agent failed to solve it. All subsequent +qualification attempts and failures must be retained. This study does not claim +that cross-rank observation is logically necessary for every successful diagnosis. +Source-based successes remain valid results. + +## Historical and current source provenance + +The upstream repository is now `deepspeedai/DeepSpeed`. +[PR #1801](https://github.com/deepspeedai/DeepSpeed/pull/1801), titled +“bf16+pipeline parallelism,” is merged. The following sources were inspected: + +- The [vulnerable parent + `a4cbf0c1d300d78c6124656458115d1050869142`](https://github.com/deepspeedai/DeepSpeed/blob/a4cbf0c1d300d78c6124656458115d1050869142/deepspeed/runtime/bf16_optimizer.py#L298) + applies the nonzero-TP-rank filtering outside `if not for_clipping`. +- The [fix commit + `e24814a10de04ce280efe2adb027b023e3336493`](https://github.com/deepspeedai/DeepSpeed/commit/e24814a10de04ce280efe2adb027b023e3336493) + moves the condition inside that branch. Its literal intermediate patch uses + `p` instead of `lp`; this commit alone is not the executable positive control. +- The [final merged source + `56c522386878df2513ccb548de76b50191ae3af7`](https://github.com/deepspeedai/DeepSpeed/blob/56c522386878df2513ccb548de76b50191ae3af7/deepspeed/runtime/bf16_optimizer.py#L406) + has the corrected branch and `lp` reference. + +**Correction to an older discovery note:** the live Ring18 source inspection for +this study found that DeepSpeed HEAD +`32d97976ce08f00228ceb55a2d93005bd8b2769a` has the correct nested condition in +`bf16_optimizer.py`. The working-tree diff moves that condition outside +`if not for_clipping`, making the inspected dirty working tree vulnerable. The +older `outputs/trace-bug-impact/checkpoint-discovery.md` described this edit as a +repair; that characterization must not be used as provenance for this study. + +The saved live diff is +`/Users/matt/gitrepos/traincheck/outputs/ring18-runtime-20261102/original-bf16-working.diff`, +SHA-256 `c04efe0f7f59bbcd973a6289a57b40fa4f832a7855b1b4ada9ebb73e3fcde8ad`. +Its two changed lines move the identical condition and `continue` out by one +indentation level. This artifact has been read and hashed locally; the complete +source snapshot and provenance manifest remain to be frozen. + +The planned candidate is an isolated snapshot of the inspected working-tree +source. The positive control uses the corresponding correct HEAD version of +`bf16_optimizer.py` while preserving the same remaining compatibility changes. +The final evidence bundle must retain the exact source diff and per-file hashes, +including a check that the control differs only in the intended logic. This is a +historical semantic reproduction on a compatible environment, not execution of +an untouched historical checkout. Shared repositories and installed environments +are inputs and must remain unchanged. + +## Planned workload and qualification + +Use native Megatron–DeepSpeed GPT training on CodeParrot: four transformer layers, +hidden width 64, TP2, BF16 and ZeRO stage 0. Sequence lengths 128 and 256 are +qualification candidates; the chosen value, microbatch size, accumulation, +optimizer settings, clipping threshold, seeds and update count remain to be +frozen. Do not infer a model parameter count or performance result from this +configuration alone. Refresh the live Ring18 environment and available GPU +inventory before execution. + +Qualification must establish all of the following before solver launch: + +1. The candidate initializes successfully and executes finite training updates. + The measured gradient norm actually activates clipping; configured clipping + without an active event does not qualify. +2. Replicated trainable parameters agree at initialization and diverge after an + affected update. Compare logical parameter identity within the relevant TP + group, not the complete partitioned `state_dict` across all ranks. +3. The isolated corrected control preserves replicated-parameter agreement and + completes the same training computation. Capture full values, shapes and + dtypes, with any numerical tolerance declared before grading. +4. Pre/post-clipping FP32 gradient observations explain the differing updates. + BF16 parameter rounding can mask an early difference; missing or unchanged + low-precision observations are not proof that clipping was equivalent. +5. Clipping-disabled and TP1 controls separate the triggering conditions from + unrelated startup or precision problems. Fresh seeds, and an additional + TP2×DP2 configuration if feasible, provide independent validation. +6. An instrumentation preflight preserves the relevant model, optimizer and RNG + behavior and establishes actual observation coverage. This preflight is + operator work, not solver use of TrainCheck. + +State observations must identify the run, global update, rank/group membership +and logical parameter. Wall-clock timestamp ordering across processes alone is +not sufficient alignment. Setup errors, nonactivating conditions, absent +observations, failed comparisons and successful comparisons remain separate +outcomes. Qualification and preflight GPU costs are recorded separately from +solver costs. + +## Neutral solver task and matched access + +The intended engineering context is an FP16-to-BF16 extension of the existing +tensor-parallel pretraining pipeline. The core prompt is: + +> We are extending this language-model pretraining pipeline to BF16 while +> retaining tensor parallelism and the existing training configuration. The +> supplied outputs show the current short training run. Assess whether this +> implementation is ready for a longer run, decide what to investigate next, +> and implement any justified revision. Explain your conclusion using the +> evidence you collected. + +Supply measured ordinary training outputs, configuration and relevant full +application/framework source. If curves show no material quality difference, +do not describe a quality regression that was not observed. Do not include this +protocol, issue identities, Git history, the private corrected source, a targeted +checker, or bug/fix annotations in solver packets. Preserve ordinary source names +and behavior; do not obscure code to make static reasoning artificially harder. +The exact allowed revision scope and preserved training contracts must be stated +and frozen before launch. + +Plan two fresh matched pairs, with launch order reversed in the second pair. +Use the same model, reasoning setting, tool interface, runtime, request allowance, +update allowance and time limits. Freeze their exact values before exposure. +Controls may use ordinary logging, assertions and custom Python diagnostics. +Treatments additionally receive the generic TrainCheck observation documentation +and implementation; no rule targets this case and no task step requires its use. + +Standard responses return native logs, training metrics and ordinary artifacts. +Do not automatically supply private grader state exports as a diagnostic shortcut. +Either arm may request and capture equivalent runtime state explicitly under the +same resource budget. Private grading begins only after all final submissions +freeze; grader feedback is not supplied during the task. + +## Milestones, costs and evidence + +| Milestone | Definition | +| --- | --- | +| First supported localization | First received visible statement containing a sufficient causal explanation supported by evidence available to the agent at that time. Source evidence qualifies; later traces cannot be credited retroactively. | +| First valid submitted implementation | Earliest captured submission later passing all prespecified independent contracts. This is a retrospective correctness measure, not proof the agent had already verified it. | +| First agent-visible runtime verification | First returned runtime evidence that supports the submitted repair under the declared checks. State exactly which conditions were checked; this can precede or follow independent qualification of that submission. | +| Final completion | Agent's final frozen implementation and report, including all accepted diagnostic requests and their outcomes. | + +Times begin at each trial's launch. Retain raw wall time, shared-service waiting, +wall time excluding only that measured wait, and CLI execution time. Source-copy, +transport, startup, diagnostics and each agent's own execution remain charged as +appropriate; shared service waiting is not cluster scheduler time. Message receipt +and submission completion are observable upper bounds, not the instant of a +private reasoning step or edit. + +Native input and output token counters are recorded at completed-turn boundaries. +For resumed sessions with cumulative counters, use validated deltas. Cached input +is a subset of input and reasoning output a subset of output; do not add either +twice. A within-turn localization can have a token interval rather than a precise +count. Missing usage is unknown, not zero. Report cache distribution and avoid +translating total-token differences into monetary savings without actual pricing. + +GPU cost is supervised worker lifetime multiplied by allocated devices, including +initialization, diagnostics and failed requests. This measures allocated +GPU-seconds, not utilization or pure training compute. Keep solver, qualification, +instrumentation-preflight and private-grading costs in distinct ledgers. Record +process termination and ownership so failed distributed workers cannot silently +continue consuming resources. + +Preserve prompts, raw CLI events with receive times, every source submission, +request/worker receipts, output manifests, environment versions and grading +artifacts. Record actual TrainCheck activation, trace coverage, failures and +produced records. Reading its manual or source is not instrumentation use; unused +availability cannot establish the effect or overhead of active use. Large data, +checkpoints and traces stay outside Git, with hashes and retrieval locations in +the final report. Source and protocol progress use signed commits and a focused +PR; no merge is part of this protocol. + +## Interpretation limits and prior result + +The [previous synthetic checkpoint pilot](checkpoint-continuation-costs-20261101.md) +had four first-turn repairs that later passed independent validation. All four +agents localized the ordering problem from source before requesting a GPU run; +neither treatment activated TrainCheck. Supported localization took +26.79–38.01 seconds, first valid submission took 99.39–111.12 seconds, and GPU +cost before those submissions was zero. These are measured results for that +particular case, not a general one-shot success rate for the paper's bugs. + +This new case has public historical provenance and may be recognized from model +training data. A small, purposefully selected pairwise pilot cannot estimate the +incidence of silent errors in agent-authored ML systems or establish a general +TrainCheck advantage. It also does not evaluate the offline inference engine: +agents may author their own checks around observed state. If agents solve the +case from source, choose ordinary diagnostics, fail to use TrainCheck, or obtain +no efficiency benefit, retain and report those outcomes without changing the +task or success criteria after seeing them. From c6362f3f8f4707ee0d2df53231665839bdf69bbe Mon Sep 17 00:00:00 2001 From: Yuxuan Jiang Date: Mon, 2 Nov 2026 03:43:09 +0800 Subject: [PATCH 02/13] Validate complete rank state and optimizer parity in runtime trials --- tests/test_runtime_pilot_evaluator.py | 483 +++++++++++++++++++++++ tools/runtime_pilot/__init__.py | 1 + tools/runtime_pilot/evaluate.py | 527 ++++++++++++++++++++++++++ 3 files changed, 1011 insertions(+) create mode 100644 tests/test_runtime_pilot_evaluator.py create mode 100644 tools/runtime_pilot/__init__.py create mode 100644 tools/runtime_pilot/evaluate.py diff --git a/tests/test_runtime_pilot_evaluator.py b/tests/test_runtime_pilot_evaluator.py new file mode 100644 index 00000000..2919913c --- /dev/null +++ b/tests/test_runtime_pilot_evaluator.py @@ -0,0 +1,483 @@ +"""Operator-only runtime evidence tests; do not export to solver packets.""" + +from __future__ import annotations + +import shutil +from collections import OrderedDict +from collections.abc import Callable +from pathlib import Path + +import pytest +import torch + +from tools.runtime_pilot.evaluate import Evaluation, evaluate + + +def artifact(rank: int, step: int, tp_size: int = 2) -> dict[str, object]: + return { + "rank": rank, + "tp_rank": rank % tp_size, + "dp_rank": rank // tp_size, + "step": step, + "parameters": { + "norm": torch.tensor([1.0, 2.0]) + step, + "partition": torch.tensor([float(rank), 3.0]) + step, + }, + "replicated_names": ["norm"], + "optimizer": { + "state": {0: {"momentum": torch.tensor([rank + 0.25, 2.0])}}, + "groups": [{"lr": 0.125, "params": (0,)}], + "active": True, + "optional": None, + }, + "rng": {"cpu": torch.tensor([1, rank + 3, step], dtype=torch.uint8)}, + } + + +def save(root: Path, rank: int, step: int, state: dict[str, object]) -> Path: + path = root / f"rank{rank}" / f"step{step}.pt" + path.parent.mkdir(parents=True, exist_ok=True) + torch.save(state, path) + return path + + +def runs( + root: Path, world_size: int = 2, tp_size: int = 2, steps: tuple[int, ...] = (0, 1) +) -> tuple[Path, Path]: + candidate, reference = root / "candidate", root / "reference" + for directory in (candidate, reference): + for rank in range(world_size): + for step in steps: + save(directory, rank, step, artifact(rank, step, tp_size)) + return candidate, reference + + +def change( + root: Path, + update: Callable[[dict[str, object]], None], + rank: int = 0, + step: int = 1, +) -> None: + path = root / f"rank{rank}" / f"step{step}.pt" + raw: object = torch.load(path, map_location="cpu", weights_only=True) + assert isinstance(raw, dict) + state: dict[str, object] = {} + for key, value in raw.items(): + assert isinstance(key, str) + state[key] = value + update(state) + torch.save(state, path) + + +def change_field( + root: Path, name: str, value: object, rank: int = 0, step: int = 1 +) -> None: + def update(state: dict[str, object]) -> None: + state[name] = value + + change(root, update, rank, step) + + +def check( + paths: tuple[Path, Path], + world_size: int = 2, + steps: list[int] | None = None, + atol: float = 0.0, + rtol: float = 0.0, +) -> Evaluation: + return evaluate(*paths, world_size, [0, 1] if steps is None else steps, atol, rtol) + + +def test_complete_partitioned_state_passes(tmp_path: Path) -> None: + report = check(runs(tmp_path)) + assert report.passed + assert report.candidate_artifacts == report.reference_artifacts == 4 + assert report.tensor_parallel_size == 2 and report.data_parallel_size == 1 + assert report.replicated_status == "passed" and report.replicated_checks > 0 + assert report.model_comparable_tensors == 8 + assert report.maximum_absolute_model_difference == 0 + assert report.failed_checks == 0 and report.failed_paths == () + + +def test_partitioned_parameter_is_still_compared_to_reference(tmp_path: Path) -> None: + paths = runs(tmp_path) + change_field( + paths[0], + "parameters", + {"norm": torch.tensor([2.0, 3.0]), "partition": torch.tensor([9.0, 4.0])}, + rank=1, + ) + report = check(paths) + assert not report.passed + assert report.replicated_status == "passed" + assert report.maximum_absolute_model_difference == 7.0 + assert any("partition" in path for path in report.failed_paths) + + +def test_replicated_parameter_failure(tmp_path: Path) -> None: + paths = runs(tmp_path) + change_field( + paths[0], + "parameters", + {"norm": torch.tensor([2.5, 3.0]), "partition": torch.tensor([2.0, 4.0])}, + rank=1, + ) + report = check(paths) + assert not report.passed and report.replicated_status == "violated" + assert report.maximum_absolute_model_difference == 0.5 + + +@pytest.mark.parametrize( + "field,value", + [ + ( + "optimizer", + { + "state": {0: {"momentum": torch.tensor([9.0, 2.0])}}, + "groups": [{"lr": 0.125, "params": (0,)}], + "active": True, + "optional": None, + }, + ), + ("rng", {"cpu": torch.tensor([1, 99, 1], dtype=torch.uint8)}), + ], +) +def test_nonmodel_state_failure(tmp_path: Path, field: str, value: object) -> None: + paths = runs(tmp_path) + change_field(paths[0], field, value) + report = check(paths, atol=100.0, rtol=100.0) + assert not report.passed + assert report.maximum_absolute_model_difference == 0 + assert report.replicated_status == "passed" + assert all(field in path for path in report.failed_paths) + + +@pytest.mark.parametrize("remove_rank", [False, True]) +def test_missing_rank_or_step_is_error(tmp_path: Path, remove_rank: bool) -> None: + paths = runs(tmp_path) + if remove_rank: + shutil.rmtree(paths[0] / "rank1") + else: + (paths[0] / "rank1/step1.pt").unlink() + with pytest.raises(ValueError, match="coverage"): + check(paths) + + +def test_duplicate_rank_step_is_error(tmp_path: Path) -> None: + paths = runs(tmp_path) + shutil.copyfile(paths[0] / "rank0/step1.pt", paths[0] / "rank0/step01.pt") + with pytest.raises(ValueError, match="duplicate rank/step"): + check(paths) + + +def test_filename_identity_must_match(tmp_path: Path) -> None: + paths = runs(tmp_path) + change_field(paths[0], "rank", 1) + with pytest.raises(ValueError, match="path disagrees"): + check(paths) + + +@pytest.mark.parametrize( + "field,value,error", + [ + ("rank", True, TypeError), + ("step", "1", TypeError), + ("tp_rank", -1, ValueError), + ("dp_rank", 0.0, TypeError), + ("parameters", {}, ValueError), + ("parameters", {"norm": [1, 2]}, TypeError), + ("parameters", {"norm": torch.tensor([float("nan")])}, ValueError), + ("parameters", {"norm": torch.tensor([float("inf")])}, ValueError), + ( + "parameters", + { + "norm": torch.sparse_coo_tensor( + torch.tensor([[0]]), torch.tensor([1.0]), (2,) + ) + }, + TypeError, + ), + ("replicated_names", "norm", TypeError), + ("replicated_names", ["missing"], ValueError), + ("replicated_names", ["norm", "norm"], ValueError), + ("optimizer", {True: 1}, TypeError), + ("optimizer", {"x": float("nan")}, ValueError), + ("optimizer", OrderedDict([("x", 1)]), TypeError), + ("rng", {}, ValueError), + ("rng", {"cpu": torch.tensor([1.0])}, ValueError), + ("rng", {"cpu": torch.zeros((1, 2), dtype=torch.uint8)}, ValueError), + ("rng", {"cpu": torch.empty(0, dtype=torch.uint8)}, ValueError), + ], +) +def test_malformed_state_rejected( + tmp_path: Path, field: str, value: object, error: type[Exception] +) -> None: + paths = runs(tmp_path) + change_field(paths[0], field, value) + with pytest.raises(error): + check(paths) + + +@pytest.mark.parametrize("extra", [False, True]) +def test_exact_required_fields(tmp_path: Path, extra: bool) -> None: + paths = runs(tmp_path) + + def update(state: dict[str, object]) -> None: + if extra: + state["unexpected"] = 1 + else: + del state["optimizer"] + + change(paths[0], update) + with pytest.raises(ValueError, match="snapshot fields differ"): + check(paths) + + +@pytest.mark.parametrize( + "tensor", + [torch.tensor([2.0, 3.0], dtype=torch.float64), torch.tensor([[2.0, 3.0]])], +) +def test_shape_dtype_failure_is_not_value_comparison( + tmp_path: Path, tensor: torch.Tensor +) -> None: + paths = runs(tmp_path) + change_field( + paths[0], "parameters", {"norm": tensor, "partition": torch.tensor([1.0, 4.0])} + ) + report = check(paths) + assert not report.passed + assert report.model_comparable_tensors == 7 + assert any("dtype_shape" in path for path in report.failed_paths) + + +def test_parameter_tolerances_do_not_change_exact_state_checks(tmp_path: Path) -> None: + paths = runs(tmp_path) + change_field( + paths[0], + "parameters", + {"norm": torch.tensor([2.001, 3.0]), "partition": torch.tensor([1.0, 4.0])}, + ) + assert not check(paths).passed + tolerant = check(paths, atol=0.002) + assert tolerant.passed and tolerant.replicated_status == "passed" + assert tolerant.maximum_absolute_model_difference is not None + assert tolerant.maximum_absolute_model_difference > 0 + + +def test_empty_replicated_coverage_is_error(tmp_path: Path) -> None: + paths = runs(tmp_path) + for directory in paths: + for rank in range(2): + for step in (0, 1): + change_field(directory, "replicated_names", [], rank, step) + with pytest.raises(ValueError, match="Empty replicated"): + check(paths) + + +def test_tp1_explicitly_not_applicable_with_multiple_dp_ranks(tmp_path: Path) -> None: + paths = runs(tmp_path, world_size=2, tp_size=1) + for directory in paths: + for rank in range(2): + for step in (0, 1): + change_field(directory, "replicated_names", [], rank, step) + report = check(paths) + assert report.passed and report.tensor_parallel_size == 1 + assert report.data_parallel_size == 2 + assert ( + report.replicated_status == "not_applicable" and report.replicated_checks == 0 + ) + + +def test_replication_compares_only_within_same_dp_group(tmp_path: Path) -> None: + paths = runs(tmp_path, world_size=4, tp_size=2) + for directory in paths: + for rank in (2, 3): + for step in (0, 1): + change_field( + directory, + "parameters", + { + "norm": torch.tensor([10.0, 20.0]), + "partition": torch.tensor([float(rank), float(step)]), + }, + rank, + step, + ) + assert check(paths, world_size=4).passed + + +def test_reference_must_satisfy_replication(tmp_path: Path) -> None: + paths = runs(tmp_path) + change_field( + paths[1], + "parameters", + {"norm": torch.tensor([99.0, 3.0]), "partition": torch.tensor([2.0, 4.0])}, + rank=1, + ) + with pytest.raises(ValueError, match="reference violates replication"): + check(paths) + + +@pytest.mark.parametrize( + "optimizer", + [ + { + "active": True, + "groups": [{"lr": 0.125, "params": (0,)}], + "state": {0: {"momentum": torch.tensor([0.25, 2.0])}}, + "optional": None, + }, + { + "state": {0: {"momentum": torch.tensor([0.25, 2.0])}}, + "groups": [{"lr": 0.125, "params": [0]}], + "active": True, + "optional": None, + }, + { + "state": {"0": {"momentum": torch.tensor([0.25, 2.0])}}, + "groups": [{"lr": 0.125, "params": (0,)}], + "active": True, + "optional": None, + }, + { + "state": {0: {"momentum": torch.tensor([0.25, 2.0])}}, + "groups": [{"lr": 0.125, "params": (0,)}], + "active": 1, + "optional": None, + }, + ], +) +def test_optimizer_order_key_and_value_types_are_significant( + tmp_path: Path, optimizer: object +) -> None: + paths = runs(tmp_path) + change_field(paths[0], "optimizer", optimizer) + report = check(paths) + assert not report.passed + assert all("optimizer" in path for path in report.failed_paths) + + +def test_optimizer_signed_zero_tensor_and_scalar_are_exact(tmp_path: Path) -> None: + paths = runs(tmp_path) + for directory in paths: + change_field(directory, "optimizer", [0.0, torch.tensor([0.0])]) + change_field(paths[0], "optimizer", [-0.0, torch.tensor([-0.0])]) + report = check(paths, atol=1.0) + assert not report.passed and report.failed_checks == 2 + + +def test_cyclic_optimizer_rejected(tmp_path: Path) -> None: + paths = runs(tmp_path) + cycle: list[object] = [] + cycle.append(cycle) + change_field(paths[0], "optimizer", cycle) + with pytest.raises(ValueError, match="cyclic"): + check(paths) + + +def test_model_integer_difference_is_not_rounded_to_zero(tmp_path: Path) -> None: + paths = runs(tmp_path) + for directory in paths: + change_field( + directory, + "parameters", + { + "norm": torch.tensor([2.0, 3.0]), + "partition": torch.tensor([2**62], dtype=torch.int64), + }, + ) + change_field( + paths[0], + "parameters", + { + "norm": torch.tensor([2.0, 3.0]), + "partition": torch.tensor([2**62 + 1], dtype=torch.int64), + }, + ) + report = check(paths, atol=10.0) + assert not report.passed and report.maximum_absolute_model_difference == 1.0 + + +def test_total_failures_are_not_truncated(tmp_path: Path) -> None: + paths = runs(tmp_path, steps=(0,)) + for rank in range(2): + change_field(paths[0], "optimizer", [1] * 120, rank, 0) + change_field(paths[1], "optimizer", [0] * 120, rank, 0) + report = check(paths, steps=[0]) + assert report.failed_checks == 240 + assert len(report.failed_paths) == 100 and report.failed_paths_truncated + + +@pytest.mark.parametrize("steps", [[], [1, 0], [0, 0], [True], [-1]]) +def test_invalid_expected_steps(tmp_path: Path, steps: list[int]) -> None: + paths = runs(tmp_path) + with pytest.raises((ValueError, TypeError)): + check(paths, steps=steps) + + +def test_unexpected_step_and_symlink_are_rejected(tmp_path: Path) -> None: + paths = runs(tmp_path) + path = save(paths[0], 0, 9, artifact(0, 9)) + with pytest.raises(ValueError, match="coverage"): + check(paths) + path.unlink() + path = paths[0] / "rank0/step1.pt" + path.unlink() + path.symlink_to(paths[1] / "rank0/step1.pt") + with pytest.raises(ValueError, match="nonsymlink"): + check(paths) + + +def test_topology_changes_and_duplicate_coordinates_are_errors(tmp_path: Path) -> None: + paths = runs(tmp_path) + change_field(paths[0], "tp_rank", 0, rank=1, step=0) + with pytest.raises(ValueError, match="topology"): + check(paths) + change_field(paths[0], "tp_rank", 1, rank=1, step=0) + change_field(paths[0], "dp_rank", 1, rank=1, step=1) + with pytest.raises(ValueError, match="topology"): + check(paths) + + +def test_extreme_topology_rejected_before_constructing_rank_grid( + tmp_path: Path, +) -> None: + paths = runs(tmp_path) + change_field(paths[0], "tp_rank", 10**20, rank=1, step=0) + with pytest.raises(ValueError, match="topology"): + check(paths) + + +def test_empty_replicated_tensors_are_not_exercised_checks(tmp_path: Path) -> None: + paths = runs(tmp_path) + for directory in paths: + for rank in range(2): + for step in (0, 1): + change_field( + directory, + "parameters", + {"norm": torch.empty(0), "partition": torch.tensor([1.0])}, + rank, + step, + ) + with pytest.raises(ValueError, match="Empty replicated tensor"): + check(paths) + + +def test_finite_values_with_unrepresentable_difference_are_explicit( + tmp_path: Path, +) -> None: + paths = runs(tmp_path) + for directory, value in zip(paths, (1.7e308, -1.7e308)): + change_field( + directory, + "parameters", + { + "norm": torch.tensor([2.0, 3.0]), + "partition": torch.tensor([value], dtype=torch.float64), + }, + ) + report = check(paths) + assert not report.passed + assert report.model_absolute_difference_overflow + assert report.maximum_absolute_model_difference is None diff --git a/tools/runtime_pilot/__init__.py b/tools/runtime_pilot/__init__.py new file mode 100644 index 00000000..1bc76f00 --- /dev/null +++ b/tools/runtime_pilot/__init__.py @@ -0,0 +1 @@ +"""Operator-owned real distributed-runtime experiment utilities.""" diff --git a/tools/runtime_pilot/evaluate.py b/tools/runtime_pilot/evaluate.py new file mode 100644 index 00000000..569f8c48 --- /dev/null +++ b/tools/runtime_pilot/evaluate.py @@ -0,0 +1,527 @@ +"""Validate independently captured rank/step state without importing candidates. + +The operator owns these artifacts and must freeze source and artifact manifests +outside this evaluator. ``weights_only=True`` limits deserialization; it does +not establish artifact provenance or defend against a candidate rewriting files. +Optimizer containers are native dict/list/tuple values. Their key/element order, +scalar types and tensor bytes remain significant. Model tolerances apply only +to floating/complex parameter comparisons, including replicated parameters. +""" + +from __future__ import annotations + +import math +import re +import struct +from dataclasses import dataclass, field +from pathlib import Path +from typing import Literal, TypeAlias + +import torch + +Optimizer: TypeAlias = ( + None + | bool + | int + | float + | str + | torch.Tensor + | list["Optimizer"] + | tuple["Optimizer", ...] + | dict[int | str, "Optimizer"] +) +Identity: TypeAlias = tuple[int, int] +Category: TypeAlias = Literal["model", "optimizer", "rng", "replicated"] + + +@dataclass(frozen=True) +class Snapshot: + rank: int + tp_rank: int + dp_rank: int + step: int + parameters: dict[str, torch.Tensor] + replicated_names: tuple[str, ...] + optimizer: Optimizer + rng: dict[str, torch.Tensor] + + +@dataclass(frozen=True) +class Evaluation: + passed: bool + world_size: int + steps: tuple[int, ...] + candidate_artifacts: int + reference_artifacts: int + tensor_parallel_size: int + data_parallel_size: int + model_checks: int + optimizer_checks: int + rng_checks: int + replicated_checks: int + replicated_status: Literal["passed", "violated", "not_applicable"] + failed_checks: int + failed_paths: tuple[str, ...] + failed_paths_truncated: bool + maximum_absolute_model_difference: float | None + model_comparable_tensors: int + model_absolute_difference_overflow: bool + atol: float + rtol: float + + +@dataclass +class _Checks: + counts: dict[Category, int] = field( + default_factory=lambda: {"model": 0, "optimizer": 0, "rng": 0, "replicated": 0} + ) + failures: int = 0 + replicated_failures: int = 0 + paths: list[str] = field(default_factory=list) + maximum_model_difference: float | None = None + comparable_model_tensors: int = 0 + model_difference_overflow: bool = False + + def record(self, category: Category, path: str, passed: bool) -> None: + self.counts[category] += 1 + if not passed: + self.failures += 1 + if category == "replicated": + self.replicated_failures += 1 + if len(self.paths) < 100: + self.paths.append(path) + + +def _integer(value: object, path: str, *, positive: bool = False) -> int: + if isinstance(value, bool) or not isinstance(value, int): + raise TypeError(f"{path}: expected integer, excluding bool") + if value < (1 if positive else 0): + raise ValueError(f"{path}: integer is out of range") + return value + + +def _tolerance(value: object, path: str) -> float: + if isinstance(value, bool) or not isinstance(value, (int, float)): + raise TypeError(f"{path}: expected finite nonnegative number") + if not math.isfinite(value) or value < 0: + raise ValueError(f"{path}: expected finite nonnegative number") + return float(value) + + +def _mapping(value: object, path: str) -> dict[str, object]: + if not isinstance(value, dict): + raise TypeError(f"{path}: expected dict") + result: dict[str, object] = {} + for key, child in value.items(): + if not isinstance(key, str) or not key: + raise TypeError(f"{path}: keys must be nonempty strings") + result[key] = child + return result + + +def _tensor(value: object, path: str) -> torch.Tensor: + if not isinstance(value, torch.Tensor): + raise TypeError(f"{path}: expected tensor") + if value.layout != torch.strided or value.is_quantized: + raise TypeError(f"{path}: expected dense nonquantized tensor") + if value.device.type != "cpu": + raise ValueError(f"{path}: deserialized tensor is not on CPU") + if not bool(torch.isfinite(value).all()): + raise ValueError(f"{path}: tensor contains nonfinite values") + return value + + +def _tensor_mapping(value: object, path: str) -> dict[str, torch.Tensor]: + mapping = _mapping(value, path) + if not mapping: + raise ValueError(f"{path}: empty mapping does not establish coverage") + return {name: _tensor(item, f"{path}[{name!r}]") for name, item in mapping.items()} + + +def _optimizer(value: object, path: str, active: set[int]) -> Optimizer: + if isinstance(value, torch.Tensor): + return _tensor(value, path) + if value is None or isinstance(value, (bool, int, str)): + return value + if isinstance(value, float): + if not math.isfinite(value): + raise ValueError(f"{path}: nonfinite scalar") + return value + # Native container types are the artifact contract. Reject subclasses rather + # than silently normalizing their type (e.g. OrderedDict) into a plain dict. + if type(value) not in (dict, list, tuple): + raise TypeError(f"{path}: unsupported optimizer value {type(value).__name__}") + identity = id(value) + if identity in active: + raise ValueError(f"{path}: cyclic optimizer containers are unsupported") + active.add(identity) + try: + if isinstance(value, list): + return [ + _optimizer(child, f"{path}[{i}]", active) + for i, child in enumerate(value) + ] + if isinstance(value, tuple): + return tuple( + _optimizer(child, f"{path}[{i}]", active) + for i, child in enumerate(value) + ) + assert isinstance(value, dict) + result: dict[int | str, Optimizer] = {} + for key, child in value.items(): + if isinstance(key, bool) or not isinstance(key, (int, str)): + raise TypeError( + f"{path}: optimizer dict keys must be int or str, excluding bool" + ) + result[key] = _optimizer(child, f"{path}[{key!r}]", active) + return result + finally: + active.remove(identity) + + +def _snapshot(value: object, path: Path) -> Snapshot: + record = _mapping(value, str(path)) + required = { + "rank", + "tp_rank", + "dp_rank", + "step", + "parameters", + "replicated_names", + "optimizer", + "rng", + } + if set(record) != required: + raise ValueError( + f"{path}: snapshot fields differ; missing={sorted(required - set(record))}, extra={sorted(set(record) - required)}" + ) + parameters = _tensor_mapping(record["parameters"], f"{path}.parameters") + names = record["replicated_names"] + if not isinstance(names, list): + raise TypeError(f"{path}.replicated_names: expected list") + replicated: list[str] = [] + for name in names: + if not isinstance(name, str) or name not in parameters: + raise ValueError( + f"{path}.replicated_names: unknown parameter name {name!r}" + ) + if name in replicated: + raise ValueError(f"{path}.replicated_names: duplicate parameter {name!r}") + replicated.append(name) + rng = _tensor_mapping(record["rng"], f"{path}.rng") + for name, tensor in rng.items(): + if tensor.dtype != torch.uint8 or tensor.ndim != 1 or tensor.numel() == 0: + raise ValueError(f"{path}.rng[{name!r}]: expected nonempty 1-D byte tensor") + return Snapshot( + rank=_integer(record["rank"], f"{path}.rank"), + tp_rank=_integer(record["tp_rank"], f"{path}.tp_rank"), + dp_rank=_integer(record["dp_rank"], f"{path}.dp_rank"), + step=_integer(record["step"], f"{path}.step"), + parameters=parameters, + replicated_names=tuple(replicated), + optimizer=_optimizer(record["optimizer"], f"{path}.optimizer", set()), + rng=rng, + ) + + +def _load( + directory: Path, world_size: int, steps: tuple[int, ...] +) -> dict[Identity, Snapshot]: + if not directory.is_dir() or directory.is_symlink(): + raise ValueError(f"{directory}: expected existing nonsymlink directory") + snapshots: dict[Identity, Snapshot] = {} + for path in sorted(directory.glob("rank*/step*.pt")): + if path.is_symlink() or path.parent.is_symlink() or not path.is_file(): + raise ValueError(f"{path}: expected regular nonsymlink artifact") + rank_match = re.fullmatch(r"rank([0-9]+)", path.parent.name) + step_match = re.fullmatch(r"step([0-9]+)\.pt", path.name) + if rank_match is None or step_match is None: + raise ValueError(f"{path}: malformed rank/step filename") + raw: object = torch.load(path, map_location="cpu", weights_only=True) + state = _snapshot(raw, path) + identity = (state.rank, state.step) + if identity in snapshots: + raise ValueError(f"{path}: duplicate rank/step identity {identity}") + if identity != (int(rank_match[1]), int(step_match[1])): + raise ValueError(f"{path}: path disagrees with recorded rank/step") + snapshots[identity] = state + expected = {(rank, step) for rank in range(world_size) for step in steps} + if set(snapshots) != expected: + raise ValueError( + f"{directory}: incomplete or unexpected rank/step coverage; missing={sorted(expected - set(snapshots))}, extra={sorted(set(snapshots) - expected)}" + ) + return snapshots + + +def _topology( + states: dict[Identity, Snapshot], world_size: int, steps: tuple[int, ...] +) -> tuple[int, int]: + first = { + rank: (states[rank, steps[0]].dp_rank, states[rank, steps[0]].tp_rank) + for rank in range(world_size) + } + dp_size = max(dp for dp, _ in first.values()) + 1 + tp_size = max(tp for _, tp in first.values()) + 1 + if dp_size * tp_size != world_size: + raise ValueError("Incomplete or duplicated data/tensor parallel topology") + expected = {(dp, tp) for dp in range(dp_size) for tp in range(tp_size)} + if len(expected) != world_size or set(first.values()) != expected: + raise ValueError("Incomplete or duplicated data/tensor parallel topology") + for (rank, _), state in states.items(): + if (state.dp_rank, state.tp_rank) != first[rank]: + raise ValueError("Rank topology changed across observed steps") + for step in steps: + for dp in range(dp_size): + group = [ + state + for state in states.values() + if state.step == step and state.dp_rank == dp + ] + names = group[0].replicated_names + if any(state.replicated_names != names for state in group): + raise ValueError("Replicated parameter names differ within a TP group") + if tp_size > 1 and not names: + raise ValueError( + "Empty replicated parameter coverage for a multi-rank TP group" + ) + if tp_size > 1 and any( + state.parameters[name].numel() == 0 for state in group for name in names + ): + raise ValueError("Empty replicated tensor does not establish coverage") + return tp_size, dp_size + + +def _same_tensor_bytes(left: torch.Tensor, right: torch.Tensor) -> bool: + return torch.equal( + left.contiguous().reshape(-1).view(torch.uint8), + right.contiguous().reshape(-1).view(torch.uint8), + ) + + +def _maximum_difference(left: torch.Tensor, right: torch.Tensor) -> float: + if left.numel() == 0: + return 0.0 + if left.is_floating_point() or left.is_complex(): + dtype = torch.complex128 if left.is_complex() else torch.float64 + return float((left.to(dtype) - right.to(dtype)).abs().max().item()) + # Python integer subtraction avoids int64 overflow and float64 rounding + # that could otherwise report zero for unequal large integer parameters. + left_values: object = left.reshape(-1).tolist() + right_values: object = right.reshape(-1).tolist() + assert isinstance(left_values, list) and isinstance(right_values, list) + maximum = 0 + for left_item, right_item in zip(left_values, right_values): + assert isinstance(left_item, (bool, int)) and isinstance( + right_item, (bool, int) + ) + maximum = max(maximum, abs(int(left_item) - int(right_item))) + return float(maximum) + + +def _compare_tensor( + candidate: torch.Tensor, + reference: torch.Tensor, + path: str, + category: Category, + checks: _Checks, + atol: float, + rtol: float, +) -> None: + comparable = ( + candidate.dtype == reference.dtype and candidate.shape == reference.shape + ) + checks.record(category, path + ".dtype_shape", comparable) + if not comparable: + return + if category == "model": + checks.comparable_model_tensors += 1 + difference = _maximum_difference(candidate, reference) + if math.isfinite(difference): + previous = checks.maximum_model_difference + checks.maximum_model_difference = ( + difference if previous is None else max(previous, difference) + ) + else: + checks.model_difference_overflow = True + if (atol or rtol) and (candidate.is_floating_point() or candidate.is_complex()): + same = bool(torch.allclose(candidate, reference, atol=atol, rtol=rtol)) + else: + same = _same_tensor_bytes(candidate, reference) + checks.record(category, path + ".value", same) + + +def _compare_optimizer( + candidate: Optimizer, reference: Optimizer, path: str, checks: _Checks +) -> None: + checks.record("optimizer", path + ".type", type(candidate) is type(reference)) + if type(candidate) is not type(reference): + return + if isinstance(candidate, torch.Tensor): + assert isinstance(reference, torch.Tensor) + _compare_tensor(candidate, reference, path, "optimizer", checks, 0.0, 0.0) + elif isinstance(candidate, dict): + assert isinstance(reference, dict) + candidate_keys = [(type(key), key) for key in candidate] + reference_keys = [(type(key), key) for key in reference] + same = candidate_keys == reference_keys + checks.record("optimizer", path + ".ordered_keys", same) + if same: + for key in reference: + _compare_optimizer( + candidate[key], reference[key], f"{path}[{key!r}]", checks + ) + elif isinstance(candidate, (list, tuple)): + assert isinstance(reference, (list, tuple)) + checks.record("optimizer", path + ".length", len(candidate) == len(reference)) + for index, (left, right) in enumerate(zip(candidate, reference)): + _compare_optimizer(left, right, f"{path}[{index}]", checks) + elif isinstance(candidate, float): + assert isinstance(reference, float) + checks.record( + "optimizer", + path + ".value", + struct.pack("!d", candidate) == struct.pack("!d", reference), + ) + else: + checks.record("optimizer", path + ".value", candidate == reference) + + +def evaluate( + candidate_directory: Path, + reference_directory: Path, + world_size: int, + steps: list[int], + atol: float = 0.0, + rtol: float = 0.0, +) -> Evaluation: + """Compare complete coverage; malformed/missing evidence raises an error. + + Replication checks use each step's TP ranks within one DP group. A TP1 + topology reports ``not_applicable`` even if several DP replicas exist. + References must independently satisfy the same replication invariant. + """ + world_size = _integer(world_size, "world_size", positive=True) + if not isinstance(steps, list) or not steps: + raise ValueError("steps must be a nonempty list") + expected_steps = tuple(_integer(step, "step") for step in steps) + if tuple(sorted(set(expected_steps))) != expected_steps: + raise ValueError("steps must be strictly increasing and unique") + atol = _tolerance(atol, "atol") + rtol = _tolerance(rtol, "rtol") + candidate = _load(candidate_directory, world_size, expected_steps) + reference = _load(reference_directory, world_size, expected_steps) + topology = _topology(candidate, world_size, expected_steps) + if topology != _topology(reference, world_size, expected_steps): + raise ValueError("Candidate and reference parallel topology differ") + checks = _Checks() + for identity in sorted(reference): + left, right = candidate[identity], reference[identity] + path = f"rank{identity[0]}/step{identity[1]}" + if (left.dp_rank, left.tp_rank) != (right.dp_rank, right.tp_rank): + raise ValueError(f"{path}: candidate/reference rank topology differs") + if left.replicated_names != right.replicated_names: + raise ValueError(f"{path}: candidate/reference replicated metadata differs") + checks.record( + "model", + path + ".parameter_names", + list(left.parameters) == list(right.parameters), + ) + for name in right.parameters: + if name in left.parameters: + _compare_tensor( + left.parameters[name], + right.parameters[name], + f"{path}.parameters[{name!r}]", + "model", + checks, + atol, + rtol, + ) + _compare_optimizer(left.optimizer, right.optimizer, path + ".optimizer", checks) + checks.record("rng", path + ".rng_names", list(left.rng) == list(right.rng)) + for name in right.rng: + if name in left.rng: + _compare_tensor( + left.rng[name], + right.rng[name], + f"{path}.rng[{name!r}]", + "rng", + checks, + 0.0, + 0.0, + ) + tp_size, dp_size = topology + if tp_size > 1: + for step in expected_steps: + for dp in range(dp_size): + group = sorted( + ( + state + for state in candidate.values() + if state.step == step and state.dp_rank == dp + ), + key=lambda state: state.tp_rank, + ) + base = group[0] + for state in group[1:]: + for name in base.replicated_names: + path = ( + f"step{step}/dp{dp}/tp{state.tp_rank}.replicated[{name!r}]" + ) + _compare_tensor( + state.parameters[name], + base.parameters[name], + path, + "replicated", + checks, + atol, + rtol, + ) + reference_base = reference[base.rank, step].parameters[name] + reference_other = reference[state.rank, step].parameters[name] + reference_checks = _Checks() + _compare_tensor( + reference_other, + reference_base, + path, + "replicated", + reference_checks, + atol, + rtol, + ) + if reference_checks.failures: + raise ValueError( + f"{path}: known-correct reference violates replication" + ) + assert checks.counts["replicated"] > 0 + status: Literal["passed", "violated", "not_applicable"] = ( + "not_applicable" + if tp_size == 1 + else "violated" if checks.replicated_failures else "passed" + ) + return Evaluation( + passed=checks.failures == 0, + world_size=world_size, + steps=expected_steps, + candidate_artifacts=len(candidate), + reference_artifacts=len(reference), + tensor_parallel_size=tp_size, + data_parallel_size=dp_size, + model_checks=checks.counts["model"], + optimizer_checks=checks.counts["optimizer"], + rng_checks=checks.counts["rng"], + replicated_checks=checks.counts["replicated"], + replicated_status=status, + failed_checks=checks.failures, + failed_paths=tuple(checks.paths), + failed_paths_truncated=checks.failures > len(checks.paths), + maximum_absolute_model_difference=( + None + if checks.model_difference_overflow + else checks.maximum_model_difference + ), + model_comparable_tensors=checks.comparable_model_tensors, + model_absolute_difference_overflow=checks.model_difference_overflow, + atol=atol, + rtol=rtol, + ) From 374ddd07144c67b6c9a3acb84dc9a2546f91e3bf Mon Sep 17 00:00:00 2001 From: Yuxuan Jiang Date: Mon, 2 Nov 2026 03:52:39 +0800 Subject: [PATCH 03/13] Keep installed TrainCheck out of ordinary control-worker imports --- tests/test_runtime_pilot_access.py | 160 +++++++++++++++++++++++++++++ tools/runtime_pilot/access.py | 64 ++++++++++++ 2 files changed, 224 insertions(+) create mode 100644 tests/test_runtime_pilot_access.py create mode 100644 tools/runtime_pilot/access.py diff --git a/tests/test_runtime_pilot_access.py b/tests/test_runtime_pilot_access.py new file mode 100644 index 00000000..f5a465e7 --- /dev/null +++ b/tests/test_runtime_pilot_access.py @@ -0,0 +1,160 @@ +"""Exercise import availability in fresh processes without changing pytest state.""" + +from __future__ import annotations + +import json +import subprocess +import sys +from pathlib import Path + +import pytest + +REPO = Path(__file__).resolve().parents[1] + + +def worker(tmp_path: Path, statements: str) -> dict[str, object]: + package = tmp_path / "traincheck" + package.mkdir() + (package / "__init__.py").write_text("IMPORTED = True\n") + (package / "child.py").write_text("IMPORTED = True\n") + (tmp_path / "trainchecker.py").write_text("VALUE = 17\n") + source = ( + "import importlib, json, sys\n" + "from types import ModuleType\n" + "sys.path.insert(0, sys.argv[1])\n" + "from tools.runtime_pilot.access import install_traincheck_blocker\n" + "sys.path.insert(0, sys.argv[2])\n" + statements + ) + completed = subprocess.run( + [sys.executable, "-I", "-c", source, str(REPO), str(tmp_path)], + check=True, + capture_output=True, + text=True, + timeout=20, + ) + raw: object = json.loads(completed.stdout) + assert isinstance(raw, dict) + result: dict[str, object] = {} + for name, value in raw.items(): + assert isinstance(name, str) + result[name] = value + return result + + +def test_installed_package_blocked_then_owned_finder_cleanup(tmp_path: Path) -> None: + result = worker( + tmp_path, + "original = tuple(sys.meta_path)\n" + "finder = install_traincheck_blocker()\n" + "try:\n" + " importlib.import_module('traincheck')\n" + "except ModuleNotFoundError as error:\n" + " missing = error.name\n" + "else:\n" + " raise AssertionError('Installed TrainCheck was not blocked')\n" + "assert 'traincheck' not in sys.modules\n" + "sys.meta_path.remove(finder)\n" + "assert tuple(sys.meta_path) == original\n" + "package = importlib.import_module('traincheck')\n" + "print(json.dumps({'missing': missing, 'imported_after_cleanup': package.IMPORTED}))\n", + ) + assert result == {"missing": "traincheck", "imported_after_cleanup": True} + + +@pytest.mark.parametrize("name", ["traincheck.child", "traincheck.instrumentor.tracer"]) +def test_submodule_namespace_blocked(tmp_path: Path, name: str) -> None: + result = worker( + tmp_path, + "finder = install_traincheck_blocker()\n" + f"name = {name!r}\n" + "try:\n" + " finder.find_spec(name, [])\n" + "except ModuleNotFoundError as error:\n" + " assert error.name == name\n" + "else:\n" + " raise AssertionError('Submodule was not blocked')\n" + "try:\n" + " importlib.import_module(name)\n" + "except ModuleNotFoundError as error:\n" + " assert error.name == 'traincheck'\n" + "else:\n" + " raise AssertionError('Normal submodule import was not blocked')\n" + "sys.meta_path.remove(finder)\n" + "print(json.dumps({'blocked': name}))\n", + ) + assert result == {"blocked": name} + + +def test_unrelated_imports_and_similar_prefix_unchanged(tmp_path: Path) -> None: + result = worker( + tmp_path, + "finder = install_traincheck_blocker()\n" + "module = importlib.import_module('trainchecker')\n" + "decimal = importlib.import_module('decimal')\n" + "assert finder.find_spec('other.traincheck', None) is None\n" + "sys.meta_path.remove(finder)\n" + "print(json.dumps({'prefix': module.VALUE, 'decimal': str(decimal.Decimal('1.25'))}))\n", + ) + assert result == {"prefix": 17, "decimal": "1.25"} + + +@pytest.mark.parametrize("name", ["traincheck", "traincheck.child"]) +@pytest.mark.parametrize("placeholder", [False, True]) +def test_preloaded_namespace_rejected_without_removal( + tmp_path: Path, name: str, placeholder: bool +) -> None: + result = worker( + tmp_path, + f"name = {name!r}\n" + f"existing = None if {placeholder!r} else ModuleType(name)\n" + "sys.modules[name] = existing\n" + "original = tuple(sys.meta_path)\n" + "try:\n" + " install_traincheck_blocker()\n" + "except RuntimeError as error:\n" + " assert name in str(error)\n" + "else:\n" + " raise AssertionError('Preloaded namespace was accepted')\n" + "assert name in sys.modules and sys.modules[name] is existing\n" + "assert tuple(sys.meta_path) == original\n" + "print(json.dumps({'preserved': name}))\n", + ) + assert result == {"preserved": name} + + +def test_repeat_install_restores_precedence_without_duplicate(tmp_path: Path) -> None: + result = worker( + tmp_path, + "original = tuple(sys.meta_path)\n" + "finder = install_traincheck_blocker()\n" + "assert install_traincheck_blocker() is finder\n" + "sys.meta_path.remove(finder)\n" + "sys.meta_path.append(finder)\n" + "assert install_traincheck_blocker() is finder\n" + "assert sys.meta_path[0] is finder\n" + "assert sum(item is finder for item in sys.meta_path) == 1\n" + "sys.meta_path.remove(finder)\n" + "assert tuple(sys.meta_path) == original\n" + "print(json.dumps({'idempotent': True}))\n", + ) + assert result == {"idempotent": True} + + +def test_repeat_install_does_not_hide_later_namespace_injection(tmp_path: Path) -> None: + result = worker( + tmp_path, + "finder = install_traincheck_blocker()\n" + "existing = ModuleType('traincheck')\n" + "sys.modules['traincheck'] = existing\n" + "try:\n" + " install_traincheck_blocker()\n" + "except RuntimeError:\n" + " pass\n" + "else:\n" + " raise AssertionError('Later namespace injection was hidden')\n" + "assert sys.modules['traincheck'] is existing\n" + "assert sum(item is finder for item in sys.meta_path) == 1\n" + "sys.meta_path.remove(finder)\n" + "print(json.dumps({'rejected': True}))\n", + ) + assert result == {"rejected": True} diff --git a/tools/runtime_pilot/access.py b/tools/runtime_pilot/access.py new file mode 100644 index 00000000..90e9115b --- /dev/null +++ b/tools/runtime_pilot/access.py @@ -0,0 +1,64 @@ +"""Process-local TrainCheck import boundary for a control worker. + +Install before dependency bootstrap, while the worker is still single-threaded. +This finder blocks normal imports, including an otherwise installed TrainCheck; +it is not an OS sandbox against code that edits import machinery, injects +``sys.modules`` entries, or executes source through a custom loader. +""" + +from __future__ import annotations + +import sys +from collections.abc import Sequence +from importlib.abc import MetaPathFinder +from importlib.machinery import ModuleSpec +from types import ModuleType + + +def _is_traincheck(fullname: str) -> bool: + return fullname == "traincheck" or fullname.startswith("traincheck.") + + +class TrainCheckImportBlocker(MetaPathFinder): + """Reject only TrainCheck's exact package namespace through normal imports.""" + + def find_spec( + self, + fullname: str, + path: Sequence[str] | None, + target: ModuleType | None = None, + ) -> ModuleSpec | None: + if _is_traincheck(fullname): + raise ModuleNotFoundError( + f"TrainCheck is unavailable in this control worker: {fullname}", + name=fullname, + ) + return None + + +def install_traincheck_blocker() -> TrainCheckImportBlocker: + """Install first in the import search order, or reuse the owned finder. + + Existing TrainCheck namespace entries, even ``None`` placeholders, are an + error: removing them would conceal earlier imports and their side effects. + Reinstallation validates that invariant again and restores the same finder + to first position if another component has moved it. + """ + loaded = sorted(name for name in tuple(sys.modules) if _is_traincheck(name)) + if loaded: + raise RuntimeError( + "Cannot install TrainCheck import blocker after namespace entries " + f"already exist: {', '.join(loaded)}" + ) + for index, finder in enumerate(sys.meta_path): + # Only reuse this exact implementation, not a subclass with different + # behavior. Do not remove or replace unrelated import finders. + if type(finder) is TrainCheckImportBlocker: + assert isinstance(finder, TrainCheckImportBlocker) + if index: + del sys.meta_path[index] + sys.meta_path.insert(0, finder) + return finder + blocker = TrainCheckImportBlocker() + sys.meta_path.insert(0, blocker) + return blocker From 511f3106b2b3df97c4e20c895b0d7f9b10ac31a4 Mon Sep 17 00:00:00 2001 From: Yuxuan Jiang Date: Mon, 2 Nov 2026 04:08:56 +0800 Subject: [PATCH 04/13] Capture native cross-rank clipping evidence for runtime qualification --- .../real-rank-runtime-pilot-20261102.md | 47 ++- tests/test_runtime_pilot_clipping.py | 337 ++++++++++++++++++ tools/runtime_pilot/clipping.py | 236 ++++++++++++ 3 files changed, 615 insertions(+), 5 deletions(-) create mode 100644 tests/test_runtime_pilot_clipping.py create mode 100644 tools/runtime_pilot/clipping.py diff --git a/docs/measurements/real-rank-runtime-pilot-20261102.md b/docs/measurements/real-rank-runtime-pilot-20261102.md index bccf2b49..0ad55544 100644 --- a/docs/measurements/real-rank-runtime-pilot-20261102.md +++ b/docs/measurements/real-rank-runtime-pilot-20261102.md @@ -1,10 +1,47 @@ # Real cross-rank runtime case: agent-efficiency protocol -**Status: in progress, 2026-11-02.** Case provenance and the experiment design -are being established. Ring18 qualification and agent trials are pending. This -document reports no new training result, repair-success rate, or TrainCheck -efficiency improvement. It is operator documentation and must not be included -in solver packets. +**Status: in progress, 2026-11-02.** The native TP2 case reproduces on Ring18; +qualification controls and the agent service are being validated. Agent trials +have not started, so no debugging-efficiency improvement is claimed. This is +operator documentation and must not be included in solver packets. + +The first two-update run had finite losses of 10.39899 and 10.62776. Among 27 +replicated trainable parameters, cross-rank mismatches increased from zero before +the first update to 6 after update 1 and 18 after update 2. The corrected control +passed 630 model, 774 optimizer, 42 RNG and 162 replicated-parameter checks against +itself. Comparing the vulnerable run against that control failed 140 checks, +with maximum absolute model difference 0.00067138671875. These are qualification +results, not measurements of agent performance. + +Removing inherited research annotations and isolating ordinary imports preserved +all archived state exactly: each vulnerable/corrected comparison covered six +rank/step snapshots, 384 tensors, 144,752,448 tensor bytes, 378 scalar values and +606 ordered dictionary keys, with no differences. Recorded CPU/CUDA/Megatron RNG +and optimizer values are included; unrecorded state and live object aliasing are +outside that comparison. Evidence: `private-state-parity.json` in the local +experiment directory below. + +Both ordinary eight-update runs complete with finite losses and zero skipped or +NaN iterations. Final FP16/BF16 losses are 10.25196/10.25426. This short difference +does not establish a quality regression. The neutral task will present measured +curves without asserting one. An issue-named installation-path prefix in exported +text is replaced with `/runtime/python-env`, identically for both arms; raw logs, +exact mappings and original/public hashes remain operator evidence. Error/warning +wording, source filenames, line numbers and numeric values are preserved. + +The private clipping observer records actual FP32 gradient selection and values +around native clipping, using the pipeline grid's slice-parallel rank. It is never +automatically provided to solvers. Its 26 CPU regressions cover call/exception +preservation, RNG and model state, flattened gradient views, real grid rank APIs, +missing/malformed metadata and duplicate evidence. Together with evaluator and +control-import tests, **91 tests pass**. Pinned hooks and explicit strict mypy +pass for the observer and tests; the repository mypy hook alone skips tools. + +Setup failures are retained, including one incorrect private grid API and native +shutdown crashes after successful updates. The worker now explicitly releases +distributed process groups after application exit. A successful rerun alone does +not prove elimination of an intermittent shutdown failure; later receipts and +failure counts remain part of the results. The development base is `codex/checkpoint-continuation-costs`, [PR #170](https://github.com/OrderLab/TrainCheck/pull/170), at diff --git a/tests/test_runtime_pilot_clipping.py b/tests/test_runtime_pilot_clipping.py new file mode 100644 index 00000000..3c24bd4e --- /dev/null +++ b/tests/test_runtime_pilot_clipping.py @@ -0,0 +1,337 @@ +"""CPU-only contracts for the operator's private clipping qualification probe.""" + +from __future__ import annotations + +import sys +from dataclasses import dataclass +from pathlib import Path +from types import ModuleType + +import pytest +import torch +import torch.nn as nn + +from tools.runtime_pilot import clipping as audit + + +class Model(nn.Module): + def __init__(self) -> None: + super().__init__() + self.rep_a = nn.Parameter(torch.ones(2, 2)) + self.rep_b = nn.Parameter(torch.ones(2, 2)) + self.shard = nn.Parameter(torch.ones(2, 2)) + self.frozen = nn.Parameter(torch.ones(2, 2), requires_grad=False) + setattr(self.shard, "tensor_model_parallel", True) + + +@dataclass +class Optimizer: + bf16_groups: list[list[torch.Tensor]] + fp32_groups_gradients: list[list[torch.Tensor]] + + +class MPU: + @staticmethod + def get_slice_parallel_rank() -> int: + return 1 + + @staticmethod + def get_model_parallel_rank() -> int: + # The native grid combines PP and TP here. Do not label this as TP. + return 5 + + @staticmethod + def get_data_parallel_rank() -> int: + return 0 + + +class Engine: + def __init__(self, native: ModuleType) -> None: + self.module = Model() + self.optimizer = Optimizer( + [ + [ + self.module.rep_a, + self.module.rep_b, + self.module.shard, + self.module.frozen, + ] + ], + [[torch.full((4,), 2.0) for _ in range(4)]], + ) + self.global_steps = 0 + self.native = native + self.mpu = MPU() + gradients = self.optimizer.fp32_groups_gradients[0] + self.clip_kwargs: dict[str, object] = { + "input_tensors": [gradients[0], gradients[2]], + "max_norm": 1.0, + "global_norm": 4.0, + "mpu": self.mpu, + "eps": 0.5, + } + self.step_calls: list[tuple[tuple[object, ...], dict[str, object]]] = [] + self.clip_return = object() + self.step_return = object() + self.observed_clip_return: object = None + + def _take_model_step(self, *args: object, **kwargs: object) -> object: + # This method stands in for an already installed state observer. + self.step_calls.append((args, kwargs)) + clip: object = getattr(self.native, "clip_tensors_by_global_norm") + assert callable(clip) + self.observed_clip_return = clip(**self.clip_kwargs) + self.global_steps += 1 + return self.step_return + + +@dataclass +class Harness: + engine: Engine + native: ModuleType + clip_calls: list[dict[str, object]] + output: Path + + def snapshot(self, step: int = 1) -> dict[object, object]: + value: object = torch.load( + self.output / "rank1" / f"step{step}.pt", weights_only=True + ) + assert isinstance(value, dict) + return value + + +@pytest.fixture +def harness(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> Harness: + native = ModuleType("deepspeed.runtime.bf16_optimizer") + engine = Engine(native) + clip_calls: list[dict[str, object]] = [] + + def native_clip(*args: object, **kwargs: object) -> object: + assert not args + clip_calls.append(kwargs) + tensors = kwargs["input_tensors"] + assert isinstance(tensors, list) + for tensor in tensors: + assert isinstance(tensor, torch.Tensor) + tensor.mul_(1.0 / 4.5) + return engine.clip_return + + engine_module = ModuleType("deepspeed.runtime.engine") + setattr(engine_module, "DeepSpeedEngine", Engine) + setattr(native, "clip_tensors_by_global_norm", native_clip) + monkeypatch.setitem(sys.modules, native.__name__, native) + monkeypatch.setitem(sys.modules, engine_module.__name__, engine_module) + monkeypatch.setattr(audit, "_installed", False) + monkeypatch.setattr(torch.distributed, "get_rank", lambda: 1) + monkeypatch.setenv("RUNTIME_CLIPPING_OUTPUT", str(tmp_path)) + # The fixture restores the class method after install replaces it. + monkeypatch.setattr(Engine, "_take_model_step", Engine._take_model_step) + return Harness(engine, native, clip_calls, tmp_path) + + +def tensor_map(value: object) -> dict[str, torch.Tensor]: + assert isinstance(value, dict) + result: dict[str, torch.Tensor] = {} + for name, tensor in value.items(): + assert isinstance(name, str) + assert isinstance(tensor, torch.Tensor) + result[name] = tensor + return result + + +def test_native_call_once_and_correct_independent_snapshots(harness: Harness) -> None: + audit.install() + marker = object() + rng_before = torch.get_rng_state().clone() + parameters_before = { + name: parameter.detach().clone() + for name, parameter in harness.engine.module.named_parameters() + } + result = harness.engine._take_model_step(marker, future_option=marker) + assert torch.equal(torch.get_rng_state(), rng_before) + assert all( + torch.equal(parameter, parameters_before[name]) + for name, parameter in harness.engine.module.named_parameters() + ) + assert result is harness.engine.step_return + assert harness.engine.observed_clip_return is harness.engine.clip_return + assert len(harness.engine.step_calls) == len(harness.clip_calls) == 1 + args, kwargs = harness.engine.step_calls[0] + assert len(args) == 1 and args[0] is marker + assert kwargs["future_option"] is marker + for name, value in harness.engine.clip_kwargs.items(): + assert harness.clip_calls[0][name] is value + snapshot = harness.snapshot() + assert snapshot["rank"] == snapshot["tp_rank"] == snapshot["step"] == 1 + assert snapshot["dp_rank"] == 0 + assert snapshot["max_norm"] == 1.0 + assert snapshot["global_norm"] == 4.0 + assert snapshot["selected"] == {"rep_a": True, "rep_b": False} + before = tensor_map(snapshot["before"]) + after = tensor_map(snapshot["after"]) + assert list(before) == list(after) == ["rep_a", "rep_b"] + assert torch.equal(before["rep_a"], torch.full((4,), 2.0)) + assert torch.equal(before["rep_b"], before["rep_a"]) + assert torch.equal(after["rep_a"], before["rep_a"] * (1.0 / 4.5)) + assert torch.equal(after["rep_b"], before["rep_b"]) + for gradient in harness.engine.optimizer.fp32_groups_gradients[0]: + gradient.zero_() + persisted = tensor_map(harness.snapshot()["after"]) + assert torch.equal(persisted["rep_a"], after["rep_a"]) + assert audit._current_engine.get() is None + + +def test_successive_steps_have_distinct_artifacts(harness: Harness) -> None: + audit.install() + harness.engine._take_model_step() + harness.engine._take_model_step() + assert harness.snapshot(1)["step"] == 1 + assert harness.snapshot(2)["step"] == 2 + assert len(harness.clip_calls) == 2 + + +def test_native_grid_tensor_slice_rank_is_used(harness: Harness) -> None: + assert not hasattr(harness.engine.mpu, "get_tensor_model_parallel_rank") + assert harness.engine.mpu.get_model_parallel_rank() != 1 + audit.install() + harness.engine._take_model_step() + assert harness.snapshot()["tp_rank"] == 1 + + +def test_original_error_is_unchanged_and_context_restored(harness: Harness) -> None: + error = RuntimeError("native clipping failed") + calls: list[None] = [] + + def failing_clip(*args: object, **kwargs: object) -> object: + calls.append(None) + raise error + + setattr(harness.native, "clip_tensors_by_global_norm", failing_clip) + audit.install() + outer = object() + token = audit._current_engine.set(outer) + try: + with pytest.raises(RuntimeError) as caught: + harness.engine._take_model_step() + assert caught.value is error + assert calls == [None] + assert audit._current_engine.get() is outer + assert not list(harness.output.rglob("*.pt")) + finally: + audit._current_engine.reset(token) + + +@pytest.mark.parametrize("field", ["input_tensors", "max_norm", "global_norm", "mpu"]) +def test_missing_required_argument_is_not_defaulted( + harness: Harness, field: str +) -> None: + del harness.engine.clip_kwargs[field] + audit.install() + with pytest.raises(TypeError, match="Missing required clipping argument"): + harness.engine._take_model_step() + assert not harness.clip_calls + assert audit._current_engine.get() is None + + +@pytest.mark.parametrize("flag", [1, None, "false"]) +def test_malformed_partition_flag_rejected(harness: Harness, flag: object) -> None: + setattr(harness.engine.module.rep_a, "tensor_model_parallel", flag) + audit.install() + with pytest.raises(TypeError, match="tensor_model_parallel"): + harness.engine._take_model_step() + assert not harness.clip_calls + + +def test_missing_replicated_gradient_rejected(harness: Harness) -> None: + harness.engine.optimizer.bf16_groups[0].pop(1) + harness.engine.optimizer.fp32_groups_gradients[0].pop(1) + audit.install() + with pytest.raises(ValueError, match="Missing replicated gradients"): + harness.engine._take_model_step() + assert not harness.clip_calls + + +def test_missing_optimizer_field_rejected(harness: Harness) -> None: + delattr(harness.engine.optimizer, "fp32_groups_gradients") + audit.install() + with pytest.raises(AttributeError, match="fp32_groups_gradients"): + harness.engine._take_model_step() + assert not harness.clip_calls + + +def test_mismatched_gradient_group_rejected(harness: Harness) -> None: + harness.engine.optimizer.fp32_groups_gradients[0].pop() + audit.install() + with pytest.raises(ValueError, match="lengths differ"): + harness.engine._take_model_step() + assert not harness.clip_calls + + +@pytest.mark.parametrize("value", [True, None, float("inf"), float("nan"), 0.0]) +def test_malformed_norm_rejected(harness: Harness, value: object) -> None: + harness.engine.clip_kwargs["global_norm"] = value + audit.install() + with pytest.raises((TypeError, ValueError), match="global_norm"): + harness.engine._take_model_step() + assert not harness.clip_calls + + +def test_duplicate_identity_rejected(harness: Harness) -> None: + optimizer = harness.engine.optimizer + optimizer.bf16_groups[0].append(optimizer.bf16_groups[0][0]) + optimizer.fp32_groups_gradients[0].append(torch.ones(4)) + audit.install() + with pytest.raises(ValueError, match="Duplicate optimizer"): + harness.engine._take_model_step() + assert not harness.clip_calls + + +@pytest.mark.parametrize("dtype", [torch.float16, torch.float64]) +def test_non_fp32_gradient_rejected(harness: Harness, dtype: torch.dtype) -> None: + harness.engine.optimizer.fp32_groups_gradients[0][1] = torch.ones(4, dtype=dtype) + audit.install() + with pytest.raises(TypeError, match="FP32"): + harness.engine._take_model_step() + assert not harness.clip_calls + + +def test_selected_view_is_not_confused_with_selected_object(harness: Harness) -> None: + gradient = harness.engine.optimizer.fp32_groups_gradients[0][0] + harness.engine.clip_kwargs["input_tensors"] = [gradient.view_as(gradient)] + audit.install() + with pytest.raises(ValueError, match="Selected gradient object"): + harness.engine._take_model_step() + assert not harness.clip_calls + + +def test_empty_selection_records_omissions(harness: Harness) -> None: + harness.engine.clip_kwargs["input_tensors"] = [] + audit.install() + harness.engine._take_model_step() + snapshot = harness.snapshot() + assert snapshot["selected"] == {"rep_a": False, "rep_b": False} + before = tensor_map(snapshot["before"]) + after = tensor_map(snapshot["after"]) + assert all(torch.equal(before[name], after[name]) for name in before) + + +def test_duplicate_artifact_rejected_before_clipping(harness: Harness) -> None: + audit.install() + harness.engine._take_model_step() + original = (harness.output / "rank1" / "step1.pt").read_bytes() + harness.engine.global_steps = 0 + with pytest.raises(FileExistsError): + harness.engine._take_model_step() + assert len(harness.clip_calls) == 1 + assert (harness.output / "rank1" / "step1.pt").read_bytes() == original + + +def test_outside_step_and_repeat_install_rejected(harness: Harness) -> None: + audit.install() + clip: object = getattr(harness.native, "clip_tensors_by_global_norm") + assert callable(clip) + with pytest.raises(RuntimeError, match="outside a model step"): + clip(**harness.engine.clip_kwargs) + with pytest.raises(RuntimeError, match="already installed"): + audit.install() + assert not harness.clip_calls diff --git a/tools/runtime_pilot/clipping.py b/tools/runtime_pilot/clipping.py new file mode 100644 index 00000000..bc1409ae --- /dev/null +++ b/tools/runtime_pilot/clipping.py @@ -0,0 +1,236 @@ +"""Private synchronous qualification of native BF16 gradient clipping. + +Install after audit_runtime so its model-step observer stays in the call chain. +This probe is intentionally limited to the native keyword-based clipping call; +it is not part of either solver arm or its performance measurements. +""" + +from __future__ import annotations + +import math +import os +from contextvars import ContextVar +from importlib import import_module +from pathlib import Path +from typing import TypedDict + +import torch +import torch.nn as nn + + +class ClippingSnapshot(TypedDict): + rank: int + tp_rank: int + dp_rank: int + step: int + max_norm: int | float + global_norm: int | float + selected: dict[str, bool] + before: dict[str, torch.Tensor] + after: dict[str, torch.Tensor] + + +_current_engine: ContextVar[object | None] = ContextVar( + "clipping_audit_engine", default=None +) +_installed = False + + +def _nonnegative_integer(value: object, field: str) -> int: + if not isinstance(value, int) or isinstance(value, bool): + raise TypeError(f"{field} must be an integer") + if value < 0: + raise ValueError(f"{field} must be nonnegative") + return value + + +def _positive_number(value: object, field: str) -> int | float: + if type(value) not in (int, float): + raise TypeError(f"{field} must be an int or float") + assert isinstance(value, (int, float)) + if not math.isfinite(value) or value <= 0: + raise ValueError(f"{field} must be finite and positive") + return value + + +def _tensor(value: object, field: str) -> torch.Tensor: + if not isinstance(value, torch.Tensor): + raise TypeError(f"{field} must be a tensor") + if value.layout != torch.strided or value.dtype != torch.float32: + raise TypeError(f"{field} must be a dense FP32 gradient") + if value.numel() == 0 or not bool(torch.isfinite(value).all()): + raise ValueError(f"{field} must be nonempty and finite") + return value + + +def _groups(value: object, field: str) -> list[list[torch.Tensor]]: + if not isinstance(value, (list, tuple)): + raise TypeError(f"{field} must contain tensor groups") + groups: list[list[torch.Tensor]] = [] + for index, group in enumerate(value): + if not isinstance(group, (list, tuple)): + raise TypeError(f"{field}[{index}] must be a tensor group") + tensors: list[torch.Tensor] = [] + for tensor in group: + if not isinstance(tensor, torch.Tensor): + raise TypeError(f"{field}[{index}] contains a non-tensor") + tensors.append(tensor) + groups.append(tensors) + return groups + + +def _replicated_gradients( + engine: object, +) -> tuple[dict[str, torch.Tensor], set[int]]: + module: object = getattr(engine, "module") + optimizer: object = getattr(engine, "optimizer") + if not isinstance(module, nn.Module): + raise TypeError("Engine module must be a torch.nn.Module") + names: dict[int, str] = {} + required: set[str] = set() + for name, parameter in module.named_parameters(): + if not isinstance(name, str) or not name: + raise TypeError("Parameter names must be nonempty strings") + if id(parameter) in names or name in names.values(): + raise ValueError("Duplicate named parameter identity or name") + names[id(parameter)] = name + # Megatron explicitly treats an absent flag as nonpartitioned. + partitioned: object = getattr(parameter, "tensor_model_parallel", False) + if not isinstance(partitioned, bool): + raise TypeError(f"{name}.tensor_model_parallel must be a bool") + if parameter.requires_grad and not partitioned: + required.add(name) + if not required: + raise ValueError("No replicated trainable parameters were observed") + + low_groups = _groups(getattr(optimizer, "bf16_groups"), "bf16_groups") + grad_groups = _groups( + getattr(optimizer, "fp32_groups_gradients"), "fp32_groups_gradients" + ) + if len(low_groups) != len(grad_groups): + raise ValueError("BF16 parameter and FP32 gradient group counts differ") + gradients: dict[str, torch.Tensor] = {} + parameter_ids: set[int] = set() + gradient_ids: set[int] = set() + for group_index, (parameters, group_gradients) in enumerate( + zip(low_groups, grad_groups) + ): + if len(parameters) != len(group_gradients): + raise ValueError( + f"Parameter/gradient lengths differ in group {group_index}" + ) + for low_parameter, gradient in zip(parameters, group_gradients): + parameter_id = id(low_parameter) + if parameter_id not in names: + raise ValueError("Optimizer parameter is absent from named_parameters") + if parameter_id in parameter_ids or id(gradient) in gradient_ids: + raise ValueError("Duplicate optimizer parameter or gradient identity") + parameter_ids.add(parameter_id) + gradient_ids.add(id(gradient)) + name = names[parameter_id] + gradient = _tensor(gradient, f"{name}.gradient") + # Native FP32 gradient views are flat, even for matrix parameters. + if low_parameter.numel() != gradient.numel(): + raise ValueError(f"Parameter/gradient element counts differ for {name}") + if name in required: + gradients[name] = gradient + missing = required.difference(gradients) + if missing: + raise ValueError(f"Missing replicated gradients: {sorted(missing)}") + return gradients, gradient_ids + + +def _mpu_rank(mpu: object, method_name: str) -> int: + method: object = getattr(mpu, method_name) + if not callable(method): + raise TypeError(f"mpu.{method_name} must be callable") + value: object = method() + return _nonnegative_integer(value, method_name) + + +def _snapshot(tensors: dict[str, torch.Tensor]) -> dict[str, torch.Tensor]: + return { + name: _tensor(tensor, f"{name}.gradient").detach().cpu().clone() + for name, tensor in tensors.items() + } + + +def install() -> None: + """Wrap the current native hooks once, without replacing prior observers.""" + global _installed + if _installed: + raise RuntimeError("Clipping audit is already installed") + optimizer_module = import_module("deepspeed.runtime.bf16_optimizer") + engine_module = import_module("deepspeed.runtime.engine") + engine_class: object = getattr(engine_module, "DeepSpeedEngine") + if not isinstance(engine_class, type): + raise TypeError("DeepSpeedEngine must be a class") + original_clip: object = getattr(optimizer_module, "clip_tensors_by_global_norm") + original_step: object = getattr(engine_class, "_take_model_step") + if not callable(original_clip) or not callable(original_step): + raise TypeError("Native clipping and model-step hooks must be callable") + + def observed_clip(*args: object, **kwargs: object) -> object: + if args: + raise TypeError("Clipping qualification requires native keyword arguments") + for field in ("input_tensors", "max_norm", "global_norm", "mpu"): + if field not in kwargs: + raise TypeError(f"Missing required clipping argument: {field}") + engine = _current_engine.get() + if engine is None: + raise RuntimeError("Clipping was called outside a model step") + input_tensors = kwargs["input_tensors"] + if not isinstance(input_tensors, (list, tuple)): + raise TypeError("input_tensors must be a list or tuple of tensors") + selected_ids: set[int] = set() + for tensor in input_tensors: + tensor = _tensor(tensor, "input_tensors entry") + if id(tensor) in selected_ids: + raise ValueError("input_tensors contains a duplicate gradient object") + selected_ids.add(id(tensor)) + gradients, known_gradient_ids = _replicated_gradients(engine) + if not selected_ids.issubset(known_gradient_ids): + raise ValueError("Selected gradient object is absent from optimizer state") + step = _nonnegative_integer(getattr(engine, "global_steps"), "global_steps") + rank = _nonnegative_integer(torch.distributed.get_rank(), "rank") + output = os.environ["RUNTIME_CLIPPING_OUTPUT"] + if not output: + raise ValueError("RUNTIME_CLIPPING_OUTPUT must not be empty") + destination = Path(output) / f"rank{rank}" / f"step{step + 1}.pt" + if destination.exists(): + raise FileExistsError(destination) + snapshot: ClippingSnapshot = { + "rank": rank, + # PipelineParallelGrid separates tensor slicing from its combined + # pipeline/model rank. The latter is not a TP rank when PP > 1. + "tp_rank": _mpu_rank(kwargs["mpu"], "get_slice_parallel_rank"), + "dp_rank": _mpu_rank(kwargs["mpu"], "get_data_parallel_rank"), + "step": step + 1, + "max_norm": _positive_number(kwargs["max_norm"], "max_norm"), + "global_norm": _positive_number(kwargs["global_norm"], "global_norm"), + "selected": { + name: id(gradient) in selected_ids + for name, gradient in gradients.items() + }, + "before": _snapshot(gradients), + "after": {}, + } + result: object = original_clip(*args, **kwargs) + snapshot["after"] = _snapshot(gradients) + destination.parent.mkdir(parents=True, exist_ok=True) + # Exclusive creation refuses duplicate evidence rather than overwriting it. + with destination.open("xb") as stream: + torch.save(snapshot, stream) + return result + + def observed_step(engine: object, *args: object, **kwargs: object) -> object: + token = _current_engine.set(engine) + try: + result: object = original_step(engine, *args, **kwargs) + return result + finally: + _current_engine.reset(token) + + setattr(optimizer_module, "clip_tensors_by_global_norm", observed_clip) + setattr(engine_class, "_take_model_step", observed_step) + _installed = True From 318175a2dc82ba2dc694230279b3020c2beba6ff Mon Sep 17 00:00:00 2001 From: Yuxuan Jiang Date: Mon, 2 Nov 2026 04:23:59 +0800 Subject: [PATCH 05/13] Record qualified runtime case and interrupted-trial recovery protocol --- .../real-rank-runtime-pilot-20261102.md | 64 ++++++++++++++++--- 1 file changed, 55 insertions(+), 9 deletions(-) diff --git a/docs/measurements/real-rank-runtime-pilot-20261102.md b/docs/measurements/real-rank-runtime-pilot-20261102.md index 0ad55544..791f8950 100644 --- a/docs/measurements/real-rank-runtime-pilot-20261102.md +++ b/docs/measurements/real-rank-runtime-pilot-20261102.md @@ -1,8 +1,9 @@ # Real cross-rank runtime case: agent-efficiency protocol **Status: in progress, 2026-11-02.** The native TP2 case reproduces on Ring18; -qualification controls and the agent service are being validated. Agent trials -have not started, so no debugging-efficiency improvement is claimed. This is +qualification passed and the first matched pair started. A transient SSH outage +interrupted result delivery; original records are retained and same-thread +recovery is being prepared. No debugging-efficiency improvement is claimed. This is operator documentation and must not be included in solver packets. The first two-update run had finite losses of 10.39899 and 10.62776. Among 27 @@ -119,13 +120,32 @@ are inputs and must remain unchanged. ## Planned workload and qualification -Use native Megatron–DeepSpeed GPT training on CodeParrot: four transformer layers, -hidden width 64, TP2, BF16 and ZeRO stage 0. Sequence lengths 128 and 256 are -qualification candidates; the chosen value, microbatch size, accumulation, -optimizer settings, clipping threshold, seeds and update count remain to be -frozen. Do not infer a model parameter count or performance result from this -configuration alone. Refresh the live Ring18 environment and available GPU -inventory before execution. +The frozen native Megatron–DeepSpeed GPT workload uses CodeParrot, four layers, +hidden width 64, eight attention heads, sequence length 128, TP2, PP1, DP1, +BF16 and ZeRO stage 0. Micro/global batch sizes are both 1. Apex FusedAdam uses +learning rate 0.0005, beta2 0.999, weight decay 0.1 and clipping threshold 1.0. +Seed 1234 is public; 5678 is the additional validation seed. Ordinary public runs +contain eight updates. Private grading uses two updates, capturing steps 0–2. + +Named parameters contain 1,722,880 elements per TP rank, including 9,856 replicated +elements. Counting each partition once and replicated parameters once gives +**3,435,904 logical parameter elements**. This is the GPT case used here, not +the earlier TorchTitan tiny-Llama trace. Ring18 uses NVIDIA A16 GPUs, Python +3.10.14 and Torch 2.2.2+cu121. GPU idleness is checked before each phase. + +Qualification confirms the cause: clipping selects 27/27 replicated gradients on +rank 0 and 0/27 on rank 1 in the vulnerable implementation. The first-step maximum +cross-rank gradient gap grows from exactly zero before clipping to +0.7917701378 after clipping; the corrected control retains zero gaps. Clipping- +disabled and TP1 candidate/control states match exactly. The fresh seed reproduces +the discrepancy. Selective TrainCheck preflight records two real clipping calls +per rank with complete entry/return pairs, no exceptions and exact recorded +model/optimizer/RNG parity. It observes scalar norm arguments, not tensor/model +tracking. Full evidence is in `qualification-summary.json`. + +Source-integrity checks cover all 271 framework/application Python files, all +139 cached native sources and four binaries. All remain unchanged. Private +snapshots stay on Ring18; compact reports, hashes and ordinary outputs are local. Qualification must establish all of the following before solver launch: @@ -246,3 +266,29 @@ agents may author their own checks around observed state. If agents solve the case from source, choose ordinary diagnostics, fail to use TrainCheck, or obtain no efficiency benefit, retain and report those outcomes without changing the task or success criteria after seeing them. + + +## Interrupted transport and recovery protocol + +The first matched pair used gpt-6-astra/high with identical limits: six remote +requests, 1–16 updates per request, 240 seconds per worker phase, 900 aggregate +CLI seconds and eight decision turns. Both agents first proposed the correct +clipping hypothesis from source and requested runtime diagnostics before editing +the implementation. This case therefore must not be described as requiring +runtime observation for every successful localization. + +An SSH connection closed after control request 3 finished training but before +artifact download. Treatment request 2 failed while creating its remote directory, +before training launched. A bounded subsequent SSH check succeeded. The completed +control phase has exit code 0, unchanged source/native binaries, empty worker/GPU +process lists, and 17.31105645187199 seconds of two-GPU worker lifetime; it must +be recovered rather than rerun. The treatment request directory was absent. + +Recovery preserves all original trial and error receipts. Pending results are +retrieved, the unlaunched request is executed once, and the same CLI thread IDs +resume with their original remaining request/turn/time budgets. New receipts are +append-only under `recovery-1/`; a composed record explicitly references both +parts. Report raw wall time including interruption and measured interruption +intervals separately. Select one effective receipt per logical request so recovery +does not double-count GPU cost. No private scoring or operator diagnosis is sent +to solvers. Independent grading remains gated on all four frozen trials. From 8788e73288c9a732f7330628c0de578ef28ace78 Mon Sep 17 00:00:00 2001 From: Yuxuan Jiang Date: Mon, 2 Nov 2026 05:15:49 +0800 Subject: [PATCH 06/13] Qualify a data-dependent DDP training case for the agent pilot --- .../ddp-runtime-state-pilot-20261102.md | 144 ++++++++++++++++++ 1 file changed, 144 insertions(+) create mode 100644 docs/measurements/ddp-runtime-state-pilot-20261102.md diff --git a/docs/measurements/ddp-runtime-state-pilot-20261102.md b/docs/measurements/ddp-runtime-state-pilot-20261102.md new file mode 100644 index 00000000..034b54d6 --- /dev/null +++ b/docs/measurements/ddp-runtime-state-pilot-20261102.md @@ -0,0 +1,144 @@ +# Data-dependent DDP training: optional TrainCheck pilot + +**Status: initial native qualification complete, 2026-11-02.** Held-out grading +qualification, optional-tool preflight and solver preparation are in progress. +This is private operator documentation, excluded from agent task packets. No +agent-efficiency benefit is established. Results from the earlier +[tensor-parallel clipping pilot](real-rank-runtime-pilot-20261102.md) are retained: +its four agents localized the cause from source before their first GPU request, +and neither treatment activated TrainCheck. + +## Why this case + +The next case reproduces the native Hugging Face Trainer behavior discussed in +[Transformers issue #34242](https://github.com/huggingface/transformers/issues/34242) +and addressed by [PR #34373](https://github.com/huggingface/transformers/pull/34373). +When ranks have different numbers of valid target tokens, averaging their local +mean-loss gradients does not yield the gradient of the global token-mean loss. +DDP can keep model replicas bitwise identical while training the wrong objective. +Replica agreement alone therefore cannot diagnose this problem. + +The relevant runtime evidence is the actual example assignment and valid target +count on each rank, the denominator reaching the model's native loss function, +and the resulting gradient before the optimizer update. The qualification uses +counts of 30 and 78 per rank versus a global denominator of 108. Equal-count +controls distinguish this condition from a general distributed-training failure. +This makes the case sensitive to cross-rank execution state; it does not prove +that an agent cannot derive the problem by reading the source and data. Such +source-based successes remain valid results. + +## Exact workload and provenance + +- Vulnerable native Transformers revision: + `a17f287ac039f92835b5cd9bd8ee28b584c9f65e`. +- Corrected native revision: `d21dbd1520937c993de1409215b1418bd6be74a1`. + The correction requires `average_tokens_across_devices=True`; the disabled + flag is a separate negative control. +- Native `Trainer` and `Qwen2ForCausalLM`, with 115,520 parameters, two layers, + hidden width 64, four attention/KV heads and sequence length 64. The model's + literal `**loss_kwargs` activates the historical Trainer path. Llama at this + revision uses a different signature and would not qualify the same path. +- Float32, AdamW, learning rate 0.001, zero weight decay, gradient clipping disabled, + constant learning rate, seed 1234, deterministic settings and TF32 disabled. +- Four examples per optimizer update, one equivalent single-GPU batch versus + two-rank DDP, with gradient accumulation of one or two microbatches. +- A controlled local text fixture derived from repository README bytes, with + fixed tokenization and equal/unequal valid-length variants. No pretrained + weights or external dataset downloads. This reproduces a real framework + defect with a small fixture; it is not a production-quality training study. +- Ring18 NVIDIA A16 GPUs 2 and 3, Torch 2.2.2+cu121, Python 3.10.14 and + Accelerate 1.0.1. Additional dependencies are installed in an isolated + experiment environment. The shared environment remains unchanged. + +All 4,715 files in each native source tree match its downloaded upstream archive. +The archive SHA-256 values are +`8977b0f525446b5ea80db0444a48b4ee5dac30373a13c64e8d979e4281638c65` +and `2d12575c66970d5f04a8fa9814ca1136f73de3d54759af20b1ea9df43fb85274`. +The shared environment package-list hash is +`70eda80894c761ff4c75e673681e86749b74158756cd8792d39027d923829fb4`. + +## Qualification and numerical error + +Sixteen completed native runs cover equal/unequal token counts, accumulation +factors one/two, vulnerable/corrected/flag-disabled code, the equivalent +single-GPU reference, repetition and observer removal. Two earlier setup failures +are retained. These 18 attempts consumed 402.53 allocated GPU-seconds, separately +from later optional-tool preflight and solver work. + +| Comparison against the equivalent single-GPU objective | Measured result | +| --- | --- | +| Vulnerable, unequal counts, accumulation 1 | Gradient relative L2 error up to 38.27%; model maximum absolute difference about 0.00582 | +| Vulnerable, unequal counts, accumulation 2 | Gradient relative L2 error up to 37.21%; model maximum absolute difference about 0.00594 | +| Corrected with global averaging | Gradient maximum absolute difference at most 8.57e-8 and relative L2 at most 3.29e-7; optimizer-state maximum absolute difference at most 1.12e-8 | +| Equal-count controls | Gradient and optimizer-state comparisons pass | +| Rank agreement within every two-rank run | Model replicas match bitwise, including in the vulnerable runs | +| Repetition and forward/loss observer removal | Bitwise equality of captured model, optimizer and CPU/CUDA/Python/NumPy RNG; 718 tensor and 10,448 scalar comparisons per comparison | + +The initial model comparison used absolute tolerance 1e-6 and relative tolerance +1e-4. It rejected some corrected results: model differences reached about +4.49e-6, while equal-count accumulation controls also reached about 3.21e-6. +Small floating-point gradient differences can be amplified by Adam near zero. +The original strict report is retained. A separate calibrated report uses model +absolute tolerance 1e-5. Gradient/optimizer absolute tolerance remains 1e-6; +relative tolerance remains 1e-4 for all three components. This is +chosen during qualification, before any agent output, with a margin of over 500 +between the model tolerance and the observed faulty-model deviation. It is not +bitwise equivalence between single-process and distributed arithmetic. + +All observed ranks, updates, examples and native calls must be present. Missing +activation, malformed snapshots and failed workers are errors, not successful +zero-difference results. No parameter, optimizer array or RNG contents are dropped +to make comparisons pass. Observer parity covers the recorded state, not +unobserved application state or live object aliasing. + +## Frozen agent comparison + +Prepare one matched pair with the same native source, local fixture, ordinary +backend logs, model and resource limits. The task asks whether the extension from +single-process to DDP training is ready for a longer run and what to do next. +It neither asserts a bug nor asks for any particular diagnostic. Logged backend +training losses are not presented as directly comparable held-out quality curves. + +The treatment additionally receives the unchanged generic TrainCheck observation +manual and implementation. It may choose custom logging or no instrumentation. +The control has the same ability to author arbitrary local diagnostics and use +the remote execution service. Neither arm receives issue IDs, this report, +private observations, a targeted invariant or grading feedback. + +Use the same native CLI model/settings and accounting protocol as the previous +pilot: gpt-6-astra/high, priority service, CLI 0.160.0, six requests, 1–16 global +updates per request, 240 seconds per worker phase, 900 aggregate CLI seconds and +eight decision turns. Both the two-GPU `current` and one-GPU `reference` execute +the agent's submitted source; the latter is not a hidden corrected source. +Allocated GPU cost uses two or one devices accordingly and includes startup and +failed requests. Qualification and independent grading costs remain separate. + +Record source-supported localization, runtime-supported localization, earliest +submitted source that later passes independent grading, agent-visible repair +verification and final completion separately. Preserve native token counters, +cache subsets, measured service waiting, every submitted source and raw events. +Within-turn token costs may only be intervals. Availability or manual reading +does not count as actual TrainCheck use. + +## Evidence and reproduction + +Local evidence root: +`/Users/matt/gitrepos/traincheck/outputs/ring18-token-normalization-20261102`. +Remote evidence root: +`/home/yuxuan/experiments/traincheck-token-normalization-20261102` on Ring18. + +- `provenance.json`: native archive/tree, environment, corpus and operator hashes. +- `qualification-strict.json` and `qualification-model-calibrated.json`: separate + full comparisons with original and calibrated model tolerances. +- `execution-ledger.jsonl`: successful runs and retained setup failures, commands, + elapsed time, devices and allocated GPU costs. +- `operator/`: versioned qualification workers, analysis, regression tests and + commands; snapshots and native outputs remain in the remote experiment tree. +- `TASK.md`, `pilot.py`, `protocol-reuse.json`: neutral task and isolated service + configuration. Packet hashes and a positive reviewed qualification gate must + be frozen before starting agents. + +The operator request service and qualification code do not modify the TrainCheck +package. Large native sources, data, checkpoints and traces remain outside Git. +The PR records reproducible evidence and limitations; it does not claim a general +debugging speedup or silently promote a source-readable case to a runtime-only one. From fb6fe178119df654dc8e0444908a4036336ad185 Mon Sep 17 00:00:00 2001 From: Yuxuan Jiang Date: Mon, 2 Nov 2026 05:18:41 +0800 Subject: [PATCH 07/13] Report source-first repairs and measured costs in the runtime pilot --- .../real-rank-runtime-pilot-20261102.md | 261 +++++++++++++++--- 1 file changed, 226 insertions(+), 35 deletions(-) diff --git a/docs/measurements/real-rank-runtime-pilot-20261102.md b/docs/measurements/real-rank-runtime-pilot-20261102.md index 791f8950..8921b60b 100644 --- a/docs/measurements/real-rank-runtime-pilot-20261102.md +++ b/docs/measurements/real-rank-runtime-pilot-20261102.md @@ -1,10 +1,13 @@ # Real cross-rank runtime case: agent-efficiency protocol -**Status: in progress, 2026-11-02.** The native TP2 case reproduces on Ring18; -qualification passed and the first matched pair started. A transient SSH outage -interrupted result delivery; original records are retained and same-thread -recovery is being prepared. No debugging-efficiency improvement is claimed. This is -operator documentation and must not be included in solver packets. +**Status: completed four-trial pilot, 2026-11-02.** All four agents identified the +clipping cause from source before their first GPU request. Each first submitted +a valid repair in turn 2, later passing all four private scenarios. No TrainCheck +instrumentation activation was observed in either treatment. This is a source-localizable +case and does not establish a TrainCheck efficiency benefit. Transport failures +and same-thread recoveries remain in the raw evidence; timing below reports both +raw and adjusted values. This is operator documentation, excluded from solver +packets. The first two-update run had finite losses of 10.39899 and 10.62776. Among 27 replicated trainable parameters, cross-rank mismatches increased from zero before @@ -24,7 +27,7 @@ experiment directory below. Both ordinary eight-update runs complete with finite losses and zero skipped or NaN iterations. Final FP16/BF16 losses are 10.25196/10.25426. This short difference -does not establish a quality regression. The neutral task will present measured +does not establish a quality regression. The neutral task presented measured curves without asserting one. An issue-named installation-path prefix in exported text is replaced with `/runtime/python-env`, identically for both arms; raw logs, exact mappings and original/public hashes remain operator evidence. Error/warning @@ -47,14 +50,14 @@ failure counts remain part of the results. The development base is `codex/checkpoint-continuation-costs`, [PR #170](https://github.com/OrderLab/TrainCheck/pull/170), at `c79218c960a823f954bf49dad354975da39e6f24`. Subsequent source revisions, workload -configuration, packet manifests, commands and raw evidence must be frozen before -the first solver sees the task. +configuration, packet manifests and commands were frozen before solver exposure. +The final evidence references are recorded below. ## Question and case selection The question is whether optional access to TrainCheck helps an agent investigate and repair a real distributed training problem more efficiently under a neutral -engineering task. The comparison will measure causal localization, independently +engineering task. The comparison measures causal localization, independently valid code, agent-visible runtime verification and final completion separately. The selected candidate is the historical DeepSpeed BF16 optimizer failure commonly @@ -106,19 +109,20 @@ The saved live diff is `/Users/matt/gitrepos/traincheck/outputs/ring18-runtime-20261102/original-bf16-working.diff`, SHA-256 `c04efe0f7f59bbcd973a6289a57b40fa4f832a7855b1b4ada9ebb73e3fcde8ad`. Its two changed lines move the identical condition and `continue` out by one -indentation level. This artifact has been read and hashed locally; the complete -source snapshot and provenance manifest remain to be frozen. +indentation level. This artifact was read and hashed locally; the complete +source snapshots were subsequently frozen in the qualification and packet +manifests. -The planned candidate is an isolated snapshot of the inspected working-tree -source. The positive control uses the corresponding correct HEAD version of +The candidate is an isolated snapshot of the inspected working-tree source. +The positive control uses the corresponding correct HEAD version of `bf16_optimizer.py` while preserving the same remaining compatibility changes. -The final evidence bundle must retain the exact source diff and per-file hashes, -including a check that the control differs only in the intended logic. This is a +The evidence bundle retains the exact source diff and per-file hashes, +including the check that the control differs only in the intended logic. This is a historical semantic reproduction on a compatible environment, not execution of an untouched historical checkout. Shared repositories and installed environments are inputs and must remain unchanged. -## Planned workload and qualification +## Workload and qualification The frozen native Megatron–DeepSpeed GPT workload uses CodeParrot, four layers, hidden width 64, eight attention heads, sequence length 128, TP2, PP1, DP1, @@ -147,7 +151,7 @@ Source-integrity checks cover all 271 framework/application Python files, all 139 cached native sources and four binaries. All remain unchanged. Private snapshots stay on Ring18; compact reports, hashes and ordinary outputs are local. -Qualification must establish all of the following before solver launch: +The prespecified qualification gates were: 1. The candidate initializes successfully and executes finite training updates. The measured gradient norm actually activates clipping; configured clipping @@ -196,9 +200,10 @@ and behavior; do not obscure code to make static reasoning artificially harder. The exact allowed revision scope and preserved training contracts must be stated and frozen before launch. -Plan two fresh matched pairs, with launch order reversed in the second pair. -Use the same model, reasoning setting, tool interface, runtime, request allowance, -update allowance and time limits. Freeze their exact values before exposure. +Two fresh matched pairs ran, with launch order reversed in the second pair. +Both used gpt-6-astra/high, the same tool interface and runtime, six remote +requests, 1–16 updates per request, 240 seconds per worker phase, 900 aggregate +CLI seconds and eight decision turns. These limits were frozen before exposure. Controls may use ordinary logging, assertions and custom Python diagnostics. Treatments additionally receive the generic TrainCheck observation documentation and implementation; no rule targets this case and no task step requires its use. @@ -206,8 +211,8 @@ and implementation; no rule targets this case and no task step requires its use. Standard responses return native logs, training metrics and ordinary artifacts. Do not automatically supply private grader state exports as a diagnostic shortcut. Either arm may request and capture equivalent runtime state explicitly under the -same resource budget. Private grading begins only after all final submissions -freeze; grader feedback is not supplied during the task. +same resource budget. Private grading began only after all final submissions +froze; grader feedback was not supplied during the task. ## Milestones, costs and evidence @@ -267,15 +272,12 @@ case from source, choose ordinary diagnostics, fail to use TrainCheck, or obtain no efficiency benefit, retain and report those outcomes without changing the task or success criteria after seeing them. - ## Interrupted transport and recovery protocol -The first matched pair used gpt-6-astra/high with identical limits: six remote -requests, 1–16 updates per request, 240 seconds per worker phase, 900 aggregate -CLI seconds and eight decision turns. Both agents first proposed the correct -clipping hypothesis from source and requested runtime diagnostics before editing -the implementation. This case therefore must not be described as requiring -runtime observation for every successful localization. +Both matched pairs used the frozen limits above. All four agents first proposed +the correct clipping hypothesis from source and requested runtime diagnostics +before editing the implementation. This case therefore must not be described +as requiring runtime observation for successful localization. An SSH connection closed after control request 3 finished training but before artifact download. Treatment request 2 failed while creating its remote directory, @@ -284,11 +286,200 @@ control phase has exit code 0, unchanged source/native binaries, empty worker/GP process lists, and 17.31105645187199 seconds of two-GPU worker lifetime; it must be recovered rather than rerun. The treatment request directory was absent. -Recovery preserves all original trial and error receipts. Pending results are -retrieved, the unlaunched request is executed once, and the same CLI thread IDs -resume with their original remaining request/turn/time budgets. New receipts are -append-only under `recovery-1/`; a composed record explicitly references both -parts. Report raw wall time including interruption and measured interruption +Recovery preserved all original trial and error receipts. Pending results were +retrieved, the unlaunched request executed once, and the same CLI thread IDs +resumed with their original remaining request/turn/time budgets. New receipts +are append-only under `recovery-1/`. The second treatment's request 1 also +completed GPU work before an artifact-download failure; `recovery-2/` retrieved +that existing result and continued its original thread. Composed records +explicitly reference every original and continuation artifact. Report raw wall time including interruption and measured interruption intervals separately. Select one effective receipt per logical request so recovery does not double-count GPU cost. No private scoring or operator diagnosis is sent -to solvers. Independent grading remains gated on all four frozen trials. +to solvers. Independent grading began only after all four trials froze and is +now complete. + +## Completed solver and independent-validation results + +All four agents first localized the same clipping-selection cause from source, +then wrote ordinary cross-rank diagnostics to test that hypothesis. No TrainCheck +instrumentation activation was observed in either treatment. Both controls and both treatments +submitted their first independently valid implementation in **turn 2**, after +one diagnostic GPU request. All later submitted implementations also passed. + +Independent grading covered every frozen submission, deduplicated by all Python +source bytes: eight distinct candidate sources and one known-correct reference, +each executed in four scenarios, for **36 GPU phases**. Each scenario compared +recorded model parameters, the complete exported optimizer state and recorded +Torch/Megatron RNG at steps 0, 1 and 2 against its fresh reference. Step 0 is +after the first backward pass. Model tolerances were `atol=0`, `rtol=0`; +optimizer container types/order, scalars and tensor contents remained significant. +Replicated parameters were also checked within their tensor-parallel group. + +| Private scenario | Seed | TP / allocated GPUs | Clipping threshold | Turn-1 sources | Turn-2 and later sources | +| --- | ---: | ---: | ---: | --- | --- | +| Base | 1234 | 2 / 2 | 1.0 | All four fail | All four pass | +| Held-out seed | 5678 | 2 / 2 | 1.0 | All four fail | All four pass | +| TP1 control | 1234 | 1 / 1 | 1.0 | All four pass | All four pass | +| Clipping disabled | 1234 | 2 / 2 | 0.0 | All four pass | All four pass | + +Thus the earliest valid-submission milestone is justified by all four gates, +not by the agent's own assertions or a successful training exit. These short +checks do not establish long-horizon convergence, checkpoint-resume correctness, +or coverage of other distributed configurations. No TP2×DP2 validation is claimed. + +### Milestones and resource costs + +All times are seconds from the trial's original start. In a `raw / adjusted` +cell, adjusted time subtracts the measured union of resource-lock waits and +operator suspension intervals. It is an accounting view, not a reconstruction +of uninterrupted execution. Source diagnosis has no prior queue/interruption. +“First valid” is the captured submission boundary, retrospectively graded; +“finish” is the final visible decision, not when private grading completed. + +| Trial | Source diagnosis | Runtime diagnosis, raw / adjusted | First valid submission, raw / adjusted | Agent-visible repair verification, raw / adjusted | Finish, raw / adjusted | +| --- | ---: | ---: | ---: | ---: | ---: | +| Control 1 | 36.118 | 134.876 / 134.875 | 162.692 / 162.691 | 232.732 / 232.729 | 1,033.415 / 310.957 | +| TrainCheck 1 | 18.708 | 201.057 / 163.998 | 286.436 / 249.376 | 1,018.755 / 316.404 | 1,121.005 / 418.653 | +| TrainCheck 2 | 35.336 | 776.050 / 125.544 | 807.169 / 156.663 | 890.986 / 240.479 | 920.339 / 269.831 | +| Control 2 | 16.462 | 127.342 / 127.341 | 387.317 / 387.316 | 451.859 / 451.857 | 539.296 / 539.293 | + +Agent-visible verification is the first completed inspection/check command that +exposes both ranks' corrected-run coverage, separate from a later prose claim. +Every corrected run actually contains all **16 updates on both ranks**, 52 +clipped gradients per rank, and zero errors for the scaling/replica fields that +its agent chose to observe. Initial diagnostics used 3, 8, 4 and 4 updates in the +table's order. Focused tests executed on both workers: respectively 5, 7, 4 and +3 tests per worker. The public diagnostics have different coverage and are not +the independent uniform oracle. Control 1, TrainCheck 1 and Control 2 also ran +eight FP16 updates; TrainCheck 2 did not, and acknowledged that limitation. + +The full-trial boundary occurs slightly after the visible finish statement. +Allocated GPU-seconds count devices × supervised worker lifetime, including +startup; they do not measure device utilization. + +| Trial | Requests | Full raw wall s | Queue s | Operator suspension s | Full adjusted wall s | CLI s | GPU-s before first valid | Full solver GPU-s | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| Control 1 | 3 | 1,037.150 | 0.003 | 722.455 | 314.692 | 173.658 | 34.731 | 104.607 | +| TrainCheck 1 | 3 | 1,124.788 | 37.062 | 665.290 | 422.436 | 271.649 | 35.239 | 94.831 | +| TrainCheck 2 | 2 | 924.628 | 42.890 | 607.618 | 274.121 | 159.723 | 34.730 | 69.355 | +| Control 2 | 3 | 543.570 | 0.002 | 0.000 | 543.568 | 397.481 | 34.720 | 94.298 | + +The four solver trials used **363.091 GPU-s total**. The separate 36-phase private +grading used **1,069.616 GPU-s**; none is charged to solver milestones. +Qualification, instrumentation preflight and earlier setup work remain separate +operator costs in their own receipts; the two totals here are not a total +campaign-cost claim. Recollecting grades from existing snapshots added no GPU +training phases. Recovered completed requests are counted once through their +effective receipt; their original failures and unknown original measurements +remain preserved. + +### Native token accounting + +The following counts are native cumulative input-plus-output tokens, including +cached input. Intervals are inclusive previous/current completed-turn bounds; +no within-turn interpolation is used. First-valid and final completed-turn +counts are exact as reported. All source diagnoses occur in turn 1 before a GPU +request. Full per-category bounds are retained in the final JSON. + +| Trial | Source-diagnosis interval | Runtime-diagnosis interval | First-valid tokens | Final completed-turn tokens | +| --- | ---: | ---: | ---: | ---: | +| Control 1 | 0–393,071 | 393,071–693,107 | 693,107 | 1,236,382 | +| TrainCheck 1 | 0–423,568 | 423,568–979,286 | 979,286 | 1,455,127 | +| TrainCheck 2 | 0–334,917 | 334,917–590,405 | 590,405 | 805,536 | +| Control 2 | 0–480,682 | 480,682–835,766 | 835,766 | 1,336,513 | + +| Trial | Final input | Of input, cached | Final output | Of output, reasoning | +| --- | ---: | ---: | ---: | ---: | +| Control 1 | 1,226,306 | 1,137,280 | 10,076 | 1,975 | +| TrainCheck 1 | 1,440,861 | 1,350,784 | 14,266 | 4,260 | +| TrainCheck 2 | 796,875 | 727,936 | 8,661 | 2,396 | +| Control 2 | 1,327,625 | 1,243,776 | 8,888 | 1,431 | + +Cached input and reasoning output are subsets, not extra tokens. These counts +are not a dollar-cost estimate. The final visible statement occurs within the +last turn; its own within-turn token interval is preserved separately from the +exact final completed-turn count. + +### TrainCheck usage and the negative finding + +TrainCheck 2 visibly read the generic manual. TrainCheck 1 only visibly listed +its filename; a listing is not counted as reading. Neither treatment emitted +observed TrainCheck trace files or added an explicit TrainCheck import in its +changed submitted Python. Both used ordinary custom state diagnostics. The +launcher option permitting TrainCheck access is not activation. No exhaustive +runtime module/dynamic-import audit was collected, so the precise conclusion +is **no instrumentation activation observed**, not proof that no incidental +import could ever have occurred. + +This real distributed case still did not meet the intended difficulty criterion: +all four agents identified the cause from source in 16–36 seconds, before runtime +observations. Runtime evidence confirmed the consequence and supported the +repair, but was not necessary for their observed localization. These results +therefore do not establish a TrainCheck debugging-efficiency benefit or an +overhead for active use. They describe optional availability that went unused. + +Two runs per arm on one selected case, unequal diagnostic lengths, one omitted +FP16 rerun, normal agent variation and asymmetric transport interruptions prevent +a causal speedup interpretation. Raw and adjusted values are both retained; no +benefit ratio is claimed. This finding also does not show that agents can solve +all TrainCheck paper cases in one shot. A subsequent case should first qualify a +failure whose cause remains ambiguous after source inspection and whose rank +state supplies the discriminating evidence, without hiding code or forcing use. +The next candidate is documented in the [DDP runtime-state +pilot](ddp-runtime-state-pilot-20261102.md). + +### Evidence, replay and collection failure + +Local operator evidence root: +`/Users/matt/gitrepos/traincheck/outputs/ring18-runtime-20261102/`. +The table below uses paths relative to that root. Private state snapshots remain +under `/home/yuxuan/experiments/traincheck-runtime-20261102/private-grading/` on +Ring18. Large snapshots, solver transcripts and datasets are not committed. + +| Evidence | SHA-256 | +| --- | --- | +| `solver-completion-composed.json` | `c7c654a72841b75f66b0ed5de0b2af7668940c77295e58a467369b4d203c8c79` | +| `grading-plan.json` | `b2dd407117d09165b93d7596ef13ef13c4dc80055b6371b9f86ab3bff002e18c` | +| `grading/result-20261101T211500631757Z.json` | `6a9810b43694cb8b13c06da6edc8feb7ab489c59fc6357ca4342d6d6864307a6` | +| `milestone-annotations-final.json` | `fe1048a72fd9a77bef24f84f2ab110e2bb4bf6265792ac2fa021b65365dce509` | +| `milestone-summary-final.json` | `a858a1003cd14c6e179efdf410723e25caa99c0ddb5eaa4af2b12d6c0dfbaeea` | +| `private-grading-milestone-join.json` | `792219847ed15062d5b8b0b29090e52068c60fcc860d3a69df8077a9b6201c77` | + +The join records every turn's submission path, source ID, Python-manifest hash, +optimizer-file hash and four scenario outcomes. Earliest valid submissions are: + +| Trial | Submission path relative to the evidence root | Python source ID / SHA-256 | +| --- | --- | --- | +| Control 1 | `trials/control_1/submissions/turn-2` | `4a6f8230a5967b40a93d1ee792427048d72405ece18abd5944d5b1caf3510b11` | +| TrainCheck 1 | `trials/traincheck_1/submissions/turn-2` | `13db280c9f102dab5b50b8d663e4f6c4637bea6a5535bc811f16e2d350fbcc51` | +| TrainCheck 2 | `recovery-2/trials/traincheck_2/submissions/turn-2` | `e51f85808dc26b77adc7f13eca38e2aa82f1b26db69114b9a12c4805f745487a` | +| Control 2 | `trials/control_2/submissions/turn-2` | `d972171d51412890b226fab07f616ad1d2819a403e23fdc83a16b67a914faab6` | + +`milestone-annotations-visible.json` and `milestone-summary-visible.json` remain +unchanged as the pre-grade analysis. Final artifacts are new files. Exact event +lines and hashes are bound in the annotations; raw transcripts are under +`trials/`, `recovery-1/trials/` and `recovery-2/trials/`. Recompute the final +accounting without training or solver interaction using: + +```sh +python summarize_milestones.py --study-root "$PWD" --annotations milestone-annotations-final.json --output milestone-summary-recomputed.json +``` + +The output path must be new. `join_private_grading.py` validates the frozen plan, +all source/occurrence mappings, original phase receipts and all four scenario +outcomes before producing final annotations; it never launches remote work. + +The initial collection produced only **unknown** outcomes because its caller +passed a tuple to an evaluator requiring a nonempty list of steps. It is retained +as `grading/result-20261101T211129986599Z.json`, SHA-256 +`2b56e92008716bf7a7bc6d6642d7cb67fab6478f5460628ae5fc5b70feedf296`. +The collector's generated argument changed from `steps=(0,1,2)` to +`steps=[0,1,2]`; the old helper is archived. A focused integration regression +exercised the actual evaluator contract. Read-only recollection of the same +snapshots produced the final result above: no evaluator contract, plan, solver +feedback or GPU phase changed. Unknown evaluation was never treated as success. + +The accounting/extraction and private-grade-join helpers pass 42 focused local +tests, explicit strict mypy and pinned formatting/lint checks. This validates +the checked contracts, not absence of every possible harness defect. Independent +state comparisons and retained failures are the basis of the reported outcomes. From d8d57d0ef153e1ce17304c25c87b0da8a37d0dcf Mon Sep 17 00:00:00 2001 From: Yuxuan Jiang Date: Mon, 2 Nov 2026 05:34:10 +0800 Subject: [PATCH 08/13] Freeze matched DDP trial gates and independent validation scenarios --- .../ddp-runtime-state-pilot-20261102.md | 76 +++++++++++++++++-- 1 file changed, 70 insertions(+), 6 deletions(-) diff --git a/docs/measurements/ddp-runtime-state-pilot-20261102.md b/docs/measurements/ddp-runtime-state-pilot-20261102.md index 034b54d6..f211ab4f 100644 --- a/docs/measurements/ddp-runtime-state-pilot-20261102.md +++ b/docs/measurements/ddp-runtime-state-pilot-20261102.md @@ -1,7 +1,8 @@ # Data-dependent DDP training: optional TrainCheck pilot -**Status: initial native qualification complete, 2026-11-02.** Held-out grading -qualification, optional-tool preflight and solver preparation are in progress. +**Status: one matched agent pair running, 2026-11-02.** Native qualification, +held-out positive controls, optional-tool preflight and packet review passed +before launch. The pair started after the 2026-11-01 21:30:49 UTC protocol freeze. This is private operator documentation, excluded from agent task packets. No agent-efficiency benefit is established. Results from the earlier [tensor-parallel clipping pilot](real-rank-runtime-pilot-20261102.md) are retained: @@ -91,9 +92,52 @@ zero-difference results. No parameter, optimizer array or RNG contents are dropp to make comparisons pass. Observer parity covers the recorded state, not unobserved application state or live object aliasing. +## Additional prelaunch checks + +The scalar-argument TrainCheck preflight observes native +`torch.nn.functional.dropout` calls: 12 entry/return pairs per rank, with no +unmatched records. Recorded model, optimizer and RNG state is bitwise identical +to the ordinary matched run (718 tensor and 10,448 scalar checks), as are all +162 gradient comparisons. This verifies the tested direct wrapper interface; +it does not qualify arbitrary tensor metadata, model tracking or the inference +engine. Its measured cost is 34.444 allocated GPU-seconds. A preceding import-only +attempt failed because the CUDA library path was missing; its GPU cost was not +measured and is retained as unknown, not zero. + +The independent grader attaches observers to the actual submitted Trainer, +model and optimizer. It does not replace the submitted training pipeline with +the qualification worker. An actual-entry replay matches every recorded state, +RNG and gradient value from the archived native parent execution bitwise. + +Fresh seed 5678 uses a separate fixture whose global valid-target counts vary +across three updates: 124, 140 and 152. Corrected actual-entry DDP runs with +accumulation 1 and 2 pass all 58 declared comparisons against the equivalent +single-GPU reference. Maximum held-out differences are 1.121e-6 for gradients, +1.0548e-5 for model parameters and 1.576e-7 for optimizer state. These pass the +unchanged **combined** absolute-plus-relative tolerance; a maximum difference +can exceed the absolute term alone. The actual-entry replay plus three held-out +controls cost 87.914 GPU-seconds, separately from solver requests. + +The frozen independent grading contract has seven scenarios: equal, unequal and +held-out data, each with DDP accumulation 1/2, plus single-GPU compatibility on +the original unequal-length data. It requires actual sample coverage, full +model/gradient/optimizer comparisons and exact recorded RNG against a matched +same-world corrected execution. It does not compare single-GPU RNG with two-GPU +RNG. A valid repartition may change rank-local sample assignments while preserving +each global four-example window; no particular normalization implementation is +required. + +Native ordinary eight-update service runs verify both backends. A first pair of +service runs revealed that native serialized training arguments contained an +operator path whose experiment name could reveal the case. The service now +passes a relative output path to the same destination. Two fresh ordinary runs +confirm neutral serialized paths and byte-identical final model files relative +to the earlier runs. Both sets of receipts are retained. This service qualification +costs 74.882 GPU-seconds in total; no binary checkpoint is edited or normalized. + ## Frozen agent comparison -Prepare one matched pair with the same native source, local fixture, ordinary +The matched pair receives the same native source, local fixture, ordinary backend logs, model and resource limits. The task asks whether the extension from single-process to DDP training is ready for a longer run and what to do next. It neither asserts a bug nor asks for any particular diagnostic. Logged backend @@ -105,7 +149,7 @@ The control has the same ability to author arbitrary local diagnostics and use the remote execution service. Neither arm receives issue IDs, this report, private observations, a targeted invariant or grading feedback. -Use the same native CLI model/settings and accounting protocol as the previous +It uses the same native CLI model/settings and accounting protocol as the previous pilot: gpt-6-astra/high, priority service, CLI 0.160.0, six requests, 1–16 global updates per request, 240 seconds per worker phase, 900 aggregate CLI seconds and eight decision turns. Both the two-GPU `current` and one-GPU `reference` execute @@ -120,6 +164,14 @@ cache subsets, measured service waiting, every submitted source and raw events. Within-turn token costs may only be intervals. Availability or manual reading does not count as actual TrainCheck use. +An independent packet comparison confirms that all 1,850 exported native runtime +source/README/license files match the upstream parent archive, and all 12 actual +training examples retain their qualified arrays. The control has 1,862 files; +the treatment has 1,894. Only the initial prompt differs among shared files, by +one sentence offering optional access. The 32 treatment-only files are the +generic manual and 31 frozen TrainCheck source files. The source and data remain +visible, so this review does not establish that runtime observation is necessary. + ## Evidence and reproduction Local evidence root: @@ -135,8 +187,20 @@ Remote evidence root: - `operator/`: versioned qualification workers, analysis, regression tests and commands; snapshots and native outputs remain in the remote experiment tree. - `TASK.md`, `pilot.py`, `protocol-reuse.json`: neutral task and isolated service - configuration. Packet hashes and a positive reviewed qualification gate must - be frozen before starting agents. + configuration. Final packets are under + `/private/tmp/traincheck-ddp-solvers-20261102-v2`. +- `launch-plan.json`, SHA-256 + `1046652737f8fe2a446f81d5c864a499a64ac9cdf9506ee6a157640f9cace98e`: + exact model/settings/budgets, execution order, code, packet and five gate hashes. +- `actual-entry-positive-gate.json`, SHA-256 + `d66be5d0c9b0acb7f61885df30e24a19d6786ead61f5d6a708a459e4f637991d`: + actual-entry parity and fresh-seed positive controls. +- `independent-grading-contract.json`, SHA-256 + `4ea06f939a254117cfb8fbb10c7b21e9f000b16f15d6e28ce520fe53da5977d5`: + the seven scenarios and unchanged numerical/coverage criteria. +- `prelaunch-neutrality-review.json`, SHA-256 + `a79346c2664a1e63796b1557bbec76906d1c08784b69fcd6274515bbc692888f`: + independent packet comparison and evidence review. The operator request service and qualification code do not modify the TrainCheck package. Large native sources, data, checkpoints and traces remain outside Git. From 0684d834a55edf164f10816375674ae9d9d2eb1b Mon Sep 17 00:00:00 2001 From: Yuxuan Jiang Date: Mon, 2 Nov 2026 06:03:27 +0800 Subject: [PATCH 09/13] Report completed DDP agent trials and independent repair validation --- .../ddp-runtime-state-pilot-20261102.md | 196 +++++++++++++++++- 1 file changed, 188 insertions(+), 8 deletions(-) diff --git a/docs/measurements/ddp-runtime-state-pilot-20261102.md b/docs/measurements/ddp-runtime-state-pilot-20261102.md index f211ab4f..af12da6d 100644 --- a/docs/measurements/ddp-runtime-state-pilot-20261102.md +++ b/docs/measurements/ddp-runtime-state-pilot-20261102.md @@ -1,13 +1,20 @@ # Data-dependent DDP training: optional TrainCheck pilot -**Status: one matched agent pair running, 2026-11-02.** Native qualification, -held-out positive controls, optional-tool preflight and packet review passed -before launch. The pair started after the 2026-11-01 21:30:49 UTC protocol freeze. -This is private operator documentation, excluded from agent task packets. No -agent-efficiency benefit is established. Results from the earlier -[tensor-parallel clipping pilot](real-rank-runtime-pilot-20261102.md) are retained: -its four agents localized the cause from source before their first GPU request, -and neither treatment activated TrainCheck. +**Status: one matched pair and independent grading completed, 2026-11-02.** +Both agents localized the cause from source before their first requested run. +Both final submissions pass all seven independent scenarios. No TrainCheck +instrumentation activation was observed; the treatment agent read its manual +and used ordinary PyTorch/Trainer diagnostics. This case did not meet the intended +difficulty of requiring runtime observation for localization, and provides no +evidence of a TrainCheck efficiency benefit. + +Native qualification, held-out positive controls, optional-tool preflight and +packet review passed before launch. The pair started after the 2026-11-01 +21:30:49 UTC protocol freeze. This is private operator documentation, excluded +from task packets. Results from the earlier +[tensor-parallel clipping pilot](real-rank-runtime-pilot-20261102.md) remain: +its four agents also localized the cause from source before their first GPU +request; no TrainCheck instrumentation activation was observed there either. ## Why this case @@ -172,6 +179,127 @@ one sentence offering optional access. The 32 treatment-only files are the generic manual and 31 frozen TrainCheck source files. The source and data remain visible, so this review does not establish that runtime observation is necessary. +## Completed pair and independent results + +The control produced its first passing implementation in turn 2; the treatment +in turn 3. Independent replay covered every one of the 13 frozen submissions, +using eight distinct complete source maps. Exact duplicate maps reuse the same +seven-scenario result; changes to diagnostic helpers still count as distinct +maps even when the ordinary training code is unchanged. The original control +source and treatment turns 1/2 fail the unequal-count and held-out comparisons, +with complete observation coverage. No earlier unknown grade blocks either +first-valid result. Both final complete source maps pass all seven scenarios. + +The grader runs the actual immutable entry point against each submitted +pipeline. It checks full model, gradient and optimizer state, native sample/mask +coverage and matched same-world RNG, under the frozen tolerances. It does not +reward a particular repair pattern or a training exit code alone. A first-valid +time is the **original submission boundary**, established retrospectively by +these grades; it excludes independent grading latency and GPU cost. + +| Milestone, elapsed seconds from trial start | Control | TrainCheck available | +| --- | ---: | ---: | +| First source-supported diagnosis | 47.928 | 38.785 | +| First runtime-supported diagnosis | 149.654 | 102.714 | +| First visible implementation edit | 157.952 | 170.017 | +| Earliest submission passing independent grading | 181.380 | 204.074 | +| First agent-visible repaired-run comparison | 287.709 | 259.778 | +| Final finish decision | 449.068 | 469.626 | +| Terminal trial boundary | 453.645 | 474.343 | + +Event times are receipt-time upper bounds, not inferred internal reasoning times. +The runtime-supported diagnosis is the first explicit statement connecting +observed local denominators of 30/78 and synchronized rank gradients to the +source hypothesis. Later gradient comparisons quantify the initial error at +about 0.332294 relative L2. The treatment's first local repaired-run comparison +failed because `safetensors` was absent; the table counts its subsequent +successful comparison, not that failed command. Agent-visible verification +remains distinct from the stronger independent grade. + +| Milestone resource cost | Control | TrainCheck available | +| --- | ---: | ---: | +| First-valid wall minus queue, seconds | 166.428 | 204.071 | +| First-valid recorded CLI time, seconds | 138.627 | 149.629 | +| First-valid allocated GPU-seconds | 25.043 | 37.548 | +| First-valid cumulative input + output tokens | 595,207 | 845,768 | +| First-valid input / cached input tokens | 586,784 / 536,192 | 835,843 / 772,480 | +| First-valid output / reasoning output tokens | 8,423 / 2,093 | 9,925 / 1,859 | +| Runtime-diagnosis allocated GPU-seconds | 25.043 | 25.025 | +| Repaired-run-comparison allocated GPU-seconds | 62.597 | 62.588 | + +Within-turn token costs are intervals, bounded by the previous and current +completed native turn. They are not interpolated from elapsed time: + +| Visible milestone, cumulative input + output interval | Control | TrainCheck available | +| --- | ---: | ---: | +| Source-supported diagnosis | 0–276,705 | 0–298,887 | +| Runtime-supported diagnosis | 276,705–595,207 | 298,887–514,729 | +| First implementation edit | 276,705–595,207 | 514,729–845,768 | +| Repaired-run comparison | 778,431–982,728 | 845,768–1,142,119 | +| Final finish decision | 1,207,772–1,471,632 | 1,558,352–1,744,783 | + +| Complete trial cost | Control | TrainCheck available | +| --- | ---: | ---: | +| Raw elapsed wall, seconds | 453.645 | 474.343 | +| Recorded resource queue, seconds | 17.777 | 5.362 | +| Wall minus recorded queue, seconds | 435.868 | 468.981 | +| Recorded CLI lifetime, seconds | 286.349 | 309.522 | +| Completed CLI invocations | 6 | 7 | +| Requested executions | 5 | 6 | +| Allocated GPU-seconds | 100.203 | 124.993 | +| Input tokens | 1,455,546 | 1,724,856 | +| Cached input tokens, subset of input | 1,373,184 | 1,641,472 | +| Output tokens | 16,086 | 19,927 | +| Reasoning output tokens, subset of output | 4,236 | 4,579 | +| Input + output tokens | 1,471,632 | 1,744,783 | + +Input counts include reused context; they are processed tokens, not unique prompt +text. Cached input is already included in input, and reasoning output in output. +The native CLI does not expose underlying model API-request counts; six/seven CLI +invocations must not be reported as six/seven model calls. The captured model +flags record requested configuration, not an independent attestation of weights. + +The paired solver total is 225.196 allocated GPU-seconds. Independent grading +added **1,416.962 operator GPU-seconds**, over 62 new supervised phases, with +three earlier references reused. This cost is separate from solver costs and +all qualification/preflight costs above. Cleanup and device assignments are +verified for known allocation cost. These short runs include process startup +and cleanup; GPU-seconds do not measure active utilization or pure training. +No failed CLI attempt or transport recovery occurred in this pair. Queue-adjusted +wall removes only recorded shared-resource waits, not all host variability. + +The treatment used one extra ordinary DDP request, costing 25.086 GPU-seconds, +which completed 16 updates on both ranks without diagnostic hooks. Its preceding +single-GPU request ran two separate eight-update configurations within a +16-update allowance, not one uninterrupted 16-update run. The control's five +requests completed 2/8/8/8/8 updates. Both agents checked accumulation 1 and 2; +their terminal validation scopes were not identical. Exported checkpoints do +not establish checkpoint-restore testing. + +The control's reference requests already executed its submitted correction; +the treatment first captured an unchanged single-GPU reference before editing. +Both follow the documented service contract. Neither reference backend is a +hidden immutable corrected pipeline. + +## Interpretation and limits + +The treatment read the generic manual at turn 1, event 13. Review of all original +completed item streams, changed source occurrences and returned public +artifacts found no TrainCheck activation. Its diagnostics instead wrapped native +Trainer calls and installed ordinary optimizer hooks. This is an observation +about the captured evidence, not proof that an unobserved import never occurred. +Tool availability and manual reading are not instrumentation activation. + +Both agents found a real data-dependent distributed-training problem, but both +localized it before any requested execution. The observed case therefore fails +the intended runtime-required localization criterion. Runtime state still +confirmed the cause and quantified the gradient error. One optional-tool trial +per arm, with no observed activation and different chosen validation scopes, +cannot establish either a causal efficiency benefit or a slowdown from TrainCheck +availability. The timing table is a descriptive result, not an improvement claim. +This does not imply that agents can solve all bugs in the TrainCheck paper or +that agent-written training pipelines cannot contain silent errors. + ## Evidence and reproduction Local evidence root: @@ -202,6 +330,58 @@ Remote evidence root: `a79346c2664a1e63796b1557bbec76906d1c08784b69fcd6274515bbc692888f`: independent packet comparison and evidence review. + +- `solver-completion.json`, SHA-256 + `c733a9fab181e100e1a19fbcaa3e7c119eaa4c2f561d6e60d91e5a5c6e5d10c6`: + both terminal records; all raw turns, source snapshots and service receipts + remain under `trials/`. +- `milestone-annotations-visible.json` and `milestone-summary-visible.json`: + preserved adjudication before private grades, with valid-patch status pending. +- `milestone-annotations-final.json`, SHA-256 + `d4359e925180995d496f604155d054be1dbac97e8c864bee4979293cabc4c744`, + and `milestone-summary-final.json`, SHA-256 + `e38bd48dd379db775572423503517af64183a4a6153f147f92e329ffec8d04d2`: + final source/runtime/verification/finish events, within-turn token intervals, + exact submitted-turn totals and backend-specific GPU accounting. +- `visible-activation-audit.json`, SHA-256 + `cba7ba4be95832c70f033612e22e84ceefd3d6f44540bb8978e6b695a9bcda6f`: + changed-source hashes and returned public artifact inventories/text checks. +- `independent-grade-v1/summary.json`, SHA-256 + `a63142a6c82cfaff98457e213a896b454a43d16eedd291c5978345526f327afa`: + all independent outcomes, exact-map deduplicates and separate operator cost. + `independent-grade-export-manifest.json` binds 89 downloaded report files. +- `private-grading-join.json`, SHA-256 + `6ef62ad56251cec1c09fb86cbbb05138cf2c6b29dcc12c623fc70d0371d8d8b1`: + verifies all 13 original complete source maps and their turn/submission hashes + against the grading plan and per-candidate manifests, including all seven + scenarios and both final identities. +- `accounting/`: the preserved typed extractor and its provenance. Forty-three + focused synthetic tests and pinned strict mypy/format/lint checks passed during + preparation; the final extraction also revalidated actual receipt chronology, + native usage, backend allocation and evidence hashes. These checks do not + independently prove every scientific annotation or eliminate all harness bugs. + +Reproduce the final accounting from the local evidence root with a new output: + +```sh +PYTHONDONTWRITEBYTECODE=1 /Users/matt/miniconda3/envs/traincheck/bin/python \ + accounting/summarize_hf.py --study-root . \ + --annotations milestone-annotations-final.json \ + --output milestone-summary-reproduced.json +``` + +The local operator bundle `frozen-operator-20261102.tar.gz` contains the 35 files +listed in `frozen-operator-manifest.json`; SHA-256 is +`6dc24b4b154a4aedecf974d539cf17fd80fa0291bb8adf0b6a9c9eaadb45937d`. +It is not a claim that every checkpoint or later report is inside that archive. +Ring18 separately retains the existing runtime/source bundle +`runtime-reproduction-20261102.tar.gz`, SHA-256 +`2f7f4c5643b919951f5f2f4bbb32455ab1b4275364e04a8046c0bb5a19f113fb`, +created in place from material already on that machine. Automatic approval review +rejected uploading the full private operator bundle because export authorization +was not established. That upload was not retried; the local bundle and existing +remote runtime archive are the retained alternative. + The operator request service and qualification code do not modify the TrainCheck package. Large native sources, data, checkpoints and traces remain outside Git. The PR records reproducible evidence and limitations; it does not claim a general From 1e7019b44e0eef01ca9376b5f1838fe900170dbc Mon Sep 17 00:00:00 2001 From: Yuxuan Jiang Date: Mon, 2 Nov 2026 06:28:41 +0800 Subject: [PATCH 10/13] Checkpoint native FSDP source-coherence evidence and qualification limits --- .../fsdp-source-consistency-pilot-20261102.md | 187 ++++++++++++++++++ 1 file changed, 187 insertions(+) create mode 100644 docs/measurements/fsdp-source-consistency-pilot-20261102.md diff --git a/docs/measurements/fsdp-source-consistency-pilot-20261102.md b/docs/measurements/fsdp-source-consistency-pilot-20261102.md new file mode 100644 index 00000000..24865102 --- /dev/null +++ b/docs/measurements/fsdp-source-consistency-pilot-20261102.md @@ -0,0 +1,187 @@ +# Native FSDP initialization: optional TrainCheck pilot + +**Status: qualification in progress; no agent comparison launched.** The earlier +[tensor-parallel clipping](real-rank-runtime-pilot-20261102.md) and +[DDP token-normalization](ddp-runtime-state-pilot-20261102.md) comparisons both +produced source-first diagnoses and no observed TrainCheck activation. They are +retained negative results. This candidate tests an external runtime condition: +different workers can resolve the same model identifier to different cached +weights before native FSDP partitions the parameters. + +## Provenance and reproduction boundary + +[Halo release PR #103](https://github.com/whitecircle/halo/pull/103) adds agreement +on model source and commit across ranks. It is a production fix with regression +coverage, contained in a larger release; the PR does not link a specific incident +report. The parent is `8bc800971e739777311862544e5f1a321e12e4ba`, and the merged +release is `5abddcb7ee12b46481f6d0facb00046c358724fb`. + +The reproduction executes byte-identical selected parent definitions, including +the native cache-loading helpers and c10d-store coordination, bound to real +installed Hugging Face Hub, Accelerate and PyTorch APIs. Native GPTNeoX loading +and Torch FSDP perform the training. It does not run the complete current Halo +package: that package requires a newer Python/framework/CUDA stack than the +shared Ring18 installation. The original source and extraction hashes remain +available for review; no fake Hub API or replacement collective implements the +failure. + +The [official Pythia model card](https://huggingface.co/EleutherAI/pythia-14m#model-details) +documents replacement of the model previously served as `EleutherAI/pythia-14m`, +moving that earlier model to `EleutherAI/pythia-14m-deduped`. The experiment uses +unchanged official artifacts: + +| Artifact | Revision | Safetensors SHA-256 | +| --- | --- | --- | +| Earlier model, now `pythia-14m-deduped` | `f33025648652797a390d8c54835273845b437161` | `821667edaab2cb38aac634fd1bbda547ecf451f4ea514f47b5a0080be24154fb` | +| Replacement `pythia-14m` | `94f7c35d5e9f2e9bac8ca839329f505b4d007d5d` | `116a02532db461f91386a5b20f942ff2c8d4de7341e21b55caafc3d7b25f49a1` | + +These are separate repository histories, not ancestor/descendant commits. The +historical cache namespace and `refs/main` mapping are reconstructed around those +official blobs. This is not a preserved incident cache, and there is no claim +that the Halo incident involved Pythia. The model migration establishes an +authentic source of different published weights, not a record of that particular +deployment. + +Two worker-local cache domains emulate logical nodes on one physical Ring18 host, +using GPUs 4/5, both NVIDIA A16. Each worker has its own local rank zero and cache +metadata; global ranks and the actual NCCL process group remain distinct. This +does not measure multi-host networking or a filesystem race. Requests use +independent copies of the official caches, not writable links to shared source +blobs. The shared repositories, environment and driver are unchanged. + +## Workload and qualification + +The workload uses the official 14M-class GPTNeoX model, explicit native FP32 +loading, eager attention, no TF32, deterministic algorithms, AdamW with learning +rate `1e-4` and weight decay zero, sequence length 64 and two global examples per +update. The qualification uses two optimizer updates. Identical pretokenized +README-derived inputs isolate the model-loading question from the official +tokenizers' different postprocessing configurations. + +The raw checkpoints have different schemas: the replacement has 76 FP16 tensors; +the earlier artifact has 88 FP16 tensors and six Boolean tensors. The additional +18 entries include legacy attention buffers. Native loading reports no missing +or unexpected keys, but that alone is insufficient: the pinned loader can filter +buffer keys from its unexpected-key report, and `state_dict()` omits nonpersistent +buffers. The qualification therefore retains all raw keys and loading messages, +and separately captures complete named runtime buffers. Raw artifact equality +and runtime equality are different claims. + +The first complete source-state check confirms that all 76 learned tensors load +exactly as native FP16-to-FP32 conversion, and both models have 19 bitwise-equal +runtime buffers. Native loading recreates some values: the earlier raw +`masked_bias` is negative infinity while the runtime uses finite `-1e9`, and raw +FP16 rotary frequencies differ from their recreated FP32 values. These are +recorded loader behaviors, not values discarded to manufacture equality. + +The bounded matrix includes uniform earlier/replacement sources, mixed and +swapped sources, their one-GPU references, and same-world runs with additional +intermediate observations removed. A missing-cache control must produce the +native offline failure. Every run retains worker lifetime, allocation, cleanup, +source/cache integrity and raw failures. Native creation of `hub/version.txt` +initially caused an overly strict cache-integrity receipt to fail; that attempt +and its cost are retained, with immutable blob/ref checks separated from the +recorded metadata initialization. + +The 18 recorded qualification attempts through the full-buffer matrix consumed +435.376 allocated GPU-seconds, including the missing-cache control and failed +integrity receipt. This is operator qualification cost, not agent cost; later +side probes, service preflight and grading must be accounted for separately. + +Independent checks of the initial parameter-focused matrix established: + +- Each uniform initial model exactly matches its corresponding one-GPU source. +- Native FSDP constructs the mixed model from 38 parameter tensors belonging to + one source and 38 belonging to the other. The result differs from each complete + authentic source in 38 parameter tensors. The swapped case follows the actual + native shard ownership too; this is checked from values and padding metadata, + not inferred from filenames or a model hash alone. +- All ranks can reconstruct the same hybrid. Post-wrap replica equality alone + therefore does not establish coherent initialization. +- Same-world observed/unobserved final parameter, gradient, optimizer, buffer and + recorded RNG values agree exactly for the checked pairs. The follow-up checked + 104 overlapping artifacts against earlier runs, 30 buffer comparisons, 34 + replica checks and 20 terminal observer comparisons, all exactly equal. + +The original cross-world model tolerance **failed** for healthy uniform controls. +With model `atol=1e-5` and `rtol=1e-4`, maximum two-update differences were about +`3.758e-4` for the earlier model and `3.328e-4` for the replacement. Initial states +were exact. First-gradient relative L2 differences are approximately 0.5275% and +0.04506%, respectively. A key-bias-only explanation was not supported: most +first-step elementwise model violations occur elsewhere, including weights. +Measured gradient differences and Adam's update rule explain update sensitivity, +but the underlying cross-world gradient discrepancy remains unresolved. It is +not reported as ordinary rounding proved harmless. + +The failed criterion remains failed; no threshold was silently relaxed. The +operator explicitly selected a separate pre-agent scope: exact initial source +coherence and full same-topology healthy trajectories, without a claim of narrow +single-versus-two-GPU numerical parity. Its gate must bind both the successful +state checks and the retained failed original criterion. Two additional bounded +single-GPU microbatch-accumulation controls are planned to characterize batch +shape effects; they do not change the submitted workload or original results. + +## Planned comparison and acceptance + +The proposed neutral task extends a pretrained fine-tuning pipeline to two-GPU +FSDP, supplies ordinary short-run outputs, and requests a readiness decision and +justified revision. It does not assert a bug, prescribe a diagnosis or require +TrainCheck. This is pretrained fine-tuning, not checkpoint restoration with +optimizer and step state. + +Both arms receive identical editable runtime source, data, ordinary diagnostics +and execution budgets. Treatment adds the same generic optional TrainCheck +manual and implementation used in the earlier pair. Both current and reference +backends execute the submitted source. A private actual-entry observer and +independent evaluation must pass their own positive controls before launch. + +Uniform-source controls must retain their respective complete initial models. +Mixed/swapped inputs may be repaired to either authentic coherent source; there +is no hidden requirement to select the newest one. The subsequent full runtime +model, gradients, optimizer, input coverage and recorded RNG must follow that +source's independently qualified same-topology trajectory. Detecting a mismatch +and aborting is recorded as detection/containment, not as a completed working +repair under the public contract. + +Keep the previous diagnosis definitions. A specific pre-run explanation of +per-worker source disagreement and FSDP assembly counts as a source-supported +diagnosis even if the agent calls it a hypothesis. Later runtime confirmation +does not make that diagnosis runtime-first. If agents again solve the problem +from source, report it; do not force execution or tool use to make the case look +harder. + +The planned pair retains six requests of 1–16 global updates, 240 seconds per +worker request, 900 seconds aggregate CLI execution and eight decision turns. +Measure source diagnosis, runtime-supported diagnosis, first independently valid +submitted repair, visible verification and final completion separately. Within- +turn tokens remain intervals; GPU cost is supervised allocated-device lifetime, +including startup and cleanup. Qualification and private grading costs are +separate from solver costs. No tool efficiency benefit is established. + +## Saved evidence and current validation + +Local evidence root: +`/Users/matt/gitrepos/traincheck/outputs/ring18-source-consistency-20261102`. +Remote root: +`/home/yuxuan/experiments/traincheck-runtime-study-c-20261102`. + +The roots retain the official source/artifact provenance, original and revised +qualification specifications, all attempted native runs, and typed operator +helpers. `evaluation/source-proof-v1.json` records raw-to-loaded tensor and buffer +checks plus exact native shard reconstruction, SHA-256 +`380d01eef23cb4e0ecc589bcb4db9c1b42d2beb120c2786cf04eb29ca174dd38`. +`evaluation/native-review-v1.json`, SHA-256 +`15eaee07020af3c9a4a7472e4976aa4690cdecbc9d552649d1cc040f06e5e856`, +records complete observer/replica overlap and numerical differences. +`evaluation/original-crossworld-gate-v1.json`, SHA-256 +`8d7b0be8c6a16504b474eac002c649b2b0e22b416752f4e8700c173a86c92dc1`, +preserves the original failed cross-world criterion with unchanged thresholds. +`prelaunch-evaluation-design.md` records the diagnosis and repair policy; +`packet-draft-preparation.json` binds local task drafts. No formal solver packet +has been exported at this checkpoint. + +The new request service and pilot adapter passed 35 focused local tests, strict +mypy for their changed helpers, and pinned formatting/lint checks. Packet +preparation passed nine additional tests and strict checks. These are local +contract checks, not a substitute for the pending native service, optional-tool +parity and independent actual-entry gates. The TrainCheck package is unchanged. From cf116d45d6cdf7313314e48bfd816253d1aadcb1 Mon Sep 17 00:00:00 2001 From: Yuxuan Jiang Date: Mon, 2 Nov 2026 07:02:50 +0800 Subject: [PATCH 11/13] Freeze qualified FSDP agent comparison and explicit evaluation scope --- .../fsdp-source-consistency-pilot-20261102.md | 105 ++++++++++++++---- 1 file changed, 86 insertions(+), 19 deletions(-) diff --git a/docs/measurements/fsdp-source-consistency-pilot-20261102.md b/docs/measurements/fsdp-source-consistency-pilot-20261102.md index 24865102..bfc46b09 100644 --- a/docs/measurements/fsdp-source-consistency-pilot-20261102.md +++ b/docs/measurements/fsdp-source-consistency-pilot-20261102.md @@ -1,6 +1,6 @@ # Native FSDP initialization: optional TrainCheck pilot -**Status: qualification in progress; no agent comparison launched.** The earlier +**Status: qualification complete; one fresh matched agent pair is running.** The earlier [tensor-parallel clipping](real-rank-runtime-pilot-20261102.md) and [DDP token-normalization](ddp-runtime-state-pilot-20261102.md) comparisons both produced source-first diagnoses and no observed TrainCheck activation. They are @@ -51,7 +51,7 @@ blobs. The shared repositories, environment and driver are unchanged. ## Workload and qualification -The workload uses the official 14M-class GPTNeoX model, explicit native FP32 +The workload uses the official 14,067,712-parameter GPTNeoX model, explicit native FP32 loading, eager attention, no TF32, deterministic algorithms, AdamW with learning rate `1e-4` and weight decay zero, sequence length 64 and two global examples per update. The qualification uses two optimizer updates. Identical pretokenized @@ -110,20 +110,27 @@ were exact. First-gradient relative L2 differences are approximately 0.5275% and 0.04506%, respectively. A key-bias-only explanation was not supported: most first-step elementwise model violations occur elsewhere, including weights. Measured gradient differences and Adam's update rule explain update sensitivity, -but the underlying cross-world gradient discrepancy remains unresolved. It is -not reported as ordinary rounding proved harmless. +but do not identify its kernel-level numerical cause. It is not reported as +ordinary rounding proved harmless. The failed criterion remains failed; no threshold was silently relaxed. The operator explicitly selected a separate pre-agent scope: exact initial source coherence and full same-topology healthy trajectories, without a claim of narrow single-versus-two-GPU numerical parity. Its gate must bind both the successful state checks and the retained failed original criterion. Two additional bounded -single-GPU microbatch-accumulation controls are planned to characterize batch -shape effects; they do not change the submitted workload or original results. - -## Planned comparison and acceptance - -The proposed neutral task extends a pretrained fine-tuning pipeline to two-GPU +single-GPU controls used one sample per microbatch and two accumulation steps, +matching each distributed worker's forward batch shape. Both models then matched +the two-GPU parameters bitwise at steps 0, 1 and 2. All buffers and recorded RNG +values matched. The replacement model's gradients and optimizer state matched +exactly; the earlier model retained one subnormal gradient/moment discrepancy, +at most about `1.5e-43`, without a parameter difference. These controls consumed +24.690 allocated GPU-seconds. They localize the discrepancy to batch shape and +accumulation order, without proving which numerical kernel causes it. Neither +the submitted workload nor the original failed criterion changed. + +## Matched comparison and acceptance + +The neutral task extends a pretrained fine-tuning pipeline to two-GPU FSDP, supplies ordinary short-run outputs, and requests a readiness decision and justified revision. It does not assert a bug, prescribe a diagnosis or require TrainCheck. This is pretrained fine-tuning, not checkpoint restoration with @@ -132,8 +139,37 @@ optimizer and step state. Both arms receive identical editable runtime source, data, ordinary diagnostics and execution budgets. Treatment adds the same generic optional TrainCheck manual and implementation used in the earlier pair. Both current and reference -backends execute the submitted source. A private actual-entry observer and -independent evaluation must pass their own positive controls before launch. +backends execute the submitted source. Both ordinary eight-update service runs +passed, using 46.958 allocated GPU-seconds in total. Their genuine logs and all +16 curve rows are provided. These are cyclic-sample short-run training losses, +not held-out model-quality measurements. + +The private actual-entry observer and six-scenario evaluation passed before +launch: six baseline runs produced the expected four healthy and two incoherent +outcomes, and six positive controls passed after changing only +`sync_module_states=True`. Mixed and swapped positives selected different +authentic sources, so the evaluator does not require the latest version. These +12 qualification jobs consumed 328.727 allocated GPU-seconds, including the +previously qualified actual-entry mixed run once. + +The observer captures complete named parameters and buffers, optimizer and +recorded RNG values, local gradients, actual forward inputs and native shard +metadata. Step 0 is explicitly after the first backward and before the first +optimizer step. It supports multiple preliminary loads and unused optimizers, +binding the trained model through actual FSDP or stepped-optimizer identity. +Multiple root FSDP constructions or multiple actually stepped optimizers are +unsupported and must produce an unknown grade. The observer does not establish +live alias/storage identity beyond its recorded native layout. + +The optional-tool preflight compared two-update runs with and without selective +TrainCheck observation of scalar `dropout` arguments. Each rank executed 38 +native calls and emitted exactly 38 entry and 38 return records. All 42 saved +state comparisons matched, covering 2,050 tensor and 5,742 scalar checks, with +no unmatched, duplicate or variable-tracker records. An independent counter-only +run also matched the saved no-counter reference. The two new jobs used 69.336 +allocated GPU-seconds. This establishes compatibility for that interface; it is +not a repeated overhead benchmark or validation of arbitrary APIs, tensor +metadata, model tracking or the inference engine. Uniform-source controls must retain their respective complete initial models. Mixed/swapped inputs may be repaired to either authentic coherent source; there @@ -150,7 +186,7 @@ does not make that diagnosis runtime-first. If agents again solve the problem from source, report it; do not force execution or tool use to make the case look harder. -The planned pair retains six requests of 1–16 global updates, 240 seconds per +The pair retains six requests of 1–16 global updates, 240 seconds per worker request, 900 seconds aggregate CLI execution and eight decision turns. Measure source diagnosis, runtime-supported diagnosis, first independently valid submitted repair, visible verification and final completion separately. Within- @@ -158,6 +194,16 @@ turn tokens remain intervals; GPU cost is supervised allocated-device lifetime, including startup and cleanup. Qualification and private grading costs are separate from solver costs. No tool efficiency benefit is established. +An independent review of the exported packets found 3,790 shared files: all +except the optional-offer prompt are byte-identical. The treatment adds 32 pinned +tool/manual files. Both arms can edit the same 3,346 Python source files and have +the same fixed entry point, data and contracts. Original source, native names, +warnings and genuine tokenizer-revision provenance remain visible; they could +enable a source-first diagnosis. The review establishes no deliberately added +cause or issue hint, not that the case requires runtime observation. Both fresh +CLI agents use `gpt-6-astra`, high reasoning and priority service. GPU requests +are serialized, with queue time reported separately. + ## Saved evidence and current validation Local evidence root: @@ -176,12 +222,33 @@ records complete observer/replica overlap and numerical differences. `evaluation/original-crossworld-gate-v1.json`, SHA-256 `8d7b0be8c6a16504b474eac002c649b2b0e22b416752f4e8700c173a86c92dc1`, preserves the original failed cross-world criterion with unchanged thresholds. -`prelaunch-evaluation-design.md` records the diagnosis and repair policy; -`packet-draft-preparation.json` binds local task drafts. No formal solver packet -has been exported at this checkpoint. +`evaluation/accumulation-diagnostic-v2.json`, SHA-256 +`7a170941297b2a8a7bc2a394d21cb6dace3d0a1b616a8a5e7565d515a721c126`, +records the additional batch-shape controls. + +The frozen launch plan has SHA-256 +`c02a5cb735b52a00b918c5dd456415eadf7d97b4ee05f61f509da4dfc20fc4b2`. +It binds the packets, environment, operator code, native qualification, service, +actual-entry positive controls, optional-tool preflight and independent grading +contract. The independent neutrality review is +`prelaunch-neutrality-review.json`, SHA-256 +`ffe22a0d859f9e571fbd5e5ff1e53a605db4a87e6e643dbdf8204564788a3b36`. +The six-scenario grading contract has SHA-256 +`6abfeddd99d16aa109424f4cdec899708c3dfa78aa847efad785487fa517a43c`. + +After both agents finish, every submitted source occurrence will be frozen and +graded. Only identical complete source maps can share a result. Evaluation +compares ordered keys, shapes, dtypes and full tensor values against independent +same-topology oracles; it does not infer correctness from the patch syntax. +Missing evidence, malformed snapshots and unsupported observer coverage remain +unknown. The bounded validation covers two updates, one seed and one small +corpus, not generalization to other workloads. The new request service and pilot adapter passed 35 focused local tests, strict mypy for their changed helpers, and pinned formatting/lint checks. Packet -preparation passed nine additional tests and strict checks. These are local -contract checks, not a substitute for the pending native service, optional-tool -parity and independent actual-entry gates. The TrainCheck package is unchanged. +preparation passed nine additional tests and strict checks. The observer passed +six regression tests, the optional-tool preflight helpers passed 20, and the +reused accounting helpers passed 50, each with scoped strict typing and pinned +format/lint checks. These counts describe separate validation scopes and should +not be summed as unique tests. The native gates also passed under the explicit +same-topology scope above. The TrainCheck package is unchanged. From 3b0e895fb05ba661daf809885b5726b9ade52c3e Mon Sep 17 00:00:00 2001 From: Yuxuan Jiang Date: Mon, 2 Nov 2026 07:14:59 +0800 Subject: [PATCH 12/13] Record completed FSDP solver runs before independent repair grading --- .../fsdp-source-consistency-pilot-20261102.md | 38 ++++++++++++++++++- 1 file changed, 37 insertions(+), 1 deletion(-) diff --git a/docs/measurements/fsdp-source-consistency-pilot-20261102.md b/docs/measurements/fsdp-source-consistency-pilot-20261102.md index bfc46b09..99e953ea 100644 --- a/docs/measurements/fsdp-source-consistency-pilot-20261102.md +++ b/docs/measurements/fsdp-source-consistency-pilot-20261102.md @@ -1,6 +1,6 @@ # Native FSDP initialization: optional TrainCheck pilot -**Status: qualification complete; one fresh matched agent pair is running.** The earlier +**Status: both fresh agents finished; independent repair grading is in progress.** The earlier [tensor-parallel clipping](real-rank-runtime-pilot-20261102.md) and [DDP token-normalization](ddp-runtime-state-pilot-20261102.md) comparisons both produced source-first diagnoses and no observed TrainCheck activation. They are @@ -204,6 +204,42 @@ cause or issue hint, not that the case requires runtime observation. Both fresh CLI agents use `gpt-6-astra`, high reasoning and priority service. GPU requests are serialized, with queue time reported separately. +## Completed solver runs; independent grades pending + +Both trials finished without CLI or transport failures. Control submitted five +decision turns and used four GPU requests; optional TrainCheck submitted four +turns and used three requests. Every submitted source is retained. These totals +are observed execution costs, not yet a statement that either repair passed the +independent acceptance criteria. + +| Full trial | Control | Optional TrainCheck | +| --- | ---: | ---: | +| Raw elapsed seconds | 432.754 | 414.381 | +| Measured GPU-service queue seconds | 23.944 | 81.287 | +| Input tokens, including cached input | 1,358,537 | 1,127,302 | +| Cached input subset | 1,298,944 | 1,066,496 | +| Output tokens | 11,210 | 9,983 | +| Reasoning output subset | 1,910 | 2,931 | +| Input plus output tokens | 1,369,747 | 1,137,285 | +| Allocated solver GPU-seconds | 118.764 | 82.527 | + +Cached input is already included in input tokens, and reasoning is already +included in output tokens. Neither subset is added again. The seven solver +requests consumed 201.291 GPU-seconds. The five operator qualification groups +above total 905.087 GPU-seconds, counting reused actual-entry/no-counter evidence +only once. Private post-trial grading remains a separate cost. + +Both agents proposed an unsafe-initialization mechanism before the first run: +unsynchronized FSDP could assemble inconsistent rank-local tensors. However, +their specific proposed origin was RAM-efficient loading skipping nonzero-rank +weights. Actual runtime diagnostics found the loading flags absent and rejected +that explanation, while confirming cross-rank initial-state differences and +38 changed tensors after sharding. Both submissions enabled state synchronization +in turn 2. Neither final answer identified the different cached model sources. +The result must therefore distinguish a source-supported mechanism hypothesis, +runtime-confirmed state inconsistency, and an unresolved upstream cause. It is +not evidence of full root-cause localization from source or from runtime. + ## Saved evidence and current validation Local evidence root: From 55305542d195086f04526bf7ace8febe250bc0e1 Mon Sep 17 00:00:00 2001 From: Yuxuan Jiang Date: Mon, 2 Nov 2026 07:39:50 +0800 Subject: [PATCH 13/13] Report independently validated FSDP agent repairs and complete costs --- .../fsdp-source-consistency-pilot-20261102.md | 152 ++++++++++++++++-- 1 file changed, 140 insertions(+), 12 deletions(-) diff --git a/docs/measurements/fsdp-source-consistency-pilot-20261102.md b/docs/measurements/fsdp-source-consistency-pilot-20261102.md index 99e953ea..3f12f5b2 100644 --- a/docs/measurements/fsdp-source-consistency-pilot-20261102.md +++ b/docs/measurements/fsdp-source-consistency-pilot-20261102.md @@ -1,6 +1,9 @@ # Native FSDP initialization: optional TrainCheck pilot -**Status: both fresh agents finished; independent repair grading is in progress.** The earlier +**Status: completed; both final repairs pass all six independent scenarios.** +Both agents used cross-rank runtime diagnostics, but the optional-tool agent did +not activate TrainCheck. This single pair does not establish a tool efficiency +benefit. The earlier [tensor-parallel clipping](real-rank-runtime-pilot-20261102.md) and [DDP token-normalization](ddp-runtime-state-pilot-20261102.md) comparisons both produced source-first diagnoses and no observed TrainCheck activation. They are @@ -38,7 +41,7 @@ unchanged official artifacts: These are separate repository histories, not ancestor/descendant commits. The historical cache namespace and `refs/main` mapping are reconstructed around those official blobs. This is not a preserved incident cache, and there is no claim -that the Halo incident involved Pythia. The model migration establishes an +that the Halo failure mode involved Pythia. The model migration establishes an authentic source of different published weights, not a record of that particular deployment. @@ -49,6 +52,11 @@ does not measure multi-host networking or a filesystem race. Requests use independent copies of the official caches, not writable links to shared source blobs. The shared repositories, environment and driver are unchanged. +The execution environment uses Python 3.10.14, PyTorch `2.2.2+cu121` and compiled +CUDA 12.1. The loaded Transformers source is pinned to +`a17f287ac039f92835b5cd9bd8ee28b584c9f65e`; the full environment and executed +source-file hashes are saved with the launch evidence. + ## Workload and qualification The workload uses the official 14,067,712-parameter GPTNeoX model, explicit native FP32 @@ -204,13 +212,58 @@ cause or issue hint, not that the case requires runtime observation. Both fresh CLI agents use `gpt-6-astra`, high reasoning and priority service. GPU requests are serialized, with queue time reported separately. -## Completed solver runs; independent grades pending +## Completed comparison and independent repair validation Both trials finished without CLI or transport failures. Control submitted five decision turns and used four GPU requests; optional TrainCheck submitted four -turns and used three requests. Every submitted source is retained. These totals -are observed execution costs, not yet a statement that either repair passed the -independent acceptance criteria. +turns and used three requests. Every submitted source is retained. Nine ordered +source occurrences reduce to five exact complete source maps. All 30 independent +scenario replays finished: 26 passed, four failed as expected for initial +unrepaired sources, and none were unknown. Both earliest valid submissions are +turn 2, with no earlier unresolved source. Both final submissions pass all six. + +| Submitted source occurrences | Independent outcome | +| --- | --- | +| Control turn 1 | Four healthy cases pass; mixed and swapped sources fail | +| Control turns 2–3 | Six of six pass; earliest valid repair is turn 2 | +| Control turns 4–5, including final | Six of six pass | +| Optional TrainCheck turn 1 | Four healthy cases pass; mixed and swapped sources fail | +| Optional TrainCheck turns 2–4, including final | Six of six pass; earliest valid repair is turn 2 | + +The replay executes the immutable ordinary training entry, with complete source +integrity and state/input coverage checks. Both repairs change the training +pipeline only by enabling `sync_module_states` and adding comments. Standalone +diagnostic/comparison helpers differ across snapshots and were retained in their +complete source identities, even when not imported by that entry. + +| Milestone, seconds | Control raw | Control queue-adjusted | Optional TrainCheck raw | Optional TrainCheck queue-adjusted | +| --- | ---: | ---: | ---: | ---: | +| Pre-run mechanism hypothesis | 30.251 | 30.251 | 33.934 | 33.934 | +| Runtime-confirmed state inconsistency | 131.170 | 131.168 | 184.793 | 134.135 | +| First independently valid submitted repair | 148.365 | 148.362 | 204.705 | 154.047 | +| First agent-visible repair verification | 228.722 | 224.517 | 271.333 | 218.093 | +| Terminal trial boundary | 432.754 | 408.810 | 414.381 | 333.094 | + +Queue-adjusted time subtracts only measured shared-service waiting. Visible +statements use received-event timestamps, an upper bound on creation time. +First-valid submission time is retrospective; it excludes later private grading +latency and is distinct from the agent's own verification. Both agents' upstream +cache-origin diagnosis remains unresolved, as detailed below. + +| Cost through first valid submission | Control | Optional TrainCheck | +| --- | ---: | ---: | +| Native CLI elapsed seconds | 93.470 | 99.253 | +| Input tokens, including cached input | 714,921 | 683,299 | +| Cached input subset | 661,504 | 628,352 | +| Output tokens | 4,994 | 5,031 | +| Input plus output tokens | 719,915 | 688,330 | +| Allocated solver GPU-seconds | 35.563 | 34.992 | + +Native counters do not identify tokens at an intermediate statement. The +runtime-confirmation milestones therefore have cumulative input-plus-output +brackets of 293,535–719,915 for control and 304,371–688,330 for treatment. The +pre-run hypotheses occur within the first turn, giving only 0–293,535 and +0–304,371. No interpolation is used. | Full trial | Control | Optional TrainCheck | | --- | ---: | ---: | @@ -227,7 +280,23 @@ Cached input is already included in input tokens, and reasoning is already included in output tokens. Neither subset is added again. The seven solver requests consumed 201.291 GPU-seconds. The five operator qualification groups above total 905.087 GPU-seconds, counting reused actual-entry/no-counter evidence -only once. Private post-trial grading remains a separate cost. +only once. Private post-trial grading used another 815.932 GPU-seconds across +30 newly executed phases; reused frozen oracles added zero GPU work. These three +categories total 1,922.310 allocated GPU-seconds for this case. GPU reservations +were released after completion. Oracle self-checks ran on the CPU controller; +their preflight elapsed time was about 52.47 seconds, including imports and hash +validation, and is not reported as pure oracle-comparison time. + +The optional-tool agent read the TrainCheck manual but neither prepared nor +activated TrainCheck instrumentation. Both arms used ordinary custom diagnostic +wrappers, and all returned artifacts contained zero native TrainCheck API or +variable-trace files. Their validation choices differed: control compared eight +updates across backends and then ran 16 ordinary distributed updates; treatment +compared 16 diagnostic updates across backends. Control checked the 76 saved +parameter tensors, while treatment also checked nonpersistent named buffers, +covering 95 records. Full-trial cost differences therefore reflect different +chosen validation work, not matched measurements of a TrainCheck repair-speed +benefit. Both agents proposed an unsafe-initialization mechanism before the first run: unsynchronized FSDP could assemble inconsistent rank-local tensors. However, @@ -237,8 +306,10 @@ that explanation, while confirming cross-rank initial-state differences and 38 changed tensors after sharding. Both submissions enabled state synchronization in turn 2. Neither final answer identified the different cached model sources. The result must therefore distinguish a source-supported mechanism hypothesis, -runtime-confirmed state inconsistency, and an unresolved upstream cause. It is -not evidence of full root-cause localization from source or from runtime. +runtime-confirmed state inconsistency, and the upstream cause unresolved by +either agent. Operator qualification did establish the cache-version divergence. +This is not evidence of full root-cause localization by either agent from source +or from runtime. ## Saved evidence and current validation @@ -272,8 +343,36 @@ contract. The independent neutrality review is The six-scenario grading contract has SHA-256 `6abfeddd99d16aa109424f4cdec899708c3dfa78aa847efad785487fa517a43c`. -After both agents finish, every submitted source occurrence will be frozen and -graded. Only identical complete source maps can share a result. Evaluation +The completed grading summary is +`grading-results-v1/independent-grade-v1/summary.json`, SHA-256 +`9b2d524617f61d549aa59a948cf7767deb5736637e5f9c5815c91bc93f2b13c3`. +Its replay plan has SHA-256 +`2890a5e00cabbdd934c4f327e9c695c5e577a40fdcad3fa4c51c162f133eb5bf`. +All 165 exported reports/receipts were downloaded and hash-verified against +`grading-result-export-manifest.json`, SHA-256 +`3e67d014a07063c258e9a7205c1e0513da2661095d137195f7ac7997c383f060`. +Full tensor snapshots remain in the isolated Ring18 evidence directory. +`grading-reproduction-and-results.json` records exact source-to-occurrence +mapping, replay commands, per-scenario results and cost scope. +Independent export verification is saved in +`evaluation/grading/export-review-v1.json`, SHA-256 +`09d00a2e71f3b1515b3ebb169223fc9ba3610ed35f6b0bfd0b82bdc90720e493`. +The final visible-milestone/grade join is `private-grading-join.json`, SHA-256 +`07caa05c568cf069157cc973fc8dd56d106dd80fbc83aaa4c1507e0b5bf63add`. +`milestone-summary-final.json`, SHA-256 +`4729939543dbec89cb0feed9d0f98cbee0040bf4437494ba422f5becca1521ec`, +records the complete timing, token and GPU accounting. Earlier visible-only +annotations remain unchanged; private grades do not retroactively create +agent-visible milestones. + +A local operator checkpoint contains 158 code, contract, review and accounting +files in `operator-evidence-final-v1.tar.gz`, SHA-256 +`d47f9017f81bbd2da95cf78e2933ea30ba9e115623f23d52408886103f3a2f8f`. +Its verified file inventory is `operator-evidence-final-v1.json`. Large native +states and original solver source snapshots remain at their recorded locations. + +Both agents' completion and every submitted source occurrence were frozen before +grading. Only identical complete source maps shared a result. Evaluation compares ordered keys, shapes, dtypes and full tensor values against independent same-topology oracles; it does not infer correctness from the patch syntax. Missing evidence, malformed snapshots and unsupported observer coverage remain @@ -287,4 +386,33 @@ six regression tests, the optional-tool preflight helpers passed 20, and the reused accounting helpers passed 50, each with scoped strict typing and pinned format/lint checks. These counts describe separate validation scopes and should not be summed as unique tests. The native gates also passed under the explicit -same-topology scope above. The TrainCheck package is unchanged. +same-topology scope above. The post-trial grader passed 14 focused tests plus +pinned strict mypy, Ruff, Black and isort checks, independently rerun by a second +reviewer. Its negative-path tests cover incomplete cost receipts, duplicate +finishes, malformed snapshots and a confirmed incorrect scenario alongside an +unknown one. The TrainCheck package is unchanged. + +The recorded solver launch, from the local evidence root after qualification +and packet export, was: + +```sh +PYTHONDONTWRITEBYTECODE=1 /Users/matt/miniconda3/envs/traincheck/bin/python pilot.py freeze +PYTHONDONTWRITEBYTECODE=1 /Users/matt/miniconda3/envs/traincheck/bin/python pilot.py check-launch +PYTHONDONTWRITEBYTECODE=1 /Users/matt/miniconda3/envs/traincheck/bin/python pilot.py run +``` + +The independent replay uses the frozen plan on Ring18: + +```sh +/home/yuxuan/experiments/traincheck-token-normalization-20261102/venv/bin/python \ + /home/yuxuan/experiments/traincheck-runtime-study-c-20261102/evaluation/grading/grade.py \ + --plan /home/yuxuan/experiments/traincheck-runtime-study-c-20261102/grading-input-v1/plan.json \ + --execute-after-both-freezes +``` + +`grading-payload-allowlist.json` binds the staged execution inputs; +`grading-source-inventory.json` retains every original submission. The payload +contains public submitted source and minimal grading inputs, excluding raw CLI +transcripts and private model snapshots. Preserve the frozen directories; a new +repetition needs fresh isolated output roots and regenerated qualification and +launch manifests. Per-request receipts retain the exact native training commands.