Skip to content

Measure agent debugging costs on distributed checkpoint continuation - #170

Open
Essoz wants to merge 8 commits into
codex/continuation-observation-pilotfrom
codex/checkpoint-continuation-costs
Open

Essoz wants to merge 8 commits into
codex/continuation-observation-pilotfrom
codex/checkpoint-continuation-costs

Conversation

@Essoz

@Essoz Essoz commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

The previous continuation pilot lacked measured debugging-efficiency costs. This adds a distributed checkpoint continuation case, fresh native CLI trials, strict accounting, and independent full-state evaluation.

Result: all four agents localized the layer-state binding error from source and submitted valid repairs before their first GPU request. Neither treatment agent activated TrainCheck. The completed experiment measures optional tool availability and establishes no stable efficiency benefit from TrainCheck.

Trial Localization (s) First valid submission (s) Tokens through first valid submission Whole-trial GPU-s
control_1 26.79 102.55 380,339 24.748
traincheck_1 38.01 111.12 334,186 24.330
control_2 34.35 99.39 304,915 24.901
traincheck_2 29.49 104.53 263,008 24.798

Tokens are cumulative input plus output, including cached input. Cached input and reasoning subsets are not added again. Localization timing is a received-event upper bound; repair timing is the earliest completed submission later passing all independent scenarios. All four first valid submissions consumed 0 GPU-s; the one remote diagnostic per agent happened afterward. The report separates cached/uncached input, raw/queue-adjusted wall time, final completion, and setup/grading costs.

Final approach and contracts

  • Neutral packets describe migration from replicated checkpoints to shared state plus per-rank layer shards and show measured curves. They do not announce a bug or provide a target invariant. Controls retain ordinary Python diagnostics; treatment adds only generic TrainCheck documentation/source.
  • Preserve existing checkpoint formats and model/data/optimizer configuration. Evaluate complete logical model/optimizer association, RNG/progress, rank/group metadata, and continued trajectories on fresh workers. An independently evaluated original implementation fails, while the known-correct control passes.
  • Record native cumulative CLI usage correctly across resume, exact source/input hashes, every request, measured shared-service waiting, failures, and unknown states. Freeze all four selections before private grading.
  • Keep required fields, runtime checks, and explicit interfaces. The implementation is a synthetic task fixture under tools; this PR does not change production tracing semantics.

Validation and limitations

  • 239 focused tests pass; pinned hooks and explicit strict mypy pass for 17 committed tool/test source files. Tools are checked explicitly because the hook alone skips them.
  • 16/16 candidate-by-scenario contracts pass, replayed locally: public two-rank checkpoint, two fresh two-rank seeds, and a fresh single-rank case for each implementation. All 25 private phases completed with unchanged frozen files. Both turns of each trial map to passing snapshots.
  • Scope/update audit covers 47 completed commands and eight edits. No observed scope/update violation or weakened production typing/contracts. Both treatments read the manual, but neither imported/activated TrainCheck; all trace directories are empty. The separate generic observation preflight succeeded and is excluded from treatment usage.
  • Two fresh matched pairs on Ring18 A16s, requested gpt-6-astra/high/priority. All 12 original severity-screen configurations failed and remain preserved. A separate manifest selects the learned checkpoint-contract case after calibration, before solver access, without changing the failed screen values. There is no large-regression qualification or general causal speedup claim.

Full workload/configuration, revisions, raw/final cost breakdown, reproduction, and limitations: measurement report.

Evidence is saved locally and on Ring18 in the recorded experiment directory. Archive final-evidence-v2.tar.xz: 3,113 evidence files plus manifest, 1,480 directories, all reopened and verified; remote hashes match. SHA-256: 829aa1424f9b1c05ccbabcb4e275411adb8d54db48b22c4e302202eb33d76b1b. Large artifacts and machine-specific operator scripts stay outside Git.

Depends on #169 (base codex/continuation-observation-pilot), itself stacked on #168. Eight signed commits checkpoint the work. Ready for review; no merge requested or performed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant