Repository navigation
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Agents previously had to implement model fingerprints and rank comparisons themselves before TrainCheck could help. This adds a typed, non-invasive model snapshot API, strict readers and pair/rank comparisons, plus runnable CPU/DDP examples that leave observation points and predicates to the caller.
The implementation reuses the existing exact SHA-256 tensor path and model-state enumeration. It preserves all registered parameter/buffer names, nonpersistent buffers and object aliases, and rejects incomplete rank coverage, malformed snapshots and unsupported tensors. It does not gather shards implicitly. Package exports now load lazily so importing the observation API does not install legacy PyTorch hooks; the four existing public exports retain their identities.
A regression also exposed a legacy proxy observer ignoring tracing suppression: capture produced four extra variable records and changed parameter timestamps. The narrow guard fix makes the same capture produce zero extra records and preserves timestamps; ordinary observations still execute.
Validation: 102 focused tests passed with five CUDA-only skips; configured Ruff, mypy, Black and isort hooks passed, with strict mypy for new helpers. The full local suite reports 1,074 passed, nine skipped, and two failures reproduced with the same assertions at parent 5530554. The two-process CPU example completes and compares all five tensors per rank. Native Ring18 current/reference snapshot parity now passes: 60 complete model/buffer/gradient/optimizer/input/RNG file comparisons and 665 independently checked tensor fingerprints across seven snapshots. Four preflight phases cost 95.853 allocated GPU-seconds, separately from the fresh optional-tool agent comparison that is being launched. No efficiency improvement is claimed yet. See docs/measurements/model-observation-workflow-20261102.md for exact scope and evidence.
Snapshots are synchronous complete-byte fingerprints, not zero-overhead tracing, numerical-distance measurements or atomic concurrent snapshots. Caller-selected full-model contexts are required for sharded training.
Depends on #171 (base codex/real-rank-runtime-pilot). Signed implementation: 498bf5f; native validation/protocol checkpoint: 4397d75. Do not merge automatically.