Skip to content

Pilot agent-authored invariants for distributed loss normalization - #168

Open
Essoz wants to merge 3 commits into
codex/parameter-observation-timefrom
codex/agent-authored-normalization
Open

Essoz wants to merge 3 commits into
codex/parameter-observation-timefrom
codex/agent-authored-normalization

Conversation

@Essoz

@Essoz Essoz commented Nov 1, 2026

Copy link
Copy Markdown
Collaborator

The distributed probe did not expose its actual normalization multiplier or window token counts to TrainCheck. This adds an opt-in native API observation at the actual loss multiplication and evaluates a separately authored Python invariant offline, making the missing objective contract directly checkable.

A fresh agent received the intended global-token mean objective, DDP semantics, and a human-selected scalar interface. It derived applied_scale ≈ world_size / global_tokens. The rule was frozen before native-trace evaluation; the final equation/tolerance were not tuned on faulty traces. This is one targeted authoring pilot, not built-in inference or autonomous bug discovery. Access isolation was instructional, not a formal blinded experiment.

The checker validates successful pre/post pairs, complete rank/update/microbatch coverage, and cross-rank count consistency. It preserves raw records and identities; malformed or incomplete evidence exits with an error and unknown violation count. The adapter retains the original multiplication and temporary tensor lifetimes. The predicate runs outside training.

Measured results

Final runtime revision: fe482640d2979481419040fe3395b894cb8600fd. Ring18, NVIDIA A16, PyTorch 2.2.2+cu121, FP32 5,504-parameter Transformer, four updates per run; one trial per configuration.

Configuration Completed rule checks Violations
Two GPUs, healthy, accumulation 1/2/3 8 / 16 / 24 0 / 0 / 0
Two GPUs, faulty local mean, accumulation 1/2/3 8 / 16 / 24 8 / 16 / 24
One GPU, local mean is correct, accumulation 2 8 0

All nine training phases completed. All three faulty configurations first violate at update 1 in the offline traces. The independent gradient oracle agrees with every healthy/faulty classification. Plain/observed complete checkpoint comparisons pass for healthy and faulty accumulation-2 runs: 4 rank-file pairs, 548 tensor pairs, matching whole-file hashes, without dropping fields.

Validation and limits

  • 113 focused tests pass locally and 113 on Ring18, including actual native tracing, gradients/RNG, exceptions, missing/duplicate evidence, and tensor lifetimes. The two lifetime regressions fail with the initial 26c77ce workload and pass after correction; preliminary runs are preserved separately and excluded from final claims.
  • Pinned pre-commit, explicit strict mypy for eight changed/new Python modules, and explicit test lint/format checks pass. Configured core mypy passes 79 files; legacy untyped-body coverage limits remain. No remote CI success is claimed.
  • 266 evidence files and 124 source identities verified after local retrieval. Archive includes raw API traces and checkpoints; SHA-256 c138f5e7e050939182ade7f60b79baad9a2c5833ca42ad4c36a237d6a549f09b.
  • Counts and reduction semantics remain supplied assumptions; the rule does not independently verify masks, custom DDP communication, or backward behavior. This tests one bug family and makes no performance, online latency, general detection-rate, or agent model-quality claim. Existing built-in inference and identity-related false alarms are unchanged.

Full commands, provenance, results, and limitations: docs/measurements/agent-authored-normalization-20261101.md.

Stack dependency: targets #167 (codex/parameter-observation-time), which depends on #166 and #164. This PR is not merged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant