Skip to content

Correct parameter observation timestamps and diagnose normalization coverage - #167

Open
Essoz wants to merge 4 commits into
codex/ring18-distributed-probefrom
codex/parameter-observation-time
Open

Essoz wants to merge 4 commits into
codex/ring18-distributed-probefrom
codex/parameter-observation-time

Conversation

@Essoz

@Essoz Essoz commented Oct 31, 2026 •

Copy link
Copy Markdown
Collaborator

Parameter sampling reused the last Python attribute-assignment timestamp. In-place optimizer updates could therefore record changed parameter values at unchanged or backdated times, corrupting observation chronology and derived variable lifetimes.

Both subclass and legacy proxy samples now capture observation-start time for sample and selective-sample, preserving mutation bookkeeping, non-sampling event times, registry eligibility, suppression, and exceptions. Repeating inference/checking also exposed a separate coverage-observer bug: valid NumPy integer lifetime endpoints were rejected. A separate signed commit adds an exact, finite numeric conversion at that dataframe boundary; generic JSON validators and checker predicates remain unchanged.

Measured correctness improvement: on a fresh matched 2×A16, four-update Transformer trial, invalid after-update times fall from 216/216 to 0/216; equal before/after times despite changed hashes fall from 54/216 to 0/216. API/VAR record counts remain 7,504/594 in both versions, with no duplicate API event keys. Across six fixed subclass trials, 1,080/1,080 parameter/update pairs have complete samples and correct chronology.

All nine GPU phases complete successfully. All eight full checkpoint comparisons pass, covering 274 tensor pairs per comparison and matching whole-file hashes, including FP32/BF16, both resume directions, and the deliberately faulty normalization run. This is a correctness change, not a performance claim.

Validation:

  • Parameter regressions: old source 8 failed / 13 passed, candidate 21 passed; real SGD disk-trace regression also fails before and passes after.
  • Observer regressions: two real-pandas cases fail before; exact nanoseconds above 2**53, observer-on/off parity, missing/malformed values, booleans, nonfinite values, and datetime/duration rejection are covered.
  • Composed local suite: 142 passed. Ring18 suites: 87 passed for tracing/model/checkpoint behavior and 55 passed for coverage.
  • Pinned hooks pass. Configured mypy passes on 79 core files; strict scoped checks cover both new test modules, the changed tracing test, and the reusable coverage tool. Explicit test/tool lint/format checks pass. Legacy untyped bodies remain outside full checker coverage.

Fresh API+VAR checking finishes with 596/605 active rules, zero dormant/error/unknown outcomes. Healthy/fault violations are 9/8 on rank 0 and 15/17 on rank 1. Two rank-1 fault-only metadata-distinctness alarms do not establish normalization detection. The raw-data diagnosis shows the applied scale was not captured, numeric loss data was excluded, and existing rules do not encode global-token normalization. Healthy object-ID-based false alarms remain visible. Frozen inputs match; every previously completed per-rule report is exactly unchanged after the observer fix.

Full experiment report · Normalization coverage diagnosis

Limits: observation times do not imply exact mutation times or atomic snapshots; an explicit after-update sample remains outside the optimizer API interval. No FSDP, stochastic/dropout, changed topology, partial accumulation windows, or agent-benefit claim. Large evidence stays outside Git: the local archive has 185 hash-verified files; both source snapshots match their committed Python files exactly. Final audit finds no remaining study processes and idle GPUs.

Dependency: stacked on #166 (codex/ring18-distributed-probe), itself stacked on #164. All four commits are signed. No merge performed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant