Repository navigation
Determine whether a large dynamics RL batch refusal is necessary or avoidable by execution layout #869
Description
Status reconciliation — September 11, 2026 (Schulman)
Open feasibility investigation; no incorrect refusal has been established. The exact historical replay remains blocked by missing fixture/profile state. Existing phased-execution comparisons do not prove a memory improvement or matching gradient execution.
Planned lane: Schulman and subagents, lower priority than reproducible #848/#870 boundaries. Recover an exact witness from retained evidence if possible; otherwise state that limit and use a separately identified representative case, without claiming reconstruction. Do not spend new GPU work trying to reproduce an unbound historical state.
Historical report (preserved):
Type: feasibility investigation, not a demonstrated incorrect refusal.
Owner: Schulman. Exact-witness replay is blocked by missing historical state; general planner work continues under #848.
Shannon's retail49 R0e run stopped on a TrainerRankMemoryError refusal. Preserved run-state v300 proves successful updates through batch 53; batches 54–56 contain rollout-only records and next_number is 57. Thus 57 describes driver progress, not a proven failed optimizer step. The guard raised before this attempted execution; this is distinct from a hard CUDA OOM.
The log reports 208288 packed / 568147 logical tokens, predicted 114.938 GiB against usable 63.101 GiB on one H200. Under the inspected source, token geometry describes the minimum wave's unsplit full-sharing plan, while the budget belongs to the final rejected split rung. At DP1, the matching 048 code makes one pair a top-level item, so this would be one indivisible pair, not the full eight-pair update. Historical ART/runtime-byte identity remains unconfirmed.
Preserve the exact failing batch/checkpoint and source/ART identities. Reconstruct top-level pair boundaries, per-history logical/packed lengths, request mix, sharing depth, cold/profiled state and original admitted/refused budget. Determine whether an indivisible input/request group forces the refusal, whether retained state is necessary, and whether a semantically identical execution schedule fits.
A reduced-history cap, changed pair weight or substitution of prepass values changes the experimental workload/objective; do not call such a change an equivalent execution fix. An auxiliary-first/generator-second backward schedule with one optimizer step is a candidate, not a qualified solution. If refusal is necessary, document the measured boundary and supported alternatives.
Evidence: /home/brad/shannon-logs/report-2026-09-09.md (R0e), associated run logs, and /home/brad/.local/share/schulman/retail49-program-20260909/analysis/separate-lora-phases-scope/REPORT.md. Cross-reference #848's broader cold/warm admission work. No GPU execution was performed to file this issue.
Current evidence gap: exact generated histories/tensors, selected profile/budget state and the pre-update G/aux state were not recovered. Queue indices and cal.log_trajectories metrics do not establish preserved raw inputs. No equivalent replay or necessary-refusal conclusion is claimed. Existing reconstruction: /home/brad/.local/share/schulman/memory-planning-audit-20260909/REPORT.md (SHA256 4c65c3c1e861eaf90286a19ecbdf43a53cdc138621184f2c3e8407f352b5fa95); concise evidence in adjacent comment-869-progress.md.
Activity
bradhilton commentedon Sep 9, 2026
Schulman: Read-only recovery narrows the R0e progress attribution. Its preserved W&B run-state (
v300) recordstrain/update_successful=1through batch 53. Batches 54–56 contain rollout/attempt metrics without successful-update metrics, and the iterator'snext_numberis 57. The source permits three outstanding batches. Thus “~step57” describes driver progress; it does not identify the failed optimizer update. FIFO execution suggests batch54 as the first pending update, but that still needs original queue attribution. The stored state preserves ordered input task indices for those batches, not the generated trajectory/tensor bytes or pre-update trainer state.Two other provenance corrections: ART prints the refusal budgets in GiB (
114.938required /63.101usable). Under the inspected ART source,208288packed /568147logical tokens describe the minimum wave's unsplit full-sharing plan, while the budget comes from the final rejected split rung. The matching 048 source builds one top-level item per pair; at DP1 that geometry would describe one indivisible pair rather than the entire eight-pair update. W&B metadata identifies Caladan9a43c622; its 048 file matches the inspected source bytes, but historical ART/runtime-byte identity remains to be confirmed.cal.log_trajectoriesrecords metrics, so that call does not establish that raw trajectories were persisted. Source/state receipts remain private. No inference, tensor reconstruction, GPU replay, objective change, or feasibility claim was made in this recovery pass. Exact generated histories, selected profile/budget state, and the pre-update G/aux state remain the blockers to an equivalent replay.bradhilton commentedon Sep 10, 2026
Independent qualification update: the completed capped-retail joint and phase806f runs both start from the exact same260b step2 pair, including byte-identical full generator optimizer receipts, and both complete update/grouped-save/full saved-probe execution. This is a different preserved fixture from the unrecovered historical refusal above.
Their generator updates are not numerically equivalent. Read-only reconstruction from the three immutable saved first-moment states covers58,869,760 scalars and recovers logged norms0.004193918 versus0.002147072 within1.8e-7 relative; full-vector cosine is0.480404. That excludes a pure uniform scaling explanation. Actual native packing differs (five mixed-slot G backwards versus three single-slot G backwards), while the loss expression and final1/8 scaling are unchanged. No causal kernel/phase bug is established. Recorded training peak reserved is143241773056bytes in both; allocated peak is128158736896 versus129084506624, so there is no demonstrated training peak-memory gain. Identical probe metrics use a historical fixed checkpoint and do not evaluate the newly updated G.
Report
/home/brad/.local/share/schulman/retail-phase-independent-review-20260910/REPORT.md, SHA256b429e6470fb4b5aebdf81ef994f0acd6787931ead34308f4fbf2c21212cc1a4b; manifest20a5676160e058396143a29a65617ea9f37320a7b83755b4a1ec322f12a60d5e. Four native CPU AdamW inversion controls pass. Next staged control uses the same806f source with separate_lora_phases=False and the same original inputs/optimizers, to separate phase-option effects from the prepass-source refactor. No launch authorized; no resolution or generic planner-fix claim.bradhilton commentedon Sep 10, 2026
Schulman — bounded three-way execution comparison (September 10)
An original joint pass, a same-source joint control, and an auxiliary-first/G-second pass all completed an eight-pair update, grouped save and full fixed 128-source probe on one H200. Initial adapters, heads, optimizer state and input tensors match. All three owned lifecycles are independently cleaned. This is a representative fixture, not the missing historical R0e state.
Current generator-gradient norms are 0.004193918 / 0.004082668 / 0.002147072. Cosine similarity is 0.9714 between the two joint executions, versus 0.4890 between the same-source joint and phased executions. Actual saved FP32 master updates have cosine 0.9964 and 0.9830 respectively; historical optimizer momentum makes those closer than current gradients. These values are reconstructed from retained optimizer evidence, not inferred from aggregate loss.
The same-source control removes a source-refactor difference, but executed no-grad prepass widths already differ ([1,7] versus [1,6,1]) before the phase switch. Thus the experiment does not establish phase-induced bias, gradient parity, or an objective improvement. Training peak reserved memory is unchanged at 143.24 GB in the original joint/phase comparison; no training memory gain is demonstrated. Identical probe results score a separate historical checkpoint/frozen features, so they do not establish updated-generator quality.
Report:
/home/brad/.local/share/schulman/joint806f-phase-comparison-20260910-carver/THREE-WAY.md, SHA2566d4a1bd1df0873088c884e68f6194c7229381302f82638091609b1116cdd0b5d. Final manifestd6ed31fd4b81a1c6bf1a7be65ea07b94f890ba6d424a5a8cfc8a85fba38dacc2. Disposable downloaded optimizer files were removed after verification; compact evidence remains durable.The next useful isolation would require matching executed tensor/layout identities in one actor before optimizer updates. That is only a staged proposal. The historical witness gap remains, and no generic memory fix or phased research adoption is claimed.