Skip to content

Add topology-portable trainer checkpoints - #784

Draft
bradhilton wants to merge 8 commits into
mainfrom
agent/topology-portable-checkpoints
Draft

Add topology-portable trainer checkpoints#784
bradhilton wants to merge 8 commits into
mainfrom
agent/topology-portable-checkpoints

Conversation

@bradhilton

@bradhilton bradhilton commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • replace separate trainable/frozen LoRA slots with checkpoint-based TrainerRank state
  • add exact canonical checkpoint serialization for LoRA + FP32 Adam state
  • restore checkpoints transactionally across supported TP/PP/EP/ETP/CP/DP topologies
  • pin and validate immutable base-model revisions through standard PEFT adapter metadata
  • add task/context checkpoint loading and service-neutral LoRA export
  • preserve logical checkpoint identities while allowing pinned local materialization
  • add bounded checkpoint-save bookkeeping and checkpoint performance harnesses

Validation

  • repository Ruff, format, ty, and lock hooks
  • exact 200-test Megatron lightweight CI slice
  • full 59-test canonical checkpoint codec suite
  • 605 backend/tinker CI-slice tests (3 skipped)
  • 1→2, 2→1, DP deduplication, PP, EP, ETP, and CP checkpoint tests
  • next-optimizer-step numerical equivalence and transactional failure coverage
  • GitHub quality, install smoke, and 2×H200 checks green on final commit

Companion

Remaining draft acceptance

The isolated cross-repo H200 acceptance/performance matrix and combined production-LOC constraint decision are still pending.

@bradhilton
bradhilton had a problem deploying to trainer-rank-gpu-validation August 11, 2026 05:40 — with GitHub Actions Error
@bradhilton
bradhilton had a problem deploying to trainer-rank-gpu-validation August 11, 2026 05:45 — with GitHub Actions Failure
@bradhilton
bradhilton temporarily deployed to trainer-rank-gpu-validation August 11, 2026 06:06 — with GitHub Actions Inactive
@bradhilton
bradhilton temporarily deployed to trainer-rank-gpu-validation August 11, 2026 07:03 — with GitHub Actions Inactive
@bradhilton
bradhilton temporarily deployed to trainer-rank-gpu-validation August 11, 2026 07:33 — with GitHub Actions Inactive
@bradhilton
bradhilton deployed to trainer-rank-gpu-validation August 11, 2026 07:55 — with GitHub Actions Active
@bradhilton
bradhilton had a problem deploying to trainer-rank-gpu-validation August 11, 2026 09:10 — with GitHub Actions Error
@bradhilton
bradhilton requested a deployment to trainer-rank-gpu-validation August 11, 2026 09:23 — with GitHub Actions In progress
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant