Skip to content

fix(checkpoint): preserve state on failed saves - #197

Draft
DevChiniwala wants to merge 2 commits into
plexe-ai:mainfrom
DevChiniwala:fix/atomic-workflow-checkpoints
Draft

DevChiniwala wants to merge 2 commits into
plexe-ai:mainfrom
DevChiniwala:fix/atomic-workflow-checkpoints

Conversation

@DevChiniwala

@DevChiniwala DevChiniwala commented Nov 1, 2026 •

Copy link
Copy Markdown

Saving a checkpoint currently opens the published phase file with "w". A serialization error, disk write error or interruption after a partial write can therefore destroy the previous resumable checkpoint and leave invalid JSON at its public path.

Serialize to a temporary file in the same checkpoint directory, close it, and atomically replace the phase file only after writing succeeds. Clean up the temporary file on failure or interruption while retaining the existing return/logging behavior and checkpoint format. This protects publication of a complete checkpoint; it does not claim power-loss durability.

Validation on Python 3.12:

  • Reproduced all six partial-write failures against upstream a1e05f6: both first saves and overwrites, for I/O errors, serialization errors and KeyboardInterrupt.
  • All eight checkpoint regression/control cases pass, including replacement failure, successful overwrite and persisted Unicode feedback.
  • 35 tests pass across the checkpoint tests and existing model/search suites.
  • Black 25.12.0 and Ruff 0.14.9 checks pass on changed files.

Draft because the repository-prescribed staged integration run requires Anthropic credentials and Spark/Java, which were not available here. Provider/model training and staged integration have not been claimed as passing.

Repository workflow metadata: the patch version is bumped from 1.4.4 to 1.4.5 as required by the source-change version gate. poetry check --lock and the branch/title/version checks pass locally. Dependencies and the lockfile are unchanged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant