Python: Prevent workflow checkpoints from being mutated outside of storage implementations

[Tao Chen (TaoChenOSU)]

Motivation & Context

Closes #7683 and #7685
Supersedes #7697, #7712, #7830, #7848

Description & Review Guide

• What are the major changes?
  • Checkpoint storage implementations should return deep copies of the stored checkpoints to prevent the owned checkpoints from being mutated outside of its boundary.
  • State export and import should also return and create deep copies of the state to prevent states from being mutated outside of its own boundary.
• What is the impact of these changes?
  • Workflow checkpoints and states are now isolated from mutations outside of their respective boundaries.
• What do you want reviewers to focus on?
  • If the checkpoint storage contract clear for developers.
  • If the default in memory checkpoint storage follows the contract correctly.

Related Issue

Fixes #7683, #7685

Contribution Checklist

• The code builds clean without any errors or warnings
• All unit tests pass, and I have added new tests where possible
• The PR follows the Contribution Guidelines
• This PR is linked to an issue and there is no other open PR for this issue (see Related Issue above).
• This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

[github-actions]

Python Test Coverage Report •

File	Stmts	Miss	Cover	Missing
packages/core/agent_framework/_harness
_loop.py	303	7	97%	493, 532, 626, 703, 746, 821, 977
packages/core/agent_framework/_workflows
_agent.py	324	32	90%	80, 84, 128–129, 278, 291, 358, 369, 371, 427, 438, 493, 563, 641, 671, 699, 746, 755, 816, 823, 829–830, 841, 873, 880, 901, 910, 914, 916–918, 925
_checkpoint.py	158	1	99%	311
_state.py	46	0	100%
TOTAL	48074	4494	90%

Python Unit Test Overview

Tests	Skipped	Failures	Errors	Time
9738	36 💤	0 ❌	0 🔥	2m 32s ⏱️

[Copilot]

Pull request overview

Strengthens Python workflow checkpoint isolation across state and storage boundaries.

Changes:

• Deep-copies state during export and restoration.
• Returns copied checkpoints from in-memory storage.
• Adds state, runner, and storage conformance tests.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
python/packages/core/agent_framework/_workflows/_state.py	Adds deep-copy state boundaries.
python/packages/core/agent_framework/_workflows/_checkpoint.py	Documents ownership and copies in-memory reads.
python/packages/core/tests/workflow/test_state.py	Tests nested state isolation.
python/packages/core/tests/workflow/test_runner.py	Tests storage-less restoration isolation.
python/packages/core/tests/workflow/test_checkpoint_storage_conformance.py	Adds backend ownership conformance tests.
python/packages/core/agent_framework/_workflows/_agent.py	Formatting-only change.
python/packages/core/agent_framework/_harness/_loop.py	Formatting-only changes.
python/packages/core/tests/core/test_harness_loop.py	Import-order-only change.

Suppressed comments (1)

python/packages/core/agent_framework/_workflows/_state.py:121

• The same custom __deepcopy__ behavior means restore isolation is still incomplete: importing a checkpoint containing Content/other SerializationMixin values copies raw_representation by reference, so later live-state mutation can mutate the caller's checkpoint. Please use a checkpoint-specific clone/sanitization policy rather than relying on generic deepcopy for the restoration boundary.

        self._committed.update(copy.deepcopy(state))

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

MAF Automated Review — Iteration 1

Result: Findings reported
Scope: full PR (1 commit(s)): a5062c6c7bc2
Model: gpt-5.6-sol

Overview

The review found 2 verified inline finding(s).

Reviewed the supplied pull-request change set across correctness, security/reliability, architecture, and failure behavior.
2 verified findings remained after source verification (2 medium) across 2 files. Details are attached to the affected lines below.

Affected areas: python/packages/core/agent_framework/_workflows/_checkpoint.py, python/packages/core/agent_framework/_workflows/_state.py

[github-actions]

MAF Automated Review — Iteration 2

Result: Findings reported
Scope: 2 net-new commit(s): 588d16565ae6, ae8cd879229b
Model: gpt-5.6-sol

Overview

The review found 2 verified inline finding(s).

Reviewed the supplied incremental change set across correctness, security/reliability, architecture, and failure behavior.
2 verified findings remained after source verification (2 medium) across 1 file. Details are attached to the affected lines below.

Affected areas: python/packages/core/agent_framework/_workflows/_state.py

[Patel Namraa (Namraa310806)]

Tao Chen (@TaoChenOSU) I also found and opened #7859 while working on the state/checkpoint behavior addressed in this PR.

#7859 is a separate issue: pending State writes from a failed or cancelled superstep are not discarded, so they can remain in the shared State and be committed by a later successful Workflow.run() on the same instance.

The underlying state consistency area is related to this PR, but the trigger and fix are different:

• Python: [Bug]: Checkpoint state is not isolated from live workflow state across restoration and storage boundaries #7683: checkpoint/live-state isolation across export/import and checkpoint storage boundaries.
• Python: [Bug]: Pending State writes from a failed superstep leak into a later successful run #7859: pending state from a failed or cancelled superstep surviving into a later run.

I can implement the #7859 fix separately with a focused regression test, or include it in this PR if you would prefer to keep the related state-consistency fixes together. #7859 is currently under triage, so please let me know which approach you prefer.

[Shivani . (Shivani767)]

Thanks for pulling this together, Tao Chen (@TaoChenOSU) — this is a solid consolidation of the ownership work from #7712 / #7697.

What looks good

• The CheckpointStorage method docs now state the copy-on-save / copy-on-read contract clearly, and InMemoryCheckpointStorage matches it on save, load, list_checkpoints, and get_latest.
• Moving conformance into test_checkpoint_storage_conformance.py (with both memory and file backends) is much easier to maintain than growing test_checkpoint.py further.
• Extending _conformance_checkpoint to cover nested mutables in state, messages, pending_request_info_events, and metadata addresses the earlier completeness gap.
• Isolating State at set / get / export_state / import_state is the right companion fix for #7685; the new state tests (including the get → mutate → set path) make the intended caller contract explicit.

Follow-up (already tracked)

• Agree with treating the intentional Content / SerializationMixin raw_representation aliasing under deepcopy as out of scope for this PR and tracking it in #7851.

Nit (non-blocking)

• The formatting-only edits in _harness/_loop.py and _workflows/_agent.py look unrelated to the ownership fix; fine if they came from repo formatters, otherwise they could be dropped to keep the diff focused.

LGTM from my side. Appreciate you taking the broader fix in one PR.

[Eduard van Valkenburg (eavanvalkenburg)]

do we really need to do a copy both on save and on read, that seems like a waste of time, easier to only do it on save then both this and the list methods do not need to copy, which is a bigger burden then on save.

[Shivani . (Shivani767)]

Good question — I don't think save-only is enough for the ownership contract this PR is targeting.

save deepcopy protects against the caller mutating the input after save. Without a second copy on load / list_checkpoints / get_latest, those APIs hand back the live object from _checkpoints. A caller can then mutate nested state / messages / etc., and the next load (or another concurrent reader) sees that corruption. That's the same class of bug as #7683 / #7685.

So the two copies defend different boundaries:

Side	Protects against
deepcopy on save	mutation of the object passed into storage
deepcopy on read APIs	mutation of the object returned from storage

If we only copied on save, we'd need a different contract (e.g. “returned checkpoints are borrowed / must not be mutated”), which is harder to enforce and easy to violate accidentally. File/Cosmos backends already get both sides “for free” via serialize/deserialize; in-memory has to be explicit on both.

Happy to hear if you'd rather document a borrowed-reference read contract instead — but for caller-owned snapshots, both copies look necessary. The list path is the hot one to watch for cost; if that becomes a real issue we could consider lazy/copy-on-write later without weakening the contract.