fix(core): validate replay operation identity

[zhongkechen]

Summary

• validate checkpoint type, subtype, and name before operation-specific replay handling
• fail mismatches with NonDeterministicExecutionError instead of consuming another operation's checkpoint
• cover type, subtype, and name drift through unit and end-to-end regression tests

Testing

• 1,587 core non-e2e tests passed
• 46 core e2e tests passed
• hatch run dev-core:typecheck
• hatch fmt --check

Closes #692

[github-actions]

Codex AI review

Found two replay-determinism gaps involving virtual and FLAT/NESTED child contexts.

Reviewed commit be1d349f153bf74d61e1104745b9ae77513f0697. Workflow run

[yaythomas]

I behavior-tested this branch against the two JS-side fixes for the same defect family (aws-durable-execution-sdk-js#867 and aws-durable-execution-sdk-js#894, and this change works perfectly!

the divergence test below returns SUCCEEDED (silent checkpoint substitution, #692) on main and flips to FAILED carrying a typed NonDeterministicExecutionError with mismatch detail on this branch, including preserving the error type into the ErrorObject, which the JS side had to fix separately in #867.

Offering two example-level test files as complementary regression coverage (parity with the test matrices those JS PRs added). All five pass on this branch; the divergence case is red on main, so it directly witnesses this PR's fix. Maybe can add?

test/map/test_map_summarized_replay_matrix.py — parity with JS #894's FLAT/NESTED/small-result matrix:

"""Tests for map summarized replay across a suspension — parity with JS PR #894.

Matrix (mirrors packages/aws-durable-execution-sdk-js-examples map/flat-summarized-replay):
- FLAT nesting, >256KB result  -> ReplayChildren rebuild across a suspension (the bug path)
- NESTED nesting, >256KB result -> control: context checkpoints exist, rebuild trivially correct
- FLAT nesting, small result    -> control: stays within one checkpoint, no rebuild at all
"""

import pytest
from aws_durable_execution_sdk_python.config import MapConfig, NestingType
from aws_durable_execution_sdk_python.context import DurableContext
from aws_durable_execution_sdk_python.execution import (
    InvocationStatus,
    durable_execution,
)
from aws_durable_execution_sdk_python.waits import Duration

from test.conftest import deserialize_operation_payload

ITEM_COUNT = 8
LARGE_ITEM_PAYLOAD_BYTES = 40 * 1024  # 8 x 40KB ≈ 320KB > 256KB limit
SMALL_ITEM_PAYLOAD_BYTES = 100  # 8 x 100B — far below the limit


def make_handler(nesting: NestingType, payload_bytes: int):
    @durable_execution
    def handler(_event, context: DurableContext):
        items = list(range(ITEM_COUNT))

        batch = context.map(
            inputs=items,
            func=lambda ctx, item, index, _: ctx.step(
                lambda _s: {"item": item, "payload": "x" * payload_bytes},
                name=f"resolve-{index}",
            ),
            name="resolve-pages",
            config=MapConfig(nesting_type=nesting),
        )

        live_result_count = context.step(
            lambda _: len(batch.get_results()), name="record-live-count"
        )

        context.wait(Duration.from_seconds(86400), name="suspend")

        replayed_result_count = context.step(
            lambda _: len(batch.get_results()), name="record-replayed-count"
        )
        replayed_items = sorted(r["item"] for r in batch.get_results())

        return {
            "live_result_count": live_result_count,
            "replayed_result_count": replayed_result_count,
            "replayed_items": replayed_items,
        }

    return handler


HANDLERS = {
    "flat-large": make_handler(NestingType.FLAT, LARGE_ITEM_PAYLOAD_BYTES),
    "nested-large": make_handler(NestingType.NESTED, LARGE_ITEM_PAYLOAD_BYTES),
    "flat-small": make_handler(NestingType.FLAT, SMALL_ITEM_PAYLOAD_BYTES),
}


@pytest.mark.example
@pytest.mark.parametrize("case", ["flat-large", "nested-large", "flat-small"])
def test_map_summarized_replay_matrix(case):
    """The batch reconstructed after suspend/resume must equal the live batch."""
    from aws_durable_execution_sdk_python_testing.runner import (
        DurableFunctionTestRunner,
    )

    with DurableFunctionTestRunner(handler=HANDLERS[case]) as runner:
        result = runner.run(input='"test"', execution_timeout=30)

    assert result.status is InvocationStatus.SUCCEEDED, (
        f"[{case}] expected SUCCEEDED, got {result.status}: {result.error}"
    )
    output = deserialize_operation_payload(result.result)
    assert output["live_result_count"] == ITEM_COUNT, f"[{case}] live count wrong"
    assert output["replayed_result_count"] == ITEM_COUNT, (
        f"[{case}] replayed count {output['replayed_result_count']} != {ITEM_COUNT} — "
        "summarized-replay rebuild lost terminal items"
    )
    assert output["replayed_items"] == list(range(ITEM_COUNT)), (
        f"[{case}] replayed items {output['replayed_items']}"
    )

test/map/test_replay_validation_parity.py — parity with JS #867's replay-validation composed tests (divergence → FAILED with typed diagnostic + stable-replay control):

"""Non-deterministic replay fails closed with diagnostic — parity with JS PR #867.

Two cases, mirroring replay-validation.composed.test.ts:
1. Divergent replay (STEP checkpointed, CHAINED_INVOKE replayed at the same
   position) -> FAILED, error type preserved as NonDeterministicExecutionError,
   never PENDING.
2. Stable control (identical replay) -> SUCCEEDED.
"""

import pytest
from aws_durable_execution_sdk_python.context import DurableContext
from aws_durable_execution_sdk_python.execution import (
    InvocationStatus,
    durable_execution,
)
from aws_durable_execution_sdk_python.waits import Duration
from aws_durable_execution_sdk_python_testing.runner import DurableFunctionTestRunner

# Deliberately module-global: DurableFunctionTestRunner replays in the SAME
# process, so this flag survives across the suspend/resume boundary — which is
# exactly what makes the divergence reachable. First invocation takes the STEP
# branch; the replay invocation (same process, flag already set) takes the
# CHAINED_INVOKE branch at the same logical position. Against the real service
# the equivalent divergence arises from any replay-visible state change
# (e.g. control flow derived from a corrupted batch rebuild, see JS PR #894).
_STATE = {"diverged_once": False}


@durable_execution
def divergent_handler(_event, context: DurableContext):
    context.step(lambda _: "anchor", name="anchor")
    if not _STATE["diverged_once"]:
        _STATE["diverged_once"] = True
        context.step(lambda _: "first-run-step", name="divergent")
        context.wait(Duration.from_seconds(86400), name="suspend")
    else:
        context.invoke(function_name="some-worker", payload={"x": 1}, name="divergent")
    return {"reached_end": True}


@durable_execution
def stable_handler(_event, context: DurableContext):
    context.step(lambda _: "anchor", name="anchor")
    context.step(lambda _: "stable-step", name="stable")
    context.wait(Duration.from_seconds(86400), name="suspend")
    context.step(lambda _: "post", name="post")
    return {"reached_end": True}


@pytest.mark.example
def test_divergent_replay_fails_closed_with_diagnostic():
    """Type mismatch on replay -> FAILED carrying NonDeterministicExecutionError."""
    _STATE["diverged_once"] = False
    with DurableFunctionTestRunner(handler=divergent_handler) as runner:
        result = runner.run(input='"test"', execution_timeout=30)

    # Never PENDING for a permanent fault (the #867 misclassification).
    assert result.status is not InvocationStatus.PENDING
    assert result.status is InvocationStatus.FAILED, (
        f"Expected FAILED, got {result.status}"
    )
    # Diagnostic must reach the caller with the error type preserved
    # (JS #867 fixed exactly this: bare Error instead of the typed error).
    assert result.error is not None, "FAILED result carries no error object"
    assert "NonDeterministicExecutionError" in (result.error.type or ""), (
        f"Error type not preserved: {result.error.type}"
    )
    assert "type checkpoint='STEP'" in (result.error.message or ""), (
        f"Diagnostic lacks mismatch detail: {result.error.message}"
    )


@pytest.mark.example
def test_stable_replay_control_succeeds():
    """Identical replay across a suspension must succeed (control case)."""
    with DurableFunctionTestRunner(handler=stable_handler) as runner:
        result = runner.run(input='"test"', execution_timeout=30)
    assert result.status is InvocationStatus.SUCCEEDED, (
        f"Control case failed: {result.status}: {result.error}"
    )

Verification on this branch (a2233e9):

test/map/test_map_summarized_replay_matrix.py::test_map_summarized_replay_matrix[flat-large] PASSED
test/map/test_map_summarized_replay_matrix.py::test_map_summarized_replay_matrix[nested-large] PASSED
test/map/test_map_summarized_replay_matrix.py::test_map_summarized_replay_matrix[flat-small] PASSED
test/map/test_replay_validation_parity.py::test_divergent_replay_fails_closed_with_diagnostic PASSED
test/map/test_replay_validation_parity.py::test_stable_replay_control_succeeds PASSED

Same five on main: the divergence case fails with SUCCEEDED (the #692 silent substitution), the other four pass — confirming this PR is the behavior change under test.

Both concerns were addressed in 687928a: the incomplete inference and O(started × operations) scan were removed, and regressions verify ambiguous histories preserve STARTED with zero operation-map reads. Requesting a fresh review of the updated PR.