Skip to content

[DurableExecution]: Invalid checkpoint token misclassified as terminal; execution fails instead of invocation #2571

Description

@yaythomas

Describe the bug

Amazon.Lambda.DurableExecution matches the stale checkpoint token rejection with a case-sensitive prefix check against "Invalid Checkpoint Token". The service emits "Invalid checkpoint token" (lowercase c and t). The check never matches, so a stale token is treated as a terminal 4xx and the whole durable execution fails instead of just the invocation.

Regression Issue

  • Select this option if this issue appears to be a regression.

Expected Behavior

When the service rejects a checkpoint call with InvalidParameterValueException because the checkpoint token is stale (a newer invocation superseded this one), IsTerminalCheckpointError returns false. The exception escapes to the host, Lambda retries the invocation, the service issues a fresh token, and the execution continues.

Current Behavior

IsTerminalCheckpointError returns true. The catch in DurableFunction.FunctionHandlerAsync returns Status = InvocationStatus.Failed. The execution is failed every time a stale token is seen.

Root cause, in Libraries/src/Amazon.Lambda.DurableExecution/DurableFunction.cs:

if (ex.ErrorCode == "InvalidParameterValueException"
&& ex.Message != null
&& ex.Message.StartsWith("Invalid Checkpoint Token", StringComparison.Ordinal))
{
return false;
}

if (ex.ErrorCode == "InvalidParameterValueException"
    && ex.Message != null
    && ex.Message.StartsWith("Invalid Checkpoint Token", StringComparison.Ordinal))
{
    return false;
}

The service constant (ValidationErrorMessages.INVALID_CHECKPOINT_TOKEN, internal) is "Invalid checkpoint token".

The existing test uses the SDK's string rather than the service's, so it cannot catch this:

new object[] { MakeServiceException("InvalidParameterValueException", HttpStatusCode.BadRequest, "Invalid Checkpoint Token: stale") },

Reproduction Steps

  1. Construct an AmazonServiceException with ErrorCode = "InvalidParameterValueException", StatusCode = 400, and message "Invalid checkpoint token: ..." (the real service message).
  2. Call IsTerminalCheckpointError with it (or run a workflow whose checkpoint flush throws it).
  3. Observe it returns true and the invocation output is Failed.

Possible Solution

  1. Change the literal to "Invalid checkpoint token", or use StringComparison.OrdinalIgnoreCase to protect against future drift.
  2. Update DurableFunctionTests to use the real service message.

Additional Information/Context

The same defect exists in the JS, Python, Java, Rust, and Go durable execution SDKs; each has its own issue. Links are in a comment below.

AWS .NET SDK and/or Package version used

Amazon.Lambda.DurableExecution 2.0.0 (master at 8ed93ce)

Targeted .NET Platform

Any

Operating System and version

Any

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions