Skip to content

[Bug]: onCheckpointComplete converts every operation on every checkpoint when no plugin is registered #795

Closed

Description

@yaythomas

Summary

When a checkpoint response changes an operation, ExecutionManager.onCheckpointComplete converts every operation in the execution with PluginInfoConverter.toOperationChangeInfo, even when no plugin is registered. The cost grows with the number of operations, so 3,000 sequential steps take 18% to 27% longer than with the fix at 128 MB and show no measurable difference at 1024 MB.

Impact

At 1024 MB, the fix makes no measurable difference up to 3,000 steps. At 128 MB, Lambda's default memory size, executions of up to 1,000 steps show no difference either. At 2,000 steps they take about 8% longer, and at 3,000 steps about 22% longer. Only run time is affected.

Real-world impact is likely less noticeable. Real steps do more work than these no-op steps, so the conversion takes a smaller share of each step.

Repro

  1. Create a durable function with the handler below. Use runtime java21, 128 MB of memory, a 900 s timeout, and durable config {"ExecutionTimeout": 3600, "RetentionPeriodInDays": 1}. Put the compiled class at the root of the deployment package. Put aws-durable-execution-sdk-java 2.2.1 and its dependencies under lib/. Set the handler to the class name.

    import software.amazon.lambda.durable.DurableContext;
    import software.amazon.lambda.durable.DurableHandler;
    
    public class SequentialSteps extends DurableHandler<SequentialSteps.Input, SequentialSteps.Output> {
        public record Input(Integer steps) {}
    
        public record Output(int steps, long sum) {}
    
        @Override
        public Output handleRequest(Input input, DurableContext context) {
            int n = input != null && input.steps() != null ? input.steps() : 3000;
            long total = 0;
            for (int i = 0; i < n; i++) {
                final int value = i;
                total += context.step("noop", Integer.class, stepCtx -> value);
            }
            return new Output(n, total);
        }
    }
  2. Start an execution. The execution timeout is over 900 s, so the invocation must be asynchronous.

    aws lambda invoke --function-name <function> --qualifier '$LATEST' \
      --invocation-type Event --durable-execution-name run-1 \
      --cli-binary-format raw-in-base64-out --payload '{"steps":3000}' out.json
  3. When the execution finishes, compute EndTimestamp - StartTimestamp from aws lambda get-durable-execution --durable-execution-arn <arn>.

  4. Repeat steps 1 to 3 with 1024 MB.

At 128 MB, the gap between consecutive StepSucceeded events grows from about 55 ms over steps 500 to 1,000 to about 70 ms over steps 2,000 to 3,000. The first 500 steps are slower because the JVM is still compiling the SDK code, which is unrelated to this issue.

Measurements

Total execution time for 3,000 steps in ap-northeast-1. In each run, both builds ran at the same time.

SDK version 128 MB, run 1 128 MB, run 2 1024 MB, run 1 1024 MB, run 2
2.2.1 211 s 218 s 113 s 122 s
2.2.1, toOperationChangeInfo skipped when no plugin is registered 179 s 172 s 121 s 120 s

Proposed fix

In ExecutionManager.onCheckpointComplete, call PluginInfoConverter.toOperationChangeInfo and PluginRunner.onOperationChange only when durableConfig.getPluginRunner().isEmpty() is false. Leave status-change detection, operationStorage updates and waiter notification unchanged. DurableExecutor.fireOnInvocationStart already skips its operation snapshot when no plugin is registered. The JS SDK skips this conversion too.

Acceptance criteria

"No plugin registered" means the DurableConfig has no plugins, so getPluginRunner().isEmpty() is true.

  1. With no plugin registered, onCheckpointComplete does not call PluginInfoConverter.toOperationChangeInfo. A unit test in ExecutionManagerTest checks this with Mockito.mockStatic(PluginInfoConverter.class).
  2. With no plugin registered, onCheckpointComplete still stores every operation from the response in operationStorage. It also still completes the futures of operations that wait on them. Existing tests pass without changes to their assertions.
  3. With at least one plugin registered, onOperationChange still fires once for each checkpoint response that changes at least one operation's status. Its OperationChangeInfo has the same updatedOperations and operations contents as before. Existing tests pass without changes to their assertions.
  4. mvn spotless:check and mvn -pl sdk test pass.
  5. No conformance tests are added or changed.

Activity

  1. self-assigned this
    on Oct 16, 2026
  2. added a commit that references this issue on Oct 16, 2026
    67bad28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingneeds-triageIssue needs triagepkg:sdkModule: sdk

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions