Repository navigation
[Bug]: onCheckpointComplete converts every operation on every checkpoint when no plugin is registered #795
Description
Summary
When a checkpoint response changes an operation, ExecutionManager.onCheckpointComplete converts every operation in the execution with PluginInfoConverter.toOperationChangeInfo, even when no plugin is registered. The cost grows with the number of operations, so 3,000 sequential steps take 18% to 27% longer than with the fix at 128 MB and show no measurable difference at 1024 MB.
Impact
At 1024 MB, the fix makes no measurable difference up to 3,000 steps. At 128 MB, Lambda's default memory size, executions of up to 1,000 steps show no difference either. At 2,000 steps they take about 8% longer, and at 3,000 steps about 22% longer. Only run time is affected.
Real-world impact is likely less noticeable. Real steps do more work than these no-op steps, so the conversion takes a smaller share of each step.
Repro
-
Create a durable function with the handler below. Use runtime
java21, 128 MB of memory, a 900 s timeout, and durable config{"ExecutionTimeout": 3600, "RetentionPeriodInDays": 1}. Put the compiled class at the root of the deployment package. Putaws-durable-execution-sdk-java2.2.1 and its dependencies underlib/. Set the handler to the class name.import software.amazon.lambda.durable.DurableContext; import software.amazon.lambda.durable.DurableHandler; public class SequentialSteps extends DurableHandler<SequentialSteps.Input, SequentialSteps.Output> { public record Input(Integer steps) {} public record Output(int steps, long sum) {} @Override public Output handleRequest(Input input, DurableContext context) { int n = input != null && input.steps() != null ? input.steps() : 3000; long total = 0; for (int i = 0; i < n; i++) { final int value = i; total += context.step("noop", Integer.class, stepCtx -> value); } return new Output(n, total); } }
-
Start an execution. The execution timeout is over 900 s, so the invocation must be asynchronous.
aws lambda invoke --function-name <function> --qualifier '$LATEST' \ --invocation-type Event --durable-execution-name run-1 \ --cli-binary-format raw-in-base64-out --payload '{"steps":3000}' out.json
-
When the execution finishes, compute
EndTimestamp - StartTimestampfromaws lambda get-durable-execution --durable-execution-arn <arn>. -
Repeat steps 1 to 3 with 1024 MB.
At 128 MB, the gap between consecutive StepSucceeded events grows from about 55 ms over steps 500 to 1,000 to about 70 ms over steps 2,000 to 3,000. The first 500 steps are slower because the JVM is still compiling the SDK code, which is unrelated to this issue.
Measurements
Total execution time for 3,000 steps in ap-northeast-1. In each run, both builds ran at the same time.
| SDK version | 128 MB, run 1 | 128 MB, run 2 | 1024 MB, run 1 | 1024 MB, run 2 |
|---|---|---|---|---|
| 2.2.1 | 211 s | 218 s | 113 s | 122 s |
2.2.1, toOperationChangeInfo skipped when no plugin is registered |
179 s | 172 s | 121 s | 120 s |
Proposed fix
In ExecutionManager.onCheckpointComplete, call PluginInfoConverter.toOperationChangeInfo and PluginRunner.onOperationChange only when durableConfig.getPluginRunner().isEmpty() is false. Leave status-change detection, operationStorage updates and waiter notification unchanged. DurableExecutor.fireOnInvocationStart already skips its operation snapshot when no plugin is registered. The JS SDK skips this conversion too.
Acceptance criteria
"No plugin registered" means the DurableConfig has no plugins, so getPluginRunner().isEmpty() is true.
- With no plugin registered,
onCheckpointCompletedoes not callPluginInfoConverter.toOperationChangeInfo. A unit test inExecutionManagerTestchecks this withMockito.mockStatic(PluginInfoConverter.class). - With no plugin registered,
onCheckpointCompletestill stores every operation from the response inoperationStorage. It also still completes the futures of operations that wait on them. Existing tests pass without changes to their assertions. - With at least one plugin registered,
onOperationChangestill fires once for each checkpoint response that changes at least one operation's status. ItsOperationChangeInfohas the sameupdatedOperationsandoperationscontents as before. Existing tests pass without changes to their assertions. mvn spotless:checkandmvn -pl sdk testpass.- No conformance tests are added or changed.
Activity
fix(sdk): skip plugin hook work without plugins