feat(extras): add the microvm operation by ParidelPooya · Pull Request #769 · aws/aws-durable-execution-sdk-python

ParidelPooya · 2026-10-13T20:22:14Z

Adds a new package, aws-durable-execution-sdk-python-extras, with the microvm operation. It runs one job in a new AWS Lambda MicroVM and returns the job's result. It is the Python counterpart of microvm() in @aws/durable-execution-sdk-js-extras.

The code in the MicroVM is the worker from #763 and #767, or the JavaScript worker. The job document is the same in both.

This is the first of two PRs for microvm. It delivers the job in the run lifecycle hook. The next PR adds delivery over HTTP for jobs that do not fit there.

Usage

from aws_durable_execution_sdk_python.config import Duration
from aws_durable_execution_sdk_python_extras.microvm import MicrovmConfig, microvm

CONFIG = MicrovmConfig(
    image_identifier=IMAGE_ARN,
    execution_role_arn=MICROVM_ROLE_ARN,
    timeout=Duration.from_hours(1),
    heartbeat_timeout=Duration.from_minutes(1),
)


@durable_execution
def handler(event, context):
    return microvm(context, "build", job_input={"repo": event["repo"]}, config=CONFIG)

What it does

The operation runs its durable operations in one child context named name:

<name>.callback creates a durable callback. The callback ID identifies the job.
<name>.launch calls RunMicrovm, with the job in the runHookPayload.
- The client token is the SHA-256 hex digest of the callback ID. So a retried or replayed launch returns the same MicroVM.
- The step checkpoints the MicroVM ID and the delivery mode (run-hook).
The child context waits on the callback. The invocation ends during the wait.
<name>.terminate calls TerminateMicrovm after success, failure, and timeout. A terminate failure is logged as an error, because the job's outcome is already recorded.

The config is validated before any durable operation starts:

An invalid config raises ValueError.
An input that is not JSON-serializable raises TypeError.

Errors

Every failure is a MicrovmError: MicrovmLaunchError, MicrovmDeliveryError, MicrovmJobFailedError, or MicrovmTimeoutError.

Inside the child context, the operation raises one of these classes.
The SDK checkpoints the class's qualified name as the error type.
The module registers each class with register_operation_error.
So the SDK rebuilds the class as the __cause__ of ChildContextError, on the first run and on replay. The operation raises that cause.
A caller's own child context, or a map item, gets the MicroVM error as __cause__ too.

MicrovmJobFailedError names the error type that the MicroVM reported, for example the job failed (RuntimeError): 3 tests failed. CallbackExternalError does not carry that type. So the operation reads it from the callback's checkpoint, through callback.state. That is internal SDK state, so the core dependency has an upper bound: >=2.0.0,<3.

Where Python needs different code from JavaScript

Suspension. callback.result() raises SuspendExecution, a BaseException, while it waits. A JS promise stays pending instead. So the terminate step runs from except Exception. A finally would terminate the MicroVM while it still runs the job.
Keyword-only arguments. job_input and config are keyword-only. A job and a name can both be strings. So swapped positional arguments would pass validation and run the wrong job.
Callback serdes. The default callback serdes returns the raw string that the MicroVM sent. So the operation sets JsonSerDes().
Code points. A Python str has one item per code point, so len() counts what RunMicrovm counts. Lone surrogates are escaped one by one, as JSON.stringify does.
Retry strategy. botocore errors have no $retryable marker. So default_microvm_retry_strategy retries by error code, by HTTP status 429 and 5xx, and by connection and timeout errors. It allows at most 5 attempts, with a delay of at most 60 seconds.

Not in this PR

Delivery over HTTP. A job whose run hook payload exceeds 4,096 code points fails with MicrovmDeliveryError before the launch. Delivery over HTTP comes in the next PR.
- The launch checkpoint already records the delivery mode, so that PR does not change the checkpoint's shape.
- A checkpoint with a delivery mode that this release does not know terminates the MicroVM and fails.
microvm_session. It comes in a later PR.
MicroVM subtypes. The operations record the SDK's default subtypes. On replay, the SDK compares each operation's subtype with its checkpoint, and a mismatch raises NonDeterministicExecutionError. So changing the subtypes later would fail every execution in flight. So the package will be published only after the SDK accepts custom subtypes (fix(sdk): load state with unlisted subtypes #762 and a follow-up), with the MicroVM subtypes from its first release. The package is not in pypi-publish.yml.

Repository wiring

The new package is under packages/aws-durable-execution-sdk-python-extras.
The dev-extras hatch env has test, cov, and typecheck scripts.
The types env now installs boto3-stubs[lambda,lambda-microvms].
The package is added to testpaths, to ruff's known-first-party, to .github/scripts/type-checks.sh, and to AGENTS.md.

Testing

hatch run dev-extras:cov: 80 tests, 100% line and branch coverage.
The operation tests run on DurableFunctionTestRunner, with a fake MicroVMs client. They cover:
- success, job failure, and timeout;
- a terminate failure, and ResourceNotFoundException from terminate;
- a throttled launch that keeps its client token;
- a rejected launch;
- a job too large for the run hook;
- a nested child context, and map;
- first-run against replay.
worker_composed_test.py runs the real Python worker in-process. The fake RunMicrovm sends the run hook to the worker, and the worker reports through a client that forwards to the runner.
A Stubber test checks the RunMicrovm and TerminateMicrovm requests against the botocore service model.
Each test added in review fails when its fix is reverted.
hatch run test:cov: 4,778 passed, 98% coverage. hatch run types:check, hatch fmt --check, and lintcommit.py pass.
Not run against AWS. Integration tests come with the examples PR. They need the MicroVM resources in the Python test account.

Add the aws-durable-execution-sdk-python-extras package with microvm(context, name, job_input, config). It runs one job in a new AWS Lambda MicroVM, in one child context: 1. A durable callback whose ID identifies the job. 2. A launch step that calls RunMicrovm with the job in the run hook payload. The client token is the SHA-256 of the callback ID, so a retried or replayed launch returns the same MicroVM. 3. A wait on the callback, which the worker in the MicroVM completes. 4. A terminate step after success, failure, and timeout. A terminate failure is logged, because the outcome is already recorded. Failures raise MicrovmLaunchError, MicrovmDeliveryError, MicrovmJobFailedError, or MicrovmTimeoutError. The operation rebuilds the class from the child context's checkpointed error type, so the first run and every replay raise the same class. A job too large for the run hook fails with MicrovmDeliveryError. HTTP delivery for larger jobs, and the session operation, follow in later changes. The operations record the default subtypes until the core SDK accepts custom ones.

- Register the MicroVM error classes with the SDK's error registry. A caller's own child context, or a map item, now rebuilds the MicroVM error as the __cause__ of ChildContextError. - Take job_input and config by keyword only. A job and a name can both be strings, so swapped positional arguments ran the wrong job. - Terminate the MicroVM and fail when the launch checkpoint names a delivery that this release does not know. - Keep the stack trace of the launch step and of the MicroVM's report. - Escape only lone surrogates in the run hook payload, as JSON.stringify does. One surrogate no longer escapes the whole input. - Depend on the core SDK >=2.0.0,<3. 2.0.0 already has the APIs, and the bound guards one read of internal state. - README: publish only with MicroVM subtypes, because a later subtype change fails executions in flight. Document the cases in which no terminate runs, and when MicrovmLaunchError can leave a MicroVM. - Fix the client token and terminate comments.

- The default lambda-microvms client has a 5-second connect timeout, a 30-second read timeout, and 3 attempts in standard retry mode. botocore's defaults are 60 seconds each, with legacy retries, so one step attempt could outlive the invocation and record no outcome. - State that the 5-minute launch margin shrinks for a timeout near the 8-hour cap. - State that the job input is sent as JSON, so a tuple arrives as a list and a non-string dict key as a string.

botocore's max_attempts counts only the retries, so the default client made 4 attempts where the docs state 3. total_max_attempts counts the first attempt too.

Vivek0712 · 2026-10-13T22:02:22Z

I ran this against the live service, since the description says it has not been yet, and read through the code. The operation behaves correctly end to end. Most of what follows is small, but two points are worth settling before the first release, because they get fixed into checkpoints.

Live run

I deployed a Python durable function (python3.13, arm64) built from this branch: the core SDK from the branch, this package, and boto3 1.43.113. The function is a port of the JS extras/e2e/function.ts for the single and parallel scenarios. I drove it with a copy of the JS run-e2e.mjs that targets a given function ARN, with its scenarios and assertions unchanged. I used two images: the Python worker from #767, and the JS e2e image with @aws/durable-execution-sdk-js-microvm-worker.

Scenario	Python worker	Node worker
succeed	passed	passed
fail	`MicrovmJobFailedError`, "the job failed (RuntimeError): job failing failed on purpose"	`MicrovmJobFailedError`, "the job failed: job failing failed on purpose"
crash	`MicrovmTimeoutError` at the heartbeat timeout	same
run-hook-cjk, run-hook-emoji	passed
parallel (two items in a `map`)	passed, 2 VMs	passed, 2 VMs
auto-http (10,468 code points)	`MicrovmDeliveryError` before the launch, as documented
terminated (VM terminated mid-job)		`MicrovmJobFailedError (MicrovmTerminatedError)`, 3 s after TerminateMicrovm

Every VM ended TERMINATED, which I checked with GetMicrovm, not from the history. The worker's error type comes through callback.state correctly on both workers.

Two things the JS runner flagged that are real differences from JS:

1. The launch checkpoint is not plain JSON. <name>.launch uses the step's default serdes, so its payload is {"t":"m","v":{"microvmId":{"t":"s","v":"microvm-..."},"delivery":{"t":"s","v":"run-hook"}}}. JS checkpoints {"microvmId":"microvm-...","delivery":"run-hook"}. Anything that reads MicroVM ids from execution history gets nothing: the e2e runner, a janitor, or an operator looking up which VM ran a job. One knock-on effect: the runner's terminated scenario could not find the VM to terminate, so I ran that one by hand. The callback already uses JsonSerDes(), and using it for the launch step too would make the history match JS. A serdes change later would break replay of executions in flight, so this is the time to decide.

2. The error type is the qualified class name. The execution's ErrorType is aws_durable_execution_sdk_python_extras.microvm.errors.MicrovmJobFailedError, where JS gives MicrovmJobFailedError. I understand this is how the core SDK rebuilds the class. Still, anything that matches on ErrorType across both SDKs sees different strings, for example a Step Functions Catch on a nested execution or a metric filter. A sentence in the README would help. The module path also becomes part of the checkpoint format, so a comment in errors.py saying it must not move would protect replay.

Code

3. InsufficientCapacityException is retried, and the test that says otherwise builds it with the wrong status. In the botocore model it is HTTP 500 with no retryable trait. _is_retryable (retry.py:88-89) retries any status of 500 or more, so the launch step retries it. tests/retry_test.py:59 constructs it with status 400, so the "not retried" case passes without exercising that path. With the default client making 3 calls per attempt, a Region with no capacity costs 15 RunMicrovm calls and about 30 s of step backoff before MicrovmLaunchError. JS fails it after one attempt. I would check a non-retryable code set before the status fallback, and build the test error with 500.

4. The README's error example raises TypeError. README.md:75 calls microvm(context, "build", job_input, CONFIG). job_input and config are keyword-only, and validation_test.py:121 asserts that this exact call raises.

5. botocore's own connection errors are not step-retried. _RETRYABLE_ERROR_TYPES (retry.py:37-44) uses the builtin ConnectionError, which botocore's HTTPClientError family does not subclass. So ProxyConnectionError, SSLError, ResponseStreamingError and a plain HTTPClientError fail the launch on the first step attempt. Catching botocore.exceptions.HTTPClientError covers them. RequestTimeout, RequestTimeoutException and PriorRequestNotComplete are also missing from the codes.

6. An empty connector tuple becomes the default. operation.py:335-341 uses or, so egress_network_connectors=() launches with INTERNET_EGRESS. The service accepts 0 to 10 connectors, and JS uses ??. is None keeps an explicit empty list.

7. Worth documenting.

A map with a completion config that abandons items leaves those items' VMs running until timeout + 5 min. Each abandoned item is suspended in callback.result() and never replays to its terminate step. I reproduced it locally with min_successful=1: 2 launches, 1 terminate. The same applies when the execution is stopped or times out. JS has the same gap.
The callback, with its heartbeat timeout, is created before the launch. With the default strategy, a throttled launch can take several minutes across client and step retries, which is longer than the README's heartbeat_timeout=1 min. The callback then times out before the VM boots. The replay still launches the VM, and terminates it at once. A note that heartbeat_timeout has to cover the launch and the boot would help.

8. Nits.

On the oversized payload path, <name>.callback is already checkpointed when MicrovmDeliveryError is raised. That callback stays open until its timeout, which can be up to 8 hours. The size without the callback id is known earlier, so the check could run first.
A non-CallbackError from callback.result(), such as a SerDesError on a non-JSON result, becomes a plain MicrovmError. JS maps it to MicrovmJobFailedError.
_reported_error_type (operation.py:494) reads internal state with no guard. A shape change in a 2.x minor release would raise AttributeError, where returning None would keep MicrovmJobFailedError.
LaunchResult.from_dict raises KeyError on a checkpoint with no delivery, before terminate can run. Reading microvmId first would let the unknown-delivery guard cover it.
StepError.message can be None, which gives "the launch failed: None". The terminate log text says "timeout plus 5 minutes", but the duration is capped at 8 hours.

The unit suite passed here: 80 tests, 100% coverage, and mypy is clean. All 80 also pass against core 2.0.0 with boto3 1.43.35, so the lower bounds hold. Happy to send a PR for 3 to 6 if that helps, and to run the same live suite on the HTTP delivery and microvm_session PRs.

- Checkpoint the launch step as plain JSON, as the JavaScript operation does, so the execution history shows the MicroVM ID. - Do not step-retry InsufficientCapacityException, which the service answers with HTTP 500. - Step-retry botocore's connection and HTTP client errors, and the RequestTimeout, RequestTimeoutException, and PriorRequestNotComplete codes. - Keep an explicit empty connector tuple empty. - Fail a job too large for the run hook before the callback exists, when it does not fit even without the callback ID. - Raise MicrovmJobFailedError for a result that is not valid JSON. - Guard the read of the reported error type from internal state. - Terminate for a launch checkpoint without a delivery. - Name the error type when a launch error has no message. - Document the qualified error type, the heartbeat timeout that must cover the launch, and the 8-hour cap; fix the README error example.

yaythomas

nit: CONTRIBUTING asks for a test tree that mirrors the source tree. Please move the tests under tests/microvm/, for example tests/microvm/job_test.py for microvm/job.py.

- Return a copy of the MicroVM error from from_child_context_error. The operation raises it from the ChildContextError, whose __cause__ was the same object. So each error was the other's __cause__, and a loop that follows __cause__ never ended. - Raise the core's ValidationError for an invalid name, config, or Region, the class that Duration raises for a negative value. - Say that the JavaScript operation fails the invocation with SERDES_FAILED for a result that is not valid JSON. - Remove a fallback that never ran. - Add tests: the cause chain ends, terminate runs only after the wait ends, and every invalid timeout raises ValidationError. - Move the tests under tests/microvm/ to mirror the source tree, with microvm_test.py renamed to operation_test.py, and move the function-level imports to the top of each module.

Closes #998. ## Summary A live-service review of the Python port ([aws-durable-execution-sdk-python#769](aws/aws-durable-execution-sdk-python#769 (comment))) found retry and documentation defects. These defects also exist in the JS operation. This PR fixes them. ### Retry: `defaultMicrovmRetryStrategy` The strategy checked only `error.name` and `$retryable`. The AWS SDK client retries a socket error inside each attempt. When those retries end, the client throws the socket error itself. That error is named `Error`, carries the code in `code`, and has no `$retryable`. So the launch, request, and terminate steps failed on their first attempt after a network failure. Changes: - Retry the Node socket error codes that the AWS SDK's own retry strategy uses: `ECONNRESET`, `ECONNREFUSED`, `EPIPE`, `ETIMEDOUT`, `EHOSTUNREACH`, `ENETUNREACH`, `ENOTFOUND`, and `EAI_AGAIN`. The list comes from the retry module of `@smithy/core`. `EAI_AGAIN` is a temporary DNS failure. - Retry `RequestTimeout` and `RequestTimeoutException`. Both are in the AWS SDK's transient error names. - Retry `TooManyRequestsException` (HTTP 429) and `ServiceException` (HTTP 500). Neither has a retryable trait in the service model, so the AWS SDK sets no `$retryable` on them. The Python operation retries both. - Keep `InsufficientCapacityException` not retried. The service answers it with HTTP 500, but a retry seconds later fails the same way. This is why the strategy does not call the AWS SDK's `isTransientError`: that function treats every HTTP 500 as transient. ### Documentation - `heartbeatTimeout` starts when the callback is created, before the launch. So it must cover the launch and the boot. The `types.ts` doc comment and the README now state this. The README example now uses 5 minutes instead of 60 seconds. - The README lists the cases where the terminate step never runs: a `map` or `parallel` that completes early, an execution that fails, times out, or is stopped, and a launch whose answers were all lost. - `MicrovmLaunchError` no longer says that no MicroVM runs the job. When the last attempt timed out, the service may have created the MicroVM anyway. - The terminate failure log and two README lines now state the 8-hour cap on the MicroVM's lifetime. ## Testing - `npm test` in `packages/aws-durable-execution-sdk-js-extras`: 162 tests pass, and `retry.ts` has 100% coverage. `biome ci` reports no errors. - New tests cover each new name and socket code, an unrelated code, and the modeled `TooManyRequestsException`, `ServiceException`, and `InsufficientCapacityException` classes. The class tests also assert that `$retryable` is unset. - A probe in plain Node used a real `LambdaMicrovmsClient` with its endpoint set to a closed port. The client threw `{ name: "Error", code: "ECONNREFUSED", $retryable: undefined }` after its own retries. `main` returns no retry for that error, and this branch returns a retry. - A Jest test cannot reproduce that probe. Jest runs the test in its own realm, and Node creates the socket error in the main realm. So the SDK's `instanceof Error` check fails, and the SDK replaces the error with a new one that has no code. The unit tests therefore build the error shape directly. A comment in the test explains this. - Not run against the live service.

github-actions · 2026-10-14T18:54:07Z

Codex AI review

No actionable findings. Tests were inspected but not run; the added tests do not exercise live AWS MicroVM integration.

Reviewed commit 7652ff9d5b8b900e97a889c71526d3ec38e35b6a. Workflow run

ParidelPooya added 4 commits October 13, 2026 12:31

fix(extras): count all attempts of the default client

e0b90a6

botocore's max_attempts counts only the retries, so the default client made 4 attempts where the docs state 3. total_max_attempts counts the first attempt too.

ParidelPooya mentioned this pull request Oct 13, 2026

MicroVM: step retry gaps and doc fixes found in the Python port review aws/aws-durable-execution-sdk-js#998

Closed

ParidelPooya force-pushed the feat/microvm-operation branch from d0a1bb8 to beed5b4 Compare October 13, 2026 22:55

ParidelPooya mentioned this pull request Oct 13, 2026

fix(extras): retry MicroVM socket errors and fix the docs aws/aws-durable-execution-sdk-js#999

Merged

yaythomas requested changes Oct 14, 2026

View reviewed changes

Vivek0712 mentioned this pull request Oct 14, 2026

test(extras): add an end-to-end test for the microvm operation #775

Closed

yaythomas approved these changes Oct 14, 2026

View reviewed changes

Merge branch 'main' into feat/microvm-operation

7652ff9

ParidelPooya marked this pull request as ready for review October 14, 2026 18:49

ParidelPooya deployed to ai-pr-review-runtime October 14, 2026 18:49 — with GitHub Actions Active

ParidelPooya deployed to ai-pr-review-runtime October 14, 2026 18:50 — with GitHub Actions Active

ParidelPooya merged commit 43780e6 into main Oct 14, 2026
28 checks passed

ParidelPooya deleted the feat/microvm-operation branch October 14, 2026 19:09

Navigation Menu

Repository navigation

feat(extras): add the microvm operation - #769

feat(extras): add the microvm operation#769
ParidelPooya merged 7 commits into
mainfrom
feat/microvm-operation

ParidelPooya commented Oct 13, 2026

Uh oh!

Vivek0712 commented Oct 13, 2026

Uh oh!

yaythomas left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Oct 14, 2026

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ParidelPooya commented Oct 13, 2026

Usage

What it does

Errors

Where Python needs different code from JavaScript

Not in this PR

Repository wiring

Testing

Uh oh!

Vivek0712 commented Oct 13, 2026

Live run

Code

Uh oh!

yaythomas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Oct 14, 2026

Codex AI review

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants