diff --git a/docs/microvm-runners.md b/docs/microvm-runners.md new file mode 100644 index 0000000000..872dd52de0 --- /dev/null +++ b/docs/microvm-runners.md @@ -0,0 +1,129 @@ +# Lambda MicroVM Runners (Experimental) + +!!! warning + Lambda MicroVM runner support is experimental. The image build, lifecycle-hook server, control-plane integration, and AWS MicroVM APIs must be configured together. Validate the complete flow in a non-production environment before relying on it for workloads. + +## Overview + +Lambda MicroVM runners provide ephemeral GitHub Actions runners backed by +Lambda MicroVMs. The runner control plane receives demand, obtains the +one-time runner configuration, starts a MicroVM from a published image, and +passes the runtime execution role to the MicroVM. + +The repository includes a combined [multi-runner webhook example](examples/multi-runner-webhook.md) +that places EC2 and Lambda MicroVM lanes behind one webhook endpoint. The +provider-specific lifecycle checks are shared where possible, so the same +deployment can validate both providers. + +## Prerequisites + +Before deploying the MicroVM runner lane, prepare all of the following in the +target AWS Region: + +1. **MicroVM foundation.** Apply the + [MicroVM foundation example](examples/microvm-foundation.md). It creates the + regional artifact bucket, Lambda Network Connectors, the image-build role, + and the reusable MicroVM usage policy. +2. **Lifecycle-hook artifact.** Build and release + `lambdas/services/microvm-lifecycle-hooks` through the same workspace + artifact process used for the repository's Lambda services. The resulting + ZIP is embedded in the MicroVM image. +3. **Published MicroVM image.** Use the + [MicroVM Ubuntu image instructions](https://github.com/github-aws-runners/terraform-aws-github-runner/blob/main/images/microvm-ubuntu/README.md) + to build and publish an image with Packer. The image must contain the + compatible lifecycle-hook server and runner entrypoint. +4. **Runner execution role.** Configure the runner role through the runner + configuration. This is different from the foundation's build role. The + control-plane TypeScript passes the execution role to `RunMicrovm`, so the + Lambda that starts the MicroVM must have permission to pass it. +5. **Runner control plane and artifacts.** Deploy the runner control plane with + the published image ARN/version, Network Connector ARNs, GitHub App + configuration, and the runner-control and webhook Lambda ZIPs. + +The foundation does not create the image or the runner execution role. The +image build does not choose the runtime role. These are separate dependencies +owned by the image build and runner-control-plane stages respectively. + +## IAM roles + +MicroVM deployments use two roles for two different operations: + +| Role | Used by | Responsibility | +| --- | --- | --- | +| Build role (`build_role_arn`) | Packer/image publisher | Creates and publishes the MicroVM image and accesses the foundation build artifacts. | +| Execution role | Runner control plane and the MicroVM | Is passed to `RunMicrovm` and provides the permissions used by the ephemeral runner at runtime. | + +Do not use the build role as the runner execution role. The control-plane +Lambda needs `iam:PassRole` for the configured execution role, and the +execution role must contain the runtime permissions required by the selected +runner lane. + +## Deployment order + +The complete dependency chain is: + +```text +MicroVM foundation + | + v +Build/release lifecycle-hook server + | + v +Packer builds and publishes image + | + v +Runner control plane resolves execution role + | + v +RunMicrovm starts an ephemeral runner +``` + +The lifecycle-hook server is part of the image artifact. Updating the hook +server therefore requires building/releasing the artifact and publishing a +new compatible image before deploying that image version to the runner lane. + +## Combined EC2 and MicroVM deployment + +The [multi-runner webhook example](examples/multi-runner-webhook.md) accepts +explicit `runners_lambda_zip` and `webhook_lambda_zip` inputs and configures +both compute providers behind one webhook. Its MicroVM settings require a +published image: + +```hcl +compute_provider = { + aws = { + microvm = { + image_arn = "arn:aws:lambda:eu-west-1:123456789012:microvm-image:gha-ubuntu-arm64" + image_version = null + ingress_network_connectors = [] + egress_network_connectors = ["arn:aws:lambda:eu-west-1:123456789012:network-connector:example"] + } + } +} +``` + +Use the example's complete Terraform configuration as the source of truth for +the current input shape. The example deploys the control plane; it does not +build the foundation, lifecycle-hook artifact, or MicroVM image for you. + +## Known limitations + +- This integration is experimental and depends on AWS Lambda MicroVM APIs and + the lifecycle-hook protocol. +- A compatible lifecycle-hook server must be present in every image used by + the MicroVM provider. +- Image publication and activation are separate from Terraform deployment; + wait for the image version to become active before starting jobs. +- The build role and execution role are intentionally separate. Changes to + either role can affect a different stage of the lifecycle. +- The combined webhook example is useful for integration testing, but a real + deployment still needs a real MicroVM image and the network/runtime IAM + configuration described above. + +## Repository examples + +- [MicroVM foundation](examples/microvm-foundation.md) +- [MicroVM image build README](https://github.com/github-aws-runners/terraform-aws-github-runner/blob/main/images/microvm-ubuntu/README.md) +- [Lifecycle-hook service README](https://github.com/github-aws-runners/terraform-aws-github-runner/blob/main/lambdas/services/microvm-lifecycle-hooks/README.md) +- [Multi-runner webhook](examples/multi-runner-webhook.md) +- [MicroVM foundation module](modules/public/microvm-foundation.md) diff --git a/examples/microvm-foundation/README.md b/examples/microvm-foundation/README.md index 7417f3c70e..c52c872d47 100644 --- a/examples/microvm-foundation/README.md +++ b/examples/microvm-foundation/README.md @@ -21,10 +21,26 @@ documented in `../../images/microvm-ubuntu/README.md`. Use the outputs as the bu - `connector_arns.ministack` -> `MICROVM_EGRESS_NETWORK_CONNECTOR_ARN` - `usage_policy_arn` -> attach to the control-plane role used by the runner example +The deployment order is: + +1. Apply this foundation to create the regional bucket, Network Connectors, + build role, and reusable runtime policy. +2. Build and release the lifecycle-hook service from + `lambdas/services/microvm-lifecycle-hooks` using the repository's normal + Lambda artifact process. +3. Build and publish the MicroVM image with Packer, passing the foundation + outputs and the released lifecycle-hook ZIP. The image builder uses the + **build role**. +4. Deploy the runner control plane, such as + `examples/multi-runner-webhook`, with the published image ARN/version. The + control plane resolves the **execution role** from the runner configuration + and passes it to `RunMicrovm` when it starts a job. + +The two roles must not be conflated: the build role creates the image, while +the execution role runs the ephemeral GitHub Actions runner inside that image. The foundation module owns regional storage, build IAM, Network Connectors, -and the reusable runtime policy. It does not publish an image or create the -runner control plane; those steps remain explicit and can be performed after -the foundation is available. +and the reusable runtime policy. It does not publish an image, create the +execution role, or create the runner control plane. ## Requirements diff --git a/examples/multi-runner-webhook/README.md b/examples/multi-runner-webhook/README.md index c6b4cf5e5d..fc653a2f70 100644 --- a/examples/multi-runner-webhook/README.md +++ b/examples/multi-runner-webhook/README.md @@ -18,6 +18,32 @@ terraform apply \ -var='webhook_lambda_zip=/path/to/webhook.zip' ``` +## MicroVM prerequisites + +The MicroVM lane expects an image that has already been built and published in +the target Region. The image is not created by this example. Prepare it in +this order: + +1. Apply `examples/microvm-foundation`. +2. Build and release the lifecycle-hook service from + `lambdas/services/microvm-lifecycle-hooks` using the same artifact process + used for the repository's Lambda services. +3. Build the image with Packer from `images/microvm-ubuntu`, passing the + foundation's bucket, connector, build-role, and lifecycle-hook artifact. +4. Set `compute_provider.aws.microvm.image_arn` (and, when applicable, + `image_version`) to the published image. + +The foundation's build role is used to create the image. It is different from +the execution role used by the runner job. The runner configuration owns that +execution role; the control-plane TypeScript passes it to `RunMicrovm` when it +starts an ephemeral runner. The control-plane Lambda therefore needs +permission to pass the configured execution role, and the role needs the +runtime permissions required by the selected runner lane. + +This example deploys both EC2 and MicroVM lanes behind one webhook endpoint, +but it does not replace the foundation, image build, lifecycle-hook release, +or execution-role setup steps. + ## Requirements @@ -61,4 +87,4 @@ No resources. |------|-------------| | [webhook\_endpoint](#output\_webhook\_endpoint) | n/a | | [webhook\_secret](#output\_webhook\_secret) | n/a | - \ No newline at end of file + diff --git a/images/microvm-ubuntu/README.md b/images/microvm-ubuntu/README.md index 84feeb24b4..218d79efab 100644 --- a/images/microvm-ubuntu/README.md +++ b/images/microvm-ubuntu/README.md @@ -7,8 +7,11 @@ Dockerfile, compiled lifecycle-hook ZIP contract, and image entrypoint. Before building the image: 1. Apply `examples/microvm-foundation` in the target AWS Region. -2. Install Packer and set the required AWS, S3, IAM, connector, and - lifecycle-hook variables. +2. Build and release the lifecycle-hook service from + `lambdas/services/microvm-lifecycle-hooks`, then set + `MICROVM_LIFECYCLE_HOOK_ZIP` to the released artifact. +3. Install Packer and set the required AWS, S3, IAM, connector, and image + variables. The image intentionally excludes the source repository's optional external telemetry and Teleport services. It contains only the Actions runner, @@ -40,7 +43,13 @@ packer build -color=false github_agent.microvm.ubuntu.pkr.hcl ``` The build role, artifact bucket, and network connector are created by the -foundation module. Keep the bucket private and versioned, use the module's -least-privilege policies, and do not put credentials in checked-in files. The -lifecycle-hook ZIP must contain the compiled `server.js` at its archive root; -any bundled dependencies must use safe relative paths. +foundation module. The build role is used only while Packer creates and +publishes the image; it is not baked into the image and is not the role used +by runner jobs. The execution role is selected by the runner control plane and +passed to `RunMicrovm` at launch time, so it is not configured by this image +build. + +Keep the bucket private and versioned, use the module's least-privilege +policies, and do not put credentials in checked-in files. The lifecycle-hook +ZIP must contain the compiled `server.js` at its archive root; any bundled +dependencies must use safe relative paths. diff --git a/lambdas/services/microvm-lifecycle-hooks/README.md b/lambdas/services/microvm-lifecycle-hooks/README.md index 5f1481c370..01a16e7732 100644 --- a/lambdas/services/microvm-lifecycle-hooks/README.md +++ b/lambdas/services/microvm-lifecycle-hooks/README.md @@ -14,6 +14,20 @@ yarn workspace @aws-github-runner/microvm-lifecycle-hooks start `build` uses esbuild to create the self-contained CommonJS server bundle `dist/server.js`. It also writes `dist/package.json` with `type: commonjs` so the bundle remains executable after it is copied outside the Yarn workspace. +## Build and release with the Lambda artifacts + +The lifecycle-hook server is a deployable image-build artifact, not a service +that is installed separately beside the runner control plane. Build and test it +through the normal Lambda workspace/release process, then provide the resulting +ZIP to the MicroVM image build as `MICROVM_LIFECYCLE_HOOK_ZIP`. Packer embeds +that artifact in the image; every published image used by the MicroVM provider +must contain a compatible hook server. + +The hook server's release lifecycle is therefore separate from the MicroVM +execution role. The image build uses the foundation's build role. When a job +starts, the runner control plane supplies the runtime execution role to +`RunMicrovm`. + To build before invoking Docker, run the workspace build above. In the existing MicroVM runner Dockerfile, which already installs s6-overlay and the GitHub runner's Node 24 runtime, copy the complete artifact and replace the old hook command with: ```dockerfile diff --git a/mkdocs.yaml b/mkdocs.yaml index ed4f2246a5..289ca9b955 100644 --- a/mkdocs.yaml +++ b/mkdocs.yaml @@ -60,6 +60,7 @@ nav: - Multi-runner v1 to v2 migration: multi-runner-v1-v2-migration.md - Getting started: getting-started.md - Security: security.md + - Lambda MicroVM runners (experimental): microvm-runners.md - Architecture decisions: - MiniStack for integration tests: adr/0001-use-ministack-for-terraform-integration-tests.md - Modules: diff --git a/modules/microvm-foundation/README.md b/modules/microvm-foundation/README.md index d87227c89f..9680f03622 100644 --- a/modules/microvm-foundation/README.md +++ b/modules/microvm-foundation/README.md @@ -12,6 +12,24 @@ It manages: - A Lambda-trusted Network Connector operator role and propagation barrier. - An unattached runtime usage policy for the reserved image namespace and connector inventory. +## Build role and execution role + +MicroVM deployments use two different IAM roles with different lifecycles: + +- The `build_role_arn` output is the **build role**. The image builder assumes + this role while it creates and publishes a MicroVM image. It grants the + image-build permissions for the foundation artifact bucket, logs, and any + configured ECR repositories. It is not the role used by a runner job. +- The **execution role** is attached to each MicroVM when the runner control + plane launches it. The control-plane TypeScript passes this role to + `RunMicrovm`; the Lambda that calls that API must be allowed to pass the + role. The MicroVM and the ephemeral runner use this role at runtime. + +The execution role is resolved by the runner configuration and is intentionally +not created by this foundation module. The foundation creates the regional +build resources and the reusable `usage_policy_arn`; the runner/control-plane +configuration owns the runtime role and its provider-specific permissions. + The module does not create MicroVM images, runner execution roles, or the runner control plane. Attach `usage_policy_arn` to the control-plane role that owns the runtime launch operations. The caller must also grant the Terraform