Repository navigation
feat(build): install nodejs workspace dependencies once per monorepo root - #9352
Merged
Merged
Conversation
…th --build-in-source In an npm workspaces monorepo npm keeps one dependency tree, hoisted to the workspace root, and every member's install reconciles that whole tree. Building N functions with --build-in-source therefore reconciled the same tree N times (aws#6567). WorkspaceAwareBuildStrategy wraps the strategy stack outermost, resolves each nodejs function's npm project root (npm's own answer, cached), installs each distinct root once through LambdaBuilder.install_shared_dependencies, and marks the grouped functions with download_dependencies=False - the lever IncrementalBuildStrategy already uses. Scope: Zip nodejs functions on the npm or esbuild workflow, building in source, not cached, not container builds, and only when npm resolves the project root outside the function's own code directory (a workspace member). Everything else, plus any group whose shared install fails or whose aws-lambda-builders does not expose the API, keeps today's per-function install. Sitting outside ParallelBuildStrategy makes the install a barrier: no grouped function build starts before the shared tree is written. Measured on an 8-function esbuild monorepo (~300 packages): 21.9s -> 1.5s; 20 functions: 57.8s -> 3.0s (bench_monorepo.py, median of 3).
…-build-in-source An npm workspaces fixture with two functions holding disjoint dependencies plus a shared workspace package; the lockfile pins lodash 4.17.20 below its ^4.17.20 range so a lockfile-ignoring build is detectable, and the shared package pins lodash 4.17.15 exactly so a hoisting mistake is detectable. An npm shim on PATH logs every invocation, asserting the whole build runs exactly one install (also under --parallel), without --omit=dev, leaving the lockfile bytes untouched; handlers execute from the artifact dirs and see the nested conflicting version; a standalone project keeps its single per-function 'npm update --omit=dev'; --use-container is mutually exclusive with --build-in-source so container builds can never group.
The gate in ApplicationBuilder.build - build_in_source and not cached and not container_manager - is what keeps every other build mode on today's behaviour, and nothing covered it: the grouping strategy had unit tests, its wiring had none. Five tests: --build-in-source wraps the delegate strategy and passes the two npm callbacks; with --parallel the wrapper sits OUTSIDE ParallelBuildStrategy, since the shared install must finish before any grouped function build starts; and --cached, a container build, and a plain build each leave the wrapper out entirely. Verified as guards rather than decoration: weakening the gate to `if self._build_in_source:` fails exactly the --cached and container tests.
aws#6567 is an esbuild report, but the integration coverage was plain npm only. The esbuild workflow reaches the same grouping - both nodejs workflows funnel their installs through the same npm project tree - while its artifact side differs: it bundles what each entry point imports instead of linking a node_modules, so the functions build with no dependency directory at all and the bundle has to have resolved through the tree npm hoisted to the workspace root. The NodeWorkspacesEsbuild fixture carries `esbuild` as the ROOT's own devDependency, which is exactly what the shared install's missing --omit=dev protects, and gives the shared package a conflicting lodash npm keeps nested. The test asserts one install for the whole workspace, that the root's esbuild survived it, that each artifact is a bundle with no node_modules, and that each bundle reports the hoisted 4.17.20 alongside the shared package's nested 4.17.15. The fixture proves its own point: with grouping disabled, the per-function installs prune the root's esbuild and EsbuildBundle fails outright.
_workspace_root_for decides whether a function is a workspace member by comparing npm's reported project root against the code directory, and the two are spelled independently. On Windows two spellings differing only in case - a drive letter included - name the same directory, so reading them as different roots would group a project that is not a workspace at all. normcase is a no-op off Windows; 310 unit and 6 workspace integration tests unchanged.
The workspace grouping changes how every function in an npm workspaces monorepo gets its dependencies, so it ships opt-in: ExperimentalFlag.NodejsMonorepo, enabled by SAM_CLI_BETA_NODEJS_MONOREPO=1, by --beta-features, or by SAM_CLI_BETA_FEATURES=1 with the rest. get_all_experimental() discovers members reflectively, so the new flag registers itself in the beta prompts, the env-var list and telemetry with no other wiring. With the flag off, ApplicationBuilder never wraps WorkspaceAwareBuildStrategy and every function installs for itself - the behaviour of every release so far. Two test changes beyond the new flag-off unit test. The five gating tests now patch the flag ON, so each still tests the gate it names rather than passing because the flag is off; and the integration tests that exercise the grouping ask for --beta-features, including the standalone-project one, where what must keep a per-function install is its own npm root rather than a disabled flag. One new integration test pins the rollout guarantee end to end: the same monorepo fixture with beta features off takes one install per function and still produces correct, disjoint artifacts.
…t hook The call site invoked the hook with the project root alone. aws-lambda-builders#941 makes prepare_shared_project refuse without experimentalNodejsMonorepo - the per-function artifacts link is gated on that same flag, so preparing the root without it would ship artifacts carrying no dependencies. Unflagged, the refusal arrives as a LambdaBuilderError, which this method already catches and turns into a silent fall back to per-function installs: correct artifacts by the slow path, with the feature inert and nothing red to show it. Renamed to follow the hook: install_shared_dependencies became prepare_shared_project upstream, because the same phase for a Cargo workspace compiles every member and installs nothing. The defect survived because nothing tested this method - one assertion pinned the callable into the strategy and stopped there. Adds a test class over what it hands the builders: the enabled flags forwarded, an empty list forwarded rather than omitted, an aws-lambda-builders too old to have the hook declined, and both failure paths falling back. Removing the kwarg again fails two of them.
mypy's no-any-return flagged both helpers: json.loads returns Any against a declared -> dict. Annotating the local satisfies it without a cast or a new import, and keeps the return type the reader wants. Caught by the repository's own lint target, which type-checks tests/ as well as samcli/ - I had run mypy over the two source modules I changed and missed it.
1 task done
The --build-in-source help said only which workflows support building in source, so nothing told a user that an npm workspaces monorepo now installs once at the workspace root, that each function's artifacts are narrowed to its own dependencies, or that any of it needs a beta flag. Scoped to `sam build`: the option is shared with `sam sync`, whose build flows pass cached=True, and the grouping requires `not self._cached`. The help names the exclusions (--use-container, --cached) so the sync case reads correctly from the same text. schema/samcli.json is regenerated from that help by `make schema`, and the build command's rich help gains one example. The buildcmd core test pins the exact list of example sections, so it names the new one.
🧪 Hands-on test: No problems observed in 10 scenariosCommit Scenarios
🤖 Automated hands-on test — not a review approval · model claude-opus-5.5 · Kiro kiro-cli 2.33.1 · build History (2)
|
licjun
approved these changes
Oct 28, 2026
Vandita2020
approved these changes
Oct 28, 2026
licjun
added a commit
to licjun/aws-sam-cli
that referenced
this pull request
Oct 28, 2026
WorkspaceAwareBuildStrategy, added on develop in aws#9352, did not implement the abstract build_single_microvm_image_definition, so it could not be created and every sam build with the NodejsMonorepo flag and --build-in-source failed. Forward MicrovmImage builds to the delegate, like layers and functions, so a wrapped parallel strategy still schedules them.
licjun
added a commit
that referenced
this pull request
Nov 3, 2026
* feat(build): Add sam build support for MicrovmImage
Add sam build support for AWS::Serverless::MicrovmImage and
AWS::Lambda::MicrovmImage. sam build runs docker build on the
resource's source directory and tags the image <full-path>:latest
for local testing.
The built template's code path is not rewritten to a build artifact:
the local image is not the deployment artifact. The source directory
is what gets zipped and uploaded, and the MicroVM service builds it
remotely. move_template only rebases the relative path.
- SamMicrovmImageProvider extracts images from the template, maps
CpuConfigurations[0].Architecture (only ARM_64 is supported), and
accepts CodeArtifact as either a string or {Uri: ...}
- MicrovmImageBuildDefinition is wired into the build graph and all
build strategies; it is not persisted to build.toml
- Explicit targets (sam build MyImage) fail on an unsupported
architecture; the build-all path warns and skips
- --output json reports built MicrovmImages
* fix(build): Resolve intrinsic CodeArtifact.Uri in the built template
update_template only replaced an intrinsic MicrovmImage code path when
the resolved value was a string. A hand-written
CodeArtifact: {Uri: !Ref Src} resolves to {"Uri": "./app"}, so the built
template kept {"Uri": {"Ref": "Src"}}. move_template cannot rebase a
non-string path, so packaging would resolve it against .aws-sam/build/.
Also accept a resolved {"Uri": <str>} and add unit tests for the
intrinsic replacement.
* feat(build): Report the MicrovmImage tag in the JSON build output
r4340571595: the local image sam build creates for a MicrovmImage is
for local testing, but nothing reported its tag. The built template
keeps the source directory, and the JSON entry had only the resource ID
and architecture, so a tool had to copy the tagging rule from
_build_microvm_image.
Add image_tag to the entry, read from the build result so it is the tag
that was actually built.
* feat(build): Warn on cross-architecture MicrovmImage builds
MicrovmImage is built for linux/arm64, but Dockerfile RUN steps execute
on the host CPU. On an x86 host without emulation they fail with
"exec format error" and no hint why. The existing cross-architecture
warning only covers functions built with --use-container.
Warn when the host architecture differs from the image architecture,
reusing _get_host_architecture. The build still proceeds, since hosts
with QEMU (e.g. Docker Desktop) can complete it.
* test(build): Exercise arm64 emulation in MicrovmImage integration tests
The existing fixture only has COPY, so building it for linux/arm64 never
runs arm64 code and the green x86 lanes did not show cross-arch builds
work. Add a fixture with a RUN step and a test that builds it.
Move the MicrovmImage integration tests from build-x86-2 to the
build-arm64 lane, which sets up QEMU. The tier1 smoke test keeps the
COPY-only fixture, since tier1 lanes do not set up emulation.
* test(build): Cover MicrovmImage inside Fn::ForEach with language extensions
MicrovmImage inside Fn::ForEach already builds with --language-extensions,
because the language-extensions code derives its artifact-property and
Mapping tables from RESOURCES_WITH_LOCAL_PATHS. Nothing pinned it, so add
an integration test: each image builds from its own directory, and the
built template keeps Fn::ForEach with per-image paths in the generated
SAMCodeUriImages Mapping.
The test fails with KeyError: 'Mappings' if AWS_SERVERLESS_MICROVMIMAGE is
removed from RESOURCES_WITH_LOCAL_PATHS.
* fix(build): Point the built MicrovmImage at the directory it was built from
With --base-dir, the MicrovmImage source is resolved from base_dir, but
the built template kept the original CodeUri and move_template rebased
it from the template's directory. For sam build -t infra/template.yaml
--base-dir . with CodeUri: src/app, Docker built ./src/app while the
built template pointed at infra/src/app, which does not exist.
Record the absolute directory each image is built from, and have
update_template write it relative to the template's directory so
move_template rebases it correctly. This also covers intrinsic code
paths and CodeArtifact: {Uri: ...}, so the separate intrinsic
resolution is removed. update_template becomes an instance method;
its only caller already calls it on an instance.
* fix(package): Fail sam package for local MicrovmImage code until supported
MicrovmImage types are in RESOURCES_WITH_LOCAL_PATHS so sam build can
rebase their source path, but sam package has no exporter for them. On an
image-only project it reported "Successfully packaged artifacts", left
the local path in the packaged template, and suggested sam deploy.
Register placeholder exporters that fail with a clear message when the
code path is local. S3 URIs and unresolved intrinsics pass through
unchanged. The upcoming sam package support replaces these with real
exporters.
* feat(build): List built MicrovmImages in the text build summary
The text output never named the local image a MicrovmImage build
creates; the tag only appeared in the Docker stream, and "Built
Artifacts" holds nothing for it. The suggested next steps also offered
sam local invoke, which fails on a project with no functions.
Add a "Built MicrovmImages" section listing each image's tag and
platform, read from the build result. When only MicrovmImages were
built, drop the sam local invoke suggestion. Output is unchanged when no
MicrovmImage is built.
* fix(build): Sanitize MicrovmImage tags and report other Docker API errors
A CDK construct ID can contain characters Docker rejects in an image
name, so a resource ID such as "ucb space img" produced the invalid tag
"ucb space img:latest" and an unhandled traceback. Build the tag from
the full path with every run of non-alphanumeric characters replaced by
"-". Existing tags are unchanged.
_build_microvm_image re-raised any docker APIError other than a
Dockerfile-not-found error, which surfaced as an unexpected error asking
for a bug report. Raise DockerBuildFailed with the daemon's explanation
instead, so it is reported as a build failure attributed to the resource.
* fix(build): Report specific MicrovmImage source errors
"Dockerfile not found in CodeUri directory" was shown when the
directory did not exist (including an unresolved parameter such as
'.../CodeUriParam'), when CodeUri was a file such as a .zip, and when
the Dockerfile was missing. It was also raised as DockerBuildFailed,
although no Docker build had run.
- Distinguish a missing directory (with a --parameter-overrides hint),
a file, and a missing Dockerfile
- Raise a new MicrovmImageSourceError for these pre-build checks
- Record why the provider skipped an image (S3 source, missing or
invalid code property), so sam build <image> reports that reason
instead of "Unable to find resource"
* feat(build): Warn when MicrovmImage Metadata Docker settings are ignored
Lambda image functions are built from Metadata Dockerfile, DockerContext,
DockerTag and DockerBuildArgs, so users may set them on a MicrovmImage
too. They have no effect: a MicrovmImage is built from the Dockerfile at
the root of CodeUri, as the MicroVM service builds it. Warn when any of
them is set.
* fix(build): Print one progress line per MicrovmImage build
Both the build strategy and _build_microvm_image logged a "Building
MicrovmImage" line for the same image. Keep the strategy's line, which
includes the source directory and is where function and layer progress
lines are logged.
* fix(build): Collect build resources once so skip warnings print once
run() collected the resources to build, and then
_check_build_method_experimental_flag collected them again. Collecting
logs a warning for each skipped MicrovmImage, so each warning printed
twice. Pass the already collected resources to the check instead.
* fix(build): Only suggest --parameter-overrides for an unset Architecture parameter
The unsupported Architecture warning always suggested
--parameter-overrides, including for a literal such as X86_64. An
unresolvable !Ref is rewritten to the parameter name, so add the hint
only when the value is the name of a template parameter, and name that
parameter.
* fix(build): Mention MicroVM images in build help and --cached output
With --cached, functions and layers report whether their cache was
valid, but MicrovmImages, which are always rebuilt with Docker's layer
cache, printed nothing. Log that SAM CLI does not cache them.
The build description and the --parallel help still described only
functions and layers. Include MicroVM images, and regenerate the
schema.
* fix(build): Skip MicrovmImages without a container runtime on build-all
A plain sam build on a template with a MicrovmImage failed when no
container runtime was available, although the local image is only for
local testing and packaging uses the source directory. On the build-all
path, warn and skip the images instead. Building a targeted image still
fails.
* feat(build): Warn when MicrovmImages map to the same local tag
The local tag is derived from the full path, so IDs that differ only in
case or punctuation map to the same tag, and the image built last
overwrites the others. Warn about such collisions within a build.
* refactor(build): Share the image build client setup
_build_lambda_image and _build_microvm_image each had the same lazy
setup that picks the BuildKit CLI client or the Docker SDK client. Move
it into _get_image_build_client so the two paths cannot drift.
* fix(build): Address MicrovmImage consistency review comments
- Return ApplicationBuildResult directly, without a temporary variable.
- Add a return type to is_microvm_image_buildable, which returns the
skip reason with the result, unlike is_layer_buildable.
- Treat a CodeUri ending in .ZIP or other casing as a .zip file.
- Track a UsedFeature telemetry event when a build includes
MicrovmImages.
* fix(build): Forward MicrovmImage builds in WorkspaceAwareBuildStrategy
WorkspaceAwareBuildStrategy, added on develop in #9352, did not
implement the abstract build_single_microvm_image_definition, so it
could not be created and every sam build with the NodejsMonorepo flag
and --build-in-source failed. Forward MicrovmImage builds to the
delegate, like layers and functions, so a wrapped parallel strategy
still schedules them.
* style: Restore import grouping in app_builder.py
The develop merge dropped the blank line between the third-party and
samcli imports, which ruff reports as I001.
* chore: Regenerate schema after the develop merge
The develop merge kept develop's schema/samcli.json, which dropped the
MicroVM images wording from the --parallel help.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue(s) does this change fix?
#6567
Why is this change necessary?
sam build --build-in-sourceon an npm workspaces monorepo is slow, and it gets slower for every function each time any function is added.npm hoists a workspace member's dependencies into one
node_modulesat the monorepo root, so an install run inside any member reconciles the entire tree.sam buildruns that once per function, which makes the install cost O(N × D_total) — quadratic in monorepo growth. Holding one function at exactly one dependency and adding only unrelated siblings around it, that function's own install goes from 0.32s at one function to 2.79s at eight: 8.7× slower to install one unchanged dependency, purely because neighbours exist.How does it address the issue?
One install per workspace root instead of one per function.
WorkspaceAwareBuildStrategygroups the function build definitions by the npm project rootnpm prefixreports, runsLambdaBuilder.prepare_shared_projectonce per distinct root, and then setsdownload_dependencies=Falseon every function in that group. Each function's artifacts are populated from the already-installed shared tree, narrowed to that function's own dependency closure.It wraps outside
ParallelBuildStrategyon purpose: the shared install has to finish before any grouped function build starts.Measured on this head with aws-lambda-builders 1.68.0, npm workspaces monorepo
fixtures of 8 and 20 functions (root tree 299/303 packages), median of 3, warm npm
cache, each sample starting with no
node_modulesand no.aws-sam.esbuild (
BuildMethod: esbuild, esbuild on PATH) — both sides produce correctbundles, so these are the like-for-like numbers:
Marginal cost per added function: 3.14s today, 0.28s grouped.
npm (no bundler): 24.86s -> 7.56s at 8 functions, 62.03s -> 14.07s at 20.
The npm "today" column is not a working build — it ships artifacts holding only
index.jsandpackage.json, zero dependencies, for every function (#933). Bothhalves of the fix sit behind the same flag, so the speed and the correctness arrive
together; this is not a speedup over a build that worked.
npm subprocess count is 1 install per workspace root against N today: 1 vs 8, and
1 vs 20.
--parallelis the sharpest result. Today it fails on this shape — concurrent npmprocesses reconciling one hoisted root tree (
EEXIST symlink,ENOTEMPTY rmdir), 2/3runs at 8 functions and 3/3 at 20. Grouped, one install finishes before any function
build starts, so the race cannot occur: 3/3 clean, 4.35s and 6.26s.
Full method, raw samples and caveats are in a comment below.
What side effects does this change have?
Opt-in while it rolls out. Gated on
ExperimentalFlag.NodejsMonorepo—--beta-features,SAM_CLI_BETA_NODEJS_MONOREPO=1, orSAM_CLI_BETA_FEATURES=1. With the flag off,ApplicationBuildernever wraps the strategy and every function installs for itself: the behaviour of every release so far, by the same code path. An integration test runs the monorepo fixture with beta features disabled and asserts both the per-function install count and the disjoint artifacts.The same flags go to
prepare_shared_project, which refuses withoutexperimentalNodejsMonorepo. That matters: the per-function artifacts link in aws-lambda-builders is gated on the same flag, so preparing the root without it would ship artifacts carrying no dependencies at all. Enabling one half without the other is now refused rather than silently wrong.Grouping is skipped, with a debug line, for:
--use-containerCodeUri; container builds reach the builders over JSON-RPC--cached.aws-sam/depsand managesdownload_dependenciesitself--build-in-sourceLayerBuildDefinitiongoes through a separate loopThe install's working directory moves from a function's
codeurito the workspace root. That root is inside the user's project and is the directory npm itself would choose; a scope check ("npm prefixresolves outside the codeuri") bounds it, compared throughnormcaseso two spellings of one Windows path match.A failure never fails the build. Anything the preparation raises is caught and the grouped functions fall back to per-function installs.
Mandatory Checklist
PRs will only be reviewed after checklist is complete
make prpassesmake update-reproducible-reqsif dependencies were changed — pending the builders release this depends on--build-in-sourcehelp now describes the workspace-root install and the beta flag,schema/samcli.jsonis regenerated from it, andsam build --helpgains a monorepo example. The published AWS SAM Developer Guide page is a separate repository and is not covered here.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.