feat(docker): write each image only the part of a shared build context its Dockerfile reads

AGENTS.md

@@ -99,7 +99,7 @@ and deserialized through a registry.
 | Package | Contents |
 |---|---|
 | `env/` | `Env` base class: a subclass declares `type`, implements `from_dict` and `async deploy(...) -> DeployedEnv`, and may override `reset` and `load_file_artifact_universe`. Built-ins in `env/envs/`: `MCPServerEnv` (`mcp_server`), `MultiEnv` (`multi`, MCP servers plus websites), `ServiceDBEnv` (`service_db`, PostgreSQL), `WebsiteEnv` (`website`), `GatewayEnv` (`gateway_server`). `env/gateway/gateway.py` is the env-side gateway server, container-only code. |
-| `artifact/` | Immutable versioned artifacts; every `put()` creates a new version. The document goes to the document store, binary payloads to the object store. Built-ins: `CliArtifact` (`cli`), `FileArtifact` (`file`), `FileArtifactUniverse` (`file_artifact_universe`), `DockerImageArtifact` (`docker_image`), `EnvironmentArtifact` (`environment`), `EnvironmentUniverseArtifact` (`environment_universe`), `SkillArtifact` (`skill`); `[artifacts] type_aliases` maps another stored spelling of a type onto one of them. A `DockerImageArtifact` holds a tar.gz of its image, or, written by `put_ref` (`--image-ref` on `env mcp-server put` and `a2a-agent put`), only a registry reference with its tag pinned to a digest, which sandboxes pull, or, written by `put_context`, only its build context, which a VM sandbox builds before running it (`VmSandbox.load_docker_images`) and Modal builds before creating its container (`ModalSandboxProvider.prepare_image`, outside a fallback chain's deadline); anything else that runs images by name (a container on the local provider, a lone server on any provider but Modal) refuses such an image. A build context (`utils/build_context.py`) is what `docker build` would send, `.dockerignore` applied, written so equal files give equal bytes and an equal `source_digest`. |
+| `artifact/` | Immutable versioned artifacts; every `put()` creates a new version. The document goes to the document store, binary payloads to the object store. Built-ins: `CliArtifact` (`cli`), `FileArtifact` (`file`), `FileArtifactUniverse` (`file_artifact_universe`), `DockerImageArtifact` (`docker_image`), `EnvironmentArtifact` (`environment`), `EnvironmentUniverseArtifact` (`environment_universe`), `SkillArtifact` (`skill`); `[artifacts] type_aliases` maps another stored spelling of a type onto one of them. A `DockerImageArtifact` holds a tar.gz of its image, or, written by `put_ref` (`--image-ref` on `env mcp-server put` and `a2a-agent put`), only a registry reference with its tag pinned to a digest, which sandboxes pull, or, written by `put_context`, only its build context, which a VM sandbox builds before running it (`VmSandbox.load_docker_images`) and Modal builds before creating its container (`ModalSandboxProvider.prepare_image`, outside a fallback chain's deadline); anything else that runs images by name (a container on the local provider, a lone server on any provider but Modal) refuses such an image. A build context (`utils/docker/context.py`) is what `docker build` would send, `.dockerignore` applied, written so equal files give equal bytes and an equal `source_digest`. |
 | `task/`, `task_step/` | A `Task` holds its `TaskStep`s inline; `Task.run()` executes them as a DAG. `depends_on` (None means all prior steps) gates a step, independent steps run concurrently, `fail_task_on_error` makes a failure fatal or tolerated, `retry_config` rolls a failed span back through the step journal and re-dispatches it. Built-in steps live in `task_step/task_steps/` (`deploy_env`, `deploy_agent`, `prompt_agent`, the verifiers under `verifiers/`, and more). `task/episode/` runs a task with some of its agents played from outside, as a training loop does (`EpisodeSpec`). |
 | `store/` | Four store ABCs with local and cloud implementations: `DocumentStore` (SQLite, MongoDB, DynamoDB, Firestore with MongoDB compatibility), `ObjectStore` (filesystem, S3, Cloud Storage), `ImageStore` (local OCI registry, ECR), `SecretStore` (env vars or file, AWS Secrets Manager, Google Cloud Secret Manager). `VersionedEntityStore` implements the shared versioned get/put logic, `QueryBuilder` is the immutable chained query API, `store/base.py` holds the error types. `store/routing.py` is the `@local` namespace: while the CLI runs, an `@local/…` id and what an `@local` task's run records go to local stores beside the configured ones, and that run's writes to the configured stores are refused. A new backend must pass the conformance kits in `tst/store/`. |
 | `config/` | The `Config` singleton (`get_config`, `configure`, `reset_config`) in `config/runtime.py`, file discovery in `config/loader.py`, and `load_impl`, which resolves `module:Class` pointers. `agent_env.store` re-exports the config names for compatibility. |

src/agent_env/artifact/artifacts/docker_image.py

@@ -23,8 +23,8 @@
 from agent_env.store.image_store.local_registry_image_store import LocalRegistryImageStore
 from agent_env.store.image_store.oci_registry_credentials import is_loopback_host, names_registry, registry_host_from_ref
 from agent_env.store.image_store.registry_api import pin_digest, pinned_from
-from agent_env.utils.build_context import BuildContext
-from agent_env.utils.docker_build import DEFAULT_BUILD_PLATFORM, normalize_platform
+from agent_env.utils.docker.context import BuildContext
+from agent_env.utils.docker.build import DEFAULT_BUILD_PLATFORM, normalize_platform
 
 logger = logging.getLogger(__name__)
 

src/agent_env/bundle/materialize.py

@@ -36,7 +36,7 @@
 from agent_env.task import Task
 from agent_env.task_step.registry import get_task_step_registry
 from agent_env.task_step.task_step import TaskStep
-from agent_env.utils.docker_build import DockerBuildError
+from agent_env.utils.docker.build import DockerBuildError
 
 from ._fs import with_article
 from .authoring import AuthoringContext, build_context_files

src/agent_env/bundle/preflight.py

@@ -71,7 +71,7 @@
 from agent_env.task_step.task_steps.deploy_env import DeployEnvTaskStep
 from agent_env.task_step.task_steps.deploy_sandbox import DeploySandboxTaskStep
 from agent_env.task_step.task_steps.verifiers.rubrics_verifier import RubricsVerifierTaskStep
-from agent_env.utils.docker_build import DEFAULT_BUILD_PLATFORM, docker_unreachable
+from agent_env.utils.docker.build import DEFAULT_BUILD_PLATFORM, docker_unreachable
 
 from .parse import BundleError, BundleKind
 from .plan import Plan

src/agent_env/cli/__init__.py

@@ -7,7 +7,7 @@
 from agent_env.plugins._cli import load_cli_plugins, load_cli_root_options
 from agent_env.store.base import NotFoundError
 from agent_env.store.routing import namespace_routing
-from agent_env.utils.docker_build import DockerBuildError
+from agent_env.utils.docker.build import DockerBuildError
 
 from .a2a_agent import a2a_agent
 from .artifact import artifact

src/agent_env/cli/a2a_agent/put.py

@@ -7,7 +7,7 @@
 from agent_env.artifact import DockerImageArtifact
 from agent_env.cli.utils import build_platform_option, detect_env_metadata, refuse_unwritable_ids
 from agent_env.store.ids import derive_id, image_repository
-from agent_env.utils.docker_build import DEFAULT_BUILD_PLATFORM, build_image
+from agent_env.utils.docker.build import DEFAULT_BUILD_PLATFORM, build_image
 
 
 @click.command()

src/agent_env/cli/env/mcp_server.py

@@ -16,7 +16,7 @@
     resolve_environment_name,
 )
 from agent_env.utils.card_naming import card_name_from_github, card_name_from_source
-from agent_env.utils.docker_build import DEFAULT_BUILD_PLATFORM, build_image
+from agent_env.utils.docker.build import DEFAULT_BUILD_PLATFORM, build_image
 from agent_env.env import Env, MCPServerEnv
 from agent_env.store.ids import derive_id, image_repository
 

src/agent_env/cli/env/website.py

@@ -16,7 +16,7 @@
     resolve_environment_name,
 )
 from agent_env.utils.card_naming import card_name_from_github, card_name_from_source
-from agent_env.utils.docker_build import DEFAULT_BUILD_PLATFORM, build_image
+from agent_env.utils.docker.build import DEFAULT_BUILD_PLATFORM, build_image
 from agent_env.env.envs.website import WebsiteEnv
 from agent_env.store.ids import derive_id, image_repository
 

src/agent_env/cli/utils.py

@@ -10,8 +10,8 @@
 from agent_env.providers.env_providers.env_provider import _env_provider_class
 from agent_env.providers.env_providers.env_server_provider import EnvironmentServerProvider
 from agent_env.store.base import NotFoundError
-from agent_env.utils.build_metadata import detect_base_metadata, detect_env_metadata  # noqa: F401  re-exported for the put commands
-from agent_env.utils.docker_build import DEFAULT_BUILD_PLATFORM
+from agent_env.utils.docker.metadata import detect_base_metadata, detect_env_metadata  # noqa: F401  re-exported for the put commands
+from agent_env.utils.docker.build import DEFAULT_BUILD_PLATFORM
 
 
 @contextmanager

src/agent_env/env/bootstrap.py

@@ -27,9 +27,9 @@
 from agent_env.store.local_state import holding_locks
 from agent_env.store.object_store import LocalFilesystemObjectStore
 from agent_env.store.routing import configured_store
-from agent_env.utils.build_context import BuildContext
-from agent_env.utils.build_metadata import detect_env_metadata
-from agent_env.utils.docker_build import build_image, docker_unreachable
+from agent_env.utils.docker.context import BuildContext
+from agent_env.utils.docker.metadata import detect_env_metadata
+from agent_env.utils.docker.build import build_image, docker_unreachable
 
 _ENV_PACKAGE = Path(__file__).parent
 GATEWAY_DOCKERFILE = _ENV_PACKAGE / "gateway" / "Dockerfile"

src/agent_env/providers/sandbox_providers/host_images.py

@@ -23,7 +23,7 @@
 from agent_env.artifact.artifacts.docker_image import DockerImageArtifact
 from agent_env.config import get_config
 from agent_env.store.local_state import holding_locks
-from agent_env.utils.docker_build import build_image, normalize_platform
+from agent_env.utils.docker.build import build_image, normalize_platform
 
 logger = logging.getLogger(__name__)
 

src/agent_env/providers/sandbox_providers/modal_image_build.py

@@ -5,9 +5,7 @@
 
 import asyncio
 import glob
-import json
 import logging
-import re
 import tarfile
 import tempfile
 import weakref
@@ -17,7 +15,8 @@
 import modal
 
 from agent_env.config import get_config
-from agent_env.utils.build_context import extract, ignore_file
+from agent_env.utils.docker.context import extract, ignore_file
+from agent_env.utils.docker.dockerfile import Instruction, copy_arguments, instructions, opens_heredoc
 
 if TYPE_CHECKING:
     from agent_env.artifact.artifacts.docker_image import DockerImageArtifact
@@ -76,7 +75,6 @@ def fmt_exc(e: BaseException) -> str:
     return f"{type_name}: {msg}" if msg else type_name
 
 
-_COPY_OR_ADD = re.compile(r"\s*(COPY|ADD)\s+(.*)", re.IGNORECASE | re.DOTALL)
 # The flags COPY takes as ADD does; ADD's own (--checksum, --keep-git-dir, --unpack) keep an ADD as it is.
 _COPY_FLAGS = ("--chown=", "--chmod=", "--link")
 
@@ -86,43 +84,29 @@ def modal_dockerfile(text: str, context: Path) -> str:
     a JSON-form COPY or ADD whose paths hold no whitespace in shell form, which Modal's context parser reads, and an
     ADD of files and folders in ``context``, none of them a tar archive, as the COPY it is, since Modal's ADD only
     fetches a URL. Every other line is left as written."""
-    lines, instruction = [], []
-    for line in text.splitlines(keepends=True):
-        instruction.append(line)
-        if not line.rstrip("\r\n").endswith("\\"):
-            lines.append(_for_modal("".join(instruction), context))
-            instruction = []
-    return "".join(lines + instruction)
-
-
-def _for_modal(instruction: str, context: Path) -> str:
-    match = _COPY_OR_ADD.fullmatch(re.sub(r"\\\r?\n", " ", instruction).strip())
-    if match is None:
-        return instruction
-    keyword, rest = match.group(1).upper(), match.group(2).strip()
-    flags = []
-    while rest.startswith("--"):
-        flag, _, rest = rest.partition(" ")
-        flags.append(flag)
-        rest = rest.strip()
-    if rest.startswith("["):
-        try:
-            args = json.loads(rest)
-        except ValueError:
-            return instruction
-        if not (isinstance(args, list) and len(args) >= 2
-                and all(isinstance(arg, str) and arg and not any(c.isspace() for c in arg) for arg in args)):
-            return instruction
-    elif keyword == "ADD" and not rest.startswith("<<") and not any(c in rest for c in "\"'"):
-        args = rest.split()
-    else:
-        return instruction
+    return "".join(_for_modal(instruction, context) for instruction in instructions(text))
+
+
+def _for_modal(instruction: Instruction, context: Path) -> str:
+    args = copy_arguments(instruction)
+    if args is None or args.heredoc:
+        return instruction.text
+    paths = (*args.sources, args.destination)
+    if args.json_form:
+        # Written in shell form, a path with whitespace would split, and one like <<EOF would open a heredoc.
+        if not all(path and not any(c.isspace() for c in path) and not opens_heredoc(path) for path in paths):
+            return instruction.text
+    elif args.keyword != "ADD":
+        return instruction.text
+    keyword = args.keyword
     if keyword == "ADD":
-        if len(args) < 2 or not all(flag.startswith(_COPY_FLAGS) for flag in flags) \
-                or not all(_local_and_not_an_archive(source, context) for source in args[:-1]):
-            return instruction
+        if not all(flag.startswith(_COPY_FLAGS) for flag in args.flags) \
+                or not all(_local_and_not_an_archive(source, context) for source in args.sources):
+            return instruction.text
         keyword = "COPY"
-    return " ".join([keyword, *flags, *args]) + "\n"
+    # The rewrite is one line, so comment lines among the continued ones go above it rather than be lost.
+    comments = [line for line in instruction.text.splitlines(keepends=True)[1:] if line.lstrip().startswith("#")]
+    return "".join(comments) + " ".join([keyword, *args.flags, *paths]) + "\n"
 
 
 def _local_and_not_an_archive(source: str, context: Path) -> bool:

src/agent_env/providers/sandbox_providers/sandbox.py

@@ -24,7 +24,7 @@
 
 from agent_env.config import get_config
 from agent_env.store.image_store.oci_registry_credentials import registry_host_from_ref
-from agent_env.utils.docker_build import DEFAULT_BUILD_PLATFORM
+from agent_env.utils.docker.build import DEFAULT_BUILD_PLATFORM
 from agent_env.utils.paths import validate_relative_filename
 
 if TYPE_CHECKING:

src/agent_env/providers/sandbox_providers/smol_vm/sandbox.py

@@ -16,7 +16,7 @@
 from agent_env.providers.sandbox_providers.local_sandbox import LocalSandboxProvider
 from agent_env.store.object_store.local.store import LocalFilesystemObjectStore
 from agent_env.store.object_store.local.tls import local_ca
-from agent_env.utils.docker_build import normalize_platform
+from agent_env.utils.docker.build import normalize_platform
 from agent_env.providers.sandbox_providers.sandbox_provider import SANDBOX_MODE_VM
 
 _PORTS_FILE = "/storage/agentenv-ports.json"

src/agent_env/utils/build_context.py → src/agent_env/utils/docker/context.py

@@ -59,7 +59,7 @@ def of(cls, root: Path, dockerfile: Path | None = None) -> BuildContext:
         named = None
         if dockerfile is not None and _inside(top, Path(os.path.realpath(dockerfile))):
             named = _inside(top, dockerfile)
-        matcher = FilePatternMatcher.from_file(ignores) if ignores else None
+        matcher = ignore_matcher(ignores.read_text(encoding="utf-8")) if ignores else None
         walk = _Walk(top, matcher, kept)
         walk.run()
         if walk.problems:
@@ -104,7 +104,7 @@ def extract(archive: Path, out: Path) -> None:
     of ``out`` is left out, as nothing there is the build's. Raises ValueError on anything else, which no context
     written by ``BuildContext.write`` holds: a hard link, a device, a name that leads out of ``out``."""
     with tarfile.open(archive, "r:gz") as tar:
-        tar.extractall(out, filter=_context_member)
+        tar.extractall(out, filter=context_member)
 
 
 class _Walk:
@@ -206,15 +206,15 @@ def hexdigest(self) -> str:
         return self._digest.hexdigest()
 
 
-def _context_member(member: tarfile.TarInfo, out: str) -> tarfile.TarInfo | None:
-    """``member`` as ``extract`` writes it, None to leave it out."""
+def context_member(member: tarfile.TarInfo, out: str, *, links_out: bool = False) -> tarfile.TarInfo | None:
+    """``member`` as ``extract`` writes it; None for a link leading out of ``out``, unless ``links_out``."""
     if not (member.isreg() or member.isdir() or member.issym()):
         raise ValueError(f"{member.name!r} in the build context is neither a file, a folder nor a link")
     try:
         member = tarfile.tar_filter(member, out)
     except tarfile.FilterError as e:
         raise ValueError(f"{member.name!r} in the build context: {e}") from e
-    if member.issym():
+    if member.issym() and not links_out:
         top = os.path.realpath(out)
         target = os.path.realpath(os.path.join(top, os.path.dirname(member.name), member.linkname))
         if os.path.commonpath([top, target]) != top:
@@ -229,6 +229,17 @@ def ignore_file(top: Path, dockerfile: Path | None) -> Path | None:
     return next((path for path in (*candidates, top / ".dockerignore") if path.is_file()), None)
 
 
+def ignore_names(dockerfile: str) -> tuple[str, str]:
+    """``ignore_file``'s candidates, in order, for a Dockerfile at ``dockerfile`` in the context."""
+    return f"{dockerfile}.dockerignore", ".dockerignore"
+
+
+def ignore_matcher(ignore_file: str) -> FilePatternMatcher:
+    """A matcher for an ignore file's patterns, as docker build reads them: a line opening with ``#`` is a comment."""
+    lines = ignore_file.removeprefix("\ufeff").splitlines()
+    return FilePatternMatcher(*(line for line in lines if not line.startswith("#")))
+
+
 def _inside(top: Path, path: Path) -> str | None:
     """``path``'s name in the context at ``top``, the folders above it resolved but not the file itself, so a link
     keeps its own name; None when it's outside the context."""