Skip to content

Remove runtime layers, sidecars, port proxy, on-demand rootfs images, and more - #109

Open
Jaana Dogan (rakyll) wants to merge 1 commit into
mainfrom
runtime-layers
Open

Jaana Dogan (rakyll) wants to merge 1 commit into
mainfrom
runtime-layers

Conversation

@rakyll

@rakyll Jaana Dogan (rakyll) commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

This removes a set of features that widened what ate-env-api and the guest can be made to do, without review in proportion to that reach.

Removed:

  • Runtime layers and sidecars: ate-env manifest template --layer, --sidecar and --sidecar-readyz, and the guest's -sidecar and -sidecar-readyz support (cmd/ate-env-guest/sidecar.go). /readyz answers once the guest is listening.
  • Port proxy (internal/portproxy) and its ate-env-api flags --atenet-connect, --port-proxy-trusted-origins and --port-proxy-response-header-timeout, plus the CONNECT support in the fake router.
  • On-demand rootfs images: CreateEnvironmentRequest.rootfs_image, the server-side derivation of <base>-<digest> ActorTemplates, ate-env create --rootfs-image, and rootfs_image= in the Python client. ate-env manifest template --rootfs-image stays: operators can still register image templates themselves.
  • The NeMo Gym integration (integrations/nemo-gym), which depended on the port proxy and on runtime layers.
  • SWE-bench benchmarking (cmd/benchmarking, internal/benchmarking, benchmarking/), which depended on on-demand rootfs images.
  • Docs for all of the above: docs/rootfs-images, examples/runtime-layers, and the related README sections.

Security:

  • The port proxy forwarded HTTP from anyone who could reach ate-env-api's listener to any TCP port of any actor the Substrate router serves, in any atespace, including actors ate-env did not create. It authenticated no caller. Its own documentation listed DNS rebinding against the listener (and the MCP endpoint) as not yet blocked. In effect, the network boundary of the API service became the only access control for every port of every actor in the cluster.
  • On-demand rootfs images let any caller of CreateEnvironment make ate-env-api create ActorTemplates that run an image of the caller's choosing under the operator's base template settings: snapshot bucket, sandbox config and resources. ate-env-api's identity needed template-creation rights for this, and the derived templates were never garbage-collected, so callers could grow them without bound.
  • Sidecars let a template make the guest, PID 1 in every actor, start and restart arbitrary binaries next to it. This added a second data-plane process per actor with no isolation or resource limits of its own, which the port proxy then exposed.

Process suggestions:

Changes of this size to a project like this, which adds new network surface, new privileges for the API service, new wire-protocol fields, and new integrations, need substantial design discussion and security review before they are merged. They should not land incrementally through feature PRs. Any of these capabilities can come back through a design proposal with a threat model and explicit sign-off.

There may be more changes to clean up and should be followed up.

Most of the changes removed by this PR belong to an examples corpus. We cannot take a dependency to a large surface of third party integrations and have to patch this repo for security vulnerabilities, etc.

NeMo Gym integration and SWE-bench benchmarking

This removes a set of features that widened what ate-env-api and the
guest can be made to do, without review in proportion to that reach.

Removed:

- Runtime layers and sidecars: `ate-env manifest template --layer`,
  `--sidecar` and `--sidecar-readyz`, and the guest's `-sidecar` and
  `-sidecar-readyz` support (cmd/ate-env-guest/sidecar.go). /readyz
  answers once the guest is listening.
- Port proxy (internal/portproxy) and its ate-env-api flags
  `--atenet-connect`, `--port-proxy-trusted-origins` and
  `--port-proxy-response-header-timeout`, plus the CONNECT support in
  the fake router.
- On-demand rootfs images: CreateEnvironmentRequest.rootfs_image
  (field 4 and its name are now reserved), the server-side derivation
  of `<base>-<digest>` ActorTemplates, `ate-env create --rootfs-image`,
  and `rootfs_image=` in the Python client. `ate-env manifest template
  --rootfs-image` stays: operators can still register image templates
  themselves.
- The NeMo Gym integration (integrations/nemo-gym), which depended on
  the port proxy and on runtime layers.
- SWE-bench benchmarking (cmd/benchmarking, internal/benchmarking,
  benchmarking/), which depended on on-demand rootfs images.
- Docs for all of the above: docs/rootfs-images, examples/runtime-layers,
  and the related README sections.

Security:

- The port proxy forwarded HTTP from anyone who could reach
  ate-env-api's listener to any TCP port of any actor the Substrate
  router serves, in any atespace, including actors ate-env did not
  create. It authenticated no caller. Its own documentation listed DNS
  rebinding against the listener (and the MCP endpoint) as not yet
  blocked. In effect, the network boundary of the API service became
  the only access control for every port of every actor in the cluster.
- On-demand rootfs images let any caller of CreateEnvironment make
  ate-env-api create ActorTemplates that run an image of the caller's
  choosing under the operator's base template settings: snapshot
  bucket, sandbox config and resources. ate-env-api's identity needed
  template-creation rights for this, and the derived templates were
  never garbage-collected, so callers could grow them without bound.
- Sidecars let a template make the guest, PID 1 in every actor, start
  and restart arbitrary binaries next to it. This added a second
  data-plane process per actor with no isolation or resource limits of
  its own, which the port proxy then exposed.

Process:

Changes of this size to a project like this, which adds new network
surface, new privileges for the API service, new wire-protocol fields,
and new integrations, need substantial design discussion and security
review before they are merged. They should not land incrementally
through feature PRs. Any of these capabilities can come back through a
design proposal with a threat model and explicit maintainer sign-off.

Compatibility:

- Clients that set rootfs_image get an environment created from the
  base template, because the server drops the unknown field. Update
  them.
- ate-env-api deployments that pass the removed port-proxy flags will
  fail to start; the generated manifest never set them.
- Templates derived by earlier versions (`<base>-<12 hex>`) are unused
  and can be deleted with `kubectl-ate delete actor-template`.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: approve after one fix. The removal is complete and consistent. No dangling reference to a removed feature remains in code, CI, the Makefile, dependabot or the docs. The proto reserves field 4 and the name rootfs_image, and the generated Go and Python code agree with it. One item must change before merge: an old client that still sends rootfs_image gets a wrong environment and no error.

Before merge

  1. internal/apiservice/server.go (CreateEnvironment) with proto/ateenv/v1alpha/env.proto:116-117: proto3 keeps a reserved field as unknown bytes and does not reject it. I marshaled a CreateEnvironmentRequest with field 4 set, as an old client sends it. proto.Unmarshal returns no error and keeps 38 unknown bytes. CreateEnvironment does not look at them. Thus the server creates the environment from the base template, which runs the guest image, not the requested image. The caller gets no error, and its commands then fail for an unrelated reason. Old Python clients (Client.create(rootfs_image=...)) and older NeMo Gym providers send this field. Do these changes:
    • Reject a request that has unknown fields with InvalidArgument, for example if len(req.ProtoReflect().GetUnknown()) > 0.
    • Name the replacement in the message: register an image template with ate-env manifest template --rootfs-image and pass its name.
    • Add a test that sends field 4 and expects InvalidArgument.

Nits

  1. PR description, "Security", sidecars: a sidecar comes from the template, and only a template author can set it. That author already controls the container image and its command, thus a sidecar adds no privilege for that author. The sidecar process also runs inside the CPU and memory limits of the actor. The exposure was the port proxy, which made the sidecar ports reachable. Consider stating it this way, because the reason for a later reintroduction depends on it.
  2. PR description: "There may be more changes to clean up". I found none. go mod tidy changes nothing, staticcheck reports nothing, and the code that created ActorTemplates in ate-env-api is gone. GetActorTemplate stays, because server.go still uses it.

Verified

  • Head f36dcf5 merges clean with main fccd920. GitHub CI is green: Go build, vet, test and license headers; Python client tests; Wiz and zizmor scans.
  • Local: go build ./..., go vet ./..., go test ./... pass; go mod tidy makes no change; staticcheck ./... is clean.
  • Local Python (the ci.yaml steps): 116 passed, 26 skipped; ruff check clients/python is clean.
  • git grep for rootfs_image, portproxy, sidecar, nemo-gym, swebench, benchmarking/, atenet-connect: the only hits are the kept ate-env manifest template --rootfs-image path and its tests.
  • No Markdown link points to a removed path. The README keeps the --rootfs-image template flow and tells operators to delete the <base>-<12 hex digits> templates that older versions derived.
  • Probe for item 1: a Go test that unmarshals a request with field 4 (removed after the run).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants