Repository navigation
Sync upstream snapshot and sandbox runtime updates - #94
Merged
Merged
Conversation
…nt-substrate#2945) ## Summary Follow-up to agent-substrate#2848 (and agent-substrate#2847). While agent-substrate#2848 fixed upload memory accounting for multi-part ranged sparse files, small single-request files (`< 64 MiB`) still allocated and reserved worst-case `64 MiB` buffers at three layers (`64 MiB` GCS lookahead + `64 MiB` GCS chunk buffer + `64 MiB` zstd worker = `192 MiB` per file, and `128 MiB` for `manifest.json`). Under the default `--upload-memory-budget=512Mi`, a small gVisor snapshot (~7 files + `manifest.json`, ~17 MB total) still required `> 1.2 GiB` of budget reservations, forcing concurrent `SuspendActor` uploads to queue behind the semaphore (2–3 files at a time). ### Changes - **Proportional GCS single-request buffers**: When the upload stream has a known size bound below `64 MiB`, size both the `PutObject` lookahead buffer and the single-request `Writer.ChunkSize` to the actual object size (rounded up to `256 KiB` multiples, `512 KiB` minimum total instead of `128 MiB`). - **Right-sized zstd compression chunks**: For sparse files with `< 8 MiB` of populated data, scale the per-frame compression chunk and zstd window down to a power of two in `[1 MiB, 8 MiB]`, reducing per-worker memory from `64 MiB` down to as low as `8 MiB`. - **Budget accounting**: Update `Uploader` planning to reserve these proportional buffer sizes (`8.5 MiB` minimum for small sparse files instead of `192 MiB`), allowing all files in a snapshot burst to upload concurrently within the default `512Mi` budget. Large files (`>= 64 MiB`) and ranged sparse uploads (`parts >= 2`) are unchanged. ## Test Plan ### 1. GKE Cluster Benchmark (`glutton_baseline_10_users`, 10 gVisor workers, default `512Mi` budget) | Metric (`grpc_SuspendActor`) | Oct 15 Baseline (pre-agent-substrate#2708) | `main` (`90de075f`, with agent-substrate#2848) | This PR | Change vs `main` | | :--- | :---: | :---: | :---: | :---: | | **p50** | 400 ms | 850 ms | **390 ms** | **-54.1%** | | **p75** | — | 1,000 ms | **420 ms** | **-58.0%** | | **p90** | 460 ms | 1,400 ms | **450 ms** | **-67.9%** | | **p95** | — | 1,600 ms | **520 ms** | **-67.5%** | | **p99** | 610 ms | 1,700 ms | **640 ms** | **-62.4%** | | **Average** | — | 905.4 ms | **414.3 ms** | **-54.2%** | | **Throughput** | — | 4.39 req/s | **5.73 req/s** | **+30.5%** | ### 2. Microbenchmarks (`pkg/objectstorage/upload_bench_test.go`) | Benchmark (`512 MiB` budget, 50 ms RTT) | `main` (agent-substrate#2848) | This PR | Speedup / Memory | | :--- | :---: | :---: | :---: | | `BenchmarkConcurrentSnapshotUploads/gcs_shared_512MiB` (80 small files) | 2,085 ms / 5,375 MiB | **73.6 ms / 83.2 MiB** | **28.4x faster, -98.5% B/op** | | `BenchmarkLargeSparseUpload/4_actors_gvisor_7files_each` (4× [256 MiB ranged + 6× 64 KiB]) | 905.5 ms / 1,613 MiB | **358.4 ms / 138 MiB** | **2.5x faster, -91.4% B/op** | | `BenchmarkLargeSparseUpload/1_actor_256MiB_ranged` (large file only) | 115.6 ms | **114.6 ms** | Unchanged | Part of agent-substrate#2553
…-substrate#2943) Terminate now cancels the actor's upload or detach in progress and waits for it to stop before it forgets the actor, and keeps new ones from starting while it runs. Once the control plane has terminated the actor, atelet is no longer uploading its snapshots in the background, so the control plane can delete their objects. If atelet restarts while Terminate waits, it resumes the upload or detach from the state file until the control plane retries Terminate. Part of agent-substrate#2069 and agent-substrate#2571 AI use: AI-assisted, I reviewed and tested it - [x] Tests pass - [x] Docs updated
…ithout the router's alert (agent-substrate#2961) `TestIngressAuthentication` fails about one e2e job in three since static-mtls became the e2e default (5 of the 15 pr-workflow runs since agent-substrate#2390 merged): the refused client's request ends with `http2: client conn could not be established` instead of the router's TLS alert, and the test calls the refusal unproven. The router closes the connection right after the alert, and the port-forward (`portforward.go:549 ... write: broken pipe`) or Go's HTTP/2 client reports that close first. Recent instances, all on the `https` listener: - https://github.com/agent-substrate/substrate/actions/runs/39402878520/job/118561649470 (untrusted CA) - https://github.com/agent-substrate/substrate/actions/runs/39408627974/job/118580615105 (unlisted SPIFFE ID) - https://github.com/agent-substrate/substrate/actions/runs/39410976322/job/118588372580 (no client certificate) - https://github.com/agent-substrate/substrate/actions/runs/39411585756/job/118590529170 (no client certificate) - https://github.com/agent-substrate/substrate/actions/runs/39413905393/job/118598129904 (unlisted SPIFFE ID) **Change.** A refusal that fails without a TLS alert is retried on a fresh port-forward, three times at most. An alert is still required, and a client the router lets through still fails the test. **Tests.** `go vet` and the suite compile; the retry only shows under the flake, which CI's e2e jobs exercise. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR (none needed)
…t safely (agent-substrate#2883) Fixes agent-substrate#2938. Synchronous suspend could retain an uploaded checkpoint while controlapi dropped its local-copy record. Keep a matching copy and its node binding while releasing the worker. - Detach pinned actors before runtime fallback so compatible workers elsewhere can preserve memory. - Allow Resume and Revert to forget an unreachable node after the latest local snapshot is durable. PENDING snapshots stay pinned, version conflicts return `Aborted`, and assigned or crashed executions still require termination. - Preserve snapshot identity through template changes, detach, and restore retries. - Record cross-node restores as `snapshot.kind=latest` and consolidate attachment rules in the API field docs. ## Validation Passed on Linux/amd64 with Go 1.27.2 as an unprivileged user: - Focused controlapi and functional regressions with `-race -count=3 -shuffle=on`: 603 passing test entries, no skips. - `go test -race -count=1 ./...`: 11,078 passing test entries across 181 packages. - `go build ./...` - `bash hack/verify-all.sh` The full race suite skipped 44 entries: 38 root-gated, three requiring IPv6 loopback, two requiring a GCS emulator, and one requiring an S3 integration environment. Functional tests use PostgreSQL, envtest, and a fake atelet. New regressions fail without the fixes. Real gVisor/micro-VM E2E and privileged root-gated tests were not run. AI assistance: Codex helped implement the review fixes, review the diff, and run validation.
…gent-substrate#2944) Image volumes were only read-only at the top mount. Mounts nested under them, or mounted under the source later, stayed writable. Current image volumes have no submounts, so this is hardening. Fixes agent-substrate#2933 - micro-VM: the share binds for image and system-info volumes are now read-only and private all the way down - imagecache: the single-layer image volume bind gets the same treatment. gVisor binds image volumes non-recursively, so gVisor actors never saw submounts, and this is hygiene The new root-gated test cases each fail without their fix. I ran them locally as root. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated
…the pool (agent-substrate#2279) Fixes agent-substrate#2082. ## The bug The layer pool is keyed by diffID and shared by every image on a node, but nothing checked a layer's content against its diffID: - The diffID comes from the image config, which the image's author controls. An image could claim a common base layer's diffID and plant its own content under it. - The old check compared a remote layer's `DiffID()` with the config's diffID, but go-containerregistry reads that `DiffID()` from the same config, so the check could never fail. - go-containerregistry checks the compressed digest only at EOF, and untar stops at the tar end marker, before EOF. ## The fix - Unpack hashes the uncompressed stream, drains it to EOF, and rejects a mismatch before the layer is renamed into the pool. - The layer flight is keyed by blob as well as diffID, so a lying image fails only its own pull, not honest pulls of the same diffID. This is safe because a pull's pins, not its flight, keep eviction off its layers, so the flight is only deduplication. - The cost is that concurrent pulls of the same diffID from different blobs (for example, the same layer recompressed) no longer share a flight. Each one downloads and unpacks the layer, and one rename wins. - A mismatch isn't retried per candidate credential. - The recorded size counts only what untar read. - The on-disk layout is unchanged, so the layout version stays at 1. Layers unpacked before this change were never verified and stay in the pool until eviction retires them. ## Not in this PR A node that already has a verified layer reuses it without downloading the new image's blob, so an image whose blob doesn't match its claimed diffID can still succeed there. Checking each image's own blob on reuse is the follow-up.
Right now it still has a bunch of logic that needs to be extended each time new functionality is added to cmd/ate-setup. Convert it to a simple wrapper that forwards args to cmd/ate-setup, and change direct references to it to instead be proper references to cmd/ate-setup
…nt-substrate#2937) Fixes agent-substrate#2917. ## Problem `ate-setup setup csi nfs` (and `--setup-csi=nfs|both`) patched the `csi-nfs-controller` Deployment with a privileged `socat` sidecar that exposed the driver's controller socket as plaintext, unauthenticated TCP: `csi-nfs-controller.kube-system.svc:50052`, and `<node>:10000` since the pod is host-networked. Anything with cluster network access could call `CreateVolume` / `DeleteVolume`. ## Change - **Envoy mTLS proxy for the NFS controller**, matching the hostpath driver and `docs/csi-deployment.md`: the sidecar terminates TLS with a serving certificate from `servicednssigner`, requires a client certificate, and accepts only the `ate-api-server` pod identity SPIFFE ID (rendered from the install namespace via `installdefaults.SPIFFEID`). It proxies raw TCP to `/csi/csi.sock`, so gRPC streams are passed end to end with no HTTP route timeout that could cut a slow `CreateVolume` short. No admin port, since the pod runs on the host network. - **`CSIDriverConfig` `nfs.csi.k8s.io`** now sets `tls: {enabled, usePodIdentity, serverName: csi-nfs-controller.kube-system.svc}`. The `.svc` form is the only DNS name `servicednssigner` issues. - **Ordering**: the Service and the Envoy ConfigMap are created before the driver, because the signer refuses to issue a certificate for a pod no Service selects yet (same rule the hostpath step already follows). - **Not privileged**: the proxy runs as root because the driver creates the socket mode 0755, but with `allowPrivilegeEscalation: false`, all capabilities dropped and a read-only root filesystem. The hostpath driver's Envoy loses `privileged: true` the same way. - **Upgrade path**: the patch carries a `$patch: delete` for the `socat` container. The upstream manifest is server-side applied, which leaves containers another field manager added in place, so re-running setup on a cluster that still has the old sidecar removes it. - **Readiness probe** on the proxy, so the Deployment rollout and the Service only count the pod once Envoy has its certificate and is accepting connections. - `hack/setup-csi-nfs-kind.sh` makes the same changes; `docs/csi-deployment.md` gets the `.svc` server name (the example previously named `.svc.cluster.local`, which fails verification), the Service-before-pod note, and a non-privileged security context in the example. ## Verification on kind `ate-setup setup csi both` and `hack/setup-csi-nfs-kind.sh` were run against a kind cluster, including over an install that still had the `socat` sidecar: - `csi-nfs-controller` runs 6/6 with `envoy` in place of `socat`, security context `{runAsUser: 0, allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities: {drop: [ALL]}}`; the hostpath proxy runs with the same context. - Issued serving certificate SAN: `DNS:csi-nfs-controller.kube-system.svc`. - TLS client without a certificate: `tlsv13 alert certificate required`. Client with a certificate for a different service account: `alert certificate unknown`. Plaintext: no bytes returned. - A gRPC client holding the `ate-api-server` pod identity, using the same TLS settings as `internal/volume/csi` (TLS 1.3, ALPN h2, servicedns trust bundle, server name from the `CSIDriverConfig`), completed `GetPluginInfo`, `ControllerGetCapabilities`, `CreateVolume` and `DeleteVolume` through the proxy. The same client with server name `csi-nfs-controller.kube-system.svc.cluster.local` fails verification, which is why the doc example changed. Not run here: the external-volume e2e suite. The local Docker VM lost outbound network partway through, so atelet could not fetch the gVisor release and no worker could host the probe actor; the suite failed before reaching volumes. CI runs it.
…agent-substrate#2796) Replaces agent-substrate#1551, which was four commits from September that no longer applied: the restore source message, the download path and the module path all changed underneath it. Same design, ported onto today's main, after agent-substrate#2756 and agent-substrate#2889, in four commits, plus three from review: the sharing derived from the snapshot's owner, the gVisor worker evicting a shared file's pages before unlinking it, and the manifest read through the cache. Milestone M2 of the node-local artifact cache from agent-substrate#690: an actor restoring from a shared snapshot, the one a tag holds and every actor created from it starts on, gets its files from a node-level cache instead of its own download. **Sharing is the control plane's knowledge.** `ObjectStorage` gains `SnapshotSharing`, the property rather than the snapshot's kind, so new shared snapshot kinds need no atelet change. ate-api-server sets it on Restore from the snapshot's recorded owner: SHARED for a tag's snapshot, borrowed at creation and immutable, PRIVATE for the actor's own, which its next suspend replaces, and for an unrecorded owner; the local-copy fallback object carries it too. Neither side infers it from the URI. **The cache.** A `filecache.Store` under `/var/lib/ate/snapshot-cache`, behind `--snapshot-cache-dir` (empty disables), `--snapshot-cache-min-age` and `--snapshot-cache-fetch-timeout`, opened at startup with crash debris swept before atelet serves. The directory must be under the node state dir: the object-store plugin writes only there, so a cache elsewhere could never be filled, and startup refuses it. Every flag is validated whether or not the cache is enabled, and the help text says which are read only while it is. The image cache's base-path check moved to `nodepath.UnderBasePath`, which both caches use. **Eviction.** The cache runs on the eviction loop every file cache on the node shares, `cmd/atelet/internal/filecachegc` from agent-substrate#2889: `--snapshot-cache-gc-period`, `--snapshot-cache-max-bytes` and `--snapshot-cache-gc-dry-run`, a budget with a 90% low water, least recently used first, never an entry a restore still links. Its size, evictions and passes appear under the existing `ate.filecache.*` instruments as a second value of `ate.filecache.name`, `snapshot`; no new instrument and no new loop. Every flag is validated whether or not the cache is enabled, as the other flags in atelet are, and the help text says which ones are read only while it is; a disabled cache starts no loop and reports no size. **The restore path.** A restore whose durable snapshot is declared SHARED stages each file through the cache; concurrent restores of one snapshot share a single download, later ones read disk. How a hit is served depends on the sandbox class the request's runtime resolved to. ateworker-gvisor gets hard links. Once the restore's page loading is done the worker evicts the memory and filesystem images from the page cache and unlinks them from its restore dir, keeping only the small descriptors linked: a file with another link is treated like a preserved restore dir, since unlinking it would free nothing and would leave its pages charged to the actor's cgroup, the case agent-substrate#1917 fixed. So a link protects an image's entry through the restore and the min age, not for the actor's life; an entry evicted later is still read to the end through the descriptors runsc holds open, and the next restore reads the images from local disk. ateworker-microvm gets private hole-preserving copies, not because a link would corrupt the cache, it replaces `config.json` by rename and its delta merge copies when the inode is shared, but because a link would cost more than it saves: an on-demand restore keeps `memory-ranges` open for the VM's life, which would pin the entry that long and turn every suspend's in-place merge into a full copy, and the worker's page-cache eviction would drop the cached pages for every other restore; agent-substrate#2948 tracks what links would take. EXDEV on a link, from a cache bind-mounted apart from the restore dirs, degrades to a copy, warned once per process. The manifest of a shared snapshot is read through the cache too, so a restore that finds the snapshot on the node makes no request to object storage; a private snapshot's manifest is read fresh, as the commit marker it is. The cache key is the object's address without its query, so a signed URL's credential reaches neither the key, the cache's errors, `meta.json` nor the logs; and a restore that fails after staging removes its restore-state dir, so the links do not hold the cache's entries until the next lifecycle call. The fetch goes through the object-store plugin as the direct path does, one file into a scratch dir on the cache's filesystem; because the cache then writes the restore dir by path rather than through the plugin's `os.Root`, a symlink waiting there fails the restore instead of being followed. **Not here.** Private snapshots stay uncached by design. Links for ateworker-microvm are agent-substrate#2948, a worker change. The open-and-sweep code the two file caches share in shape is still duplicated, since the snapshot cache can be disabled and degrades to copies across mounts where the asset cache refuses to start. **Tests.** ate-api functional: a golden restore declares SHARED, the actor's own snapshot PRIVATE. ate-api unit: the detach-and-resume test from agent-substrate#2852 compares the whole Restore snapshot, so its fixture now puts the snapshot under the actor's own UID, and it expects PRIVATE for the downloaded copy and nothing for a local one. atelet: flag validation, enabled or not, including the eviction flags, the fetch timeout, and a directory outside the node state dir; two shared manifest reads download once and a private one reads fresh; open disabled, creating its root, sweeping debris; the sharing gate and the class modes; through the plugin-backed test atelet, two link restores share one download and one inode and the file is read-only, copy mode gives private inodes and survives a consumer's mutation, a nil cache downloads, a missing file's error reaches the caller with nothing left behind, a plugin error through the cached modes keeps its code and leaks no credential from a signed URI, a restore that fails at the worker after staging leaves no file in restore-state and no link on the cache's entry, four concurrent restores download once, a file in the checkpoint subdirectory lands there, and symlinks in place of the file or the subdirectory fail; on Linux, a restore dir on another mount gets a copy with no second download and one warning for two restores. ateworker-gvisor, on Linux: staged images hard-linked elsewhere are evicted after the restore wait and only then unlinked, with the other links intact. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…2886) - kubectl ate logs uses the singular 'actor' subcommand; the not-running error names atespace/name; quote the crash log message exactly. - GKE telemetry goes to the GKE managed OTel collector, not straight to GMP. - Workers normally export through the atelet relay, so an endpoint change reaches them when atelet restarts; the controller copy is the fallback. - List why atelet may start no relay (https, conflicting per-signal endpoints or compression, unparsable headers), per-signal endpoints as a fallback destination, and that ate-controller injects the endpoint only when it has one. A socket left by an earlier atelet keeps ateworker on the relay, failing every export instead of falling back. - The probes moved to :9091 (agent-substrate#2424); agent-substrate#1558 was closed unmerged. - Drop the removed pause from the lifecycle and snapshot-kind text. - The registry table now lists events.yaml. - Kind Prometheus does not show the metrics that ateworker and atecontroller only push over OTLP; the credential provider does not read ate-otel-config, so its OTLP push never reaches the collector. - Tracing is head-sampled, and the roadmap metrics already exist.
…ILABLE (agent-substrate#2824) Part of agent-substrate#799. The test could read the worker after registration but before it was marked ACTIVE. It failed twice on main this week. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
<!-- The PR title becomes the release note. Write it for users. --> Go has a critical security fix. Longterm we're going to automate these but dependabot can't handle it and we're undecided on solution, @MushuEE / @stshrive are looking. NOTE: we don't _have_ to bump the dockerfiles for go bumps, but in this case TLS is vulnerable so the download mechanism for gotoolchain is impacted, so I'm being extra-comprehensive. <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: Yes, I reviewed everything. Trivial diff, and tests were run <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [x] Tests pass - [x] Docs updated
…r a failed pull (agent-substrate#2989) Follow-up to agent-substrate#1509 (comment). `retireLayer` stats a layer before checking its pin. If the layer lands, and its pull then fails and unpins in between, eviction reports the layer gone and drops the image record, so the unpacked layer has no record until the image is pulled again. A missing layer now goes through `renameAsideUnlessPinned`, which checks the pin and stats again under `pinMu`. `dryRunRetire` already checks the pin first. There is no new test, since the window can't be hit without a hook between the two checks. `TestRetireLayerStatuses` still covers the missing and missing+pinned cases. AI use: AI-assisted, I reviewed the change. - [x] Tests pass
After mega-worker, this demo is no longer correct. We need more though on how workers autoscaling would work, so let's remove this demo in the meantime.
…ent-substrate#2924) Part of agent-substrate#777 Every snapshot fetch and upload on GCS now runs on a token limited to the actor's own snapshot prefixes, read-write on its own and read-only on the tag it was created from, instead of the node's project-wide credential. atelet mints a worker-for-actor JWT per actor through `MintWorkerActorJwt`, caches it until five minutes before expiry, and sends it on every transfer. The node plugin on GCS trades it at the exchange the control sidecar serves (agent-substrate#2762) for a downscoped token, caches the token per actor until twenty minutes before expiry, and runs each transfer on a client built with it, closed when the transfer ends. A request GCS refuses is retried once with a token the exchange re-derives, which is what carries the first upload after a repoint, and a second refusal is `PERMISSION_DENIED`. S3 ignores the JWT and behaves as before. The atelet manifest gives the node sidecar the pod namespace and the service DNS trust bundle it dials the exchange with. Two e2e tests, GCS only, prove it against the bucket: the token reads the golden and cannot write under it, writes only under the actor's own prefix, reads its own snapshot and not a sibling's, and the exchange refuses the actor's own token kind, another audience, a stale UID, and a node the actor has left. With the `PerTemplateSnapshotLocation` gate, a repoint to another location is followed through the first suspend. Neither runs in PR CI, whose lane is kind over S3. Verified on a GKE dev cluster, installed from this branch with the gate on: the full e2e run passes, every suspend and resume in the demo and identity suites going through the new path, and both new tests pass. The six egress HTTPS tests were skipped, since they fail on main there too for lack of a GKE equivalent of the kind TLS setup script. Memory on the node sidecar was sampled through two demo suite runs: it peaks during transfers on the upload buffers, returns to its idle level between runs, and a thousand idle cached clients measure at 0.1 MiB locally. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
Fixes: agent-substrate#2659 ate-setup already enforces authorization by default, and there's no compatibility promise before v1.0, so the off switch isn't worth keeping. This removes `--experimental-enable-authz` from ate-api-server and `ateapi.authz.enabled` (`ATE_API_EXPERIMENTAL_ENABLE_AUTHZ`) from ate-setup. AccessPolicy RPCs keep the same checks. #### Breaking change ate-api-server won't start without `--authz-bootstrap-owners`. ate-setup always sets it, so only installs that apply `manifests/ate-install` directly need to add it. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…etach on crash (agent-substrate#2611) Some storage backends (e.g. Cloud Filestore IP export rules) restrict a volume to the node it is published to via CSI `ControllerPublishVolume` / `ControllerUnpublishVolume`, so every publish must be matched by an unpublish to the same node. Since agent-substrate#2222 , detach targets `ActorStatus.assigned_node`, which survives crashes and worker loss. This PR adds checks before attach and test coverage for the paths where detach used to be skipped. Follows agent-substrate#2094, which covers the publish side. - *ateapi: check mounted volumes are attachable before publishing* - *test(ateapi): cover volume detach via assigned_node after crash or worker loss* #### Changes - Worker without a node name: `assignWorkerAttempt` rejects such a worker with when the actor mounts external volumes. Previously the resume failed in `ensureVolumesAttached`, after the worker was booked. - Volume checks before attach: `ensureVolumesAttached` checks that every mounted volume is `STATUS_CREATED` and has a `storage_volume_id` before any attach call, so one bad volume can't leave earlier volumes published. - Unit tests, functional tests - E2E: `assigned_node` equals the worker's `node_name` while RUNNING and is cleared after suspend and revert. The OSS NFS driver doesn't implement `ControllerPublishVolume`. - [x] Tests pass
…nt-substrate#2486) After a node is deleted, revert can still hang for a CRASHED actor or a paused actor whose latest upload is PENDING. Delete can hang for any actor still bound to that node. agent-substrate#2883 already lets revert and resume forget an unreachable node when the actor has no worker assignment or crash and the latest local snapshot has a COMPLETED durable copy. When no atelet is indexed on the assigned node, `ensureAteletTerminated` uses `NodeGone` to check the Node through the live Kubernetes API. It skips termination only on NotFound, after checking again that no atelet appeared. Existing nodes and lookup failures stay retryable. The API server gains `get` permission on nodes. External volumes still go through `DetachVolume`: nil and NotFound succeed, other errors propagate. Kubernetes Node deletion does not fence a machine that is still running; resources on such a machine may remain. Rebased onto main `872ab2077`. The whole-workflow test comment now identifies its paused actor's PENDING upload. Validation: - Targeted revert/delete/detach and `TestNodeGone` tests pass with `-race -count=3 -shuffle=on` and Docker-backed database tests enabled. - The full race suite passes all 15 ateapi packages with tests. Overall it reports 10,601 passing test/subtest events, 8 skipped, and two failures in `cmd/ateworker-microvm/internal/child`: `TestReapedWhenItDiesOnItsOwn` and `TestStop`. Both report `unsupported operation` on macOS and reproduce from unchanged main `872ab2077`; the non-Linux pidfd implementation returns `errors.ErrUnsupported`. - `hack/verify-all.sh` passes (using local Python 3.13 and clang-format 23.1.3). - A fresh independent review found no issues in the rebased patch. `git diff --check` passes. Earlier kind runs exercised real worker deletion, paused/crashed revert and delete, and the kept-node control. Those live runs were not repeated for this rebase. Current volume tests use a mock plugin; no real CSI-driver validation is claimed. Part of agent-substrate#2470. Resume without a completed latest upload remains separate. AI assistance: Claude Code helped write the original code and tests. Codex clarified the fixture comment, rebased the branch, reviewed the integration, and ran validation.
* If DeleteActor fails midway, it leaves the actor in DELETING state. * If the worker pod goes away, or the worker restarts, before the retry, crashBoundActor moves the actor to CRASHED. * RevertActor accepts CRASHED, so someone can revert the actor and resume it. This is not safe, because there's no guarantee that a DELETING actor can be recovered (volumes may have already been detached, or snapshots cleared). _____________________________ This also addresses agent-substrate#1981 (comment)
…#2781) Part of agent-substrate#1756. - ate-api-server's `--actor-jwt-issuer` again accepts any OpenID issuer identifier: https, a host, and no user info, query, or fragment. A path is allowed. ate-api-server only writes the value into `iss`, so it shouldn't depend on where the discovery document is served. - The `https://<host>` rule stays in ate-idp-server and in ate-setup, which deploys ate-idp-server with every install. ate-idp-server serves the discovery document and key set at fixed root paths, so it only works with an issuer that has no path. - The flag's help text says the same. For reviewers: installs through ate-setup behave as before. Only an operator who runs ate-api-server from their own manifests, and serves the documents somewhere else, can now use an issuer with a path. Testing: unit tests for the issuer check. I didn't run e2e, since nothing an install deploys changes. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…tions (agent-substrate#3007) agent-substrate#2718 removed the legacy `--deploy-ate-system`, `--delete-ate-system`, and `--setup-csi=nfs` flag-translation shim from `hack/install-ate.sh` and `hack/install-ate-kind.sh`, forwarding arguments directly to `go run ./cmd/ate-setup "$@"`. Update the remaining callers in `benchmarking/` and `hack/run-microvm-demo.sh` to pass the `ate-setup` subcommands (`deploy ate-system`, `delete ate-system`, `setup csi nfs`) instead of the removed flags. Fixes agent-substrate#3006 - [x] Tests pass - [x] Docs updated
…nt-substrate#2732) ### What Raises the circuit breaker limits on three clusters in `manifests/ate-install/atenet-egress.yaml`: | Cluster | Before | After | |---|---|---| | `mitm_internal` | 16,384 | 1,048,400 | | `egress_forward_proxy` (HTTPS to destinations) | unset (Envoy default 1,024) | 65,536 | | `egress_forward_proxy_cleartext` (HTTP to destinations) | unset (Envoy default 1,024) | 65,536 | Each cluster gets the same value for `max_connections`, `max_pending_requests` and `max_requests`. ### Why **`mitm_internal` caps the tunnels per replica.** Each open tunnel holds one connection and one request on this cluster. HTTP/2 multiplexing of CONNECT doesn't change that, because it only shares the actor-to-gateway connections. At 16,384, the next CONNECT gets a 503. **The destination clusters were running at Envoy's default of 1,024 connections.** Envoy keeps a separate connection pool per worker thread per destination, so the count grows with threads × destinations, not with tunnels. At 1,000 destinations and 16 threads that's already 16,000 pools. Past the limit, requests queue and then fail with `503 UO`. - Scale tests measured 16–17k destination connections at 16 threads, with 250k–500k tunnels. - 65,536 covers 1,000 destinations × 32 threads × 2 requests at once per destination per thread. ### Consequence - **Increased memory usage**: Up to 1,048,400 tunnels, about 49–56 GiB at ~49 KiB (HTTP) or ~56 KiB (HTTPS) each. - Currently the Envoy container has no memory request or limit. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…n merge (agent-substrate#2623) If a checkpoint attempt fails mid-flight, retrying `CheckpointWorkload` can fail due to leftover or displaced files on disk: 1. **gVisor**: `checkpoint-state/` was not cleared before checkpointing, leaving partial files from a failed attempt behind. 2. **microVM**: `MergeDeltaIntoBase` moved `restore-state/memory-ranges` into `checkpoint-state/` before merging, so a failed checkpoint followed by a retry (`os.RemoveAll(checkpointDir)`) permanently deleted the restore base. **Fix:** - Clear `checkpoint-state/` before running gVisor checkpoint (matching microVM). - In microVM `MergeDeltaIntoBase`, merge delta pages onto `baseFile` in place under `restore-state/` and rename into `checkpoint-state/` only after the merge succeeds.
…uest (agent-substrate#2162) A benchmark that wants an actor to do a whole on-sandbox activity, a build, a test run, a file churn, had to issue every resource operation as its own request through the router. That costs a round trip per operation, and it leaves the driver unable to hold the actor busy for a chosen length of time without knowing how long each operation takes on the sandbox at hand. RunScript takes the activity as a script and runs it in-process, holding the request open until it finishes. A script is a tree. A Block runs its steps in order and may cycle them for a wall-clock budget; a Step is a nested Block or an Operation; an Operation runs its requests at the same time and waits for all of them. A budget is checked between passes: the pass under way finishes, so a loop overruns by at most one pass and no request is cut short, and a nested block stops cycling once any ancestor's budget has elapsed the same way. The only errors are the caller's context ending and a failing request, which names the request's path in the tree and keeps its status code. The request set is the service's work requests plus sleep. UseCPU and OpenFD are left out because they install a background load that outlives the script; BurnCPU is the CPU spike. The response mirrors the tree with summed vital stats at every node: requests run, bytes written and read to disk and RAM, burn iterations, time slept, and the node's own wall clock and entry count. A node inside a looped block sums over its passes, so the response is the size of the script however long it ran, and the root carries the totals. The fake actor answers a script with the mirrored tree, so drivers can be tested against it. Fixes #<issue_number_goes_here> > It's a good idea to open an issue first for discussion. - [ x ] Tests pass - [ x ] Appropriate changes to documentation are included in the PR
When a Checkpoint with an immediate upload, or a PromoteSnapshot, makes a snapshot durable, atelet now cancels any upload of an older snapshot still in progress and waits for it to stop before returning. As during a PromoteSnapshot, no background upload starts while a Checkpoint with an immediate upload runs. Once the control plane learns that a newer snapshot is durable, atelet is no longer uploading an older snapshot in the background, so the control plane can delete its objects. Once the new snapshot is committed, the canceled upload deletes the older snapshot's local copy before the Checkpoint or PromoteSnapshot returns. Only a Run or Restore in progress, which may read any local snapshot, keeps a superseded local copy. Part of agent-substrate#2069 and agent-substrate#2571 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
… data plane (agent-substrate#2686) Adds what it takes to measure ate-api-server's and Postgres's cold resume capacity on the fake data plane from the benchmark automation, plus the first tests that use it. - **boomer: record server-side latency.** `resumepaused` records `locust_server_duration_milliseconds` from the `x-server-elapsed-us` trailer that `ateinterceptors.ServerUnaryInterceptor` sets, next to the client latency. The gap between the two is network and client time. - **Scrape boomer in automation runner Jobs.** The runner container declares port 8001 and `monitoring.yaml`'s `boomer-worker` job keeps `substrate-benchmark-runner` pods, so automated runs get the same latency histograms as manual ones. - **Shape the control plane for capacity runs.** A locust test can carry a `controlPlane:` block (`benchmarking/automation/controlplane.py`): - `apiServer:` sets ate-api-server's replicas, CPU and memory (requests = limits) and its Postgres pool size. - `postgres:` loads `pg_stat_statements` with a `postgres_exporter` sidecar that Prometheus scrapes, and passes extra Postgres settings. Placement is left to the install: capacity tests install with `--cordon-control-plane`, which keeps ate-api-server and Postgres on their own tainted pools, away from the load. The block patches the installed manifests after the deploy; a TODO in `controlplane.py` records the drift guards that shim still lacks. - **Summarize ladder runs per step, on the runner.** For a ladder run, `benchmarking/locust/ladder_steps.py` reads each step's boundaries from the runner's log, queries the in-cluster Prometheus over the step's steady window, and appends a `ladder_step_NNN` row per step to `stats.jsonl`: throughput, client and server latency, CPU and memory of ate-api-server, Postgres and fake-atelet, and Postgres connections, commits and waits. It is additive, like the server telemetry harvest it follows: a failure is logged and the run's other results are kept. - **Cold resume tests.** `tests.yaml` adds two: - `ateapi_cold_resume_pause_fake_2cpu`: a 100–1500/s cold resume ladder against one 2 CPU replica. - `ateapi_cold_resume_pause_fake_smoke`: the same setup at 50 actors and 20 → 50/s, a quick end-to-end check. Each actor books 10m CPU, since a fake Worker reports its node's allocatable as its capacity and placement must never bind. The benchmarking README explains the per-step rows. **Running them.** The target cluster needs the `ate-control-plane` and `ate-postgres` node pools `--cordon-control-plane` expects, and a node with room for the runner. On current `main` the target cluster's env also needs `SNAPSHOT_LOCATION` (agent-substrate#2544) and `ATE_INGRESS_ALLOWED_SPIFFE_IDS` (agent-substrate#2390), which every install now requires. Addresses agent-substrate#2261 - [x] Tests pass (`go test -race ./internal/benchmarking/...`, the `benchmarking/automation` and `benchmarking/locust` Python tests, `hack/verify-all.sh`) - [x] Appropriate changes to documentation are included in the PR
…-substrate#2973) The version skew policy said each release ships its SandboxConfig under its own name and keeps the earlier ones. That is not how it works: it now holds a list of versions with a defaultVersion, and every runtime bump has replaced the list, so the upgrade's `deploy sandboxconfig` removes the old version from the cluster. atelet restores a snapshot on the version it was taken with, so a release that keeps the previous version lets actors suspended on it keep their memory across the upgrade. Write that down as the rule for every sandbox class: a bump adds the new version as the default, keeps the previous one unless it has a known vulnerability, and leaves its assets in place. Step 4d and the step 4 rollback in the runbook now say what suspended actors see, removing the TODOs. Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.) Fixes agent-substrate#1686 AI use: No, written by human. - [ ] Tests pass - [ ] Docs updated
Follow up after agent-substrate#2879 Removes `sandbox_class` from the `Worker` spec (`ateapipb.Worker`) and reads the worker's sandbox class from `Worker.status.default_runtime.sandbox_class`. Each worker already reports its `default_runtime` (including `sandbox_class`) to `WorkerService.RegisterWorker` via `atelet` alongside its capacity before it can take any actors. Carrying `sandbox_class` on the `Worker` spec required `ate-controller` to copy it from the `WorkerPool` on `CreateWorker` and duplicated what the worker itself reports in `WorkerStatus`. Using `status.default_runtime.sandbox_class` as the single source of truth removes that duplication across `ate-controller`, `ate-api-server`, `fake-workersync`, and `kubectl-ate`. #### Behavior changes - **Class is fixed by the first registration.** `RegisterWorker` records the class from the first report and refuses a later report with a different `default_runtime.sandbox_class` (restorable runtimes must match it too). The pool's declared class no longer cross-checks the worker's report: the pod's shape comes from the pool, and the report is the pod's own authenticated statement. The proto comments on `WorkerStatus.default_runtime` and `RegisterWorkerRequest.default_runtime` document this. - **Before a worker registers it has no class.** `kubectl ate get/top workers` show an empty CLASS for it and `--sandbox-class` filters it out;`ate.workerpool.workers` counts it under `ate.sandbox.class=unknown` (registry note updated). The scheduler never placed on such a worker, and still doesn't:`Applies` rejects an empty class, which also covers the "no default runtime" guard agent-substrate#2879 added, so that guard is folded into the class check. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…e#2892) The egress handler only ever answers `dial=name`, so the `dial=address` routes, the three ORIGINAL_DST clusters behind them and the `original_dst_address` attributes in atenet-egress.yaml could never match. This removes them plus the extproc constants only they used, and fixes comments and READMEs that still describe removed behavior (ActorIdentity extension, one pool per tunnel, MITM-only injection, egress-only dataplane image). `OriginalDstFilterStateKey` stays since the ingress upstream CONNECT sets it. The passthrough manifest test now rejects any ORIGINAL_DST cluster, and the edited manifest passes `envoy --mode validate` in the dataplane image (11 clusters down to 8). Split out of agent-substrate#2166. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…ger cross (agent-substrate#3003) Taking over agent-substrate#2726 from @ericcurtin so it can land before tomorrow's code freeze. The fix is all Eric's and his commits are kept as-is, thanks Eric! I only reworked the e2e test. CONNECT tunnels to different ports or actors no longer share one Envoy connection pool, so a tunnel can't inherit another tunnel's port or actor. The test now opens the :80 and :9090 CONNECTs as two streams on one HTTP/2 connection, so they always land on the same Envoy worker instead of by chance, and it's one connection through the port-forward, so no more retries or draining. AI use: AI-assisted to move this PR and fix a test Fixes agent-substrate#2705 - [x] Tests pass - [x] Docs updated (none needed) --------- Co-authored-by: Eric Curtin <eric.curtin@docker.com>
…d states (agent-substrate#2967) ResumeActor restored a paused actor's local snapshot and a suspended actor's durable snapshot through separate paths, but both code paths share mostly the same logic. Commits: 1. `ateapi: resolve the resume source only when restoring`: `ensureAteletRestored` resolves the restore source itself, so `loadActorForResume` only loads the actor and its template. `assignWorkerAttempt` checks the snapshot URIs with `checkSnapshotURI`, so a malformed one still fails with DataLoss before a worker is claimed, and the actor stays SUSPENDED. 2. `ateapi: build the Restore request in one branch`: one Restore request for both snapshot kinds, still local first. A local snapshot whose runtime the worker does not run drops to ROOTFS, as a durable one already did. 3. `ateapi: stop restoring an older local snapshot over a newer durable one`: restore from `snapshotToResume`, the newest snapshot that is local or has a completed durable copy. 4. `ateapi: place a resume by the snapshot it restores`: placement takes the sandbox class and runtime from `snapshotToResume` too. Placement still pins a paused actor to `status.assignedNode`. A follow-up PR makes it prefer the node of the snapshot it restores. AI use: Claude Code split this out of a larger branch, wrote the commit messages and drafted this description. I reviewed the diff and the messages. - [ ] Tests pass - [ ] Docs updated
…gent-substrate#2976) PR agent-substrate#2390 removed the hardcoded `--ingress-allowed-spiffe-ids` flag from `manifests/ate-install/atenet-router.yaml` and made `ingressAllowedSPIFFEIDs` (`ATE_INGRESS_ALLOWED_SPIFFE_IDS` / `--ingress-allowed-spiffe-ids`) required in `ate-setup deploy ate-system` when `--atenet-dataplane=envoy`. `deploy_substrate()` in `benchmarking/automation/orchestrator.py` does not set `ATE_INGRESS_ALLOWED_SPIFFE_IDS`, so automated benchmark runs fail at `deploy_ate_system`. Set `ATE_INGRESS_ALLOWED_SPIFFE_IDS` to `BENCHMARK_RUNNER_PRINCIPAL` alongside `ATE_API_AUTHZ_BOOTSTRAP_OWNERS`, and update the benchmark install docs. Fixes agent-substrate#2975 - [x] Tests pass - [x] Docs updated
…up (agent-substrate#2998) Every component logged this at startup: ``` WARN partial telemetry resource err="error detecting resource: conflicting Schema URL: https://opentelemetry.io/schemas/1.40.0 and https://opentelemetry.io/schemas/1.43.0" ``` `serverboot.newResource` sets the resource's schema with semconv v1.40.0, while the vendored SDK's own resource detector now uses v1.43.0. The merge then drops the schema URL. All attributes were kept: on ate-dev, atelet's `target_info` still had `service.*` and `k8s.*`. Only the schema URL was lost, and the warning made the resource look partial. This moves `serverboot` to semconv v1.43.0, and adds a check to `TestNewResourceDefaults` that the resource's schema URL equals the SDK's (`resource.Default().SchemaURL()`). With the old import, the test fails with `schema URL = "", want the SDK's "https://opentelemetry.io/schemas/1.43.0"`, so the next SDK upgrade that moves its semconv version fails a test instead of warning in every pod. boomer-worker's tracer resource had the same pin (`internal/benchmarking/boomer/trace`); it gets the same fix, its resource moves into `newResource`, and a new test checks its schema URL the same way. The only other resource with a pinned schema, in `ateclient`, runs no detectors, so it cannot conflict. Found while testing on ate-dev. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR (none needed)
…substrate#2926) Depends on agent-substrate#2883, which makes a synchronous suspend record `locality` and keep `assigned_node`. The docs here describe that behavior after it merges. ## Summary Following the removal of `PauseActor` in agent-substrate#2813 and defaulting `SuspendActor` to asynchronous mode in agent-substrate#2805, update documentation, protobuf comments (and generated code), production Go comments, and test/benchmark comments across the repository that still referred to `PauseActor` or "paused" actors. ### Key Updates - **Documentation (`docs/`)**: - Remove the standalone `Pause` glossary entry in `docs/glossary.md` and update `Actor`, `Checkpoint`, `Restore`, and `Suspend` definitions to describe `SuspendActor`'s two durability modes (`DURABILITY_MODE_SYNCHRONOUS` and `DURABILITY_MODE_ASYNCHRONOUS`, with async as the default) and clarify that snapshots are always uploaded to object storage. - Update `docs/api-guide.md`, `docs/architecture.md`, `docs/api-style-guide.md`, `docs/authorization.md`, `docs/observability.md`, and `docs/metrics/registry/metrics.yaml` to describe synchronous vs. asynchronous `SuspendActor` and node-local snapshot pinning instead of `PauseActor`. - **Protobuf & Generated Code (`pkg/proto/ateapipb/`, `internal/proto/ateletpb/`)**: - Update comments on `SuspendActor`, `RevertActor`, `DeleteSnapshotTag`, `ActorStatus.assigned_node`, `Snapshot.locality`, and `ateletpb` RPCs; regenerate `*.pb.go` and `*_grpc.pb.go`. - **Production Code & Tests (`internal/`, `cmd/ateapi/`, `cmd/atelet/`, `benchmarking/`)**: - Update comments and error message hints referring to `PauseActor` or "paused" actors to reflect asynchronous `SuspendActor` and node-local snapshots.
This is agent-substrate#1981, from @juli4n, but rebased + the review comments addressed. The code has changed a bit since then. Some Delete<Resource> methods need to orchestrate calls to other systems. Today this happens synchronously, within the inbound RPC call. If something fails half-way through, we need a way to denote that the resource is being deleted. Resources that don't have long running workflows for deletion have a simpler implementation (i.e. no Mark<Resource>ForDeletion) as they are removed in a single tx. If they ever need a long-running delete we can add support for them.
…nt-substrate#3056) Each ateapi replica reports the whole fleet on ate.workerpool.workers, and the default install runs two, so a sum across replicas counts every worker twice. The registry told readers they could add the counts together. Say that a sum across replicas double counts, and to take the max across replicas by pool namespace, pool, state and sandbox class before adding. To mitigate agent-substrate#3055 - [x] Tests pass - [x] Docs updated
…e project (agent-substrate#3042) Part of agent-substrate#777 setup-gcp granted the nodes' default compute service account `roles/storage.objectViewer` on the whole project, for pulling images from the GCS-backed Container Registry. Container Registry is gone, and gcr.io is now served by Artifact Registry, which the existing `roles/artifactregistry.reader` grant covers. Root on a node can use this account, so the grant let a compromised node read every snapshot. Existing installs keep the grant, since setup-gcp only adds bindings. Remove it with: ```bash gcloud projects remove-iam-policy-binding "${PROJECT_ID}" \ --member="serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com" \ --role=roles/storage.objectViewer --condition=None ``` Tested on a GKE cluster: after removing the grant, a host-network pod running as the node account was refused on the snapshot bucket, and a redeploy pulled every image from gcr.io without errors. AI use: AI-assisted, I reviewed it. - [x] Tests pass - [x] Docs updated
…e#3073) Inspired by agent-substrate#2979 AI use: No. written by human - [x] Tests pass - [x] Docs updated
…#3010) agent-substrate#2883 lets resume forget an unreachable node when a paused actor's latest snapshot is durable. When that upload is still PENDING, the actor stays pinned. If the node was deleted, every resume returns `Unavailable`, and nothing says the snapshot is lost. Now, when resume finds no atelet on the pinned node and `NodeGone` reports the Node object deleted, it reads the actor again. If the upload was recorded during the lookup, it detaches as usual. Otherwise it crashes the actor. The crash keeps `assigned_node` and the PENDING snapshot. `RevertActor` then skips the terminate, deletes the pending objects, and returns the actor to its last completed durable snapshot. A node that still exists, or a failed Node lookup, keeps the current retry. There are two known limits. The check runs only after resume finds room on another node, so with no room anywhere resume still returns `RESOURCE_EXHAUSTED`. And a Node object deleted while the machine keeps running crashes the actor even though its checkpoint is still on that machine. That's the same trade agent-substrate#2486 makes. A PENDING copy may already be fully uploaded if atelet died before reporting it. Resume still crashes the actor, and revert deletes that copy. Recovering it needs ate-api-server to check for the manifest, which is a follow-up. No proto changes. `ResumeActor` can now crash the actor and return `FAILED_PRECONDITION` in this case, and `docs/api-guide.md` says so. Part of agent-substrate#2470 Supersedes agent-substrate#3005. Its "upload completes during lookup" regression case is adapted here. AI use: Claude Code helped write the change and tests. I reviewed them. - [ ] Tests pass - [x] Docs updated
…bstrate#2159) Golden initialization can need network access that runtime actors should not inherit. `ActorTemplate.goldenEgressPolicy` configures that access behind the `GoldenEgressPolicy` preview gate, with client preview opt-in. The template reconciler creates the golden actor suspended, creates its policy through the existing `CreateActorEgressPolicy` RPC, then calls `ResumeActor`. Retries recover partial creation and verify that existing rules match the template. Policy errors or conflicts block the controller from resuming the golden. Ordinary actors do not inherit this policy. The template remains immutable and keeps the metadata-free `EgressPolicyTemplate` wrapper. It requires at least one rule and rejects a policy when golden snapshots are disabled. Existing templates can finish initialization after the preview gate is disabled. Actor and policy creation are separate calls, so callers must leave golden startup to the controller. Motivation and API rationale: agent-substrate#2358. Related to agent-substrate#1324. - [ ] Tests pass — affected API, functional, storage, authorization, and preview suites pass with race detection; focused cases passed three shuffled runs. All repository verifiers and API validation pass. CI is pending. The full root-module race suite was not rerun locally due limited scratch space. - [x] Appropriate changes to documentation are included in the PR AI assistance: Codex helped implement, rebase, and test these changes. Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
This change enables H/2 for origins in Envoy egress. Also separate connection pools to origin servers for distinct actors, to avoid any problems if origins maintain actor specific state per connection. Switching H/2 on results in 20% (2600->3200) QPS increase from cold when benchmarking single core Envoy egress with H/2 traffic. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Yan Avlasov <yavlasov@google.com>
…ubstrate#3071) A duplicate entry error from validate.PtrSliceUnique carries the whole entry as its bad value, and apimachinery renders it as JSON in the error text. For a template or Run request with two env vars of the same name that is env[1]: Duplicate value: {"name":"API_KEY","value":"sk-..."} The RPC interceptors log the error with slog.Any("err", err). It is a string by then, so the logging handler's proto redaction cannot mask it, and the env value reaches the Info "Handle RPC" line even though the request in the same record shows value:"[REDACTED]". ToAPIError and ToGRPCStatusError now replace a proto bad value with its protoredact copy before building the text, so the error still names the duplicate but shows value:"[REDACTED]". The caller's ErrorList is left unmodified. <!-- The PR title becomes the release note. Write it for users. --> Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.) Fixes agent-substrate#3070 <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [ ] Tests pass - [ ] Docs updated
Directs agents to create all bash scripts in standalone `.sh` scripts to improve shellcheck coverage.
…gent-substrate#2979) Follow-up to agent-substrate#2279. @BenTheElder's last review asked for these, and they were pushed to agent-substrate#2279 before it merged, but main got the earlier commit. - Layout version moves to 2. Layers in a version 1 pool may not have been hashed against their diffID, and eviction never retires a layer that images keep using, so atelet refuses the old pool. - `pullWith` logs a diffID mismatch at WARN with ref, digest, and diffID. The RPC error was its only trace. ## Breaking change After an upgrade or rollback, atelet refuses the existing image cache. Before 1.0 a minor upgrade is a reinstall, so the upgrade runbook does not change. agent-substrate#2392 also moves the layout version to 2. Whichever lands second takes 3. Written with Claude Code.
…ubstrate#3077) The mitm and cleartext legs' access logs printed the path with %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%, which includes the query. The mitm leg logs requests the gateway has decrypted, and actor traffic can carry credentials in the query (API keys, signed URL tokens, OAuth codes). These lines go to stdout and the cluster log backend for every failed request and a 1% sample of the rest. Use %PATH(NQ:ORIG_OR_PATH)%, the same original-or-path choice without the query, which the router's access log already does for this reason. TestEgressManifestsLogPathsWithoutTheQuery fails if any access log format in the egress bootstrap prints the path with its query. <!-- The PR title becomes the release note. Write it for users. --> Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.) Fixes agent-substrate#3076 <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [x] Tests pass - [ ] Docs updated
…s to the snapshot bucket (agent-substrate#3037) Part of agent-substrate#777 setup-gcp gave atelet object admin on the snapshot bucket and on the whole project, so root on a node could read and write every snapshot without the credential exchange. atelet only needs its own identity for micro-VM sandbox assets, which lived in the snapshot bucket. On GKE they now move to a bucket of their own, `<BUCKET_NAME>-assets`, which setup-gcp creates and lets atelet read, and atelet gets no other storage role. Snapshot transfers already run on per-actor tokens from the exchange. #### Breaking change On GKE with micro-VM workers, recreate the cluster. Substrate does not upgrade a cluster in place, and a node keeps fetching micro-VM assets from the snapshot bucket, since their digests do not change when they move. Run `go run ./tools/setup-gcp create bucket` and `create iam` before `hack/install-microvm-deps.sh --install` or a micro-VM demo, since assets are now staged in `<BUCKET_NAME>-assets`, which those commands create. On any existing install, gVisor-only included, run `create bucket` before rerunning `create iam`, which now also grants on the asset bucket and stops if it is missing. setup-gcp also refuses a `BUCKET_NAME` longer than 56 characters, since the asset bucket name would exceed GCS's limit. Existing installs keep atelet's old grants, since setup-gcp only adds bindings. Once Substrate runs a release whose node plugin gets snapshot credentials from the credential exchange, and micro-VM assets are in the asset bucket, remove them: ```bash ATELET="principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${PROJECT_ID}.svc.id.goog/subject/ns/ate-system/sa/atelet" gcloud projects remove-iam-policy-binding "${PROJECT_ID}" --member="${ATELET}" --role=roles/storage.objectAdmin --condition=None gcloud storage buckets remove-iam-policy-binding "gs://${BUCKET_NAME}" --member="${ATELET}" --role=roles/storage.objectAdmin gcloud storage buckets remove-iam-policy-binding "gs://${BUCKET_NAME}" --member="${ATELET}" --role=roles/storage.bucketViewer ``` Tested on a GKE cluster with atelet's storage grants removed: atelet's identity was refused on the snapshot bucket and could read the asset bucket, a micro-VM actor booted from the asset bucket, suspended to a completed durable snapshot and resumed, and the full e2e suite passed. AI use: AI-assisted, I reviewed it. - [x] Tests pass - [x] Docs updated
…t-substrate#3019) Fixes agent-substrate#3083 `GetFromName` passed the name straight to `vishvananda/netns`, which joins it onto `/run/netns`. A name like `../../proc/1/ns/net` opened a namespace outside that directory. `CreateNamed` and `RemoveNamed` already reject these names. The check moves into `validateName`, and `GetFromName` calls it too. Today only tests call `GetFromName`, with names from `nodepath.ActorNetNSName`. - [x] Tests pass - [ ] Docs updated
… request times out (agent-substrate#2940) Fixes agent-substrate#2910. `DetachActor` ran its detach on the RPC's context. When a resume moved an actor off a full node (agent-substrate#2881) and the caller left mid-upload, for example when the router's resume budget ran out, the upload was cancelled and the next resume started it over. A snapshot slower to upload than that budget never moved. And because `DetachActor` cancelled atelet's report, nothing corrected the record if the caller failed to write it. Now the detach runs on the tracker's context and always makes atelet's report due. ```mermaid sequenceDiagram participant A as ate-api-server participant L as atelet participant O as object storage A->>L: DetachActor L->>O: upload snapshot Note over A: resume budget runs out L-->>A: Canceled Note over L: the detach keeps running O-->>L: upload done L->>L: delete local copies L->>A: ReportActorSnapshotState A->>A: record the detach ``` - If the detach finishes in time, `DetachActor` returns its state as before. Otherwise it returns the caller's error and the detach runs on. A second call waits for it. - Every detach makes the report due, including when the upload succeeds and the prune fails. - The upload timeout from agent-substrate#2831 now bounds a `DetachActor` call's upload, since the caller's context no longer does. - atelet's shutdown waits for a detach that outlived its call. Terminate cancels it and waits, as agent-substrate#2943 does for any detach. #### API changes - `Atelet.DetachActor` (atelet internal). The proto comment now says atelet reports the detach and that the detach outlives the call. Comment only, with the regenerated `atelet_grpc.pb.go`. #### Worth knowing - Each move now sends one redundant report. It gets `Aborted` while the actor is RESUMING, then `NotFound` once it runs elsewhere, and atelet drops it. - agent-substrate#2831's subtest asserted that a `DetachActor` call ignores the upload timeout. It now asserts the timeout applies.
…strate#3072) EgressIdleConns couldn't get past 64,512 connections. All of them came from the egress actor in one sandbox, and atunnel's redirect sends every connection to the same listener, so only the source port tells them apart. That's a per-actor limit, not a per-worker one. The worker now runs an egress actor in every sandbox, and EgressIdleConns splits its connections across them. The other egress scenarios still use actor-0's, so their numbers stay comparable with earlier runs. Also adds a 100k run, with atunnel at 4 CPUs / 16 GiB so neither limit is reached: | idle connections | per connection (after GC) | goroutines/conn | atunnel cgroup memory | CPU opening 200 conns/s | |---|---|---|---|---| | 25,000 | 26.9 KiB | 4.0 | 1.3 GiB | 0.15 cores | | 50,000 | 27.1 KiB | 4.0 | 2.6 GiB | 0.14 cores | | 100,000 | 26.8 KiB | 4.0 | 5.2 GiB | 0.19 cores | Memory per connection stays flat. Before agent-substrate#2908 it was about 157 KiB. The CPU is what opening new connections costs (about 1 ms each); holding the idle ones costs almost none. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…gent-substrate#3054) Flaky test fix, test-only change. `TestDownloadExternalCheckpointCachedSubdirAndSymlinks` fetches its files in parallel; on a cold cache every fetch mints, and `fakeJWTMinter` appended to its record from several goroutines. The race detector failed `cmd/atelet` on it four times in the last day, once in the merge queue. **Change.** The fake records under a mutex. **Tests.** `go test -race -run TestDownloadExternalCheckpointCachedSubdirAndSymlinks -count=300 -cpu 1,2,4,8 ./cmd/atelet`: a race report before, none after, on macOS and Linux. `go test -race ./cmd/atelet` passes. AI use: AI-assisted. - [x] Tests pass - [x] Docs updated (none needed)
…rate#3068) sweperf always sent a synchronous SuspendActor and ignored the shared `--lifecycle-mode` flag that agentsession, ateenv and glutton honor. Lifecycle v2 defaults to async suspend, so sweperf couldn't measure the path most users take without a code change. sweperf now embeds `dynconfig.Lifecycle` like the other classes. With `--lifecycle-mode` pause, it sends an asynchronous SuspendActor, recorded as SuspendActorAsync. The default (suspend) keeps today's behavior, so existing runs are unchanged. `--lifecycle-mode` none and `--resume-mode` implicit are refused, since every sweperf cycle ends in a suspend and resumes with ResumeActor. note: pause is the existing `--lifecycle-mode` value for async SuspendActor according to how the other benchmarks do it; was wondering if we should change the naming for all since PauseActor is gone per agent-substrate#2813 AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [ ] Docs updated --------- Co-authored-by: Haowei Cai (Roy) <haoweic@google.com>
EItanya
marked this pull request as ready for review
October 21, 2026 22:25
EItanya
force-pushed
the
sync/upstream-main-20261021
branch
3 times, most recently
from
October 21, 2026 22:53
f9594f4 to
926ca01
Compare
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
EItanya
force-pushed
the
sync/upstream-main-20261021
branch
from
October 21, 2026 23:28
926ca01 to
32b11c1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bring the fork up to upstream
4b3e0003b(63 commits), preserving both histories and the fork's agentgateway configuration.Includes snapshot lifecycle fixes, actor-scoped GCS transfers, gVisor without a pause container, golden egress policies, and Go 1.27.3. Resolve API and workflow conflicts, update Helm permissions and snapshot-plugin configuration, and migrate Helm CI to the new installer commands.
Upgrade
substrate-crdsbefore the application chart. RemovepauseImagefrom custom SandboxConfigs andateApi.authorization.enabledfrom Helm values; authorization is always enforced. GKE microVM installs now use a separate asset bucket; follow the updated setup-gcp instructions.Validation: the full race suite passed locally with
NO_COLORunset and packages run serially.hack/verify-all.sh, all 36 Helm chart tests, manifest rendering, CRD parity, and the arm64 build passed. CI exposed stale API references and an obsolete Helm flag, now fixed.The pinned gateway image includes the HTTP/2 CONNECT authority fix. The regression reproduced locally before the fix and passes three runs each on gVisor and microVM with the new image. Local gVisor and microVM validation is complete: 160 main cases per runtime with four skips, plus nine egress-credential cases and one identity case each. The microVM credential case passed after adding its namespace grant to the local test values. All CI checks pass on
32b11c114: unit tests and verification, Helm checks and E2E, and gVisor plus microVM in both standard and preview installs.Use a merge commit to retain upstream ancestry. All integration fixes are amended into the single merge commit on this branch.
Used Codex to prepare the merge, resolve conflicts, update Helm and CI, and review and validate the result, including an independent agent review.