diff --git a/.agents/skills/detect-flaky-tests/SKILL.md b/.agents/skills/detect-flaky-tests/SKILL.md index 09fa827361..34462311a1 100644 --- a/.agents/skills/detect-flaky-tests/SKILL.md +++ b/.agents/skills/detect-flaky-tests/SKILL.md @@ -74,7 +74,7 @@ The two sandbox-class lanes are NOT a matrix — each `e2e-test` job runs the st `hack/run-e2e-kind.sh` command, the second with `E2E_SANDBOX_CLASS: microvm`). Split the each `e2e-test` job log at the "Run E2E tests (micro-VM)" step boundary: PASS/FAIL lines before it belong to the gVisor lane, lines after it to the microVM lane. Because the -steps are sequential, a gVisor-lane failure means the micro-VM step never ran — record +steps are sequential, a gVisor-lane failure means the microVM step never ran — record no microVM results for that run rather than counting them as failures. Parse `go test -v` output from each log: diff --git a/.agents/skills/triage-issues/SKILL.md b/.agents/skills/triage-issues/SKILL.md index be21b1b5dd..a2f283261a 100644 --- a/.agents/skills/triage-issues/SKILL.md +++ b/.agents/skills/triage-issues/SKILL.md @@ -53,7 +53,7 @@ Apply all areas that are touched. Most issues need one or two; a few need more. | `area/network` | Networking: atenet-router, Envoy, ext_proc, DNS, xDS, ingress (`cmd/atenet/`) | | `area/node` | Node agent and worker lifecycle: atelet, sandbox launch, OCI, cgroups (`cmd/atelet/`) | | `area/gvisor` | gVisor sandbox specifics: runsc integration, GPU in gVisor, gVisor OCI (`cmd/ateworker-gvisor/`) | -| `area/microVM` | Micro-VM sandbox specifics: cloud-hypervisor, kata, snapshot/restore (`cmd/ateworker-microvm/`) | +| `area/microVM` | MicroVM sandbox specifics: cloud-hypervisor, kata, snapshot/restore (`cmd/ateworker-microvm/`) | | `area/storage` | Snapshot storage, image cache, GCS/S3 backends, retention (`internal/imagecache/`, ActorSnapshot) | | `area/scheduling` | WorkerPool sizing, actor placement, HPA, resource allocation | | `area/observability` | Metrics, tracing, logging, OTLP, OTel SDK integration | diff --git a/.github/scripts/release/attach-binaries.sh b/.github/scripts/release/attach-binaries.sh new file mode 100755 index 0000000000..17679be480 --- /dev/null +++ b/.github/scripts/release/attach-binaries.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash + +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Attaches every file in to the draft release for TAG in +# GITHUB_REPOSITORY, replacing files of the same name. A release that is +# already published is left unchanged, so a rerun never changes its files. +# +# GITHUB_REPOSITORY= TAG= GH_TOKEN= \ +# .github/scripts/release/attach-binaries.sh +# +# Used by the draft job in .github/workflows/release.yaml. + +set -o errexit -o nounset -o pipefail + +[[ $# -eq 1 ]] || { + echo "usage: GITHUB_REPOSITORY= TAG= $0 " >&2 + exit 2 +} +dir="$1" + +# An assignment, so a failed query (API error, rate limit) stops the script +# under errexit instead of being read as "published" and skipping the upload. +is_draft="$(gh release view "${TAG}" --repo "${GITHUB_REPOSITORY}" --json isDraft --jq .isDraft)" +if [[ "${is_draft}" != true ]]; then + echo "${TAG} is published; its binaries are left unchanged." + exit 0 +fi +gh release upload "${TAG}" "${dir}"/* --repo "${GITHUB_REPOSITORY}" --clobber diff --git a/.github/scripts/release/ghcr-login.sh b/.github/scripts/release/ghcr-login.sh new file mode 100755 index 0000000000..9d6dadbbb8 --- /dev/null +++ b/.github/scripts/release/ghcr-login.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash + +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Logs docker in to ghcr.io as GITHUB_ACTOR with the token in TOKEN. ko and +# ate-setup's registry checks read the credentials docker login stores. +# +# GITHUB_ACTOR= TOKEN= .github/scripts/release/ghcr-login.sh +# +# Used by the images job in .github/workflows/release.yaml. + +set -o errexit -o nounset -o pipefail + +docker login ghcr.io --username "${GITHUB_ACTOR}" --password-stdin <<<"${TOKEN}" diff --git a/.github/scripts/release/refuse-published-release.sh b/.github/scripts/release/refuse-published-release.sh new file mode 100755 index 0000000000..35fae73965 --- /dev/null +++ b/.github/scripts/release/refuse-published-release.sh @@ -0,0 +1,44 @@ +#!/usr/bin/env bash + +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Fails if the tag VERSION has a published (non-draft) release in +# GITHUB_REPOSITORY. Publishing a release's images again replaces the digests +# that operators install by tag, so it is refused once the release is +# published, whether or not the run asked to overwrite. +# +# GITHUB_REPOSITORY= VERSION= GH_TOKEN= \ +# .github/scripts/release/refuse-published-release.sh +# +# A draft is not returned by tag, so a 404 means there is no published +# release. Any other failure stops the script. A release the API returns is +# treated as published unless it says it is a draft. +# +# Used by the images job in .github/workflows/release.yaml. + +set -o errexit -o nounset -o pipefail + +errfile="$(mktemp)" +trap 'rm -f "${errfile}"' EXIT + +if draft="$(gh api "repos/${GITHUB_REPOSITORY}/releases/tags/${VERSION}" --jq .draft 2>"${errfile}")"; then + if [[ "${draft}" != true ]]; then + echo "::error::${VERSION} has a published release; its images are not published again" + exit 1 + fi +elif ! grep -q 'HTTP 404' "${errfile}"; then + echo "::error::checking for a published ${VERSION} release: $(<"${errfile}")" + exit 1 +fi diff --git a/.github/workflows/pr-workflow.yaml b/.github/workflows/pr-workflow.yaml index 81a95b20b4..3cff0baf96 100644 --- a/.github/workflows/pr-workflow.yaml +++ b/.github/workflows/pr-workflow.yaml @@ -51,10 +51,6 @@ jobs: uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 with: go-version-file: 'go.mod' - # Unconditional: a released migration becoming permanent is exactly what - # this guards, and that happens on the push to main, not on the PR. - - name: Verify immutable PostgreSQL migrations - run: hack/verify/postgresql-migrations.sh # we only test on amd64, make sure everything still builds for arm64 - name: build for arm64 run: GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build ./... @@ -145,6 +141,20 @@ jobs: if-no-files-found: error - name: verify run: hack/verify-all.sh + verify-migrations: + if: github.event_name == 'pull_request' || github.event_name == 'merge_group' + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - name: Checkout + uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0 + with: + fetch-depth: 0 + # A merge group has no base_ref. It exposes the base branch as refs/heads/. + - name: Verify immutable PostgreSQL migrations + env: + BASE_REF: ${{ github.base_ref || github.event.merge_group.base_ref }} + run: hack/verify/postgresql-migrations.sh "origin/${BASE_REF#refs/heads/}" # One kind cluster exercises BOTH runtimes. Free x86-64 ubuntu-latest runners # expose /dev/kvm (with a udev rule), so create-kind-cluster.sh mounts it and the # micro-VM (kata + cloud-hypervisor) sandbox class works alongside gVisor. diff --git a/.github/workflows/release-fork.yaml b/.github/workflows/release-fork.yaml new file mode 100644 index 0000000000..5c2709b40c --- /dev/null +++ b/.github/workflows/release-fork.yaml @@ -0,0 +1,159 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +name: release-fork + +on: + workflow_dispatch: + inputs: + tag: + description: 'Image tag (e.g. v1.2.3-rc1). Leave blank to auto-generate from branch+SHA.' + required: false + create_release: + description: 'Create a GitHub release' + type: boolean + default: false + +permissions: + contents: write + packages: write + +jobs: + release: + runs-on: ubuntu-latest + + steps: + - name: Checkout + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + + - name: Validate and resolve tag + id: tag + run: | + TAG="${{ inputs.tag }}" + if [[ -z "${TAG}" ]]; then + BRANCH="${GITHUB_REF_NAME//\//-}" + SHA="$(git rev-parse --short HEAD)" + TAG="${BRANCH}-${SHA}" + fi + if [[ "${{ inputs.create_release }}" == "true" ]]; then + if [[ ! "${TAG}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9._-]+)?$ ]]; then + echo "::error::Tag '${TAG}' must match vMAJOR.MINOR.PATCH[-prerelease] when creating a release (e.g. v1.2.3 or v1.2.3-rc1)" + exit 1 + fi + fi + echo "value=${TAG}" >> "$GITHUB_OUTPUT" + if [[ "${{ inputs.create_release }}" == "true" ]]; then + echo "tags=${TAG},latest" >> "$GITHUB_OUTPUT" + else + echo "tags=${TAG}" >> "$GITHUB_OUTPUT" + fi + + - name: Setup Go + uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff # v5.6.0 + with: + go-version-file: 'go.mod' + + - name: Install ko + uses: ko-build/setup-ko@3aebd0597dc1e9d1a26bcfdb7cbeb19c131d3037 # v0.7 + + - name: Install Helm + uses: azure/setup-helm@1a275c3b69536ee54be43f2070a358922e12c8d4 # v4.3.1 + + - name: Log in to GHCR + uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3.7.0 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} + + - name: Set up QEMU (multi-arch) + uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3.7.0 + + - name: Build and push images + env: + # ghcr.io// — resolves correctly in forks + IMAGE_REPOSITORY: ghcr.io/${{ github.repository }} + IMAGE_TAGS: ${{ steps.tag.outputs.tags }} + run: | + set -o errexit -o nounset -o pipefail + + for component in ateapi ate-idp-server atecontroller atelet ateworker-gvisor ateworker-microvm podcertcontroller atenet objectstore-plugin credential-provider/kubernetes-secrets; do + KO_DOCKER_REPO="${IMAGE_REPOSITORY}/${component##*/}" \ + ./hack/run-tool.sh ko build \ + --tags "${IMAGE_TAGS}" \ + --platform linux/amd64,linux/arm64 \ + --bare \ + "./cmd/${component}" + done + + - name: Package and push Helm charts + if: inputs.create_release + env: + HELM_EXPERIMENTAL_OCI: "1" + CHART_REPOSITORY: oci://ghcr.io/kagent-dev/substrate/helm + run: | + set -o errexit -o nounset -o pipefail + + tag="${{ steps.tag.outputs.value }}" + chart_version="${tag#v}" + package_dir="${RUNNER_TEMP}/helm-packages" + mkdir -p "${package_dir}" + + echo "${{ secrets.GITHUB_TOKEN }}" \ + | helm registry login ghcr.io \ + --username "${{ github.actor }}" \ + --password-stdin + + helm package charts/substrate-crds \ + --destination "${package_dir}" \ + --version "${chart_version}" \ + --app-version "${tag}" + helm package charts/substrate \ + --destination "${package_dir}" \ + --version "${chart_version}" \ + --app-version "${tag}" + helm package charts/substrate-podcert \ + --destination "${package_dir}" \ + --version "${chart_version}" \ + --app-version "${tag}" + + helm push "${package_dir}/substrate-crds-${chart_version}.tgz" "${CHART_REPOSITORY}" + helm push "${package_dir}/substrate-${chart_version}.tgz" "${CHART_REPOSITORY}" + helm push "${package_dir}/substrate-podcert-${chart_version}.tgz" "${CHART_REPOSITORY}" + + - name: Build kubectl-ate release binaries + if: inputs.create_release + env: + VERSION: ${{ steps.tag.outputs.value }} + run: | + set -o errexit -o nounset -o pipefail + + mkdir -p dist + for os in linux darwin; do + for arch in amd64 arm64; do + CGO_ENABLED=0 GOOS="${os}" GOARCH="${arch}" go build \ + -trimpath \ + -ldflags="-s -w -X=go.ate.dev/substrate/internal/version.Version=${VERSION}" \ + -o "dist/kubectl-ate-${os}-${arch}" \ + ./cmd/kubectl-ate + done + done + + - name: Create GitHub Release + if: inputs.create_release + uses: softprops/action-gh-release@3bb12739c298aeb8a4eeaf626c5b8d85266b0e65 # v2.6.2 + with: + tag_name: ${{ steps.tag.outputs.value }} + generate_release_notes: true + files: dist/kubectl-ate-* diff --git a/.github/workflows/release-notes.yaml b/.github/workflows/release-notes.yaml deleted file mode 100644 index 8434e51991..0000000000 --- a/.github/workflows/release-notes.yaml +++ /dev/null @@ -1,48 +0,0 @@ -# Copyright 2026 Google LLC -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Pushing a vX.Y.Z[-PRERELEASE] tag creates a DRAFT release with generated -# notes (.github/release.yml). A maintainer edits and publishes it; see -# docs/dev/release-notes.md. Run manually to draft a release for a tag that -# has none; an existing release is never modified. -name: release-notes -on: - push: - tags: ['v*'] - workflow_dispatch: - inputs: - tag: - description: 'Existing tag to draft a release for (e.g. v0.3.0)' - required: true -# Creating a release needs contents: write. -permissions: - contents: write -concurrency: - group: release-${{ inputs.tag || github.ref_name }} -jobs: - draft: - runs-on: ubuntu-latest - steps: - - name: Checkout - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0 - with: - # The committer list is read from local history back to the - # previous release tag. - fetch-depth: 0 - persist-credentials: false - - name: Draft release - env: - GH_TOKEN: ${{ github.token }} - TAG: ${{ inputs.tag || github.ref_name }} - run: hack/release/draft-release.sh "${TAG}" diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml index bcb2ab88b2..b1b622c544 100644 --- a/.github/workflows/release.yaml +++ b/.github/workflows/release.yaml @@ -12,148 +12,173 @@ # See the License for the specific language governing permissions and # limitations under the License. +# Pushing a vX.Y.Z[-PRERELEASE] tag publishes the release images and then +# creates a DRAFT release with generated notes (.github/release.yml), with the +# kubectl-ate binaries attached. A maintainer edits and publishes it; see +# docs/dev/releasing.md. +# +# Images are published only from agent-substrate/substrate, to +# ghcr.io/agent-substrate/substrate. In any other repository (a fork) the +# images job is skipped, because GitHub bills package storage to the fork's +# owner, and the draft is created as before. +# +# Run manually to publish images or draft a release for an existing tag. +# A complete image set is left unchanged unless overwrite is set; an incomplete +# one is rebuilt in full. Nothing is published for a tag whose release is +# published, and only a draft gets its binaries replaced. name: release - on: + push: + # Starts on tags that begin vMAJOR.MINOR.PATCH. The filter cannot express + # the whole release tag, so ate-setup and draft-release.sh reject anything + # that is not vMAJOR.MINOR.PATCH[-PRERELEASE] before publishing. + tags: ['v[0-9]+.[0-9]+.[0-9]+*'] workflow_dispatch: inputs: tag: - description: 'Image tag (e.g. v1.2.3-rc1). Leave blank to auto-generate from branch+SHA.' - required: false - create_release: - description: 'Create a GitHub release' + description: 'Existing tag to draft a release for (e.g. v0.3.0)' + required: true + overwrite: + description: 'Rebuild and replace images that already exist for the tag' type: boolean default: false - + dry_run_images: + description: 'Build the images into a registry inside the job and discard them; no images published, no draft' + type: boolean + default: false +# Each job requests the permissions it needs. permissions: - contents: write - packages: write - + contents: read +concurrency: + group: release-${{ inputs.tag || github.ref_name }} jobs: - release: + # The only job that writes to a registry. + images: + if: github.repository == 'agent-substrate/substrate' && !inputs.dry_run_images runs-on: ubuntu-latest - + timeout-minutes: 30 + permissions: + contents: read + packages: write + # Read by ate-setup publish release-images. + env: + VERSION: ${{ inputs.tag || github.ref_name }} + KO_DOCKER_REPO: ghcr.io/agent-substrate/substrate + KO_DEFAULTPLATFORMS: linux/amd64,linux/arm64 + # inputs.overwrite is empty on a tag push; this makes the value explicit. + ATE_PUBLISH_OVERWRITE: ${{ inputs.overwrite == true }} steps: - - name: Checkout - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 - - - name: Validate and resolve tag - id: tag - run: | - TAG="${{ inputs.tag }}" - if [[ -z "${TAG}" ]]; then - BRANCH="${GITHUB_REF_NAME//\//-}" - SHA="$(git rev-parse --short HEAD)" - TAG="${BRANCH}-${SHA}" - fi - if [[ "${{ inputs.create_release }}" == "true" ]]; then - if [[ ! "${TAG}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9._-]+)?$ ]]; then - echo "::error::Tag '${TAG}' must match vMAJOR.MINOR.PATCH[-prerelease] when creating a release (e.g. v1.2.3 or v1.2.3-rc1)" - exit 1 - fi - fi - echo "value=${TAG}" >> "$GITHUB_OUTPUT" - if [[ "${{ inputs.create_release }}" == "true" ]]; then - echo "tags=${TAG},latest" >> "$GITHUB_OUTPUT" - else - echo "tags=${TAG}" >> "$GITHUB_OUTPUT" - fi - - - name: Setup Go - uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff # v5.6.0 - with: - go-version-file: 'go.mod' - - - name: Install ko - uses: ko-build/setup-ko@3aebd0597dc1e9d1a26bcfdb7cbeb19c131d3037 # v0.7 - - - name: Install Helm - uses: azure/setup-helm@1a275c3b69536ee54be43f2070a358922e12c8d4 # v4.3.1 - - - name: Log in to GHCR - uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3.7.0 - with: - registry: ghcr.io - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} - - - name: Set up QEMU (multi-arch) - uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3.7.0 - - - name: Build and push images - env: - # ghcr.io// — resolves correctly in forks - IMAGE_REPOSITORY: ghcr.io/${{ github.repository }} - IMAGE_TAGS: ${{ steps.tag.outputs.tags }} - run: | - set -o errexit -o nounset -o pipefail - - for component in ateapi ate-idp-server atecontroller atelet ateworker-gvisor ateworker-microvm podcertcontroller atenet objectstore-plugin credential-provider/kubernetes-secrets; do - KO_DOCKER_REPO="${IMAGE_REPOSITORY}/${component##*/}" \ - ./hack/run-tool.sh ko build \ - --tags "${IMAGE_TAGS}" \ - --platform linux/amd64,linux/arm64 \ - --bare \ - "./cmd/${component}" - done - - - name: Package and push Helm charts - if: inputs.create_release - env: - HELM_EXPERIMENTAL_OCI: "1" - CHART_REPOSITORY: oci://ghcr.io/kagent-dev/substrate/helm - run: | - set -o errexit -o nounset -o pipefail - - tag="${{ steps.tag.outputs.value }}" - chart_version="${tag#v}" - package_dir="${RUNNER_TEMP}/helm-packages" - mkdir -p "${package_dir}" - - echo "${{ secrets.GITHUB_TOKEN }}" \ - | helm registry login ghcr.io \ - --username "${{ github.actor }}" \ - --password-stdin - - helm package charts/substrate-crds \ - --destination "${package_dir}" \ - --version "${chart_version}" \ - --app-version "${tag}" - helm package charts/substrate \ - --destination "${package_dir}" \ - --version "${chart_version}" \ - --app-version "${tag}" - helm package charts/substrate-podcert \ - --destination "${package_dir}" \ - --version "${chart_version}" \ - --app-version "${tag}" - - helm push "${package_dir}/substrate-crds-${chart_version}.tgz" "${CHART_REPOSITORY}" - helm push "${package_dir}/substrate-${chart_version}.tgz" "${CHART_REPOSITORY}" - helm push "${package_dir}/substrate-podcert-${chart_version}.tgz" "${CHART_REPOSITORY}" - - - name: Build kubectl-ate release binaries - if: inputs.create_release - env: - VERSION: ${{ steps.tag.outputs.value }} - run: | - set -o errexit -o nounset -o pipefail - - mkdir -p dist - for os in linux darwin; do - for arch in amd64 arm64; do - CGO_ENABLED=0 GOOS="${os}" GOARCH="${arch}" go build \ - -trimpath \ - -ldflags="-s -w -X=go.ate.dev/substrate/internal/version.Version=${VERSION}" \ - -o "dist/kubectl-ate-${os}-${arch}" \ - ./cmd/kubectl-ate - done - done - - - name: Create GitHub Release - if: inputs.create_release - uses: softprops/action-gh-release@3bb12739c298aeb8a4eeaf626c5b8d85266b0e65 # v2.6.2 - with: - tag_name: ${{ steps.tag.outputs.value }} - generate_release_notes: true - files: dist/kubectl-ate-* + - name: Checkout + uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0 + with: + ref: refs/tags/${{ inputs.tag || github.ref_name }} + persist-credentials: false + - name: Setup Go + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + go-version-file: 'go.mod' + # These jobs can write packages or releases, so they restore no cache + # that another workflow could have written. + cache: false + # Every release image is built with ko, which cross-compiles Go for arm64 + # without emulation, so the job needs neither QEMU nor buildx. ko and + # ate-setup's registry checks read the credentials docker login stores. + - name: Log in to ghcr.io + env: + TOKEN: ${{ github.token }} + run: .github/scripts/release/ghcr-login.sh + # Publishing replaces the digests that operators of a published release + # install by tag, so nothing is published once the release is. This runs + # whether or not overwrite is set: without it, an incomplete set (an image + # deleted after release) is still rebuilt in full. + - name: Refuse to publish images for a published release + env: + GH_TOKEN: ${{ github.token }} + run: .github/scripts/release/refuse-published-release.sh + # Builds nothing if every image already exists for the tag, unless overwrite + # is set, and after pushing checks that every tag names the pushed image on + # every platform. + - name: Publish images + run: go run ./cmd/ate-setup publish release-images + # The same build and checks against a registry that exists only for the + # duration of this job. Runs only when started manually with dry_run_images. + images-dry-run: + if: inputs.dry_run_images + runs-on: ubuntu-latest + timeout-minutes: 30 + services: + registry: + # The index registry:3 named on 2026-10-19, pinned like the actions above. + image: registry:3@sha256:ddf754342cfc8acc51a56d5d0ab6af06826461864460636d8bd5c546dab2a7b8 + ports: ['5000:5000'] + env: + VERSION: ${{ inputs.tag }} + KO_DOCKER_REPO: localhost:5000/substrate + KO_DEFAULTPLATFORMS: linux/amd64,linux/arm64 + ATE_PUBLISH_OVERWRITE: ${{ inputs.overwrite == true }} + steps: + - name: Checkout + uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0 + with: + ref: refs/tags/${{ inputs.tag }} + persist-credentials: false + - name: Setup Go + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + go-version-file: 'go.mod' + # These jobs can write packages or releases, so they restore no cache + # that another workflow could have written. + cache: false + - name: Publish images + run: go run ./cmd/ate-setup publish release-images + draft: + needs: images + # Runs after images succeeded, or when images was skipped (any repository + # other than agent-substrate/substrate). If images failed, no draft is + # created, so a release cannot be published without its images. Skipped + # in a dry run. + if: ${{ !cancelled() && !inputs.dry_run_images && (needs.images.result == 'success' || needs.images.result == 'skipped') }} + runs-on: ubuntu-latest + timeout-minutes: 30 + # Creating a release needs contents: write. + permissions: + contents: write + steps: + - name: Checkout + uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0 + with: + # Check out the tag being released, as the images job does. A manual + # run otherwise checks out the branch it was started on. + ref: refs/tags/${{ inputs.tag || github.ref_name }} + # The committer list is read from local history back to the + # previous release tag. + fetch-depth: 0 + persist-credentials: false + - name: Setup Go + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + go-version-file: 'go.mod' + # These jobs can write packages or releases, so they restore no cache + # that another workflow could have written. + cache: false + # Built before the draft exists, so a build failure leaves no draft, as an + # images failure does. A fork attaches no binaries, so it builds none. + - name: Build kubectl-ate + if: github.repository == 'agent-substrate/substrate' + env: + VERSION: ${{ inputs.tag || github.ref_name }} + run: go run ./cmd/ate-setup publish release-binaries --out bin/release + - name: Draft release + env: + GH_TOKEN: ${{ github.token }} + TAG: ${{ inputs.tag || github.ref_name }} + run: hack/release/draft-release.sh "${TAG}" + # Only agent-substrate/substrate attaches binaries; a fork's draft gets + # none. Assets are replaced only while the release is a draft, so a rerun + # never changes the files of a published release. + - name: Attach kubectl-ate + if: github.repository == 'agent-substrate/substrate' + env: + GH_TOKEN: ${{ github.token }} + TAG: ${{ inputs.tag || github.ref_name }} + run: .github/scripts/release/attach-binaries.sh bin/release diff --git a/.ko.yaml b/.ko.yaml index a16adb0c23..80d54c7973 100644 --- a/.ko.yaml +++ b/.ko.yaml @@ -12,14 +12,14 @@ # See the License for the specific language governing permissions and # limitations under the License. -defaultBaseImage: gcr.io/distroless/static-debian13:latest@sha256:f2ea2709ac8db56323cbd7d014277f32cb572d9ea124b0076f7aafe5980678fe +defaultBaseImage: gcr.io/distroless/static-debian13:nonroot@sha256:18f4cc26c9ffa7f20a359298be7deadac5c607c8e03a02b8119f1a5fc2f82c99 defaultPlatforms: - linux/amd64 - linux/arm64 baseImageOverrides: - go.ate.dev/substrate/demos/sandbox: alpine:3.24.2@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6 + go.ate.dev/substrate/demos/sandbox: alpine:3.24.3@sha256:be2656061b536c1883c4034790e21d0a09c5aaaf51968b89368a3c0371ea2c9a # atelet needs glibc to run the node's kubelet credential provider, which # some platforms ship dynamically linked (e.g. AKS, GKE Ubuntu). - go.ate.dev/substrate/cmd/atelet: gcr.io/distroless/base-debian13:latest@sha256:389cad21f73e4c37b94ffe5b13736d5a92bd5bd3c6c6b38c2be1c881e14ba2bd + go.ate.dev/substrate/cmd/atelet: gcr.io/distroless/base-debian13:latest@sha256:f5e95195e998742ff590b7569bae479accce1226e51ae6c3118954030121cdcb diff --git a/AGENTS.md b/AGENTS.md index 6494966a1c..75833863d9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,6 +12,7 @@ - Prefer verbs to abstract nouns: "evicts" instead of "performs eviction". - Say each thing once. Don't restate the diff, open with "This PR …" or "This commit …", or repeat the same explanation in several comments. - Explain why only when the code doesn't make it obvious, in a clause or a line. + - Write `microVM` (`MicroVM` at the start of a sentence) without a hyphen in docs, comments, and commits. Use `microvm` for API strings and flags. - Avoid: - em-dashes, and chains of semicolons - headings, bold, or test plans in commit messages @@ -44,7 +45,7 @@ - Check every claim in comments, docs and the PR description against the code. Don't claim a guarantee the code doesn't enforce. When you change behavior, update the comments, docs and scripts elsewhere that describe it. Link to existing docs instead of copying them. - Find the root cause before fixing a symptom. Back performance claims with a measurement. - Keep atelet and the worker runtimes opaque to each other. The only knowledge they may share is `internal/proto/ateworkerpb` and the paths in `internal/nodepath`. Using the same library code is fine; depending on how the other side is built is not. We want atelet to roll ahead of workers eventually, so don't widen that interface casually. Until v1.0.0 nothing enforces this; see Backward Compatibility. -- Actor features work on both gVisor and micro-VM, with e2e coverage for both, unless the issue or PR description scopes the change to one runtime. +- Actor features work on both gVisor and microVM, with e2e coverage for both, unless the issue or PR description scopes the change to one runtime. - Don't assume kind, GKE, amd64 or a newer kernel than the repo already needs. Dev scripts must work on arm64 macOS. Don't change host-wide state. - Put shell code in standalone `.sh` scripts so shellcheck can lint them. Place each script near what uses it, such as `hack/` for dev scripts or a directory under `.github/` for a workflow. Don't inline it in YAML, such as a GitHub workflow `run:` block, in JSON, or in another language's source, such as Go or Python. Have those call the script. Get a reviewer's agreement before making an exception. - Read the `CI` environment variable only inside a named fail-closed predicate, never inline. See Testing Instructions 5. @@ -104,7 +105,8 @@ The dev entry points are the scripts under `hack/` and the `cmd/ate-setup` insta ### Building and Installing - **Install from source**: `go run ./cmd/ate-setup deploy ate-system` (or the `hack/install-ate.sh` shim) builds every image with ko as part of the install; see `cmd/ate-setup/README.md` - **Binaries**: `go build -o bin/kubectl-ate ./cmd/kubectl-ate`, `go build -o bin/ate-setup ./cmd/ate-setup` -- **Release images**: `VERSION=TAG go run ./cmd/ate-setup publish release-images --ko-docker-repo REPO` (every image a pre-built install needs, all tagged `TAG`, including the docker-built `envoy-dataplane`) +- **Release images**: `VERSION=TAG go run ./cmd/ate-setup publish release-images --ko-docker-repo REPO` (the release set, `images.ReleaseComponents`, all tagged `TAG`; see `docs/dev/releasing.md`) +- **Release binaries**: `VERSION=TAG go run ./cmd/ate-setup publish release-binaries --out DIR` (`kubectl-ate` for linux and darwin, amd64 and arm64, with `.sha256` files) ### Testing and Verification - **Run Unit Tests**: `go test -race ./...` diff --git a/README.md b/README.md index dc686a2640..22ec50d785 100644 --- a/README.md +++ b/README.md @@ -272,7 +272,7 @@ We provide several sample applications demonstrating Agent Substrate's capabilit * `cmd/ate-idp-server`: Serves the OpenID discovery document and key set that relying parties use to verify actor JWTs. * `cmd/atenet`: A combined networking controller providing Envoy routing and proxy sidecars. * `cmd/ateworker-gvisor`: An interior-pod helper running inside sandboxed worker pods to execute `runsc` checkpoint and restore commands. -* `cmd/ateworker-microvm`: The micro-VM peer of `ateworker-gvisor`, running actors as cloud-hypervisor VMs. +* `cmd/ateworker-microvm`: The microVM peer of `ateworker-gvisor`, running actors as cloud-hypervisor VMs. * `cmd/podcertcontroller`: A "polyfill" that provides Pod Certificate signers that will eventually ship in upstream Kubernetes (with different names). * `cmd/kubectl-ate`: A CLI tool for managing Agent Substrate resources. See its [README](cmd/kubectl-ate/README.md). diff --git a/api/infra/k8s/ate/v1/workerpool_types.go b/api/infra/k8s/ate/v1/workerpool_types.go index f34ebb7091..a0af976cfa 100644 --- a/api/infra/k8s/ate/v1/workerpool_types.go +++ b/api/infra/k8s/ate/v1/workerpool_types.go @@ -263,6 +263,7 @@ const ( // +kubebuilder:printcolumn:name="SandboxConfig",type=string,JSONPath=`.status.sandboxClasses[*].configRef.name` // +kubebuilder:printcolumn:name="Resolved",type=string,JSONPath=`.status.sandboxClasses[*].conditions[?(@.type=="Resolved")].status` // +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp` +// +kubebuilder:validation:XValidation:rule="size(self.metadata.name) <= 63",message="metadata.name must be at most 63 characters, because worker pods use it as a label value" type WorkerPool struct { metav1.TypeMeta `json:",inline"` diff --git a/api/infra/k8s/ate/v1/workerpool_validation_test.go b/api/infra/k8s/ate/v1/workerpool_validation_test.go index ce2712268a..3da16c9d40 100644 --- a/api/infra/k8s/ate/v1/workerpool_validation_test.go +++ b/api/infra/k8s/ate/v1/workerpool_validation_test.go @@ -57,6 +57,19 @@ func TestWorkerPoolValidation(t *testing.T) { name: "base worker pool", mutate: func(wp *WorkerPool) {}, wantErr: false, + }, { + name: "name of 63 characters", + mutate: func(wp *WorkerPool) { + wp.Name = strings.Repeat("a", 63) + }, + wantErr: false, + }, { + name: "name of 64 characters", + mutate: func(wp *WorkerPool) { + wp.Name = strings.Repeat("a", 64) + }, + wantErr: true, + errMsg: "metadata.name must be at most 63 characters", }, { name: "replicas below minimum", mutate: func(wp *WorkerPool) { diff --git a/benchmarking/README.md b/benchmarking/README.md index 1455a9653f..c584d1aad3 100644 --- a/benchmarking/README.md +++ b/benchmarking/README.md @@ -466,7 +466,7 @@ sandbox, so each of those turns needs an external wake. `personal-assistant-light` is the same day with the RAM arrays scaled down to 28Mi and everything else unchanged (files, reads, CPU, dwell and the 61 wakes), for runs that want the always-on wake pattern with a small -memory image: about 40 to 95 MB per pause under gVisor, and under micro-VM +memory image: about 40 to 95 MB per pause under gVisor, and under microVM about 100 MB of guest before the script writes anything, bounded by the actor's memory limit. It fits a 256Mi actor. Run it with the same fractional think scale. diff --git a/benchmarking/analysis/README.md b/benchmarking/analysis/README.md index b2a6735bec..23bff10085 100644 --- a/benchmarking/analysis/README.md +++ b/benchmarking/analysis/README.md @@ -74,14 +74,14 @@ A failed atelet operation still writes its record, marked with `error.type` **Phase percentiles.** Per layer, operation, sandbox class, snapshot kind, scope and phase: count, p50, p90, p95 and max. A `local` restore reads a node-local snapshot and a `latest` one a durable snapshot from object -storage, so they are separate rows, as are gVisor and micro-VM operations. +storage, so they are separate rows, as are gVisor and microVM operations. A restore from a template's golden snapshot reports `latest`, like the actor's own durable snapshot. The atelet rows split a checkpoint between `sandbox_assets`, `worker_checkpoint`, `persist` and `cleanup`, and a restore between `volume_mount`, `manifest_fetch`, `sandbox_assets`, `download`, `oci_unpack` and `worker_restore`. The ateworker rows split the -`worker_*` phase further with the runtime's own phases: for micro-VM +`worker_*` phase further with the runtime's own phases: for microVM `prep` / `pause` / `snapshot` / `durable_dir` / `rootfs_upper` / `teardown` on a checkpoint and `prep` / … / `vm_restore` / `wakeup_probe` on a restore; for gVisor `pause` / `checkpoint` / `fs_checkpoint` / `resume` / `teardown` and diff --git a/benchmarking/automation/README.md b/benchmarking/automation/README.md index 28515298a4..7f36ddf5be 100644 --- a/benchmarking/automation/README.md +++ b/benchmarking/automation/README.md @@ -47,7 +47,7 @@ the router capacity benchmark — see variable (a JSON object) if set. A `--metadata` in the test's own `flags` is merged in too, its keys winning on conflict. - Polls until complete/failed/timeout; tails logs; deletes the Job. - - Tears down workloads + micro-VM deps (if any) + substrate. + - Tears down workloads + microVM deps (if any) + substrate. - If not the last test, redeploys them so the next run starts clean. ## Choosing a sandbox class @@ -57,7 +57,7 @@ Each entry in `tests.yaml` may set `sandboxClass: gvisor | microvm` (default and its `workerImage` (`ateworker-gvisor` vs `ateworker-microvm`). For `microvm` tests the target cluster must have KVM-capable nodes. The -orchestrator uploads the micro-VM assets, so its Workload Identity principal +orchestrator uploads the microVM assets, so its Workload Identity principal must be able to write to the bucket they go to. On GKE that is `-assets`, where `BUCKET_NAME` comes from `.ate-dev-env.sh`. diff --git a/benchmarking/locust/tests/counter_demo.py b/benchmarking/locust/tests/counter_demo.py index afeed39a70..b8bd622f3e 100644 --- a/benchmarking/locust/tests/counter_demo.py +++ b/benchmarking/locust/tests/counter_demo.py @@ -25,7 +25,7 @@ import requests from common import ateapi_pb2 from common import ateapi_pb2_grpc -from common.ateapi_channel import ateapi_channel +from common.ateapi_channel import CA_FILE, CRED_BUNDLE, ateapi_channel from common.atespace import ATESPACE, ensure_atespace from common.grpc_tracing import traced_grpc @@ -50,7 +50,7 @@ # Atenet router fronts all actor traffic. The actor routing header selects # the actor whose current worker pod the router resolves. -ROUTER_URL = "http://atenet-router.ate-system.svc.cluster.local" +ROUTER_URL = "https://atenet-router.ate-system.svc" class CounterUser(User): @@ -97,8 +97,10 @@ def on_start(self) -> None: except Exception as e: logger.error(f"Failed to create actor {self.actor_name}: {e}") - # One HTTP session per user, talking to the router. + # One HTTP session per user, talking to the router over mTLS. self.http_session = requests.Session() + self.http_session.verify = CA_FILE + self.http_session.cert = CRED_BUNDLE self.run_url = f"{ROUTER_URL}/" def on_stop(self) -> None: diff --git a/benchmarking/nighthawk-egress/README.md b/benchmarking/nighthawk-egress/README.md index 24c3fcc2ac..df62df9284 100644 --- a/benchmarking/nighthawk-egress/README.md +++ b/benchmarking/nighthawk-egress/README.md @@ -8,7 +8,7 @@ Benchmarks the Agent Substrate egress dataplane (`atunnel` -> `atenet-egress`) u flowchart LR client["nighthawk_client
(host)"] invoker["traffic-driver-invoker
(:19980, adds ate-target-actor)"] - router["atenet-router
(:80)"] + router["atenet-router
(:443)"] subgraph worker["WorkerPod (ateworker-gvisor)"] atunnel-ingress["ingress atunnel"] diff --git a/benchmarking/nighthawk-egress/traffic-driver-invoker.yaml b/benchmarking/nighthawk-egress/traffic-driver-invoker.yaml index 4b37bc243d..194511e04d 100644 --- a/benchmarking/nighthawk-egress/traffic-driver-invoker.yaml +++ b/benchmarking/nighthawk-egress/traffic-driver-invoker.yaml @@ -157,7 +157,7 @@ spec: containers: - name: envoy # Same envoyproxy/envoy digest as cmd/dataplane/envoy/Dockerfile. - image: envoyproxy/envoy:v1.39.4@sha256:74bd4eb5629708d3fadca1a5e6b85d208f338d7bb54e06b69565905c5294cc01 + image: envoyproxy/envoy:v1.39.5@sha256:8fc594a5d85eb76dca24155cfb9208882406873932b20cf33649a0a611194c43 command: - /usr/local/bin/envoy args: diff --git a/benchmarking/nighthawk-ingress/README.md b/benchmarking/nighthawk-ingress/README.md index d5a94ac329..1dd1d9bb14 100644 --- a/benchmarking/nighthawk-ingress/README.md +++ b/benchmarking/nighthawk-ingress/README.md @@ -42,7 +42,7 @@ flowchart LR end subgraph pod["atenet-router pod — pinned to envoyCpu CPUs"] - envoy["envoy :8080
--concurrency envoyCpu"] + envoy["envoy :8443
--concurrency envoyCpu"] extproc["atenet-router sidecar
ext_proc routing decision"] envoy -->|request headers| extproc extproc -->|x-ate-original-dst| envoy @@ -54,7 +54,7 @@ flowchart LR atunnel["atunnel :443"] --> glutton["glutton actor
POST /ping :80"] end - svc -->|"HTTP :80
ate-target-actor"| envoy + svc -->|"mTLS :443
ate-target-actor"| envoy extproc -->|ResumeActor| ateapi envoy -->|"mTLS :443"| atunnel ``` diff --git a/benchmarking/nighthawk-ingress/actors.py b/benchmarking/nighthawk-ingress/actors.py index b9b6d681c2..bcebb1aa1c 100644 --- a/benchmarking/nighthawk-ingress/actors.py +++ b/benchmarking/nighthawk-ingress/actors.py @@ -28,7 +28,7 @@ from common import ateapi_pb2 from common import ateapi_pb2_grpc -from common.ateapi_channel import CA_FILE, SERVER_NAME, ateapi_channel +from common.ateapi_channel import CA_FILE, CRED_BUNDLE, SERVER_NAME, ateapi_channel import spec as spec_mod @@ -115,6 +115,8 @@ def _warm_actor( returns FailedPrecondition/Unavailable until a worker frees up.""" ref = ateapi_pb2.ObjectRef(atespace=atespace, name=name) session = requests.Session() + session.verify = CA_FILE + session.cert = CRED_BUNDLE last_err: str = "not attempted" while time.time() < deadline: try: diff --git a/benchmarking/nighthawk-ingress/runner.py b/benchmarking/nighthawk-ingress/runner.py index 921673f8fc..cf90716eef 100644 --- a/benchmarking/nighthawk-ingress/runner.py +++ b/benchmarking/nighthawk-ingress/runner.py @@ -48,7 +48,7 @@ ADAPTIVE_CLIENT = "nighthawk_adaptive_load_client" SERVICE_ADDRESS = "127.0.0.1:8443" -DEFAULT_ROUTER_URL = "http://atenet-router.ate-system.svc.cluster.local:80" +DEFAULT_ROUTER_URL = "https://atenet-router.ate-system.svc:443" DESC_PATH = os.environ.get("NIGHTHAWK_DESC", "/app/nighthawk.desc") diff --git a/benchmarking/nighthawk-ingress/spec.py b/benchmarking/nighthawk-ingress/spec.py index debd4308a4..acecb0e21d 100644 --- a/benchmarking/nighthawk-ingress/spec.py +++ b/benchmarking/nighthawk-ingress/spec.py @@ -38,6 +38,10 @@ "type.googleapis.com/nighthawk.request_source.InLineOptionsListRequestSourceConfig" ) +ROUTER_SNI = "atenet-router.ate-system.svc" +CA_FILE = "/run/servicedns-ca/ca.crt" +CRED_BUNDLE = "/run/podidentity.podcert.ate.dev/credential-bundle.pem" + # The default failure predicates (all 0) abort an execution on the first # 4xx/5xx; measuring periods must instead run to completion so overload # scores as degraded success-rate. All five defaults overridden. @@ -143,26 +147,50 @@ def build_spec_dict( upper_threshold=tail_latency_slo_ms * 1_000_000.0, ) ) - return { - "nighthawk_traffic_template": { - "uri": uri, - "concurrency": str(client_concurrency), - "connections": connections, - "max_pending_requests": max_pending_requests, - "open_loop": True, - "failure_predicates": dict(PERMISSIVE_FAILURE_PREDICATES), - # request_options shares a oneof with this field: method and - # headers must come from the per-variant RequestOptions. - "request_source_plugin_config": { - "name": REQUEST_SOURCE_PLUGIN, - "typed_config": { - "@type": REQUEST_SOURCE_TYPE_URL, - "options_list": {"options": request_variants}, - # 0 = cycle through options_list indefinitely. - "num_requests": 0, + if not uri.startswith("https://"): + raise ValueError(f"router URI must use https://, got {uri!r}") + + traffic_template: dict = { + "uri": uri, + "concurrency": str(client_concurrency), + "connections": connections, + "max_pending_requests": max_pending_requests, + "open_loop": True, + "failure_predicates": dict(PERMISSIVE_FAILURE_PREDICATES), + # request_options shares a oneof with this field: method and + # headers must come from the per-variant RequestOptions. + "request_source_plugin_config": { + "name": REQUEST_SOURCE_PLUGIN, + "typed_config": { + "@type": REQUEST_SOURCE_TYPE_URL, + "options_list": {"options": request_variants}, + # 0 = cycle through options_list indefinitely. + "num_requests": 0, + }, + }, + "tls_context": { + "sni": ROUTER_SNI, + "common_tls_context": { + "tls_certificates": [ + { + "certificate_chain": {"filename": CRED_BUNDLE}, + "private_key": {"filename": CRED_BUNDLE}, + }, + ], + "validation_context": { + "trusted_ca": {"filename": CA_FILE}, + "match_typed_subject_alt_names": [ + { + "san_type": "DNS", + "matcher": {"exact": ROUTER_SNI}, + }, + ], }, }, }, + } + return { + "nighthawk_traffic_template": traffic_template, "metric_thresholds": thresholds, "informational_metric_specs": [ _metric_spec(m) for m in INFORMATIONAL_METRICS diff --git a/benchmarking/nighthawk-ingress/tests/test_spec.py b/benchmarking/nighthawk-ingress/tests/test_spec.py index cf45513809..d78c550cdc 100644 --- a/benchmarking/nighthawk-ingress/tests/test_spec.py +++ b/benchmarking/nighthawk-ingress/tests/test_spec.py @@ -32,7 +32,7 @@ def build(**overrides): kwargs = dict( - uri="http://atenet-router.ate-system.svc.cluster.local:80/ping", + uri="https://atenet-router.ate-system.svc:443/ping", actor_names=[f"sb-{i}" for i in range(3)], atespace="benchmark", client_concurrency=4, @@ -64,6 +64,39 @@ def test_traffic_template_shape(): "benchmark.stream_resets", "requestsource.upstream_rq_5xx", } + assert template["tls_context"] == { + "sni": "atenet-router.ate-system.svc", + "common_tls_context": { + "tls_certificates": [ + { + "certificate_chain": { + "filename": "/run/podidentity.podcert.ate.dev/credential-bundle.pem" + }, + "private_key": { + "filename": "/run/podidentity.podcert.ate.dev/credential-bundle.pem" + }, + }, + ], + "validation_context": { + "trusted_ca": {"filename": "/run/servicedns-ca/ca.crt"}, + "match_typed_subject_alt_names": [ + { + "san_type": "DNS", + "matcher": {"exact": "atenet-router.ate-system.svc"}, + }, + ], + }, + }, + } + + +def test_rejects_non_https_uri(): + try: + build(uri="http://atenet-router.ate-system.svc:80/ping") + except ValueError: + pass + else: + raise AssertionError("expected ValueError for plaintext http:// URI") def test_actor_reference_rotation_covers_all_actors(): diff --git a/charts/substrate-crds/templates/ate.dev_workerpools.yaml b/charts/substrate-crds/templates/ate.dev_workerpools.yaml index d964ac6f1f..f16043ebb4 100644 --- a/charts/substrate-crds/templates/ate.dev_workerpools.yaml +++ b/charts/substrate-crds/templates/ate.dev_workerpools.yaml @@ -624,6 +624,10 @@ spec: required: - spec type: object + x-kubernetes-validations: + - message: metadata.name must be at most 63 characters, because worker pods + use it as a label value + rule: size(self.metadata.name) <= 63 served: true storage: true subresources: diff --git a/charts/substrate-podcert/templates/pod-certificate-controller.yaml b/charts/substrate-podcert/templates/pod-certificate-controller.yaml index acf802d53c..80284bb991 100644 --- a/charts/substrate-podcert/templates/pod-certificate-controller.yaml +++ b/charts/substrate-podcert/templates/pod-certificate-controller.yaml @@ -191,6 +191,7 @@ spec: mountPath: "/run/ca-state" securityContext: allowPrivilegeEscalation: false + runAsNonRoot: true capabilities: add: - NET_BIND_SERVICE diff --git a/charts/substrate/README.md b/charts/substrate/README.md index 4ae503fd25..5b955ccb98 100644 --- a/charts/substrate/README.md +++ b/charts/substrate/README.md @@ -140,10 +140,19 @@ snapshot bucket, and let the API server's identity impersonate it. See [the GCP setup guide](../../tools/setup-gcp/README.md#what-create-iam-actually-grants) for the required IAM grants. S3 and RustFS do not need this setting. -The default gVisor config now uses the 2026-10-17 nightly (`v4`) for separate +The default gVisor config now uses release `20261019.0` (`v5`) for separate container-rootfs and durable-directory checkpoint bundles. Upgrade the sandbox config together with ateworker-gvisor; custom configs must pin a release with the same checkpoint support and supply the `gvisor` release tarball asset. Bare `runsc` assets are no longer accepted. +This replaces the previous `v4` config. Existing `v4` memory snapshots need a +compatible runtime; falling back to a durable filesystem snapshot does not +restore the actor's memory. + +When upgrading an installation with RustFS enabled, delete its completed bucket +initializer Job before running `helm upgrade`. The default Job name is +`rustfs-bucket-init` in the release namespace. This update changes the AWS CLI +image, and Kubernetes cannot patch an existing Job's pod template. Helm +recreates the Job, which keeps the existing bucket and its contents. The chart now denies ingress to the control-plane namespace by default and allows the component, router, egress, storage, and monitoring traffic it needs. @@ -161,6 +170,10 @@ clients and components together. ## PostgreSQL setup +Upstream squashed the migrations through v0.6.0 into `000001_initial.sql`. +Databases created before that squash cannot be upgraded in place and must be +recreated. See [PostgreSQL schema evolution](../../docs/dev/postgresql-schema-evolution.md). + The chart does not deploy or initialize PostgreSQL. Prepare the database, schema, login users, and the configured `readWriteRole` and `ownerRole` before installing the chart. For development Helm installations in `ate-system`, diff --git a/charts/substrate/templates/ate-api-server.yaml b/charts/substrate/templates/ate-api-server.yaml index d76ccf8301..8089720d3e 100644 --- a/charts/substrate/templates/ate-api-server.yaml +++ b/charts/substrate/templates/ate-api-server.yaml @@ -108,6 +108,8 @@ spec: periodSeconds: 2 failureThreshold: 30 securityContext: + allowPrivilegeEscalation: false + runAsNonRoot: true capabilities: drop: [ALL] resources: @@ -139,6 +141,12 @@ spec: - { name: actor-id-jwt-jwks, mountPath: /run/actor-id-jwt-jwks, readOnly: true } containers: - name: ate-api-server + securityContext: + allowPrivilegeEscalation: false + runAsNonRoot: true + capabilities: + add: [NET_BIND_SERVICE] + drop: [ALL] image: {{ include "substrate.componentImage" (list "ateapi" .) }} resources: {{ toYaml .Values.ateApi.resources | indent 10 }} diff --git a/charts/substrate/templates/ate-controller.yaml b/charts/substrate/templates/ate-controller.yaml index 1470f4c7c1..7ac38e2409 100644 --- a/charts/substrate/templates/ate-controller.yaml +++ b/charts/substrate/templates/ate-controller.yaml @@ -84,6 +84,12 @@ spec: {{- with include "substrate.imagePullSecrets" . }}{{- . | nindent 6 }}{{- end }} containers: - name: ate-controller + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + capabilities: + drop: [ALL] image: {{ include "substrate.componentImage" (list "atecontroller" .) }} resources: {{ toYaml .Values.ateController.resources | indent 10 }} diff --git a/charts/substrate/templates/ate-idp-server.yaml b/charts/substrate/templates/ate-idp-server.yaml index 67f7c16741..62f09032c3 100644 --- a/charts/substrate/templates/ate-idp-server.yaml +++ b/charts/substrate/templates/ate-idp-server.yaml @@ -67,6 +67,8 @@ spec: scheme: HTTPS periodSeconds: 10 securityContext: + allowPrivilegeEscalation: false + runAsNonRoot: true capabilities: drop: - ALL diff --git a/charts/substrate/templates/atelet.yaml b/charts/substrate/templates/atelet.yaml index 42d0ed60e3..9207c69ac9 100644 --- a/charts/substrate/templates/atelet.yaml +++ b/charts/substrate/templates/atelet.yaml @@ -159,6 +159,7 @@ spec: periodSeconds: 2 failureThreshold: 30 securityContext: + allowPrivilegeEscalation: false runAsUser: 0 runAsGroup: 0 capabilities: @@ -203,6 +204,7 @@ spec: {{ toYaml . | indent 8 }} {{- end }} securityContext: + allowPrivilegeEscalation: false runAsUser: 0 runAsGroup: 0 capabilities: diff --git a/charts/substrate/templates/sandboxconfig-gvisor.yaml b/charts/substrate/templates/sandboxconfig-gvisor.yaml index 911edf0c3d..03f8cdab7d 100644 --- a/charts/substrate/templates/sandboxconfig-gvisor.yaml +++ b/charts/substrate/templates/sandboxconfig-gvisor.yaml @@ -33,21 +33,21 @@ metadata: sandboxconfig.ate.dev/is-class-default: "true" spec: sandboxClass: gvisor - defaultVersion: v4 + defaultVersion: v5 versions: - # gVisor nightly 2026-10-17, the first to include google/gvisor@b79256080dec - # (save and restore multiple filesystem checkpoint bundles). ateworker-gvisor - # uses it to save container rootfs overlays (rootfs/) and durable-dir volumes - # (fs/) as separate bundles, and to run the sandbox with no root container - # (google/gvisor#15551). - - name: v4 + # gVisor release 20261019.0, the first tagged release with + # google/gvisor@b79256080dec (save and restore multiple filesystem checkpoint + # bundles). ateworker-gvisor uses it to save container rootfs overlays + # (rootfs/) and durable-dir volumes (fs/) as separate bundles, and to run the + # sandbox with no root container (google/gvisor#15551). + - name: v5 state: Enabled assets: amd64: gvisor: - url: "gs://gvisor/releases/nightly/2026-10-17/x86_64/gvisor.tar.zstd" - sha256: "fa72f0d45756c69a30beec21222c78241b0d330c36f4f2185648617cc23415c8" + url: "gs://gvisor/releases/release/20261019.0/x86_64/gvisor.tar.zstd" + sha256: "8637abfb50f81552af25ca08f7172589c09292a7c881efe8d9e09a9f3b6bec7b" arm64: gvisor: - url: "gs://gvisor/releases/nightly/2026-10-17/aarch64/gvisor.tar.zstd" - sha256: "238634978d99f0dd3e631c4f88eae7b19fb36937065428467f0b6d6b10697689" + url: "gs://gvisor/releases/release/20261019.0/aarch64/gvisor.tar.zstd" + sha256: "88556c21468a066d872a055425958c83fcbafc79c6477a195b18070c77f856d0" diff --git a/charts/substrate/values.yaml b/charts/substrate/values.yaml index 336fd71cac..ca2d2b4e39 100644 --- a/charts/substrate/values.yaml +++ b/charts/substrate/values.yaml @@ -233,7 +233,7 @@ image: tag: "" images: - rustfs: rustfs/rustfs:1.0.0-beta.3@sha256:378642b05b7dcb4849fb77ebe6aca4ced1c3f66e7e504247df95a5c9018d3358 - awsCli: amazon/aws-cli:2.17.0@sha256:643507c10ada7964ca6157b3d799f030b90577643da9955d319a77399ed80d73 + rustfs: rustfs/rustfs:1.0.3@sha256:9ca9559533b537d10482fe923288b676ad08f92a7069d9a17327fbae3cab688b + awsCli: amazon/aws-cli:2.38.5@sha256:c8156a0e74e13929a42d7e60f0d4e14e7d28d980cd21822505d4bff80e5460c1 agentgateway: ghcr.io/kagent-dev/substrate/agentgateway:c8550278b2f2@sha256:5703311439f92a898603e2abed32edcbaf6ec659dbb4d617568cef6d9c6fc789 busybox: busybox:1.36 diff --git a/cmd/ate-setup/README.md b/cmd/ate-setup/README.md index 9d99f27b31..dafb7671ad 100644 --- a/cmd/ate-setup/README.md +++ b/cmd/ate-setup/README.md @@ -11,22 +11,29 @@ go build -o bin/ate-setup ./cmd/ate-setup that existing command lines and CI jobs keep working; it holds no install logic of its own. -## Installing a release +## Installing pre-built images By default every image is built from the checkout with `ko`, which is what a -developer wants and what the shell installer always did. To install published -images instead, name the registry they were pushed to: +developer wants and what the shell installer always did. To install images +that are already published instead, name the registry that holds them and the +tag they share: ``` ate-setup deploy ate-system \ --credential-provider='{"name":"k8s.io"}' \ --image-repo registry.example.com/substrate \ - --image-tag v0.0.0 + --image-tag vX.Y.Z ``` -Nothing is built and no registry is pushed to; the manifests still come from the -checkout. Each reference is pinned to the digest its tag names, so the registry -has to be readable from here as well as from the cluster. +The registry has to hold every image the install deploys. The images a release +publishes to `ghcr.io/agent-substrate/substrate` are not enough on their own; +see [`docs/dev/releasing.md`](../../docs/dev/releasing.md#release-images). An +install that needs an image the registry lacks fails naming it. + +Nothing is built and no registry is pushed to; the manifests still come from +the checkout, which should be the tag being installed. Each reference is pinned +to the digest its tag names, so the registry has to be readable from here as +well as from the cluster. - [`commands.md`](commands.md) — every command with its `hack/install-ate.sh` equivalent. diff --git a/cmd/ate-setup/commands.md b/cmd/ate-setup/commands.md index 54a80ea7f1..c47ce3da89 100644 --- a/cmd/ate-setup/commands.md +++ b/cmd/ate-setup/commands.md @@ -60,7 +60,7 @@ rollout waits still diagnose workloads that cannot become ready. The shell shim runs actions in argument order, so put `--deploy-ate-system` before demo or benchmark deployment flags when combining them on one command line. -## Installing a release +## Installing pre-built images Without `--image-repo`, `ate-setup` builds every image from the checkout with `ko` and pushes it to `KO_DOCKER_REPO`. That is the developer install and is @@ -69,29 +69,32 @@ unchanged. ``` ate-setup deploy ate-system \ --credential-provider='{"name":"k8s.io"}' \ - --image-repo registry.example.com/substrate \ - --image-tag v0.0.0 + --image-repo REPO \ + --image-tag TAG ``` -installs published images instead, and never invokes `ko`. The manifests still -come from the checkout, so this needs one; what it removes is the build, the Go -toolchain, and write access to a registry. - -`REPO` has to hold every component image the manifests reference, all under the -same tag, which is how a release publishes them. A release that adds a component -has to publish it alongside the others before a pre-built install can use it. -`VERSION=TAG ate-setup publish release-images --ko-docker-repo REPO` publishes -the full set, including `envoy-dataplane`, which is built from a Dockerfile with -`docker buildx` rather than with `ko`. A build from source builds that image itself, so -it needs `docker` as well as `ko`. -Each reference is then pinned to the digest its tag names, which takes one HEAD +installs images already published to `REPO` instead, and never invokes `ko`. +The manifests still come from the checkout, so this needs one; what it removes +is the build, the Go toolchain, and write access to a registry. + +`REPO` has to hold every image the install deploys, all under `TAG`, including +`envoy-dataplane`, which a build from source builds from a Dockerfile with +`docker buildx` (so that build needs `docker` as well as `ko`). An install that +needs an image `REPO` lacks fails naming it. + +The images a release publishes to `ghcr.io/agent-substrate/substrate`, with +`publish release-images`, are not a complete install; see +[`docs/dev/releasing.md`](../../docs/dev/releasing.md#release-images). When an +image releases do not publish is missing from `REPO`, the error says so. + +Each reference is pinned to the digest its tag names, which takes one HEAD request per image, so the installer needs read access to `REPO` and not only the cluster does. That read is authenticated with the docker config file and the credential helpers it names, plus gcloud's own credentials for GCR and Artifact Registry: -Application Default Credentials, falling back to the `gcloud` CLI. Installing a -release onto GKE therefore needs no `~/.docker/config.json` entry. Amazon ECR +Application Default Credentials, falling back to the `gcloud` CLI. Installing +from GCR or Artifact Registry therefore needs no `~/.docker/config.json` entry. Amazon ECR and Azure Container Registry need credential-helper modules this repository does not depend on, so those registries need a `docker login` first. @@ -123,14 +126,21 @@ a running cluster. |---|---| | `publish worker-images` | (no shell equivalent) | | `publish release-images` | (no shell equivalent) | +| `publish release-binaries` | (no shell equivalent) | `publish worker-images` builds and pushes the worker images for the checked-out build and prints their refs; a WorkerPool points `spec.workerImage` to a build to use the ateworker. -`publish release-images` builds and pushes every image a pre-built install -needs, all tagged with the build version; see [Installing a -release](#installing-a-release). +`publish release-images` builds and pushes every image a release publishes, +all tagged with the build version; see [Installing pre-built +images](#installing-pre-built-images). + +`publish release-binaries` builds `kubectl-ate` for linux and darwin on amd64 +and arm64 into `--out` (default `bin/release`), stamped with the build version, commit, +and commit time, each with a `.sha256` file that `shasum -a 256 -c` checks. The +version has to be a release tag. It uploads nothing; the release workflow +attaches the files to the draft release. ## Delete @@ -198,9 +208,9 @@ See |---|---|---| | `deploy demo counter` | `--deploy-demo-counter` | A counter actor exercising snapshot, resume, and atenet ingress | | `deploy demo counter --with-external-volume [--storage-class NAME]` | `--deploy-demo-counter-with-external-volume` (`STORAGE_CLASS=NAME`) | The same, plus an external volume and a pre-seeded file to validate (requires `--preview=ExternalVolumes`). Run `setup csi` first and name the class it created, e.g. `csi-nfs-sc`; defaults to `standard` | -| `deploy demo counter-microvm` | `--deploy-demo-counter-microvm` | The counter demo on micro-VM workers. Run `hack/install-microvm-deps.sh --install` first | +| `deploy demo counter-microvm` | `--deploy-demo-counter-microvm` | The counter demo on microVM workers. Run `hack/install-microvm-deps.sh --install` first | | `deploy demo egress` | `--deploy-demo-egress` | Egress policy enforcement through atenet; the actors trust the egress gateway's CA | -| `deploy demo egress-microvm` | `--deploy-demo-egress-microvm` | The same on micro-VM workers. Run `hack/install-microvm-deps.sh --install` first | +| `deploy demo egress-microvm` | `--deploy-demo-egress-microvm` | The same on microVM workers. Run `hack/install-microvm-deps.sh --install` first | | `deploy demo jupyter` | `--deploy-demo-jupyter` | A Jupyter notebook server per actor, reached through atenet ingress | | `deploy demo sandbox` | `--deploy-demo-sandbox` | An on-demand sandbox actor driven by the sandbox client | | `deploy demo multi-template` | `--deploy-demo-multi-template` | Two ActorTemplates sharing one WorkerPool | diff --git a/cmd/ate-setup/differences.md b/cmd/ate-setup/differences.md index e54eb99e31..9c53e93872 100644 --- a/cmd/ate-setup/differences.md +++ b/cmd/ate-setup/differences.md @@ -143,7 +143,16 @@ listed package would otherwise inherit its image. `images.Components` is checked against the real manifests by a test, so adding a component fails a test rather than an install — though only the list is checked that way. The release also has to publish the new image under the same repository and tag as the rest, and -until it does, the component can only be installed by building it. +until it does, an install has to build it or take it from a registry that +holds it. + +Releases publish only `images.ReleaseComponents`, an allowlist: a component is +published only once it is added to it, so a new entry in `images.Components` +stays out of releases until someone adds it on purpose. See +[`docs/dev/releasing.md`](../../docs/dev/releasing.md#release-images) for what +releases cover. Under `--image-repo`, a component that is not on the allowlist +is looked up like any other, so a registry that holds it works. If the lookup +fails, the error adds that releases do not publish it. ## Cluster access diff --git a/cmd/ate-setup/internal/cmd/publish.go b/cmd/ate-setup/internal/cmd/publish.go index 8f9ce0bef8..5294157ef2 100644 --- a/cmd/ate-setup/internal/cmd/publish.go +++ b/cmd/ate-setup/internal/cmd/publish.go @@ -16,11 +16,22 @@ package cmd import ( "github.com/spf13/cobra" + + "go.ate.dev/substrate/cmd/ate-setup/internal/config" + "go.ate.dev/substrate/cmd/ate-setup/internal/steps" +) + +const ( + publishReleaseImagesPath = "publish release-images" + publishReleaseBinariesPath = "publish release-binaries" ) var publishCmd = &cobra.Command{ Use: "publish", - Short: "Build and push images that no manifest references", + Short: "Build release images and binaries, and images that no manifest references", + // Publishing builds images and binaries and never contacts a cluster, so + // it runs where there is no kubeconfig, such as a release job. + Annotations: map[string]string{clusterAnnotation: "no"}, } var publishWorkerImagesCmd = &cobra.Command{ @@ -40,17 +51,42 @@ worker ref.`, var publishReleaseImagesCmd = &cobra.Command{ Use: "release-images", - Short: "Build and push every image a pre-built install needs, tagged with the build version", - Long: `Build and push every component image, including the Dockerfile-built -envoy-dataplane, to KO_DOCKER_REPO, all tagged with the build version (VERSION, -else git describe), and print their pushed references. + Short: "Build and push the release images, tagged with the release version", + Long: `Build and push the images a release publishes (images.ReleaseComponents) to +KO_DOCKER_REPO, all tagged with the build version, and print their pushed +references. The version comes from VERSION, else git describe, and has to be a +release tag (vMAJOR.MINOR.PATCH[-PRERELEASE]). + + VERSION=TAG ate-setup publish release-images --ko-docker-repo REPO -The result is what "deploy --image-repo REPO --image-tag TAG" installs: +The release images are not a complete install; docs/dev/releasing.md lists +what they cover. - VERSION=TAG ate-setup publish release-images --ko-docker-repo REPO`, +If REPO already holds every image for the tag on every platform, nothing is +built unless --overwrite is set. After pushing, every image is checked for +every platform, and a missing one fails the command.`, Args: cobra.NoArgs, RunE: func(cmd *cobra.Command, _ []string) error { - return env.PublishReleaseImages(cmd.Context(), cmd.OutOrStdout()) + opts := steps.PublishReleaseOptions{Overwrite: env.Cfg.Resolved().Bool("publish.overwrite")} + return env.PublishReleaseImages(cmd.Context(), cmd.OutOrStdout(), opts) + }, +} + +var publishReleaseBinariesCmd = &cobra.Command{ + Use: "release-binaries", + Short: "Build the kubectl-ate release binaries and their checksums into a directory", + Long: `Build kubectl-ate for linux and darwin on amd64 and arm64, statically and +stamped with the build version (VERSION, else git describe), the commit, and +the commit time, into --out. Beside each kubectl-ate-- binary it +writes a .sha256 file that "shasum -a 256 -c" checks. + + VERSION=TAG ate-setup publish release-binaries + +The version has to be a release tag. Nothing is uploaded; the release +workflow attaches the files to the draft release.`, + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, _ []string) error { + return env.PublishReleaseBinaries(cmd.Context(), cmd.OutOrStdout(), env.Cfg.Resolved().String("publish.binariesDir")) }, } @@ -58,4 +94,21 @@ func init() { rootCmd.AddCommand(publishCmd) publishCmd.AddCommand(publishWorkerImagesCmd) publishCmd.AddCommand(publishReleaseImagesCmd) + publishCmd.AddCommand(publishReleaseBinariesCmd) + + config.RegisterCommand(config.Setting{ + Key: "publish.binariesDir", Env: "ATE_PUBLISH_BINARIES_DIR", Flag: "out", + Kind: config.KindString, Default: "bin/release", + Commands: []string{publishReleaseBinariesPath}, + Usage: "Directory the release binaries and their checksums are written to", + }) + config.BindCommandFlags(publishReleaseBinariesPath, publishReleaseBinariesCmd.Flags()) + + config.RegisterCommand(config.Setting{ + Key: "publish.overwrite", Env: "ATE_PUBLISH_OVERWRITE", Flag: "overwrite", + Kind: config.KindBool, Default: "false", + Commands: []string{publishReleaseImagesPath}, + Usage: "Rebuild and replace images the registry already holds for the tag", + }) + config.BindCommandFlags(publishReleaseImagesPath, publishReleaseImagesCmd.Flags()) } diff --git a/cmd/ate-setup/internal/cmd/record_test.go b/cmd/ate-setup/internal/cmd/record_test.go index 0787e14669..838509a5c9 100644 --- a/cmd/ate-setup/internal/cmd/record_test.go +++ b/cmd/ate-setup/internal/cmd/record_test.go @@ -60,6 +60,35 @@ func TestOnlySystemDeploysRecordTheirRun(t *testing.T) { } } +// The publish commands build and push without a cluster, so the release job +// runs them with no kubeconfig. Every other command connects to one. +func TestOnlyPublishSkipsTheCluster(t *testing.T) { + for _, tc := range []struct { + path string + want bool + }{ + {path: "publish release-images", want: false}, + {path: "publish release-binaries", want: false}, + {path: "publish worker-images", want: false}, + {path: "deploy ate-system", want: true}, + {path: "delete ate-system", want: true}, + {path: "setup csi", want: true}, + } { + t.Run(tc.path, func(t *testing.T) { + target, _, err := rootCmd.Find(strings.Fields(tc.path)) + if err != nil { + t.Fatalf("Find(%q): %v", tc.path, err) + } + if target.CommandPath() != "ate-setup "+tc.path { + t.Fatalf("%q resolved to %q", tc.path, target.CommandPath()) + } + if got := needsCluster(target); got != tc.want { + t.Errorf("needsCluster(%q) = %v, want %v", tc.path, got, tc.want) + } + }) + } +} + // Nothing is recorded before the configuration is resolved: a run that failed // parsing its flags has no settings to record, and reaching for them would // dereference nil. diff --git a/cmd/ate-setup/internal/cmd/root.go b/cmd/ate-setup/internal/cmd/root.go index e08a206739..4fa51941ed 100644 --- a/cmd/ate-setup/internal/cmd/root.go +++ b/cmd/ate-setup/internal/cmd/root.go @@ -59,15 +59,35 @@ var resolved *config.Resolved // Commands that install nothing -- delete, publish, setup -- are not marked. const recordAnnotation = "record" -// recordsRun reports whether the nearest annotated command, cmd or an -// ancestor, opted in. -func recordsRun(cmd *cobra.Command) bool { +// clusterAnnotation says whether a command contacts a cluster: "no" for one +// that does not. As with recordAnnotation, the nearest annotated command +// decides. A command marked "no" gets an environment without a Kubernetes +// client, so it runs on a host with no kubeconfig. +const clusterAnnotation = "cluster" + +// nearestAnnotation returns the value of key on cmd or its nearest ancestor +// that carries it. +func nearestAnnotation(cmd *cobra.Command, key string) (string, bool) { for c := cmd; c != nil; c = c.Parent() { - if v, ok := c.Annotations[recordAnnotation]; ok { - return v == "yes" + if v, ok := c.Annotations[key]; ok { + return v, true } } - return false + return "", false +} + +// recordsRun reports whether the nearest annotated command, cmd or an +// ancestor, opted in. +func recordsRun(cmd *cobra.Command) bool { + v, _ := nearestAnnotation(cmd, recordAnnotation) + return v == "yes" +} + +// needsCluster reports whether cmd contacts a cluster. Commands do unless +// marked otherwise. +func needsCluster(cmd *cobra.Command) bool { + v, _ := nearestAnnotation(cmd, clusterAnnotation) + return v != "no" } var rootCmd = &cobra.Command{ @@ -107,6 +127,11 @@ is present (skipped for --kind, and by NO_DEV_ENV=1).`, // screen even when connecting fails. resolved.Report(log.Writer(), config.ReportOptions{OmitDefaults: noReportDefaults}) + if !needsCluster(cmd) { + env = steps.NewLocalEnv(cfg) + return nil + } + // Populate the kubeconfig before any client is built, for the // GKE-from-.ate-dev-env.sh flow. if err := cfg.EnsureClusterCredentials(cmd.Context()); err != nil { diff --git a/cmd/ate-setup/internal/images/dockerfile.go b/cmd/ate-setup/internal/images/dockerfile.go index 5b0549d126..78211a3c3c 100644 --- a/cmd/ate-setup/internal/images/dockerfile.go +++ b/cmd/ate-setup/internal/images/dockerfile.go @@ -84,14 +84,8 @@ func dockerBuildArgs(platforms string, extraFlags []string, tag, contextPath str // push to; the returned reference always uses the digest, so a stale tag can // never be resolved by accident. func BuildDockerfileImage(ctx context.Context, rootDir, dockerRepo, imageName, contextPath, koDefaultPlatforms string, extraFlags []string) (string, error) { - return PublishDockerfileImage(ctx, rootDir, dockerRepo, imageName, fmt.Sprintf("build-%d", time.Now().Unix()), contextPath, koDefaultPlatforms, extraFlags) -} - -// PublishDockerfileImage is BuildDockerfileImage with the tag chosen by the -// caller, for a release that publishes every image under one tag. -func PublishDockerfileImage(ctx context.Context, rootDir, dockerRepo, imageName, tag, contextPath, koDefaultPlatforms string, extraFlags []string) (string, error) { repo := strings.TrimSuffix(dockerRepo, "/") + "/" + imageName - stageTag := repo + ":" + tag + stageTag := fmt.Sprintf("%s:build-%d", repo, time.Now().Unix()) platforms, err := dockerfilePlatforms(rootDir, koDefaultPlatforms) if err != nil { diff --git a/cmd/ate-setup/internal/images/images.go b/cmd/ate-setup/internal/images/images.go index 79ae3fd69b..2110fe26b9 100644 --- a/cmd/ate-setup/internal/images/images.go +++ b/cmd/ate-setup/internal/images/images.go @@ -24,6 +24,7 @@ package images import ( "fmt" "path" + "slices" ) // ModulePath is this repository's module path. A ko:// reference is it joined @@ -38,7 +39,7 @@ const ModulePath = "go.ate.dev/substrate" // that was never pushed. TestComponentsCoverTheManifests keeps it in step with // the manifests, which is half of it: a new component also has to be published // under the same repository and tag as the rest before a pre-built install can -// use it. +// use it, and a release publishes it only once it is added to released. var Components = []string{ "cmd/ate-idp-server", "cmd/ateapi", @@ -58,6 +59,29 @@ var Components = []string{ "demos/sandbox", } +// released is the allowlist of Components a release publishes. A component is +// published only after it is added here, so one added to Components stays out +// of releases until it is added here in review. A pre-built install still +// looks an unlisted image up in its registry; if the lookup fails, the error +// says releases do not publish it. +var released = []string{ + "cmd/ate-idp-server", + "cmd/ateapi", + "cmd/atecontroller", + "cmd/atelet", + "cmd/atenet", + "cmd/ateworker-gvisor", + "cmd/ateworker-microvm", + "cmd/objectstore-plugin", + "cmd/podcertcontroller", +} + +// ReleaseComponents returns the Components a release publishes. +func ReleaseComponents() []string { return slices.Clone(released) } + +// IsReleased reports whether a release publishes pkg. +func IsReleased(pkg string) bool { return slices.Contains(released, pkg) } + // KoReference is the ko:// reference the manifests name pkg by. // // ko resolves a manifest reference as written, so a "./"-relative import path diff --git a/cmd/ate-setup/internal/images/images_test.go b/cmd/ate-setup/internal/images/images_test.go index 293da7e64d..85c2e1f47d 100644 --- a/cmd/ate-setup/internal/images/images_test.go +++ b/cmd/ate-setup/internal/images/images_test.go @@ -134,3 +134,11 @@ func TestImageNamesAreUnique(t *testing.T) { byName[name] = pkg } } + +// A nested package publishes under its last path element, which is how a +// build from source names it and how --image-repo has to find it. +func TestImageNameOfANestedPackage(t *testing.T) { + if got := images.ImageName("demos/multi-template/fspersist"); got != "fspersist" { + t.Errorf("ImageName() = %q, want fspersist", got) + } +} diff --git a/cmd/ate-setup/internal/images/prebuilt.go b/cmd/ate-setup/internal/images/prebuilt.go index 873dcb01e8..5d36a1bbd7 100644 --- a/cmd/ate-setup/internal/images/prebuilt.go +++ b/cmd/ate-setup/internal/images/prebuilt.go @@ -129,6 +129,13 @@ func (p *Prebuilt) rewrite(ctx context.Context, manifest []byte) ([]byte, error) } ref, err := p.Pin(ctx, image) if err != nil { + // The registry may hold an image releases do not publish, so it is + // looked up like any other, and only a failed lookup says why it + // might be missing. + if pkg := strings.TrimPrefix(string(match), "ko://"+ModulePath+"/"); !IsReleased(pkg) { + err = fmt.Errorf("%w; releases do not publish %s, so publish it to %s yourself or build it from source by dropping --image-repo", + err, image, p.src.Repo) + } unpinned[image] = err return match } diff --git a/cmd/ate-setup/internal/images/prebuilt_test.go b/cmd/ate-setup/internal/images/prebuilt_test.go index 41a36bf181..ccf5bc48ce 100644 --- a/cmd/ate-setup/internal/images/prebuilt_test.go +++ b/cmd/ate-setup/internal/images/prebuilt_test.go @@ -83,8 +83,8 @@ func TestPrebuiltResolveBytes(t *testing.T) { }, { name: "flow sequence", - in: `images: [ko://go.ate.dev/substrate/demos/counter, other]`, - want: `images: [example.com/substrate/counter:v1.2.3` + digestSuffix + `, other]`, + in: `images: [ko://go.ate.dev/substrate/cmd/atecontroller, other]`, + want: `images: [example.com/substrate/atecontroller:v1.2.3` + digestSuffix + `, other]`, }, { // The reference is a plain CRD field here, not a pod spec, which is @@ -102,15 +102,10 @@ func TestPrebuiltResolveBytes(t *testing.T) { in: "image: ko://go.ate.dev/substrate/cmd/ateapi\n", want: "image: example.com/substrate/ateapi:v1.2.3" + digestSuffix + "\n", }, - { - name: "nested package maps to its image name", - in: "image: ko://go.ate.dev/substrate/demos/multi-template/fspersist\n", - want: "image: example.com/substrate/fspersist:v1.2.3" + digestSuffix + "\n", - }, { name: "two references on one line", - in: "a: ko://go.ate.dev/substrate/cmd/ateapi, b: ko://go.ate.dev/substrate/demos/egress\n", - want: "a: example.com/substrate/ateapi:v1.2.3" + digestSuffix + ", b: example.com/substrate/egress:v1.2.3" + digestSuffix + "\n", + in: "a: ko://go.ate.dev/substrate/cmd/ateapi, b: ko://go.ate.dev/substrate/cmd/atelet\n", + want: "a: example.com/substrate/ateapi:v1.2.3" + digestSuffix + ", b: example.com/substrate/atelet:v1.2.3" + digestSuffix + "\n", }, { // The demo templates carry "# ko:// image reference" as prose. The @@ -233,6 +228,41 @@ func TestPrebuiltReportsEveryFailure(t *testing.T) { } } +// An image releases do not publish resolves from a registry that holds it, +// such as one the operator published to. When the lookup fails, the error says +// releases do not publish it; for a released image it does not. +func TestPrebuiltUnreleasedImages(t *testing.T) { + const ref = "image: ko://go.ate.dev/substrate/cmd/credential-provider/kubernetes-secrets\n" + const hint = "releases do not publish kubernetes-secrets" + + got, err := images.NewPrebuilt(testSource, newStubRegistry().digest). + ResolveBytes(context.Background(), []byte(ref)) + if err != nil { + t.Fatalf("ResolveBytes() from a registry that holds it = %v", err) + } + if want := "image: example.com/substrate/kubernetes-secrets:v1.2.3" + digestSuffix + "\n"; string(got) != want { + t.Errorf("ResolveBytes() = %q, want %q", got, want) + } + + missing := newStubRegistry() + missing.err = errors.New("MANIFEST_UNKNOWN") + _, err = images.NewPrebuilt(testSource, missing.digest).ResolveBytes(context.Background(), []byte(ref)) + if err == nil { + t.Fatal("ResolveBytes() from a registry without it = nil error, want one") + } + for _, want := range []string{"MANIFEST_UNKNOWN", hint, "dropping --image-repo"} { + if !strings.Contains(err.Error(), want) { + t.Errorf("error is missing %q:\n%v", want, err) + } + } + + _, err = images.NewPrebuilt(testSource, missing.digest). + ResolveBytes(context.Background(), []byte("image: ko://go.ate.dev/substrate/cmd/ateapi\n")) + if err == nil || strings.Contains(err.Error(), "releases do not publish") { + t.Errorf("ResolveBytes() for a released image = %v, want a lookup error without the release hint", err) + } +} + // A repeated bad reference is one problem, not one per occurrence. func TestPrebuiltDeduplicatesFailures(t *testing.T) { ref := "image: ko://go.ate.dev/substrate/cmd/nope\n" diff --git a/cmd/ate-setup/internal/images/release.go b/cmd/ate-setup/internal/images/release.go new file mode 100644 index 0000000000..72fc7f9d95 --- /dev/null +++ b/cmd/ate-setup/internal/images/release.go @@ -0,0 +1,239 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package images + +import ( + "context" + "errors" + "fmt" + "net/http" + "regexp" + "strings" + + "github.com/google/go-containerregistry/pkg/name" + v1 "github.com/google/go-containerregistry/pkg/v1" + "github.com/google/go-containerregistry/pkg/v1/remote" + "github.com/google/go-containerregistry/pkg/v1/remote/transport" +) + +// SourceLabel is the OCI image label naming the repository an image was built +// from. ghcr.io connects a new package carrying it to that repository, which +// lets the repository's workflows push later versions with their GITHUB_TOKEN. +const SourceLabel = "org.opencontainers.image.source" + +// SourceURL is the SourceLabel value for images built from this repository: +// its GitHub URL, which ghcr.io matches. It cannot be derived from ModulePath, +// which is a vanity import path. +const SourceURL = "https://github.com/agent-substrate/substrate" + +// RevisionLabel is the OCI image label naming the commit an image was built +// from. +const RevisionLabel = "org.opencontainers.image.revision" + +// releaseTag matches a release tag, vMAJOR.MINOR.PATCH, or a release candidate, +// vMAJOR.MINOR.PATCH-PRERELEASE. hack/release/draft-release.sh accepts the same. +var releaseTag = regexp.MustCompile(`^v[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$`) + +// CheckReleaseVersion returns an error unless version is a release tag. +func CheckReleaseVersion(version string) error { + if !releaseTag.MatchString(version) { + return fmt.Errorf("%q is not a release tag (vMAJOR.MINOR.PATCH[-PRERELEASE])", version) + } + return nil +} + +// CheckReleaseTag returns an error unless tag is a release tag that makes a +// valid image reference under repo. It runs before anything is built, so a +// stray tag or a value ko would split, such as "v1.2.3,latest", publishes +// nothing. +func CheckReleaseTag(repo, tag string) error { + if err := CheckReleaseVersion(tag); err != nil { + return err + } + if _, err := name.NewTag(strings.TrimSuffix(repo, "/")+"/image:"+tag, name.StrictValidation); err != nil { + return fmt.Errorf("%s with tag %s is not a valid image reference: %w", repo, tag, err) + } + return nil +} + +// StaleTags returns a line for each of tagged whose tag does not name the +// digest pushed for its repository. pushed are digest references, one per +// repository, as ko and buildx print them (repo/name:tag@sha256:... or +// repo/name@sha256:...). It catches a tag that still names an earlier push, +// which a platform check alone would pass. +func StaleTags(ctx context.Context, tagged, pushed []string) ([]string, error) { + want := make(map[string]string, len(pushed)) + for _, ref := range pushed { + d, err := name.NewDigest(ref) + if err != nil { + return nil, fmt.Errorf("%s is not a digest reference: %w", ref, err) + } + want[d.Context().Name()] = d.DigestStr() + } + var stale []string + for _, ref := range tagged { + t, err := name.NewTag(ref) + if err != nil { + return nil, fmt.Errorf("%s is not a tagged reference: %w", ref, err) + } + expected, ok := want[t.Context().Name()] + if !ok { + stale = append(stale, fmt.Sprintf("%s: nothing was pushed for it", ref)) + continue + } + got, err := RemoteDigest(ctx, ref) + if err != nil { + return nil, err + } + if got != expected { + stale = append(stale, fmt.Sprintf("%s names %s, not the pushed %s", ref, got, expected)) + } + } + return stale, nil +} + +// Platforms returns the platforms a build produces, the way ko chooses them: +// KO_DEFAULTPLATFORMS, else defaultPlatforms from the ko config, else +// linux/amd64. Dockerfile-built images use the same set. +func Platforms(rootDir, koDefaultPlatforms string) ([]string, error) { + platforms, err := dockerfilePlatforms(rootDir, koDefaultPlatforms) + if err != nil { + return nil, err + } + return strings.Split(platforms, ","), nil +} + +// Incomplete is an image reference that is missing from its registry, or is +// present without every wanted platform. +type Incomplete struct { + Ref string + // Missing lists the wanted platforms the reference lacks. A reference that + // does not exist lacks all of them. + Missing []string +} + +func (i Incomplete) String() string { + return fmt.Sprintf("%s lacks %s", i.Ref, strings.Join(i.Missing, ", ")) +} + +// CheckPublished returns the refs that do not hold an image for each of +// platforms. An error is returned only when a registry +// cannot answer (network, credentials), since that says nothing about whether +// an image exists; treating it as missing would lead a caller to rebuild and +// replace images that are already published. +func CheckPublished(ctx context.Context, refs, platforms []string) ([]Incomplete, error) { + want := make([]v1.Platform, 0, len(platforms)) + for _, p := range platforms { + parsed, err := v1.ParsePlatform(p) + if err != nil { + return nil, fmt.Errorf("%q is not a platform: %w", p, err) + } + want = append(want, *parsed) + } + + var incomplete []Incomplete + for _, ref := range refs { + have, err := indexPlatforms(ctx, ref) + if err != nil { + return nil, err + } + var missing []string + for i, w := range want { + if !satisfiedBy(w, have) { + missing = append(missing, platforms[i]) + } + } + if len(missing) > 0 { + incomplete = append(incomplete, Incomplete{Ref: ref, Missing: missing}) + } + } + return incomplete, nil +} + +// indexPlatforms returns the platforms of the images ref names: each image in +// an index, or the one image of a plain manifest, which ko pushes when a build +// targets a single platform. It returns none, without an error, if ref does +// not exist. +func indexPlatforms(ctx context.Context, ref string) ([]v1.Platform, error) { + parsed, err := name.ParseReference(ref) + if err != nil { + return nil, fmt.Errorf("%s is not a valid image reference: %w", ref, err) + } + desc, err := remote.Get(parsed, + remote.WithContext(ctx), + remote.WithAuthFromKeychain(keychain), + ) + if isNotFound(err) { + return nil, nil + } + if err != nil { + return nil, fmt.Errorf("while inspecting %s: %w", ref, err) + } + if !desc.MediaType.IsIndex() { + img, err := desc.Image() + if err != nil { + return nil, fmt.Errorf("while reading the image %s names: %w", ref, err) + } + config, err := img.ConfigFile() + if err != nil { + return nil, fmt.Errorf("while reading the config of %s: %w", ref, err) + } + return []v1.Platform{{OS: config.OS, Architecture: config.Architecture, Variant: config.Variant}}, nil + } + index, err := desc.ImageIndex() + if err != nil { + return nil, fmt.Errorf("while reading the index of %s: %w", ref, err) + } + manifest, err := index.IndexManifest() + if err != nil { + return nil, fmt.Errorf("while reading the index of %s: %w", ref, err) + } + var have []v1.Platform + for _, m := range manifest.Manifests { + if m.Platform != nil { + have = append(have, *m.Platform) + } + } + return have, nil +} + +// isNotFound reports whether err is a registry saying the repository or the +// tag does not exist. +func isNotFound(err error) bool { + var terr *transport.Error + if !errors.As(err, &terr) { + return false + } + if terr.StatusCode == http.StatusNotFound { + return true + } + for _, d := range terr.Errors { + if d.Code == transport.ManifestUnknownErrorCode || d.Code == transport.NameUnknownErrorCode { + return true + } + } + return false +} + +// satisfiedBy reports whether any of have is an image for want. A want without +// a variant, such as linux/arm64, accepts any variant. +func satisfiedBy(want v1.Platform, have []v1.Platform) bool { + for _, h := range have { + if h.Satisfies(want) { + return true + } + } + return false +} diff --git a/cmd/ate-setup/internal/images/release_test.go b/cmd/ate-setup/internal/images/release_test.go new file mode 100644 index 0000000000..1df4f73213 --- /dev/null +++ b/cmd/ate-setup/internal/images/release_test.go @@ -0,0 +1,207 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package images + +import ( + "io" + "log" + "net/http" + "net/http/httptest" + "slices" + "strings" + "testing" + + "github.com/google/go-containerregistry/pkg/name" + "github.com/google/go-containerregistry/pkg/registry" + v1 "github.com/google/go-containerregistry/pkg/v1" + "github.com/google/go-containerregistry/pkg/v1/empty" + "github.com/google/go-containerregistry/pkg/v1/mutate" + "github.com/google/go-containerregistry/pkg/v1/remote" +) + +// An allowlist entry that is not a component publishes nothing and hides that +// the component it meant was renamed or removed. +func TestReleasedAreComponents(t *testing.T) { + seen := make(map[string]bool, len(released)) + for _, pkg := range released { + if !slices.Contains(Components, pkg) { + t.Errorf("released names %s, which is not in Components", pkg) + } + if seen[pkg] { + t.Errorf("released lists %s twice", pkg) + } + seen[pkg] = true + } +} + +// Only a release tag that makes a valid reference passes. A comma would be +// split by ko into two tags, and a stray tag would publish a full image set to +// the release repository. +func TestCheckReleaseTag(t *testing.T) { + for _, tag := range []string{"v1.2.3", "v0.5.0-rc", "v0.5.0-rc.1", "v0.5.0-pre", "v0.5.0-pre.4", "v10.0.0-alpha.2-x"} { + if err := CheckReleaseTag("ghcr.io/agent-substrate/substrate", tag); err != nil { + t.Errorf("CheckReleaseTag(%q) = %v, want nil", tag, err) + } + } + for _, tag := range []string{"v1-test", "vnext", "v1.2.3,latest", "1.2.3", "v1.2", "v1.2.3-", "v1.2.3foo", ""} { + if err := CheckReleaseTag("ghcr.io/agent-substrate/substrate", tag); err == nil { + t.Errorf("CheckReleaseTag(%q) = nil, want an error", tag) + } + } + if err := CheckReleaseTag("not a repo", "v1.2.3"); err == nil { + t.Error("CheckReleaseTag() with an invalid repository = nil, want an error") + } +} + +// A tag that names an earlier push passes a platform check, so the digest is +// compared too. +func TestStaleTags(t *testing.T) { + srv := httptest.NewServer(registry.New(registry.Logger(log.New(io.Discard, "", 0)))) + t.Cleanup(srv.Close) + repo := strings.TrimPrefix(srv.URL, "http://") + "/substrate" + + amd64 := v1.Platform{OS: "linux", Architecture: "amd64"} + arm64 := v1.Platform{OS: "linux", Architecture: "arm64"} + pushIndex(t, repo+"/current:v1", amd64) + pushIndex(t, repo+"/stale:v1", amd64) + current := indexDigest(t, repo+"/current:v1") + // What the run pushed for "stale": a different index, never tagged. + pushed := mutate.AppendManifests(empty.Index, mutate.IndexAddendum{ + Add: empty.Image, Descriptor: v1.Descriptor{Platform: &arm64}, + }) + pushedDigest, err := pushed.Digest() + if err != nil { + t.Fatal(err) + } + + got, err := StaleTags(t.Context(), + []string{repo + "/current:v1", repo + "/stale:v1", repo + "/missing:v1"}, + []string{repo + "/current:v1@" + current, repo + "/stale@" + pushedDigest.String()}, + ) + if err != nil { + t.Fatalf("StaleTags() error = %v", err) + } + if len(got) != 2 || !strings.Contains(got[0], "stale:v1 names") || !strings.Contains(got[1], "missing:v1: nothing was pushed") { + t.Errorf("StaleTags() = %q, want the stale and the missing tag", got) + } +} + +// indexDigest is the digest ref's tag names in the test registry. +func indexDigest(t *testing.T, ref string) string { + t.Helper() + d, err := RemoteDigest(t.Context(), ref) + if err != nil { + t.Fatal(err) + } + return d +} + +func parseRef(t *testing.T, ref string) name.Reference { + t.Helper() + parsed, err := name.ParseReference(ref) + if err != nil { + t.Fatal(err) + } + return parsed +} + +// pushIndex pushes an index to ref with one empty image per platform. +func pushIndex(t *testing.T, ref string, platforms ...v1.Platform) { + t.Helper() + var index v1.ImageIndex = empty.Index + for _, p := range platforms { + index = mutate.AppendManifests(index, mutate.IndexAddendum{ + Add: empty.Image, + Descriptor: v1.Descriptor{Platform: &p}, + }) + } + if err := remote.WriteIndex(parseRef(t, ref), index); err != nil { + t.Fatal(err) + } +} + +// The check decides whether a release is rebuilt, so each way an image can +// fall short has to be reported, and a complete image must not be. +func TestCheckPublished(t *testing.T) { + srv := httptest.NewServer(registry.New(registry.Logger(log.New(io.Discard, "", 0)))) + t.Cleanup(srv.Close) + repo := strings.TrimPrefix(srv.URL, "http://") + "/substrate" + + amd64 := v1.Platform{OS: "linux", Architecture: "amd64"} + arm64v8 := v1.Platform{OS: "linux", Architecture: "arm64", Variant: "v8"} + pushIndex(t, repo+"/complete:v1", amd64, arm64v8) + pushIndex(t, repo+"/amd64-only:v1", amd64) + if err := remote.Write(parseRef(t, repo+"/single:v1"), empty.Image); err != nil { + t.Fatal(err) + } + // ko pushes a plain image, not an index, when a build targets one platform. + config, err := empty.Image.ConfigFile() + if err != nil { + t.Fatal(err) + } + config.OS, config.Architecture = "linux", "amd64" + plain, err := mutate.ConfigFile(empty.Image, config) + if err != nil { + t.Fatal(err) + } + if err := remote.Write(parseRef(t, repo+"/plain-amd64:v1"), plain); err != nil { + t.Fatal(err) + } + + refs := []string{ + repo + "/complete:v1", + repo + "/amd64-only:v1", + repo + "/single:v1", + repo + "/plain-amd64:v1", + repo + "/complete:v2", + repo + "/absent:v1", + } + got, err := CheckPublished(t.Context(), refs, []string{"linux/amd64", "linux/arm64"}) + if err != nil { + t.Fatalf("CheckPublished() error = %v", err) + } + want := []Incomplete{ + {Ref: repo + "/amd64-only:v1", Missing: []string{"linux/arm64"}}, + {Ref: repo + "/single:v1", Missing: []string{"linux/amd64", "linux/arm64"}}, + {Ref: repo + "/plain-amd64:v1", Missing: []string{"linux/arm64"}}, + {Ref: repo + "/complete:v2", Missing: []string{"linux/amd64", "linux/arm64"}}, + {Ref: repo + "/absent:v1", Missing: []string{"linux/amd64", "linux/arm64"}}, + } + if !slices.EqualFunc(got, want, func(a, b Incomplete) bool { + return a.Ref == b.Ref && slices.Equal(a.Missing, b.Missing) + }) { + t.Errorf("CheckPublished() = %v, want %v", got, want) + } + + // A single-platform release is complete as a plain image. + got, err = CheckPublished(t.Context(), []string{repo + "/plain-amd64:v1"}, []string{"linux/amd64"}) + if err != nil || len(got) != 0 { + t.Errorf("CheckPublished() for one platform = %v, %v; want complete", got, err) + } +} + +// A registry that cannot answer says nothing about whether an image exists. +// Reporting it as missing would get published images rebuilt and replaced. +func TestCheckPublishedFailsWhenTheRegistryCannotAnswer(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + http.Error(w, "denied", http.StatusForbidden) + })) + t.Cleanup(srv.Close) + ref := strings.TrimPrefix(srv.URL, "http://") + "/substrate/ateapi:v1" + + if got, err := CheckPublished(t.Context(), []string{ref}, []string{"linux/amd64"}); err == nil { + t.Errorf("CheckPublished() = %v, nil; want an error", got) + } +} diff --git a/cmd/ate-setup/internal/ko/ko.go b/cmd/ate-setup/internal/ko/ko.go index 5b6fea21e4..139fa6bd63 100644 --- a/cmd/ate-setup/internal/ko/ko.go +++ b/cmd/ate-setup/internal/ko/ko.go @@ -138,10 +138,15 @@ func (r *Runner) Build(ctx context.Context, importPath string) (string, error) { } // BuildTagged builds and publishes the images for importPaths in one ko -// invocation, tagging each with tag, and returns their pushed references. -func (r *Runner) BuildTagged(ctx context.Context, tag string, importPaths ...string) ([]string, error) { +// invocation, tagging each with tag and setting each "key=value" in labels as +// an image label, and returns their pushed references. +func (r *Runner) BuildTagged(ctx context.Context, tag string, labels []string, importPaths ...string) ([]string, error) { defer log.Elapsed(time.Now(), "ko build --tags="+tag) - cmd := exec.CommandContext(ctx, r.binary, append(r.args("build", importPaths...), "--tags="+tag)...) + args := append(r.args("build", importPaths...), "--tags="+tag) + for _, label := range labels { + args = append(args, "--image-label="+label) + } + cmd := exec.CommandContext(ctx, r.binary, args...) cmd.Dir = r.Root cmd.Env = append(os.Environ(), r.Env...) var stdout bytes.Buffer @@ -175,7 +180,38 @@ func (r *Runner) args(subcommand string, target ...string) []string { // ldflags returns the version stamp ko should bake into binaries. func (r *Runner) ldflags() []string { - return []string{fmt.Sprintf("-X=%s.Version=%s", versionPkg, BuildVersion(r.Root))} + return Ldflags(r.Root) +} + +// Ldflags returns the -X flags that stamp a binary built from root with +// internal/version's Version, and with Commit and BuildDate when root has a +// commit checked out. internal/version replaces a stamped Version with the +// module's build info unless Commit is stamped too. BuildDate is the commit's +// time, so rebuilding one commit gives the same binary. +func Ldflags(root string) []string { + flags := []string{fmt.Sprintf("-X=%s.Version=%s", versionPkg, BuildVersion(root))} + if commit, date, ok := CommitStamp(root); ok { + flags = append(flags, + fmt.Sprintf("-X=%s.Commit=%s", versionPkg, commit), + fmt.Sprintf("-X=%s.BuildDate=%s", versionPkg, date)) + } + return flags +} + +// CommitStamp returns the commit checked out at root and its committer date in +// RFC 3339 form. ok is false when root has no commit, as in a source tarball. +func CommitStamp(root string) (commit, date string, ok bool) { + cmd := exec.Command("git", "show", "-s", "--format=%H %cI", "HEAD") + cmd.Dir = root + out, err := cmd.Output() + if err != nil { + return "", "", false + } + fields := strings.Fields(string(out)) + if len(fields) != 2 { + return "", "", false + } + return fields[0], fields[1], true } // BuildVersion is the VERSION environment variable, else `git describe`, or diff --git a/cmd/ate-setup/internal/ko/ko_test.go b/cmd/ate-setup/internal/ko/ko_test.go index dfe5d250af..8c5377a60b 100644 --- a/cmd/ate-setup/internal/ko/ko_test.go +++ b/cmd/ate-setup/internal/ko/ko_test.go @@ -16,6 +16,7 @@ package ko import ( "os" + "os/exec" "path/filepath" "slices" "strings" @@ -78,6 +79,48 @@ func TestBuildVersionFallsBackToDev(t *testing.T) { } } +// internal/version discards a stamped Version unless Commit is stamped too, so +// a checkout with a commit stamps the commit and its time beside the version. +// Without a commit only the version can be stamped. +func TestLdflagsStampTheCommit(t *testing.T) { + t.Setenv("VERSION", "v1.2.3") + dir := t.TempDir() + for _, args := range [][]string{ + {"init", "-q"}, + {"-c", "user.name=test", "-c", "user.email=test@example.com", "-c", "commit.gpgsign=false", "commit", "-q", "--allow-empty", "-m", "test"}, + } { + cmd := exec.Command("git", args...) + cmd.Dir = dir + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("git %v: %v\n%s", args, err, out) + } + } + commit, date, ok := CommitStamp(dir) + if !ok || len(commit) != 40 || date == "" { + t.Fatalf("CommitStamp() = %q, %q, %v; want a commit and its date", commit, date, ok) + } + + want := []string{ + "-X=go.ate.dev/substrate/internal/version.Version=v1.2.3", + "-X=go.ate.dev/substrate/internal/version.Commit=" + commit, + "-X=go.ate.dev/substrate/internal/version.BuildDate=" + date, + } + got := Ldflags(dir) + if !slices.Equal(got, want) { + t.Errorf("Ldflags() = %v, want %v", got, want) + } + // ko splits --ldflags on commas, so no stamp may contain one. + for _, flag := range got { + if strings.Contains(flag, ",") { + t.Errorf("%s contains a comma, which ko would split", flag) + } + } + + if got := Ldflags(t.TempDir()); len(got) != 1 { + t.Errorf("Ldflags() without a commit = %v, want only the version", got) + } +} + // fakeKo writes a ko stand-in that records its arguments, one per line, and // prints stdout. func fakeKo(t *testing.T, stdout string) (binary, argsFile string) { @@ -99,7 +142,7 @@ func TestBuildTagged(t *testing.T) { binary, argsFile := fakeKo(t, "repo/ateapi:v1.2.3@sha256:a\\nrepo/atelet:v1.2.3@sha256:b\\n") r := &Runner{Root: t.TempDir(), Stderr: os.Stderr, binary: binary} - refs, err := r.BuildTagged(t.Context(), "v1.2.3", "./cmd/ateapi", "./cmd/atelet") + refs, err := r.BuildTagged(t.Context(), "v1.2.3", []string{"org.example/source=https://example.com"}, "./cmd/ateapi", "./cmd/atelet") if err != nil { t.Fatalf("BuildTagged() error = %v", err) } @@ -114,7 +157,7 @@ func TestBuildTagged(t *testing.T) { if got := args[:3]; !slices.Equal(got, []string{"build", "./cmd/ateapi", "./cmd/atelet"}) { t.Errorf("args[:3] = %v, want [build ./cmd/ateapi ./cmd/atelet]", got) } - for _, want := range []string{"--tags=v1.2.3", "--base-import-paths"} { + for _, want := range []string{"--tags=v1.2.3", "--base-import-paths", "--image-label=org.example/source=https://example.com"} { if !slices.Contains(args, want) { t.Errorf("args = %v, want %s", args, want) } @@ -127,7 +170,7 @@ func TestBuildTaggedRejectsMissingRefs(t *testing.T) { binary, _ := fakeKo(t, "repo/ateapi:v1@sha256:a\\n") r := &Runner{Root: t.TempDir(), Stderr: os.Stderr, binary: binary} - if _, err := r.BuildTagged(t.Context(), "v1", "./cmd/ateapi", "./cmd/atelet"); err == nil { + if _, err := r.BuildTagged(t.Context(), "v1", nil, "./cmd/ateapi", "./cmd/atelet"); err == nil { t.Error("BuildTagged() error = nil, want one for 1 ref from 2 packages") } } diff --git a/cmd/ate-setup/internal/steps/csi.go b/cmd/ate-setup/internal/steps/csi.go index cb06345e9d..f42183257b 100644 --- a/cmd/ate-setup/internal/steps/csi.go +++ b/cmd/ate-setup/internal/steps/csi.go @@ -403,7 +403,7 @@ spec: - $patch: delete name: socat - name: envoy - image: envoyproxy/envoy:v1.34-latest + image: envoyproxy/envoy:v1.39.5@sha256:8fc594a5d85eb76dca24155cfb9208882406873932b20cf33649a0a611194c43 command: - /usr/local/bin/envoy args: diff --git a/cmd/ate-setup/internal/steps/env.go b/cmd/ate-setup/internal/steps/env.go index 8cfba859a7..6af68a5c06 100644 --- a/cmd/ate-setup/internal/steps/env.go +++ b/cmd/ate-setup/internal/steps/env.go @@ -117,6 +117,13 @@ func NewEnv(cfg *config.Config) (*Env, error) { return &Env{Cfg: cfg, Kube: client}, nil } +// NewLocalEnv returns an Env with no Kubernetes client, for the publish +// commands, which build images and binaries without a cluster. Kube is nil, so +// a step that uses it must not run under such an Env. +func NewLocalEnv(cfg *config.Config) *Env { + return &Env{Cfg: cfg} +} + // imageResolver returns the image resolver, creating it on first use. func (e *Env) imageResolver() (imageResolver, error) { if e.resolver != nil { @@ -165,8 +172,8 @@ func (e *Env) koRunner() (*ko.Runner, error) { // KO_DOCKER_REPO. A pre-built install builds nothing: --image-repo installs // images someone else published, and needs neither docker nor a registry to // push to, so the image is pinned from REPO/name:TAG like every ko image. -// That only works if the release published it there, which -// `ate-setup publish release-images` does. +// That only works if someone published it there: `ate-setup publish +// release-images` does not, so an install from release images fails here. func (e *Env) dockerfileImage(ctx context.Context, name, contextDir string) (string, error) { if ref, ok := e.dockerfileImages[name]; ok { return ref, nil @@ -186,8 +193,9 @@ func (e *Env) dockerfileImage(ctx context.Context, name, contextDir string) (str } ref, err = prebuilt.Pin(ctx, name) if err != nil { - return "", fmt.Errorf("%s is built from a Dockerfile, not with ko, so a release has to publish it "+ - "alongside the other images (ate-setup publish release-images): %w", name, err) + return "", fmt.Errorf("%s is built from a Dockerfile, and releases do not publish it; "+ + "build it from source by dropping --image-repo, or choose --atenet-dataplane=agentgateway, "+ + "which does not use it: %w", name, err) } } if err != nil { diff --git a/cmd/ate-setup/internal/steps/overlay_test.go b/cmd/ate-setup/internal/steps/overlay_test.go index 9faa884ee0..a626532e64 100644 --- a/cmd/ate-setup/internal/steps/overlay_test.go +++ b/cmd/ate-setup/internal/steps/overlay_test.go @@ -1133,8 +1133,10 @@ func TestApplyOtelEndpointOverride(t *testing.T) { } // A pre-built install renders the envoy router, egress, and system manifests -// without building anything: envoy-dataplane is pinned from the release like -// every ko image, so neither docker nor KO_DOCKER_REPO is needed. +// without building anything: envoy-dataplane is pinned from the image +// repository like every ko image, so neither docker nor KO_DOCKER_REPO is +// needed. Releases do not publish it, so this holds for a repository someone +// published it to. func TestRenderAtenetEgressManifestPrebuilt(t *testing.T) { const digest = "sha256:2222222222222222222222222222222222222222222222222222222222222222" src := images.Source{Repo: "example.com/substrate", Tag: "v1.2.3"} @@ -1186,9 +1188,9 @@ func TestRenderAtenetEgressManifestPrebuilt(t *testing.T) { } } -// A release that did not publish envoy-dataplane fails the install with a -// message naming the image and the command that publishes it, rather than a bare -// registry error. +// An install from release images that needs envoy-dataplane fails with a +// message saying releases do not publish it and how to proceed, rather than a +// bare registry error. func TestDockerfileImagePrebuiltNotPublished(t *testing.T) { src := images.Source{Repo: "example.com/substrate", Tag: "v1.2.3"} e := &Env{ @@ -1203,7 +1205,8 @@ func TestDockerfileImagePrebuiltNotPublished(t *testing.T) { t.Fatal("dockerfileImage() error = nil, want one") } for _, want := range []string{ - "ate-setup publish release-images", + "envoy-dataplane is built from a Dockerfile, and releases do not publish it", + "--atenet-dataplane=agentgateway", "resolving example.com/substrate/envoy-dataplane:v1.2.3 to a digest: MANIFEST_UNKNOWN", } { if !strings.Contains(err.Error(), want) { diff --git a/cmd/ate-setup/internal/steps/postgres_test.go b/cmd/ate-setup/internal/steps/postgres_test.go index d0022be87b..a9bfdc1cd3 100644 --- a/cmd/ate-setup/internal/steps/postgres_test.go +++ b/cmd/ate-setup/internal/steps/postgres_test.go @@ -97,7 +97,7 @@ func TestApplyPostgresSize10Overrides(t *testing.T) { cm := findObject(objs, "ConfigMap", "postgres-config") data, _, _ := unstructured.NestedStringMap(cm.Object, "data") - if !strings.Contains(data["postgresql.conf"], "max_connections = 300") { + if !strings.Contains(data["postgresql.conf"], "max_connections = 500") { t.Errorf("postgresql.conf was not replaced with the size10 tuning:\n%s", data["postgresql.conf"]) } // The merge semantics: the base's other keys survive untouched. diff --git a/cmd/ate-setup/internal/steps/publish.go b/cmd/ate-setup/internal/steps/publish.go index 52c5d264bc..cf36277fae 100644 --- a/cmd/ate-setup/internal/steps/publish.go +++ b/cmd/ate-setup/internal/steps/publish.go @@ -16,13 +16,114 @@ package steps import ( "context" + "crypto/sha256" + "encoding/hex" "fmt" "io" + "os" + "os/exec" + "path/filepath" + "strings" "go.ate.dev/substrate/cmd/ate-setup/internal/images" + "go.ate.dev/substrate/cmd/ate-setup/internal/ko" "go.ate.dev/substrate/cmd/ate-setup/internal/log" ) +// releaseBinary is the binary a release attaches. +const releaseBinary = "kubectl-ate" + +// releaseBinaryPlatforms returns the GOOS/GOARCH pairs releaseBinary is built +// for. Windows is not among them. +func releaseBinaryPlatforms() []string { + return []string{"darwin/amd64", "darwin/arm64", "linux/amd64", "linux/arm64"} +} + +// releaseBinaryName is the file name of the release binary for platform, e.g. +// kubectl-ate-linux-amd64. +func releaseBinaryName(platform string) string { + return releaseBinary + "-" + strings.ReplaceAll(platform, "/", "-") +} + +// PublishReleaseBinaries builds kubectl-ate for every releaseBinaryPlatforms +// entry into outDir, statically (CGO_ENABLED=0) and stamped with the release +// version, commit, and commit time, and writes beside each binary a .sha256 +// file that `shasum -a 256 -c` checks. It writes the paths it produced to w. +// The version has to be a release tag, and the checkout has to have a commit, +// since a binary without one reports the wrong version. +func (e *Env) PublishReleaseBinaries(ctx context.Context, w io.Writer, outDir string) error { + return e.publishReleaseBinaries(ctx, w, outDir, releaseBinaryPlatforms()) +} + +// publishReleaseBinaries is PublishReleaseBinaries for the given GOOS/GOARCH +// pairs. +func (e *Env) publishReleaseBinaries(ctx context.Context, w io.Writer, outDir string, platforms []string) error { + version := e.SubstrateVersion() + if err := images.CheckReleaseVersion(version); err != nil { + return fmt.Errorf("the build version: %w; set VERSION to the release tag", err) + } + if _, _, ok := ko.CommitStamp(e.Cfg.Root); !ok { + return fmt.Errorf("release binaries are stamped with the commit they were built from, and %s has none checked out", e.Cfg.Root) + } + log.Stepf("publish_release_binaries (%s)", version) + if err := os.MkdirAll(outDir, 0o755); err != nil { + return fmt.Errorf("while creating %s: %w", outDir, err) + } + ldflags := strings.Join(ko.Ldflags(e.Cfg.Root), " ") + var written []string + for _, platform := range platforms { + goos, goarch, _ := strings.Cut(platform, "/") + out, err := filepath.Abs(filepath.Join(outDir, releaseBinaryName(platform))) + if err != nil { + return err + } + build := exec.CommandContext(ctx, "go", "build", "-trimpath", "-ldflags="+ldflags, "-o", out, "./cmd/"+releaseBinary) + build.Dir = e.Cfg.Root + build.Env = append(os.Environ(), "CGO_ENABLED=0", "GOOS="+goos, "GOARCH="+goarch) + build.Stdout = os.Stderr + build.Stderr = os.Stderr + if err := build.Run(); err != nil { + return fmt.Errorf("while building %s for %s: %w", releaseBinary, platform, err) + } + sum, err := writeChecksum(out) + if err != nil { + return err + } + written = append(written, out, sum) + } + if _, err := fmt.Fprintf(w, "\nRelease binaries for %s:\n", version); err != nil { + return err + } + for _, path := range written { + if _, err := fmt.Fprintln(w, path); err != nil { + return err + } + } + return nil +} + +// writeChecksum writes path's SHA-256 to path.sha256 in the " " +// form `shasum -a 256 -c` and `sha256sum -c` read, naming the file without +// its directory so the check runs wherever both are downloaded. It returns +// the checksum file's path. +func writeChecksum(path string) (string, error) { + f, err := os.Open(path) + if err != nil { + return "", err + } + defer f.Close() + h := sha256.New() + if _, err := io.Copy(h, f); err != nil { + return "", fmt.Errorf("while hashing %s: %w", path, err) + } + sum := path + ".sha256" + line := hex.EncodeToString(h.Sum(nil)) + " " + filepath.Base(path) + "\n" + if err := os.WriteFile(sum, []byte(line), 0o644); err != nil { + return "", fmt.Errorf("while writing %s: %w", sum, err) + } + return sum, nil +} + // workerImages are the worker images a WorkerPool points at through // workerImage, one per sandbox class. No manifest references them, so the // install never publishes them as a side effect. @@ -59,35 +160,104 @@ func (e *Env) PublishWorkerImages(ctx context.Context, w io.Writer) error { return nil } -// PublishReleaseImages builds and pushes every image a pre-built install -// needs, all tagged with the build version, and writes their pushed references -// to w. That is what `deploy --image-repo KO_DOCKER_REPO --image-tag VERSION` -// installs: the ko images in images.Components, plus envoy-dataplane, which is -// built from a Dockerfile. -func (e *Env) PublishReleaseImages(ctx context.Context, w io.Writer) error { +// PublishReleaseOptions are the publish release-images settings. +type PublishReleaseOptions struct { + // Overwrite rebuilds and replaces images the registry already holds for + // the tag. Without it, a complete set is left alone. + Overwrite bool +} + +// releaseImageRefs returns the tagged reference of every image a release +// publishes to repo: one per images.ReleaseComponents entry. +func releaseImageRefs(repo, tag string) []string { + repo = strings.TrimSuffix(repo, "/") + pkgs := images.ReleaseComponents() + refs := make([]string, 0, len(pkgs)) + for _, pkg := range pkgs { + refs = append(refs, repo+"/"+images.ImageName(pkg)+":"+tag) + } + return refs +} + +// PublishReleaseImages builds and pushes every image a release publishes, the +// ko images in images.ReleaseComponents, all tagged with the build version, +// and writes their pushed references to w. The version has to be a release +// tag. Releases do not publish envoy-dataplane, the one Dockerfile-built +// image, so nothing here needs docker. +// +// If the registry already holds every image for the tag on every platform, +// nothing is built unless opts.Overwrite is set, so rerunning a release never +// replaces published digests by accident. An incomplete set, left by a failed +// run, is rebuilt in full. After pushing, every tag is checked to name the +// digest just pushed, on every platform. +func (e *Env) PublishReleaseImages(ctx context.Context, w io.Writer, opts PublishReleaseOptions) error { if e.Cfg.KODockerRepo == "" { return fmt.Errorf("publishing release images needs a registry to push to; set KO_DOCKER_REPO or --ko-docker-repo") } tag := e.SubstrateVersion() + if err := images.CheckReleaseTag(e.Cfg.KODockerRepo, tag); err != nil { + return fmt.Errorf("the build version: %w; set VERSION to the release tag", err) + } log.Stepf("publish_release_images (%s)", tag) + platforms, err := images.Platforms(e.Cfg.Root, e.Cfg.KODefaultPlatforms) + if err != nil { + return err + } + tagged := releaseImageRefs(e.Cfg.KODockerRepo, tag) + if !opts.Overwrite { + incomplete, err := images.CheckPublished(ctx, tagged, platforms) + if err != nil { + return err + } + if len(incomplete) == 0 { + log.Infof("%s already holds every image for %s; nothing built. Pass --overwrite to rebuild and replace them.", + e.Cfg.KODockerRepo, tag) + return nil + } + for _, i := range incomplete { + log.Infof("to build: %s", i) + } + } + runner, err := e.koRunner() if err != nil { return err } - pkgs := make([]string, 0, len(images.Components)) - for _, pkg := range images.Components { + var pkgs []string + for _, pkg := range images.ReleaseComponents() { pkgs = append(pkgs, "./"+pkg) } - refs, err := runner.BuildTagged(ctx, tag, pkgs...) + revision, _, ok := ko.CommitStamp(e.Cfg.Root) + if !ok { + return fmt.Errorf("release images are labeled with the commit they were built from, and %s has none checked out", e.Cfg.Root) + } + labels := []string{ + images.SourceLabel + "=" + images.SourceURL, + images.RevisionLabel + "=" + revision, + } + refs, err := runner.BuildTagged(ctx, tag, labels, pkgs...) if err != nil { return err } - envoy, err := images.PublishDockerfileImage(ctx, e.Cfg.Root, e.Cfg.KODockerRepo, envoyDataplaneImage, tag, - e.Cfg.Path(envoyDataplaneDockefile), e.Cfg.KODefaultPlatforms, e.Cfg.DockerBuildFlags) + + incomplete, err := images.CheckPublished(ctx, tagged, platforms) if err != nil { return err } - refs = append(refs, envoy) + if len(incomplete) > 0 { + lines := make([]string, 0, len(incomplete)) + for _, i := range incomplete { + lines = append(lines, i.String()) + } + return fmt.Errorf("after publishing, %d image(s) are incomplete:\n %s", len(incomplete), strings.Join(lines, "\n ")) + } + stale, err := images.StaleTags(ctx, tagged, refs) + if err != nil { + return err + } + if len(stale) > 0 { + return fmt.Errorf("after publishing, %d tag(s) do not name the pushed image:\n %s", len(stale), strings.Join(stale, "\n ")) + } if _, err := fmt.Fprintf(w, "\nRelease images for %s:\n", tag); err != nil { return err } diff --git a/cmd/ate-setup/internal/steps/publish_test.go b/cmd/ate-setup/internal/steps/publish_test.go index f47a47549a..90288247a9 100644 --- a/cmd/ate-setup/internal/steps/publish_test.go +++ b/cmd/ate-setup/internal/steps/publish_test.go @@ -15,12 +15,29 @@ package steps import ( + "crypto/sha256" + "encoding/hex" "io" + "log" + "net/http/httptest" + "os" + "os/exec" + "path/filepath" + "runtime" + "slices" "strings" "testing" + "github.com/google/go-containerregistry/pkg/name" + "github.com/google/go-containerregistry/pkg/registry" + v1 "github.com/google/go-containerregistry/pkg/v1" + "github.com/google/go-containerregistry/pkg/v1/empty" + "github.com/google/go-containerregistry/pkg/v1/mutate" + "github.com/google/go-containerregistry/pkg/v1/remote" + "go.ate.dev/substrate/cmd/ate-setup/internal/config" "go.ate.dev/substrate/cmd/ate-setup/internal/images" + "go.ate.dev/substrate/cmd/ate-setup/internal/ko" ) // TestKoRunnerPrebuilt covers the one resolver that cannot build. A pre-built @@ -48,8 +65,160 @@ func TestKoRunnerPrebuilt(t *testing.T) { func TestPublishReleaseImagesNeedsARegistry(t *testing.T) { e := &Env{Cfg: &config.Config{Root: t.TempDir()}} - err := e.PublishReleaseImages(t.Context(), io.Discard) + err := e.PublishReleaseImages(t.Context(), io.Discard, PublishReleaseOptions{}) if err == nil || !strings.Contains(err.Error(), "KO_DOCKER_REPO") { t.Errorf("PublishReleaseImages() error = %v, want one naming KO_DOCKER_REPO", err) } } + +// Rerunning a release must not replace images that are already published, so +// a complete set is left alone. The Root has no ko tool, so reaching the build +// would fail this test. +func TestPublishReleaseImagesSkipsACompleteRelease(t *testing.T) { + srv := httptest.NewServer(registry.New(registry.Logger(log.New(io.Discard, "", 0)))) + t.Cleanup(srv.Close) + repo := strings.TrimPrefix(srv.URL, "http://") + "/substrate" + t.Setenv("VERSION", "v1.2.3") + + index := mutate.AppendManifests(empty.Index, + mutate.IndexAddendum{Add: empty.Image, Descriptor: v1.Descriptor{Platform: &v1.Platform{OS: "linux", Architecture: "amd64"}}}, + mutate.IndexAddendum{Add: empty.Image, Descriptor: v1.Descriptor{Platform: &v1.Platform{OS: "linux", Architecture: "arm64"}}}, + ) + for _, ref := range releaseImageRefs(repo, "v1.2.3") { + parsed, err := name.ParseReference(ref) + if err != nil { + t.Fatal(err) + } + if err := remote.WriteIndex(parsed, index); err != nil { + t.Fatal(err) + } + } + + e := &Env{Cfg: &config.Config{ + Root: t.TempDir(), + KODockerRepo: repo, + KODefaultPlatforms: "linux/amd64,linux/arm64", + }} + if err := e.PublishReleaseImages(t.Context(), io.Discard, PublishReleaseOptions{}); err != nil { + t.Errorf("PublishReleaseImages() error = %v, want nil with every image published", err) + } +} + +// A stray tag or a value ko would split publishes nothing, with --overwrite as +// well as without: the check runs before the registry is read or ko is +// located, and the Root has no ko tool. +func TestPublishReleaseImagesRejectsANonReleaseTag(t *testing.T) { + for _, version := range []string{"v1-test", "v1.2.3,latest", "dev"} { + t.Run(version, func(t *testing.T) { + t.Setenv("VERSION", version) + e := &Env{Cfg: &config.Config{Root: t.TempDir(), KODockerRepo: "example.com/substrate"}} + err := e.PublishReleaseImages(t.Context(), io.Discard, PublishReleaseOptions{Overwrite: true}) + if err == nil || !strings.Contains(err.Error(), "not a release tag") { + t.Errorf("PublishReleaseImages() error = %v, want one saying %q is not a release tag", err, version) + } + }) + } +} + +// The release covers every released component, each under the name and tag +// that `deploy --image-repo --image-tag` looks up, and nothing releases leave +// out. +func TestReleaseImageRefs(t *testing.T) { + refs := releaseImageRefs("example.com/substrate/", "v1") + if want := len(images.ReleaseComponents()); len(refs) != want { + t.Fatalf("releaseImageRefs() returned %d refs, want %d", len(refs), want) + } + for _, want := range []string{"example.com/substrate/ateapi:v1", "example.com/substrate/objectstore-plugin:v1"} { + if !slices.Contains(refs, want) { + t.Errorf("releaseImageRefs() = %v, want %s", refs, want) + } + } + for _, unwanted := range []string{ + "example.com/substrate/counter:v1", + "example.com/substrate/kubernetes-secrets:v1", + "example.com/substrate/envoy-dataplane:v1", + } { + if slices.Contains(refs, unwanted) { + t.Errorf("releaseImageRefs() = %v, want no %s", refs, unwanted) + } + } +} + +// The release names each binary for its platform, and publishes no Windows +// build. +func TestReleaseBinaryNames(t *testing.T) { + var got []string + for _, platform := range releaseBinaryPlatforms() { + got = append(got, releaseBinaryName(platform)) + } + want := []string{"kubectl-ate-darwin-amd64", "kubectl-ate-darwin-arm64", "kubectl-ate-linux-amd64", "kubectl-ate-linux-arm64"} + if !slices.Equal(got, want) { + t.Errorf("release binaries = %v, want %v", got, want) + } +} + +// The checksum file names the binary without its directory, in the form +// `shasum -a 256 -c` reads, so the check works wherever both are downloaded. +func TestWriteChecksum(t *testing.T) { + path := filepath.Join(t.TempDir(), "kubectl-ate-linux-amd64") + if err := os.WriteFile(path, []byte("binary"), 0o755); err != nil { + t.Fatal(err) + } + sum, err := writeChecksum(path) + if err != nil { + t.Fatalf("writeChecksum() error = %v", err) + } + if sum != path+".sha256" { + t.Errorf("writeChecksum() = %q, want %q", sum, path+".sha256") + } + got, err := os.ReadFile(sum) + if err != nil { + t.Fatal(err) + } + digest := sha256.Sum256([]byte("binary")) + if want := hex.EncodeToString(digest[:]) + " kubectl-ate-linux-amd64\n"; string(got) != want { + t.Errorf("checksum file = %q, want %q", got, want) + } +} + +// A binary for a version that is not a release tag is never built. +func TestPublishReleaseBinariesRejectsANonReleaseTag(t *testing.T) { + t.Setenv("VERSION", "dev") + e := &Env{Cfg: &config.Config{Root: t.TempDir()}} + err := e.PublishReleaseBinaries(t.Context(), io.Discard, t.TempDir()) + if err == nil || !strings.Contains(err.Error(), "not a release tag") { + t.Errorf("PublishReleaseBinaries() error = %v, want one saying dev is not a release tag", err) + } +} + +// A release binary reports the release version and the commit it was built +// from. internal/version discards a stamped Version without a stamped Commit, +// which is what this guards. Only the host platform is built, to keep the test +// to one compile. +func TestPublishReleaseBinaries(t *testing.T) { + root := repoRoot(t) + commit, _, ok := ko.CommitStamp(root) + if !ok { + t.Fatalf("%s has no commit checked out", root) + } + platform := runtime.GOOS + "/" + runtime.GOARCH + t.Setenv("VERSION", "v1.2.3") + + out := t.TempDir() + e := &Env{Cfg: &config.Config{Root: root}} + if err := e.publishReleaseBinaries(t.Context(), io.Discard, out, []string{platform}); err != nil { + t.Fatalf("publishReleaseBinaries() error = %v", err) + } + + binary := filepath.Join(out, releaseBinaryName(platform)) + if _, err := os.Stat(binary + ".sha256"); err != nil { + t.Errorf("no checksum beside the binary: %v", err) + } + version, err := exec.Command(binary, "--version").CombinedOutput() + if err != nil { + t.Fatalf("%s --version: %v\n%s", binary, err, version) + } + if want := "version v1.2.3 commit=" + commit; !strings.Contains(string(version), want) { + t.Errorf("%s --version = %q, want it to contain %q", binary, version, want) + } +} diff --git a/cmd/ateapi/internal/authz/registry.go b/cmd/ateapi/internal/authz/registry.go index 3c8affb82f..741e716bbd 100644 --- a/cmd/ateapi/internal/authz/registry.go +++ b/cmd/ateapi/internal/authz/registry.go @@ -333,5 +333,5 @@ var defaultRPCPermissions = map[string]rpcRule{ ateapi.WorkerService_RequestActorSuspend_FullMethodName: inHandler("ateletauth: caller is atelet on the node of the Worker hosting the Actor"), ateapi.WorkerService_MintWorkerActorCertificate_FullMethodName: inHandler("ateletauth: caller is atelet on the node of the Worker the Actor is assigned to"), ateapi.WorkerService_MintWorkerActorJwt_FullMethodName: inHandler("ateletauth: caller is atelet on the node hosting the Actor"), - ateapi.WorkerService_ReportActorSnapshotState_FullMethodName: inHandler("ateletauth: caller is atelet on the node the Actor is placed on or holds a local snapshot of it"), + ateapi.WorkerService_ReportActorSnapshotState_FullMethodName: inHandler("ateletauth: caller is atelet on the node the Actor is placed on"), } diff --git a/cmd/ateapi/internal/controlapi/actor.go b/cmd/ateapi/internal/controlapi/actor.go index ecdef8e9f9..ffee8080ea 100644 --- a/cmd/ateapi/internal/controlapi/actor.go +++ b/cmd/ateapi/internal/controlapi/actor.go @@ -257,6 +257,12 @@ func (s *RPCService) UpdateActor(ctx context.Context, req *ateapi.UpdateActorReq actorRef := resources.ActorRefFromActor(inActor) setSpanActorRefAttributes(ctx, actorRef) + ctx, lease, err := acquireLease(ctx, s.impl, actorLeaseKey(actorRef), "actor") + if err != nil { + return nil, err + } + defer lease.Close() + storedActor, err := s.impl.UpdateActor(ctx, actorRef, store.PreconditionFrom(inActor), func(toUpdate *ateapi.Actor) error { // Status and Metadata are server-owned fields. status, metadata := toUpdate.GetStatus(), toUpdate.GetMetadata() diff --git a/cmd/ateapi/internal/controlapi/actor_test.go b/cmd/ateapi/internal/controlapi/actor_test.go index d6296b1f10..29b4057422 100644 --- a/cmd/ateapi/internal/controlapi/actor_test.go +++ b/cmd/ateapi/internal/controlapi/actor_test.go @@ -139,6 +139,9 @@ func TestUpdateActor(t *testing.T) { req *ateapi.Actor want *ateapi.Actor wantCode codes.Code + // leaseConflict holds the actor lease across the update, + // as a running workflow does. + leaseConflict bool }{ { name: "sets a worker_selector the stored actor does not have", @@ -206,6 +209,16 @@ func TestUpdateActor(t *testing.T) { }, wantCode: codes.InvalidArgument, }, + { + name: "refused while another operation holds the actor", + stored: &ateapi.Actor{}, + req: &ateapi.Actor{ + ActorTemplate: &ateapi.ObjectRef{Atespace: templateNS, Name: templateName}, + WorkerSelector: &ateapi.Selector{MatchLabels: map[string]string{"tier": "paid"}}, + }, + leaseConflict: true, + wantCode: codes.Aborted, + }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -216,6 +229,15 @@ func TestUpdateActor(t *testing.T) { } svc, created := rpcServiceWithActor(t, tt.stored) + actorRef := resources.ActorRefFromActor(created) + + if tt.leaseConflict { + lease, err := svc.impl.AcquireLease(context.Background(), actorLeaseKey(actorRef)) + if err != nil { + t.Fatalf("AcquireLease() failed: %v", err) + } + defer lease.Close() + } tt.req.Metadata = created.GetMetadata() updated, err := svc.UpdateActor(context.Background(), &ateapi.UpdateActorRequest{Actor: tt.req}) @@ -224,6 +246,13 @@ func TestUpdateActor(t *testing.T) { if code := apierror.Code(err); code != tt.wantCode { t.Errorf("UpdateActor error = %v (code %v), want code %v", err, code, tt.wantCode) } + got, err := svc.impl.GetActor(context.Background(), actorRef) + if err != nil { + t.Fatal(err) + } + if got.GetMetadata().GetVersion() != created.GetMetadata().GetVersion() { + t.Errorf("a refused update wrote the actor: version %d, want %d", got.GetMetadata().GetVersion(), created.GetMetadata().GetVersion()) + } return } if err != nil { diff --git a/cmd/ateapi/internal/controlapi/snapshot_report.go b/cmd/ateapi/internal/controlapi/snapshot_report.go index 286b425ef7..b03b5ece68 100644 --- a/cmd/ateapi/internal/controlapi/snapshot_report.go +++ b/cmd/ateapi/internal/controlapi/snapshot_report.go @@ -37,9 +37,9 @@ func (s *RPCService) ApplyActorSnapshotState(ctx context.Context, actorRef resou } // ApplyReportedSnapshotState applies a report from the atelet on node to the -// Actor. It holds the Actor's lease and applies reports only to a SUSPENDED or -// RUNNING Actor, so a report never interleaves with a workflow. A report that -// changes nothing writes nothing. +// Actor. It holds the Actor's lease and applies reports only to a SUSPENDED, +// RUNNING, or CRASHED Actor, so a report never interleaves with a workflow. A +// report that changes nothing writes nothing. func (w *ActorWorkflow) ApplyReportedSnapshotState(ctx context.Context, actorRef resources.ActorRef, actorUID, node string, state *ateapi.ActorSnapshotState) error { ctx = preview.ContextWithOptIn(ctx) // Check before taking the lease, so an atelet can't hold the lease of an @@ -58,11 +58,9 @@ func (w *ActorWorkflow) ApplyReportedSnapshotState(ctx context.Context, actorRef } // A workflow owns a transitioning Actor's snapshots, even after its lease - // lapses. A crashed Actor's snapshots wait for a revert. + // lapses. switch st := before.GetStatus().GetState(); st { - case ateapi.ActorState_ACTOR_STATE_SUSPENDED, ateapi.ActorState_ACTOR_STATE_RUNNING: - case ateapi.ActorState_ACTOR_STATE_CRASHED: - return apierror.FailedPrecondition("actor %s is %s; revert it first", actorRef, st) + case ateapi.ActorState_ACTOR_STATE_SUSPENDED, ateapi.ActorState_ACTOR_STATE_RUNNING, ateapi.ActorState_ACTOR_STATE_CRASHED: case ateapi.ActorState_ACTOR_STATE_SUSPENDING, ateapi.ActorState_ACTOR_STATE_RESUMING, ateapi.ActorState_ACTOR_STATE_DELETING, ateapi.ActorState_ACTOR_STATE_REVERTING: return apierror.Aborted("actor %s is %s; retry once that finishes", actorRef, st) @@ -118,26 +116,16 @@ func (w *ActorWorkflow) loadReportedActor(ctx context.Context, actorRef resource return actor, nil } -// actorOnNode reports whether the Actor is placed on node or has a local -// snapshot there. Unlike HostingNode, it includes a node that holds only -// older snapshots of an Actor placed elsewhere, since that node's atelet -// reports them. +// actorOnNode reports whether the Actor is placed on node. func actorOnNode(actor *ateapi.Actor, node string) bool { - if actor.GetStatus().GetAssignedNode() == node { - return true - } - for _, snap := range actor.GetStatus().GetSnapshots() { - if snap.GetLocality() == node { - return true - } - } - return false + return actor.GetStatus().GetAssignedNode() == node } // Messages logged for a reported durable snapshot that changes nothing. const ( - msgDurableOlder = "Ignoring a durable snapshot older than the latest completed one" - msgDurableNotRecorded = "Ignoring a durable snapshot at a generation the actor no longer records" + msgDurableOlder = "Ignoring a durable snapshot older than the latest completed one" + msgDurableNotRecorded = "Ignoring a durable snapshot at a generation the actor no longer records" + msgDurableCrashedSuspend = "Ignoring the upload of a snapshot whose suspend crashed" ) // applyActorSnapshotState applies the two facts a report carries: an upload @@ -169,8 +157,10 @@ func applyAteletResponseState(ctx context.Context, status *ateapi.ActorStatus, s // // A late report changes nothing and is logged at Info. A report is late when // its generation is older than the latest COMPLETED one, or is no longer in -// status.snapshots, for example because a revert removed it. A repeat of the -// latest COMPLETED generation also changes nothing, and isn't logged. +// status.snapshots, for example because a revert removed it. A CRASHED +// Actor's report of a snapshot whose suspend crashed is logged the same way. A +// repeat of the latest COMPLETED generation also changes nothing, and isn't +// logged. func applyLatestDurable(ctx context.Context, status *ateapi.ActorStatus, reported *ateapi.SnapshotReport) error { if reported == nil { return nil @@ -207,6 +197,16 @@ func applyLatestDurable(ctx context.Context, status *ateapi.ActorStatus, reporte if reported.GetFidelity() == ateapi.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED { return apierror.InvalidArgument("snapshot generation %d reports no fidelity", reported.GetGeneration()) } + // A crashed Actor keeps its AssignedNode, so only Locality tells a crashed + // suspend's snapshot from an upload that finished on node: only a finalized + // suspend sets Locality. Its upload is not recorded, and a revert deletes + // it with the rest of the discarded execution. + if status.GetState() == ateapi.ActorState_ACTOR_STATE_CRASHED && snap.GetLocality() == "" { + slog.InfoContext(ctx, msgDurableCrashedSuspend, + slog.Int("generation", int(reported.GetGeneration())), + slog.String("uuid", reported.GetUuid())) + return nil + } snap.Fidelity = reported.GetFidelity() snap.DurableSnapshot.Status = ateapi.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED return nil @@ -217,10 +217,11 @@ func applyLatestDurable(ctx context.Context, status *ateapi.ActorStatus, reporte // durable. A newer snapshot on node postdates the detach, so it stays. A // SUSPENDED Actor is released from node once none of its snapshots are there. // It refuses if the latest deleted snapshot isn't durable, so the Actor can't -// resume from an older one. +// resume from an older one. A CRASHED Actor resumes only after a revert, which +// drops its local snapshots and pending uploads, so it is not refused. func applyDetach(status *ateapi.ActorStatus, node string, upTo int32) error { deleted := func(s *ateapi.Snapshot) bool { return s.GetLocality() == node && s.GetGeneration() <= upTo } - if latest := latestSnapshot(status, deleted); latest != nil && + if latest := latestSnapshot(status, deleted); status.GetState() != ateapi.ActorState_ACTOR_STATE_CRASHED && latest != nil && latest.GetDurableSnapshot().GetStatus() != ateapi.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED { return apierror.FailedPrecondition("snapshot generation %d on node %s is not durable", latest.GetGeneration(), node) } diff --git a/cmd/ateapi/internal/controlapi/snapshot_report_test.go b/cmd/ateapi/internal/controlapi/snapshot_report_test.go index bdb1049ba8..2536dde820 100644 --- a/cmd/ateapi/internal/controlapi/snapshot_report_test.go +++ b/cmd/ateapi/internal/controlapi/snapshot_report_test.go @@ -319,10 +319,10 @@ func TestApplyActorSnapshotState(t *testing.T) { } } -// A reported durable snapshot older than the latest COMPLETED one, or at a -// generation status no longer has, is logged at Info, once, with the -// generation it named. A repeat of the latest, and one that is applied, aren't -// logged. +// A reported durable snapshot older than the latest COMPLETED one, at a +// generation status no longer has, or of a CRASHED actor's crashed suspend, is +// logged at Info, once, with the generation it named. A repeat of the latest, +// and one that is applied, aren't logged. func TestApplyLatestDurableLogsStaleReports(t *testing.T) { const ( pending = ateapi.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING @@ -351,6 +351,15 @@ func TestApplyLatestDurableLogsStaleReports(t *testing.T) { gen: 2, wantMsg: msgDurableNotRecorded, wantAttrs: map[string]string{"generation": "2", "uuid": snapName(2)}, + }, { + name: "a CRASHED actor's crashed suspend", + status: &ateapi.ActorStatus{State: ateapi.ActorState_ACTOR_STATE_CRASHED, LastAssignedGeneration: 2, Snapshots: []*ateapi.Snapshot{ + withDurable(t, 1, reportNode, completed), + withDurable(t, 2, "", pending), + }}, + gen: 2, + wantMsg: msgDurableCrashedSuspend, + wantAttrs: map[string]string{"generation": "2", "uuid": snapName(2)}, }, { name: "the latest COMPLETED generation itself", status: &ateapi.ActorStatus{LastAssignedGeneration: 2, Snapshots: []*ateapi.Snapshot{ @@ -373,7 +382,7 @@ func TestApplyLatestDurableLogsStaleReports(t *testing.T) { var got []logged prev := slog.Default() slog.SetDefault(slog.New(slogHandlerFunc(func(r slog.Record) { - if r.Message != msgDurableOlder && r.Message != msgDurableNotRecorded { + if r.Message != msgDurableOlder && r.Message != msgDurableNotRecorded && r.Message != msgDurableCrashedSuspend { return } l := logged{msg: r.Message, attrs: map[string]string{}} @@ -468,26 +477,22 @@ func TestApplyReportedSnapshotState(t *testing.T) { assertUnwritten(t) }) } - // A transitioning actor gets Aborted, and a crashed one FailedPrecondition. - for _, tc := range []struct { - state ateapi.ActorState - want codes.Code - }{ - {ateapi.ActorState_ACTOR_STATE_SUSPENDING, codes.Aborted}, - {ateapi.ActorState_ACTOR_STATE_RESUMING, codes.Aborted}, - {ateapi.ActorState_ACTOR_STATE_DELETING, codes.Aborted}, - {ateapi.ActorState_ACTOR_STATE_REVERTING, codes.Aborted}, - {ateapi.ActorState_ACTOR_STATE_CRASHED, codes.FailedPrecondition}, + // A transitioning actor gets Aborted. + for _, st := range []ateapi.ActorState{ + ateapi.ActorState_ACTOR_STATE_SUSPENDING, + ateapi.ActorState_ACTOR_STATE_RESUMING, + ateapi.ActorState_ACTOR_STATE_DELETING, + ateapi.ActorState_ACTOR_STATE_REVERTING, } { - t.Run("refused for a "+tc.state.String()+" actor", func(t *testing.T) { - actor = mustUpdateActorStatus(t, ctx, persistence, actor, func(s *ateapi.ActorStatus) { s.State = tc.state }) + t.Run("refused for a "+st.String()+" actor", func(t *testing.T) { + actor = mustUpdateActorStatus(t, ctx, persistence, actor, func(s *ateapi.ActorStatus) { s.State = st }) defer func() { actor = mustUpdateActorStatus(t, ctx, persistence, actor, func(s *ateapi.ActorStatus) { s.State = ateapi.ActorState_ACTOR_STATE_SUSPENDED }) }() - if err := w.ApplyReportedSnapshotState(ctx, actorRef, uid, reportNode, state); apierror.Code(err) != tc.want { - t.Fatalf("ApplyReportedSnapshotState() = %v, want %v", err, tc.want) + if err := w.ApplyReportedSnapshotState(ctx, actorRef, uid, reportNode, state); apierror.Code(err) != codes.Aborted { + t.Fatalf("ApplyReportedSnapshotState() = %v, want Aborted", err) } assertUnwritten(t) }) @@ -500,23 +505,96 @@ func TestApplyReportedSnapshotState(t *testing.T) { assertUnwritten(t) }) - if err := w.ApplyReportedSnapshotState(ctx, actorRef, uid, reportNode, state); err != nil { - t.Fatalf("ApplyReportedSnapshotState() failed: %v", err) - } - got, err := persistence.GetActor(ctx, actorRef) - if err != nil { - t.Fatal(err) - } - if st := got.GetStatus().GetSnapshots()[0].GetDurableSnapshot().GetStatus(); st != ateapi.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED { - t.Errorf("durable snapshot status = %v, want COMPLETED", st) + for _, st := range []ateapi.ActorState{ateapi.ActorState_ACTOR_STATE_SUSPENDED, ateapi.ActorState_ACTOR_STATE_CRASHED} { + t.Run("applied to a "+st.String()+" actor", func(t *testing.T) { + actor = mustUpdateActorStatus(t, ctx, persistence, actor, func(s *ateapi.ActorStatus) { + s.State = st + s.Snapshots[0].DurableSnapshot.Status = ateapi.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING + }) + if err := w.ApplyReportedSnapshotState(ctx, actorRef, uid, reportNode, state); err != nil { + t.Fatalf("ApplyReportedSnapshotState() failed: %v", err) + } + got, err := persistence.GetActor(ctx, actorRef) + if err != nil { + t.Fatal(err) + } + if st := got.GetStatus().GetSnapshots()[0].GetDurableSnapshot().GetStatus(); st != ateapi.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED { + t.Errorf("durable snapshot status = %v, want COMPLETED", st) + } + + // A report that changes nothing writes nothing. + actor = got + if err := w.ApplyReportedSnapshotState(ctx, actorRef, uid, reportNode, state); err != nil { + t.Fatalf("repeated ApplyReportedSnapshotState() failed: %v", err) + } + assertUnwritten(t) + }) } - // A report that changes nothing writes nothing. - actor = got - if err := w.ApplyReportedSnapshotState(ctx, actorRef, uid, reportNode, state); err != nil { - t.Fatalf("repeated ApplyReportedSnapshotState() failed: %v", err) + // A suspend that crashed before it finalized leaves its snapshot on no node. + older := mustActorSnapshotURI(t, tmpl, actor, "snap-1") + snapshot := func(gen int32, uri resources.SnapshotURI, locality string, storage ateapi.SnapshotStorageStatus) *ateapi.Snapshot { + snap := newDurableSnapshot(gen, ateapi.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapi.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", uri.Name(), uri.String(), storage) + snap.Locality = locality + return snap + } + for _, tc := range []struct { + name string + snapshots []*ateapi.Snapshot + state *ateapi.ActorSnapshotState + want []*ateapi.Snapshot + }{{ + name: "a crashed suspend's upload stays pending", + snapshots: []*ateapi.Snapshot{ + snapshot(1, older, reportNode, ateapi.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED), + snapshot(2, pending, "", ateapi.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING), + }, + state: state, + want: []*ateapi.Snapshot{ + snapshot(1, older, reportNode, ateapi.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED), + snapshot(2, pending, "", ateapi.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING), + }, + }, { + name: "a crashed suspend's detach releases the older snapshot", + snapshots: []*ateapi.Snapshot{ + snapshot(1, older, reportNode, ateapi.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED), + snapshot(2, pending, "", ateapi.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING), + }, + state: &ateapi.ActorSnapshotState{LatestDurable: uploaded}, + want: []*ateapi.Snapshot{ + snapshot(1, older, "", ateapi.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED), + snapshot(2, pending, "", ateapi.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING), + }, + }, { + name: "a crashed suspend's detach releases an older snapshot that isn't durable", + snapshots: []*ateapi.Snapshot{ + snapshot(1, older, reportNode, ateapi.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING), + snapshot(2, pending, "", ateapi.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING), + }, + state: &ateapi.ActorSnapshotState{LatestDurable: uploaded}, + want: []*ateapi.Snapshot{ + snapshot(1, older, "", ateapi.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING), + snapshot(2, pending, "", ateapi.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING), + }, + }} { + t.Run(tc.name, func(t *testing.T) { + actor = mustUpdateActorStatus(t, ctx, persistence, actor, func(s *ateapi.ActorStatus) { + s.State = ateapi.ActorState_ACTOR_STATE_CRASHED + s.Snapshots = tc.snapshots + }) + if err := w.ApplyReportedSnapshotState(ctx, actorRef, uid, reportNode, tc.state); err != nil { + t.Fatalf("ApplyReportedSnapshotState() failed: %v", err) + } + got, err := persistence.GetActor(ctx, actorRef) + if err != nil { + t.Fatal(err) + } + if diff := cmp.Diff(tc.want, got.GetStatus().GetSnapshots(), protocmp.Transform()); diff != "" { + t.Errorf("snapshots mismatch (-want +got):\n%s", diff) + } + actor = got + }) } - assertUnwritten(t) } func TestActorOnNode(t *testing.T) { @@ -533,7 +611,7 @@ func TestActorOnNode(t *testing.T) { want bool }{ {"placed on the node", actor(reportNode), true}, - {"holds a snapshot on the node", actor(reportOther, "", reportNode), true}, + {"holds only a snapshot on the node", actor(reportOther, "", reportNode), false}, {"neither", actor(reportOther, reportOther), false}, } { t.Run(tc.name, func(t *testing.T) { diff --git a/cmd/ateapi/internal/store/atepg/migrations/000001_initial.sql b/cmd/ateapi/internal/store/atepg/migrations/000001_initial.sql index 66ebbee243..77b026e21f 100644 --- a/cmd/ateapi/internal/store/atepg/migrations/000001_initial.sql +++ b/cmd/ateapi/internal/store/atepg/migrations/000001_initial.sql @@ -1,4 +1,4 @@ --- Copyright 2026 Google LLC +-- Copyright 2026 Google LLC and The OpenFGA Authors -- -- Licensed under the Apache License, Version 2.0 (the "License"); -- you may not use this file except in compliance with the License. @@ -124,3 +124,97 @@ CREATE TABLE leases ( ); CREATE INDEX leases_expires_at_idx ON leases (expires_at); + +-- OpenFGA storage schema (pinned to github.com/openfga/openfga PostgreSQL +-- migration version 6). Kept in the same schema and Goose migration directory +-- as Substrate tables so resource mutations and authorization tuple updates +-- execute within the same PostgreSQL schema and transaction. + +CREATE TABLE tuple ( + store TEXT NOT NULL, + object_type TEXT NOT NULL, + object_id TEXT NOT NULL, + relation TEXT NOT NULL, + _user TEXT NOT NULL, + user_type TEXT NOT NULL, + ulid TEXT NOT NULL, + inserted_at TIMESTAMPTZ NOT NULL, + condition_name TEXT, + condition_context BYTEA, + PRIMARY KEY (store, object_type, object_id, relation, _user) +); + +CREATE INDEX idx_tuple_partial_user + ON tuple (store, object_type, object_id, relation, _user) + WHERE user_type = 'user'; + +CREATE INDEX idx_tuple_partial_userset + ON tuple (store, object_type, object_id, relation, _user) + WHERE user_type = 'userset'; + +CREATE UNIQUE INDEX idx_tuple_ulid ON tuple (ulid); + +CREATE INDEX idx_user_lookup ON tuple ( + store, + _user, + relation, + object_type, + object_id COLLATE "C" +); + +CREATE TABLE authorization_model ( + store TEXT NOT NULL, + authorization_model_id TEXT NOT NULL, + type TEXT NOT NULL, + type_definition BYTEA, + schema_version TEXT NOT NULL DEFAULT '1.0', + serialized_protobuf BYTEA, + PRIMARY KEY (store, authorization_model_id, type) +); + +CREATE TABLE store ( + id TEXT PRIMARY KEY, + name TEXT NOT NULL, + created_at TIMESTAMPTZ NOT NULL, + updated_at TIMESTAMPTZ, + deleted_at TIMESTAMPTZ +); + +CREATE TABLE assertion ( + store TEXT NOT NULL, + authorization_model_id TEXT NOT NULL, + assertions BYTEA, + PRIMARY KEY (store, authorization_model_id) +); + +CREATE TABLE changelog ( + store TEXT NOT NULL, + object_type TEXT NOT NULL, + object_id TEXT NOT NULL, + relation TEXT NOT NULL, + _user TEXT NOT NULL, + operation INTEGER NOT NULL, + ulid TEXT NOT NULL, + inserted_at TIMESTAMPTZ NOT NULL, + condition_name TEXT, + condition_context BYTEA, + PRIMARY KEY (store, ulid, object_type) +); + +-- Singleton: the id column only admits true, so the table holds at most one row. +CREATE TABLE global_access_policy ( + id boolean PRIMARY KEY DEFAULT true CHECK (id), + uid text NOT NULL, + version bigint NOT NULL, + proto bytea NOT NULL +); + +CREATE TABLE atespace_access_policies ( + atespace_name text NOT NULL, + uid text NOT NULL, + version bigint NOT NULL, + proto bytea NOT NULL, + PRIMARY KEY (atespace_name), + FOREIGN KEY (atespace_name) + REFERENCES atespaces(name) ON DELETE CASCADE +); diff --git a/cmd/ateapi/internal/store/atepg/migrations/000002_openfga.sql b/cmd/ateapi/internal/store/atepg/migrations/000002_openfga.sql deleted file mode 100644 index 9d6234e891..0000000000 --- a/cmd/ateapi/internal/store/atepg/migrations/000002_openfga.sql +++ /dev/null @@ -1,92 +0,0 @@ --- Copyright 2026 Google LLC and The OpenFGA Authors --- --- Licensed under the Apache License, Version 2.0 (the "License"); --- you may not use this file except in compliance with the License. --- You may obtain a copy of the License at --- --- http://www.apache.org/licenses/LICENSE-2.0 --- --- Unless required by applicable law or agreed to in writing, software --- distributed under the License is distributed on an "AS IS" BASIS, --- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. --- See the License for the specific language governing permissions and --- limitations under the License. - --- +goose Up - --- OpenFGA storage schema (pinned to github.com/openfga/openfga PostgreSQL --- migration version 6). Kept in the same schema and Goose migration directory --- as Substrate tables so resource mutations and authorization tuple updates --- execute within the same PostgreSQL schema and transaction. - -CREATE TABLE tuple ( - store TEXT NOT NULL, - object_type TEXT NOT NULL, - object_id TEXT NOT NULL, - relation TEXT NOT NULL, - _user TEXT NOT NULL, - user_type TEXT NOT NULL, - ulid TEXT NOT NULL, - inserted_at TIMESTAMPTZ NOT NULL, - condition_name TEXT, - condition_context BYTEA, - PRIMARY KEY (store, object_type, object_id, relation, _user) -); - -CREATE INDEX idx_tuple_partial_user - ON tuple (store, object_type, object_id, relation, _user) - WHERE user_type = 'user'; - -CREATE INDEX idx_tuple_partial_userset - ON tuple (store, object_type, object_id, relation, _user) - WHERE user_type = 'userset'; - -CREATE UNIQUE INDEX idx_tuple_ulid ON tuple (ulid); - -CREATE INDEX idx_user_lookup ON tuple ( - store, - _user, - relation, - object_type, - object_id COLLATE "C" -); - -CREATE TABLE authorization_model ( - store TEXT NOT NULL, - authorization_model_id TEXT NOT NULL, - type TEXT NOT NULL, - type_definition BYTEA, - schema_version TEXT NOT NULL DEFAULT '1.0', - serialized_protobuf BYTEA, - PRIMARY KEY (store, authorization_model_id, type) -); - -CREATE TABLE store ( - id TEXT PRIMARY KEY, - name TEXT NOT NULL, - created_at TIMESTAMPTZ NOT NULL, - updated_at TIMESTAMPTZ, - deleted_at TIMESTAMPTZ -); - -CREATE TABLE assertion ( - store TEXT NOT NULL, - authorization_model_id TEXT NOT NULL, - assertions BYTEA, - PRIMARY KEY (store, authorization_model_id) -); - -CREATE TABLE changelog ( - store TEXT NOT NULL, - object_type TEXT NOT NULL, - object_id TEXT NOT NULL, - relation TEXT NOT NULL, - _user TEXT NOT NULL, - operation INTEGER NOT NULL, - ulid TEXT NOT NULL, - inserted_at TIMESTAMPTZ NOT NULL, - condition_name TEXT, - condition_context BYTEA, - PRIMARY KEY (store, ulid, object_type) -); - diff --git a/cmd/ateapi/internal/store/atepg/migrations/000003_access_policies.sql b/cmd/ateapi/internal/store/atepg/migrations/000003_access_policies.sql deleted file mode 100644 index f61b84d5ea..0000000000 --- a/cmd/ateapi/internal/store/atepg/migrations/000003_access_policies.sql +++ /dev/null @@ -1,33 +0,0 @@ --- Copyright 2026 Google LLC --- --- Licensed under the Apache License, Version 2.0 (the "License"); --- you may not use this file except in compliance with the License. --- You may obtain a copy of the License at --- --- http://www.apache.org/licenses/LICENSE-2.0 --- --- Unless required by applicable law or agreed to in writing, software --- distributed under the License is distributed on an "AS IS" BASIS, --- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. --- See the License for the specific language governing permissions and --- limitations under the License. - --- +goose Up - --- Singleton: the id column only admits true, so the table holds at most one row. -CREATE TABLE global_access_policy ( - id boolean PRIMARY KEY DEFAULT true CHECK (id), - uid text NOT NULL, - version bigint NOT NULL, - proto bytea NOT NULL -); - -CREATE TABLE atespace_access_policies ( - atespace_name text NOT NULL, - uid text NOT NULL, - version bigint NOT NULL, - proto bytea NOT NULL, - PRIMARY KEY (atespace_name), - FOREIGN KEY (atespace_name) - REFERENCES atespaces(name) ON DELETE CASCADE -); diff --git a/cmd/ateapi/internal/store/atepg/migrations_test.go b/cmd/ateapi/internal/store/atepg/migrations_test.go index 6b3fb705d8..5987e128be 100644 --- a/cmd/ateapi/internal/store/atepg/migrations_test.go +++ b/cmd/ateapi/internal/store/atepg/migrations_test.go @@ -41,7 +41,7 @@ var transactionControl = regexp.MustCompile(`(?im)^\s*(BEGIN|START\s+TRANSACTION // // Why this is needed: // 1. Substrate manages the OpenFGA PostgreSQL tables directly in -// migrations/000002_openfga.sql (rather than running OpenFGA's embedded Goose +// migrations/000001_initial.sql (rather than running OpenFGA's embedded Goose // migrations) so that OpenFGA tables and Substrate resource tables live in // the same PostgreSQL schema and migration ledger. // 2. cmd/ateapi/internal/authz/datastore.go adapts upstream postgres.Datastore @@ -75,7 +75,7 @@ func TestOpenFGAMigrationVersionGuard(t *testing.T) { } if maxVersion != pinnedOpenFGAMigrationVersion { t.Fatalf( - "OpenFGA embedded PostgreSQL migrations are at version %d, but 000002_openfga.sql and cmd/ateapi/internal/authz/datastore.go are pinned to version %d; port any new OpenFGA DDL to cmd/ateapi/internal/store/atepg/migrations/ and verify TransactionalDatastore before updating pinnedOpenFGAMigrationVersion", + "OpenFGA embedded PostgreSQL migrations are at version %d, but 000001_initial.sql and cmd/ateapi/internal/authz/datastore.go are pinned to version %d; port any new OpenFGA DDL to cmd/ateapi/internal/store/atepg/migrations/ and verify TransactionalDatastore before updating pinnedOpenFGAMigrationVersion", maxVersion, pinnedOpenFGAMigrationVersion, ) } diff --git a/cmd/atelet/internal/ateletpath/ateletpath.go b/cmd/atelet/internal/ateletpath/ateletpath.go index d38ad6241e..67bab746c1 100644 --- a/cmd/atelet/internal/ateletpath/ateletpath.go +++ b/cmd/atelet/internal/ateletpath/ateletpath.go @@ -43,9 +43,10 @@ var ( SandboxAssetCacheDir = filepath.Join(nodepath.BasePath, "sandbox-asset-cache") // SandboxAssetScratchDir holds short-lived destination directories for - // asset fetches that have no actor, such as prewarming a SandboxConfig's - // default version. Wiped at startup. Under BasePath so the links the - // cache serves into it are on the cache's mount. + // asset fetches that have no actor and no worker directory to link into: + // the assets the worker sandbox syncer stages when checking a version. + // Wiped at startup. Under BasePath so the links the cache serves into it + // are on the cache's mount. SandboxAssetScratchDir = filepath.Join(nodepath.BasePath, "sandbox-asset-scratch") // SandboxRuntimeIndexFile holds every sandbox runtime a worker on this diff --git a/cmd/atelet/main.go b/cmd/atelet/main.go index 1d2d846c3d..fb145da374 100644 --- a/cmd/atelet/main.go +++ b/cmd/atelet/main.go @@ -519,8 +519,9 @@ func main() { o.FieldSelector = "spec.nodeName=" + nodeName o.LabelSelector = workerPoolLabel })) - // Pre-download each local worker's default runtime, as the syncer plans - // the worker, so the first Run/Restore on this node hits the cache. + // Link each local worker's default runtime under the worker's directory + // as the syncer plans the worker, so it stays cached while no actor runs + // it. // Best-effort: on failure the on-demand fetch in ensureSandboxAssets // still covers correctness. prewarmer := newRuntimePrewarmer(runtimes, wmService) diff --git a/cmd/atelet/runtime_prewarm.go b/cmd/atelet/runtime_prewarm.go index ec648b138f..b58c851ecc 100644 --- a/cmd/atelet/runtime_prewarm.go +++ b/cmd/atelet/runtime_prewarm.go @@ -38,9 +38,9 @@ const defaultPrewarmAttemptTimeout = 10 * time.Minute // without hammering a permanently broken runtime forever. const prewarmMaxRetries = 8 -// sandboxAssetFetcher is the slice of Atelet the prewarmer and the syncer's -// probe need. Staging through the same methods as the Run/Restore path -// keeps the fetches from ever diverging on cache layout or validation. +// sandboxAssetFetcher is the slice of Atelet the prewarmer and the syncer +// need. Staging through the same methods as the Run/Restore path keeps the +// fetches from ever diverging on cache layout or validation. type sandboxAssetFetcher interface { ensureSandboxAssets(ctx context.Context, rec *sandboxAssetsRecord, dstDir string) (map[string]string, error) stageSandboxAssets(ctx context.Context, rec *sandboxAssetsRecord, dstDir string) (map[string]string, error) @@ -54,12 +54,13 @@ type prewarmItem struct { dir string } -// runtimePrewarmer fetches the default runtime of each worker on this node -// into the asset cache before any actor asks for it, so the -// fetch inside the first Run/Restore on the node is a cache hit instead of -// a download+extract on the critical path. The worker sandbox syncer hands -// it each default runtime a worker advertises, once the worker's file says -// so, with a directory to stage it into; the index holds the record. +// runtimePrewarmer links the default runtime of each worker on this node +// from the asset cache into the worker's default-runtime directory (fetching +// on a cache miss after eviction), so the fetch inside the first Run/Restore +// on the node is a cache hit instead of a download+extract on the critical +// path. The worker sandbox syncer hands it each default runtime a worker +// advertises, once the worker's file says so, with a directory to stage it +// into; the index holds the record. // // The assets are linked into that directory and left there: the cache // never evicts a linked entry, so a default runtime stays cached while no @@ -72,15 +73,13 @@ type prewarmItem struct { // // Prewarming is purely a latency optimization: every failure is logged and // left to the on-demand fetch in ensureSandboxAssets, which remains the -// correctness path. Only default runtimes are prewarmed; the first restore of -// another version a worker advertises pays its download. +// correctness path. Only default runtimes are linked here; non-default +// versions stay unlinked while no actor runs them. type runtimePrewarmer struct { index *runtimeIndex assets sandboxAssetFetcher // queue carries items. It dedupes while an item waits, and it - // rate-limits retries after failures. A single worker drains it, which - // also serializes downloads so concurrent prewarms never compete for - // node bandwidth. + // rate-limits retries after failures. A single worker drains it. queue workqueue.TypedRateLimitingInterface[prewarmItem] // done is closed when the worker goroutine exits; wait blocks on it. done chan struct{} @@ -174,11 +173,11 @@ func (p *runtimePrewarmer) process(ctx context.Context, item prewarmItem) { p.queue.Forget(item) } -// prewarm fetches the runtime's assets into the asset cache, linked into -// the item's directory. Racing an on-demand ensureSandboxAssets for the same -// assets is safe: the asset cache runs one download per asset and serves -// every caller from it. A directory the syncer already filled costs a stat -// per asset. +// prewarm links the runtime's assets from the asset cache into the item's +// directory, fetching on a cache miss after eviction. Racing an on-demand +// ensureSandboxAssets for the same assets is safe: the asset cache runs one +// download per asset and serves every caller from it. A directory the syncer +// already filled costs a stat per asset. // // Nothing here creates a directory: the staging goes into the item's // directory as it is, so a directory removed before or during the work, by diff --git a/cmd/atelet/sandbox_asset_cache_test.go b/cmd/atelet/sandbox_asset_cache_test.go index 625e9985f2..790e2af965 100644 --- a/cmd/atelet/sandbox_asset_cache_test.go +++ b/cmd/atelet/sandbox_asset_cache_test.go @@ -49,12 +49,12 @@ func TestValidateSandboxAssetCacheFlags(t *testing.T) { } // TestOpenSandboxAssetCacheResetsScratch pins the startup contract: the -// cache opens under the given dir, and whatever a crashed prewarm left in -// the scratch dir is gone. +// cache opens under the given dir, and whatever a crashed sync left in the +// scratch dir is gone. func TestOpenSandboxAssetCacheResetsScratch(t *testing.T) { useTempNodeDirs(t) useTempAssetScratch(t) - leftover := filepath.Join(ateletpath.SandboxAssetScratchDir, "prewarm-123", "runsc") + leftover := filepath.Join(ateletpath.SandboxAssetScratchDir, "cpu-features-123", "runsc") if err := os.MkdirAll(filepath.Dir(leftover), 0o755); err != nil { t.Fatal(err) } diff --git a/cmd/atelet/snapshotcache.go b/cmd/atelet/snapshotcache.go index 3cf2474fef..4e7a5dd0a0 100644 --- a/cmd/atelet/snapshotcache.go +++ b/cmd/atelet/snapshotcache.go @@ -53,7 +53,7 @@ import ( ) var ( - snapshotCacheDir = pflag.String("snapshot-cache-dir", ateletpath.SnapshotCacheDir, "Directory for the node-local shared snapshot file cache. Empty disables caching: every restore downloads its snapshot files, and the other --snapshot-cache-* flags are validated but not read. Must be under the node state dir, "+nodepath.BasePath+": the object-store plugin writes only there, and cache hits are served as hard links into the per-actor restore dirs on the same mount.") + snapshotCacheDir = pflag.String("snapshot-cache-dir", ateletpath.SnapshotCacheDir, "Directory for the node-local shared snapshot file cache. Empty disables caching: every restore downloads its snapshot files, and the other --snapshot-cache-* flags are validated but not read. Must be under the node state dir, "+nodepath.BasePath+": the object-store plugin writes only there. Cache hits are served as hard links into the per-actor restore dirs, or as copies when those are on another mount.") snapshotCacheMinAge = pflag.Duration("snapshot-cache-min-age", 10*time.Minute, "Cached snapshot files younger than this are never evicted, protecting files fetched but not yet linked into a restore dir. Read only while the cache is enabled.") snapshotCacheFetchTimeout = pflag.Duration("snapshot-cache-fetch-timeout", 10*time.Minute, "Bound on one cached snapshot file download. Downloads run detached from the restores waiting on them, so this is the only bound on how long one can run; a restore arriving while a download hangs waits on it until then. Sized for multi-GiB memory images on a busy node. Read only while the cache is enabled.") // snapshotCacheGCFlags are --snapshot-cache-gc-period, diff --git a/cmd/atelet/worker_sandbox_syncer.go b/cmd/atelet/worker_sandbox_syncer.go index fb94e04d29..45a82a7488 100644 --- a/cmd/atelet/worker_sandbox_syncer.go +++ b/cmd/atelet/worker_sandbox_syncer.go @@ -109,8 +109,9 @@ type workerSandboxSyncer struct { // pods holds this node's worker pods, indexed by podUIDIndex. pods cache.Indexer assets sandboxAssetFetcher - // scratchDir is where the runsc probe stages the binary it runs, read - // from ateletpath once here so a test can point a syncer elsewhere. + // scratchDir is where buildFile stages assets to verify them (and run + // the runsc probe), read from ateletpath once here so a test can point + // a syncer elsewhere. scratchDir string // queue carries pod UIDs. A workqueue never hands the same key to two // workers at once, so a pod's file is never written and its directory @@ -124,16 +125,18 @@ type workerSandboxSyncer struct { probeCPUIdentity func() (vendor, model string, err error) // runCPUFeatures is runRunscCPUFeatures. Overridable in tests. runCPUFeatures func(ctx context.Context, runscPath string) ([]string, error) - // mu guards hostCPU and cpuFeatures across the syncWorkers in run: - // hostCPU is the probed host CPU identity, nil until a probe succeeds, - // and cpuFeatures the encoded cpu_features of each runsc binary by its - // gVisor asset's SHA256 digest. A binary is fetched and run once per - // digest, except that two workers meeting a digest before either has - // memoized it both probe it; the duplicate costs a fetch from the asset - // cache and a run of runsc, and both store the same mask. - mu sync.Mutex - hostCPU *cpuIdentity - cpuFeatures map[string]string + // mu guards hostCPU, cpuFeatures and microvmAssets across the syncWorkers + // in run: hostCPU is the probed host CPU identity, nil until a probe + // succeeds; cpuFeatures is the encoded cpu_features of each runsc binary + // by its gVisor asset's SHA256 digest; and microvmAssets holds the + // bundle hashes of microVM versions whose assets have been verified in + // this atelet process. Restorable microVM versions are verified once per + // atelet process (except when two workers meet a version before either + // memoizes it); the default version is staged on every sync. + mu sync.Mutex + hostCPU *cpuIdentity + cpuFeatures map[string]string + microvmAssets map[string]bool } func newWorkerSandboxSyncer(resolver *workerSandboxResolver, pods cache.Indexer, assets sandboxAssetFetcher, workersDir string, index *runtimeIndex, prewarm runtimePrewarm) *workerSandboxSyncer { @@ -155,6 +158,7 @@ func newWorkerSandboxSyncer(resolver *workerSandboxResolver, pods cache.Indexer, probeCPUIdentity: hardware.ProbeCPUIdentity, runCPUFeatures: runRunscCPUFeatures, cpuFeatures: make(map[string]string), + microvmAssets: make(map[string]bool), } } diff --git a/cmd/atelet/worker_sandbox_syncer_test.go b/cmd/atelet/worker_sandbox_syncer_test.go index f88a146df0..0f0d1b108e 100644 --- a/cmd/atelet/worker_sandbox_syncer_test.go +++ b/cmd/atelet/worker_sandbox_syncer_test.go @@ -75,8 +75,7 @@ func (f *fakeAssetFetcher) stageSandboxAssets(ctx context.Context, rec *sandboxA } // newTestAssetFetcher returns a sandboxAssetFetcher that maps every asset of -// a record to a path under its digest without fetching anything, for syncers -// whose runsc probe is faked. +// a record to a path under its digest without fetching anything. func newTestAssetFetcher() sandboxAssetFetcher { return &fakeAssetFetcher{ fn: func(_ context.Context, rec *sandboxAssetsRecord) (map[string]string, error) { @@ -175,7 +174,7 @@ func newTestSyncer(t *testing.T, objs ...any) *workerSandboxSyncer { }, pods, newTestAssetFetcher(), t.TempDir(), newRuntimeIndex(), nil) // The queue starts a goroutine; stop it with the test. t.Cleanup(s.queue.ShutDown) - s.scratchDir = t.TempDir() // the runsc probe stages the binary there + s.scratchDir = t.TempDir() // buildFile stages assets there s.probeCPUIdentity = func() (string, string, error) { return testCPUVendor, testCPUModel, nil } s.runCPUFeatures = func(context.Context, string) ([]string, error) { return testCPUFeatures(), nil } // Tests that check the jitter set it. diff --git a/cmd/atelet/worker_sandbox_versions.go b/cmd/atelet/worker_sandbox_versions.go index 57ee9d2281..6449dcbba0 100644 --- a/cmd/atelet/worker_sandbox_versions.go +++ b/cmd/atelet/worker_sandbox_versions.go @@ -19,9 +19,9 @@ package main // - resolve: find the SandboxConfig of the pod's WorkerPool // - plan: pick the config's versions this node can run // - build: hash each version's asset digests into its sandbox_bundle_hash, -// and probe what a snapshot taken on the runtime must find again: the -// host CPU identity for microVM, the runsc binary's CPU features for -// gVisor +// verify its assets in the cache, and probe what a snapshot taken on the +// runtime must find again: the host CPU identity for microVM, the runsc +// binary's CPU features for gVisor // - write: replace the file in the worker directory if its content changed // // worker_sandbox_syncer.go decides when each pod is synced, and creates and @@ -231,14 +231,18 @@ func hashVersions(plan *sandboxPlan) []hashedVersion { // // TODO: write the file as soon as the default version's attributes are known // and add restorable runtimes on a later pass. On a fresh node the first -// worker cannot register until every restorable version's runsc has been +// worker cannot register until every restorable version's assets have been // fetched and probed, serially. // -// Listing a runtime does not promise its assets are on the node: probing -// fetches a gVisor version's runsc the first time this atelet sees it, and -// nothing of a microVM version. The syncer has the default runtime prewarmed -// once the file is written, and Run/Restore fetch any other version on -// demand. +// The first time this atelet process sees a gVisor runsc digest or a +// restorable microVM bundle hash, buildFile fetches the version's assets into +// the cache (and runs runsc for gVisor); a microVM default version is staged +// on every build so promoting an evicted version to default verifies it in the +// cache before unpinning the previous default. Once the file is written, the +// syncer links the default runtime under the worker's directory so cache GC +// never evicts it. Run and Restore fetch any evicted restorable version on +// demand. A version with a bad sha256 fails verification without being cached +// or memoized, so retries re-download it until the SandboxConfig is fixed. func (s *workerSandboxSyncer) buildFile(ctx context.Context, plan *sandboxPlan, hashed []hashedVersion) (*sandboxruntimes.File, error) { class := sandboxruntimes.Class{SandboxClass: string(plan.sandboxClass)} seen := make(map[string]bool, len(hashed)) @@ -247,7 +251,7 @@ func (s *workerSandboxSyncer) buildFile(ctx context.Context, plan *sandboxPlan, if seen[v.hash] { continue } - attrs, err := s.versionAttrs(ctx, plan.sandboxClass, v.rec, v.hash) + attrs, err := s.versionAttrs(ctx, plan.sandboxClass, v.rec, v.hash, v.isDefault) if err != nil { err = fmt.Errorf("SandboxConfig %q, version %q: %w", plan.config, v.name, err) if v.isDefault { @@ -280,9 +284,10 @@ func (s *workerSandboxSyncer) buildFile(ctx context.Context, plan *sandboxPlan, // versionAttrs returns the attributes of one version's runtime, whose // sandbox_bundle_hash is hash: architecture and the hash for every class, -// the host CPU identity for a microVM runtime, and the runsc binary's -// cpu_features for a gVisor one. -func (s *workerSandboxSyncer) versionAttrs(ctx context.Context, class atev1.SandboxClass, rec *sandboxAssetsRecord, hash string) (sandboxruntimes.Attributes, error) { +// the host CPU identity for a microVM runtime (with its assets verified in +// the local cache), and the runsc binary's cpu_features for a gVisor one. +// isDefault makes the microVM asset check skip the memo. +func (s *workerSandboxSyncer) versionAttrs(ctx context.Context, class atev1.SandboxClass, rec *sandboxAssetsRecord, hash string, isDefault bool) (sandboxruntimes.Attributes, error) { attrs := sandboxruntimes.Attributes{ hardware.AttrArchitecture: runtime.GOARCH, hardware.AttrSandboxBundleHash: hash, @@ -293,6 +298,9 @@ func (s *workerSandboxSyncer) versionAttrs(ctx context.Context, class atev1.Sand if err != nil { return nil, fmt.Errorf("while probing host CPU identity: %w", err) } + if err := s.ensureMicroVMAssets(ctx, rec, hash, isDefault); err != nil { + return nil, err + } attrs[hardware.AttrCPUVendor], attrs[hardware.AttrCPUModel] = vendor, model case atev1.SandboxClassGvisor: mask, err := s.gvisorCPUFeatures(ctx, rec) @@ -325,6 +333,38 @@ func (s *workerSandboxSyncer) hostCPUIdentity() (vendor, model string, err error return s.hostCPU.vendor, s.hostCPU.model, nil } +// ensureMicroVMAssets stages a microVM version's assets into a scratch +// directory so a version whose assets cannot be fetched or fail SHA256 +// verification is not advertised in the worker's sandbox runtimes file. +// Restorable versions are checked once per atelet process; the default version +// is checked on every sync (a cache hit once prewarmed) so promoting an +// evicted version to default verifies it in the cache before unpinning the +// previous default. +func (s *workerSandboxSyncer) ensureMicroVMAssets(ctx context.Context, rec *sandboxAssetsRecord, hash string, isDefault bool) error { + if !isDefault { + s.mu.Lock() + ok := s.microvmAssets[hash] + s.mu.Unlock() + if ok { + return nil + } + } + ctx, cancel := context.WithTimeout(ctx, probeFetchTimeout) + defer cancel() + scratch, err := os.MkdirTemp(s.scratchDir, "microvm-assets-") + if err != nil { + return fmt.Errorf("while creating microvm-assets scratch dir: %w", err) + } + defer func() { _ = os.RemoveAll(scratch) }() + if _, err := s.assets.ensureSandboxAssets(ctx, rec, scratch); err != nil { + return err + } + s.mu.Lock() + s.microvmAssets[hash] = true + s.mu.Unlock() + return nil +} + // gvisorCPUFeatures returns the encoded cpu_features of rec's runsc binary. // The first time this atelet sees the binary's digest it fetches the binary // into the local asset cache and runs ` cpu-features`. @@ -367,14 +407,19 @@ func (s *workerSandboxSyncer) gvisorCPUFeatures(ctx context.Context, rec *sandbo return mask, nil } -// probeFetchTimeout bounds the fetch of a runsc binary for its probe, from -// the first cache lookup to the staged link. The asset cache bounds each -// download with its own fetch timeout, and the exec below has -// cpuFeaturesTimeout; this is the backstop for everything around them, -// such as waiting on a download another caller started, so a probe cannot -// hold one of the syncer's few workers for longer than one download may -// take. -const probeFetchTimeout = 5 * time.Minute +// probeFetchTimeout bounds staging a version's assets during buildFile (a +// gVisor version's runsc binary for its probe, or a microVM version's kernel, +// guest image, Cloud Hypervisor and virtiofsd), from the first cache lookup to +// the staged links. The asset cache bounds each individual download with its +// own detached --sandbox-asset-fetch-timeout, and the exec below has +// cpuFeaturesTimeout. This is the backstop for the whole version so a slow +// bucket or a download another caller started cannot hold one of the syncer's +// few workers for longer than this before the sync fails and retries. Twice +// the flag's default: one guest image at the flag's bound, then the three +// small binaries. Raising the flag past this makes a cold build time out and +// retry; its downloads continue detached, so the retry finds them cached or +// joins one still in flight. +const probeFetchTimeout = 10 * time.Minute // cpuFeaturesTimeout bounds one `runsc cpu-features` run, which takes // milliseconds. A runsc that hangs would hold a sync worker. A var so tests @@ -431,12 +476,13 @@ func bundleHash(rec *sandboxAssetsRecord) string { // - plan: resolve the pool's config and its versions for this node // - hash: hash each version's runtime, and make each resolvable by its // hash before the file can let the worker register it -// - build: probe what a snapshot taken on each runtime must find again +// - build: verify each version's assets in the cache and probe what a +// snapshot taken on the runtime must find again // - write: replace the file if its content changed, then record what // the worker now advertises in the index // - prewarm: make the pod's default-runtime directories match what the -// file now advertises, and have each default's assets fetched and -// linked into its directory, off this goroutine +// file now advertises, and have each default's assets linked into its +// directory, off this goroutine // // A file missing restorable runtimes for a transient reason is written all // the same, and the errIncomplete from build is returned so finish retries. @@ -466,9 +512,10 @@ func (s *workerSandboxSyncer) syncPod(ctx context.Context, pod *corev1.Pod) erro return err } // Only now, with the file naming its defaults: a sync that failed above - // left the file, and so the directories, as they were. For a gVisor - // default the build fetched the release for its probe, so the prewarm - // is links; a micro-VM default downloads here. + // left the file, and so the directories, as they were. The build staged + // a microVM default just now and fetched a gVisor default the first time + // it saw its digest, so the prewarm is links unless an evicted gVisor + // version was promoted to default. if s.prewarm != nil { var defaults []string for _, v := range writtenVersions(hashed, want) { diff --git a/cmd/atelet/worker_sandbox_versions_test.go b/cmd/atelet/worker_sandbox_versions_test.go index f381b0fc12..827ad39c17 100644 --- a/cmd/atelet/worker_sandbox_versions_test.go +++ b/cmd/atelet/worker_sandbox_versions_test.go @@ -33,6 +33,7 @@ import ( "k8s.io/utils/ptr" atev1 "go.ate.dev/substrate/api/infra/k8s/ate/v1" + "go.ate.dev/substrate/cmd/atelet/internal/filecache" "go.ate.dev/substrate/internal/hardware" "go.ate.dev/substrate/internal/sandboxruntimes" ) @@ -390,10 +391,16 @@ func TestPlanAndBuildMicroVM(t *testing.T) { } s := newTestSyncer(t, pool, cfg) + var fetched []string + var failSHA string s.assets = &fakeAssetFetcher{ - fn: func(context.Context, *sandboxAssetsRecord) (map[string]string, error) { - t.Fatal("microVM buildFile must not fetch sandbox assets") - return nil, nil + fn: func(ctx context.Context, rec *sandboxAssetsRecord) (map[string]string, error) { + sha := rec.Assets["kernel"].SHA256 + fetched = append(fetched, sha) + if sha == failSHA { + return nil, fmt.Errorf("bad sha256 for %s", sha) + } + return newTestAssetFetcher().ensureSandboxAssets(ctx, rec, "") }, } probeCalls := 0 @@ -412,20 +419,199 @@ func TestPlanAndBuildMicroVM(t *testing.T) { t.Fatalf("planAndBuild() on probe failure = %v, want %v and not permanent", err, probeErr) } - // Once a probe succeeds, every later sync reuses it. + // A failed fetch on the default version fails the build without writing. probeErr = nil + failSHA = "sha-v2" + if _, err := planAndBuild(s, workerPod("pod-a", "microvm-pool")); err == nil || errors.Is(err, errPermanent) || errors.Is(err, errIncomplete) { + t.Fatalf("planAndBuild() on default asset fetch failure = %v, want retryable error", err) + } + + // A failed fetch on a restorable version returns the default runtime + // alongside errIncomplete, leaving the unverified restorable version out. + failSHA = "sha-v1" + partial, err := planAndBuild(s, workerPod("pod-a", "microvm-pool")) + if !errors.Is(err, errIncomplete) || errors.Is(err, errPermanent) { + t.Fatalf("planAndBuild() on restorable asset fetch failure = %v, want errIncomplete", err) + } + if diff := cmp.Diff(classFile(atev1.SandboxClassMicroVM, wantMicroVM("sha-v2")), partial); diff != "" { + t.Errorf("planAndBuild() partial file mismatch (-want +got):\n%s", diff) + } + + // Once a restorable version's assets are verified, the check is memoized + // by bundle hash alongside the host CPU identity probe. + failSHA = "" want := classFile(atev1.SandboxClassMicroVM, wantMicroVM("sha-v2"), wantMicroVM("sha-v1")) - for _, uid := range []string{"pod-a", "pod-b"} { - got, err := planAndBuild(s, workerPod(uid, "microvm-pool")) + got, err := planAndBuild(s, workerPod("pod-a", "microvm-pool")) + if err != nil { + t.Fatalf("planAndBuild(pod-a) error = %v", err) + } + if diff := cmp.Diff(want, got); diff != "" { + t.Errorf("planAndBuild(pod-a) mismatch (-want +got):\n%s", diff) + } + + // Syncing pod-b stages the default (v2) again and reuses the memoized check + // for the restorable version (v1) without fetching v1 again. + failSHA = "sha-v1" + got, err = planAndBuild(s, workerPod("pod-b", "microvm-pool")) + if err != nil { + t.Fatalf("planAndBuild(pod-b) error = %v", err) + } + if diff := cmp.Diff(want, got); diff != "" { + t.Errorf("planAndBuild(pod-b) mismatch (-want +got):\n%s", diff) + } + if probeCalls != 2 { + t.Errorf("probe calls = %d, want 2: one that failed, one that succeeded and was reused", probeCalls) + } + // The probe-failure build fetches nothing. The default-failure build + // fetches v2, the restorable-failure build fetches v2 then fails on v1, + // pod-a fetches v2 and v1 (memoizing v1), and pod-b fetches only the + // default v2. + wantFetched := []string{"sha-v2", "sha-v2", "sha-v1", "sha-v2", "sha-v1", "sha-v2"} + if diff := cmp.Diff(wantFetched, fetched); diff != "" { + t.Errorf("fetched assets mismatch (-want +got):\n%s", diff) + } +} + +// TestSyncPodMicroVMAssetCacheAndEviction runs syncPod for a microVM pool +// against the real sandboxAssetCache and runtimePrewarmer: +// - buildFile stages each version's assets into scratchDir and cleans up; +// - the prewarmer links only the default version under the worker's directory, +// so EvictUnused reclaims the unlinked restorable version; +// - once memoized, syncing a new worker pod does not re-download the evicted +// restorable version, and re-syncing while the bucket is failing keeps the +// restorable version in sandbox-runtimes.json; +// - promoting the evicted restorable version to default while the bucket is +// failing fails the sync and leaves the file and the old default's +// directory as they were. +func TestSyncPodMicroVMAssetCacheAndEviction(t *testing.T) { + ctx := context.Background() + v1Bytes, v2Bytes := []byte("kernel v1"), []byte("kernel v2") + v1SHA := fmt.Sprintf("%x", sha256.Sum256(v1Bytes)) + v2SHA := fmt.Sprintf("%x", sha256.Sum256(v2Bytes)) + + pool := resolvedPoolOf(atev1.SandboxClassMicroVM, "microvm-pool", "microvm-release") + version := func(name, sha string) atev1.SandboxVersionConfig { + return atev1.SandboxVersionConfig{ + Name: name, + Assets: map[string]map[string]atev1.AssetFile{ + runtime.GOARCH: {"kernel": {URL: "gs://assets/kernel-" + name, SHA256: sha}}, + }, + } + } + cfg := &atev1.SandboxConfig{ + ObjectMeta: metav1.ObjectMeta{Name: "microvm-release"}, + Spec: atev1.SandboxConfigSpec{ + SandboxClass: atev1.SandboxClassMicroVM, + DefaultVersion: "v2", + Versions: []atev1.SandboxVersionConfig{ + version("v1", v1SHA), + version("v2", v2SHA), + }, + }, + } + wantMicroVM := func(sha string) sandboxruntimes.Attributes { + rec := &sandboxAssetsRecord{Assets: map[string]assetEntry{"kernel": {SHA256: sha}}} + return sandboxruntimes.Attributes{ + hardware.AttrArchitecture: runtime.GOARCH, + hardware.AttrCPUVendor: testCPUVendor, + hardware.AttrCPUModel: testCPUModel, + hardware.AttrSandboxBundleHash: bundleHash(rec), + } + } + want := classFile(atev1.SandboxClassMicroVM, wantMicroVM(v2SHA), wantMicroVM(v1SHA)) + + root := t.TempDir() + storage := &bucketStorage{ + objects: map[string][]byte{ + "assets/kernel-v1": v1Bytes, + "assets/kernel-v2": v2Bytes, + }, + } + herder := &Atelet{ + sandboxAssetCache: newTestAssetCacheAt(t, root, filecache.WithMinAge(0)), + anonGCSClient: storage, + } + s := newTestSyncer(t, pool, cfg) + s.assets = herder + s.workersDir = filepath.Join(root, "workers") + s.scratchDir = filepath.Join(root, "scratch") + if err := os.MkdirAll(s.scratchDir, 0o700); err != nil { + t.Fatal(err) + } + recPrewarm := &recordingPrewarm{} + s.prewarm = recPrewarm + + if err := s.syncPod(ctx, workerPod("pod-a", "microvm-pool")); err != nil { + t.Fatalf("syncPod(pod-a): %v", err) + } + if left, err := os.ReadDir(s.scratchDir); err != nil || len(left) != 0 { + t.Errorf("scratchDir entries after syncPod = %v, %v; want empty", left, err) + } + if got := storage.gets.Load(); got != 2 { + t.Fatalf("bucket gets after first sync = %d, want 2", got) + } + if n := cachedAssets(t, root); n != 2 { + t.Fatalf("cached assets after first sync = %d, want 2", n) + } + + // Link the default version (v2) into pod-a's default-runtime directory via + // the prewarmer, leaving the restorable version (v1) unlinked. + p := newRuntimePrewarmer(s.index, herder) + for _, item := range recPrewarm.items() { + p.process(ctx, item) + } + stats, err := herder.sandboxAssetCache.EvictUnused(ctx, 1<<40, false) + if err != nil { + t.Fatalf("EvictUnused: %v", err) + } + if stats.Retired != 1 || stats.SkippedLinked != 1 || cachedAssets(t, root) != 1 { + t.Fatalf("after EvictUnused: stats=%+v, cached=%d; want 1 retired (v1) and 1 linked kept (v2)", stats, cachedAssets(t, root)) + } + + // Take the bucket offline and verify that syncing a new pod (pod-b) and + // re-syncing pod-a neither re-download v1 nor drop it from the written file. + storage.mu.Lock() + clear(storage.objects) + storage.mu.Unlock() + + for _, uid := range []string{"pod-b", "pod-a"} { + if err := s.syncPod(ctx, workerPod(uid, "microvm-pool")); err != nil { + t.Fatalf("syncPod(%s) after eviction with bucket offline: %v", uid, err) + } + got, err := sandboxruntimes.Read(s.runtimesFile(uid)) if err != nil { - t.Fatalf("planAndBuild(%s) error = %v", uid, err) + t.Fatalf("sandboxruntimes.Read(%s): %v", uid, err) } if diff := cmp.Diff(want, got); diff != "" { - t.Errorf("planAndBuild(%s) mismatch (-want +got):\n%s", uid, diff) + t.Errorf("sandbox-runtimes.json for %s mismatch (-want +got):\n%s", uid, diff) } } - if probeCalls != 2 { - t.Errorf("probe calls = %d, want 2: one that failed, one that succeeded and was reused", probeCalls) + if got := storage.gets.Load(); got != 2 { + t.Errorf("bucket gets after syncing pod-b and re-syncing pod-a = %d, want 2 (no re-downloads)", got) + } + + // Promoting the evicted v1 to default while the bucket is offline must fail + // the sync and keep pod-a's previous file and default-runtime pin in place. + cfg.Spec.DefaultVersion = "v1" + if err := s.syncPod(ctx, workerPod("pod-a", "microvm-pool")); err == nil { + t.Fatal("syncPod(pod-a) after promoting evicted v1 to default with bucket offline = nil, want error") + } + got, err := sandboxruntimes.Read(s.runtimesFile("pod-a")) + if err != nil { + t.Fatalf("sandboxruntimes.Read(pod-a): %v", err) + } + if diff := cmp.Diff(want, got); diff != "" { + t.Errorf("sandbox-runtimes.json for pod-a after failed default promotion (-want +got):\n%s", diff) + } + v2Hash := want.SandboxClasses[0].DefaultRuntime.Version[hardware.AttrSandboxBundleHash] + if !exists(s.defaultRuntimeDir("pod-a", v2Hash)) { + t.Error("a failed sync removed pod-a's default-runtime directory for v2") + } + if n := cachedAssets(t, root); n != 1 { + t.Errorf("cached assets after failed default promotion = %d, want 1 (v2 still linked)", n) + } + if got := storage.gets.Load(); got != 3 { + t.Errorf("bucket gets after promoting evicted v1 to default = %d, want 3 (the default is checked, not memoized)", got) } } diff --git a/cmd/ateworker-gvisor/ckptcgroup.go b/cmd/ateworker-gvisor/ckptcgroup.go index e5e26eefa9..f2a5a6055f 100644 --- a/cmd/ateworker-gvisor/ckptcgroup.go +++ b/cmd/ateworker-gvisor/ckptcgroup.go @@ -38,7 +38,7 @@ import ( // During checkpoint, moving the sentry from -_sandbox into a sibling // -_ckpt cgroup (memory.max = "max", cpu.max copied from -_sandbox) // prevents newly written pages.img cache from evicting clean pages or -// OOM-killing the sentry under the actor's memory.max. Removing -_ckpt +// throttling the sentry under the actor's memory.high. Removing -_ckpt // after sandbox teardown reparents the cache to the worker scope for fast // local Resume, and dropActorCheckpointCacheAsync unlinks non-preserved files // in the restore dir or evicts preserved checkpoint page cache via @@ -148,27 +148,22 @@ func (c *checkpointCgroup) remove() error { return nil } -// checkpointIsolated isolates the sentry into a sibling cgroup (when the -// worker's cgroup is delegated), runs checkpoint, and tears down the workload +// checkpointIsolated isolates the sentry into a sibling cgroup, runs +// checkpoint, and tears down the workload // before removing the sibling cgroup so its page cache reparents cleanly. If // checkpoint fails, it moves the sentry back into its sandbox leaf (or terminates // the workload if that rollback fails). func checkpointIsolated( ctx context.Context, - cgroupDelegated bool, cgroupRoot, actorUID string, actorDirs *ateworkerpb.ActorDirs, checkpoint func(context.Context) error, terminate func() error, ) error { - var ckptCgroup *checkpointCgroup - if cgroupDelegated { - var err error - ckptCgroup, err = isolateCheckpointCache(cgroupRoot, actorDirs, actorUID) - if err != nil { - slog.WarnContext(ctx, "Checkpointing without page cache isolation", - slog.String("actorUID", actorUID), slog.Any("err", err)) - } + ckptCgroup, err := isolateCheckpointCache(cgroupRoot, actorDirs, actorUID) + if err != nil { + slog.WarnContext(ctx, "Checkpointing without page cache isolation", + slog.String("actorUID", actorUID), slog.Any("err", err)) } terminateOnExit := true defer func() { diff --git a/cmd/ateworker-gvisor/ckptcgroup_test.go b/cmd/ateworker-gvisor/ckptcgroup_test.go index dfca0796e9..910c98987e 100644 --- a/cmd/ateworker-gvisor/ckptcgroup_test.go +++ b/cmd/ateworker-gvisor/ckptcgroup_test.go @@ -221,7 +221,7 @@ func TestCheckpointIsolated(t *testing.T) { err := checkpointIsolated( context.Background(), - true, f.root, testActorUID, f.actorDirs, + f.root, testActorUID, f.actorDirs, func(context.Context) error { if got := mustRead(t, filepath.Join(ckptDir, "cgroup.procs")); got != "101" { t.Errorf("checkpoint cgroup.procs during checkpoint = %q, want 101", got) @@ -246,17 +246,17 @@ func TestCheckpointIsolated(t *testing.T) { } }) - t.Run("skips cgroup isolation when cgroup is not delegated", func(t *testing.T) { + t.Run("checkpoints without isolation when the sandbox pid is unknown", func(t *testing.T) { f := newCkptFixture(t, "") ckptDir := checkpointCgroupDir(f.root, testActorUID) var terminated bool err := checkpointIsolated( context.Background(), - false, f.root, testActorUID, f.actorDirs, + f.root, testActorUID, f.actorDirs, func(context.Context) error { if _, err := os.Stat(ckptDir); !os.IsNotExist(err) { - t.Errorf("checkpoint cgroup should not be created when !cgroupDelegated: %v", err) + t.Errorf("checkpoint cgroup should not be created without a sandbox pid: %v", err) } return nil }, @@ -280,7 +280,7 @@ func TestCheckpointIsolated(t *testing.T) { err := checkpointIsolated( context.Background(), - true, f.root, testActorUID, f.actorDirs, + f.root, testActorUID, f.actorDirs, func(context.Context) error { return errors.New("checkpoint boom") }, @@ -310,7 +310,7 @@ func TestCheckpointIsolated(t *testing.T) { err := checkpointIsolated( context.Background(), - true, f.root, testActorUID, f.actorDirs, + f.root, testActorUID, f.actorDirs, func(context.Context) error { movePID = func(string, int) error { return unix.EACCES } return errors.New("checkpoint boom") @@ -339,7 +339,7 @@ func TestCheckpointIsolated(t *testing.T) { err := checkpointIsolated( context.Background(), - true, f.root, testActorUID, f.actorDirs, + f.root, testActorUID, f.actorDirs, func(context.Context) error { if err := os.RemoveAll(f.leaf); err != nil { t.Fatal(err) @@ -369,7 +369,7 @@ func TestCheckpointIsolated(t *testing.T) { err := checkpointIsolated( context.Background(), - true, f.root, testActorUID, f.actorDirs, + f.root, testActorUID, f.actorDirs, func(context.Context) error { return nil }, func() error { // Non-empty ckptDir emulates a sentry still alive in the cgroup. diff --git a/cmd/ateworker-gvisor/crash.go b/cmd/ateworker-gvisor/crash.go index e0bc39ca6e..e6e1b83cb8 100644 --- a/cmd/ateworker-gvisor/crash.go +++ b/cmd/ateworker-gvisor/crash.go @@ -57,7 +57,10 @@ func (s *AteWorkerService) watchSandbox(ctx context.Context, hosted *hostedActor return } ctx := context.Background() - const message = "gVisor sandbox exited" + message := "gVisor sandbox exited" + if hosted.oomKilled.Load() { + message = "killed for exceeding its memory limit" + } // stopping is checked under the actor's lock, so an exit during a // checkpoint that then fails is still handled once it gives up. diff --git a/cmd/ateworker-gvisor/hosted.go b/cmd/ateworker-gvisor/hosted.go index a1c764ebe2..47e9b56e1d 100644 --- a/cmd/ateworker-gvisor/hosted.go +++ b/cmd/ateworker-gvisor/hosted.go @@ -21,11 +21,13 @@ import ( "fmt" "log/slog" "net" + "sync" "sync/atomic" "time" "go.ate.dev/substrate/internal/apierror" "go.ate.dev/substrate/internal/ateworker/actornet" + "go.ate.dev/substrate/internal/ateworker/cgroup" "go.ate.dev/substrate/internal/atunnel/ingress" "go.ate.dev/substrate/internal/proto/ateworkerpb" "go.ate.dev/substrate/internal/resources" @@ -53,6 +55,15 @@ type hostedActor struct { // nothing that waits on it later can be fooled by a reused pid. Nil if it // could not be pinned. Set once, under the actor's lock. sandbox *sandboxProcess + // memoryLimit is the sandbox leaf's memory.high, or 0 while watchMemory + // should leave it alone. + memoryLimit atomic.Uint64 + // oomKilled is set before watchMemory kills the sandbox. + oomKilled atomic.Bool + // memoryMu is held while watchMemory checks or kills the sandbox, and + // guards memoryWatch. + memoryMu sync.Mutex + memoryWatch cgroup.MemoryWatch } // admitActor reserves capacity before network setup. An actor that is already diff --git a/cmd/ateworker-gvisor/main.go b/cmd/ateworker-gvisor/main.go index 55136406fe..14f386a144 100644 --- a/cmd/ateworker-gvisor/main.go +++ b/cmd/ateworker-gvisor/main.go @@ -40,6 +40,7 @@ import ( "go.ate.dev/substrate/internal/actorlog" "go.ate.dev/substrate/internal/apierror" "go.ate.dev/substrate/internal/ateinterceptors" + "go.ate.dev/substrate/internal/ateworker" "go.ate.dev/substrate/internal/ateworker/actornet" "go.ate.dev/substrate/internal/ateworker/cgroup" "go.ate.dev/substrate/internal/ateworker/phaselog" @@ -205,13 +206,10 @@ func do(ctx context.Context) error { // Prepare the pod cgroup so runsc can create per-actor-container leaves under // it with real accounting. - cgroupDelegated, err := cgroup.Delegate(ctx) - if err != nil { + if err := cgroup.Delegate(ctx); err != nil { return fmt.Errorf("while setting up cgroup delegation: %w", err) } - if cgroupDelegated { - removeStaleCheckpointCgroups(ctx, defaultCgroupRoot) - } + removeStaleCheckpointCgroups(ctx, defaultCgroupRoot) // Clean up any old socket. sockPath := nodepath.WorkerSocketPath(*podUID) @@ -240,7 +238,10 @@ func do(ctx context.Context) error { return err } ateworkerService := NewService(tunnel, actorLogger, *maxActors, flags) - ateworkerService.cgroupDelegated = cgroupDelegated + ateworkerService.throttleMemory = psiEnabled(defaultCgroupRoot) + if !ateworkerService.throttleMemory { + slog.WarnContext(ctx, "Kernel has no cgroup memory pressure (PSI), so sandboxes keep memory.max and an OOM kill in one can restart the worker") + } // The controller sets both from the downward API. pool := workerstats.Pool{Namespace: os.Getenv("POD_NAMESPACE"), Name: os.Getenv("WORKER_POOL_NAME")} if pool.Namespace == "" || pool.Name == "" { @@ -250,6 +251,9 @@ func do(ctx context.Context) error { defer usageStdout.Close() ateworkerService.usage = workerstats.NewUsageEmitter(lp, usageStdout, pool) defer workerstats.StartSampler(ctx, *usageSampleInterval, func(ctx context.Context) { ateworkerService.sweepUsage(ctx) })() + if ateworkerService.throttleMemory { + go ateworkerService.watchMemory(ctx) + } svr := grpc.NewServer( grpc.StatsHandler(otelgrpc.NewServerHandler()), @@ -365,9 +369,9 @@ type AteWorkerService struct { // than a constant so tests can point GetWorkloadStats at a fixture tree. cgroupRoot string - // cgroupDelegated is set when the worker's cgroup is delegated (false on - // privileged workers that inherit the host cgroup namespace). - cgroupDelegated bool + // throttleMemory is set when sandboxes get memory.high in place of + // memory.max, with watchMemory killing the ones stuck there. + throttleMemory bool // readSandboxCgroup overrides cgroupstats.Read when set. Only tests set it: // it is the seam that lets them interleave a lifecycle transition with the @@ -629,6 +633,9 @@ func (s *AteWorkerService) RunWorkload(ctx context.Context, req *ateworkerpb.Run if err := resetRunscStateAndPidFileDirs(req.GetActorDirs()); err != nil { return nil, err } + if err := s.removeStaleSandboxLeaf(ctx, req.GetActorUid()); err != nil { + return nil, err + } if err := s.tunnel.Deactivate(ctx, workerstats.ActorAttributionFromRequest(req)); err != nil { return nil, err @@ -663,6 +670,9 @@ func (s *AteWorkerService) RunWorkload(ctx context.Context, req *ateworkerpb.Run var containersToDelete []string defer func() { if retErr != nil { + if hosted.oomKilled.Load() { + retErr = fmt.Errorf("killed for exceeding its memory limit: %w", retErr) + } cleanupCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second) defer cancel() if err := s.tunnel.Deactivate(cleanupCtx, workerstats.ActorAttributionFromRequest(req)); err != nil { @@ -698,6 +708,7 @@ func (s *AteWorkerService) RunWorkload(ctx context.Context, req *ateworkerpb.Run return nil, fmt.Errorf("while creating the sandbox: %w", err) } s.pinSandbox(ctx, hosted, req.GetActorDirs()) + s.throttleSandboxMemory(ctx, hosted) if err := rcmd.cmdStart(ctx, pw, ocispec.SandboxID); err != nil { return nil, fmt.Errorf("while starting the sandbox: %w", err) } @@ -761,6 +772,13 @@ func (s *AteWorkerService) CheckpointWorkload(ctx context.Context, req *ateworke } defer s.locks.Unlock(req.GetActorUid()) + if s.lookupActor(req.GetActorUid()) == nil { + if resp := ateworker.ReuseCheckpoint(req.GetActorDirs().GetCheckpointDir(), req.GetFidelity()); resp != nil { + slog.InfoContext(ctx, "Reusing completed checkpoint from previous attempt", slog.String("actorUID", req.GetActorUid())) + return resp, nil + } + } + ctx, cancel := context.WithCancel(ctx) defer cancel() // Not cancelable: a checkpoint is saving the actor's state. @@ -901,7 +919,15 @@ func (s *AteWorkerService) CheckpointWorkload(ctx context.Context, req *ateworke default: return nil, fmt.Errorf("unsupported snapshot fidelity: %v", fidelity) } - if err := checkpointIsolated(ctx, s.cgroupDelegated, s.cgroupRoot, req.GetActorUid(), req.GetActorDirs(), checkpointFn, terminate); err != nil { + // the sentry leaves its sandbox leaf for the checkpoint cgroup + var memoryLimit uint64 + if hosted != nil { + memoryLimit = pauseMemoryWatch(hosted) + } + if err := checkpointIsolated(ctx, s.cgroupRoot, req.GetActorUid(), req.GetActorDirs(), checkpointFn, terminate); err != nil { + if hosted != nil && memoryLimit != 0 { + resumeMemoryWatch(hosted, memoryLimit) + } return nil, err } @@ -916,7 +942,11 @@ func (s *AteWorkerService) CheckpointWorkload(ctx context.Context, req *ateworke s.actorLogger.EmitLifecycleLog(ctx, "Actor checkpointed", attribution) - return &ateworkerpb.CheckpointWorkloadResponse{SnapshotFiles: snapshotFiles, DataSnapshotFiles: durableFiles}, nil + resp := &ateworkerpb.CheckpointWorkloadResponse{SnapshotFiles: snapshotFiles, DataSnapshotFiles: durableFiles} + if err := ateworker.RecordCheckpoint(checkpointPath, fidelity, resp); err != nil { + slog.WarnContext(ctx, "Failed to record checkpoint completion marker", slog.String("actorUID", req.GetActorUid()), slog.Any("err", err)) + } + return resp, nil } // listSnapshotFiles returns the (relative) names of regular files directly under @@ -1051,6 +1081,9 @@ func (s *AteWorkerService) RestoreWorkload(ctx context.Context, req *ateworkerpb if err := resetRunscStateAndPidFileDirs(req.GetActorDirs()); err != nil { return nil, err } + if err := s.removeStaleSandboxLeaf(ctx, req.GetActorUid()); err != nil { + return nil, err + } err = s.tunnel.Deactivate(ctx, attribution) timing.prep = lap(&tLast) @@ -1088,6 +1121,9 @@ func (s *AteWorkerService) RestoreWorkload(ctx context.Context, req *ateworkerpb var containersToDelete []string defer func() { if retErr != nil { + if hosted.oomKilled.Load() { + retErr = fmt.Errorf("killed for exceeding its memory limit: %w", retErr) + } cleanupCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second) defer cancel() if err := s.tunnel.Deactivate(cleanupCtx, attribution); err != nil { @@ -1131,6 +1167,7 @@ func (s *AteWorkerService) RestoreWorkload(ctx context.Context, req *ateworkerpb return nil, fmt.Errorf("while creating the sandbox: %w", err) } s.pinSandbox(ctx, hosted, req.GetActorDirs()) + s.throttleSandboxMemory(ctx, hosted) err = rcmd.cmdStart(ctx, pw, ocispec.SandboxID) timing.sandboxRestore = lap(&tLast) if err != nil { @@ -1145,6 +1182,7 @@ func (s *AteWorkerService) RestoreWorkload(ctx context.Context, req *ateworkerpb return nil, fmt.Errorf("while creating the sandbox: %w", err) } s.pinSandbox(ctx, hosted, req.GetActorDirs()) + s.throttleSandboxMemory(ctx, hosted) err = rcmd.cmdRestore(ctx, pw, ocispec.SandboxID, checkpointDir) timing.sandboxRestore = lap(&tLast) if err != nil { diff --git a/cmd/ateworker-gvisor/memlimit.go b/cmd/ateworker-gvisor/memlimit.go new file mode 100644 index 0000000000..3a9ed5551f --- /dev/null +++ b/cmd/ateworker-gvisor/memlimit.go @@ -0,0 +1,187 @@ +//go:build linux + +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "context" + "errors" + "fmt" + "io/fs" + "log/slog" + "os" + "path/filepath" + "strconv" + "strings" + "time" + + "go.ate.dev/substrate/internal/ateattr" + "go.ate.dev/substrate/internal/ateworker/cgroup" + "go.ate.dev/substrate/internal/ocispec" +) + +// With runc 1.5.2+ on systemd 253-258 the worker container's scope has +// OOMPolicy=kill, so any kernel OOM kill under it stops the whole worker, even +// one confined to a single actor's leaf. So the sandbox leaf gets memory.high +// at the actor's limit, which throttles without invoking the OOM killer, and +// we kill a sandbox that stays over with cgroup.kill, which systemd ignores. +// https://github.com/opencontainers/cgroups/blob/19a241326c83a5100dff94732733791a0c9e4720/systemd/v2.go#L366-L382 + +// memoryWatchInterval is how often sandbox leaves are checked. +const memoryWatchInterval = 250 * time.Millisecond + +// throttleSandboxMemory swaps the memory.max runsc set on the sandbox leaf for +// a memory.high at the same limit. runsc reads memory.max back to size the +// sentry's --total-memory when create launches the sandbox, so this runs after +// the sandbox's create. On failure the leaf keeps its memory.max. +func (s *AteWorkerService) throttleSandboxMemory(ctx context.Context, hosted *hostedActor) { + if !s.throttleMemory { + return + } + limit, err := swapMemoryMaxForHigh(s.sandboxLeaf(hosted.attribution.UID)) + if err != nil { + attrs := append(ateattr.ActorLogAttrs(hosted.attribution), slog.Any("err", err)) + slog.LogAttrs(ctx, slog.LevelWarn, "Sandbox keeps its memory.max, an OOM kill in it will restart the worker", attrs...) + return + } + hosted.memoryLimit.Store(limit) +} + +// psiEnabled reports whether the kernel tracks memory pressure for cgroups +// under root, which the watch needs to tell a stuck sandbox from a busy one. +func psiEnabled(root string) bool { + _, err := os.Stat(filepath.Join(root, "memory.pressure")) + return err == nil +} + +// removeStaleSandboxLeaf removes a sandbox leaf an earlier activation of the +// actor left behind when its runsc delete never ran. runsc joins an existing +// cgroup as is, so the new sandbox would get no memory.max, a sentry sized to +// the host's RAM, and no watch. +// https://github.com/google/gvisor/blob/0068142bdeb9de91fe72bf9794204de6dec92cb4/runsc/cgroup/cgroup_v2.go#L138-L153 +func (s *AteWorkerService) removeStaleSandboxLeaf(ctx context.Context, actorUID string) error { + dir := s.sandboxLeaf(actorUID) + killCtx, cancel := context.WithTimeout(ctx, 10*time.Second) + defer cancel() + if err := cgroup.KillLeafUnder(killCtx, s.cgroupRoot, filepath.Base(dir)); err != nil { + return fmt.Errorf("while clearing a leftover sandbox cgroup: %w", err) + } + if err := os.Remove(dir); err != nil && !errors.Is(err, fs.ErrNotExist) { + return fmt.Errorf("while removing leftover sandbox cgroup %q: %w", dir, err) + } + return nil +} + +func (s *AteWorkerService) sandboxLeaf(actorUID string) string { + return filepath.Join(s.cgroupRoot, ocispec.GVisorCgroupLeaf(actorUID, sandboxCgroupContainer)) +} + +// swapMemoryMaxForHigh returns the limit it moved, or 0 if the leaf has none. +func swapMemoryMaxForHigh(dir string) (uint64, error) { + b, err := os.ReadFile(filepath.Join(dir, "memory.max")) + if errors.Is(err, fs.ErrNotExist) { + // memory controller not delegated + return 0, nil + } + if err != nil { + return 0, err + } + v := strings.TrimSpace(string(b)) + if v == "max" { + return 0, nil + } + limit, err := strconv.ParseUint(v, 10, 64) + if err != nil { + return 0, fmt.Errorf("parsing memory.max %q: %w", v, err) + } + // high first, so the leaf is never unlimited + if err := os.WriteFile(filepath.Join(dir, "memory.high"), []byte(v), 0o644); err != nil { + return 0, err + } + if err := os.WriteFile(filepath.Join(dir, "memory.max"), []byte("max"), 0o644); err != nil { + return 0, err + } + return limit, nil +} + +// watchMemory kills sandboxes that stay over their memory limit, until ctx is +// done. +func (s *AteWorkerService) watchMemory(ctx context.Context) { + ticker := time.NewTicker(memoryWatchInterval) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return + case <-ticker.C: + for _, h := range s.hostedActors() { + s.checkMemory(ctx, h) + } + } + } +} + +// checkMemory kills h's sandbox if the watch finds it stuck at its limit. The +// kill runs in the background, holding h.memoryMu until the sandbox is gone. +func (s *AteWorkerService) checkMemory(ctx context.Context, h *hostedActor) { + if !h.memoryMu.TryLock() { + // being killed, or paused for a checkpoint + return + } + limit := h.memoryLimit.Load() + if limit == 0 || h.stopping.Load() || h.oomKilled.Load() { + h.memoryMu.Unlock() + return + } + dir := s.sandboxLeaf(h.attribution.UID) + over, current, err := h.memoryWatch.Check(dir, limit, time.Now()) + if err != nil || !over { + // a read fails when the sandbox is going away + h.memoryMu.Unlock() + return + } + h.oomKilled.Store(true) + attrs := append(ateattr.ActorLogAttrs(h.attribution), + slog.Uint64("memoryLimitBytes", limit), + slog.Uint64(string(ateattr.StatsMemoryUsageKey), current)) + slog.LogAttrs(ctx, slog.LevelWarn, "Killing a sandbox over its memory limit", attrs...) + go func() { + defer h.memoryMu.Unlock() + killCtx, cancel := context.WithTimeout(ctx, 10*time.Second) + defer cancel() + if err := cgroup.KillLeafUnder(killCtx, s.cgroupRoot, filepath.Base(dir)); err != nil { + attrs = append(attrs, slog.Any("err", err)) + slog.LogAttrs(ctx, slog.LevelError, "Failed to kill a sandbox over its memory limit", attrs...) + } + }() +} + +// pauseMemoryWatch stops the watch on h's sandbox leaf, e.g. while its sentry +// sits in the checkpoint cgroup, waiting out a kill in progress. It returns +// the limit to resume with. +func pauseMemoryWatch(h *hostedActor) uint64 { + h.memoryMu.Lock() + defer h.memoryMu.Unlock() + return h.memoryLimit.Swap(0) +} + +// resumeMemoryWatch watches h's sandbox leaf again, from a fresh window. +func resumeMemoryWatch(h *hostedActor, limit uint64) { + h.memoryMu.Lock() + defer h.memoryMu.Unlock() + h.memoryWatch = cgroup.MemoryWatch{} + h.memoryLimit.Store(limit) +} diff --git a/cmd/ateworker-gvisor/memlimit_test.go b/cmd/ateworker-gvisor/memlimit_test.go new file mode 100644 index 0000000000..c6fc94a761 --- /dev/null +++ b/cmd/ateworker-gvisor/memlimit_test.go @@ -0,0 +1,135 @@ +//go:build linux + +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "context" + "os" + "path/filepath" + "strconv" + "strings" + "testing" + + "go.ate.dev/substrate/internal/resources" +) + +func writeCgroupFiles(t *testing.T, dir string, files map[string]string) { + t.Helper() + for name, content := range files { + if err := os.WriteFile(filepath.Join(dir, name), []byte(content), 0o600); err != nil { + t.Fatal(err) + } + } +} + +func readCgroupFile(t *testing.T, dir, name string) string { + t.Helper() + b, err := os.ReadFile(filepath.Join(dir, name)) + if err != nil { + t.Fatal(err) + } + return strings.TrimSpace(string(b)) +} + +func TestSwapMemoryMaxForHigh(t *testing.T) { + dir := t.TempDir() + writeCgroupFiles(t, dir, map[string]string{"memory.max": "268435456\n", "memory.high": "max\n"}) + limit, err := swapMemoryMaxForHigh(dir) + if err != nil || limit != 268435456 { + t.Fatalf("swapMemoryMaxForHigh() = %d, %v, want 268435456", limit, err) + } + if got := readCgroupFile(t, dir, "memory.high"); got != "268435456" { + t.Errorf("memory.high = %q, want 268435456", got) + } + if got := readCgroupFile(t, dir, "memory.max"); got != "max" { + t.Errorf("memory.max = %q, want max", got) + } +} + +func TestSwapMemoryMaxForHighNoLimit(t *testing.T) { + for name, files := range map[string]map[string]string{ + "unlimited": {"memory.max": "max\n", "memory.high": "max\n"}, + "no memory controller": {}, + } { + t.Run(name, func(t *testing.T) { + dir := t.TempDir() + writeCgroupFiles(t, dir, files) + if limit, err := swapMemoryMaxForHigh(dir); err != nil || limit != 0 { + t.Errorf("swapMemoryMaxForHigh() = %d, %v, want 0", limit, err) + } + if _, err := os.Stat(filepath.Join(dir, "memory.high")); len(files) == 0 && err == nil { + t.Error("memory.high was created") + } + }) + } +} + +func TestRemoveStaleSandboxLeaf(t *testing.T) { + root := t.TempDir() + s := &AteWorkerService{cgroupRoot: root} + leaf := s.sandboxLeaf("uid-a") + if err := os.Mkdir(leaf, 0o755); err != nil { + t.Fatal(err) + } + if err := s.removeStaleSandboxLeaf(context.Background(), "uid-a"); err != nil { + t.Fatalf("removeStaleSandboxLeaf() = %v", err) + } + if _, err := os.Stat(leaf); !os.IsNotExist(err) { + t.Errorf("leftover leaf still there (stat err: %v)", err) + } + if err := s.removeStaleSandboxLeaf(context.Background(), "uid-a"); err != nil { + t.Errorf("removeStaleSandboxLeaf() without a leaf = %v", err) + } +} + +// A sandbox past the overshoot cap is killed once, and a checkpoint pausing +// the watch waits for the kill. +func TestCheckMemoryKillsOncePausesForCheckpoint(t *testing.T) { + root := t.TempDir() + s := &AteWorkerService{cgroupRoot: root, throttleMemory: true} + h := &hostedActor{attribution: resources.ActorAttribution{UID: "uid-a"}} + leaf := s.sandboxLeaf("uid-a") + if err := os.Mkdir(leaf, 0o755); err != nil { + t.Fatal(err) + } + const limit = 256 << 20 + writeCgroupFiles(t, leaf, map[string]string{ + "memory.current": strconv.Itoa(limit+32<<20) + "\n", + "cgroup.kill": "", + "cgroup.events": "populated 0\nfrozen 0\n", + }) + h.memoryLimit.Store(limit) + + s.checkMemory(context.Background(), h) + if !h.oomKilled.Load() { + t.Fatal("sandbox over its limit was not killed") + } + if got := pauseMemoryWatch(h); got != limit { + t.Errorf("pauseMemoryWatch() = %d, want %d", got, limit) + } + if got := readCgroupFile(t, leaf, "cgroup.kill"); got != "1" { + t.Errorf("cgroup.kill = %q after pausing, want the kill done", got) + } + + writeCgroupFiles(t, leaf, map[string]string{"cgroup.kill": ""}) + resumeMemoryWatch(h, limit) + s.checkMemory(context.Background(), h) + pauseMemoryWatch(h) + if got := readCgroupFile(t, leaf, "cgroup.kill"); got != "" { + t.Errorf("sandbox was killed twice") + } +} diff --git a/cmd/ateworker-gvisor/stats.go b/cmd/ateworker-gvisor/stats.go index 6ddd9bf49d..2b49088485 100644 --- a/cmd/ateworker-gvisor/stats.go +++ b/cmd/ateworker-gvisor/stats.go @@ -302,9 +302,6 @@ func (s *AteWorkerService) sampleSandbox(active *resources.ActorAttribution) (*a // and holds the actor's lock while it does, so a busy actor is skipped. The // cgroup was read before the lock was checked, and h is looked up again after, // so a teardown that starts or finishes in between is never reported. -// -// oom_kill counts over the cgroup's lifetime. A leaf left behind by an earlier -// activation of the same actor carries its count forward. func (s *AteWorkerService) warnDeadSandbox(ctx context.Context, h *hostedActor, raw cgroupstats.Sample) { if h.deadReported.Load() { return diff --git a/cmd/ateworker-microvm/checkpoint.go b/cmd/ateworker-microvm/checkpoint.go index 27a0a77092..4686651f98 100644 --- a/cmd/ateworker-microvm/checkpoint.go +++ b/cmd/ateworker-microvm/checkpoint.go @@ -30,6 +30,7 @@ import ( "go.ate.dev/substrate/cmd/ateworker-microvm/internal/ch" "go.ate.dev/substrate/cmd/ateworker-microvm/internal/kata" + "go.ate.dev/substrate/internal/ateworker" "go.ate.dev/substrate/internal/ateworker/phaselog" "go.ate.dev/substrate/internal/ateworkervalidation" "go.ate.dev/substrate/internal/imagecache" @@ -78,6 +79,13 @@ func (s *AteWorkerService) CheckpointWorkload(ctx context.Context, req *ateworke } defer s.locks.Unlock(req.GetActorUid()) + if s.lookupActor(req.GetActorUid()) == nil { + if resp := ateworker.ReuseCheckpoint(req.GetActorDirs().GetCheckpointDir(), req.GetFidelity()); resp != nil { + slog.InfoContext(ctx, "Reusing completed checkpoint from previous attempt", slog.String("actorUID", req.GetActorUid())) + return resp, nil + } + } + ctx, cancel := context.WithCancel(ctx) defer cancel() defer s.inFlight.Add(req.GetActorUid(), rpcCheckpointWorkload, nil)() @@ -252,7 +260,11 @@ func (s *AteWorkerService) CheckpointWorkload(ctx context.Context, req *ateworke slog.Duration("durable_dir", dDurable), slog.Duration("rootfs_upper", dUpper), slog.Duration("teardown", dTeardown)) s.recordFinalIfEnded(ctx, hosted) - return &ateworkerpb.CheckpointWorkloadResponse{SnapshotFiles: snapshotFiles, DataSnapshotFiles: durableFiles}, nil + resp := &ateworkerpb.CheckpointWorkloadResponse{SnapshotFiles: snapshotFiles, DataSnapshotFiles: durableFiles} + if err := ateworker.RecordCheckpoint(checkpointDir, scope, resp); err != nil { + slog.WarnContext(ctx, "Failed to record checkpoint completion marker", slog.String("actorUID", actorUID), slog.Any("err", err)) + } + return resp, nil } // snapshotVMState captures the paused guest into checkpointDir: the CH snapshot diff --git a/cmd/ateworker-microvm/hosted.go b/cmd/ateworker-microvm/hosted.go index 3447e9ed9b..9b2fe6c69c 100644 --- a/cmd/ateworker-microvm/hosted.go +++ b/cmd/ateworker-microvm/hosted.go @@ -30,7 +30,6 @@ import ( "go.ate.dev/substrate/internal/atunnel/ingress" "go.ate.dev/substrate/internal/netns" "go.ate.dev/substrate/internal/resources" - "go.ate.dev/substrate/internal/sizing" "go.ate.dev/substrate/internal/workerstats" ) @@ -106,24 +105,14 @@ func (s *AteWorkerService) hostActor(ctx context.Context, attribution resources. return hosted, nil } -// actorLeaf opens the actor's cgroup, or returns nil when the worker has none. -func (s *AteWorkerService) actorLeaf(actorUID string, size sizing.SandboxSize) (*cgroup.ActorLeaf, error) { - if !s.actorCgroups { - return nil, nil - } - return cgroup.OpenActorLeaf(actorUID, size.MilliCPU) -} - // cleanupSandboxState kills what is left of the actor's host processes and // clears its sandbox directories. func (s *AteWorkerService) cleanupSandboxState(ctx context.Context, actorUID string) { - if s.actorCgroups { - killCtx, cancel := context.WithTimeout(ctx, 5*time.Second) - if err := cgroup.KillActorLeaf(killCtx, actorUID); err != nil { - slog.WarnContext(ctx, "Failed to kill the actor's leftover processes", slog.Any("err", err)) - } - cancel() + killCtx, cancel := context.WithTimeout(ctx, 5*time.Second) + if err := cgroup.KillActorLeaf(killCtx, actorUID); err != nil { + slog.WarnContext(ctx, "Failed to kill the actor's leftover processes", slog.Any("err", err)) } + cancel() kata.CleanupSandboxState(ctx, actorUID) } @@ -147,10 +136,8 @@ func (s *AteWorkerService) unhostActor(ctx context.Context, actorUID string) err s.actorsMu.Unlock() }() - if s.actorCgroups { - if err := cgroup.RemoveActorLeaf(actorUID); err != nil { - slog.WarnContext(ctx, "Failed to remove the actor's cgroup", slog.Any("err", err)) - } + if err := cgroup.RemoveActorLeaf(actorUID); err != nil { + slog.WarnContext(ctx, "Failed to remove the actor's cgroup", slog.Any("err", err)) } if hosted.network == nil { return nil diff --git a/cmd/ateworker-microvm/internal/third_party/kata/PROVENANCE.md b/cmd/ateworker-microvm/internal/third_party/kata/PROVENANCE.md index b884f7f647..d75a1d5d1e 100644 --- a/cmd/ateworker-microvm/internal/third_party/kata/PROVENANCE.md +++ b/cmd/ateworker-microvm/internal/third_party/kata/PROVENANCE.md @@ -5,10 +5,10 @@ and used by `cmd/ateworker-microvm`. The upstream `LICENSE` in this directory co everything under it. - **Upstream:** github.com/kata-containers/kata-containers -- **Version:** tag `3.31.0`. The micro-VM demo's runtime assets track a newer kata - (see `hack/microvm-assets/assemble.sh` `KATA_VER`, currently `4.1.0`); the four - vendored `.proto` files are byte-identical between `3.31.0` and `4.1.0` (verified - 2026-09-16), so the generated Go needs no re-vendor. +- **Version:** tag `3.31.0`. The microVM demo's runtime assets track a newer kata + (see `hack/microvm-assets/assemble.sh` `KATA_VER`, currently `4.3.1`); the four + vendored `.proto` files are byte-identical between `3.31.0` and `4.3.1` (verified + 2026-10-23), so the generated Go needs no re-vendor. - **License:** Apache-2.0 — `./LICENSE` is the upstream license verbatim (mirrored to `LICENSES/third_party/kata/LICENSE` by `hack/update/licenses.sh`); the per-file copyright headers (HyperHQ, Ant Group, Intel, Databricks) are retained verbatim in diff --git a/cmd/ateworker-microvm/main.go b/cmd/ateworker-microvm/main.go index 2e8cdc3830..9f8aac1783 100644 --- a/cmd/ateworker-microvm/main.go +++ b/cmd/ateworker-microvm/main.go @@ -225,8 +225,7 @@ func do(ctx context.Context) error { actorLogger := actorlog.NewActorLogger(logWriter, metadata.OnGCE()) // Give each actor's VMM and virtiofsd a cgroup of their own, so one busy // guest cannot starve the rest. - actorCgroups, err := cgroup.Delegate(ctx) - if err != nil { + if err := cgroup.Delegate(ctx); err != nil { return fmt.Errorf("while delegating the worker cgroup: %w", err) } @@ -235,7 +234,6 @@ func do(ctx context.Context) error { return err } ateworkerService := NewService(*podUID, *chBinary, *guestDebug, *vmmMemReserve, *maxActors, tunnel, actorLogger) - ateworkerService.actorCgroups = actorCgroups // The controller sets both from the downward API. pool := workerstats.Pool{Namespace: os.Getenv("POD_NAMESPACE"), Name: os.Getenv("WORKER_POOL_NAME")} if pool.Namespace == "" || pool.Name == "" { @@ -397,9 +395,6 @@ type AteWorkerService struct { // Actors undergoing network cleanup still count against capacity. draining int maxActors int - // actorCgroups is set when the worker's cgroup is delegated, so each actor's - // VMM and virtiofsd run in a leaf of their own. - actorCgroups bool } var _ ateworkerpb.WorkerServer = (*AteWorkerService)(nil) diff --git a/cmd/ateworker-microvm/restore.go b/cmd/ateworker-microvm/restore.go index 98a9b0e1c1..fae3772575 100644 --- a/cmd/ateworker-microvm/restore.go +++ b/cmd/ateworker-microvm/restore.go @@ -33,6 +33,7 @@ import ( "go.ate.dev/substrate/internal/workerstats" "go.ate.dev/substrate/internal/ateworker/actornet" + "go.ate.dev/substrate/internal/ateworker/cgroup" "go.ate.dev/substrate/cmd/ateworker-microvm/internal/ch" "go.ate.dev/substrate/cmd/ateworker-microvm/internal/kata" @@ -339,7 +340,7 @@ func (s *AteWorkerService) restoreMemoryFidelity(ctx context.Context, p actorBoo return untarErr } tUpper := time.Now() - leaf, err := s.actorLeaf(actorUID, p.size) + leaf, err := cgroup.OpenActorLeaf(actorUID, p.size.MilliCPU) if err != nil { return err } diff --git a/cmd/ateworker-microvm/run.go b/cmd/ateworker-microvm/run.go index 926dcbfa83..1b2fda9fc3 100644 --- a/cmd/ateworker-microvm/run.go +++ b/cmd/ateworker-microvm/run.go @@ -33,6 +33,7 @@ import ( "go.ate.dev/substrate/internal/workerstats" "go.ate.dev/substrate/internal/ateworker/actornet" + "go.ate.dev/substrate/internal/ateworker/cgroup" specs "github.com/opencontainers/runtime-spec/specs-go" "go.ate.dev/substrate/cmd/ateworker-microvm/internal/ch" @@ -498,7 +499,7 @@ func (s *AteWorkerService) coldBootActor(ctx context.Context, p actorBootParams) // host upper, mounted into the shared dir) + durable-dir and CSI volumes (if any), // and start the ONE virtiofsd that serves them all. CH connects to it at vm.create // and demand-pages for the actor's lifetime, so ateworker owns the process (killed in teardownActor). - leaf, err := s.actorLeaf(actorUID, p.size) + leaf, err := cgroup.OpenActorLeaf(actorUID, p.size.MilliCPU) if err != nil { return err } diff --git a/cmd/benchmarking/boomer-worker/main.go b/cmd/benchmarking/boomer-worker/main.go index f07b6d5ed2..5611ec3fd3 100644 --- a/cmd/benchmarking/boomer-worker/main.go +++ b/cmd/benchmarking/boomer-worker/main.go @@ -46,7 +46,7 @@ import ( func main() { var ( apiEndpoint = flag.String("api-endpoint", "k8s:///api.ate-system.svc.cluster.local:443", "ateapi gRPC dial target.") - routerURL = flag.String("router-url", "http://atenet-router.ate-system.svc.cluster.local", "atenet HTTP router base URL (no trailing slash).") + routerURL = flag.String("router-url", boomerutil.DefaultRouterURL, "atenet HTTPS router base URL (no trailing slash).") atespace = flag.String("atespace", "benchmark", "Atespace every actor this worker creates lives in. Ensured (CreateAtespace, AlreadyExists is ok) at startup.") promAddr = flag.String("prometheus-addr", ":8001", "Address for the Prometheus /metrics endpoint.") configJSON = flag.String("config-json", "", "Initial dynconfig as a JSON object keyed by locust flag name in snake_case (trace_probability, min_wait_time, ...); the selected user class reads the keys it documents and ignores the rest. Unset keys keep the class's built-in defaults. Each payload the master serves later stands on its own in the same way, so a key it leaves null is back at the default.") @@ -90,6 +90,12 @@ func main() { os.Exit(1) } + if !strings.HasPrefix(*routerURL, "https://") { + slog.Error("fatal: --router-url must use https://", + slog.String("router_url", *routerURL)) + os.Exit(1) + } + slog.SetDefault(slog.New(slog.NewJSONHandler(os.Stdout, nil))) if *checkAgentSessionScript != "" { @@ -143,6 +149,12 @@ func main() { } defer conn.Close() + routerTLS, err := boomerutil.RouterTLSConfig() + if err != nil { + slog.Error("failed to configure router TLS", slog.String("err", err.Error())) + os.Exit(1) + } + // Every VU talks to the one router address, so the transport must keep // as many idle connections as there are VUs. The default keeps two per // host; at thousands of VUs each request then opens and closes its own @@ -151,7 +163,9 @@ func main() { // outlast a VU's gap between requests, which is the wake window plus the // suspend/resume round trip. transport := http.DefaultTransport.(*http.Transport).Clone() - transport.MaxIdleConns = 0 // no total cap; the per-host cap governs + transport.TLSClientConfig = routerTLS + transport.ForceAttemptHTTP2 = false // keep one HTTP/1.1 connection per VU across Envoy threads + transport.MaxIdleConns = 0 // no total cap; the per-host cap governs transport.MaxIdleConnsPerHost = *httpMaxIdleConnsPerHost transport.IdleConnTimeout = 5 * time.Minute httpClient := &http.Client{Timeout: 30 * time.Second, Transport: transport} diff --git a/cmd/benchmarking/egress/README.md b/cmd/benchmarking/egress/README.md new file mode 100644 index 0000000000..2c82bd8110 --- /dev/null +++ b/cmd/benchmarking/egress/README.md @@ -0,0 +1,514 @@ +# benchmark-egress + +`benchmark-egress` is the workload of the egress scale test. It holds many +CONNECT tunnels open through the egress gateway at a fixed pace and reports what +each tunnel saw, so that a run can price what one tunnel costs the gateway. + +## 1. What the benchmark measures + +The benchmark sizes one `atenet-egress` replica for a target fleet: A actors, +each reaching C destinations. It reports what one tunnel and one request cost +the gateway, so that the replica count for a fleet follows from fleet size, +one replica's CPU and memory, and these costs. + +Three terms: + +- **Tunnel.** One kept-alive CONNECT from one actor to one destination + Service, opened by the actor's first request and reused by every later one. + It is the unit the gateway's cost scales with: a fleet of A actors over C + Services holds A × C tunnels. +- **Blended.** The window from the first actor's start until the last actor + has opened its last tunnel. New tunnels are being set up while tunnels + already open carry repeat requests, so the gateway's CPU mixes setup with + ordinary requests. +- **Settled.** The period, after the blended window, in which every tunnel + has been reused and every request reuses one. Nothing is set up; the + gateway's cost is per request. + +The settled period gives the cost of one request directly. The blended window +gives the cost of setting up one tunnel once its requests are priced at the +settled rate. + +```mermaid +flowchart LR + subgraph run ["one cell, in time"] + t0(["t0
first /start"]) --> B["blended window
A tunnels open per interval,
repeats on tunnels already open"] + B --> t1(["t1
last tunnel open"]) + t1 --> W["until every tunnel is reused
(rounds ≥ 2, newConns flat)"] + W --> t2(["t2"]) + t2 --> S["settled window
A requests per interval,
nothing set up"] + S --> t3(["t3
/stop"]) + end + subgraph results ["what the cell reports"] + RB["gateway CPU per tunnel opened"] + RS["settled req/s per core
p50, p99, p99.9, success rate
Envoy heap"] + end + B -. "gateway CPU over t0..t1,
minus its requests at the settled price" .-> RB + S -. "read over t2..t3" .-> RS +``` + +A cell is one fleet size and one protocol. It reports: + +- **Settled req/s per core.** Throughput per gateway core once every tunnel + exists. Its inverse is the CPU cost of one request. +- **Gateway CPU per tunnel opened.** CPU milliseconds to set up one tunnel, the + cost of its requests taken out. +- **Envoy heap per tunnel.** KiB of heap one held tunnel adds. +- **Settled p50, p99, p99.9 and success rate.** Latency of repeat requests as + the actor sees it, and the share that got HTTP 200. Flat across fleet sizes + means the gateway was not saturated while the costs were read. + +The costs price held tunnels, not tunnels that reconnect. Section 6 lists the +checks a cell must pass before its numbers count. + +One replica's figures size any fleet, on the assumption that the gateway's +cost is linear in tunnels and requests; the heap slope and the latencies across +fleet sizes are the checks on it. The fleet's memory is A × C tunnels times +heap per tunnel; its CPU is the request rate times CPU per request, plus the +tunnel open rate times CPU per tunnel opened. Each, divided by what one +replica has, is a replica count, and the fleet needs the larger of the two. + +```mermaid +flowchart LR + subgraph fleet ["fleet to size"] + T["A × C tunnels"] + R["request rate,
tunnel open rate"] + end + subgraph measured ["measured here, on one replica"] + H["heap per tunnel"] + Q["CPU per request,
CPU per tunnel opened"] + end + T --> M["memory needed
= A × C × heap per tunnel"] + H --> M + R --> P["CPU needed
= rate × CPU per request
+ open rate × CPU per tunnel opened"] + Q --> P + M -- "÷ one replica's memory" --> NM["replicas, by memory"] + P -- "÷ one replica's cores" --> NC["replicas, by CPU"] + NM --> N["replica count = max of the two"] + NC --> N +``` + +## 2. Topology + +```mermaid +flowchart LR + D[driver] -- "/start, /stats, /collect, /stop
through the router" --> A["A actors"] + D -- "gateway CPU" --> P[Prometheus] + A -- "C CONNECT tunnels,
one GET per period on each" --> G["egress gateway
(proxy, the system under test)"] + G -. "forwarded" .-> S["Services 0 to C-1"] --> T[target Deployment] + P -- "scrapes cAdvisor
on the gateway's node" --> G +``` + +Each dotted arrow is a connection an actor asked for: the actor dials a +Service name, and the gateway checks the egress policy and forwards the bytes. + +- **`benchmark-egress actor`** runs inside each Actor and serves its control + routes on port 80. Once started, it holds one CONNECT tunnel per endpoint + through the gateway and sends a `GET` on each at a fixed pace, and it serves + what it measures as window cuts over `/collect`. The actor's egress policy + must name every endpoint hostname. Section 3 has the routes and the + workflow, section 5 the window cuts. +- **Egress gateway.** One `atenet-egress` replica in its sdsmint deployment, + the system under test. It carries one CONNECT tunnel per actor and + destination, terminates the actor's TLS on HTTPS runs, and checks the egress + policy on every request. +- **Endpoints and `benchmark-egress target`.** C ClusterIP Services, + `egress-target-0` to `egress-target-`, in namespace + `egress-tests-targets`, all in front of one Deployment of + `benchmark-egress target`. Each Service is a distinct hostname, so each is + its own tunnel. The target answers every `GET /` with a 200 and a small fixed + body, over plain HTTP and, given a certificate, HTTPS. Each Service maps to + its own backend port, from the ranges in `--listen-ports` and + `--tls-listen-ports`: Services on distinct ClusterIPs that share one backend + port can hand the target two connections with the same 4-tuple. The driver + issues the HTTPS certificate for `*.egress-tests-targets.svc.cluster.local` + from a throwaway CA that is name-constrained to + `egress-tests-targets.svc.cluster.local`. +- **Prometheus.** Scrapes the gateway node's cAdvisor for the gateway's CPU. + It belongs to the benchmark's deployment, not to the actor, which does not + know it exists. +- **Driver.** Sets up the run and starts the A actors. It reaches each actor + through the atenet router: it posts `/start`, polls `/stats` for the gates, + posts `/collect` at the instants that bound each window, and posts `/stop`. + It merges the cuts and reads the gateway's CPU from Prometheus at the same + instants, section 6. + +On HTTPS runs one request crosses two TLS sessions, each trusting a different +CA: + +```mermaid +sequenceDiagram + participant A as actor + participant G as egress gateway + participant T as target + A->>G: CONNECT egress-target-i:443 + Note over A,G: egress policy checked + A->>G: TLS handshake + Note over A,G: gateway terminates with a MITM cert
actor trusts the MITM CA via SSL_CERT_FILE + G->>T: TLS handshake + Note over G,T: target presents *.egress-tests-targets.svc.cluster.local
gateway trusts the driver's throwaway CA + A->>G: GET / + G->>T: GET / + T-->>G: 200 + G-->>A: 200 +``` + +The deploy script and the telemetry stack, in `benchmarking/egress-tunnels/`, +and the driver, in this directory, are later PRs. + +## 3. What the actor does + +The actor serves `/readyz`, `/start`, `/stats`, `/collect` and `/stop` on +port 80, and every answer carries `Benchmark-Egress-Protocol: 2`, which the +driver requires so another contract is refused rather than misread: the images +that pushed their measurements over OTLP sent 1 and served no `/collect`. It +has two states, idle and running, and every code the driver sees is in this +table: + +| Route | Idle | Running | +|---|---|---| +| `GET /readyz` | 200 | 200 | +| `POST /start` | 204, or 400 for a bad body | 409 | +| `GET /stats` | 404 | 200 | +| `POST /collect?window=N` | 404 | 200 with the window; 400 for a missing or malformed `window`, 409 for an id out of sequence | +| `POST /stop` | 200 with the last loop's final counters and tail window, 404 if none | 200 with the final counters once every goroutine has exited | + +A run, from the actor's side: + +1. **Boot idle.** The actor sends nothing until `/start`, because actors + restore from a golden snapshot taken before any egress policy exists. + `/readyz` answers 200 from boot. +2. **`POST /start`.** The driver sends one JSON object; unknown fields are + ignored. `endpoints` is required, the rest default. + + | Field | Meaning | + |---|---| + | `endpoints` | C, 1 to 1024. | + | `intervalMs` | Pacing. Endpoint *i* opens *i* intervals after `/start`, and each endpoint is then requested once per C × `intervalMs`. Zero sends back to back. | + | `offsetMs` | This actor's phase: endpoint *i* opens at `offsetMs` + *i* × `intervalMs`. At most C × `intervalMs`; see below for how a driver picks it. | + | `scheme` | `http` (default) or `https`. | + | `requestTimeoutMs` | Bound on one request, handshake included; default 5000. | + + ```json + {"endpoints": 1000, "intervalMs": 100, "scheme": "https"} + ``` + + The actor creates its instruments, starts one goroutine per endpoint, and + answers 204 as soon as they exist, before any request is sent. It does not + probe the endpoints. +3. **The loop.** Goroutine *i* waits `offsetMs` + *i* × `intervalMs`, so C + handshakes do not reach the gateway at once, then sends its first `GET`. + When that completes, success or not, the endpoint counts as opened; when + the last one opens, the actor stamps `openingDone`. From then on it sends + one `GET` per period on a ticker started at the open, with at most one + request in flight per endpoint. A request that outlasts the period delays + only its own endpoint, whose next request starts as soon as the slow one + returns. The client keeps one connection per endpoint and never closes it + for idling. +4. **`GET /stats`.** The driver polls it for the gates of section 6. A read + consumes nothing; the payload is in section 5. +5. **`POST /collect?window=N`.** The driver posts it at the window edges. It + cuts window N: the counters and the four distributions since the previous + cut, under the lock every record takes, so no sample lands between the two. + Window ids start at 1 and the first cut covers everything since `/start`. + The id after the last cut's cuts a new window; the last cut's id answers + that cut again from a cache, so a lost answer costs a retry rather than a + window; any other id is 409 and the body names the next id. A missing or + malformed `window` is 400. `?endpoints=1` works as on `/stats`. The payload + is in section 5. +6. **`POST /stop`.** The actor waits for every goroutine, releases the + instruments, and answers like a `/collect` for what is left since the last + cut, the tail window, with the final counters of section 5. A request cut + off by the stop is not counted. + +What goes wrong, and what each side does about it: + +- A missing egress policy or Service does not fail `/start`; it shows up as + errors in `/stats`. +- A restarted actor is idle again with nothing kept, and after the next + `/start` its counters begin from zero. The 404 or the drop tells the driver + the actor restarted; section 6 says what it does with that. +- A `/collect` whose answer was lost is asked again with the same id and + answered from the cache. A second reader, or a skipped id, gets 409. +- A `/stop` whose answer was lost is asked again: the actor keeps the final + counters and the tail window until the next `/start`. +- A redial, one more in `newConns`, means the gateway cut the tunnel or a + request on it timed out. The gateway resets a tunnel that stays quiet for + its stream idle timeout, 5 minutes by default, so keep the per-endpoint + period, C × `intervalMs`, under it. + +Only an actor's phase modulo `intervalMs` matters to the fleet: once started +it sends at `/start` + `offsetMs` + k × `intervalMs`. A driver gives actor k +of A `offsetMs` = ⌊k × `intervalMs` / A⌋, which spreads the phases evenly; a +uniform draw from [0, `intervalMs`) does about as well but needs its seed +recorded. Four actors with `intervalMs` 100: + +| Offsets | Fleet sends in each interval | +|---|---| +| 0, 0, 0, 0 | 4 at 0 ms | +| 0, 25, 50, 75 | 1 at 0, 25, 50, 75 ms | + +The offset lives on the actor because the router's fan-out of `/start` sets +each actor's real phase, and the driver does not control it. + +With these rules an actor sends one request per interval throughout the run, +and holds C tunnels. + +## 4. Traffic shape over a run + +The fleet's load on the gateway is meant to be flat within each phase and to +change only at the phase edges, so that a window's cost is the cost of one +kind of work. A run has four phases: + +| Phase | Ends when | Each actor sends | The fleet sends per interval | The gateway sets up | +|---|---|---|---|---| +| Start | Every actor has answered `/start`. | Nothing yet. | Nothing. | Nothing. The router fans `/start` out, and each actor's real phase is the moment its `/start` arrived plus its offset. | +| Opening | The last actor opens its last endpoint: the end of the blended window. | Its first request to one more endpoint per interval, and repeat requests on the endpoints already open. | A opens plus the repeats. | A tunnels per interval, for C intervals. | +| Settled | The driver posts `/stop`. | One request per interval; each endpoint once per C intervals. | A requests, spread across the interval by the offsets. | Nothing. Every tunnel is held and reused. | +| Stop | Every actor has answered `/stop`. | Nothing; requests in flight finish. | Nothing. | Nothing. A × C tunnels close as the actors' clients exit. | + +Within an actor the C endpoints are one interval apart, and each keeps its +own period from the moment its open started, so the actor's rate is one +request per interval however long any request takes. Across the fleet the +offsets put the A actors on distinct points of one interval, so the gateway +sees A requests spread over each interval rather than A at once. + +A few things in a run are coordinated across actors and would, left alone, +put a periodic burst on top of the flat rate: + +| Event | Period | Load | How the design keeps it out of a window's cost | +|---|---|---|---| +| The driver's `/stats` poll | The poll period. | A requests through the ingress router, none through the gateway. | The driver spreads the polls over the poll period; a read consumes nothing. | +| Tunnel resets on idle | C × `intervalMs`. | Every endpoint redials once per period: a second opening wave each period. | A cell keeps C × `intervalMs` under the gateway's stream idle timeout, 5 minutes by default; `longestGap` and `newConns` show when it did not. | +| The gateway's DNS refresh | 60 s per hostname. | One lookup per hostname. | The hostnames were first resolved one per interval, so their refreshes are spread the same way. | +| `/stop` | Once. | A × C tunnel closes within the fan-out window. | The stop edge lies outside both windows; nothing is read across it. | +| The collect round | Once per window edge: `/collect` at `t1` and `t2`, `/stop` at `t3`. | A requests through the ingress router, none through the gateway. | The edges are the only instants read; an actor that timed out is retried by id and is the same window, and nothing periodic runs in the settled window. | + +## 5. What the actor reports + +The actor reports on two paths. `GET /stats` answers one small JSON object, +the counters cumulative since `/start`, for a driver's gates and progress: +poll it, it consumes nothing. Everything a result is computed from, the same +counters and the four latency distributions, the actor serves as window cuts +over `POST /collect?window=N`, one cut per window edge, and the driver merges +them. The actor keeps one piece of window state, the last cut keyed by its id, +so a lost answer is asked for again and costs a retry and nothing else. + +### 5.1 `GET /stats` and `POST /stop` + +Durations are Go `time.Duration` values, nanoseconds. `/collect` and `/stop` +answer the same object with `distributions` added, section 5.2. The +per-endpoint list comes only with `?endpoints=1`, on any of the three, a +repeated `/collect` or `/stop` included. + +| Field | What it holds | Used for | +|---|---|---| +| `scheme` | `http` or `https`. | The scheme in use. | +| `elapsed` | Time since `/start`. | Rates between two polls. | +| `requests`, `successes` | Requests finished; HTTP 200 with no error is a success. | Progress; a `requests` that went down means a restart. | +| `newConns` | Requests sent on a freshly dialed connection. | One per endpoint is the tunnel. Any more is a reconnect after the gateway cut a tunnel or a request on it timed out. | +| `opened` | Endpoints whose first request has completed. | Progress of this actor's opens; the gate on C. | +| `openingDone` | Elapsed when the last endpoint opened; zero until then. | Added to this actor's `/start` time, the latest across actors ends the blended window. | +| `requestsAtOpeningDone` | `requests` at that same moment; zero until then. | This actor's request count for its own opening, section 6. | +| `rounds` | Requests completed by the least-visited endpoint. | Two means every tunnel has been reused; the settled gate. | +| `longestGap` | Longest time between two consecutive requests on one endpoint. | Whether any tunnel went quiet long enough to be reset. | +| `errors` | Failures by class, such as `timeout`, `connection reset`, `HTTP 403`; one entry per class seen. | One class per failure mode. | +| `endpoints` | Per endpoint, by number: `requests`, `errors`, `longestGap`. `?endpoints=1` only. | Naming a failing or quiet tunnel. | + +At C = 1024 a `/stats` is about 250 bytes and the endpoint list about 60 KB +more. + +```json +{ + "scheme": "https", "elapsed": 612000000000, + "requests": 61440, "successes": 61437, "newConns": 1024, "opened": 1024, + "openingDone": 102400000000, "requestsAtOpeningDone": 1536, + "rounds": 60, "longestGap": 102600000000, + "errors": {"timeout": 3} +} +``` + +### 5.2 Window cuts + +A cut, the answer to `/collect` and `/stop`, holds the counters of 5.1 and +four distributions under `distributions`, each recorded since the previous +cut, or since `/start` for the first one: + +| Distribution | What it holds | +|---|---| +| `opening` | Successful first requests: tunnel setup latency. | +| `latency` | Successful repeat requests: the settled quantiles, no handshakes in them. | +| `dns`, `tls` | Lookup and handshake times, when a request did one. Both are absent from a settled window's cut. | + +Each distribution is the OTel histogram data point of its window: + +| Field | What it holds | +|---|---| +| `bounds` | Upper bounds of the finite buckets, seconds. | +| `bucketCounts` | One count per bucket, plus the overflow bucket last. Per bucket, not cumulative. | +| `count`, `sum` | Samples in the window and their total, seconds. | +| `min`, `max` | Smallest and largest sample in the window. | + +A distribution with no sample in the window is absent. `max` is exact. + +The histograms share one set of bounds, 20 per decade from 1 µs to 63.1 s, +157 values, the same in every actor and every image, and the bounds travel in +the payload, so a merge across actors with differing bounds is refused and +names the differing index. Neighboring bounds differ by about 12%, which sets +the resolution: a quantile is within one bucket step of the true value, and +two runs whose quantiles fall within one step are indistinguishable. + +The per-endpoint rows are not in a cut unless `?endpoints=1` asks for them: at +C = 1024 across a thousand actors they would be a million rows in every round. +They stay for a person debugging one actor. + +At C = 1024 a `/collect` is about 7 KB, and about 63 KB with `?endpoints=1`. + +```json +{ + "scheme": "https", "elapsed": 612000000000, + "requests": 61440, "successes": 61437, "newConns": 1024, "opened": 1024, + "openingDone": 102400000000, "requestsAtOpeningDone": 1536, + "rounds": 60, "longestGap": 102600000000, + "errors": {"timeout": 3}, + "distributions": { + "latency": { + "bounds": [0.000001, 0.000001122, "... 157 values ...", 63.1], + "bucketCounts": [0, 0, "... 158 values ...", 0], + "count": 60413, "sum": 181.2, "min": 0.00091, "max": 2.31 + } + } +} +``` + +## 6. What a run computes + +The driver is the next PR. This section is the contract it implements. + +The driver polls every actor's `/stats` for the gates and records four +instants: `t0`, the first `/start`; `t1`, when every actor's `opened` is C, +the end of the blended window; `t2`, when every actor's `rounds` is at least +two and `newConns` is flat between two polls, the start of the settled window; +`t3`, `t2` plus the cell's duration, when it posts `/stop`. At `t1` and `t2` +it posts one `/collect` to every actor with the same window id, so a window is +one collect round across the fleet; a round that fails or times out for some +actors is retried for those actors with the same id and is the same window. +The first cut covers everything since `/start`, and the `/stop` at `t3` is the +last cut, the settled window. The gateway's CPU is +read from Prometheus at the same instants, so the gateway's CPU and the +actors' counts are read at the instants the driver chose, and no lag +correction is needed. + +The merge across actors happens in the driver, per field: + +| Rule | Fields | +|---|---| +| sum | `requests`, `successes`, `newConns`, `opened`, `errors` per class | +| max | `openingDone`, `longestGap` | +| min | `rounds` | +| `Merge`, bucket by bucket | each of `distributions`: bounds must be equal; `bucketCounts`, `count` and `sum` are added; `min` and `max` taken | +| every actor answered | presence: a round is merged only once every actor has answered it | + +| Result | From the round | +|---|---| +| Requests in a window | `requests` summed over the window's cuts | +| Success rate | `successes` over `requests`, both summed | +| Settled p50, p99, p99.9 | `Quantile` on the merged `latency`: the upper bound of the bucket that holds the rank, capped at `max`; a rank in the overflow bucket reads as `max` | +| Settled maximum | `max` of the merged `latency` | +| Blended end | per actor, its `/start` time plus `openingDone`; the latest is `t1` | +| Gateway CPU in a window | `sum(container_cpu_usage_seconds_total{pod=~"atenet-egress.*", container=~"envoy|ext-proc|sdsmint"})` as an instant query at each edge, from the cAdvisor scrape of the gateway's node; the window is the difference | +| Presence | every actor answered the round | + +`rate` and `increase` are not used for the gateway's CPU: they extrapolate +over the range and lose the first sample. + +The results of a cell come from two windows. The blended window runs from the +first `/start` to the latest `openingDone`, each actor's shifted by its own +`/start` time. An actor's `openingDone` counts from its `/start`, so each +actor's window also covers its `offsetMs`. The settled window starts once +every actor has opened, every actor's `rounds` has reached two, checked per +actor before the sum, and `newConns` is flat between two polls. Two actors and +two endpoints, 100 ms apart, both started at the same time, actor 2 with a +50 ms offset and a slow second open: + +```mermaid +sequenceDiagram + autonumber + participant A1 as Actor 1 + participant A2 as Actor 2 + participant S0 as Service 0 + participant S1 as Service 1 + rect rgb(240, 248, 255) + A1->>+S0: open (t=0ms) + S0-->>-A1: done (t=20ms) + A2->>+S0: open (t=50ms, offset) + S0-->>-A2: done (t=70ms) + A1->>+S1: open (t=100ms) + S1-->>-A1: done (t=120ms) + A2->>+S1: open, SLOW (t=150ms) + Note over A1,S1: Repeat requests begin while the slow open is still pending + A1->>S0: repeat (t=200ms) + A2->>S0: repeat (t=250ms) + A1->>S1: repeat (t=300ms) + S1-->>-A2: latest openingDone (t=320ms) + end + Note over A1,S1: Settled, after the blended window + A2->>S1: repeat (t=350ms) + A1->>S0: repeat (t=400ms) + A2->>S0: repeat (t=450ms) + A1->>S1: repeat (t=500ms) +``` + +Both actors' first repeats fall inside the blended window, because it ends +when the slowest actor finishes its last open. The offset puts actor 2's sends +halfway between actor 1's, so the two together send one request every 50 ms +rather than two every 100 ms; without it every send of actor 2 would coincide +with one of actor 1's. The gateway's CPU in that window paid for four tunnel +setups and for every request in it; the settled rate prices every request, the +first on each tunnel included, and what is left is the setups. `t1` is the +edge the driver reads the gateway's CPU against, and the first cut's +`requests`, summed across actors, is the window's request count over +`t0..t1`; `requestsAtOpeningDone`, summed from `/stats`, is the count at each +actor's own opening end, for the opening alone. + +| Result | Source | How it is computed | +|---|---|---| +| Settled req/s per gateway core | actors + gateway | Requests completed in the settled window divided by the gateway pod's CPU-seconds in the same window, both over `t2..t3`: the settled cut's `requests` summed, and the gateway's CPU read at the two edges. | +| Gateway CPU per tunnel opened | actors + gateway | The blended window's gateway CPU, minus every request in it priced at the settled per-request cost, divided by the tunnels opened. What remains is setup alone, independent of pacing, on the assumption that a request on an open tunnel costs the same in either window. Computed after the settled window, since it needs the settled rate. | +| Envoy heap | gateway | The maximum of `server.memory_allocated` in the cell; per-tunnel cost is the slope across fleet sizes. | +| Settled p50, p99, p99.9 | actors | `Quantile` on the settled cut's merged `latency`, reported as a bucket upper edge with a bucket resolution of 12%. | +| Settled maximum | actors | `max` of the settled cut's merged `latency`. | +| Success rate | actors | `successes` over `requests` in the window's cuts. | +| Run validity | actors + gateway | `opened` summed equals A × C and `newConns` does not move over `t2..t3` (a first request that failed counts its endpoint as opened and dials its tunnel one period later, so the two need not be equal); `max(longestGap)` stays under the stream idle timeout; `dns` and `tls` are absent from the settled window's cut; every actor answered the round; `cluster.mitm_internal.upstream_cx_active` equals A × C and `upstream_cx_total` is flat over `t2..t3`; `http.egress_connect.downstream_rq_idle_timeout`, `downstream_cx_idle_timeout` and `downstream_cx_protocol_error` do not move. | + +Two limits of this design are accepted and stated here rather than worked +around. The window edges are exact cuts and `max` is exact, so neither a read +lag nor a lost maximum is among them. + +- **A collect round is A round trips.** A round is A requests through the + ingress router, each inside the router's client timeout. A slow fleet makes + a round take seconds, and an actor that timed out is retried by id; the + retry is the same window. +- **A restart inside a window invalidates the cell.** A `requests` that went + down between two reads, or a 404, means the actor restarted. Its windows + since the last cut are gone, and the driver marks the cell invalid rather + than reasoning about the gap. + +## 7. Decisions and alternatives + +| Decision | Chosen | Instead of | Because | +|---|---|---|---| +| Programs | One binary, `benchmark-egress`, with one subcommand per program. | One image per program. | One image to build, ship and pin. | +| Contract | Plain JSON over HTTP; the Go types live in the actor package. | Protobuf, or the OpenTelemetry data types. | A driver decodes a report with the standard library alone. | +| `/stats` payload | Cumulative counters only, about 250 bytes at C = 1024. | Counters plus the four latency distributions. | Cheap enough to poll from a thousand actors; the distributions are not in the payload, the package names them and holds their bounds. | +| Contract version | `Benchmark-Egress-Protocol: 2` on every answer. | A version in the path or the body. | A driver refuses an actor that speaks another contract instead of misreading its report; the OTLP images sent 1. | +| Target certificate | A throwaway CA name-constrained to `egress-tests-targets.svc.cluster.local`. | A cluster CA, or a self-signed leaf. | A certificate the driver hands out cannot vouch for anything but the test's own Services. | +| Loop start | On `POST /start`. | At boot. | Every actor restores from a golden snapshot taken before any egress policy exists. | +| Phase offset | On the actor, `offsetMs`. | Staggered `/start` posts from the driver. | The router's fan-out sets each actor's real start, so the actor is the only place the phase can be set. | +| Pacing | One ticker per endpoint, never reset. | A shared ticker, or a sleep after each request. | A slow request skips its slot instead of shifting the phase; the actor's rate stays one request per interval. | +| Measurement path | Window cuts served by the actor over `/collect?window=N`, merged in the driver. | Cumulative OTLP push to a collector read from Prometheus, and Prometheus scraping the actors. | Exact window edges and an exact maximum; no measurement traffic through the system under test; nothing to deploy or prove on a cluster beyond the actor image and the policy; a lost answer is a retry by id. The push was built and measured on GKE: its cost on the gateway was below run-to-run noise and PromQL agreed with the actors' own counters within one percent, but the maximum was lost and the pipeline added five moving parts. The router reaches an actor only by header, so no scraper can target one. | +| Window state | The last cut, cached, keyed by its id. | None on the actor, or every cut kept. | A lost answer is asked for again by id and costs nothing; a skipped or older id is 409, so a second reader is seen. | +| Temporality | Delta, per cut. | Cumulative. | A cut is the window: nothing to subtract, and the edges are exact because the cut happens under the lock every record takes. | +| Reading results | Cuts at the instants the driver chose. | Instant queries against a store, read after a lag. | The actors' counts and the gateway's CPU are read at the same instants, and no lag correction is needed. | +| Merging across actors | `Merge` in the driver. | In PromQL. | Every actor shares one bound set and the bounds travel in the payload, so the merge is exact and a differing bound set is refused rather than mixed. | +| Per-endpoint rows | Behind `?endpoints=1` only. | In every cut, or exported with an `endpoint` attribute. | A million rows a round at C = 1024 across a thousand actors. | diff --git a/cmd/benchmarking/egress/internal/cmd/actor/actor.go b/cmd/benchmarking/egress/internal/cmd/actor/actor.go new file mode 100644 index 0000000000..e7ff76f3bc --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/actor.go @@ -0,0 +1,623 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actor + +import ( + "cmp" + "context" + "crypto/tls" + "crypto/x509" + "encoding/json" + "errors" + "fmt" + "io" + "log/slog" + "maps" + "net" + "net/http" + "net/http/httptrace" + "slices" + "strconv" + "strings" + "sync" + "time" + + "github.com/spf13/cobra" +) + +// Command is the workload of the egress scale test. Once told to start, +// it opens every endpoint, one per interval, then on one goroutine per +// endpoint sends it an HTTP GET once per period of C intervals, and counts +// what happened under one lock. +// +// The loop starts on POST /start rather than at boot: every actor restores +// from a golden snapshot taken after the template's warmup, so a loop started +// at boot would run in the golden actor, which has no egress policy, and be +// frozen into every actor's snapshot, failed requests included. +func Command() *cobra.Command { + var listen string + cmd := &cobra.Command{ + Use: "actor", + Short: "Hold one tunnel per endpoint open through the gateway and count what happened", + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, _ []string) error { + return runActor(cmd.Context(), listen) + }, + } + cmd.Flags().StringVar(&listen, "listen", ":80", "Address to serve the readyz, start, stats, collect and stop routes on.") + return cmd +} + +// runActor serves the actor's routes on listen until ctx ends. +func runActor(ctx context.Context, listen string) error { + srv := &http.Server{ + Addr: listen, + Handler: newServer().handler(), + ReadHeaderTimeout: 10 * time.Second, + } + go func() { + <-ctx.Done() + shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + _ = srv.Shutdown(shutdownCtx) + }() + + slog.Info("benchmark-egress actor listening", "addr", listen) + if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { + return fmt.Errorf("serving: %w", err) + } + return nil +} + +// maxBodyBytes bounds what the actor reads, from a /start body or from an +// endpoint's response. +const maxBodyBytes = 1 << 20 + +// server serves the actor routes. At most one loop runs at a time. +type server struct { + // endpointURL maps an endpoint number and scheme to the URL the loop + // requests. Tests point it at local servers. + endpointURL func(i int, scheme string) string + // rootCAs verifies HTTPS endpoints; nil uses the system roots, which + // SSL_CERT_FILE extends with the gateway's MITM CA. Tests set their own. + rootCAs *x509.CertPool + // dial opens the loop's connections; nil uses a plain net.Dialer. Tests + // send the in-cluster endpoint names to local listeners through it. + dial dialFunc + + mu sync.Mutex + run *loopRun // nil when no loop is running + // final is the last stopped loop's Stats, endpoints included, kept until + // the next /start so a repeated /stop answers it again. + final *Stats +} + +func newServer() *server { + return &server{endpointURL: EndpointURL} +} + +func (s *server) handler() http.Handler { + mux := http.NewServeMux() + mux.HandleFunc("GET "+ReadyzRoute, func(w http.ResponseWriter, _ *http.Request) { + w.Write([]byte("ok\n")) + }) + mux.HandleFunc("POST "+StartRoute, s.handleStart) + mux.HandleFunc("GET "+StatsRoute, s.handleStats) + mux.HandleFunc("POST "+CollectRoute, s.handleCollect) + mux.HandleFunc("POST "+StopRoute, s.handleStop) + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set(ProtocolHeader, ProtocolVersion) + mux.ServeHTTP(w, r) + }) +} + +func (s *server) handleStart(w http.ResponseWriter, r *http.Request) { + var req StartRequest + if err := json.NewDecoder(io.LimitReader(r.Body, maxBodyBytes)).Decode(&req); err != nil { + http.Error(w, "decoding start request: "+err.Error(), http.StatusBadRequest) + return + } + if err := req.Validate(); err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + + s.mu.Lock() + defer s.mu.Unlock() + if s.run != nil { + http.Error(w, "a loop is already running; POST "+StopRoute+" first", http.StatusConflict) + return + } + urls := make([]string, req.Endpoints) + for i := range urls { + urls[i] = s.endpointURL(i, req.URLScheme()) + } + run, err := startLoop(req, urls, s.rootCAs, s.dial) + if err != nil { + http.Error(w, "starting loop: "+err.Error(), http.StatusInternalServerError) + return + } + s.run, s.final = run, nil + w.WriteHeader(http.StatusNoContent) +} + +// withEndpoints reports whether the request asked for the per-endpoint +// counts. +func withEndpoints(r *http.Request) bool { + return r.URL.Query().Get("endpoints") == "1" +} + +// view is a kept Stats as a request asked for it: the per-endpoint slice +// stays only with ?endpoints=1. The kept value is not touched. +func view(s Stats, endpoints bool) Stats { + if !endpoints { + s.Endpoints = nil + } + return s +} + +// runOr404 returns the loop, or answers 404 and returns nil. +func (s *server) runOr404(w http.ResponseWriter) *loopRun { + s.mu.Lock() + run := s.run + s.mu.Unlock() + if run == nil { + http.Error(w, NoLoopMessage, http.StatusNotFound) + } + return run +} + +func (s *server) handleStats(w http.ResponseWriter, r *http.Request) { + run := s.runOr404(w) + if run == nil { + return + } + st, ok := run.snapshot(withEndpoints(r)) + if !ok { + http.Error(w, NoLoopMessage, http.StatusNotFound) + return + } + writeJSON(w, st) +} + +func (s *server) handleCollect(w http.ResponseWriter, r *http.Request) { + run := s.runOr404(w) + if run == nil { + return + } + raw := r.URL.Query().Get(WindowParam) + window, err := strconv.ParseInt(raw, 10, 64) + if err != nil || window < 1 { + http.Error(w, fmt.Sprintf("%s must be a positive integer, got %q; the next window is %d", WindowParam, raw, run.nextWindow()), http.StatusBadRequest) + return + } + st, err := run.collect(window, withEndpoints(r)) + var seq *sequenceError + switch { + case errors.Is(err, errNoLoop): + http.Error(w, NoLoopMessage, http.StatusNotFound) + case errors.As(err, &seq): + http.Error(w, err.Error(), http.StatusConflict) + case err != nil: + http.Error(w, "collecting: "+err.Error(), http.StatusInternalServerError) + default: + writeJSON(w, st) + } +} + +func (s *server) handleStop(w http.ResponseWriter, r *http.Request) { + // Held across the wait so a /start cannot slip in before the loop exits. + s.mu.Lock() + defer s.mu.Unlock() + if s.run == nil { + if s.final == nil { + http.Error(w, NoLoopMessage, http.StatusNotFound) + return + } + writeJSON(w, view(*s.final, withEndpoints(r))) + return + } + st, err := s.run.stop() + s.run = nil + if err != nil { + http.Error(w, "collecting: "+err.Error(), http.StatusInternalServerError) + return + } + s.final = &st + writeJSON(w, view(st, withEndpoints(r))) +} + +func writeJSON(w http.ResponseWriter, v any) { + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(v) +} + +// loopRun is one started loop and what it has measured: the counters under +// its lock, the distributions in its recorder. +type loopRun struct { + cancel context.CancelFunc + wg sync.WaitGroup // one goroutine per endpoint + started time.Time + transport *http.Transport + rec *recorder + + mu sync.Mutex + // stopped is set by stop before it releases the recorder, so a collect + // that raced the stop answers 404 instead of reading a shut-down reader. + stopped bool + // stats holds the counters; Elapsed and Rounds are derived at snapshot, + // Distributions at collect. + stats Stats + last []time.Time // start of each endpoint's most recent request + // rounds is Stats.Rounds kept current: the least-visited endpoint's + // request count. + rounds int64 + // window is the id of the last cut, 0 before the first, and cut its + // Stats with the endpoints, so the same id asked again is answered + // without a new cut. + window int64 + cut Stats +} + +// errNoLoop is a read of a loop that has been stopped. +var errNoLoop = errors.New(NoLoopMessage) + +// sequenceError is a window id that is neither the last cut nor the next. +type sequenceError struct{ got, last int64 } + +func (e *sequenceError) Error() string { + return fmt.Sprintf("window %d is out of sequence: the last cut is %d, the next window is %d", e.got, e.last, e.last+1) +} + +// newLoopRun is a loop over req's endpoints with a fresh recorder and no +// goroutines yet; startLoop runs it. +func newLoopRun(req StartRequest) (*loopRun, error) { + rec, err := newRecorder() + if err != nil { + return nil, err + } + return &loopRun{ + started: time.Now(), + rec: rec, + stats: Stats{Scheme: req.URLScheme(), Endpoints: make([]Endpoint, req.Endpoints)}, + last: make([]time.Time, req.Endpoints), + }, nil +} + +func startLoop(req StartRequest, urls []string, roots *x509.CertPool, dial dialFunc) (*loopRun, error) { + timeoutMs := req.RequestTimeoutMs + if timeoutMs == 0 { + timeoutMs = DefaultRequestTimeoutMs + } + r, err := newLoopRun(req) + if err != nil { + return nil, err + } + ctx, cancel := context.WithCancel(context.Background()) + r.cancel, r.transport = cancel, newTransport(roots, dial) + client := &http.Client{ + Transport: r.transport, + // Count a redirect as the endpoint's answer rather than following it + // somewhere the egress policy may not allow. + CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }, + } + timeout := time.Duration(timeoutMs) * time.Millisecond + interval := time.Duration(req.IntervalMs) * time.Millisecond + offset := time.Duration(req.OffsetMs) * time.Millisecond + period := interval * time.Duration(len(urls)) + for i, u := range urls { + r.wg.Go(func() { r.endpoint(ctx, client, i, u, timeout, offset+interval*time.Duration(i), period) }) + } + return r, nil +} + +// dialFunc opens one connection, as net.Dialer.DialContext does. +type dialFunc = func(ctx context.Context, network, addr string) (net.Conn, error) + +// newTransport keeps one connection alive per endpoint and verifies HTTPS +// endpoints against roots. It ignores proxy environment variables: the +// sandbox's egress capture is the only proxy under test. +func newTransport(roots *x509.CertPool, dial dialFunc) *http.Transport { + if dial == nil { + dial = (&net.Dialer{Timeout: 30 * time.Second, KeepAlive: 30 * time.Second}).DialContext + } + return &http.Transport{ + DialContext: dial, + TLSClientConfig: &tls.Config{RootCAs: roots}, + ForceAttemptHTTP2: false, // HTTP/1.1 over either scheme: one request per connection at a time + MaxIdleConns: 0, // no total cap, so C endpoints keep C connections + MaxIdleConnsPerHost: 1, // one request in flight per endpoint, so one per endpoint is enough + // No IdleConnTimeout: a kept-alive connection the client closed for + // idling would redial like one the gateway reset, and the gateway's + // stream idle timeout is what the run measures. + IdleConnTimeout: 0, + // No TLSHandshakeTimeout: the request's own timeout bounds the + // handshake, so a stalled one is counted as a timeout like any other. + } +} + +// endpoint waits offset, opens endpoint i, then requests it once per period +// until ctx ends. The opens are one interval apart across endpoints, so C +// handshakes do not reach the gateway at once. At most one request is in +// flight per endpoint, and the period is counted from the start of the open: +// a request that outlasts the period delays only this endpoint, whose next +// request starts as soon as the slow one returns. +func (r *loopRun) endpoint(ctx context.Context, client *http.Client, i int, u string, timeout, offset, period time.Duration) { + if !pause(ctx, offset) { + return + } + var tick <-chan time.Time + if period > 0 { + ticker := time.NewTicker(period) + defer ticker.Stop() + tick = ticker.C + } + r.request(ctx, client, i, u, timeout) + if ctx.Err() != nil { + return // stopped mid-open: not an open + } + r.mu.Lock() + r.stats.Opened++ + if r.stats.Opened == int64(len(r.stats.Endpoints)) { + r.stats.OpeningDone = time.Since(r.started) + r.stats.RequestsAtOpeningDone = r.stats.Requests + } + r.mu.Unlock() + for ctx.Err() == nil { + if tick != nil { + select { + case <-ctx.Done(): + return + case <-tick: + } + } + r.request(ctx, client, i, u, timeout) + } +} + +// pause waits d, or until ctx ends, and reports whether to go on. +func pause(ctx context.Context, d time.Duration) bool { + if d <= 0 { + return ctx.Err() == nil + } + select { + case <-ctx.Done(): + return false + case <-time.After(d): + return true + } +} + +// requestTrace collects what httptrace reports about one request, as the +// times its steps happened: a zero time is a step that did not. The dial, +// and with it the DNS and TLS callbacks, can still be running on another +// goroutine after a timed-out request returns, hence the lock. +type requestTrace struct { + mu sync.Mutex + // newConn is set when the request went out on a connection opened for + // it rather than a kept-alive one. + newConn bool + dnsStart time.Time + dnsDone time.Time + tlsStart time.Time + tlsDone time.Time // set only when the handshake completed +} + +func (t *requestTrace) clientTrace() *httptrace.ClientTrace { + return &httptrace.ClientTrace{ + GotConn: func(info httptrace.GotConnInfo) { + t.mu.Lock() + defer t.mu.Unlock() + t.newConn = !info.Reused + }, + DNSStart: func(httptrace.DNSStartInfo) { + t.mu.Lock() + defer t.mu.Unlock() + t.dnsStart = time.Now() + }, + DNSDone: func(httptrace.DNSDoneInfo) { + t.mu.Lock() + defer t.mu.Unlock() + t.dnsDone = time.Now() + }, + TLSHandshakeStart: func() { + t.mu.Lock() + defer t.mu.Unlock() + t.tlsStart = time.Now() + }, + TLSHandshakeDone: func(_ tls.ConnectionState, err error) { + t.mu.Lock() + defer t.mu.Unlock() + if err == nil { + t.tlsDone = time.Now() + } + }, + } +} + +func (r *loopRun) request(ctx context.Context, client *http.Client, i int, u string, timeout time.Duration) { + reqCtx, cancel := context.WithTimeout(ctx, timeout) + defer cancel() + trace := &requestTrace{} + req, err := http.NewRequestWithContext(httptrace.WithClientTrace(reqCtx, trace.clientTrace()), http.MethodGet, u, nil) + if err != nil { + r.record(i, 0, 0, err, trace) + return + } + + start := time.Now() + r.mu.Lock() + if last := r.last[i]; !last.IsZero() { + gap := start.Sub(last) + r.stats.LongestGap = max(r.stats.LongestGap, gap) + r.stats.Endpoints[i].LongestGap = max(r.stats.Endpoints[i].LongestGap, gap) + } + r.last[i] = start + r.mu.Unlock() + status := 0 + resp, err := client.Do(req) + if err == nil { + status = resp.StatusCode + _, err = io.Copy(io.Discard, io.LimitReader(resp.Body, maxBodyBytes)) + resp.Body.Close() + } + latency := time.Since(start) + if ctx.Err() != nil { + // Stopped mid-request: the failure is the stop, not the egress path. + return + } + r.record(i, latency, status, err, trace) +} + +func (r *loopRun) record(i int, latency time.Duration, status int, err error, trace *requestTrace) { + trace.mu.Lock() + newConn := trace.newConn + didDNS, dns := !trace.dnsDone.IsZero(), trace.dnsDone.Sub(trace.dnsStart) + tlsStarted, didTLS, tlsTime := !trace.tlsStart.IsZero(), !trace.tlsDone.IsZero(), trace.tlsDone.Sub(trace.tlsStart) + trace.mu.Unlock() + + r.mu.Lock() + defer r.mu.Unlock() + s, ctx := &r.stats, context.Background() + first := s.Endpoints[i].Requests == 0 + s.Requests++ + s.Endpoints[i].Requests++ + if s.Endpoints[i].Requests-1 == r.rounds { + // Only the least-visited endpoint can raise the minimum. + r.rounds = slices.MinFunc(s.Endpoints, func(a, b Endpoint) int { return cmp.Compare(a.Requests, b.Requests) }).Requests + } + if newConn { + s.NewConns++ + } + if didDNS { + r.rec.record(ctx, DNS, dns.Seconds()) + } + if didTLS { + r.rec.record(ctx, TLS, tlsTime.Seconds()) + } + class := classify(err, status) + if class != "" && tlsStarted && !didTLS && !strings.HasPrefix(class, "tls: ") { + // A failure inside the handshake, such as the gateway closing a + // connection whose SNI no rule allows, is told apart from the same + // error in an HTTP exchange. + class = "tls handshake: " + class + } + if class == "" { + s.Successes++ + if first { + r.rec.record(ctx, Opening, latency.Seconds()) + } else { + r.rec.record(ctx, Latency, latency.Seconds()) + } + return + } + if s.Errors == nil { + s.Errors = make(map[string]int64) + } + s.Errors[class]++ + s.Endpoints[i].Errors++ +} + +// snapshotLocked copies the counters so far, with Elapsed derived. The +// caller holds r.mu. +func (r *loopRun) snapshotLocked(endpoints bool) Stats { + s := r.stats + s.Elapsed, s.Rounds = time.Since(r.started), r.rounds + s.Errors = maps.Clone(s.Errors) + if endpoints { + s.Endpoints = slices.Clone(s.Endpoints) + } else { + s.Endpoints = nil + } + return s +} + +// snapshot answers GET /stats: the counters alone. ok is false once the +// loop is stopped. +func (r *loopRun) snapshot(endpoints bool) (Stats, bool) { + r.mu.Lock() + defer r.mu.Unlock() + if r.stopped { + return Stats{}, false + } + return r.snapshotLocked(endpoints), true +} + +// nextWindow is the id the next cut must carry. +func (r *loopRun) nextWindow() int64 { + r.mu.Lock() + defer r.mu.Unlock() + return r.window + 1 +} + +// collect answers POST /collect for window id w, at least 1. The id after the +// last cut's cuts a new window: the counters and the distributions since the +// previous cut, under the lock every record takes, so no sample lands between +// the two. The last cut's id answers that cut again, so a lost answer costs +// the driver a retry rather than a window. Any other id is a sequenceError, +// and a stopped loop errNoLoop. +func (r *loopRun) collect(w int64, endpoints bool) (Stats, error) { + r.mu.Lock() + defer r.mu.Unlock() + switch { + case r.stopped: + return Stats{}, errNoLoop + case w == r.window: + return view(r.cut, endpoints), nil + case w != r.window+1: + return Stats{}, &sequenceError{got: w, last: r.window} + } + s, err := r.collectLocked() + if err != nil { + return Stats{}, err + } + r.window, r.cut = w, s + return view(s, endpoints), nil +} + +// collectLocked cuts a window with r.mu held, endpoints included. The cut uses +// its own context: a client that went away must not cost the driver a window. +func (r *loopRun) collectLocked() (Stats, error) { + s := r.snapshotLocked(true) + d, err := r.rec.collect(context.Background()) + if err != nil { + return Stats{}, err + } + s.Distributions = d + return s, nil +} + +// stop cancels the loop, waits for every endpoint's goroutine to exit, and +// returns the final Stats: the counters, endpoints included, and the tail +// window. It then releases the recorder; a collect that raced the stop +// answers 404. +func (r *loopRun) stop() (Stats, error) { + if r.cancel != nil { + r.cancel() + } + r.wg.Wait() + if r.transport != nil { + r.transport.CloseIdleConnections() + } + r.mu.Lock() + defer r.mu.Unlock() + r.stopped = true + s, err := r.collectLocked() + // Nothing is attached to the provider, so there is nothing to flush and + // a failed shutdown loses nothing. Its own context, as for the collect: a + // client that hung up must not leave the provider running. + _ = r.rec.shutdown(context.Background()) + return s, err +} diff --git a/cmd/benchmarking/egress/internal/cmd/actor/actor_test.go b/cmd/benchmarking/egress/internal/cmd/actor/actor_test.go new file mode 100644 index 0000000000..a73f5229d7 --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/actor_test.go @@ -0,0 +1,1437 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actor + +import ( + "bytes" + "context" + "crypto/tls" + "crypto/x509" + "encoding/json" + "fmt" + "io" + "log" + "net" + "net/http" + "net/http/httptest" + "net/url" + "slices" + "strings" + "sync" + "sync/atomic" + "testing" + "testing/synctest" + "time" + + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/targetcert" +) + +// target is an endpoint that counts the TCP connections opened to it. +type target struct { + *httptest.Server + conns atomic.Int64 + requests atomic.Int64 +} + +func newTarget(t *testing.T, h http.HandlerFunc) *target { + t.Helper() + return newTargetConnState(t, h, nil) +} + +// newTargetConnState is newTarget with onState called on every connection +// state change, after the connection count. +func newTargetConnState(t *testing.T, h http.HandlerFunc, onState func(net.Conn, http.ConnState)) *target { + t.Helper() + tg := &target{} + tg.Server = httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + tg.requests.Add(1) + h(w, r) + })) + tg.Config.ConnState = func(c net.Conn, s http.ConnState) { + if s == http.StateNew { + tg.conns.Add(1) + } + if onState != nil { + onState(c, s) + } + } + tg.Start() + t.Cleanup(tg.Close) + return tg +} + +func okHandler(w http.ResponseWriter, _ *http.Request) { w.Write([]byte("ok")) } + +// quietLog discards a test server's own error log. +var quietLog = log.New(io.Discard, "", 0) + +// host is the target's host:port. +func (tg *target) host() string { return tg.Listener.Addr().String() } + +// serve returns the actor's handler with endpoint i mapped to hosts[i], a +// host:port, in place of the in-cluster Service names. The URL keeps the +// scheme the loop asks for. +func serve(hosts ...string) http.Handler { + return serveTLS(nil, hosts...) +} + +// serveTLS is serve, verifying HTTPS endpoints against roots. +func serveTLS(roots *x509.CertPool, hosts ...string) http.Handler { + s := newServer() + s.rootCAs = roots + s.endpointURL = func(i int, scheme string) string { return scheme + "://" + hosts[i] + "/" } + return s.handler() +} + +// serveTargets is serve over n fresh targets, one per endpoint. Endpoints +// that share one host would share the transport's single idle connection per +// host and redial on every request. +func serveTargets(t *testing.T, n int) http.Handler { + t.Helper() + hosts := make([]string, n) + for i := range hosts { + hosts[i] = newTarget(t, okHandler).host() + } + return serve(hosts...) +} + +// newTLSTarget is newTarget serving HTTPS with httptest's certificate, which +// the returned pool trusts. +func newTLSTarget(t *testing.T, h http.HandlerFunc) (*target, *x509.CertPool) { + t.Helper() + tg := &target{} + tg.Server = httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + tg.requests.Add(1) + h(w, r) + })) + tg.Config.ConnState = func(_ net.Conn, s http.ConnState) { + if s == http.StateNew { + tg.conns.Add(1) + } + } + tg.Config.ErrorLog = quietLog // failed handshakes are what some tests expect + tg.StartTLS() + t.Cleanup(tg.Close) + roots := x509.NewCertPool() + roots.AddCert(tg.Certificate()) + return tg, roots +} + +func post(t *testing.T, h http.Handler, path string, body any) *httptest.ResponseRecorder { + t.Helper() + var buf bytes.Buffer + if body != nil { + if err := json.NewEncoder(&buf).Encode(body); err != nil { + t.Fatal(err) + } + } + rec := httptest.NewRecorder() + h.ServeHTTP(rec, httptest.NewRequest(http.MethodPost, path, &buf)) + return rec +} + +func get(h http.Handler, path string) *httptest.ResponseRecorder { + rec := httptest.NewRecorder() + h.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, path, nil)) + return rec +} + +// decodeStats reads a 200 answer's Stats. +func decodeStats(t *testing.T, what string, rec *httptest.ResponseRecorder) *Stats { + t.Helper() + if rec.Code != http.StatusOK { + t.Fatalf("%s = %d %s", what, rec.Code, rec.Body) + } + var s Stats + if err := json.Unmarshal(rec.Body.Bytes(), &s); err != nil { + t.Fatalf("%s: decoding %s: %v", what, rec.Body, err) + } + return &s +} + +// The poll and the stop of a running loop, each with the per-endpoint counts. +const ( + statsPath = StatsRoute + "?endpoints=1" + stopPath = StopRoute + "?endpoints=1" +) + +// collectPath is POST /collect's path for window id w, with the per-endpoint +// counts when asked. +func collectPath(w int64, endpoints bool) string { + path := fmt.Sprintf("%s?%s=%d", CollectRoute, WindowParam, w) + if endpoints { + path += "&endpoints=1" + } + return path +} + +// start posts /start with req. +func start(t *testing.T, h http.Handler, req StartRequest) *httptest.ResponseRecorder { + t.Helper() + return post(t, h, StartRoute, req) +} + +// readStats polls the counters. +func readStats(t *testing.T, h http.Handler) *Stats { + t.Helper() + return decodeStats(t, "GET /stats", get(h, statsPath)) +} + +// collectStats cuts window w, as the driver numbers them from 1: the counters +// and the distributions since the previous cut. +func collectStats(t *testing.T, h http.Handler, w int64) *Stats { + t.Helper() + return decodeStats(t, "POST /collect", post(t, h, collectPath(w, true), nil)) +} + +// stopLoop stops the loop and returns its final Stats, whose distributions +// cover everything since the previous cut. +func stopLoop(t *testing.T, h http.Handler) *Stats { + t.Helper() + return decodeStats(t, "POST /stop", post(t, h, stopPath, nil)) +} + +// runLoop starts a loop with req, lets it run until cond holds on the live +// counters (or fails after 10s), then stops it and returns the final Stats. +func runLoop(t *testing.T, h http.Handler, req StartRequest, cond func(*Stats) bool) *Stats { + t.Helper() + if rec := start(t, h, req); rec.Code != http.StatusNoContent { + t.Fatalf("POST /start = %d %s", rec.Code, rec.Body) + } + waitFor(t, h, cond) + return stopLoop(t, h) +} + +// waitFor polls the counters until cond holds, or fails after 10s. +func waitFor(t *testing.T, h http.Handler, cond func(*Stats) bool) { + t.Helper() + deadline := time.Now().Add(10 * time.Second) + for { + s := readStats(t, h) + if cond(s) { + return + } + if time.Now().After(deadline) { + t.Fatalf("condition not reached; last stats %+v", s) + } + time.Sleep(5 * time.Millisecond) + } +} + +func atLeast(n int64) func(*Stats) bool { + return func(s *Stats) bool { return s.Requests >= n } +} + +// samples is the count of a distribution in s, 0 when absent. +func samples(s *Stats, name Distribution) uint64 { + return s.Distributions[name].Count +} + +func TestKeepAliveReusesOneConnectionPerEndpoint(t *testing.T) { + t.Parallel() + targets := []*target{newTarget(t, okHandler), newTarget(t, okHandler), newTarget(t, okHandler)} + var hosts []string + for _, tg := range targets { + hosts = append(hosts, tg.host()) + } + h := serve(hosts...) + + s := runLoop(t, h, StartRequest{Endpoints: len(hosts)}, func(s *Stats) bool { return s.Rounds >= 10 }) + if s.Scheme != SchemeHTTP || samples(s, TLS) != 0 || samples(s, DNS) != 0 { + t.Errorf("a start with no scheme against IP hosts ran over %q with %d TLS handshakes and %d DNS lookups, want http with neither", s.Scheme, samples(s, TLS), samples(s, DNS)) + } + + if s.Successes != s.Requests || len(s.Errors) != 0 { + t.Errorf("successes %d of %d requests, errors %v", s.Successes, s.Requests, s.Errors) + } + for i, tg := range targets { + if got := tg.conns.Load(); got != 1 { + t.Errorf("target %d saw %d connections, want 1 kept alive", i, got) + } + } + if s.NewConns != int64(len(targets)) { + t.Errorf("NewConns = %d, want %d (one per endpoint)", s.NewConns, len(targets)) + } + if opening, latency := samples(s, Opening), samples(s, Latency); opening != uint64(len(targets)) || int64(opening+latency) != s.Successes { + t.Errorf("opening %d and latency %d samples for %d successes; want one opening per endpoint and the rest latency", opening, latency, s.Successes) + } + for i, e := range s.Endpoints { + if e.Requests < 10 { + t.Errorf("endpoint %d got %d requests, want at least the %d rounds the loop was stopped at", i, e.Requests, 10) + } + } +} + +func TestStopDuringOpeningDoesNotCountTheCancelledOpen(t *testing.T) { + t.Parallel() + first := newTarget(t, okHandler) + blocked := newTarget(t, func(w http.ResponseWriter, r *http.Request) { + <-r.Context().Done() // holds the second open until the client gives up + }) + h := serve(first.host(), blocked.host()) + + s := runLoop(t, h, StartRequest{Endpoints: 2}, func(s *Stats) bool { return s.Opened >= 1 }) + if s.Opened != 1 { + t.Errorf("Opened = %d after stopping during the second open, want 1", s.Opened) + } + if s.Rounds != 0 || s.OpeningDone != 0 || s.Endpoints[1].Requests != 0 { + t.Errorf("Rounds = %d, OpeningDone = %v, endpoint 1 requests = %d during the opening, want 0, 0 and 0", s.Rounds, s.OpeningDone, s.Endpoints[1].Requests) + } +} + +func TestRequestTimeoutIsClassified(t *testing.T) { + t.Parallel() + release := make(chan struct{}) + slow := newTarget(t, func(w http.ResponseWriter, r *http.Request) { + select { + case <-release: + case <-r.Context().Done(): + } + }) + t.Cleanup(func() { close(release) }) + h := serve(slow.host()) + + s := runLoop(t, h, StartRequest{Endpoints: 1, RequestTimeoutMs: 20}, atLeast(2)) + + if s.Errors["timeout"] != s.Requests || s.Successes != 0 { + t.Errorf("errors %v, successes %d for %d requests; want every request a timeout", s.Errors, s.Successes, s.Requests) + } + if n := samples(s, Latency); n != 0 { + t.Errorf("failed requests recorded %d latency samples, want 0", n) + } + if s.Endpoints[0].Errors != s.Requests { + t.Errorf("endpoint errors = %d, want %d", s.Endpoints[0].Errors, s.Requests) + } +} + +func TestNon200IsAnError(t *testing.T) { + t.Parallel() + ok := newTarget(t, okHandler) + failing := newTarget(t, func(w http.ResponseWriter, _ *http.Request) { + http.Error(w, "egress denied", http.StatusServiceUnavailable) + }) + h := serve(ok.host(), failing.host()) + + s := runLoop(t, h, StartRequest{Endpoints: 2}, atLeast(10)) + + if s.Errors["HTTP 503"] == 0 { + t.Errorf("errors = %v, want HTTP 503 counted", s.Errors) + } + if s.Endpoints[0].Errors != 0 || s.Endpoints[1].Errors != s.Endpoints[1].Requests { + t.Errorf("endpoints = %+v, want only the second one failing", s.Endpoints) + } +} + +func TestStartStopLifecycle(t *testing.T) { + t.Parallel() + tg := newTarget(t, okHandler) + h := serve(tg.host()) + req := StartRequest{Endpoints: 1} + + if rec := get(h, StatsRoute); rec.Code != http.StatusNotFound { + t.Errorf("GET /stats before start = %d, want 404", rec.Code) + } + if rec := post(t, h, StopRoute, nil); rec.Code != http.StatusNotFound { + t.Errorf("POST /stop before start = %d, want 404", rec.Code) + } + + first := runLoop(t, h, req, atLeast(5)) + time.Sleep(20 * time.Millisecond) + // A request the stop cut short can still reach the target after /stop + // returns; nothing else may. + if extra := tg.requests.Load() - first.Requests; extra > 1 { + t.Errorf("target got %d requests the stopped loop did not count, want at most the one cut short by the stop", extra) + } + + if rec := start(t, h, req); rec.Code != http.StatusNoContent { + t.Fatalf("restart = %d %s", rec.Code, rec.Body) + } + if rec := start(t, h, req); rec.Code != http.StatusConflict { + t.Errorf("second /start while running = %d, want 409", rec.Code) + } + second := stopLoop(t, h) + // The target saw every request of both runs (plus any cut short by a + // stop), so a second run that started from zero counts no more than the + // target saw after the first run. + if sinceRestart := tg.requests.Load() - first.Requests; second.Requests > sinceRestart { + t.Errorf("restarted loop counted %d requests, more than the %d sent since the restart: the old counts were kept", second.Requests, sinceRestart) + } +} + +func TestStartRejectsBadRequests(t *testing.T) { + t.Parallel() + h := newServer().handler() + for _, tc := range []struct{ name, body string }{ + {"not json", `not json`}, + {"empty object", `{}`}, + {"invalid", `{"endpoints":1,"scheme":"ftp"}`}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + rec := httptest.NewRecorder() + h.ServeHTTP(rec, httptest.NewRequest(http.MethodPost, StartRoute, strings.NewReader(tc.body))) + if rec.Code != http.StatusBadRequest { + t.Errorf("POST /start %s = %d, want 400", tc.body, rec.Code) + } + }) + } + if rec := get(h, ReadyzRoute); rec.Code != http.StatusOK { + t.Errorf("GET /readyz = %d, want 200", rec.Code) + } +} + +func TestDefaultURLsAreTheEndpointServices(t *testing.T) { + t.Parallel() + s := newServer() + for _, i := range []int{0, 9, 255} { + for _, scheme := range []string{SchemeHTTP, SchemeHTTPS} { + if got, want := s.endpointURL(i, scheme), EndpointURL(i, scheme); got != want { + t.Errorf("endpoint %d over %s = %q, want %q", i, scheme, got, want) + } + } + } + if s.rootCAs != nil { + t.Error("the actor's roots are set by default; they must stay nil so SSL_CERT_FILE applies") + } + if s.dial != nil { + t.Error("the actor dials through a hook by default; it must stay nil so a plain dialer is used") + } +} + +func TestHTTPSFailuresAreClassified(t *testing.T) { + t.Parallel() + bundle, err := targetcert.Generate(time.Now(), EndpointDomain) + if err != nil { + t.Fatal(err) + } + endpointCert, err := tls.X509KeyPair(bundle.Cert, bundle.Key) + if err != nil { + t.Fatal(err) + } + endpointRoots := x509.NewCertPool() + endpointRoots.AppendCertsFromPEM(bundle.CA) + // Serves the endpoints' certificate, which names no IP address. + wrongName := &target{Server: httptest.NewUnstartedServer(http.HandlerFunc(okHandler))} + wrongName.TLS = &tls.Config{Certificates: []tls.Certificate{endpointCert}} + wrongName.Config.ErrorLog = quietLog + wrongName.StartTLS() + t.Cleanup(wrongName.Close) + + untrusted, _ := newTLSTarget(t, okHandler) + plain := newTarget(t, okHandler) + + // Accepts, reads the ClientHello so closing sends a FIN rather than a + // reset, and closes: a peer that refuses the handshake, as the gateway + // does for an SNI no rule allows. + closer, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { closer.Close() }) + go func() { + for { + conn, err := closer.Accept() + if err != nil { + return + } + _, _ = conn.Read(make([]byte, 4096)) + conn.Close() + } + }() + + for _, tc := range []struct { + name string + roots *x509.CertPool + host string + want string + }{ + {"untrusted CA", x509.NewCertPool(), untrusted.host(), "tls: unknown authority"}, + {"hostname mismatch", endpointRoots, wrongName.host(), "tls: hostname mismatch"}, + {"plain HTTP server", x509.NewCertPool(), plain.host(), "tls: not a TLS server"}, + {"peer closes during handshake", x509.NewCertPool(), closer.Addr().String(), "tls handshake: "}, + } { + t.Run(tc.name, func(t *testing.T) { + h := serveTLS(tc.roots, tc.host) + s := runLoop(t, h, StartRequest{Endpoints: 1, Scheme: SchemeHTTPS}, atLeast(2)) + var matched int64 + for class, n := range s.Errors { + if strings.HasPrefix(class, tc.want) { + matched += n + } + } + if matched != s.Requests || s.Successes != 0 || samples(s, TLS) != 0 { + t.Errorf("errors %v, %d successes, %d handshakes for %d requests; want every request classed %q", s.Errors, s.Successes, samples(s, TLS), s.Requests, tc.want) + } + }) + } +} + +// dialTo returns a dial function that connects host:port to addrs[host] +// instead, so a loop can request the in-cluster endpoint names against local +// listeners. TLS still verifies the name the loop asked for. +func dialTo(addrs map[string]string) dialFunc { + return func(ctx context.Context, network, addr string) (net.Conn, error) { + host, _, err := net.SplitHostPort(addr) + if err != nil { + return nil, err + } + real, ok := addrs[host] + if !ok { + return nil, fmt.Errorf("no local listener for %s", addr) + } + return (&net.Dialer{}).DialContext(ctx, network, real) + } +} + +// targetCertServer is an HTTPS target serving the endpoints' certificate. +func targetCertServer(t *testing.T, bundle *targetcert.Bundle) *target { + t.Helper() + cert, err := tls.X509KeyPair(bundle.Cert, bundle.Key) + if err != nil { + t.Fatal(err) + } + tg := &target{Server: httptest.NewUnstartedServer(http.HandlerFunc(okHandler))} + tg.Config.ConnState = func(_ net.Conn, s http.ConnState) { + if s == http.StateNew { + tg.conns.Add(1) + } + } + tg.TLS = &tls.Config{Certificates: []tls.Certificate{cert}} + tg.StartTLS() + t.Cleanup(tg.Close) + return tg +} + +// TestEndpointsByNameOverBothSchemes runs the loop as the actor does in the +// cluster: the default endpoint URLs, eight endpoints, and over HTTPS the +// target's own certificate verified under each endpoint's name. +func TestEndpointsByNameOverBothSchemes(t *testing.T) { + t.Parallel() + const c = 8 + bundle, err := targetcert.Generate(time.Now(), EndpointDomain) + if err != nil { + t.Fatal(err) + } + roots := x509.NewCertPool() + if !roots.AppendCertsFromPEM(bundle.CA) { + t.Fatal("no CA in the bundle") + } + for _, scheme := range []string{SchemeHTTP, SchemeHTTPS} { + t.Run(scheme, func(t *testing.T) { + t.Parallel() + addrs := make(map[string]string, c) + targets := make([]*target, c) + for i := range c { + if scheme == SchemeHTTPS { + targets[i] = targetCertServer(t, bundle) + } else { + targets[i] = newTarget(t, okHandler) + } + addrs[EndpointHost(i)] = targets[i].host() + } + s := newServer() + s.rootCAs = roots + s.dial = dialTo(addrs) + h := s.handler() + + const interval = 2 * time.Millisecond + st := runLoop(t, h, StartRequest{Endpoints: c, Scheme: scheme, IntervalMs: int64(interval / time.Millisecond)}, + func(s *Stats) bool { return s.Rounds >= 3 }) + + if st.Scheme != scheme || st.Opened != c || st.Rounds < 3 { + t.Errorf("Scheme = %q, Opened = %d, Rounds = %d, want %s, %d and at least 3", st.Scheme, st.Opened, st.Rounds, scheme, c) + } + if st.Successes != st.Requests || len(st.Errors) != 0 { + t.Errorf("successes %d of %d requests, errors %v", st.Successes, st.Requests, st.Errors) + } + if st.NewConns != c { + t.Errorf("NewConns = %d, want one per endpoint (%d)", st.NewConns, c) + } + wantTLS := uint64(0) + if scheme == SchemeHTTPS { + wantTLS = c + } + if n := samples(st, TLS); n != wantTLS { + t.Errorf("TLS handshakes = %d, want %d", n, wantTLS) + } + for i, tg := range targets { + if got := tg.conns.Load(); got != 1 { + t.Errorf("endpoint %d saw %d connections, want 1 kept alive", i, got) + } + } + // Every endpoint is requested once per period of C intervals; a gap + // is about one period, never anywhere near Elapsed. + if period := c * interval; st.LongestGap < period/2 || st.LongestGap > 5*time.Second { + t.Errorf("LongestGap = %v, want about one period (%v) and well under Elapsed", st.LongestGap, period) + } + }) + } +} + +// TestResetConnectionIsRedialedWithoutAnError pins how a tunnel the gateway +// reset shows up: the loop redials, net/http retries the GET, and NewConns +// grows by one while Errors stays empty. +func TestResetConnectionIsRedialedWithoutAnError(t *testing.T) { + t.Parallel() + var cutOnce atomic.Bool + cut := newTargetConnState(t, okHandler, func(c net.Conn, s http.ConnState) { + // Closes the server side once the first response is out, as the + // gateway does to an idle tunnel. + if s == http.StateIdle && cutOnce.CompareAndSwap(false, true) { + c.Close() + } + }) + steady := []*target{newTarget(t, okHandler), newTarget(t, okHandler)} + h := serve(steady[0].host(), cut.host(), steady[1].host()) + + s := runLoop(t, h, StartRequest{Endpoints: 3, IntervalMs: 2}, func(s *Stats) bool { return s.Rounds >= 2 }) + + if s.Successes != s.Requests || len(s.Errors) != 0 { + t.Errorf("successes %d of %d requests, errors %v; want the reset absorbed", s.Successes, s.Requests, s.Errors) + } + if s.NewConns != 4 { + t.Errorf("NewConns = %d, want 4: one per endpoint plus the redial", s.NewConns) + } + if got := cut.conns.Load(); got != 2 { + t.Errorf("the cut endpoint saw %d connections, want 2", got) + } + for i, tg := range steady { + if got := tg.conns.Load(); got != 1 { + t.Errorf("steady endpoint %d saw %d connections, want 1", i, got) + } + } +} + +// pipeNet serves a handler over in-memory connections, so a loop can run +// inside a synctest bubble, where a real socket would not count as blocked. +type pipeNet struct { + conns chan net.Conn + done chan struct{} + once sync.Once + srv *http.Server +} + +func newPipeNet(h http.Handler) *pipeNet { + p := &pipeNet{conns: make(chan net.Conn), done: make(chan struct{}), srv: &http.Server{Handler: h, ErrorLog: quietLog}} + go p.srv.Serve(p) + return p +} + +func (p *pipeNet) dial(ctx context.Context, _, _ string) (net.Conn, error) { + client, server := net.Pipe() + select { + case p.conns <- server: + return client, nil + case <-ctx.Done(): + return nil, ctx.Err() + } +} + +func (p *pipeNet) Accept() (net.Conn, error) { + select { + case c := <-p.conns: + return c, nil + case <-p.done: + return nil, net.ErrClosed + } +} + +func (p *pipeNet) Close() error { + p.once.Do(func() { close(p.done) }) + return nil +} + +func (p *pipeNet) Addr() net.Addr { return pipeAddr{} } + +type pipeAddr struct{} + +func (pipeAddr) Network() string { return "pipe" } +func (pipeAddr) String() string { return "pipe" } + +// startPipeLoop serves the actor over pn, with endpoint i named endpoint-i, +// and starts a loop with req. It returns the handler and a counters reader +// that waits for the bubble to settle first. +func startPipeLoop(t *testing.T, pn *pipeNet, req StartRequest) (http.Handler, func() *Stats) { + t.Helper() + s := newServer() + s.dial = pn.dial + s.endpointURL = func(i int, scheme string) string { return fmt.Sprintf("%s://endpoint-%d/", scheme, i) } + h := s.handler() + if rec := start(t, h, req); rec.Code != http.StatusNoContent { + t.Fatalf("POST /start = %d %s", rec.Code, rec.Body) + } + return h, func() *Stats { + synctest.Wait() + return readStats(t, h) + } +} + +// TestPacingAndGapsAreExact runs the loop on a fake clock with instant +// endpoints, so the schedule can be checked to the millisecond: endpoint i +// opens i intervals in, each endpoint is then requested once per period of C +// intervals, and so every gap is one period. Each step cuts a window, and the +// windows add up to one opening per opened endpoint and the rest latency. +func TestPacingAndGapsAreExact(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + pn := newPipeNet(http.HandlerFunc(okHandler)) + defer pn.srv.Close() + const c, interval = 4, 100 * time.Millisecond + const period = c * interval + start := time.Now() + h, _ := startPipeLoop(t, pn, StartRequest{Endpoints: c, IntervalMs: int64(interval / time.Millisecond)}) + var opening, latency uint64 + for step, tc := range []struct { + at time.Duration + requests, opened, rounds int64 + openingDone, longestGap time.Duration + why string + }{ + {interval / 2, 1, 1, 0, 0, 0, "endpoint 0 opens at once, the others wait their turn"}, + {period - interval/2, c, c, 1, (c - 1) * interval, 0, "one open per interval; the last completes at (C-1) intervals"}, + {2*period - interval/2, 2 * c, c, 2, (c - 1) * interval, period, "each endpoint's second request comes one period after its first"}, + {3*period - interval/2, 3 * c, c, 3, (c - 1) * interval, period, "steady state: every gap is one period"}, + } { + time.Sleep(tc.at - time.Since(start)) + synctest.Wait() + st := collectStats(t, h, int64(step+1)) + if st.Requests != tc.requests || st.Opened != tc.opened || st.Rounds != tc.rounds || st.OpeningDone != tc.openingDone || st.LongestGap != tc.longestGap { + t.Errorf("at %v: requests %d, opened %d, rounds %d, opening done %v, longest gap %v; want %d, %d, %d, %v, %v (%s)", + tc.at, st.Requests, st.Opened, st.Rounds, st.OpeningDone, st.LongestGap, tc.requests, tc.opened, tc.rounds, tc.openingDone, tc.longestGap, tc.why) + } + opening += samples(st, Opening) + latency += samples(st, Latency) + if int64(opening) != tc.opened || int64(opening+latency) != st.Successes { + t.Errorf("at %v: %d opening and %d latency samples so far for %d successes; want one opening per opened endpoint (%d) and the rest latency", + tc.at, opening, latency, st.Successes, tc.opened) + } + } + synctest.Wait() + for i, e := range readStats(t, h).Endpoints { + if e.LongestGap != period { + t.Errorf("endpoint %d: longest gap %v, want one period (%v)", i, e.LongestGap, period) + } + } + stopLoop(t, h) + }) +} + +// TestOffsetShiftsEveryEndpoint pins that offsetMs delays the whole schedule +// and nothing else: endpoint 0 opens at the offset, the last endpoint at the +// offset plus C-1 intervals, and every gap is still one period. +func TestOffsetShiftsEveryEndpoint(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + pn := newPipeNet(http.HandlerFunc(okHandler)) + defer pn.srv.Close() + const c, interval, offset = 4, 100 * time.Millisecond, 50 * time.Millisecond + const period = c * interval + start := time.Now() + h, stats := startPipeLoop(t, pn, StartRequest{Endpoints: c, IntervalMs: int64(interval / time.Millisecond), OffsetMs: int64(offset / time.Millisecond)}) + for _, tc := range []struct { + at time.Duration + requests, opened, rounds int64 + openingDone, longestGap time.Duration + why string + }{ + {offset / 2, 0, 0, 0, 0, 0, "nothing opens before the offset"}, + {offset + interval/2, 1, 1, 0, 0, 0, "endpoint 0 opens at the offset"}, + {offset + period - interval/2, c, c, 1, offset + (c-1)*interval, 0, "the last open completes at the offset plus C-1 intervals"}, + {offset + 2*period - interval/2, 2 * c, c, 2, offset + (c-1)*interval, period, "each repeat comes one period after the open it follows"}, + } { + time.Sleep(tc.at - time.Since(start)) + st := stats() + if st.Requests != tc.requests || st.Opened != tc.opened || st.Rounds != tc.rounds || st.OpeningDone != tc.openingDone || st.LongestGap != tc.longestGap { + t.Errorf("at %v: requests %d, opened %d, rounds %d, opening done %v, longest gap %v; want %d, %d, %d, %v, %v (%s)", + tc.at, st.Requests, st.Opened, st.Rounds, st.OpeningDone, st.LongestGap, tc.requests, tc.opened, tc.rounds, tc.openingDone, tc.longestGap, tc.why) + } + } + for i, e := range stats().Endpoints { + if e.LongestGap != period { + t.Errorf("endpoint %d: longest gap %v, want one period (%v)", i, e.LongestGap, period) + } + } + stopLoop(t, h) + }) +} + +// TestSlowEndpointDelaysOnlyItself pins that a request hanging until its +// timeout on one endpoint leaves the other endpoints' gaps at one period, and +// that the hung endpoint's own gap is the timeout: its next request starts as +// soon as the slow one returns. +func TestSlowEndpointDelaysOnlyItself(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + pn := newPipeNet(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Host == "endpoint-0" { + <-r.Context().Done() + return + } + okHandler(w, r) + })) + defer pn.srv.Close() + const c, interval, timeout = 3, 50 * time.Millisecond, 300 * time.Millisecond + const period = c * interval + h, stats := startPipeLoop(t, pn, StartRequest{Endpoints: c, IntervalMs: int64(interval / time.Millisecond), RequestTimeoutMs: int64(timeout / time.Millisecond)}) + + time.Sleep(time.Second + interval/2) + st := stats() + for i := 1; i < c; i++ { + if got := st.Endpoints[i].LongestGap; got != period { + t.Errorf("endpoint %d: longest gap %v, want one period (%v) with endpoint 0 hung", i, got, period) + } + if st.Endpoints[i].Errors != 0 { + t.Errorf("endpoint %d: %d errors, want 0", i, st.Endpoints[i].Errors) + } + } + if st.Endpoints[0].LongestGap != timeout || st.LongestGap != timeout { + t.Errorf("endpoint 0 longest gap %v, overall %v; want the timeout (%v) for both", st.Endpoints[0].LongestGap, st.LongestGap, timeout) + } + if st.Errors["timeout"] == 0 || st.Errors["timeout"] != st.Endpoints[0].Errors { + t.Errorf("errors %v, endpoint 0 errors %d; want every timeout on endpoint 0", st.Errors, st.Endpoints[0].Errors) + } + if st.Opened != c || st.OpeningDone != timeout { + t.Errorf("Opened = %d, OpeningDone = %v; want %d and %v: the hung open counts once it times out, and it completes last", st.Opened, st.OpeningDone, c, timeout) + } + stopLoop(t, h) + }) +} + +// TestSlowOpensDoNotDelayOpenedEndpoints pins the schedule when opens are +// slow, as they are while a gateway works through a fleet's handshakes: an +// endpoint already opened is requested one period after its open while the +// others are still opening, and a slow open stretches only its own +// endpoint's first gap, to the open's latency. +func TestSlowOpensDoNotDelayOpenedEndpoints(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + const c, interval, openLatency = 4, 100 * time.Millisecond, 500 * time.Millisecond + const period = c * interval + var mu sync.Mutex + opened := map[string]bool{} + pn := newPipeNet(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + first := !opened[r.Host] + opened[r.Host] = true + mu.Unlock() + if first && r.Host != "endpoint-0" { + select { + case <-time.After(openLatency): + case <-r.Context().Done(): + return + } + } + okHandler(w, r) + })) + defer pn.srv.Close() + start := time.Now() + h, stats := startPipeLoop(t, pn, StartRequest{Endpoints: c, IntervalMs: int64(interval / time.Millisecond)}) + + // endpoint 0 opened at once; its second request is due at one period, + // while endpoints 1 to 3 are still inside their first. + time.Sleep(period + interval/2) + st := stats() + if st.Opened != 1 || st.Endpoints[0].Requests != 2 || st.Endpoints[0].LongestGap != period || st.Requests != 2 { + t.Errorf("at %v: opened %d, endpoint 0 %d requests with gap %v, %d requests in all; want 1, 2, %v, 2: the opened endpoint keeps its period while the others open", + time.Since(start), st.Opened, st.Endpoints[0].Requests, st.Endpoints[0].LongestGap, st.Requests, period) + } + + // The last open, endpoint 3's, started at 3 intervals and took + // openLatency; after that each slow endpoint's next request came + // at once, since its period had already passed. + time.Sleep(3*period + interval/2 - time.Since(start)) + st = stats() + if st.Opened != c || st.OpeningDone != (c-1)*interval+openLatency { + t.Errorf("Opened = %d, OpeningDone = %v; want %d and %v", st.Opened, st.OpeningDone, c, (c-1)*interval+openLatency) + } + if st.Endpoints[0].LongestGap != period || st.Endpoints[0].Requests != 4 { + t.Errorf("endpoint 0: longest gap %v, %d requests; want %v and 4: the slow opens of the others did not touch it", st.Endpoints[0].LongestGap, st.Endpoints[0].Requests, period) + } + for i := 1; i < c; i++ { + if got := st.Endpoints[i].LongestGap; got != openLatency { + t.Errorf("endpoint %d: longest gap %v, want its own open latency (%v)", i, got, openLatency) + } + } + final := stopLoop(t, h) + if op := final.Distributions[Opening]; st.LongestGap != openLatency || op.Count != c || op.Max != openLatency.Seconds() { + t.Errorf("LongestGap = %v, opening samples %d with max %v; want %v, %d and %v", st.LongestGap, op.Count, op.Max, openLatency, c, openLatency.Seconds()) + } + }) +} + +// TestSlowRequestResumesOnTheNextTick pins what a request longer than the +// period does to its own endpoint: the next request starts as soon as the +// slow one returns, so the gap is the request's latency, not latency plus a +// period. +func TestSlowRequestResumesOnTheNextTick(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + const period, latency = 50 * time.Millisecond, 120 * time.Millisecond + var mu sync.Mutex + var starts []time.Duration + start := time.Now() + pn := newPipeNet(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + starts = append(starts, time.Since(start)) + mu.Unlock() + select { + case <-time.After(latency): + okHandler(w, r) + case <-r.Context().Done(): + } + })) + defer pn.srv.Close() + h, stats := startPipeLoop(t, pn, StartRequest{Endpoints: 1, IntervalMs: int64(period / time.Millisecond)}) + + time.Sleep(4*latency + period/2) + st := stats() + mu.Lock() + got := slices.Clone(starts) + mu.Unlock() + want := []time.Duration{0, latency, 2 * latency, 3 * latency, 4 * latency} + if !slices.Equal(got, want) { + t.Errorf("request starts = %v, want %v: back to back once each outlasts the period", got, want) + } + if st.LongestGap != latency { + t.Errorf("LongestGap = %v, want the request latency (%v)", st.LongestGap, latency) + } + stopLoop(t, h) + }) +} + +// TestStopWithRequestsInFlightReturnsAtOnce pins that a stop neither waits +// for the in-flight requests' timeouts nor leaks their goroutines. +func TestStopWithRequestsInFlightReturnsAtOnce(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + pn := newPipeNet(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) { <-r.Context().Done() })) + defer pn.srv.Close() + const c = 8 + h, _ := startPipeLoop(t, pn, StartRequest{Endpoints: c, RequestTimeoutMs: 5000}) + time.Sleep(10 * time.Millisecond) + synctest.Wait() + + before := time.Now() + st := stopLoop(t, h) + if waited := time.Since(before); waited != 0 { + t.Errorf("/stop took %v with %d requests in flight, want 0: the stop must not wait for a timeout", waited, c) + } + if st.Requests != 0 || st.Opened != 0 || len(st.Distributions) != 0 { + t.Errorf("Requests = %d, Opened = %d, distributions %v after stopping mid-open, want 0, 0 and none", st.Requests, st.Opened, st.Distributions) + } + if rec := get(h, StatsRoute); rec.Code != http.StatusNotFound { + t.Errorf("GET /stats after stop = %d, want 404", rec.Code) + } + }) +} + +// TestTwoCutsPartitionTheRun pins what a window is: each /collect answers +// with the samples since the previous one, so the windows of a run add up to +// the whole, and a window's min and max come from its own samples alone. +func TestTwoCutsPartitionTheRun(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + const c, interval = 2, 100 * time.Millisecond + const fast, slow = 10 * time.Millisecond, 100 * time.Millisecond + var delay atomic.Int64 + delay.Store(int64(fast)) + pn := newPipeNet(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + select { + case <-time.After(time.Duration(delay.Load())): + okHandler(w, r) + case <-r.Context().Done(): + } + })) + defer pn.srv.Close() + start := time.Now() + h, _ := startPipeLoop(t, pn, StartRequest{Endpoints: c, IntervalMs: int64(interval / time.Millisecond)}) + cut := func(w int64, at time.Duration) *Stats { + time.Sleep(at - time.Since(start)) + synctest.Wait() + return collectStats(t, h, w) + } + + // Two opens and one repeat each, all at the fast latency. + first := cut(1, 350*time.Millisecond) + if op, l := first.Distributions[Opening], first.Distributions[Latency]; first.Successes != 4 || op.Count != 2 || l.Count != 2 || l.Min != fast.Seconds() || l.Max != fast.Seconds() { + t.Errorf("first window: %d successes, opening %d, latency %d in [%v, %v]; want 4, 2, 2 in [%v, %v]", first.Successes, op.Count, l.Count, l.Min, l.Max, fast.Seconds(), fast.Seconds()) + } + + // Four repeats at the slow latency: the second window must not + // remember the fast ones, and a /stats poll inside it must not + // consume them: only /collect cuts. + delay.Store(int64(slow)) + time.Sleep(650*time.Millisecond - time.Since(start)) + synctest.Wait() + if polled := readStats(t, h); polled.Successes != 6 || polled.Distributions != nil { + t.Errorf("/stats inside the second window: %d successes, distributions %v; want 6 and none", polled.Successes, polled.Distributions) + } + second := cut(2, 850*time.Millisecond) + if _, ok := second.Distributions[Opening]; ok { + t.Errorf("second window has an opening distribution: %+v", second.Distributions[Opening]) + } + if l := second.Distributions[Latency]; second.Successes != 8 || l.Count != 4 || l.Min != slow.Seconds() || l.Max != slow.Seconds() { + t.Errorf("second window: %d successes, latency %d in [%v, %v]; want 8, 4 in [%v, %v]", second.Successes, l.Count, l.Min, l.Max, slow.Seconds(), slow.Seconds()) + } + + // Three more repeats, then the stop's tail window. + time.Sleep(1150*time.Millisecond - time.Since(start)) + synctest.Wait() + final := stopLoop(t, h) + var total uint64 + for _, s := range []*Stats{first, second, final} { + total += samples(s, Opening) + samples(s, Latency) + } + if final.Successes != 11 || int64(total) != final.Successes { + t.Errorf("windows hold %d samples for %d successes in all; want 11 and 11", total, final.Successes) + } + }) +} + +// TestSettledWindowHasNoDNSOrTLS pins the emptiness the driver checks in a +// settled window: once every tunnel is open, a window with no opens in it +// carries no lookup and no handshake. +func TestSettledWindowHasNoDNSOrTLS(t *testing.T) { + t.Parallel() + byName := newTarget(t, okHandler) + u, err := url.Parse(byName.URL) + if err != nil { + t.Fatal(err) + } + tlsTarget, roots := newTLSTarget(t, okHandler) + for _, tc := range []struct { + name string + h http.Handler + req StartRequest + dist Distribution + }{ + {"dns over http by name", serve("localhost:" + u.Port()), StartRequest{Endpoints: 1}, DNS}, + {"tls over https", serveTLS(roots, tlsTarget.host()), StartRequest{Endpoints: 1, Scheme: SchemeHTTPS}, TLS}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + if rec := start(t, tc.h, tc.req); rec.Code != http.StatusNoContent { + t.Fatalf("POST /start = %d %s", rec.Code, rec.Body) + } + waitFor(t, tc.h, atLeast(2)) + first := collectStats(t, tc.h, 1) + if samples(first, tc.dist) == 0 { + t.Errorf("the window with the open has no %s sample: %+v", tc.dist, first.Distributions) + } + waitFor(t, tc.h, atLeast(first.Requests+2)) + second := collectStats(t, tc.h, 2) + if p, ok := second.Distributions[tc.dist]; ok { + t.Errorf("a window with no open carries %s: %+v", tc.dist, p) + } + if samples(second, Latency) == 0 || second.Successes <= first.Successes { + t.Errorf("second window: latency %d samples, successes %d after %d; want repeats in it", samples(second, Latency), second.Successes, first.Successes) + } + stopLoop(t, tc.h) + }) + } +} + +func TestStopReturnsFinalStats(t *testing.T) { + t.Parallel() + tg := newTarget(t, okHandler) + h := serve(tg.host()) + for _, tc := range []struct { + name string + rec *httptest.ResponseRecorder + }{ + {"GET /stats", get(h, StatsRoute)}, + {"POST /collect", post(t, h, CollectRoute, nil)}, + {"POST /stop", post(t, h, StopRoute, nil)}, + } { + if tc.rec.Code != http.StatusNotFound || strings.TrimSpace(tc.rec.Body.String()) != NoLoopMessage { + t.Errorf("%s with no loop = %d %q, want 404 %q", tc.name, tc.rec.Code, tc.rec.Body, NoLoopMessage) + } + } + if rec := start(t, h, StartRequest{Endpoints: 1}); rec.Code != http.StatusNoContent { + t.Fatalf("POST /start = %d %s", rec.Code, rec.Body) + } + waitFor(t, h, atLeast(1)) + rec := post(t, h, StopRoute, nil) + final := decodeStats(t, "POST /stop", rec) + if ct := rec.Header().Get("Content-Type"); ct != "application/json" || final.Scheme != SchemeHTTP || final.Endpoints != nil || final.Requests == 0 { + t.Errorf("POST /stop answered %q with scheme %q, endpoints %v and %d requests; want JSON, http, no endpoints unless asked, and the requests made", ct, final.Scheme, final.Endpoints, final.Requests) + } + for _, tc := range []struct { + name string + rec *httptest.ResponseRecorder + }{ + {"GET /stats", get(h, StatsRoute)}, + {"POST /collect", post(t, h, collectPath(1, false), nil)}, + } { + if tc.rec.Code != http.StatusNotFound { + t.Errorf("%s after stop = %d, want 404", tc.name, tc.rec.Code) + } + } + + // The final report is kept: a /stop the driver did not hear is answered + // again, byte for byte, with or without the endpoints, until a new loop + // replaces it. + if again := post(t, h, StopRoute, nil); again.Code != http.StatusOK || !bytes.Equal(again.Body.Bytes(), rec.Body.Bytes()) { + t.Errorf("second POST /stop = %d %s; want 200 and the first answer %s", again.Code, again.Body, rec.Body) + } + if withEP := decodeStats(t, "third POST /stop", post(t, h, stopPath, nil)); len(withEP.Endpoints) != 1 || withEP.Requests != final.Requests { + t.Errorf("POST /stop?endpoints=1 after stop: %d endpoints, %d requests; want 1 and %d", len(withEP.Endpoints), withEP.Requests, final.Requests) + } + if rec := start(t, h, StartRequest{Endpoints: 1, IntervalMs: 60_000, OffsetMs: 60_000}); rec.Code != http.StatusNoContent { + t.Fatalf("second POST /start = %d %s", rec.Code, rec.Body) + } + if next := stopLoop(t, h); next.Requests != 0 { + t.Errorf("the stop after a new /start reports %d requests; want 0, the new loop's own report", next.Requests) + } +} + +// TestCollectWindowIDs drives /collect through the ids a driver sends and the +// ones it must not: the first cut is 1, the last id is answered again, the +// next id cuts, and anything else is refused with the next id named. +func TestCollectWindowIDs(t *testing.T) { + t.Parallel() + h := serveTargets(t, 1) + if rec := start(t, h, StartRequest{Endpoints: 1, IntervalMs: 60_000, OffsetMs: 60_000}); rec.Code != http.StatusNoContent { + t.Fatalf("POST /start = %d %s", rec.Code, rec.Body) + } + t.Cleanup(func() { stopLoop(t, h) }) + route := CollectRoute + "?" + WindowParam + "=" + // The steps run in order on one loop; each names the next id the body + // must carry when the cut is refused. + for _, tc := range []struct { + name string + path string + want int + next int64 + }{ + {"missing", CollectRoute, http.StatusBadRequest, 1}, + {"zero", route + "0", http.StatusBadRequest, 1}, + {"negative", route + "-1", http.StatusBadRequest, 1}, + {"non-numeric", route + "one", http.StatusBadRequest, 1}, + {"skip ahead before the first", route + "2", http.StatusConflict, 1}, + {"first cut is 1", route + "1", http.StatusOK, 0}, + {"last again", route + "1", http.StatusOK, 0}, + {"last again with endpoints", route + "1&endpoints=1", http.StatusOK, 0}, + {"next cuts", route + "2", http.StatusOK, 0}, + {"lower", route + "1", http.StatusConflict, 3}, + {"zero after cuts", route + "0", http.StatusBadRequest, 3}, + {"skip ahead", route + "4", http.StatusConflict, 3}, + {"next cuts again", route + "3", http.StatusOK, 0}, + } { + t.Run(tc.name, func(t *testing.T) { + rec := post(t, h, tc.path, nil) + if rec.Code != tc.want { + t.Fatalf("POST %s = %d %s, want %d", tc.path, rec.Code, rec.Body, tc.want) + } + if tc.next == 0 { + decodeStats(t, "POST "+tc.path, rec) + return + } + if want := fmt.Sprintf("the next window is %d", tc.next); !strings.Contains(rec.Body.String(), want) { + t.Errorf("POST %s body %q does not say %q", tc.path, rec.Body, want) + } + }) + } +} + +// TestRepeatedCollectConsumesNothing pins the retry: the same id answered +// again is the same bytes, the sample recorded between the two answers not +// in it, and that sample is in the next window, not lost. +func TestRepeatedCollectConsumesNothing(t *testing.T) { + t.Parallel() + run, err := newLoopRun(StartRequest{Endpoints: 1}) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = run.rec.shutdown(context.Background()) }) + s := newServer() + s.run = run + h := s.handler() + run.record(0, 5*time.Millisecond, http.StatusOK, nil, &requestTrace{}) + first := post(t, h, collectPath(1, false), nil) + run.record(0, 7*time.Millisecond, http.StatusOK, nil, &requestTrace{}) + again := post(t, h, collectPath(1, false), nil) + if first.Code != http.StatusOK || again.Code != http.StatusOK || !bytes.Equal(first.Body.Bytes(), again.Body.Bytes()) { + t.Errorf("window 1 twice = %d %s and %d %s; want 200 and the same bytes", first.Code, first.Body, again.Code, again.Body) + } + withEP := decodeStats(t, "window 1 with endpoints", post(t, h, collectPath(1, true), nil)) + if withEP.Requests != 1 || len(withEP.Endpoints) != 1 || samples(withEP, Opening) != 1 { + t.Errorf("window 1 with endpoints: %d requests, %d endpoints, %d opening samples; want 1, 1, 1", withEP.Requests, len(withEP.Endpoints), samples(withEP, Opening)) + } + second := collectStats(t, h, 2) + if l := second.Distributions[Latency]; second.Requests != 2 || l.Count != 1 || l.Min != 0.007 { + t.Errorf("window 2: %d requests, latency %d samples with min %v; want 2, 1 and 0.007: the sample between the two answers", second.Requests, l.Count, l.Min) + } +} + +// TestRequestsAtOpeningDone pins the count the blended window prices: the +// requests this actor had finished when its last endpoint opened, repeats on +// the endpoints already open included, and fixed from then on. +func TestRequestsAtOpeningDone(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + const c, interval, openLatency = 2, 100 * time.Millisecond, 250 * time.Millisecond + var mu sync.Mutex + opened := map[string]bool{} + pn := newPipeNet(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + first := !opened[r.Host] + opened[r.Host] = true + mu.Unlock() + if first && r.Host == "endpoint-1" { + select { + case <-time.After(openLatency): + case <-r.Context().Done(): + return + } + } + okHandler(w, r) + })) + defer pn.srv.Close() + start := time.Now() + h, stats := startPipeLoop(t, pn, StartRequest{Endpoints: c, IntervalMs: int64(interval / time.Millisecond)}) + + // Endpoint 0 opens at 0 and repeats at 200; endpoint 1 opens at 100 + // and finishes at 350: the third request. Its first repeat follows at + // once, since its period passed during the open. + time.Sleep(interval + openLatency - time.Since(start)) + st := stats() + if st.Opened != c || st.OpeningDone != interval+openLatency || st.RequestsAtOpeningDone != 3 || st.Requests != 4 { + t.Errorf("at %v: opened %d at %v with %d requests, %d requests now; want %d at %v with 3, 4", time.Since(start), st.Opened, st.OpeningDone, st.RequestsAtOpeningDone, st.Requests, c, interval+openLatency) + } + time.Sleep(time.Second - time.Since(start)) + if st = stats(); st.RequestsAtOpeningDone != 3 || st.Requests <= 4 { + t.Errorf("at %v: requestsAtOpeningDone %d with %d requests; want still 3 with more than 4", time.Since(start), st.RequestsAtOpeningDone, st.Requests) + } + if final := stopLoop(t, h); final.RequestsAtOpeningDone != 3 { + t.Errorf("final requestsAtOpeningDone = %d, want 3", final.RequestsAtOpeningDone) + } + }) +} + +// TestFreshRecorderPerLoop pins that a loop starts from nothing: the first +// collect after start-stop-start is an empty window. +func TestFreshRecorderPerLoop(t *testing.T) { + t.Parallel() + h := serveTargets(t, 2) + first := runLoop(t, h, StartRequest{Endpoints: 2}, atLeast(10)) + if samples(first, Latency) == 0 { + t.Fatalf("first loop recorded no latency over %d requests", first.Requests) + } + // Nothing in the new loop fires for a minute, so its first collect sees + // only what the previous loop might have left behind. + if rec := start(t, h, StartRequest{Endpoints: 2, IntervalMs: 60_000, OffsetMs: 60_000}); rec.Code != http.StatusNoContent { + t.Fatalf("POST /start = %d %s", rec.Code, rec.Body) + } + second := collectStats(t, h, 1) + if second.Requests != 0 || second.Opened != 0 || len(second.Distributions) != 0 { + t.Errorf("second loop's first collect: %d requests, %d opened, distributions %v; want an empty window", second.Requests, second.Opened, second.Distributions) + } + stopLoop(t, h) +} + +// TestCollectInvariantUnderRace cuts windows while requests land: every +// success records exactly one opening or latency sample, so each window's +// samples equal the growth of Successes since the previous cut, and the +// windows add up to the final count. Every tenth cut is asked for twice, as +// a driver retries it, and must come back unchanged. +func TestCollectInvariantUnderRace(t *testing.T) { + t.Parallel() + h := serveTargets(t, 8) + if rec := start(t, h, StartRequest{Endpoints: 8}); rec.Code != http.StatusNoContent { + t.Fatalf("POST /start = %d %s", rec.Code, rec.Body) + } + var wg sync.WaitGroup + var badReads atomic.Int64 + for range 4 { + wg.Go(func() { + for range 50 { + st := readStats(t, h) + var failed int64 + for _, n := range st.Errors { + failed += n + } + if st.Successes+failed != st.Requests || st.Distributions != nil { + badReads.Add(1) + } + } + }) + } + var total uint64 + var prev int64 + var badCuts, badRetries int + for id := int64(1); id <= 200; id++ { + rec := post(t, h, collectPath(id, true), nil) + w := decodeStats(t, "POST /collect", rec) + if id%10 == 0 { + if again := post(t, h, collectPath(id, true), nil); !bytes.Equal(again.Body.Bytes(), rec.Body.Bytes()) { + badRetries++ + } + } + got := samples(w, Opening) + samples(w, Latency) + if int64(got) != w.Successes-prev { + badCuts++ + } + prev, total = w.Successes, total+got + } + wg.Wait() + final := stopLoop(t, h) + total += samples(final, Opening) + samples(final, Latency) + if badReads.Load() > 0 || badCuts > 0 || badRetries > 0 { + t.Errorf("%d reads broke successes + errors = requests, %d cuts broke window = growth of successes, %d retried cuts came back changed", badReads.Load(), badCuts, badRetries) + } + if int64(total) != final.Successes || final.Successes == 0 { + t.Errorf("windows hold %d samples for %d successes in all", total, final.Successes) + } +} + +// TestCollectDuringStopIsNever500 races a driver's cuts and many reads +// against a stop: each one answers 200 while the loop runs or 404 once it is +// stopped, never a 500 from a shut-down reader. +func TestCollectDuringStopIsNever500(t *testing.T) { + t.Parallel() + h := serveTargets(t, 8) + if rec := start(t, h, StartRequest{Endpoints: 8}); rec.Code != http.StatusNoContent { + t.Fatalf("POST /start = %d %s", rec.Code, rec.Body) + } + var wg sync.WaitGroup + var ok, gone, other atomic.Int64 + stopped := make(chan struct{}) + for i := range 16 { + wg.Go(func() { + var window int64 + for { + var rec *httptest.ResponseRecorder + if i == 0 { + window++ + rec = post(t, h, collectPath(window, false), nil) + } else { + rec = get(h, StatsRoute) + } + switch { + case rec.Code == http.StatusOK: + ok.Add(1) + case rec.Code == http.StatusNotFound && strings.TrimSpace(rec.Body.String()) == NoLoopMessage: + gone.Add(1) + default: + other.Add(1) + t.Errorf("answered %d %s", rec.Code, rec.Body) + } + select { + case <-stopped: + return + default: + } + } + }) + } + time.Sleep(20 * time.Millisecond) + stopLoop(t, h) + close(stopped) + wg.Wait() + if rec := post(t, h, collectPath(1, false), nil); rec.Code != http.StatusNotFound { + t.Errorf("POST /collect after stop = %d, want 404", rec.Code) + } + if ok.Load() == 0 || other.Load() != 0 { + t.Errorf("%d answers were 200, %d 404, %d something else", ok.Load(), gone.Load(), other.Load()) + } +} + +// TestEveryRouteCarriesProtocolHeader checks the header a driver reads to +// tell this actor from an image built before this contract, on every answer +// the actor gives, errors and unknown routes included. The value is 2: the +// images that sent 1 took no window id on /collect. +func TestEveryRouteCarriesProtocolHeader(t *testing.T) { + t.Parallel() + if ProtocolVersion != "2" { + t.Errorf("ProtocolVersion = %q, want 2; a driver requires it", ProtocolVersion) + } + h := newServer().handler() + for _, tc := range []struct { + name string + rec *httptest.ResponseRecorder + want int + }{ + {"readyz", get(h, ReadyzRoute), http.StatusOK}, + {"start rejected", post(t, h, StartRoute, map[string]int{"endpoints": 0}), http.StatusBadRequest}, + {"stats without a loop", get(h, StatsRoute), http.StatusNotFound}, + {"collect without a loop", post(t, h, collectPath(1, false), nil), http.StatusNotFound}, + {"stop without a loop", post(t, h, StopRoute, nil), http.StatusNotFound}, + {"unknown route", get(h, "/metrics"), http.StatusNotFound}, + } { + if tc.rec.Code != tc.want { + t.Errorf("%s: status %d, want %d", tc.name, tc.rec.Code, tc.want) + } + if got := tc.rec.Header().Get(ProtocolHeader); got != ProtocolVersion { + t.Errorf("%s: %s = %q, want %q", tc.name, ProtocolHeader, got, ProtocolVersion) + } + } +} + +// TestPayloadBudget pins the answer sizes at the largest C, since the driver +// polls every actor: /stats stays under 1 KB, a /collect carries the four +// distributions in a few KB, and the per-endpoint counts come only when +// asked for. +func TestPayloadBudget(t *testing.T) { + t.Parallel() + const c = MaxEndpoints + run, err := newLoopRun(StartRequest{Endpoints: c, Scheme: SchemeHTTPS}) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = run.rec.shutdown(context.Background()) }) + run.stats.Requests, run.stats.Successes, run.stats.NewConns, run.stats.Opened = 12345*c, 12339*c, c, c + run.stats.OpeningDone, run.stats.LongestGap = 102400*time.Millisecond, 12345678*time.Microsecond + run.stats.Errors = map[string]int64{"timeout": 3 * c, "HTTP 503": 3 * c} + run.rounds = 12345 + for i := range c { + run.stats.Endpoints[i] = Endpoint{Requests: 12345, Errors: 6, LongestGap: 12345678 * time.Microsecond} + } + // Every bucket non-zero, so no count is a short "0". + fill := func() { + for _, name := range DistributionNames { + for _, b := range Bounds { + run.rec.record(context.Background(), name, b) + } + } + } + s := newServer() + s.run = run + h := s.handler() + + fill() + collect := post(t, h, collectPath(1, false), nil) + fill() + collectEndpoints := post(t, h, collectPath(2, true), nil) + stats := get(h, StatsRoute) + t.Logf("at C=%d: /stats %d B, /collect %d B, /collect?endpoints=1 %d B", c, stats.Body.Len(), collect.Body.Len(), collectEndpoints.Body.Len()) + for _, tc := range []struct { + name string + rec *httptest.ResponseRecorder + budget int + endpoints bool + distributions bool + }{ + {"/stats", stats, 1 << 10, false, false}, + {"/collect", collect, 12 << 10, false, true}, + {"/collect?endpoints=1", collectEndpoints, 80 << 10, true, true}, + } { + if tc.rec.Code != http.StatusOK { + t.Fatalf("%s = %d %s", tc.name, tc.rec.Code, tc.rec.Body) + } + if tc.rec.Body.Len() >= tc.budget { + t.Errorf("%s = %d B, budget %d B", tc.name, tc.rec.Body.Len(), tc.budget) + } + var st Stats + if err := json.Unmarshal(tc.rec.Body.Bytes(), &st); err != nil { + t.Fatal(err) + } + if (len(st.Endpoints) == c) != tc.endpoints || (len(st.Distributions) == len(DistributionNames)) != tc.distributions || st.Rounds != 12345 { + t.Errorf("%s carries %d endpoints, %d distributions and rounds %d; want endpoints %v, distributions %v and rounds 12345", tc.name, len(st.Endpoints), len(st.Distributions), st.Rounds, tc.endpoints, tc.distributions) + } + } +} diff --git a/cmd/benchmarking/egress/internal/cmd/actor/api.go b/cmd/benchmarking/egress/internal/cmd/actor/api.go new file mode 100644 index 0000000000..afd0bad3c5 --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/api.go @@ -0,0 +1,129 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package actor is the workload of the egress scale test: the subcommand +// that holds one tunnel per endpoint open through the egress gateway and +// counts what happened. Its contract, what the programs of the test agree +// on, is in api.go, endpoints.go, stats.go and hist.go: the endpoint names +// the loop calls, the control routes and the /start body a driver sends it, +// and the Stats it answers with, down to the bounds of its distributions. +package actor + +import "fmt" + +// Control routes the actor serves on port 80, the only inbound port of a +// sandbox. +const ( + ReadyzRoute = "/readyz" + StartRoute = "/start" + // StatsRoute, GET, serves the running loop's Stats without its + // distributions: the counters, cheap enough to poll, and nothing is + // consumed by a read. ?endpoints=1 adds the per-endpoint counts. + StatsRoute = "/stats" + // CollectRoute, POST, cuts a window: it serves the Stats with the + // distributions recorded since the previous cut, or since /start for the + // first. Each sample is served once, so the route has one consumer: the + // driver, which cuts the run into windows with it. The driver numbers the + // cuts with WindowParam so a cut it did not hear can be asked for again. + // ?endpoints=1 as on /stats. + CollectRoute = "/collect" + // StopRoute, POST, ends the loop and serves its final Stats: the counters + // and the distributions since the previous cut. The actor keeps that + // final Stats once the loop is gone, so a repeated /stop answers it again + // with 200 until the next /start, while /stats and /collect answer 404. + // ?endpoints=1 as on /stats. + StopRoute = "/stop" +) + +// WindowParam is the query parameter of POST /collect: the id of the window +// the driver asks for, a positive integer. The first cut is 1 and each cut +// is the previous id plus one. The actor keeps the last cut's Stats, so the +// same id asked again is answered from it, byte for byte, and consumes +// nothing. A missing or malformed value is 400; an id that is neither the +// last cut nor the next is 409; either body names the next id. +const WindowParam = "window" + +// NoLoopMessage is the body of the 404 that /stats, /collect and /stop answer +// while no loop runs. +const NoLoopMessage = "no loop is running" + +// ProtocolHeader is on every answer the actor gives, ProtocolVersion its +// value, so a driver can tell what it is talking to before it starts +// anything: an actor whose image speaks another contract is refused rather +// than misread. The images that pushed their measurements over OTLP sent 1 +// and served no /collect; a driver must require 2. +const ( + ProtocolHeader = "Benchmark-Egress-Protocol" + ProtocolVersion = "2" +) + +// DefaultRequestTimeoutMs applies when StartRequest.RequestTimeoutMs is zero. +const DefaultRequestTimeoutMs = 5000 + +// The schemes the loop can request endpoints over. +const ( + SchemeHTTP = "http" + SchemeHTTPS = "https" +) + +// StartRequest is the JSON body of POST /start. +type StartRequest struct { + // Endpoints is C: the loop requests EndpointURL(0) through + // EndpointURL(Endpoints-1), each on its own goroutine, until it is stopped. + Endpoints int `json:"endpoints"` + // RequestTimeoutMs bounds one request; zero means DefaultRequestTimeoutMs. + RequestTimeoutMs int64 `json:"requestTimeoutMs"` + // IntervalMs paces the actor: endpoint i's first request starts OffsetMs + // plus i intervals after the loop, and each endpoint is then requested once + // per period of C intervals, so the actor sends one request per interval + // throughout. Zero sends every endpoint's requests back to back. + IntervalMs int64 `json:"intervalMs"` + // OffsetMs delays the whole schedule: endpoint i's first request starts + // OffsetMs plus i intervals after the loop. A driver gives each actor its + // own value so a fleet started at once does not send in lockstep. + OffsetMs int64 `json:"offsetMs,omitempty"` + // Scheme is SchemeHTTP or SchemeHTTPS; empty means SchemeHTTP. + Scheme string `json:"scheme,omitempty"` +} + +// URLScheme is the scheme the loop requests endpoints over. +func (r *StartRequest) URLScheme() string { + if r.Scheme == "" { + return SchemeHTTP + } + return r.Scheme +} + +// Validate reports whether r can start a loop. +func (r *StartRequest) Validate() error { + if r.Endpoints < 1 || r.Endpoints > MaxEndpoints { + return fmt.Errorf("endpoints must be between 1 and %d, got %d", MaxEndpoints, r.Endpoints) + } + if r.RequestTimeoutMs < 0 { + return fmt.Errorf("requestTimeoutMs cannot be negative: %d", r.RequestTimeoutMs) + } + if r.IntervalMs < 0 { + return fmt.Errorf("intervalMs cannot be negative: %d", r.IntervalMs) + } + if r.OffsetMs < 0 { + return fmt.Errorf("offsetMs cannot be negative: %d", r.OffsetMs) + } + if period := int64(r.Endpoints) * r.IntervalMs; r.OffsetMs > period { + return fmt.Errorf("offsetMs cannot exceed one period, endpoints x intervalMs = %d: %d", period, r.OffsetMs) + } + if r.Scheme != "" && r.Scheme != SchemeHTTP && r.Scheme != SchemeHTTPS { + return fmt.Errorf("scheme must be %s or %s, got %q", SchemeHTTP, SchemeHTTPS, r.Scheme) + } + return nil +} diff --git a/cmd/benchmarking/egress/internal/cmd/actor/api_test.go b/cmd/benchmarking/egress/internal/cmd/actor/api_test.go new file mode 100644 index 0000000000..97cf164871 --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/api_test.go @@ -0,0 +1,58 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actor + +import "testing" + +func TestStartRequestValidate(t *testing.T) { + t.Parallel() + for _, tc := range []struct { + name string + req StartRequest + wantErr bool + }{ + {"ok", StartRequest{Endpoints: 10}, false}, + {"http", StartRequest{Endpoints: 10, Scheme: SchemeHTTP}, false}, + {"https", StartRequest{Endpoints: 10, Scheme: SchemeHTTPS}, false}, + {"bad scheme", StartRequest{Endpoints: 10, Scheme: "ftp"}, true}, + {"most endpoints", StartRequest{Endpoints: MaxEndpoints}, false}, + {"no endpoints", StartRequest{}, true}, + {"too many endpoints", StartRequest{Endpoints: MaxEndpoints + 1}, true}, + {"negative timeout", StartRequest{Endpoints: 1, RequestTimeoutMs: -1}, true}, + {"negative interval", StartRequest{Endpoints: 1, IntervalMs: -1}, true}, + {"offset", StartRequest{Endpoints: 3, IntervalMs: 100, OffsetMs: 50}, false}, + {"offset of one period", StartRequest{Endpoints: 3, IntervalMs: 100, OffsetMs: 300}, false}, + {"offset over one period", StartRequest{Endpoints: 3, IntervalMs: 100, OffsetMs: 301}, true}, + {"negative offset", StartRequest{Endpoints: 1, IntervalMs: 100, OffsetMs: -1}, true}, + {"offset without interval", StartRequest{Endpoints: 1, OffsetMs: 1}, true}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + if err := tc.req.Validate(); (err != nil) != tc.wantErr { + t.Errorf("Validate() = %v, wantErr %v", err, tc.wantErr) + } + }) + } +} + +func TestURLScheme(t *testing.T) { + t.Parallel() + if got := (&StartRequest{}).URLScheme(); got != SchemeHTTP { + t.Errorf("URLScheme() with no scheme = %q, want %q", got, SchemeHTTP) + } + if got := (&StartRequest{Scheme: SchemeHTTPS}).URLScheme(); got != SchemeHTTPS { + t.Errorf("URLScheme() with https = %q, want %q", got, SchemeHTTPS) + } +} diff --git a/cmd/benchmarking/egress/internal/cmd/actor/classify.go b/cmd/benchmarking/egress/internal/cmd/actor/classify.go new file mode 100644 index 0000000000..09e752a18f --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/classify.go @@ -0,0 +1,95 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actor + +import ( + "context" + "crypto/tls" + "crypto/x509" + "errors" + "fmt" + "io" + "net" + "net/url" + "strings" + "syscall" +) + +// maxOtherLen bounds the message kept for an error classify does not +// recognize, so one odd error cannot flood the report. +const maxOtherLen = 80 + +// classify names the outcome of a request in a few low-cardinality classes. +// It returns "" for a success: no error and HTTP 200. +func classify(err error, status int) string { + if err == nil { + if status == 200 { + return "" + } + return fmt.Sprintf("HTTP %d", status) + } + var netErr net.Error + var dnsErr *net.DNSError + var opErr *net.OpError + switch { + case errors.As(err, &dnsErr): + if dnsErr.IsNotFound { + return "dns: not found" + } + if dnsErr.IsTimeout { + return "dns: timeout" + } + return "dns: error" + case errors.As(err, new(x509.UnknownAuthorityError)): + return "tls: unknown authority" + case errors.As(err, new(x509.HostnameError)): + return "tls: hostname mismatch" + case errors.As(err, new(x509.CertificateInvalidError)): + return "tls: invalid certificate" + case errors.As(err, new(tls.RecordHeaderError)), + // net/http replaces the record error with this when the server + // answered in plain HTTP, without wrapping it. + strings.Contains(err.Error(), "server gave HTTP response to HTTPS client"): + return "tls: not a TLS server" + case errors.As(err, &opErr) && opErr.Op == "remote error": + // crypto/tls reports an alert from the peer this way; its message, + // such as "tls: handshake failure", names the alert. + return "tls: peer alert: " + strings.TrimPrefix(opErr.Err.Error(), "tls: ") + case errors.Is(err, context.DeadlineExceeded), errors.As(err, &netErr) && netErr.Timeout(): + return "timeout" + case errors.Is(err, syscall.ECONNREFUSED): + return "connection refused" + case errors.Is(err, syscall.ECONNRESET): + return "connection reset" + case errors.Is(err, io.EOF), errors.Is(err, io.ErrUnexpectedEOF): + return "EOF" + } + // *url.Error repeats the method and URL, and *net.OpError the addresses + // and the local port; both only add cardinality. + var urlErr *url.Error + if errors.As(err, &urlErr) { + err = urlErr.Err + } + if errors.As(err, &opErr) { + return "net: " + opErr.Op + ": " + opErr.Err.Error() + } + msg := err.Error() + if len(msg) > maxOtherLen { + // Cut on a character boundary, or one message could yield two + // classes depending on where its bytes fall. + msg = strings.ToValidUTF8(msg[:maxOtherLen], "") + } + return "other: " + msg +} diff --git a/cmd/benchmarking/egress/internal/cmd/actor/classify_test.go b/cmd/benchmarking/egress/internal/cmd/actor/classify_test.go new file mode 100644 index 0000000000..0728af73e3 --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/classify_test.go @@ -0,0 +1,79 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actor + +import ( + "context" + "crypto/tls" + "crypto/x509" + "errors" + "io" + "net" + "net/url" + "strings" + "syscall" + "testing" +) + +type timeoutErr struct{} + +func (timeoutErr) Error() string { return "i/o timeout" } +func (timeoutErr) Timeout() bool { return true } +func (timeoutErr) Temporary() bool { return true } + +func TestClassify(t *testing.T) { + t.Parallel() + wrap := func(err error) error { return &url.Error{Op: "Get", URL: "http://x/", Err: err} } + for _, tc := range []struct { + name string + err error + status int + want string + }{ + {"success", nil, 200, ""}, + {"http status", nil, 503, "HTTP 503"}, + {"deadline", wrap(context.DeadlineExceeded), 0, "timeout"}, + {"net timeout", wrap(&net.OpError{Op: "dial", Err: timeoutErr{}}), 0, "timeout"}, + {"refused", wrap(&net.OpError{Op: "dial", Err: syscall.ECONNREFUSED}), 0, "connection refused"}, + {"reset", wrap(&net.OpError{Op: "read", Err: syscall.ECONNRESET}), 0, "connection reset"}, + {"eof", wrap(io.EOF), 0, "EOF"}, + {"unexpected eof", wrap(io.ErrUnexpectedEOF), 0, "EOF"}, + {"dns not found", wrap(&net.DNSError{Err: "no such host", Name: "x", IsNotFound: true}), 0, "dns: not found"}, + {"dns other", wrap(&net.DNSError{Err: "server misbehaving", Name: "x"}), 0, "dns: error"}, + {"dns timeout", wrap(&net.DNSError{Err: "i/o timeout", Name: "x", IsTimeout: true}), 0, "dns: timeout"}, + {"long other is cut", wrap(errors.New(strings.Repeat("x", maxOtherLen+20))), 0, "other: " + strings.Repeat("x", maxOtherLen)}, + {"cut keeps whole characters", wrap(errors.New(strings.Repeat("a", maxOtherLen-1) + "é!")), 0, "other: " + strings.Repeat("a", maxOtherLen-1)}, + {"other strips url", wrap(errors.New("boom")), 0, "other: boom"}, + {"unknown authority", wrap(&tls.CertificateVerificationError{Err: x509.UnknownAuthorityError{}}), 0, "tls: unknown authority"}, + {"hostname mismatch", wrap(&tls.CertificateVerificationError{Err: x509.HostnameError{Host: "x"}}), 0, "tls: hostname mismatch"}, + {"expired", wrap(&tls.CertificateVerificationError{Err: x509.CertificateInvalidError{Reason: x509.Expired}}), 0, "tls: invalid certificate"}, + {"not tls", wrap(tls.RecordHeaderError{Msg: "first record does not look like a TLS handshake"}), 0, "tls: not a TLS server"}, + {"plain http answer", wrap(errors.New("http: server gave HTTP response to HTTPS client")), 0, "tls: not a TLS server"}, + {"peer alert", wrap(&net.OpError{Op: "remote error", Err: errors.New("tls: handshake failure")}), 0, "tls: peer alert: handshake failure"}, + {"net error without its addresses", &url.Error{Op: "Get", URL: "http://10.0.0.1:8080/", Err: &net.OpError{ + Op: "write", Net: "tcp", + Source: &net.TCPAddr{IP: net.IPv4(10, 0, 0, 2), Port: 51234}, + Addr: &net.TCPAddr{IP: net.IPv4(10, 0, 0, 1), Port: 8080}, + Err: syscall.EPIPE, + }}, 0, "net: write: broken pipe"}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + if got := classify(tc.err, tc.status); got != tc.want { + t.Errorf("classify() = %q, want %q", got, tc.want) + } + }) + } +} diff --git a/cmd/benchmarking/egress/internal/cmd/actor/endpoints.go b/cmd/benchmarking/egress/internal/cmd/actor/endpoints.go new file mode 100644 index 0000000000..9ef1278a9c --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/endpoints.go @@ -0,0 +1,49 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actor + +import "fmt" + +// TargetNamespace holds the endpoint Services the benchmark deploys. +const TargetNamespace = "egress-tests-targets" + +// MaxEndpoints is the most endpoints a run can use. +const MaxEndpoints = 1024 + +// ServiceName is the name of endpoint i's Service. +func ServiceName(i int) string { + return fmt.Sprintf("egress-target-%d", i) +} + +// EndpointDomain is the DNS domain every endpoint name sits directly under. +const EndpointDomain = TargetNamespace + ".svc.cluster.local" + +// EndpointHostPattern matches every endpoint name and nothing below them: a +// wildcard stands for exactly one leftmost label. The target's certificate is +// issued for it. +const EndpointHostPattern = "*." + EndpointDomain + +// EndpointHost is the DNS name of endpoint i. It is the one place the +// endpoint names are defined: the actor's loop calls these names and the +// driver's egress policy allows them. +func EndpointHost(i int) string { + return ServiceName(i) + "." + EndpointDomain +} + +// EndpointURL is the URL the actor's loop requests for endpoint i over scheme, +// http or https, each on its default port. +func EndpointURL(i int, scheme string) string { + return scheme + "://" + EndpointHost(i) + "/" +} diff --git a/cmd/benchmarking/egress/internal/cmd/actor/endpoints_test.go b/cmd/benchmarking/egress/internal/cmd/actor/endpoints_test.go new file mode 100644 index 0000000000..3c7dab31a7 --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/endpoints_test.go @@ -0,0 +1,36 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actor + +import "testing" + +func TestEndpointNames(t *testing.T) { + t.Parallel() + if got, want := ServiceName(7), "egress-target-7"; got != want { + t.Errorf("ServiceName(7) = %q, want %q", got, want) + } + if got, want := EndpointHost(7), "egress-target-7.egress-tests-targets.svc.cluster.local"; got != want { + t.Errorf("EndpointHost(7) = %q, want %q", got, want) + } + if got, want := EndpointURL(7, SchemeHTTP), "http://egress-target-7.egress-tests-targets.svc.cluster.local/"; got != want { + t.Errorf("EndpointURL(7, http) = %q, want %q", got, want) + } + if got, want := EndpointURL(7, SchemeHTTPS), "https://egress-target-7.egress-tests-targets.svc.cluster.local/"; got != want { + t.Errorf("EndpointURL(7, https) = %q, want %q", got, want) + } + if got, want := EndpointHostPattern, "*.egress-tests-targets.svc.cluster.local"; got != want { + t.Errorf("EndpointHostPattern = %q, want %q", got, want) + } +} diff --git a/cmd/benchmarking/egress/internal/cmd/actor/hist.go b/cmd/benchmarking/egress/internal/cmd/actor/hist.go new file mode 100644 index 0000000000..c16c563040 --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/hist.go @@ -0,0 +1,145 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actor + +import ( + "fmt" + "math" + "slices" + "strconv" +) + +const ( + bucketsPerDecade = 20 + // maxBucketSeconds stops the bounds: the first bound at or past it is the + // last. Slower samples count in the overflow bucket. + maxBucketSeconds = 60 +) + +// mantissas are 10^(j/20) for j = 0..19, to four significant digits. +var mantissas = [bucketsPerDecade]string{ + "1", "1.122", "1.259", "1.413", "1.585", "1.778", "1.995", "2.239", "2.512", "2.818", + "3.162", "3.548", "3.981", "4.467", "5.012", "5.623", "6.31", "7.079", "7.943", "8.913", +} + +// Bounds are the upper bounds, in seconds, of every distribution: 20 per +// decade from 1 µs to 63.1 s. Each is parsed from a decimal string, so every +// build on every architecture holds the same 157 values. Neighboring bounds +// differ by about 12%, the resolution of a quantile read from buckets. +var Bounds = func() []float64 { + var bounds []float64 + for k := 0; ; k++ { + b, err := strconv.ParseFloat(fmt.Sprintf("%se%d", mantissas[k%bucketsPerDecade], k/bucketsPerDecade-6), 64) + if err != nil { + panic(err) + } + bounds = append(bounds, b) + if b >= maxBucketSeconds { + return bounds + } + } +}() + +// Distribution names one of the four distributions of a Stats, the key of +// its Point in Stats.Distributions. +type Distribution string + +const ( + Opening Distribution = "opening" // end-to-end time of an endpoint's first request, when it succeeded + Latency Distribution = "latency" // end-to-end time of later successful requests + DNS Distribution = "dns" // lookup time, for requests that did a lookup + TLS Distribution = "tls" // handshake time, for handshakes that completed +) + +// DistributionNames lists the four distributions. +var DistributionNames = []Distribution{Opening, Latency, DNS, TLS} + +// Point is one distribution over one window: the fields of an OTel +// HistogramDataPoint that a driver adds across actors. The SDK's type marshals +// to JSON but does not unmarshal, so both sides use this mirror. The zero +// Point is empty. +type Point struct { + // Bounds are the upper bounds of the finite buckets; BucketCounts has one + // more entry, the overflow bucket. Counts are per bucket, not cumulative. + Bounds []float64 `json:"bounds,omitempty"` + BucketCounts []uint64 `json:"bucketCounts,omitempty"` + Count uint64 `json:"count"` + Sum float64 `json:"sum"` + Min float64 `json:"min,omitempty"` + Max float64 `json:"max,omitempty"` +} + +// Merge adds o into p. It refuses, leaving p as it was, a point whose bounds +// are not p's, or whose counts do not fit its bounds. +func (p *Point) Merge(o Point) error { + if o.Count == 0 { + return nil + } + if len(o.BucketCounts) != len(o.Bounds)+1 { + return fmt.Errorf("%d bucket counts for %d bounds", len(o.BucketCounts), len(o.Bounds)) + } + if p.Count == 0 { + *p = Point{Bounds: slices.Clone(o.Bounds), BucketCounts: slices.Clone(o.BucketCounts), Count: o.Count, Sum: o.Sum, Min: o.Min, Max: o.Max} + return nil + } + if len(p.BucketCounts) != len(p.Bounds)+1 { + return fmt.Errorf("receiver has %d bucket counts for %d bounds", len(p.BucketCounts), len(p.Bounds)) + } + if len(p.Bounds) != len(o.Bounds) { + return fmt.Errorf("%d bounds, want %d", len(o.Bounds), len(p.Bounds)) + } + for i := range p.Bounds { + if p.Bounds[i] != o.Bounds[i] { + return fmt.Errorf("bounds differ at index %d: %v vs %v", i, p.Bounds[i], o.Bounds[i]) + } + } + for i := range p.BucketCounts { + p.BucketCounts[i] += o.BucketCounts[i] + } + p.Count += o.Count + p.Sum += o.Sum + p.Min = min(p.Min, o.Min) + p.Max = max(p.Max, o.Max) + return nil +} + +// Quantile returns the upper bound of the bucket holding the q-th sample, +// capped at the largest sample seen. ok is false for an empty point. +func (p Point) Quantile(q float64) (seconds float64, ok bool) { + if p.Count == 0 { + return 0, false + } + rank := max(uint64(math.Ceil(q*float64(p.Count))), 1) + var seen uint64 + for i, c := range p.BucketCounts { + seen += c + if seen < rank { + continue + } + if i == len(p.Bounds) { + return p.Max, true // the overflow bucket: only the maximum bounds it + } + return min(p.Bounds[i], p.Max), true + } + return p.Max, true +} + +// Mean returns the average sample, or 0 for an empty point. +func (p Point) Mean() float64 { + if p.Count == 0 { + return 0 + } + return p.Sum / float64(p.Count) +} diff --git a/cmd/benchmarking/egress/internal/cmd/actor/hist_test.go b/cmd/benchmarking/egress/internal/cmd/actor/hist_test.go new file mode 100644 index 0000000000..3dc88130af --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/hist_test.go @@ -0,0 +1,201 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actor + +import ( + "encoding/json" + "reflect" + "slices" + "strconv" + "strings" + "testing" + "time" +) + +// record puts one sample into p the way the SDK buckets it, so a test can +// build a Point without an instrument. +func record(p *Point, seconds float64) { + if p.Count == 0 { + p.Bounds = slices.Clone(Bounds) + p.BucketCounts = make([]uint64, len(Bounds)+1) + p.Min, p.Max = seconds, seconds + } + i, found := slices.BinarySearch(p.Bounds, seconds) + if !found && i == len(p.Bounds) { + i = len(p.Bounds) // overflow + } + p.BucketCounts[i]++ + p.Count++ + p.Sum += seconds + p.Min, p.Max = min(p.Min, seconds), max(p.Max, seconds) +} + +// pointOf is record over every sample. +func pointWith(samples ...float64) Point { + var p Point + for _, s := range samples { + record(&p, s) + } + return p +} + +func TestBounds(t *testing.T) { + t.Parallel() + if n := len(Bounds); n != 157 { + t.Fatalf("len(Bounds) = %d, want 157", n) + } + if Bounds[0] != 1e-6 || Bounds[60] != 0.001 || Bounds[156] != 63.1 { + t.Errorf("Bounds[0,60,156] = %v %v %v, want 1e-06 0.001 63.1", Bounds[0], Bounds[60], Bounds[156]) + } + for i, b := range Bounds { + if i > 0 && b <= Bounds[i-1] { + t.Errorf("Bounds[%d] = %v does not exceed Bounds[%d] = %v", i, b, i-1, Bounds[i-1]) + } + if again, _ := strconv.ParseFloat(strconv.FormatFloat(b, 'g', 4, 64), 64); again != b { + t.Errorf("Bounds[%d] = %v has more than four significant digits", i, b) + } + } +} + +func TestPointJSONRoundTrip(t *testing.T) { + t.Parallel() + in := map[Distribution]Point{TLS: pointWith(0.006, 0.009), DNS: {}} + b, err := json.Marshal(in) + if err != nil { + t.Fatal(err) + } + var back map[Distribution]Point + if err := json.Unmarshal(b, &back); err != nil { + t.Fatal(err) + } + if !reflect.DeepEqual(back, in) { + t.Errorf("round trip = %+v, want %+v", back, in) + } +} + +func TestPointMergeAndQuantile(t *testing.T) { + t.Parallel() + var a, b Point + for i := 1; i <= 90; i++ { + record(&a, float64(i)*time.Millisecond.Seconds()) // 1..90 ms + } + for range 10 { + record(&b, 5) // ten slow ones at 5 s + } + var sum Point + for _, p := range []Point{a, b, {}} { // an empty point merges as a no-op + if err := sum.Merge(p); err != nil { + t.Fatal(err) + } + } + if sum.Count != 100 || sum.Max != 5 || sum.Min != 0.001 { + t.Errorf("merged count %d min %v max %v, want 100, 0.001, 5", sum.Count, sum.Min, sum.Max) + } + p50, _ := sum.Quantile(0.5) + p99, _ := sum.Quantile(0.99) + p999, _ := sum.Quantile(0.999) + if p50 < 0.045 || p50 > 0.0502 || p99 != 5 || p999 != 5 { + t.Errorf("p50 %v p99 %v p99.9 %v; want p50 near 45-50 ms, p99 and p99.9 = 5 (the maximum)", p50, p99, p999) + } + if mean := sum.Mean(); mean < 0.54 || mean > 0.55 { + t.Errorf("Mean() = %v, want about 0.5409", mean) + } + if (Point{}).Mean() != 0 { + t.Errorf("Mean() of an empty point = %v, want 0", (Point{}).Mean()) + } +} + +func TestPointQuantile(t *testing.T) { + t.Parallel() + edge := pointWith(0.001, 0.001, 0.002, 0.002, 0.002) // two in the 1 ms bucket, three in the 2 ms one + for _, tc := range []struct { + name string + p Point + q float64 + want float64 + wantOK bool + }{ + {"empty", Point{}, 0.5, 0, false}, + {"one sample is its own bound", pointWith(0.0015), 0.5, 0.0015, true}, + {"rank on a bucket edge", edge, 0.4, 0.001, true}, + {"rank just past a bucket edge", edge, 0.41, 0.002, true}, + {"q of 0 is the first bucket", edge, 0, 0.001, true}, + {"q of 1 is the last bucket", edge, 1, 0.002, true}, + {"overflow bucket is the maximum", pointWith(0.001, 100), 0.99, 100, true}, + {"capped at the maximum", pointWith(0.001, 0.0021), 1, 0.0021, true}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + got, ok := tc.p.Quantile(tc.q) + if got != tc.want || ok != tc.wantOK { + t.Errorf("Quantile(%v) = %v, %v; want %v, %v", tc.q, got, ok, tc.want, tc.wantOK) + } + }) + } +} + +func TestPointMerge(t *testing.T) { + t.Parallel() + // Samples are powers of two, so the sums compare exactly whatever the + // order of addition. + full := pointWith(0.25, 0.5) + other := pointWith(0.125) + other.Bounds = slices.Clone(other.Bounds) + other.Bounds[0] = 0.5e-6 + short := pointWith(0.125) + short.BucketCounts = short.BucketCounts[:len(short.BucketCounts)-1] + fewer := pointWith(0.125) + fewer.Bounds, fewer.BucketCounts = fewer.Bounds[:10], fewer.BucketCounts[:11] + for _, tc := range []struct { + name string + into Point + from Point + want Point + wantErr string // a substring of the error; "" means no error + }{ + {"empty into empty", Point{}, Point{}, Point{}, ""}, + {"empty into full", full, Point{}, full, ""}, + {"full into empty", Point{}, full, full, ""}, + {"equal bounds", full, pointWith(0.125, 2), pointWith(0.25, 0.5, 0.125, 2), ""}, + {"different bounds refused, naming the index", full, other, full, "bounds differ at index 0: 1e-06 vs 5e-07"}, + {"fewer bounds refused, naming both lengths", full, fewer, full, "10 bounds, want 157"}, + {"counts that do not fit the bounds refused", full, short, full, "157 bucket counts for 157 bounds"}, + {"receiver that does not fit its bounds refused", short, full, short, "receiver has 157 bucket counts for 157 bounds"}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + got := tc.into + got.BucketCounts = slices.Clone(got.BucketCounts) + err := got.Merge(tc.from) + if (err != nil) != (tc.wantErr != "") || (err != nil && !strings.Contains(err.Error(), tc.wantErr)) { + t.Fatalf("Merge() error = %v, want %q", err, tc.wantErr) + } + if !reflect.DeepEqual(got, tc.want) { + t.Errorf("after Merge: count %d sum %v min %v max %v, want count %d sum %v min %v max %v", + got.Count, got.Sum, got.Min, got.Max, tc.want.Count, tc.want.Sum, tc.want.Min, tc.want.Max) + } + }) + } + // A merge into an empty point copies, so the source's buckets are its own. + var sum Point + if err := sum.Merge(full); err != nil { + t.Fatal(err) + } + i, _ := slices.BinarySearch(Bounds, 0.25) + sum.BucketCounts[i]++ + if full.BucketCounts[i] != 1 { + t.Errorf("merging into an empty point shared the source's buckets") + } +} diff --git a/cmd/benchmarking/egress/internal/cmd/actor/recorder.go b/cmd/benchmarking/egress/internal/cmd/actor/recorder.go new file mode 100644 index 0000000000..7b4da76028 --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/recorder.go @@ -0,0 +1,89 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actor + +import ( + "context" + "fmt" + + "go.opentelemetry.io/otel/metric" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/metric/metricdata" + "go.opentelemetry.io/otel/sdk/resource" +) + +// recorder holds one loop's four distributions as OTel histograms on a +// manual reader in delta temporality: Collect returns what was recorded since +// the previous Collect, so each call is one window. Nothing is pushed anywhere; +// the points travel inside the loop's Stats. +type recorder struct { + reader *sdkmetric.ManualReader + provider *sdkmetric.MeterProvider + hists map[Distribution]metric.Float64Histogram +} + +// newRecorder creates the four instruments on a private MeterProvider. +func newRecorder() (*recorder, error) { + reader := sdkmetric.NewManualReader(sdkmetric.WithTemporalitySelector( + func(sdkmetric.InstrumentKind) metricdata.Temporality { return metricdata.DeltaTemporality })) + provider := sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader), sdkmetric.WithResource(resource.Empty())) + meter := provider.Meter("benchmark-egress") + r := &recorder{reader: reader, provider: provider, hists: make(map[Distribution]metric.Float64Histogram, len(DistributionNames))} + for _, name := range DistributionNames { + h, err := meter.Float64Histogram(string(name), metric.WithUnit("s"), metric.WithExplicitBucketBoundaries(Bounds...)) + if err != nil { + _ = provider.Shutdown(context.Background()) + return nil, fmt.Errorf("creating the %s histogram: %w", name, err) + } + r.hists[name] = h + } + return r, nil +} + +// record adds one sample, in seconds, to the named distribution. +func (r *recorder) record(ctx context.Context, name Distribution, seconds float64) { + r.hists[name].Record(ctx, seconds) +} + +// collect returns each distribution's samples since the previous collect, +// keyed by name. A distribution with no new sample is absent. +func (r *recorder) collect(ctx context.Context) (map[Distribution]Point, error) { + var rm metricdata.ResourceMetrics + if err := r.reader.Collect(ctx, &rm); err != nil { + return nil, err + } + out := make(map[Distribution]Point, len(DistributionNames)) + for _, sm := range rm.ScopeMetrics { + for _, m := range sm.Metrics { + h, ok := m.Data.(metricdata.Histogram[float64]) + if !ok || len(h.DataPoints) == 0 { + continue + } + out[Distribution(m.Name)] = pointOf(h.DataPoints[0]) + } + } + return out, nil +} + +// shutdown releases the provider. record is a no-op afterwards. +func (r *recorder) shutdown(ctx context.Context) error { return r.provider.Shutdown(ctx) } + +// pointOf copies the merge fields out of an SDK data point. +func pointOf(dp metricdata.HistogramDataPoint[float64]) Point { + p := Point{Bounds: dp.Bounds, BucketCounts: dp.BucketCounts, Count: dp.Count, Sum: dp.Sum} + p.Min, _ = dp.Min.Value() + p.Max, _ = dp.Max.Value() + return p +} diff --git a/cmd/benchmarking/egress/internal/cmd/actor/recorder_test.go b/cmd/benchmarking/egress/internal/cmd/actor/recorder_test.go new file mode 100644 index 0000000000..4d78344e76 --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/recorder_test.go @@ -0,0 +1,155 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actor + +import ( + "context" + "encoding/json" + "reflect" + "testing" + "time" +) + +func newTestRecorder(t *testing.T) *recorder { + t.Helper() + r, err := newRecorder() + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = r.shutdown(context.Background()) }) + return r +} + +func collectOf(t *testing.T, r *recorder) map[Distribution]Point { + t.Helper() + got, err := r.collect(context.Background()) + if err != nil { + t.Fatal(err) + } + return got +} + +func TestCollectIsOneWindow(t *testing.T) { + t.Parallel() + ctx := context.Background() + r := newTestRecorder(t) + for _, ms := range []float64{1, 2, 3} { + r.record(ctx, Latency, (time.Duration(ms) * time.Millisecond).Seconds()) + } + first := collectOf(t, r) + if p := first[Latency]; p.Count != 3 || p.Min != 0.001 || p.Max != 0.003 { + t.Errorf("first window = count %d min %v max %v, want 3, 0.001, 0.003", p.Count, p.Min, p.Max) + } + r.record(ctx, Latency, 0.5) + second := collectOf(t, r) + if p := second[Latency]; p.Count != 1 || p.Min != 0.5 || p.Max != 0.5 { + t.Errorf("second window = count %d min %v max %v, want only the 0.5 s sample", p.Count, p.Min, p.Max) + } + if third := collectOf(t, r); len(third) != 0 { + t.Errorf("third window has data with nothing recorded: %+v", third) + } +} + +func TestIdleDistributionsAreAbsent(t *testing.T) { + t.Parallel() + r := newTestRecorder(t) + if got := collectOf(t, r); len(got) != 0 { + t.Errorf("fresh recorder collected %v, want nothing", got) + } + r.record(context.Background(), DNS, 0.002) + got := collectOf(t, r) + for _, name := range DistributionNames { + if _, ok := got[name]; ok != (name == DNS) { + t.Errorf("%s present = %v after one dns sample, want %v", name, ok, name == DNS) + } + } +} + +func TestOneSampleLandsInOneBucket(t *testing.T) { + t.Parallel() + r := newTestRecorder(t) + r.record(context.Background(), Opening, time.Millisecond.Seconds()) + got := collectOf(t, r) + p := got[Opening] + // 1 ms is a bound: the sample goes in the bucket whose bound is 0.001, + // not the one above it. + if len(p.BucketCounts) != len(Bounds)+1 || p.BucketCounts[60] != 1 || p.Count != 1 || p.Sum != 0.001 { + t.Errorf("1 ms sample: %d buckets, bucket 60 = %d, count %d, sum %v; want %d, 1, 1, 0.001", len(p.BucketCounts), p.BucketCounts[60], p.Count, p.Sum, len(Bounds)+1) + } + if !reflect.DeepEqual(p.Bounds, Bounds) { + t.Errorf("point carries other bounds: %v", p.Bounds) + } + b, err := json.Marshal(got) + if err != nil { + t.Fatal(err) + } + var back map[Distribution]Point + if err := json.Unmarshal(b, &back); err != nil { + t.Fatal(err) + } + if !reflect.DeepEqual(back, got) { + t.Errorf("JSON round trip = %+v, want %+v", back, got) + } +} + +func TestRecordAfterShutdownDoesNotPanic(t *testing.T) { + t.Parallel() + ctx := context.Background() + r, err := newRecorder() + if err != nil { + t.Fatal(err) + } + r.record(ctx, TLS, 0.01) + if err := r.shutdown(ctx); err != nil { + t.Fatal(err) + } + r.record(ctx, TLS, 0.01) + if _, err := r.collect(ctx); err == nil { + t.Errorf("Collect after Shutdown returned no error") + } +} + +func BenchmarkRecord(b *testing.B) { + r, err := newRecorder() + if err != nil { + b.Fatal(err) + } + defer r.shutdown(context.Background()) + ctx := context.Background() + for i := 0; b.Loop(); i++ { + r.record(ctx, Latency, float64(i%1000)*1e-5) + } +} + +func BenchmarkCollect(b *testing.B) { + r, err := newRecorder() + if err != nil { + b.Fatal(err) + } + defer r.shutdown(context.Background()) + ctx := context.Background() + for b.Loop() { + b.StopTimer() + for _, name := range DistributionNames { + for i := range 64 { + r.record(ctx, name, float64(1+i)*1e-3) + } + } + b.StartTimer() + if _, err := r.collect(ctx); err != nil { + b.Fatal(err) + } + } +} diff --git a/cmd/benchmarking/egress/internal/cmd/actor/stats.go b/cmd/benchmarking/egress/internal/cmd/actor/stats.go new file mode 100644 index 0000000000..863eb5b83b --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/stats.go @@ -0,0 +1,73 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actor + +import "time" + +// Stats is what one actor's loop measured. The counters are cumulative since +// /start. GET /stats serves them alone; POST /collect and POST /stop add the +// distributions recorded since the previous cut. +type Stats struct { + // Scheme is the scheme the loop requested endpoints over. + Scheme string `json:"scheme,omitempty"` + Elapsed time.Duration `json:"elapsed"` + // Requests counts every request the loop finished, failed ones included. + Requests int64 `json:"requests"` + Successes int64 `json:"successes"` + // NewConns counts requests that went out on a connection opened for them + // rather than on a kept-alive one. A tunnel the gateway reset shows up + // here rather than in Errors: net/http redials, and retries a GET that + // failed before any of the response arrived. + NewConns int64 `json:"newConns"` + // Opened counts endpoints whose first request has completed, so a reader + // can tell the opening from the steady state. + Opened int64 `json:"opened"` + // OpeningDone is Elapsed at the moment the last endpoint's first request + // completed; zero until every endpoint has opened. + OpeningDone time.Duration `json:"openingDone"` + // RequestsAtOpeningDone is Requests at that same moment, so a reader can + // price this actor's opening alone; zero until every endpoint has opened. + RequestsAtOpeningDone int64 `json:"requestsAtOpeningDone"` + // Rounds is the number of requests the least-visited endpoint has + // completed: whether this actor is still making progress. + Rounds int64 `json:"rounds"` + // LongestGap is the longest time between the starts of two consecutive + // requests on one endpoint: how quiet the loop has left a tunnel. Each + // endpoint is requested once per period, C times the interval, so with + // prompt endpoints every gap is one period; a request that outlasts the + // period delays only that endpoint's next request, so a longer gap means a + // request on that tunnel took about that long. Keep the period under the + // gateway's stream idle timeout. + LongestGap time.Duration `json:"longestGap"` + // Errors counts failed requests by class, as the actor names them: one + // low-cardinality class per failure mode. + Errors map[string]int64 `json:"errors,omitempty"` + // Endpoints holds per-endpoint counts, indexed by endpoint number. It is + // served only when asked for with ?endpoints=1. + Endpoints []Endpoint `json:"endpoints,omitempty"` + // Distributions are the samples recorded since the previous cut, keyed + // by DistributionNames. A distribution with no sample in the window is + // absent. GET /stats leaves them out. + Distributions map[Distribution]Point `json:"distributions,omitempty"` +} + +// Endpoint holds the counts for one endpoint. +type Endpoint struct { + Requests int64 `json:"requests"` + Errors int64 `json:"errors"` + // LongestGap is Stats.LongestGap for this endpoint alone, so a quiet + // tunnel can be named. + LongestGap time.Duration `json:"longestGap"` +} diff --git a/cmd/benchmarking/egress/internal/cmd/actor/stats_test.go b/cmd/benchmarking/egress/internal/cmd/actor/stats_test.go new file mode 100644 index 0000000000..c83eca71e1 --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/stats_test.go @@ -0,0 +1,135 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package actor + +import ( + "encoding/json" + "flag" + "os" + "path/filepath" + "reflect" + "regexp" + "strings" + "testing" + "time" +) + +var updateGolden = flag.Bool("update", false, "rewrite testdata goldens") + +// numberList matches an indented JSON array of numbers, so a golden can keep +// each Point's bounds and counts on one line. +var numberList = regexp.MustCompile(`\[\s+(?:-?[0-9.e+-]+,?\s+)+\]`) + +// marshalGolden is MarshalIndent with the number arrays folded onto one line. +func marshalGolden(v any) ([]byte, error) { + b, err := json.MarshalIndent(v, "", " ") + if err != nil { + return nil, err + } + return numberList.ReplaceAllFunc(b, func(m []byte) []byte { + return []byte("[" + strings.Join(strings.Fields(string(m[1:len(m)-1])), " ") + "]") + }), nil +} + +// scriptedStats is a fixed run: 60 requests over 4 endpoints, each opened +// with a lookup and a handshake, a few 503s and timeouts, and two +// distributions in the window. +func scriptedStats() Stats { + const c = 4 + s := Stats{ + Scheme: SchemeHTTPS, + Elapsed: 61200 * time.Millisecond, + OpeningDone: 350 * time.Millisecond, + RequestsAtOpeningDone: c, + LongestGap: 400 * time.Millisecond, + Errors: map[string]int64{}, + Endpoints: make([]Endpoint, c), + Distributions: map[Distribution]Point{}, + } + opening, latency := Point{}, Point{} + for n := range 60 { + i := n % c + s.Requests++ + s.Endpoints[i].Requests++ + s.Endpoints[i].LongestGap = 400 * time.Millisecond + if n < c { + s.NewConns++ + s.Opened++ + } + seconds := (time.Duration(1+n) * time.Millisecond).Seconds() + switch { + case n%17 == 3: + s.Errors["HTTP 503"]++ + s.Endpoints[i].Errors++ + case n%23 == 5: + s.Errors["timeout"]++ + s.Endpoints[i].Errors++ + case n < c: + s.Successes++ + record(&opening, seconds) + default: + s.Successes++ + record(&latency, seconds) + } + } + s.Rounds = 60 / c + s.Distributions[Opening] = opening + s.Distributions[Latency] = latency + return s +} + +// TestStatsGolden pins the JSON a /collect answers with, as a driver decodes +// it: field names, duration units, and the shape of a Point. +func TestStatsGolden(t *testing.T) { + t.Parallel() + in := scriptedStats() + got, err := marshalGolden(in) + if err != nil { + t.Fatal(err) + } + path := filepath.Join("testdata", "collect.json") + if *updateGolden { + if err := os.WriteFile(path, got, 0o644); err != nil { + t.Fatal(err) + } + } + want, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if string(got) != string(want) { + t.Errorf("Stats JSON differs from %s (run with -update to accept):\n%s", path, got) + } + var back Stats + if err := json.Unmarshal(want, &back); err != nil { + t.Fatalf("decoding %s: %v", path, err) + } + if !reflect.DeepEqual(back, in) { + t.Errorf("%s decodes to\n%+v\nwant\n%+v", path, back, in) + } +} + +func TestStatsWithoutDistributionsOmitsThem(t *testing.T) { + t.Parallel() + b, err := json.Marshal(Stats{Scheme: SchemeHTTP, Requests: 3}) + if err != nil { + t.Fatal(err) + } + for _, absent := range []string{"distributions", "endpoints", "errors"} { + if strings.Contains(string(b), absent) { + t.Errorf("counters-only Stats carries %q: %s", absent, b) + } + } +} diff --git a/cmd/benchmarking/egress/internal/cmd/actor/testdata/collect.json b/cmd/benchmarking/egress/internal/cmd/actor/testdata/collect.json new file mode 100644 index 0000000000..9b176e3fa8 --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/actor/testdata/collect.json @@ -0,0 +1,56 @@ +{ + "scheme": "https", + "elapsed": 61200000000, + "requests": 60, + "successes": 53, + "newConns": 4, + "opened": 4, + "openingDone": 350000000, + "requestsAtOpeningDone": 4, + "rounds": 15, + "longestGap": 400000000, + "errors": { + "HTTP 503": 4, + "timeout": 3 + }, + "endpoints": [ + { + "requests": 15, + "errors": 2, + "longestGap": 400000000 + }, + { + "requests": 15, + "errors": 2, + "longestGap": 400000000 + }, + { + "requests": 15, + "errors": 1, + "longestGap": 400000000 + }, + { + "requests": 15, + "errors": 2, + "longestGap": 400000000 + } + ], + "distributions": { + "latency": { + "bounds": [0.000001, 0.000001122, 0.000001259, 0.000001413, 0.000001585, 0.000001778, 0.000001995, 0.000002239, 0.000002512, 0.000002818, 0.000003162, 0.000003548, 0.000003981, 0.000004467, 0.000005012, 0.000005623, 0.00000631, 0.000007079, 0.000007943, 0.000008913, 0.00001, 0.00001122, 0.00001259, 0.00001413, 0.00001585, 0.00001778, 0.00001995, 0.00002239, 0.00002512, 0.00002818, 0.00003162, 0.00003548, 0.00003981, 0.00004467, 0.00005012, 0.00005623, 0.0000631, 0.00007079, 0.00007943, 0.00008913, 0.0001, 0.0001122, 0.0001259, 0.0001413, 0.0001585, 0.0001778, 0.0001995, 0.0002239, 0.0002512, 0.0002818, 0.0003162, 0.0003548, 0.0003981, 0.0004467, 0.0005012, 0.0005623, 0.000631, 0.0007079, 0.0007943, 0.0008913, 0.001, 0.001122, 0.001259, 0.001413, 0.001585, 0.001778, 0.001995, 0.002239, 0.002512, 0.002818, 0.003162, 0.003548, 0.003981, 0.004467, 0.005012, 0.005623, 0.00631, 0.007079, 0.007943, 0.008913, 0.01, 0.01122, 0.01259, 0.01413, 0.01585, 0.01778, 0.01995, 0.02239, 0.02512, 0.02818, 0.03162, 0.03548, 0.03981, 0.04467, 0.05012, 0.05623, 0.0631, 0.07079, 0.07943, 0.08913, 0.1, 0.1122, 0.1259, 0.1413, 0.1585, 0.1778, 0.1995, 0.2239, 0.2512, 0.2818, 0.3162, 0.3548, 0.3981, 0.4467, 0.5012, 0.5623, 0.631, 0.7079, 0.7943, 0.8913, 1, 1.122, 1.259, 1.413, 1.585, 1.778, 1.995, 2.239, 2.512, 2.818, 3.162, 3.548, 3.981, 4.467, 5.012, 5.623, 6.31, 7.079, 7.943, 8.913, 10, 11.22, 12.59, 14.13, 15.85, 17.78, 19.95, 22.39, 25.12, 28.18, 31.62, 35.48, 39.81, 44.67, 50.12, 56.23, 63.1], + "bucketCounts": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 2, 1, 1, 2, 1, 2, 2, 2, 3, 3, 2, 4, 3, 5, 6, 4, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], + "count": 50, + "sum": 1.6190000000000004, + "min": 0.005, + "max": 0.06 + }, + "opening": { + "bounds": [0.000001, 0.000001122, 0.000001259, 0.000001413, 0.000001585, 0.000001778, 0.000001995, 0.000002239, 0.000002512, 0.000002818, 0.000003162, 0.000003548, 0.000003981, 0.000004467, 0.000005012, 0.000005623, 0.00000631, 0.000007079, 0.000007943, 0.000008913, 0.00001, 0.00001122, 0.00001259, 0.00001413, 0.00001585, 0.00001778, 0.00001995, 0.00002239, 0.00002512, 0.00002818, 0.00003162, 0.00003548, 0.00003981, 0.00004467, 0.00005012, 0.00005623, 0.0000631, 0.00007079, 0.00007943, 0.00008913, 0.0001, 0.0001122, 0.0001259, 0.0001413, 0.0001585, 0.0001778, 0.0001995, 0.0002239, 0.0002512, 0.0002818, 0.0003162, 0.0003548, 0.0003981, 0.0004467, 0.0005012, 0.0005623, 0.000631, 0.0007079, 0.0007943, 0.0008913, 0.001, 0.001122, 0.001259, 0.001413, 0.001585, 0.001778, 0.001995, 0.002239, 0.002512, 0.002818, 0.003162, 0.003548, 0.003981, 0.004467, 0.005012, 0.005623, 0.00631, 0.007079, 0.007943, 0.008913, 0.01, 0.01122, 0.01259, 0.01413, 0.01585, 0.01778, 0.01995, 0.02239, 0.02512, 0.02818, 0.03162, 0.03548, 0.03981, 0.04467, 0.05012, 0.05623, 0.0631, 0.07079, 0.07943, 0.08913, 0.1, 0.1122, 0.1259, 0.1413, 0.1585, 0.1778, 0.1995, 0.2239, 0.2512, 0.2818, 0.3162, 0.3548, 0.3981, 0.4467, 0.5012, 0.5623, 0.631, 0.7079, 0.7943, 0.8913, 1, 1.122, 1.259, 1.413, 1.585, 1.778, 1.995, 2.239, 2.512, 2.818, 3.162, 3.548, 3.981, 4.467, 5.012, 5.623, 6.31, 7.079, 7.943, 8.913, 10, 11.22, 12.59, 14.13, 15.85, 17.78, 19.95, 22.39, 25.12, 28.18, 31.62, 35.48, 39.81, 44.67, 50.12, 56.23, 63.1], + "bucketCounts": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], + "count": 3, + "sum": 0.006, + "min": 0.001, + "max": 0.003 + } + } +} \ No newline at end of file diff --git a/cmd/benchmarking/egress/internal/cmd/target/target.go b/cmd/benchmarking/egress/internal/cmd/target/target.go new file mode 100644 index 0000000000..1932184105 --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/target/target.go @@ -0,0 +1,208 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package target is the endpoint every tunnel of the egress scale test leads +// to: the subcommand that answers each request with a small fixed body. +package target + +import ( + "bytes" + "context" + "crypto/tls" + "errors" + "fmt" + "log/slog" + "net" + "net/http" + "strconv" + "strings" + "sync" + "time" + + "github.com/spf13/cobra" +) + +// targetOptions are the flags of the target subcommand. +type targetOptions struct { + listen, listenPorts string + tlsListen, tlsListenPorts string + tlsCertFile, tlsKeyFile string + responseBytes int +} + +// Command is the endpoint the benchmark-egress actors call: every request +// gets a 200 with a small fixed body, over plain HTTP and, when given a +// certificate, HTTPS. One Deployment of it sits behind all of the test's +// Services. --listen-ports and --tls-listen-ports give each Service its own +// backend port: Services on distinct ClusterIPs that share one backend port +// can hand the target two connections with the same 4-tuple. +func Command() *cobra.Command { + var o targetOptions + cmd := &cobra.Command{ + Use: "target", + Short: "Answer every request the actors' tunnels carry with a small fixed body", + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, _ []string) error { + return runTarget(cmd.Context(), o) + }, + } + cmd.Flags().StringVar(&o.listen, "listen", ":8080", "Address to serve plain HTTP on.") + cmd.Flags().StringVar(&o.listenPorts, "listen-ports", "", "Also serve plain HTTP on each port of this range, A-B.") + cmd.Flags().StringVar(&o.tlsListen, "tls-listen", "", "Address to serve HTTPS on; empty serves no HTTPS.") + cmd.Flags().StringVar(&o.tlsListenPorts, "tls-listen-ports", "", "Also serve HTTPS on each port of this range, A-B.") + cmd.Flags().StringVar(&o.tlsCertFile, "tls-cert-file", "", "PEM certificate to serve HTTPS with.") + cmd.Flags().StringVar(&o.tlsKeyFile, "tls-key-file", "", "PEM private key of --tls-cert-file.") + cmd.Flags().IntVar(&o.responseBytes, "response-bytes", 16, "Size of the body every request gets back.") + return cmd +} + +func newHandler(responseBytes int) http.Handler { + body := bytes.Repeat([]byte("x"), responseBytes) + mux := http.NewServeMux() + mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, _ *http.Request) { + w.Write([]byte("ok\n")) + }) + mux.HandleFunc("GET /", func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "text/plain") + w.Write(body) + }) + return mux +} + +// newTLSConfig serves the certificate in certFile with the key in keyFile. +func newTLSConfig(certFile, keyFile string) (*tls.Config, error) { + cert, err := tls.LoadX509KeyPair(certFile, keyFile) + if err != nil { + return nil, fmt.Errorf("loading TLS certificate: %w", err) + } + return &tls.Config{Certificates: []tls.Certificate{cert}, MinVersion: tls.VersionTLS12}, nil +} + +// portRange parses "A-B" or "A" into the ports A through B; "" is none. +func portRange(s string) ([]int, error) { + if s == "" { + return nil, nil + } + lo, hi, found := strings.Cut(s, "-") + if !found { + hi = lo + } + a, errA := strconv.Atoi(lo) + b, errB := strconv.Atoi(hi) + if errA != nil || errB != nil || a < 1 || b > 65535 || a > b { + return nil, fmt.Errorf("%q is not a port or a port range A-B", s) + } + ports := make([]int, 0, b-a+1) + for p := a; p <= b; p++ { + ports = append(ports, p) + } + return ports, nil +} + +// openListeners opens addr, when set, and every port in ports on all interfaces. +// On an error it closes what it opened. +func openListeners(addr, ports string) ([]net.Listener, error) { + addrs, err := listenAddrs(addr, ports) + if err != nil { + return nil, err + } + return listenAll(addrs) +} + +// listenAddrs is addr, when set, followed by every port of ports on the +// wildcard address. +func listenAddrs(addr, ports string) ([]string, error) { + ps, err := portRange(ports) + if err != nil { + return nil, err + } + var addrs []string + if addr != "" { + addrs = append(addrs, addr) + } + for _, p := range ps { + addrs = append(addrs, ":"+strconv.Itoa(p)) + } + return addrs, nil +} + +// listenAll listens on every address, or on none. +func listenAll(addrs []string) ([]net.Listener, error) { + lns := make([]net.Listener, 0, len(addrs)) + for _, a := range addrs { + ln, err := net.Listen("tcp", a) + if err != nil { + for _, l := range lns { + l.Close() + } + return nil, fmt.Errorf("listening on %s: %w", a, err) + } + lns = append(lns, ln) + } + return lns, nil +} + +// runTarget serves until ctx ends or a listener fails. +func runTarget(ctx context.Context, o targetOptions) error { + handler := newHandler(o.responseBytes) + plain := &http.Server{Handler: handler, ReadHeaderTimeout: 10 * time.Second} + plainLns, err := openListeners(o.listen, o.listenPorts) + if err != nil { + return fmt.Errorf("opening HTTP listeners: %w", err) + } + servers := []*http.Server{plain} + var tlsSrv *http.Server + var tlsLns []net.Listener + if o.tlsListen != "" || o.tlsListenPorts != "" { + cfg, err := newTLSConfig(o.tlsCertFile, o.tlsKeyFile) + if err != nil { + return err + } + tlsSrv = &http.Server{Handler: handler, TLSConfig: cfg, ReadHeaderTimeout: 10 * time.Second} + if tlsLns, err = openListeners(o.tlsListen, o.tlsListenPorts); err != nil { + return fmt.Errorf("opening HTTPS listeners: %w", err) + } + servers = append(servers, tlsSrv) + } + slog.Info("benchmark-egress target serving", "http_ports", len(plainLns), "https_ports", len(tlsLns)) + + // The first listener to fail stops them all, as the signal does. + failed := make(chan error, len(plainLns)+len(tlsLns)) + var wg sync.WaitGroup + serve := func(ln net.Listener, run func(net.Listener) error) { + wg.Go(func() { + if err := run(ln); !errors.Is(err, http.ErrServerClosed) { + failed <- fmt.Errorf("serving on %s: %w", ln.Addr(), err) + } + }) + } + for _, ln := range plainLns { + serve(ln, plain.Serve) + } + for _, ln := range tlsLns { + serve(ln, func(ln net.Listener) error { return tlsSrv.ServeTLS(ln, "", "") }) + } + + select { + case <-ctx.Done(): + case err = <-failed: + } + shutdownCtx, cancelShutdown := context.WithTimeout(context.Background(), 5*time.Second) + defer cancelShutdown() + for _, srv := range servers { + _ = srv.Shutdown(shutdownCtx) + } + wg.Wait() + return err +} diff --git a/cmd/benchmarking/egress/internal/cmd/target/target_test.go b/cmd/benchmarking/egress/internal/cmd/target/target_test.go new file mode 100644 index 0000000000..4cd01fdef1 --- /dev/null +++ b/cmd/benchmarking/egress/internal/cmd/target/target_test.go @@ -0,0 +1,159 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package target + +import ( + "context" + "crypto/tls" + "crypto/x509" + "errors" + "io" + "net" + "net/http" + "net/http/httptest" + "path/filepath" + "slices" + "testing" + "time" + + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/cmd/actor" + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/targetcert" +) + +func TestHandler(t *testing.T) { + t.Parallel() + h := newHandler(32) + for _, tc := range []struct { + path string + wantBody int + }{ + {"/", 32}, + {"/anything", 32}, + {"/healthz", len("ok\n")}, + } { + t.Run(tc.path, func(t *testing.T) { + t.Parallel() + rec := httptest.NewRecorder() + h.ServeHTTP(rec, httptest.NewRequest(http.MethodGet, tc.path, nil)) + if rec.Code != http.StatusOK || rec.Body.Len() != tc.wantBody { + t.Errorf("GET %s = %d with %d bytes, want 200 with %d", tc.path, rec.Code, rec.Body.Len(), tc.wantBody) + } + }) + } +} + +func TestHandlerOverTLS(t *testing.T) { + t.Parallel() + bundle, err := targetcert.Generate(time.Now(), actor.EndpointDomain) + if err != nil { + t.Fatal(err) + } + dir := t.TempDir() + if err := bundle.Write(dir); err != nil { + t.Fatal(err) + } + cfg, err := newTLSConfig(filepath.Join(dir, "tls.crt"), filepath.Join(dir, "tls.key")) + if err != nil { + t.Fatal(err) + } + srv := httptest.NewUnstartedServer(newHandler(32)) + srv.TLS = cfg + srv.StartTLS() + t.Cleanup(srv.Close) + + roots := x509.NewCertPool() + if !roots.AppendCertsFromPEM(bundle.CA) { + t.Fatal("bundle CA does not parse") + } + // Dial the test server whatever the URL names, so the name can be an + // endpoint's while the TLS check runs for real. + client := &http.Client{Transport: &http.Transport{ + TLSClientConfig: &tls.Config{RootCAs: roots}, + DialContext: func(ctx context.Context, network, _ string) (net.Conn, error) { + return (&net.Dialer{}).DialContext(ctx, network, srv.Listener.Addr().String()) + }, + }} + + resp, err := client.Get(actor.EndpointURL(42, actor.SchemeHTTPS)) + if err != nil { + t.Fatalf("GET over TLS as an endpoint name: %v", err) + } + body, _ := io.ReadAll(resp.Body) + resp.Body.Close() + if resp.StatusCode != http.StatusOK || len(body) != 32 { + t.Errorf("GET over TLS = %d with %d bytes, want 200 with 32", resp.StatusCode, len(body)) + } + + _, err = client.Get("https://example.com/") + var hostErr x509.HostnameError + if !errors.As(err, &hostErr) { + t.Errorf("GET over TLS as another name = %v, want a hostname mismatch", err) + } +} + +func TestPortRange(t *testing.T) { + t.Parallel() + for _, tc := range []struct { + in string + want []int + wantErr bool + }{ + {"", nil, false}, + {"8080", []int{8080}, false}, + {"10000-10002", []int{10000, 10001, 10002}, false}, + {"10002-10000", nil, true}, + {"0-3", nil, true}, + {"65535-65536", nil, true}, + {"a-b", nil, true}, + {"10000-", nil, true}, + } { + t.Run(tc.in, func(t *testing.T) { + t.Parallel() + got, err := portRange(tc.in) + if (err != nil) != tc.wantErr || !slices.Equal(got, tc.want) { + t.Errorf("portRange(%q) = %v, %v; want %v, error %v", tc.in, got, err, tc.want, tc.wantErr) + } + }) + } +} + +func TestListenAddrs(t *testing.T) { + t.Parallel() + got, err := listenAddrs("127.0.0.1:0", "10000-10002") + if err != nil { + t.Fatal(err) + } + if want := []string{"127.0.0.1:0", ":10000", ":10001", ":10002"}; !slices.Equal(got, want) { + t.Errorf("listenAddrs() = %v, want %v", got, want) + } + if _, err := listenAddrs("", "a-b"); err == nil { + t.Error("listenAddrs accepted a bad port range") + } +} + +func TestListenAllClosesWhatItOpenedOnFailure(t *testing.T) { + t.Parallel() + held, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { held.Close() }) + if lns, err := listenAll([]string{"127.0.0.1:0", held.Addr().String()}); err == nil { + for _, ln := range lns { + ln.Close() + } + t.Fatal("listenAll bound a port that was taken") + } +} diff --git a/cmd/benchmarking/egress/internal/targetcert/targetcert.go b/cmd/benchmarking/egress/internal/targetcert/targetcert.go new file mode 100644 index 0000000000..2c0052b1f0 --- /dev/null +++ b/cmd/benchmarking/egress/internal/targetcert/targetcert.go @@ -0,0 +1,158 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package targetcert issues the name-constrained CA and the certificate the +// target serves over HTTPS, for the driver to deploy and the actor to verify +// against. +package targetcert + +import ( + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/x509" + "crypto/x509/pkix" + "encoding/pem" + "fmt" + "math/big" + "os" + "path/filepath" + "time" +) + +// validity is long enough that a test cluster never sees the certificate +// expire; there is no rotation, since nothing can sign a replacement. +const validity = 5 * 365 * 24 * time.Hour + +// clockSkew backdates NotBefore so a node whose clock runs slightly behind +// still accepts a certificate issued moments ago. +const clockSkew = time.Hour + +// Bundle is a CA certificate and the target certificate and key it signed, +// all PEM-encoded. +type Bundle struct { + CA, Cert, Key []byte +} + +// Generate issues a new CA and the certificate the target serves over HTTPS, +// valid from now, for every name directly under domain, the DNS domain the +// endpoint names sit in. The egress gateway re-originates TLS to the target +// and has to trust the CA, and the gateway is shared, so the CA vouches for +// nothing but the endpoint names: it is name-constrained to domain, and its +// private key is thrown away once the target's certificate is signed. +func Generate(now time.Time, domain string) (*Bundle, error) { + b, _, _, err := generateWithCA(now, domain) + return b, err +} + +// generateWithCA is Generate, also returning the CA certificate and key so tests +// can show what the CA refuses to vouch for. +func generateWithCA(now time.Time, domain string) (*Bundle, *x509.Certificate, *ecdsa.PrivateKey, error) { + caKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + return nil, nil, nil, fmt.Errorf("generating CA key: %w", err) + } + caSerial, err := serialNumber() + if err != nil { + return nil, nil, nil, err + } + caTemplate := &x509.Certificate{ + SerialNumber: caSerial, + Subject: pkix.Name{CommonName: "egress-tests target CA"}, + NotBefore: now.Add(-clockSkew), + NotAfter: now.Add(validity), + KeyUsage: x509.KeyUsageCertSign | x509.KeyUsageCRLSign, + BasicConstraintsValid: true, + IsCA: true, + MaxPathLenZero: true, + // Critical, so a verifier that cannot enforce the constraint rejects + // the CA instead of trusting it for every name. + PermittedDNSDomainsCritical: true, + PermittedDNSDomains: []string{domain}, + } + caDER, err := x509.CreateCertificate(rand.Reader, caTemplate, caTemplate, &caKey.PublicKey, caKey) + if err != nil { + return nil, nil, nil, fmt.Errorf("creating CA certificate: %w", err) + } + ca, err := x509.ParseCertificate(caDER) + if err != nil { + return nil, nil, nil, fmt.Errorf("parsing CA certificate: %w", err) + } + + leafKey, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + return nil, nil, nil, fmt.Errorf("generating target key: %w", err) + } + leafSerial, err := serialNumber() + if err != nil { + return nil, nil, nil, err + } + leafTemplate := &x509.Certificate{ + SerialNumber: leafSerial, + Subject: pkix.Name{CommonName: "egress-tests target"}, + // One wildcard covers every endpoint name, which differ only in the + // leftmost label. + DNSNames: []string{"*." + domain}, + NotBefore: now.Add(-clockSkew), + NotAfter: now.Add(validity), + KeyUsage: x509.KeyUsageDigitalSignature, + ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}, + } + leafDER, err := x509.CreateCertificate(rand.Reader, leafTemplate, ca, &leafKey.PublicKey, caKey) + if err != nil { + return nil, nil, nil, fmt.Errorf("creating target certificate: %w", err) + } + keyDER, err := x509.MarshalPKCS8PrivateKey(leafKey) + if err != nil { + return nil, nil, nil, fmt.Errorf("encoding target key: %w", err) + } + + b := &Bundle{ + CA: pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: caDER}), + Cert: pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: leafDER}), + Key: pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: keyDER}), + } + return b, ca, caKey, nil +} + +// Write stores the bundle in dir as ca.crt, tls.crt and tls.key; only the key +// is private. The files are what the deploy script turns into the target's +// TLS Secret, named as its keys. +func (b *Bundle) Write(dir string) error { + if err := os.MkdirAll(dir, 0o700); err != nil { + return err + } + for _, f := range []struct { + name string + data []byte + mode os.FileMode + }{ + {"ca.crt", b.CA, 0o644}, + {"tls.crt", b.Cert, 0o644}, + {"tls.key", b.Key, 0o600}, + } { + if err := os.WriteFile(filepath.Join(dir, f.name), f.data, f.mode); err != nil { + return err + } + } + return nil +} + +func serialNumber() (*big.Int, error) { + n, err := rand.Int(rand.Reader, new(big.Int).Lsh(big.NewInt(1), 128)) + if err != nil { + return nil, fmt.Errorf("generating serial number: %w", err) + } + return n, nil +} diff --git a/cmd/benchmarking/egress/internal/targetcert/targetcert_test.go b/cmd/benchmarking/egress/internal/targetcert/targetcert_test.go new file mode 100644 index 0000000000..0cf7db3a15 --- /dev/null +++ b/cmd/benchmarking/egress/internal/targetcert/targetcert_test.go @@ -0,0 +1,187 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package targetcert + +import ( + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/tls" + "crypto/x509" + "crypto/x509/pkix" + "encoding/pem" + "errors" + "math/big" + "os" + "path/filepath" + "testing" + "time" +) + +// domain is what the tests issue for; the actor passes the endpoint domain. +const domain = "egress-tests-targets.svc.cluster.local" + +func parseCert(t *testing.T, data []byte) *x509.Certificate { + t.Helper() + block, _ := pem.Decode(data) + if block == nil || block.Type != "CERTIFICATE" { + t.Fatalf("not a PEM certificate: %q", data) + } + cert, err := x509.ParseCertificate(block.Bytes) + if err != nil { + t.Fatal(err) + } + return cert +} + +func verify(t *testing.T, b *Bundle, name string, at time.Time) error { + t.Helper() + roots := x509.NewCertPool() + roots.AddCert(parseCert(t, b.CA)) + _, err := parseCert(t, b.Cert).Verify(x509.VerifyOptions{ + DNSName: name, + Roots: roots, + CurrentTime: at, + KeyUsages: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}, + }) + return err +} + +func TestCertificateCoversEveryNameUnderTheDomain(t *testing.T) { + t.Parallel() + now := time.Now() + b, err := Generate(now, domain) + if err != nil { + t.Fatal(err) + } + for _, name := range []string{"egress-target-0." + domain, "egress-target-1023." + domain, "x." + domain} { + if err := verify(t, b, name, now); err != nil { + t.Fatalf("certificate does not verify for %s: %v", name, err) + } + } +} + +func TestCertificateRejectsOtherNames(t *testing.T) { + t.Parallel() + now := time.Now() + b, err := Generate(now, domain) + if err != nil { + t.Fatal(err) + } + for _, name := range []string{ + "a.egress-target-0." + domain, // a wildcard covers one label only + "egress-target-0.other-namespace.svc.cluster.local", + "example.com", + } { + t.Run(name, func(t *testing.T) { + t.Parallel() + var hostErr x509.HostnameError + if err := verify(t, b, name, now); !errors.As(err, &hostErr) { + t.Errorf("verifying for %s = %v, want a HostnameError", name, err) + } + }) + } +} + +func TestCertificateValidityWindow(t *testing.T) { + t.Parallel() + now := time.Now() + b, err := Generate(now, domain) + if err != nil { + t.Fatal(err) + } + name := "egress-target-0." + domain + if err := verify(t, b, name, now.Add(-30*time.Minute)); err != nil { + t.Errorf("certificate rejected half an hour before issue, inside the clock-skew allowance: %v", err) + } + for _, tc := range []struct { + name string + at time.Time + }{ + {"before the skew allowance", now.Add(-2 * time.Hour)}, + {"after expiry", now.Add(validity + time.Hour)}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + var invalid x509.CertificateInvalidError + if err := verify(t, b, name, tc.at); !errors.As(err, &invalid) || invalid.Reason != x509.Expired { + t.Errorf("verifying at %v = %v, want it outside the validity window", tc.at, err) + } + }) + } +} + +// The gateway trusts this CA for every upstream it dials, so the CA must be +// unable to vouch for any name outside the endpoints' domain, even in a +// certificate it signed itself. +func TestCARefusesNamesOutsideTheEndpointDomain(t *testing.T) { + t.Parallel() + now := time.Now() + _, ca, caKey, err := generateWithCA(now, domain) + if err != nil { + t.Fatal(err) + } + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + t.Fatal(err) + } + der, err := x509.CreateCertificate(rand.Reader, &x509.Certificate{ + SerialNumber: big.NewInt(1), + Subject: pkix.Name{CommonName: "evil"}, + DNSNames: []string{"evil.example.com"}, + NotBefore: now.Add(-time.Hour), + NotAfter: now.Add(time.Hour), + ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}, + }, ca, &key.PublicKey, caKey) + if err != nil { + t.Fatal(err) + } + evil, err := x509.ParseCertificate(der) + if err != nil { + t.Fatal(err) + } + roots := x509.NewCertPool() + roots.AddCert(ca) + _, err = evil.Verify(x509.VerifyOptions{DNSName: "evil.example.com", Roots: roots, CurrentTime: now}) + var invalid x509.CertificateInvalidError + if !errors.As(err, &invalid) || invalid.Reason != x509.CANotAuthorizedForThisName { + t.Errorf("a certificate for evil.example.com signed by the CA verified with %v, want CANotAuthorizedForThisName", err) + } +} + +func TestWrite(t *testing.T) { + t.Parallel() + b, err := Generate(time.Now(), domain) + if err != nil { + t.Fatal(err) + } + dir := filepath.Join(t.TempDir(), "certs") + if err := b.Write(dir); err != nil { + t.Fatal(err) + } + if _, err := tls.LoadX509KeyPair(filepath.Join(dir, "tls.crt"), filepath.Join(dir, "tls.key")); err != nil { + t.Errorf("written certificate and key do not load as a pair: %v", err) + } + info, err := os.Stat(filepath.Join(dir, "tls.key")) + if err != nil { + t.Fatal(err) + } + if mode := info.Mode().Perm(); mode != 0o600 { + t.Errorf("key file mode = %o, want 600", mode) + } + if ca, err := os.ReadFile(filepath.Join(dir, "ca.crt")); err != nil || string(ca) != string(b.CA) { + t.Errorf("CA file = %q, %v; want the bundle's CA", ca, err) + } +} diff --git a/cmd/benchmarking/egress/main.go b/cmd/benchmarking/egress/main.go new file mode 100644 index 0000000000..fa3f068bb6 --- /dev/null +++ b/cmd/benchmarking/egress/main.go @@ -0,0 +1,56 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// benchmark-egress is the workload of the egress scale test, which holds many +// CONNECT tunnels open through the egress gateway at a fixed pace and measures +// what they cost. Each of its programs is a subcommand: the actor that opens +// the tunnels and counts what happened, and the target they call. +package main + +import ( + "context" + "fmt" + "os" + "os/signal" + "syscall" + + "github.com/spf13/cobra" + + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/cmd/actor" + "go.ate.dev/substrate/cmd/benchmarking/egress/internal/cmd/target" +) + +func main() { + root := &cobra.Command{ + Use: "benchmark-egress", + Short: "The actor and target of the egress scale test", + Long: `benchmark-egress is the workload of the egress scale test. The actor holds +one tunnel per endpoint open through the egress gateway and counts what +happened; the target is the endpoint every tunnel leads to.`, + // No usage text on an error; the error alone names what went wrong. + SilenceUsage: true, + // main prints the error itself, once. + SilenceErrors: true, + // Nobody types these commands often enough to want shell completion. + CompletionOptions: cobra.CompletionOptions{DisableDefaultCmd: true}, + } + root.AddCommand(actor.Command(), target.Command()) + + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) + defer stop() + if err := root.ExecuteContext(ctx); err != nil { + fmt.Fprintln(os.Stderr, "benchmark-egress:", err) + os.Exit(1) + } +} diff --git a/cmd/dataplane/envoy/Dockerfile b/cmd/dataplane/envoy/Dockerfile index b179a711d4..b464a15316 100644 --- a/cmd/dataplane/envoy/Dockerfile +++ b/cmd/dataplane/envoy/Dockerfile @@ -13,28 +13,48 @@ # limitations under the License. # Stage 1: Build the release version of the Rust dynamic modules -FROM rust:bookworm@sha256:93ce27a88655056a51dbdd8f5f2d7ddc071c7b0070fb288a37b5a285fc83971e AS builder +# This stage runs on the build host and cross-compiles for the target platform, +# so building for another architecture avoids slow emulation. +FROM --platform=$BUILDPLATFORM rust:bookworm@sha256:93ce27a88655056a51dbdd8f5f2d7ddc071c7b0070fb288a37b5a285fc83971e AS builder +ARG BUILDARCH +ARG TARGETARCH WORKDIR /usr/src/dynamic-modules -# Install build dependencies required by bindgen for generating Envoy ABI bindings -RUN apt-get update && apt-get install -y --no-install-recommends \ +# Install build dependencies required by bindgen for generating Envoy ABI bindings, +# and when cross-compiling, the target's linker and libc +RUN case "$TARGETARCH" in \ + amd64) echo x86_64 > /target-arch ;; \ + arm64) echo aarch64 > /target-arch ;; \ + *) echo "unsupported TARGETARCH: $TARGETARCH" >&2; exit 1 ;; \ + esac \ + && arch="$(cat /target-arch)" \ + && if [ "$BUILDARCH" != "$TARGETARCH" ]; then cross="gcc-$(echo "$arch" | tr _ -)-linux-gnu libc6-dev-${TARGETARCH}-cross"; fi \ + && apt-get update && apt-get install -y --no-install-recommends \ clang \ libclang-dev \ - && rm -rf /var/lib/apt/lists/* + $cross \ + && rm -rf /var/lib/apt/lists/* \ + && rustup target add "${arch}-unknown-linux-gnu" # Copy rust workspace source code COPY dynamic-modules/ . # Build release version of the dynamic module shared libraries -RUN cargo build --release --locked +RUN arch="$(cat /target-arch)" \ + && if [ "$BUILDARCH" != "$TARGETARCH" ]; then \ + export "CARGO_TARGET_$(echo "$arch" | tr a-z A-Z)_UNKNOWN_LINUX_GNU_LINKER=${arch}-linux-gnu-gcc"; \ + fi \ + && cargo build --release --locked --target "${arch}-unknown-linux-gnu" \ + && mkdir -p target/release \ + && cp "target/${arch}-unknown-linux-gnu/release/libenvoy_substrate_egress_policy.so" target/release/ # Stage 2: Create the runtime image containing Envoy and the built dynamic module shared libraries # When changing Envoy version, the version of SDK in the Cargo.toml files SHOULD match. # Pin the multi-platform image index, not one platform's manifest, so that the # --platform the installer passes (the host architecture on kind, linux/amd64 # otherwise) selects a matching Envoy binary. -FROM envoyproxy/envoy:v1.39.4@sha256:74bd4eb5629708d3fadca1a5e6b85d208f338d7bb54e06b69565905c5294cc01 +FROM envoyproxy/envoy:v1.39.5@sha256:8fc594a5d85eb76dca24155cfb9208882406873932b20cf33649a0a611194c43 # Copy the built dynamic module shared library from the builder stage COPY --from=builder /usr/src/dynamic-modules/target/release/libenvoy_substrate_egress_policy.so /usr/local/lib/libenvoy_substrate_egress_policy.so diff --git a/cmd/dataplane/envoy/README.md b/cmd/dataplane/envoy/README.md index b543cce1d9..7399619fae 100644 --- a/cmd/dataplane/envoy/README.md +++ b/cmd/dataplane/envoy/README.md @@ -14,18 +14,14 @@ cmd/dataplane/envoy/ - **`Dockerfile`**: Multi-stage container build that: 1. Compiles the Rust dynamic module (`envoy-substrate-egress-policy`) into a shared library (`libenvoy_substrate_egress_policy.so`) in a `rust:bookworm` builder stage. - 2. Packages the compiled `.so` into the `envoyproxy/envoy:v1.39.4` runtime image under `/usr/local/lib/libenvoy_substrate_egress_policy.so` and sets `ENVOY_DYNAMIC_MODULES_SEARCH_PATH=/usr/local/lib`. + 2. Packages the compiled `.so` into the `envoyproxy/envoy:v1.39.5` runtime image under `/usr/local/lib/libenvoy_substrate_egress_policy.so` and sets `ENVOY_DYNAMIC_MODULES_SEARCH_PATH=/usr/local/lib`. - **`dynamic-modules/egress-policy/`**: A Rust crate using the Envoy Dynamic Modules SDK (`envoy-proxy-dynamic-modules-rust-sdk`) that implements a custom Envoy listener filter for Substrate egress policy evaluation. See [`dynamic-modules/egress-policy/README.md`](dynamic-modules/egress-policy/README.md) for module-specific build, test, and Envoy configuration details. ## Building and Deployment During a build from source, `ate-setup` builds the image from this directory via `docker buildx`, pushes it to `$KO_DOCKER_REPO/envoy-dataplane`, and its resolved digest replaces the `${ENVOY_DATAPLANE_IMAGE}` placeholder in the router and egress manifests (`manifests/ate-install/atenet-router.yaml`, `manifests/ate-install/atenet-egress.yaml`). -A pre-built install (`ate-setup deploy --image-repo REPO --image-tag TAG`) builds nothing and pins `REPO/envoy-dataplane:TAG` instead, so a release publishes it with the other images: - -```bash -VERSION=TAG go run ./cmd/ate-setup publish release-images --ko-docker-repo REPO -``` +A pre-built install (`ate-setup deploy --image-repo REPO --image-tag TAG`) builds nothing and pins `REPO/envoy-dataplane:TAG` instead, so `REPO` has to hold it. Releases do not publish it; see [`docs/dev/releasing.md`](../../../docs/dev/releasing.md#release-images). The image targets the same platforms as the ko images: `KO_DEFAULTPLATFORMS`, else `defaultPlatforms` in `.ko.yaml` (linux/amd64 and linux/arm64), else `linux/amd64`. Building another architecture compiles Rust under QEMU, which must be registered with binfmt (e.g. `docker run --privileged --rm tonistiigi/binfmt --install arm64`). diff --git a/cmd/kubectl-ate/README.md b/cmd/kubectl-ate/README.md index 21c27a77e4..d0ac8f2528 100644 --- a/cmd/kubectl-ate/README.md +++ b/cmd/kubectl-ate/README.md @@ -49,7 +49,7 @@ gcloud beta container clusters update CLUSTER_NAME \ **Local (kind):** -The kind overlay installed by `hack/install-ate-kind.sh --deploy-ate-system` already provisions an in-cluster OpenTelemetry Collector and a Jaeger all-in-one in the `otel-system` namespace. No additional setup is required. +The kind overlay installed by `hack/install-ate-kind.sh --deploy-ate-system` already provisions an in-cluster OpenTelemetry Collector and Jaeger in the `otel-system` namespace. No additional setup is required. Port-forward the Jaeger UI and invoke any command with `--trace`: ```bash diff --git a/demos/counter/README.md b/demos/counter/README.md index 161997d7df..98bb4b5329 100644 --- a/demos/counter/README.md +++ b/demos/counter/README.md @@ -25,16 +25,16 @@ The demo uses two kinds of resources: the `WorkerPool` is a Kubernetes CRD, whil
Installation on MicroVM -The in-RAM-counter suspend/resume-continuity demo be run on the micro-VM +The in-RAM-counter suspend/resume-continuity demo be run on the microVM sandbox class (`ateworker-microvm`: a Kata guest on Cloud Hypervisor). - [`demos/counter/counter-microvm.yaml.tmpl`](counter-microvm.yaml.tmpl) — - the `WorkerPool` for the micro-VM sandbox class. + the `WorkerPool` for the microVM sandbox class. - [`demos/counter/counter-microvm-template.yaml.tmpl`](counter-microvm-template.yaml.tmpl) — the `counter-microvm` actor template, created through the ate API in the `ate-demo-counter-microvm` atespace. - [`hack/run-microvm-demo.sh`](../../hack/run-microvm-demo.sh) — one-shot bring-up - that builds the micro-VM worker image, stages the guest assets, deploys the + that builds the microVM worker image, stages the guest assets, deploys the control plane, and applies the manifests above. Like the other hack scripts it reads `.ate-dev-env.sh` for GKE; use the kind wrapper for a local cluster. @@ -48,7 +48,7 @@ Run it and follow the printed next steps: KIND_CLUSTER_NAME= ./hack/run-microvm-demo-kind.sh ``` -On a cluster that already has the micro-VM deps (`hack/install-microvm-deps.sh`), +On a cluster that already has the microVM deps (`hack/install-microvm-deps.sh`), deploy directly instead: ```bash diff --git a/docs/api-guide.md b/docs/api-guide.md index ef7a3db7af..5d98d077b5 100644 --- a/docs/api-guide.md +++ b/docs/api-guide.md @@ -105,7 +105,7 @@ The `ActorTemplate` defines the code, environment, and state-management policies | `resources` | `*ResourceRequirements` | Required. Declares each actor's compute size via `limits`, which must set both `cpuMillicores` and `memoryBytes` — see [Sandbox Right-Sizing](#sandbox-right-sizing-resources). Immutable, like the rest of the template. | | `goldenEgressPolicy` | `EgressPolicyTemplate` | **Preview.** Optional egress policy for the golden actor during initialization. Requires at least one rule, using the same rule defaults and validation as `EgressPolicy`. See [Golden Egress Policy](#golden-egress-policy-goldenegresspolicy). | -The sandbox runtime and its binaries (e.g. the gVisor `runsc` binary, or a micro-VM kernel/firmware/config) are configured on the [`WorkerPool`](#1-workerpool-the-compute-foundation) via its `sandboxClasses` and the cluster-scoped [`SandboxConfig`](#3-sandboxconfig-the-sandbox-itself) it resolves, rather than on the `ActorTemplate`. An `ActorTemplate` selects its `WorkerPool`s via `workerSelector` (AND'd with the actor's `worker_selector`). When an actor runs on a worker, the runtime it runs under is stamped onto the actor's assignment and snapshots (`sandboxRuntime`: the sandbox class and the runtime's compatibility version) so subsequent resumes prefer runtime-compatible workers. +The sandbox runtime and its binaries (e.g. the gVisor `runsc` binary, or a microVM kernel/firmware/config) are configured on the [`WorkerPool`](#1-workerpool-the-compute-foundation) via its `sandboxClasses` and the cluster-scoped [`SandboxConfig`](#3-sandboxconfig-the-sandbox-itself) it resolves, rather than on the `ActorTemplate`. An `ActorTemplate` selects its `WorkerPool`s via `workerSelector` (AND'd with the actor's `worker_selector`). When an actor runs on a worker, the runtime it runs under is stamped onto the actor's assignment and snapshots (`sandboxRuntime`: the sandbox class and the runtime's compatibility version) so subsequent resumes prefer runtime-compatible workers. ### Golden Egress Policy (`goldenEgressPolicy`) @@ -159,11 +159,11 @@ continue to work. Unlike a Pod, an actor is sized by its **`limits`** (CPU and Memory): the size is a property of the template, baked into snapshots, so it lives on the immutable `ActorTemplate`. Both are required. Declared limits do two things: 1. **Size the sandbox.** The limits are supplied to the sandbox over the actor RPCs (control plane → atelet → ateworker): - - **gVisor (`ateworker-gvisor`)** — applied to the container OCI spec: `limits.cpuMillicores` sets the cgroup v2 CPU quota (`cpu.max`) and the Sentry vCPU count (`--cpu-num-from-quota`); `limits.memoryBytes` sets the cgroup v2 memory limit (`memory.max`) and bounds the virtual total memory the sandbox reports (so JVM/Go do not over-allocate from host RAM). - - **Micro-VM (`ateworker-microvm`)** — `limits.cpuMillicores` sets Cloud Hypervisor `BootVcpus` / `MaxVcpus` (rounded up to whole vCPUs); `limits.memoryBytes` sets guest RAM, reserving a small configurable margin (default 256 MiB, `--vmm-mem-reserve-mib`) for the VMM and virtiofsd so the pod cgroup does not OOM. + - **gVisor (`ateworker-gvisor`)** — applied to the container OCI spec: `limits.cpuMillicores` sets the cgroup v2 CPU quota (`cpu.max`) and the Sentry vCPU count (`--cpu-num-from-quota`); `limits.memoryBytes` caps the sandbox's memory and bounds the virtual total memory it reports (so JVM/Go do not over-allocate from host RAM). When the node's kernel has cgroup memory pressure (PSI) enabled, an actor that stays over its limit is killed and reported crashed, and other actors on its worker keep running. + - **microVM (`ateworker-microvm`)** — `limits.cpuMillicores` sets Cloud Hypervisor `BootVcpus` / `MaxVcpus` (rounded up to whole vCPUs); `limits.memoryBytes` sets guest RAM, reserving a small configurable margin (default 256 MiB, `--vmm-mem-reserve-mib`) for the VMM and virtiofsd so the pod cgroup does not OOM. 2. **Gate scheduling.** An actor is only placed on a `WorkerPool` whose [worker capacity](#worker-capacity-spectemplateresources) is `>=` these limits. -`requests` are not consulted today (an actor occupies its whole worker). Because the size is baked into snapshots, a **micro-VM MEMORY-fidelity restore reuses the size in the snapshot**; changing an actor's limits takes effect on its next cold boot. +`requests` are not consulted today (an actor occupies its whole worker). Because the size is baked into snapshots, a **microVM MEMORY-fidelity restore reuses the size in the snapshot**; changing an actor's limits takes effect on its next cold boot. Container environment variables support literal `value` entries only. Values are not interpolated (`$(VAR)` references are not expanded), and Kubernetes `envFrom`/`valueFrom` sources are not supported. @@ -304,11 +304,11 @@ Each container runs with a default set of Linux capabilities — `AUDIT_WRITE`, - **`ALL` in `add` is rejected.** Kubernetes accepts it in the API and relies on PodSecurity admission to deny it; Substrate has no equivalent policy layer yet, so it is refused at admission instead. Name the capabilities the container needs. - **Ambient capabilities are not supported** ([gvisor#3166](https://github.com/google/gvisor/issues/3166)). -The sandbox — gVisor or micro-VM — remains the isolation boundary; capabilities constrain the workload *inside* it. +The sandbox — gVisor or microVM — remains the isolation boundary; capabilities constrain the workload *inside* it. ### Per-container limits -A container may cap its own CPU and memory so it cannot starve or kill its siblings in the same actor (on a template whose `workerSelector` targets a micro-VM pool): +A container may cap its own CPU and memory so it cannot starve or kill its siblings in the same actor (on a template whose `workerSelector` targets a microVM pool): ```yaml containers: @@ -322,9 +322,9 @@ containers: A container that exceeds its memory limit is OOM-killed on its own; the actor's other containers are unaffected. A `cpu` limit below `10m` is raised to `10m`, because the kernel rejects a CFS quota under 1ms. -Per-container limits are micro-VM only today. gVisor applies cgroup limits at the sandbox level: one sentry backs every container in the actor, so a per-container cgroup is created and then stays empty ([google/gvisor#190](https://github.com/google/gvisor/issues/190)). On a gVisor worker the per-container limits are accepted but not enforced; only the sandbox-level size applies. +Per-container limits are microVM only today. gVisor applies cgroup limits at the sandbox level: one sentry backs every container in the actor, so a per-container cgroup is created and then stays empty ([google/gvisor#190](https://github.com/google/gvisor/issues/190)). On a gVisor worker the per-container limits are accepted but not enforced; only the sandbox-level size applies. -These limits subdivide the sandbox that [`resources`](#sandbox-right-sizing-resources) already sized; a container that declares none is bounded by the guest as a whole, not by a copy of the actor's total. A micro-VM guest's memory is `resources.limits.memoryBytes` minus the VMM reserve, and its vCPU count is `resources.limits.cpuMillicores` rounded up to whole cores. A container limit above either ceiling can never bind, so the actor fails to start with an error naming both the limit and the ceiling. +These limits subdivide the sandbox that [`resources`](#sandbox-right-sizing-resources) already sized; a container that declares none is bounded by the guest as a whole, not by a copy of the actor's total. A microVM guest's memory is `resources.limits.memoryBytes` minus the VMM reserve, and its vCPU count is `resources.limits.cpuMillicores` rounded up to whole cores. A container limit above either ceiling can never bind, so the actor fails to start with an error naming both the limit and the ceiling. Each limit is validated on its own at apply, but the sum across the actor's containers is only checked when the actor first runs, against the real guest size. A template whose limits do not fit is accepted by the API server and fails on its first actor. @@ -487,7 +487,7 @@ One consequence worth planning for: **a published snapshot is read from the ates ## 3. SandboxConfig: The Sandbox Itself -`SandboxConfig` is a **cluster-scoped** resource that decouples the sandbox binaries (the gVisor `runsc` binary, or a micro-VM kernel/firmware/config) from the workload definition in the `ActorTemplate`. A `WorkerPool`'s `sandboxClasses` entry resolves its `SandboxConfig` (either by explicit `configRef` or via the class default), and `atelet` provisions each worker pod with the enabled versions of that config, using `defaultVersion` for cold boots and keeping other enabled versions in `restorableRuntimes` so existing snapshots can still restore across upgrades. +`SandboxConfig` is a **cluster-scoped** resource that decouples the sandbox binaries (the gVisor `runsc` binary, or a microVM kernel/firmware/config) from the workload definition in the `ActorTemplate`. A `WorkerPool`'s `sandboxClasses` entry resolves its `SandboxConfig` (either by explicit `configRef` or via the class default), and `atelet` provisions each worker pod with the enabled versions of that config, using `defaultVersion` for cold boots and keeping other enabled versions in `restorableRuntimes` so existing snapshots can still restore across upgrades. ### Specification (`SandboxConfigSpec`) @@ -503,7 +503,7 @@ Each entry of `versions` (`SandboxVersionConfig`): | :--- | :--- | :--- | | `name` | `string` | **Required.** Identifies the version within the config. A DNS label (lower-case alphanumerics and `-`, at most 63 characters). | | `state` | `string` | Optional. `Enabled` (default) or `Disabled`. The version named by `defaultVersion` cannot be `Disabled`. | -| `assets` | `map[arch]map[name]AssetFile` | Content-addressed files atelet fetches, keyed by architecture (`amd64`, `arm64`) then asset name. gVisor expects a `gvisor` asset (the release's `gvisor.tar.zstd`), which atelet auto-extracts. A micro-VM backend expects several. Each `AssetFile` is a `{ url, sha256 }` pair; `url` is at most 2048 bytes. A `ValidatingAdmissionPolicy` enforces each class's required assets on every version. atelet holds no storage credentials: it fetches every URL through the node's object-store plugin (`AssetProvider`), with the plugin's credentials. Which schemes work depends on the plugin; the in-tree one serves only its storage backend's scheme, `gs://` on GCS and `s3://` on S3, as `/`, and rejects other schemes without reading them. A `gs://` URL the plugin cannot serve (it has no such object, rejects the URL, does not implement `AssetProvider`, is unreachable, or fails with an error it does not recognize, such as GCS refusing its credentials for a public bucket) is then read anonymously, which serves public buckets such as `gs://gvisor`; any other refusal by the plugin is final. The plugin returns an object only if its content matches `sha256`, and refuses any URL under a snapshot location (an `ActorTemplate`'s `snapshotConfig.location` followed by `/atespaces/`), so assets must not be stored there. On GKE, setup-gcp lets the plugin read only the asset bucket, `-assets`, so keep assets there. atelet verifies the sha256 again before using an asset. Its logs and errors show an asset URL as `scheme://host/path`. | +| `assets` | `map[arch]map[name]AssetFile` | Content-addressed files atelet fetches, keyed by architecture (`amd64`, `arm64`) then asset name. gVisor expects a `gvisor` asset (the release's `gvisor.tar.zstd`), which atelet auto-extracts. A microVM backend expects several. Each `AssetFile` is a `{ url, sha256 }` pair; `url` is at most 2048 bytes. A `ValidatingAdmissionPolicy` enforces each class's required assets on every version. atelet holds no storage credentials: it fetches every URL through the node's object-store plugin (`AssetProvider`), with the plugin's credentials. Which schemes work depends on the plugin; the in-tree one serves only its storage backend's scheme, `gs://` on GCS and `s3://` on S3, as `/`, and rejects other schemes without reading them. A `gs://` URL the plugin cannot serve (it has no such object, rejects the URL, does not implement `AssetProvider`, is unreachable, or fails with an error it does not recognize, such as GCS refusing its credentials for a public bucket) is then read anonymously, which serves public buckets such as `gs://gvisor`; any other refusal by the plugin is final. The plugin returns an object only if its content matches `sha256`, and refuses any URL under a snapshot location (an `ActorTemplate`'s `snapshotConfig.location` followed by `/atespaces/`), so assets must not be stored there. On GKE, setup-gcp lets the plugin read only the asset bucket, `-assets`, so keep assets there. atelet verifies the sha256 again before using an asset. Its logs and errors show an asset URL as `scheme://host/path`. | A cluster-wide gVisor `SandboxConfig` (`gvisor-default`, annotated as the class default) is installed with the platform, so gVisor `WorkerPool`s use it automatically without any extra setup. @@ -542,24 +542,24 @@ metadata: sandboxconfig.ate.dev/is-class-default: "true" spec: sandboxClass: gvisor - defaultVersion: v4 + defaultVersion: v5 versions: - - name: v4 + - name: v5 state: Enabled assets: amd64: gvisor: - url: "gs://gvisor/releases/nightly/2026-10-17/x86_64/gvisor.tar.zstd" - sha256: "fa72f0d45756c69a30beec21222c78241b0d330c36f4f2185648617cc23415c8" + url: "gs://gvisor/releases/release/20261019.0/x86_64/gvisor.tar.zstd" + sha256: "8637abfb50f81552af25ca08f7172589c09292a7c881efe8d9e09a9f3b6bec7b" arm64: gvisor: - url: "gs://gvisor/releases/nightly/2026-10-17/aarch64/gvisor.tar.zstd" - sha256: "238634978d99f0dd3e631c4f88eae7b19fb36937065428467f0b6d6b10697689" + url: "gs://gvisor/releases/release/20261019.0/aarch64/gvisor.tar.zstd" + sha256: "88556c21468a066d872a055425958c83fcbafc79c6477a195b18070c77f856d0" ``` -### Micro-VM SandboxConfig +### MicroVM SandboxConfig -A `microvm` `SandboxConfig` supplies the [Kata Containers](https://katacontainers.io/) + [Cloud Hypervisor](https://www.cloudhypervisor.org/) toolchain instead of `runsc`. Each architecture of each version must define the full asset set — `cloud-hypervisor`, `virtiofsd`, `kata-kernel`, and `kata-image` — which a `ValidatingAdmissionPolicy` enforces at apply time. Worker pods for a micro-VM pool require `/dev/kvm` and nested-virtualization-capable nodes. The controller requests those devices on the pod automatically, and atelet advertises them only where they exist, so placement follows the hardware rather than a node label. Clusters that reserve nested-virt nodes with an `ate.dev/sandboxClass=microvm` taint are still tolerated: advertising a device attracts these pods to capable nodes but repels nothing else from them. The same convention applies to every class: worker pods of a pool tolerate `ate.dev/sandboxClass=:NoSchedule`, so a cluster can reserve a node pool per sandbox class with that taint, and the atelet DaemonSet tolerates the key for any value. On GKE, a pool can also select a ComputeClass with `template.nodeSelector` (`cloud.google.com/compute-class: `): GKE taints the nodes it creates for the class `cloud.google.com/compute-class=:NoSchedule` and adds the matching toleration to the worker pods, and the atelet DaemonSet tolerates that key as well. +A `microvm` `SandboxConfig` supplies the [Kata Containers](https://katacontainers.io/) + [Cloud Hypervisor](https://www.cloudhypervisor.org/) toolchain instead of `runsc`. Each architecture of each version must define the full asset set — `cloud-hypervisor`, `virtiofsd`, `kata-kernel`, and `kata-image` — which a `ValidatingAdmissionPolicy` enforces at apply time. Worker pods for a microVM pool require `/dev/kvm` and nested-virtualization-capable nodes. The controller requests those devices on the pod automatically, and atelet advertises them only where they exist, so placement follows the hardware rather than a node label. Clusters that reserve nested-virt nodes with an `ate.dev/sandboxClass=microvm` taint are still tolerated: advertising a device attracts these pods to capable nodes but repels nothing else from them. The same convention applies to every class: worker pods of a pool tolerate `ate.dev/sandboxClass=:NoSchedule`, so a cluster can reserve a node pool per sandbox class with that taint, and the atelet DaemonSet tolerates the key for any value. On GKE, a pool can also select a ComputeClass with `template.nodeSelector` (`cloud.google.com/compute-class: `): GKE taints the nodes it creates for the class `cloud.google.com/compute-class=:NoSchedule` and adds the matching toleration to the worker pods, and the atelet DaemonSet tolerates that key as well. See [`hack/microvm-assets/`](../hack/microvm-assets/) for scripts that assemble and stage these assets, plus a worked counter demo (`demos/counter/counter-microvm.yaml.tmpl`) that suspends and resumes an in-RAM counter across worker pods. @@ -657,10 +657,10 @@ Replaces the mutable fields of an existing actor with the ones in the request (` * **Request:** `UpdateActorRequest` * `actor`: `Actor` — the complete replacement actor. `metadata.atespace` and `metadata.name` identify the resource; `metadata.uid` and `metadata.version` are **required** preconditions. `metadata` and `status` are server-owned and whatever the request carries in them is ignored. `source_tag` is immutable. * `actor_template` may change only while the actor is `SUSPENDED`. The new template must declare the same volumes and volume mounts. Once the actor owns an external snapshot, the new template must also store snapshots under the same location: its `snapshotConfig.storageLocation` with the `PerTemplateSnapshotLocation` preview gate, otherwise ate-api-server's `--snapshot-location` (see [Snapshot Storage Layout](#snapshot-storage-layout)). - * `worker_selector` may be updated or cleared (`nil`) in any state; the updated selector takes effect on the next `ResumeActor`. + * `worker_selector` may be updated or cleared (`nil`) in any state, but not while another operation is changing the actor. The updated selector takes effect on the next `ResumeActor`. * A suspended actor with a local snapshot is initially pinned to its node (see `locality` in [Snapshot Storage Layout](#snapshot-storage-layout)). `ResumeActor` first looks for a worker there matching the template and actor selectors. If none is available but a worker satisfies the remaining constraints without node or runtime restrictions, resume detaches the checkpoint and retries placement. If no eligible worker has room, it returns `RESOURCE_EXHAUSTED`. * **Response:** the updated `Actor`. -* **Errors:** `INVALID_ARGUMENT` if `uid` or `version` is unset, or if the request changes an immutable field — including by leaving one unset; `FAILED_PRECONDITION` if a change to `actor_template` breaks one of the rules above; `ABORTED` if either guard no longer matches the stored resource. +* **Errors:** `INVALID_ARGUMENT` if `uid` or `version` is unset, or if the request changes an immutable field — including by leaving one unset; `FAILED_PRECONDITION` if a change to `actor_template` breaks one of the rules above; `ABORTED` if either guard no longer matches the stored resource, or while another operation is changing the actor (retry the same request). Because the guards are required and only a read supplies them, an update is always a read-modify-write. To Update an `Actor`, you must first `GetActor`/`CreateActor`, instead of building a new one — see [§7.2 of the API style guide](api-style-guide.md#72-using-version-and-uid-to-guard-writes) for why reconstructing the message can silently drop data. diff --git a/docs/architecture.md b/docs/architecture.md index 708aaea427..8336ab2659 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -35,7 +35,7 @@ instead use the term "actor" to refer to an instance of an agent-like workload. Since many agents run untrusted code, we need to run them in sandboxes of some sort. Agent Substrate is designed to support multiple sandbox technologies with -[gVisor](http://gvisor.dev) and micro-VMs such as +[gVisor](http://gvisor.dev) and microVMs such as [Kata Containers](https://katacontainers.io/) being popular options. Both of those happen to support a notion of suspend and resume which is core for the Agent Substrate functionality. @@ -327,7 +327,7 @@ The node-level subsystem manages the physical execution of sandboxes and the mov * **ateworker**: A specialized sandbox-herder container image — one per sandbox class (`ateworker-gvisor`, `ateworker-microvm`) — that runs inside the physical worker pods. It provides a gRPC interface for `atelet` to trigger `RunWorkload`, `CheckpointWorkload`, and `RestoreWorkload` operations. This separation ensures that the physical pod lifecycle remains decoupled from the sandboxed agent process. - * **Lifecycle Management**: The `ateworker` process invokes the sandbox runtime to checkpoint or restore processes within the physical pod boundaries — `runsc` for gVisor, or the Kata + Cloud Hypervisor stack for micro-VMs. (Note: the gVisor backend currently requires a `runsc` version with the `--allow-connected-on-save` flag to work around a bug in networking resumption during checkpointing.) + * **Lifecycle Management**: The `ateworker` process invokes the sandbox runtime to checkpoint or restore processes within the physical pod boundaries — `runsc` for gVisor, or the Kata + Cloud Hypervisor stack for microVMs. (Note: the gVisor backend currently requires a `runsc` version with the `--allow-connected-on-save` flag to work around a bug in networking resumption during checkpointing.) * **Storage Mover**: External snapshots move through an *object-store plugin*, a sidecar that serves the `objectstore.v1` gRPC API over a Unix socket in its pod. `atelet`'s sidecar (`NodeProvider`) uploads and fetches a snapshot's files; `ate-api-server`'s sidecar (`ControlProvider`) copies and deletes whole snapshots. Neither main process talks to object storage for snapshots. The in-tree plugin, `cmd/objectstore-plugin`, backs the API with GCS or S3; `ATE_STORAGE_BACKEND` on the sidecar (unset or `gcs`, or `s3` with the `AWS_*` variables) selects which, see `manifests/ate-install/atelet.yaml`, `ate-api-server.yaml`, and the kind overlay for an S3 example. The same `atelet` sidecar serves `AssetProvider`, which fetches every sandbox asset; `atelet` reads a `gs://` asset anonymously only when the plugin cannot serve it, which covers public buckets such as `gs://gvisor`. The plugin stages each download where `atelet` cannot read it, and copies it into `atelet`'s cache only if its sha256 matches; it refuses any URL under a snapshot location. `atelet` verifies the sha256 again and has no object storage credentials or settings of its own. @@ -339,7 +339,7 @@ A `WorkerPool` selects a **sandbox class** (`spec.sandboxClasses[].name`), and e * **gVisor** (`ateworker-gvisor`, the default): Runs the workload under `runsc` for kernel-level sandboxing. Suspend and resume leverage gVisor's native checkpoint/restore of the sandboxed process tree. `DurableDir` volumes and each container's rootfs overlay are saved as separate filesystem checkpoint bundles (`fs/` and `rootfs/`). A `MEMORY` snapshot splits them out of the process checkpoint; `VOLUMES` and `ROOTFS` snapshots save only the bundles they need with `runsc fscheckpoint`, without memory, and restore them on a cold boot. - * **micro-VM** (`ateworker-microvm`): Runs the workload inside a [Kata Containers](https://katacontainers.io/) guest on the [Cloud Hypervisor](https://www.cloudhypervisor.org/) VMM. Suspend and resume capture a memory-only VM snapshot and restore it on-demand using `userfaultfd` memory demand-paging. Container rootfs writes are host-backed: the overlay is assembled on the host (read-only OCI image lower plus a per-actor writable upper) and served to the guest over the single virtio-fs share, so they cost reclaimable host page cache rather than guest RAM, and `ROOTFS` and `MEMORY` snapshots ship the upper as its own tar (a `ROOTFS` restore puts it back and cold-boots on it). `DurableDir` volumes travel over that same share and are likewise shipped as a tar, so a `VOLUMES` snapshot can capture them without any guest memory. Each volume is a subdirectory of the share, so an actor can have several at no extra cost in devices. + * **microVM** (`ateworker-microvm`): Runs the workload inside a [Kata Containers](https://katacontainers.io/) guest on the [Cloud Hypervisor](https://www.cloudhypervisor.org/) VMM. Suspend and resume capture a memory-only VM snapshot and restore it on-demand using `userfaultfd` memory demand-paging. Container rootfs writes are host-backed: the overlay is assembled on the host (read-only OCI image lower plus a per-actor writable upper) and served to the guest over the single virtio-fs share, so they cost reclaimable host page cache rather than guest RAM, and `ROOTFS` and `MEMORY` snapshots ship the upper as its own tar (a `ROOTFS` restore puts it back and cold-boots on it). `DurableDir` volumes travel over that same share and are likewise shipped as a tar, so a `VOLUMES` snapshot can capture them without any guest memory. Each volume is a subdirectory of the share, so an actor can have several at no extra cost in devices. ### Networking Stack (`atenet` + `atunnel`) diff --git a/docs/authorization.md b/docs/authorization.md index 8dbf7b1ffa..271123faa9 100644 --- a/docs/authorization.md +++ b/docs/authorization.md @@ -223,7 +223,7 @@ certificate. | `RequestActorSuspend` | `atelet` on the node of the worker hosting the actor | | `MintWorkerActorJwt` | `atelet` on the node hosting the actor | | `MintWorkerActorCertificate` | `atelet` on the node of the worker the actor is assigned to | -| `ReportActorSnapshotState` | `atelet` on the node the actor is placed on or holds a local snapshot of it | +| `ReportActorSnapshotState` | `atelet` on the node the actor is placed on | ### Every RPC diff --git a/docs/csi-deployment.md b/docs/csi-deployment.md index b646ce1fb5..813d563e25 100644 --- a/docs/csi-deployment.md +++ b/docs/csi-deployment.md @@ -205,7 +205,7 @@ spec: mountPath: /csi # Envoy proxy sidecar terminating mTLS and forwarding to /csi/csi.sock - name: envoy-proxy - image: envoyproxy/envoy:v1.34-latest + image: envoyproxy/envoy:v1.39.5@sha256:8fc594a5d85eb76dca24155cfb9208882406873932b20cf33649a0a611194c43 command: - "/usr/local/bin/envoy" args: diff --git a/docs/dev/best-practices/metrics.md b/docs/dev/best-practices/metrics.md index 83f7e9cbb1..8f3997a6b5 100644 --- a/docs/dev/best-practices/metrics.md +++ b/docs/dev/best-practices/metrics.md @@ -67,7 +67,7 @@ Rules of thumb: it ends, you subtract one. `ate.workerpool.workers` is one because ateapi assigns and releases every worker, so it is the one keeping that tally. A gauge is a dial you read: memory working set is whatever the sandbox reports - when atelet polls it (a cgroup on gVisor, the guest agent on micro-VM), + when atelet polls it (a cgroup on gVisor, the guest agent on microVM), written down per template. Ask "did I get this number by adding and subtracting, or by looking?" and the answer is the instrument. How a dashboard later aggregates the number does not enter into it. The diff --git a/docs/dev/best-practices/otel-collector.md b/docs/dev/best-practices/otel-collector.md index 01448f2de2..fe4582bfc0 100644 --- a/docs/dev/best-practices/otel-collector.md +++ b/docs/dev/best-practices/otel-collector.md @@ -505,7 +505,7 @@ failure. ## Local Development (kind) -`hack/install-ate-kind.sh` installs a collector plus a Jaeger all-in-one into +`hack/install-ate-kind.sh` installs a collector plus Jaeger into the `otel-system` namespace — see `manifests/ate-install/kind/otel-collector.yaml`. It is a single-replica Deployment, which is appropriate for a one-node kind cluster. The kind diff --git a/docs/dev/code-layout.md b/docs/dev/code-layout.md index ffbde39fee..088e83fa4d 100644 --- a/docs/dev/code-layout.md +++ b/docs/dev/code-layout.md @@ -73,7 +73,7 @@ Each subdirectory of `cmd/` corresponds to one compiled binary: | `cmd/atelet` | Node supervisor (DaemonSet) | | `cmd/atenet` | Network proxy / Envoy external-processing server | | `cmd/ateworker-gvisor` | In-pod gVisor container image entry point | -| `cmd/ateworker-microvm` | In-pod kata + cloud-hypervisor micro-VM container image entry point | +| `cmd/ateworker-microvm` | In-pod kata + cloud-hypervisor microVM container image entry point | | `cmd/kubectl-ate` | `kubectl` plugin for interacting with Substrate | | `cmd/podcertcontroller` | Controller that issues pod TLS certificates | diff --git a/docs/dev/microvm-local.md b/docs/dev/microvm-local.md index cbbb25e541..66aba29851 100644 --- a/docs/dev/microvm-local.md +++ b/docs/dev/microvm-local.md @@ -40,11 +40,11 @@ instead, or open up the device with `sudo chmod 666 /dev/kvm`. ```sh ./hack/create-kind-cluster.sh -# Look for: "/dev/kvm found: micro-VM (kata + cloud-hypervisor) support will be enabled." +# Look for: "/dev/kvm found: microVM (kata + cloud-hypervisor) support will be enabled." ``` Once the control plane is up, atelet advertises the device on each KVM-capable -node, which is what places micro-VM workers: +node, which is what places microVM workers: ```sh kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.capacity.ate\.dev/kvm}{"\n"}{end}' @@ -185,7 +185,7 @@ resume it and confirm the count continues — proving the guest-memory snapshot round-tripped. The flow is the same as the [README Quickstart](../../README.md#quickstart-development), just with the microVM template; see the -[counter demo's micro-VM variant](../../demos/counter/README.md#micro-vm-variant) +[counter demo's microVM steps](../../demos/counter/README.md#1-build-and-deploy) for background. Note that an actor template showing a GOLDEN SNAPSHOT in the verify step already exercises the runtime end-to-end — the golden snapshot requires a full guest boot and checkpoint. diff --git a/docs/dev/mitm-egress-test-runbook.md b/docs/dev/mitm-egress-test-runbook.md index 912e823104..789ed24497 100644 --- a/docs/dev/mitm-egress-test-runbook.md +++ b/docs/dev/mitm-egress-test-runbook.md @@ -8,13 +8,13 @@ normal public roots. The CI workflow installs both egress ActorTemplates, adds the service-DNS CA to the gateway's upstream roots with `hack/setup-e2e-egress-tls-kind.sh`, and -runs the networking suite on gVisor and micro-VM. The commands below reproduce +runs the networking suite on gVisor and microVM. The commands below reproduce that setup locally. ## Prerequisites Follow the [development quickstart](../../README.md#quickstart-development) -for Go, Docker, `kubectl`, and Kind. The micro-VM lane also needs KVM and the +for Go, Docker, `kubectl`, and Kind. The microVM lane also needs KVM and the setup in [Running the microVM runtime locally](microvm-local.md). ## Install the cluster and fixtures @@ -25,7 +25,7 @@ kubectl config use-context kind-kind ATE_CREDENTIAL_PROVIDER='{"name":"k8s.io"}' ./hack/install-ate-kind.sh --deploy-ate-system ``` -For micro-VM, install its assets before deploying the fixture: +For microVM, install its assets before deploying the fixture: ```bash NO_DEV_ENV=true ATE_INSTALL_KIND=true KUBECTL_CONTEXT=kind-kind \ @@ -54,7 +54,7 @@ Run the gVisor lane: hack/run-e2e-kind.sh ./internal/e2e/suites/networking -run '^TestActorEgress' -v -args --no-color ``` -Run the micro-VM lane sequentially on the same cluster: +Run the microVM lane sequentially on the same cluster: ```bash E2E_SANDBOX_CLASS=microvm hack/run-e2e-kind.sh ./internal/e2e/suites/networking -run '^TestActorEgress|^TestMicroVMClockAfterDelayedRestore' -v -args --no-color @@ -64,7 +64,7 @@ The HTTPS protocol test checks a verified HTTP/1.1 request to a local TLS origin on port 8443. The WebSocket tests check that both `ws` and `wss` upgrades are denied with HTTP 403 before the origin receives any messages. The delayed-restore test issues a fresh origin certificate after the Actor is -suspended, then checks that the restored micro-VM can complete the TLS request. +suspended, then checks that the restored microVM can complete the TLS request. ## Troubleshooting diff --git a/docs/dev/postgresql-schema-evolution.md b/docs/dev/postgresql-schema-evolution.md index b41afcadc0..9193a7444c 100644 --- a/docs/dev/postgresql-schema-evolution.md +++ b/docs/dev/postgresql-schema-evolution.md @@ -61,9 +61,9 @@ Standalone Substrate defaults to the `substrate` schema, including under Kagent' - Keep each startup migration short. - A database administrator must configure the owner role's default table and sequence privileges for the read/write role before migrations. These grants do not go to `PUBLIC` or Kagent's roles. In this first pass, migration ledgers receive the same table privileges as other objects. -Before the first stable v1 release, developers can change or squash migration files. Recreate a development database after its migration history changes. +Migrations up to v0.6.0 were squashed into `000001_initial.sql`. A database created before the squash cannot be upgraded. Recreate it. -After the first stable v1 release, do not change or delete a released migration file. Add a new migration to correct it. +Do not change or delete a migration file after it merges. Add a new migration to correct it. Do not edit the `schema_migrations` ledger manually. @@ -76,6 +76,6 @@ Do not edit the `schema_migrations` ledger manually. 5. Run the migration verifier and PostgreSQL store tests. ```sh -hack/verify/postgresql-migrations.sh +hack/verify/postgresql-migrations.sh origin/main go test ./cmd/ateapi/internal/store/atepg ``` diff --git a/docs/dev/release-notes.md b/docs/dev/release-notes.md index d9f6d153ad..97081d62af 100644 --- a/docs/dev/release-notes.md +++ b/docs/dev/release-notes.md @@ -1,6 +1,6 @@ # Release notes -GitHub generates the release notes for each Substrate release from the labels and titles of the merged pull requests. [`.github/release.yml`](../../.github/release.yml) defines the sections. Pushing a release tag creates a draft release with the generated notes. A maintainer edits the draft and publishes it. +GitHub generates the release notes for each Substrate release from the labels and titles of the merged pull requests. [`.github/release.yml`](../../.github/release.yml) defines the sections. Pushing a release tag creates a draft release with the generated notes. A maintainer edits the draft and publishes it; see [Releasing Substrate](releasing.md). The notes are only as good as the labels. You can fetch the current labels and their descriptions using `gh`. @@ -38,24 +38,3 @@ The generated notes list each pull request as ` by @author in <link>`. Wr | `Fix #1234` | `Fix the kind install on arm64 hosts` | For a breaking change, describe the upgrade step in the pull request description and add the `breaking-change` label. The release manager uses it to write the note. - -## Cut a release - -1. Tag the release commit `vX.Y.Z`, or `vX.Y.Z-rc.N` for a release candidate, and push the tag. The [`release-notes`](../../.github/workflows/release-notes.yaml) workflow runs [`hack/release/draft-release.sh`](../../hack/release/draft-release.sh), which creates a draft release. The notes cover every change since the previous `vX.Y.Z` release and end with a list of committers. A release candidate is marked as a prerelease. -2. Edit the draft: - - Write a summary at the top. - - Rewrite each breaking change with its upgrade step. - - Optionally, group the feature sections under one `## Features` heading. - - Remove Other Changes entries users do not need, and move entries that landed in the wrong section. -3. Publish the release. - -If the workflow did not run, start it from the Actions tab with the tag as input, or run the script with an authenticated `gh` that has write access to the repository: - -```sh -hack/release/draft-release.sh --dry-run v0.3.0 # print the notes only -hack/release/draft-release.sh v0.3.0 # create the draft -``` - -The script never modifies a release that already exists. - -To preview the notes before tagging, run the dry run with the tag you plan to push. If the tag does not exist yet, the notes run up to your local `HEAD`, which must already be on a branch in the repository. diff --git a/docs/dev/releasing.md b/docs/dev/releasing.md new file mode 100644 index 0000000000..8fb606cfca --- /dev/null +++ b/docs/dev/releasing.md @@ -0,0 +1,92 @@ +# Releasing Substrate + +A release is a `vX.Y.Z` tag, or `vX.Y.Z-rc.N` for a release candidate, with container images in `ghcr.io/agent-substrate/substrate` and a GitHub release whose notes are generated from the merged pull requests. How the notes are generated, and how to label and title a pull request for them, is in [Release notes](release-notes.md). + +## Who may release + +Pushing a `vX.Y.Z` or `vX.Y.Z-PRERELEASE` tag publishes that commit's images to `ghcr.io/agent-substrate/substrate` and creates a release, under the repository's token. Whoever can push such a tag can therefore publish images for any commit. Only release managers may push one. + +A tag ruleset on `agent-substrate/substrate` that restricts creating, updating, and deleting `v[0-9]*` tags to release managers enforces this for tag pushes. An organization admin maintains it, and it has to be in place before the first release that publishes images. + +The ruleset limits tag pushes only. Anyone with write access can still publish images: they can start this workflow from the Actions tab, from any branch, and publish or overwrite the images of a tag whose release is still a draft, and a workflow on a branch they push can request `packages: write` as well. Once a tag's release is published, this workflow publishes nothing for it. + +## Cut a release + +1. Tag the release commit and push the tag. The [`release`](../../.github/workflows/release.yaml) workflow does two things, in order: + 1. The `images` job runs `ate-setup publish release-images`, which builds every release image for `linux/amd64` and `linux/arm64`, pushes it to `ghcr.io/agent-substrate/substrate/<image>:<tag>`, and then checks that every tag names the image just pushed, for both platforms. Every release image is built with `ko`, which cross-compiles for arm64 without emulation. + 2. Once the images are verified, the `draft` job builds the `kubectl-ate` binaries with `ate-setup publish release-binaries`, then runs [`hack/release/draft-release.sh`](../../hack/release/draft-release.sh), which creates a draft release, and attaches the binaries to it. The notes cover every change since the previous `vX.Y.Z` release, say where the release images are published and link to the list below, give the steps to check and install `kubectl-ate`, and end with a list of committers. A release candidate is marked as a prerelease. + + If the `images` job or the binary build fails, no draft is created. If attaching the binaries fails, the draft exists without them. Either way, fix the cause and run the workflow again from the Actions tab with the tag as input. +2. Edit the draft: + - Write a summary at the top. + - Rewrite each breaking change with its upgrade step. + - Optionally, group the feature sections under one `## Features` heading. + - Remove Other Changes entries users do not need, and move entries that landed in the wrong section. +3. Publish the release. + +If the workflow did not run, start it from the Actions tab with the tag as input, or run the script with an authenticated `gh` that has write access to the repository: + +```sh +hack/release/draft-release.sh --dry-run v0.3.0 # print the notes only +hack/release/draft-release.sh v0.3.0 # create the draft +``` + +The script never modifies a release that already exists. + +To preview the notes before tagging, run the dry run with the tag you plan to push. If the tag does not exist yet, the notes run up to `origin/main` in your clone, which must already be in the repository. + +## Release images + +A release publishes `ate-idp-server`, `ateapi`, `atecontroller`, `atelet`, `atenet`, `ateworker-gvisor`, `ateworker-microvm`, `objectstore-plugin`, and `podcertcontroller`, as `ghcr.io/agent-substrate/substrate/<image>:<tag>` (`images.ReleaseComponents` in [`cmd/ate-setup/internal/images`](../../cmd/ate-setup/internal/images/images.go)). The object storage plugin ships with releases until it moves out of this repository. + +The list is an allowlist, `released` in `images.go`. A new component is not published until a pull request adds it there, so publishing an image is always an explicit, reviewed change. + +It leaves out `envoy-dataplane`, the Kubernetes Secrets credential provider, the demos, and the benchmarking fakes. The release images are therefore not a complete install of the default configuration, and the notes give no install command: an install has to take the images releases leave out from another registry, or build them from source. `ate-setup --image-repo` looks those images up like any other; if one is missing, the error says releases do not publish it. + +Tags that begin `vX.Y.Z` start the workflow. `ate-setup` refuses any version that is not `vX.Y.Z` or `vX.Y.Z-PRERELEASE` before building, so a tag like `v1.2.3foo` publishes nothing. + +Only `agent-substrate/substrate` publishes images. In a fork the `images` job is skipped, since GitHub bills package storage to the fork's owner, and the draft is created without them. + +The kagent-dev fork also provides the manually dispatched +[`release-fork`](../../.github/workflows/release-fork.yaml) workflow. It publishes +the fork's component images, including the Kubernetes Secrets credential +provider, to `ghcr.io/kagent-dev/substrate`. With `create_release` enabled, it +also publishes the three Helm charts and attaches `kubectl-ate` binaries to +the release. Without that option it publishes only images, using the supplied +tag or a branch-and-commit tag. The separate +[`agentgateway-image`](../../.github/workflows/agentgateway-image.yaml) workflow +builds the fork's agentgateway image from a selected source revision. + +A rerun builds nothing if every image already exists for the tag on both platforms. An incomplete set, left by a failed run, is rebuilt in full. To rebuild and replace a complete set, run the workflow manually with `overwrite` checked. Replacing images changes their digests, so once the tag has a published (non-draft) release the `images` job publishes nothing and fails, with or without `overwrite`. + +To test the image build without publishing anything, for example in a fork, run the workflow manually with `dry_run_images` checked. The images are pushed to a registry that exists only inside the job and are checked there. No draft is created. + +GitHub creates a new package as private. After the first release that adds an image, an owner of the `agent-substrate` organization must make the package public in its package settings, otherwise `ate-setup --image-repo` cannot read it without credentials. + +To publish images by hand, check out the tag in a clean clone, log in to the registry with `docker login`, and run: + +```sh +VERSION=v0.3.0 KO_DEFAULTPLATFORMS=linux/amd64,linux/arm64 \ + go run ./cmd/ate-setup publish release-images --ko-docker-repo <registry path> +``` + +The images are built from the work tree, so it must hold the tag and nothing else. The build needs only Go and `ko`, which `ate-setup` runs from `hack/tools/ko`. + +## Release binaries + +The `draft` job attaches `kubectl-ate` for `linux/amd64`, `linux/arm64`, `darwin/amd64`, and `darwin/arm64`, as raw binaries named `kubectl-ate-<os>-<arch>`, each with a `<name>.sha256` file. There is no Windows build. The binaries are static (`CGO_ENABLED=0`) and stamped with the tag, the commit, and the commit time, which `kubectl ate --version` prints. + +Only `agent-substrate/substrate` attaches binaries; a fork's draft has none. A rerun replaces the binaries only while the release is still a draft. + +To check and install one: + +```sh +shasum -a 256 -c kubectl-ate-linux-amd64.sha256 +install -m 0755 kubectl-ate-linux-amd64 ~/.local/bin/kubectl-ate +``` + +To build them by hand, check out the tag in a clean clone and run: + +```sh +VERSION=v0.3.0 go run ./cmd/ate-setup publish release-binaries +``` diff --git a/docs/dev/suspend-resume-phase-breakdown.md b/docs/dev/suspend-resume-phase-breakdown.md index 50cee957cb..2983693382 100644 --- a/docs/dev/suspend-resume-phase-breakdown.md +++ b/docs/dev/suspend-resume-phase-breakdown.md @@ -24,10 +24,10 @@ the report. or the [GKE Quickstart](../../README.md#gke-quickstart-development). For GKE the scripts below read `.ate-dev-env.sh` from the repository root (`collect_logs.sh` sources it itself; set `NO_DEV_ENV` to opt out). -* A **micro-VM** or **gVisor** pool: both ateworkers write the inner records, +* A **microVM** or **gVisor** pool: both ateworkers write the inner records, so the report shows both layers for either class (a worker image that predates the gVisor records shows the atelet layer only on a gVisor pool). - For micro-VM see [microvm-local.md](microvm-local.md). + For microVM see [microvm-local.md](microvm-local.md). * `kubectl`, `kubectl-ate` (`go install ./cmd/kubectl-ate`), and `python3` (standard library only). diff --git a/docs/examples/substrate-install.yaml b/docs/examples/substrate-install.yaml index 7d45899d03..e1077f6563 100644 --- a/docs/examples/substrate-install.yaml +++ b/docs/examples/substrate-install.yaml @@ -42,9 +42,11 @@ atenet: # Prebuilt images. Both halves are required together: a tag with no repository # would be dropped, leaving a build from source that looks like the release the -# tag names. +# tag names. The repository has to hold every image the install deploys; the +# release images in ghcr.io/agent-substrate/substrate are not a complete set +# (see docs/dev/releasing.md). images: - repo: ghcr.io/agent-substrate/substrate + repo: registry.example.com/substrate tag: v0.4.1 # Snapshot storage. diff --git a/docs/glossary.md b/docs/glossary.md index 4fe6787cb5..9d97989d7f 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -24,7 +24,7 @@ For how the pieces fit together, see the [Architecture](architecture.md) and - **SandboxConfig** (Kubernetes CRD): a cluster-scoped resource holding the sandbox binaries for one runtime family (the gVisor `runsc` binary, or a - micro-VM kernel/firmware/config). An actor resolves its sandbox at first cold + microVM kernel/firmware/config). An actor resolves its sandbox at first cold boot from the config its `ActorTemplate` names (naming one is currently required; per-class cluster defaults are planned), so one config pins the runtime version for many templates. diff --git a/docs/metrics/registry/events.yaml b/docs/metrics/registry/events.yaml index e119254217..c7802bc237 100644 --- a/docs/metrics/registry/events.yaml +++ b/docs/metrics/registry/events.yaml @@ -115,7 +115,7 @@ groups: initial record is absent when the first reading fails. The final record is written when the ateworker tears the activation down: a checkpoint, a terminate, a graceful shutdown of the worker, or a re-host on a - micro-VM. It is read just before the teardown and repeats the newest + microVM. It is read just before the teardown and repeats the newest sample when that read fails or times out, or is pending when none was measured. It is absent when the activation ends otherwise, as by a crash, a lost worker, or a re-host on gVisor. Thus take the highest diff --git a/docs/metrics/registry/metrics.yaml b/docs/metrics/registry/metrics.yaml index 5201d98c32..41f9171fdf 100644 --- a/docs/metrics/registry/metrics.yaml +++ b/docs/metrics/registry/metrics.yaml @@ -261,7 +261,7 @@ groups: - id: microvm stability: development value: microvm - brief: A micro-VM sandbox. + brief: A microVM sandbox. - id: unknown stability: development value: unknown @@ -1057,7 +1057,7 @@ groups: brief: The uncompressed allocated size in bytes of each snapshot image that a checkpoint writes. note: > atelet records one measurement for each image file from its allocated disk - blocks (st_blocks * 512, excluding sparse holes in micro-VM + blocks (st_blocks * 512, excluding sparse holes in microVM memory-ranges). It does not record one measurement for each checkpoint. Use the file.name key to compare the same type of image. annotations: @@ -1257,10 +1257,10 @@ groups: It does not count usage twice. It can count less: the end of an activation after the last sample atelet read of it, which is up to one poll interval plus one ateworker sample interval plus the sweep's own time - (up to 45s on a micro-VM), so an activation shorter than that can lose + (up to 45s on a microVM), so an activation shorter than that can lose most or all of its CPU here, while its usage records keep it; the time around an atelet restart; and, in a - micro-VM restored from a full snapshot, a container's use before its + microVM restored from a full snapshot, a container's use before its first successful reading. A node clock that steps back across an atelet restart can count one activation in full a second time. annotations: @@ -1406,7 +1406,7 @@ groups: brief: The number of entries that the eviction passes removed from a file cache on the node, by cache. note: > An entry is one cached file or tree, such as a runsc binary or a - micro-VM kernel. A pass removes only an entry that nothing links, + microVM kernel. A pass removes only an entry that nothing links, oldest first, and a dry run removes none. A rate that stays above zero with a size at the budget means that the cache thrashes: an entry leaves and returns on each pass, and each return pays for a download. diff --git a/docs/observability.md b/docs/observability.md index f94c0b2480..97cf4d8bbc 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -232,7 +232,7 @@ The attributes are the same flat `ate.*` keys as the stdout record, so they arri ### Per-Actor Usage Events -Each ateworker samples its actors on its own timer and writes [`ate.actor.usage_sampled`](#the-same-records-over-otlp) for each one, in this order: an `initial` record when an activation becomes measurable, a `periodic` record per interval, and a `final` record when the ateworker tears the activation down: a checkpoint, a terminate, a graceful shutdown, or a re-host on a micro-VM. The sampler reads an actor only after its initial reading and until its final record; until then the discovery read reports it as pending. A failed first reading writes no `initial` record, and periodic records start anyway, so an unreachable guest is still reported. These records and the [`ate.actor.stats.*` metrics](#the-metric-registry) are the halves of one split: the metrics aggregate to the bounded template and pool label set a TSDB can hold, and everything carrying actor identity travels here, on the log stream, where cardinality is free. +Each ateworker samples its actors on its own timer and writes [`ate.actor.usage_sampled`](#the-same-records-over-otlp) for each one, in this order: an `initial` record when an activation becomes measurable, a `periodic` record per interval, and a `final` record when the ateworker tears the activation down: a checkpoint, a terminate, a graceful shutdown, or a re-host on a microVM. The sampler reads an actor only after its initial reading and until its final record; until then the discovery read reports it as pending. A failed first reading writes no `initial` record, and periodic records start anyway, so an unreachable guest is still reported. These records and the [`ate.actor.stats.*` metrics](#the-metric-registry) are the halves of one split: the metrics aggregate to the bounded template and pool label set a TSDB can hold, and everything carrying actor identity travels here, on the log stream, where cardinality is free. The stdout record carries the same flat `ate.*` keys as the OTLP event, dated when the measurement was read: @@ -253,9 +253,9 @@ The stdout record carries the same flat `ate.*` keys as the OTLP event, dated wh jsonPayload.msg="Actor usage sampled" AND jsonPayload."ate.actor.uid"="8f2a…" ``` -* **The measurements** follow [the rules above](#the-same-records-over-otlp): absent while the actor is pending, CPU in seconds since the activation began, memory usage and working set absolute. An activation that ends any other way, as by a crash, has no `final` record, so take the highest CPU per epoch rather than wait for one. The `final` reading is taken just before the teardown, bounded on a micro-VM to 10ms. When it fails or times out, the `final` record repeats the newest measured sample, dated when that sample was read, or is pending when none was measured. +* **The measurements** follow [the rules above](#the-same-records-over-otlp): absent while the actor is pending, CPU in seconds since the activation began, memory usage and working set absolute. An activation that ends any other way, as by a crash, has no `final` record, so take the highest CPU per epoch rather than wait for one. The `final` reading is taken just before the teardown, bounded on a microVM to 10ms. When it fails or times out, the `final` record repeats the newest measured sample, dated when that sample was read, or is pending when none was measured. * **Cadence**: `--usage-sample-interval` on the ateworker sets the timer (default `1m`, at least `50s`). Each sweep writes one periodic record per actor, and `GetActiveWorkloadStats` serves the latest sample rather than reading again. -* **Trace context**: `initial` records come from the Run or Restore call (on a micro-VM, from a goroutine the call starts, so the record can land just before or after the call returns) and `final` records from the call that tears the activation down, a Checkpoint, a Terminate, or, on a micro-VM, the Run or Restore that re-hosts it; both carry that call's span, like any record in [Joining Logs to Traces](#joining-logs-to-traces). `periodic` records come from the sampler, and a shutdown's `final` records from the drain, which serve no request, so they carry none. +* **Trace context**: `initial` records come from the Run or Restore call (on a microVM, from a goroutine the call starts, so the record can land just before or after the call returns) and `final` records from the call that tears the activation down, a Checkpoint, a Terminate, or, on a microVM, the Run or Restore that re-hosts it; both carry that call's span, like any record in [Joining Logs to Traces](#joining-logs-to-traces). `periodic` records come from the sampler, and a shutdown's `final` records from the drain, which serve no request, so they carry none. * **Delivery is best-effort**: the stdout record ignores `--log-level` and sits behind a bounded queue, so a stalled log consumer costs records, with a warning, never the sampler. **Do not put actor identity on a log-based metric.** Aggregating these events in the log store is what they are for, but a log-based metric built over them must label only by the bounded set (template, sandbox class, source, pool) — promoting `ate.actor.uid` or `ate.actor.name` into a metric label reintroduces exactly the per-actor cardinality this split keeps out of the TSDB. @@ -279,7 +279,7 @@ Agent Substrate emits foundational OpenTelemetry system and server metrics to mo | `rpc.server.call.duration` | ateapi, atelet, atenet-router, ateworker-gvisor, ateworker-microvm & credprovider (gRPC servers, via `otelgrpc`) | histogram | per-method gRPC latency, request rate, and errors (labels `rpc.method`, `rpc.response.status_code`) | | `ate.actor.crashes` | ateapi | counter | Number of times actors transitioned to `ACTOR_STATE_CRASHED` (labels `ate.actor.operation.name`, `ate.template.atespace`, `ate.template.name`, `ate.workerpool.namespace`, `ate.workerpool.name`, `ate.sandbox.class`) | | `atenet.router.route.duration` | atenet-router | histogram | Substrate E2E — Envoy receiving a request to Envoy forwarding it to the resolved worker, excluding actor compute and the response (labels `ate.template.atespace`, `ate.template.name`, `ate.router.outcome`, `ate.router.resume`) | -| `atelet.snapshot.size` | atelet | histogram | uncompressed allocated size in bytes of each snapshot image written during checkpoint (`st_blocks * 512`, excluding sparse holes in micro-VM `memory-ranges`) (labels `file.name`, `ate.template.atespace`, `ate.template.name`) | +| `atelet.snapshot.size` | atelet | histogram | uncompressed allocated size in bytes of each snapshot image written during checkpoint (`st_blocks * 512`, excluding sparse holes in microVM `memory-ranges`) (labels `file.name`, `ate.template.atespace`, `ate.template.name`) | | `ate.workerpool.desired_workers` | atecontroller | up/down counter | number of worker pods requested for a WorkerPool, from `spec.replicas` (labels `ate.workerpool.namespace`, `ate.workerpool.name`) | | `ate.workerpool.ready_workers` | atecontroller | up/down counter | number of worker pods currently ready for a WorkerPool, from `status.readyReplicas` (labels diff --git a/docs/operator-install.md b/docs/operator-install.md index bbf0678424..86b46f6d5b 100644 --- a/docs/operator-install.md +++ b/docs/operator-install.md @@ -15,7 +15,7 @@ namespace: ate-system atenet: dataplane: envoy images: - repo: ghcr.io/agent-substrate/substrate + repo: registry.example.com/substrate tag: v0.4.1 ``` @@ -315,6 +315,8 @@ demos, the [benchmarks](../benchmarking/README.md#settings), and the | `otlpEndpoint` | `ATE_OTLP_ENDPOINT` | `--otlp-endpoint` | — | (global) | | `podcert.workersPerSigner` | `ATE_INSTALL_PODCERT_WORKERS_PER_SIGNER` | `--podcert-workers-per-signer` | `0` | (global) | | `preview` | `ATE_PREVIEW` | `--preview` | — | (global) | +| `publish.binariesDir` | `ATE_PUBLISH_BINARIES_DIR` | `--out` | `bin/release` | `publish release-binaries` | +| `publish.overwrite` | `ATE_PUBLISH_OVERWRITE` | `--overwrite` | `false` | `publish release-images` | | `record.dir` | `ATE_RECORD_DIR` | `--record-dir` | — | (global) | | `rolloutTimeout` | `ATE_INSTALL_ROLLOUT_TIMEOUT` | `--rollout-timeout` | `1m0s` | (global) | | `storage.bucketName` | `BUCKET_NAME` | `--storage-bucket-name` | — | (global) | @@ -352,13 +354,13 @@ does not create the bucket or grant access to it: run [tools/setup-gcp](../tools/setup-gcp/README.md)). setup-gcp also creates an asset bucket, `<BUCKET_NAME>-assets`, where -`hack/install-microvm-deps.sh` stages micro-VM sandbox assets, and gives atelet +`hack/install-microvm-deps.sh` stages microVM sandbox assets, and gives atelet read access to it. atelet gets no access to the snapshot bucket: every snapshot transfer runs on a token scoped to its actor, issued by ate-api-server's credential exchange. A GCP project set up before the asset bucket existed keeps atelet's old grants, since setup-gcp only adds bindings. Remove them as [tools/setup-gcp](../tools/setup-gcp/README.md#upgrading-an-existing-install) -shows, which also covers recreating a cluster with micro-VM workers. +shows, which also covers recreating a cluster with microVM workers. With the `PerTemplateSnapshotLocation` preview gate, an ActorTemplate may set `snapshotConfig.storageLocation` to store its snapshots elsewhere; templates diff --git a/docs/sandbox-asset-cache.md b/docs/sandbox-asset-cache.md index 13df476b5a..1b42d9524a 100644 --- a/docs/sandbox-asset-cache.md +++ b/docs/sandbox-asset-cache.md @@ -3,7 +3,7 @@ ## Summary Every actor runs on sandbox binaries that come from a cluster-scoped -`SandboxConfig`: a gVisor release, or a micro-VM kernel, VMM and guest image. +`SandboxConfig`: a gVisor release, or a microVM kernel, VMM and guest image. `atelet` keeps **one node-local, content-addressed cache** of these assets and gives each actor **hard links** into it. Nothing is copied per actor, an asset is downloaded once per node however many actors and workers use it, and the @@ -26,7 +26,7 @@ cannot cross mounts. | `actors/<actor uid>/sandbox-assets/` | The actor's assets, each a hard link into the cache, named by asset name. The only location a worker runs a sandbox binary from; the worker checks that a request names the actor's own directory. `sandbox-assets.staging/` beside it exists only while a Run fetches. | | `workers/<pod uid>/sandbox-runtimes.json` | What each worker advertises to the control plane, written by `atelet`. | | `workers/<pod uid>/default-runtime/<hash>/` | The worker's default runtimes, one directory per default its file advertises, by bundle hash, one per sandbox class, linked in from the cache by the prewarmer. These links are what keeps a default runtime cached while no actor runs it; the syncer removes a directory when the file stops naming its runtime, and they all go with the worker's directory when the pod does. | -| `sandbox-asset-scratch/` | Short-lived destinations for the one fetch that has no actor and no worker directory to link into: the `runsc` binary the worker sandbox syncer probes for CPU features. Wiped at startup. | +| `sandbox-asset-scratch/` | Short-lived destinations for fetches that have no actor and no worker directory to link into: the assets the worker sandbox syncer stages when checking a version (and probes `runsc` for CPU features). Wiped at startup. | | `sandbox-runtime-index.json` | The runtime index serialized: every runtime a worker on this node has advertised, by hash, with the asset record the hash was computed from. | A cache entry is keyed by the asset's sha256, so the same binaries under @@ -67,15 +67,18 @@ pool runs, hashes every enabled version, and: - **registers** every hash in the runtime index, with its record, and writes the index file durably, *before* anything else, so a hash is resolvable before the worker can register it with the control plane; -- probes what a snapshot taken on each runtime must find again (CPU features - for gVisor, host CPU identity for micro-VM) and writes the worker's - `sandbox-runtimes.json`, from which the worker registers; +- fetches each version's assets into the cache (every sync for a microVM + default, and once per `atelet` process for restorable microVM versions and + gVisor `runsc` digests), probes what a snapshot taken on the runtime must + find again (CPU features for gVisor, host CPU identity for microVM), and + writes the worker's `sandbox-runtimes.json`, from which the worker + registers; - records what the worker now advertises, which is what the written file says; - makes the worker's `default-runtime/` directory match the file's - **default** runtimes, and hands each to the **prewarmer**, which fetches - its assets into the cache and links them into the runtime's directory, so - the first actor on the node finds a warm cache. + **default** runtimes, and hands each to the **prewarmer**, which links + the default's cached assets into the runtime's directory so they stay + cached while no actor runs them. A `SandboxConfig` or `WorkerPool` change registers the affected pools' runtimes at once and rewrites their files after a jitter of up to thirty @@ -84,8 +87,14 @@ restart the index loads from its file before the port opens, so a request for a runtime a worker registered earlier resolves with no wait on the syncer. The in-memory map is the read path; the file is write-through persistence. -Only default runtimes are prewarmed and linked. The first actor restored on -another enabled version a worker advertises pays that version's download. +Only default runtimes stay linked under the worker's directory. A +non-default version's assets are fetched when the syncer first checks the +version in that `atelet` process, but nothing links them while no actor runs +it; if cache eviction reclaims them, the next actor restored on that version +(or the first sync after an `atelet` restart) downloads them again. A sync +whose download fails writes the file without the version until a retry +succeeds, so after a restart a worker can stop advertising a version it held +before. ### The default runtime directories @@ -103,9 +112,9 @@ which directories exist. Nothing counts references and nothing prunes. - The hand-over comes after the build on purpose. A build that fails, as a permanent `runsc` probe failure does, leaves the file and so the directories as they were, with the previous default still linked and still - advertised. The prewarm starts late by only a little: a gVisor default's - release is fetched by the build's probe, so the prewarm is links, and a - micro-VM build takes milliseconds. + advertised. Unless an evicted gVisor version was promoted to default, the + build has already fetched the default's assets into the cache, so the + prewarm is just hard links. - The prewarm fills what exists and creates nothing. It checks that its directory is there, stages into it, and on an error checks again: a directory gone, after a later sync or the pod's removal, means the item @@ -113,24 +122,23 @@ which directories exist. Nothing counts references and nothing prunes. anything. A directory the syncer already filled costs a stat per asset, so a re-sync with an unchanged default downloads nothing and leaves the links alone. -- A failed download retries with backoff and gives up after eight attempts. +- A failed prewarm retries with backoff and gives up after eight attempts. The directory stays, empty or partial, until the next sync; an actor - placed on that default meanwhile fetches on demand and shares the - download. + placed on that default meanwhile stages on demand. - A pod's removal removes its whole directory, links included. After a restart the directories of pods still on the node keep protecting their defaults with no wait on the syncer, and the startup sweep removes the directories of pods that are gone. -The old default is not kept while the new one downloads: by then the file -names the new default and the worker has registered it, so nothing new is -placed on the old one. Actors already running on it hold their own links, -and restores of snapshots taken on it go through the on-demand path like -every non-default version. Several pods of one pool each hold their own -links to the same entries; the entry becomes evictable when the last of -them goes. One queue and one worker drain the prewarm items in order, so -defaults download one at a time across the node and never compete for its -bandwidth. +The old default's directory is removed as soon as the file names the new +default: by then the worker has registered the new default (and unless an +evicted gVisor version was promoted to default, the build has already fetched +it into the cache), so nothing new is placed on the old one. Actors already +running on it hold their own links, and restores of snapshots taken on it go +through the on-demand path like every non-default version. Several pods of +one pool each hold their own links to the same entries; the entry becomes +evictable when the last of them goes. One queue and one worker drain the +prewarm items in order. ## Eviction diff --git a/docs/security.md b/docs/security.md index 1287df7fc5..675d511ab8 100644 --- a/docs/security.md +++ b/docs/security.md @@ -40,7 +40,7 @@ Substrate is designed to safely run mutually hostile, untrusted actors from disp | Principle | Description | Status | | :--- | :--- | :--- | -| **Always Sandboxed** | Every actor is required to run inside a strong compute sandbox. A linux container is not sufficient or supported. Our security design relies on strong containment. | Met (gVisor, CHV micro-VM) | +| **Always Sandboxed** | Every actor is required to run inside a strong compute sandbox. A linux container is not sufficient or supported. Our security design relies on strong containment. | Met (gVisor, CHV microVM) | | **Per-Actor Storage Segregation** | Storage and persistent volume state must be strictly segregated per actor. Actors cannot traverse, enumerate, or access persistence belonging to other actors. Restoring a snapshot must re-validate the actor's current tenant authorization before execution resumes. | Met (Segregation)<br>Future (Per-Actor Encryption, Restore Authorization) | | **Per-Actor Network Isolation** | Guest network stacks are segregated per actor. Lateral peer-to-peer network communication between co-located actors on the same host or across the cluster is blocked by default; inter-actor communication must route through authenticated, policy-enforced gateways. | Met (Private netns, atunnel) | | **Delete Guarantee** | Reset, scrub, and sanitize local ephemeral volumes, cached state, and worker runtime environments between actor scheduling cycles to prevent cross-tenant data leakage. | In Progress (atelet resetActorDirs) | diff --git a/hack/microvm-assets/README.md b/hack/microvm-assets/README.md index 61dccd116e..1436ef4e68 100644 --- a/hack/microvm-assets/README.md +++ b/hack/microvm-assets/README.md @@ -1,4 +1,4 @@ -# Micro-VM runtime assets + counter demo (kind, fetch-not-bake) +# MicroVM runtime assets + counter demo (kind, fetch-not-bake) The `microvm` runtime (`cmd/ateworker-microvm`, kata + cloud-hypervisor) fetches its toolchain at runtime — nothing kata-specific is baked into the worker image. ateworker drives @@ -15,7 +15,7 @@ set. The asset set is four files: These helpers assemble the asset set for your node arch, stage it into the cluster's rustfs S3 bucket, and the demo manifest's `SandboxConfig` points at it. When `/dev/kvm` is available, `hack/create-kind-cluster.sh` mounts it into the node; atelet then advertises -it as a device, which is what places micro-VM workers there. +it as a device, which is what places microVM workers there. > [!TIP] > `hack/run-microvm-demo.sh` automates the full bring-up below (assets, control plane, diff --git a/hack/microvm-assets/assemble.sh b/hack/microvm-assets/assemble.sh index 68483b4de8..fc1e09f095 100755 --- a/hack/microvm-assets/assemble.sh +++ b/hack/microvm-assets/assemble.sh @@ -27,12 +27,12 @@ # ateworker drives the kata-agent directly (the kata containerd shim is NOT an asset). The # actor rootfs is overlay(virtio-fs RO lower + guest-tmpfs upper), so virtiofsd IS an # asset. CH's restore handshake hangs against virtiofsd v1.13.3, which kata bundled up -# to and including 4.0.0; kata 4.1.0 bundles v1.14.0, the first release carrying the +# to and including 4.0.0; kata 4.1.0 and later bundle v1.14.0, the first release carrying the # vhost-0.16 / vhost-user-backend-0.22 snapshot-restore fix (REPLY_ACK). So virtiofsd # now comes out of kata-static with the kernel and rootfs instead of being sourced # separately per arch. # -# Env: ARCH (arm64|amd64, default arm64), KATA_VER (4.1.0), CH_VER (v53.0), +# Env: ARCH (arm64|amd64, default arm64), KATA_VER (4.3.1), CH_VER (v53.0), # OUT (default ./bin/microvm-assets/$ARCH, under the gitignored bin/). # # Always re-downloads and overwrites — there is no incremental mode. It clears @@ -47,7 +47,7 @@ set -o errexit -o nounset -o pipefail ROOT="$(git rev-parse --show-toplevel)" ARCH="${ARCH:-arm64}" -KATA_VER="${KATA_VER:-4.1.0}" +KATA_VER="${KATA_VER:-4.3.1}" CH_VER="${CH_VER:-v53.0}" # Not env-overridable: this is whatever KATA_VER bundles. It is declared rather than # read off the binary because --print-stamp has to answer before anything is diff --git a/hack/microvm-assets/stage-to-rustfs.sh b/hack/microvm-assets/stage-to-rustfs.sh index 2f8c66f7b2..5fbd9c1e8b 100755 --- a/hack/microvm-assets/stage-to-rustfs.sh +++ b/hack/microvm-assets/stage-to-rustfs.sh @@ -49,7 +49,7 @@ KIND_CLUSTER_NAME="${KIND_CLUSTER_NAME:-kind}" # Keep in sync with the rustfs-bucket-init Job in # manifests/ate-install/kind/rustfs.yaml, which creates the bucket we upload into. -AWS_CLI_IMAGE="amazon/aws-cli:2.17.0@sha256:643507c10ada7964ca6157b3d799f030b90577643da9955d319a77399ed80d73" +AWS_CLI_IMAGE="amazon/aws-cli:2.38.5@sha256:c8156a0e74e13929a42d7e60f0d4e14e7d28d980cd21822505d4bff80e5460c1" ASSETS=(cloud-hypervisor virtiofsd vmlinux rootfs.img) diff --git a/hack/release/draft-release.sh b/hack/release/draft-release.sh index 0dc643403d..7980da2dd2 100755 --- a/hack/release/draft-release.sh +++ b/hack/release/draft-release.sh @@ -16,7 +16,7 @@ # Creates a DRAFT GitHub release for an existing tag, with notes generated # from .github/release.yml plus a committer list. A maintainer edits and -# publishes the draft (see docs/dev/release-notes.md). +# publishes the draft (see docs/dev/releasing.md). # # hack/release/draft-release.sh [--dry-run] <tag> # @@ -33,9 +33,13 @@ set -o errexit -o nounset -o pipefail ROOT="$(git rev-parse --show-toplevel)" -REPO="${GITHUB_REPOSITORY:-agent-substrate/substrate}" +UPSTREAM_REPO='agent-substrate/substrate' +REPO="${GITHUB_REPOSITORY:-${UPSTREAM_REPO}}" SEMVER='^v[0-9]+\.[0-9]+\.[0-9]+$' SEMVER_PRE='^v[0-9]+\.[0-9]+\.[0-9]+-[0-9A-Za-z.-]+$' +# Where the upstream release workflow publishes images (KO_DOCKER_REPO in +# .github/workflows/release.yaml). +RELEASE_IMAGE_REPO='ghcr.io/agent-substrate/substrate' usage() { echo "usage: $0 [--dry-run] <tag>" >&2 @@ -142,6 +146,26 @@ if ! notes="$(api "generating notes" "repos/${REPO}/releases/generate-notes" \ exit 1 fi +# Only the upstream release workflow publishes images (see +# .github/workflows/release.yaml), so only its notes say where they are. The +# notes give no install command because the release images are not a complete +# install; docs/dev/releasing.md lists what they cover. +if [[ "${REPO}" == "${UPSTREAM_REPO}" ]]; then + notes+=$'\n\n## Images\n\n' + notes+="Container images for this release are published as \`${RELEASE_IMAGE_REPO}/<image>:${tag}\`" + notes+=" for \`linux/amd64\` and \`linux/arm64\`. The images a release publishes are listed in" + notes+=" [Release images](https://github.com/${REPO}/blob/${tag}/docs/dev/releasing.md#release-images)." + # The workflow attaches these after creating the draft; see the + # "Attach kubectl-ate" step in .github/workflows/release.yaml. + notes+=$'\n\n## kubectl-ate\n\n' + notes+="\`kubectl-ate\` is attached for Linux and macOS on amd64 and arm64." + notes+=" Download the binary for your platform and its \`.sha256\` file, check it, and put it on your \`PATH\` as \`kubectl-ate\`:" + notes+=$'\n\n```sh\n' + notes+=$'shasum -a 256 -c kubectl-ate-linux-amd64.sha256\n' + notes+=$'install -m 0755 kubectl-ate-linux-amd64 ~/.local/bin/kubectl-ate\n' + notes+='```' +fi + # GitHub only lists first-time contributors; list everyone, as v0.1.0 did. # get-contributors.sh reads the range from the local clone, so both ends must # be present locally (the release workflow checks out full history). diff --git a/hack/setup-csi-nfs-kind.sh b/hack/setup-csi-nfs-kind.sh index 027bd9ac2f..37828e5c87 100755 --- a/hack/setup-csi-nfs-kind.sh +++ b/hack/setup-csi-nfs-kind.sh @@ -173,7 +173,7 @@ spec: - $patch: delete name: socat - name: envoy - image: envoyproxy/envoy:v1.34-latest + image: envoyproxy/envoy:v1.39.5@sha256:8fc594a5d85eb76dca24155cfb9208882406873932b20cf33649a0a611194c43 command: - /usr/local/bin/envoy args: diff --git a/hack/setup-e2e-egress-tls-kind.sh b/hack/setup-e2e-egress-tls-kind.sh index ebfc6906d4..adb6e0cbd4 100755 --- a/hack/setup-e2e-egress-tls-kind.sh +++ b/hack/setup-e2e-egress-tls-kind.sh @@ -42,7 +42,7 @@ fi # this E2E test setup; production keeps the gateway's normal root behavior. root_image="${image}" if [[ "${gateway}" == agentgateway ]]; then - root_image="envoyproxy/envoy:v1.39.4@sha256:74bd4eb5629708d3fadca1a5e6b85d208f338d7bb54e06b69565905c5294cc01" + root_image="envoyproxy/envoy:v1.39.5@sha256:8fc594a5d85eb76dca24155cfb9208882406873932b20cf33649a0a611194c43" fi # Use the helper image's public roots on every Pod start. The separate init diff --git a/hack/third_party/csi-driver-host-path/deploy/csi-hostpath-testing.yaml b/hack/third_party/csi-driver-host-path/deploy/csi-hostpath-testing.yaml index 5a72d350c9..2862b5b93d 100644 --- a/hack/third_party/csi-driver-host-path/deploy/csi-hostpath-testing.yaml +++ b/hack/third_party/csi-driver-host-path/deploy/csi-hostpath-testing.yaml @@ -155,7 +155,7 @@ spec: topologyKey: kubernetes.io/hostname containers: - name: envoy - image: envoyproxy/envoy:v1.34-latest + image: envoyproxy/envoy:v1.39.5@sha256:8fc594a5d85eb76dca24155cfb9208882406873932b20cf33649a0a611194c43 command: - "/usr/local/bin/envoy" args: diff --git a/hack/third_party/kubernetes/verify-shellcheck.sh b/hack/third_party/kubernetes/verify-shellcheck.sh index 7c1250c0de..e516cef9d4 100755 --- a/hack/third_party/kubernetes/verify-shellcheck.sh +++ b/hack/third_party/kubernetes/verify-shellcheck.sh @@ -61,7 +61,7 @@ if [[ "$#" == 0 ]]; then done < <(find . -name '*.sh' \ -not \( \ -path './_*' -o \ - -path './.git*' -o \ + -path './.git/*' -o \ -path '*/vendor/*' -o \ -path '*/third_party/*' \ \)) diff --git a/hack/verify/postgresql-migrations.sh b/hack/verify/postgresql-migrations.sh index 4894cf0e61..0f4759ee9a 100755 --- a/hack/verify/postgresql-migrations.sh +++ b/hack/verify/postgresql-migrations.sh @@ -24,7 +24,7 @@ export LC_ALL=C MIGRATIONS_DIR="cmd/ateapi/internal/store/atepg/migrations" if (( $# > 1 )); then - echo "Usage: $0 [released-ref]" >&2 + echo "Usage: $0 [base-ref]" >&2 exit 2 fi @@ -74,21 +74,14 @@ for migration in "${migrations[@]}"; do expected=$((expected + 1)) done -released_ref="${1:-}" -if [[ -z "${released_ref}" ]]; then - while read -r tag; do - if [[ ! "${tag}" =~ ^v[1-9][0-9]*\.[0-9]+\.[0-9]+$ ]]; then - continue - fi - if [[ "$(git ls-tree -r --name-only "${tag}" -- "${MIGRATIONS_DIR}")" == *".sql"* ]]; then - released_ref="${tag}" - break - fi - done < <(git tag --merged HEAD --sort=-version:refname) +# Without a base ref, as from verify-all.sh, only the file rules above are checked. +if (( $# == 0 )); then + exit 0 fi -if [[ -n "${released_ref}" ]] && ! git diff --quiet --no-renames --diff-filter=MD "${released_ref}" HEAD -- "${MIGRATIONS_DIR}"; then - echo "Do not change or delete released PostgreSQL migrations." >&2 - git diff --name-only --no-renames --diff-filter=MD "${released_ref}" HEAD -- "${MIGRATIONS_DIR}" >&2 +base="$(git merge-base HEAD "$1")" +if ! git diff --quiet --no-renames --diff-filter=MD "${base}" HEAD -- "${MIGRATIONS_DIR}"; then + echo "Do not change or delete merged PostgreSQL migrations." >&2 + git diff --name-only --no-renames --diff-filter=MD "${base}" HEAD -- "${MIGRATIONS_DIR}" >&2 exit 1 fi diff --git a/internal/ateworker/cgroup/actor.go b/internal/ateworker/cgroup/actor.go index 54e4b29f32..c71bdce014 100644 --- a/internal/ateworker/cgroup/actor.go +++ b/internal/ateworker/cgroup/actor.go @@ -87,21 +87,14 @@ func cpuMax(milliCPU int64) string { return fmt.Sprintf("%d %d", max(milliCPU*cpuPeriodUS/1000, cpuMinQuotaUS), cpuPeriodUS) } -// SysProcAttr starts a process directly inside the leaf. A nil leaf starts it -// where the caller is. +// SysProcAttr starts a process directly inside the leaf. func (l *ActorLeaf) SysProcAttr() *syscall.SysProcAttr { - if l == nil { - return nil - } return &syscall.SysProcAttr{UseCgroupFD: true, CgroupFD: int(l.dir.Fd())} } // Close releases the leaf's handle. The leaf itself stays until // RemoveActorLeaf. func (l *ActorLeaf) Close() error { - if l == nil { - return nil - } return l.dir.Close() } diff --git a/internal/ateworker/cgroup/actor_linux_test.go b/internal/ateworker/cgroup/actor_linux_test.go index a38ce75144..1e3f38a2ee 100644 --- a/internal/ateworker/cgroup/actor_linux_test.go +++ b/internal/ateworker/cgroup/actor_linux_test.go @@ -92,15 +92,9 @@ func TestKillActorLeaf(t *testing.T) { // ownCgroup is the test process's own cgroup, under which it may create leaves. func ownCgroup(t *testing.T) string { t.Helper() - b, err := os.ReadFile("/proc/self/cgroup") + p, err := cgroupPath("/proc/self/cgroup") if err != nil { - t.Fatal(err) - } - for _, line := range strings.Split(strings.TrimSpace(string(b)), "\n") { - if p, ok := strings.CutPrefix(line, "0::"); ok { - return filepath.Join(Root, p) - } + t.Skip(err) } - t.Skip("no cgroup v2") - return "" + return filepath.Join(Root, p) } diff --git a/internal/ateworker/cgroup/actor_test.go b/internal/ateworker/cgroup/actor_test.go index 05e5fa2dae..19aa017772 100644 --- a/internal/ateworker/cgroup/actor_test.go +++ b/internal/ateworker/cgroup/actor_test.go @@ -95,13 +95,3 @@ func TestActorLeafRejectsUnsafeNames(t *testing.T) { } } } - -func TestNilActorLeaf(t *testing.T) { - var leaf *ActorLeaf - if leaf.SysProcAttr() != nil { - t.Error("a nil leaf set SysProcAttr") - } - if err := leaf.Close(); err != nil { - t.Error(err) - } -} diff --git a/internal/ateworker/cgroup/cgroup.go b/internal/ateworker/cgroup/cgroup.go index 776856c394..44f076ae5f 100644 --- a/internal/ateworker/cgroup/cgroup.go +++ b/internal/ateworker/cgroup/cgroup.go @@ -36,8 +36,9 @@ const Root = "/sys/fs/cgroup" const workerLeaf = "worker" // Delegate prepares the worker pod's cgroup so the runtime can create per-actor -// leaves under it with real cpu/memory/pids accounting. It reports whether it -// did: a worker outside a private cgroup namespace is left alone. +// leaves under it with real cpu/memory/pids accounting. A worker outside a +// private cgroup namespace fails here rather than running actors without their +// cgroups. // // The unprivileged worker runs in a private cgroup namespace, so /sys/fs/cgroup // is the pod's own cgroup scope rather than the host root. Two things must be @@ -54,21 +55,20 @@ const workerLeaf = "worker" // worker holds CAP_SYS_ADMIN with no user namespace, so the ro flag is not // locked: clear it and leave it writable (the runtime writes here on every // create/restore). -func Delegate(ctx context.Context) (bool, error) { - const leaf = Root + "/" + workerLeaf +func Delegate(ctx context.Context) error { + return delegate(ctx, "/proc/self/cgroup", Root) +} + +func delegate(ctx context.Context, procSelfCgroup, root string) error { + leaf := root + "/" + workerLeaf - // Delegation only makes sense inside a private cgroup namespace, where - // /sys/fs/cgroup is the pod's own scope. A privileged worker instead inherits - // the host cgroup namespace, so /sys/fs/cgroup is the true host root: it holds - // unmovable kernel threads (cgroup.procs would never drain) and must not be - // carved up. Detect the namespace via /proc/self/cgroup, which reads "0::/" - // only at a cgroup-namespace root, and skip delegation otherwise (the runtime - // then falls back to its own cgroup handling). - if private, err := inPrivateCgroupNamespace(); err != nil { - return false, fmt.Errorf("while detecting cgroup namespace: %w", err) - } else if !private { - slog.InfoContext(ctx, "not in a private cgroup namespace; skipping cgroup delegation (worker is likely privileged)") - return false, nil + // Outside a private cgroup namespace, as in a privileged pod, /sys/fs/cgroup + // is the host's root: it holds unmovable kernel threads and must not be + // carved up. /proc/self/cgroup reads "0::/" only at a namespace root. + if path, err := cgroupPath(procSelfCgroup); err != nil { + return fmt.Errorf("while detecting cgroup namespace: %w", err) + } else if path != "/" { + return fmt.Errorf("worker needs a private cgroup namespace (cgroup path %q, want /): workers must run unprivileged", path) } if err := os.Mkdir(leaf, 0o755); err != nil && !os.IsExist(err) { @@ -77,59 +77,58 @@ func Delegate(ctx context.Context) (bool, error) { // that permits mount. The gVisor worker runs AppArmor-unconfined, which // runsc's own mounts require anyway; on nodes that do enforce the default // profile (GKE COS) this mount is otherwise denied with EPERM. - if err := unix.Mount("none", Root, "", unix.MS_BIND|unix.MS_REMOUNT, ""); err != nil { - return false, fmt.Errorf("while remounting %q read-write: %w", Root, err) + if err := unix.Mount("none", root, "", unix.MS_BIND|unix.MS_REMOUNT, ""); err != nil { + return fmt.Errorf("while remounting %q read-write: %w", root, err) } if err := os.Mkdir(leaf, 0o755); err != nil && !os.IsExist(err) { - return false, fmt.Errorf("while creating cgroup leaf %q: %w", leaf, err) + return fmt.Errorf("while creating cgroup leaf %q: %w", leaf, err) } } - if err := moveProcs(ctx, Root+"/cgroup.procs", leaf+"/cgroup.procs"); err != nil { - return false, fmt.Errorf("while moving worker processes into %q: %w", leaf, err) + if err := moveProcs(ctx, root+"/cgroup.procs", leaf+"/cgroup.procs"); err != nil { + return fmt.Errorf("while moving worker processes into %q: %w", leaf, err) } - avail, err := os.ReadFile(Root + "/cgroup.controllers") + avail, err := os.ReadFile(root + "/cgroup.controllers") if err != nil { - return false, fmt.Errorf("while reading available cgroup controllers: %w", err) + return fmt.Errorf("while reading available cgroup controllers: %w", err) } // Enable controllers one at a time so a single controller the node cannot // delegate (for example cpuset without an assigned cpu set) does not prevent // the others from being enabled. var enabled []string for _, c := range strings.Fields(string(avail)) { - if err := os.WriteFile(Root+"/cgroup.subtree_control", []byte("+"+c), 0o644); err != nil { + if err := os.WriteFile(root+"/cgroup.subtree_control", []byte("+"+c), 0o644); err != nil { slog.WarnContext(ctx, "could not enable cgroup controller for delegation", slog.String("controller", c), slog.Any("err", err)) continue } enabled = append(enabled, c) } slog.InfoContext(ctx, "cgroup delegation ready", slog.Any("controllers", enabled)) - return true, nil + return nil } -// inPrivateCgroupNamespace reports whether the process sits at the root of its -// own cgroup namespace. The cgroup v2 line of /proc/self/cgroup ("0::<path>") -// reports the path relative to the namespace root, so it reads exactly "/" only -// when /sys/fs/cgroup is the namespace's own (pod-scoped) cgroup. A privileged +// cgroupPath returns the cgroup v2 path from a /proc/<pid>/cgroup file +// ("0::<path>"). It is relative to the namespace root, so it reads exactly "/" +// only when /sys/fs/cgroup is the namespace's own (pod-scoped) cgroup. A privileged // worker inheriting the host cgroup namespace instead sees its full host path // (for example "/kubepods.slice/.../cri-containerd-<id>.scope"). -func inPrivateCgroupNamespace() (bool, error) { - b, err := os.ReadFile("/proc/self/cgroup") +func cgroupPath(procCgroup string) (string, error) { + b, err := os.ReadFile(procCgroup) if err != nil { - return false, fmt.Errorf("while reading /proc/self/cgroup: %w", err) + return "", fmt.Errorf("while reading %q: %w", procCgroup, err) } - return atNamespaceRoot(string(b)) + return parseCgroupPath(string(b)) } -// atNamespaceRoot parses the contents of /proc/self/cgroup. -func atNamespaceRoot(procSelfCgroup string) (bool, error) { +// parseCgroupPath parses the contents of /proc/self/cgroup. +func parseCgroupPath(procSelfCgroup string) (string, error) { for _, line := range strings.Split(strings.TrimSpace(procSelfCgroup), "\n") { if path, ok := strings.CutPrefix(line, "0::"); ok { - return path == "/", nil + return path, nil } } - return false, fmt.Errorf("no cgroup v2 (0::) entry in /proc/self/cgroup") + return "", fmt.Errorf("no cgroup v2 (0::) entry in /proc/self/cgroup") } // moveProcs relocates every process listed in srcProcs into dstProcs. cgroup.procs diff --git a/internal/ateworker/cgroup/cgroup_test.go b/internal/ateworker/cgroup/cgroup_test.go index a5fbb791bc..0cf0b9ddd6 100644 --- a/internal/ateworker/cgroup/cgroup_test.go +++ b/internal/ateworker/cgroup/cgroup_test.go @@ -16,28 +16,51 @@ package cgroup -import "testing" +import ( + "context" + "os" + "path/filepath" + "testing" +) -func TestAtNamespaceRoot(t *testing.T) { +func TestParseCgroupPath(t *testing.T) { for _, tc := range []struct { name string in string - want bool + want string wantErr bool }{ - {name: "private namespace", in: "0::/\n", want: true}, - {name: "host namespace", in: "0::/kubepods.slice/kubepods-pod1.slice/cri-containerd-abc.scope\n", want: false}, - {name: "hybrid hierarchy", in: "12:pids:/\n0::/\n", want: true}, + {name: "private namespace", in: "0::/\n", want: "/"}, + {name: "host namespace", in: "0::/kubepods.slice/kubepods-pod1.slice/cri-containerd-abc.scope\n", want: "/kubepods.slice/kubepods-pod1.slice/cri-containerd-abc.scope"}, + {name: "hybrid hierarchy", in: "12:pids:/\n0::/\n", want: "/"}, {name: "cgroup v1 only", in: "12:pids:/\n11:memory:/\n", wantErr: true}, } { t.Run(tc.name, func(t *testing.T) { - got, err := atNamespaceRoot(tc.in) + got, err := parseCgroupPath(tc.in) if (err != nil) != tc.wantErr { - t.Fatalf("atNamespaceRoot() error = %v, wantErr %v", err, tc.wantErr) + t.Fatalf("parseCgroupPath() error = %v, wantErr %v", err, tc.wantErr) } if got != tc.want { - t.Errorf("atNamespaceRoot() = %v, want %v", got, tc.want) + t.Errorf("parseCgroupPath() = %q, want %q", got, tc.want) } }) } } + +func TestDelegateRejectsSharedNamespace(t *testing.T) { + dir := t.TempDir() + procCgroup := filepath.Join(dir, "cgroup") + if err := os.WriteFile(procCgroup, []byte("0::/kubepods.slice/kubepods-pod1.slice/cri-containerd-abc.scope\n"), 0o600); err != nil { + t.Fatal(err) + } + root := filepath.Join(dir, "root") + if err := os.Mkdir(root, 0o755); err != nil { + t.Fatal(err) + } + if err := delegate(context.Background(), procCgroup, root); err == nil { + t.Fatal("delegate() succeeded outside a private cgroup namespace") + } + if _, err := os.Stat(filepath.Join(root, workerLeaf)); !os.IsNotExist(err) { + t.Errorf("delegate() created the worker leaf anyway (stat err: %v)", err) + } +} diff --git a/internal/ateworker/cgroup/memwatch.go b/internal/ateworker/cgroup/memwatch.go new file mode 100644 index 0000000000..15caed14aa --- /dev/null +++ b/internal/ateworker/cgroup/memwatch.go @@ -0,0 +1,179 @@ +//go:build linux + +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package cgroup + +import ( + "bytes" + "fmt" + "io/fs" + "os" + "path/filepath" + "strconv" + "strings" + "time" +) + +const ( + // memoryStallWindow is how far back a cgroup's memory stalls are counted, + // and memoryStallKill the share of it it may spend stalled before it is + // stuck. Throttling at memory.high counts as a stall. Short, because a + // throttled cgroup still creeps up, by ~128KiB/s per allocating thread. + memoryStallWindow = 2 * time.Second + memoryStallKill = 0.25 + // memoryStallMinWindow is the least it is watched before it is stuck. + memoryStallMinWindow = time.Second + // memoryOvershootMax is the most a cgroup may go over its limit at all, so + // a few of them can't use up what their parent keeps in reserve. + memoryOvershootMax = 16 << 20 +) + +// MemoryWatch decides when a cgroup with memory.high at its limit is stuck +// there and should be killed, from how much of the recent past it spent +// stalled on memory. Check it on a timer, one MemoryWatch per cgroup. +type MemoryWatch struct { + samples []stallSample +} + +type stallSample struct { + at time.Time + stall uint64 // memory.pressure some total, in microseconds + high uint64 // memory.events.local high +} + +// Check reports whether the cgroup in dir is stuck at limit, along with its +// memory.current: stalled on memory for at least memoryStallKill of the last +// memoryStallWindow while hitting its own memory.high with at most 1/16 of the +// limit in reclaimable page cache, or over the limit by more than 1/16 of it +// or memoryOvershootMax at all. +// +// Stalls rather than memory.current over the limit, because reclaim keeps +// pulling a throttled cgroup back under memory.high between allocations, and +// at only a little over it a cgroup stalls about half the time. "some" rather +// than "full", because busy threads alongside a throttled one hide a full +// stall. Its own memory.high events, so stalls from node-wide reclaim don't +// count. +func (w *MemoryWatch) Check(dir string, limit uint64, now time.Time) (bool, uint64, error) { + current, err := memoryCurrent(dir) + if err != nil { + return false, 0, err + } + if current > limit+min(limit/16, memoryOvershootMax) { + return true, current, nil + } + if current < limit-limit/16 { + w.samples = w.samples[:0] + return false, current, nil + } + cache, err := fileCache(dir) + if err != nil { + return false, current, err + } + if current-min(cache, current) < limit-limit/16 { + w.samples = w.samples[:0] + return false, current, nil + } + stall, err := memoryStall(dir) + if err != nil { + return false, current, err + } + high, err := memoryHighEvents(dir) + if err != nil { + return false, current, err + } + w.samples = append(w.samples, stallSample{at: now, stall: stall, high: high}) + // drop what fell out of the window, keeping one sample at or before its start + for len(w.samples) > 1 && now.Sub(w.samples[1].at) >= memoryStallWindow { + w.samples = w.samples[1:] + } + first := w.samples[0] + span := now.Sub(first.at) + if span < memoryStallMinWindow { + return false, current, nil + } + stalled := time.Duration(stall-first.stall) * time.Microsecond + return high > first.high && float64(stalled) >= memoryStallKill*float64(span), current, nil +} + +// memoryCurrent reads memory.current. A missing file is an error wrapping +// fs.ErrNotExist. +func memoryCurrent(dir string) (uint64, error) { + b, err := os.ReadFile(filepath.Join(dir, "memory.current")) + if err != nil { + return 0, err + } + return strconv.ParseUint(strings.TrimSpace(string(b)), 10, 64) +} + +// fileCache is memory.stat's active_file plus inactive_file: the page cache on +// the file LRU, which the kernel can reclaim. shmem sits on the anon LRU, so +// memfd-backed memory such as a gVisor sentry's is not counted. +func fileCache(dir string) (uint64, error) { + path := filepath.Join(dir, "memory.stat") + active, err := keyed(path, "active_file") + if err != nil { + return 0, err + } + inactive, err := keyed(path, "inactive_file") + if err != nil { + return 0, err + } + return active + inactive, nil +} + +// memoryStall is memory.pressure's some total: microseconds during which at +// least one task in the cgroup was stalled on memory. +func memoryStall(dir string) (uint64, error) { + path := filepath.Join(dir, "memory.pressure") + b, err := os.ReadFile(path) + if err != nil { + return 0, err + } + for line := range bytes.Lines(b) { + f := bytes.Fields(line) + if len(f) == 0 || string(f[0]) != "some" { + continue + } + for _, kv := range f[1:] { + if v, ok := bytes.CutPrefix(kv, []byte("total=")); ok { + return strconv.ParseUint(string(v), 10, 64) + } + } + } + return 0, fmt.Errorf("no some total in %q", path) +} + +// memoryHighEvents is memory.events.local's high: how many times the cgroup +// itself, not a descendant, was over memory.high and reclaimed. +func memoryHighEvents(dir string) (uint64, error) { + return keyed(filepath.Join(dir, "memory.events.local"), "high") +} + +// keyed reads one key from a flat keyed cgroup file. A key the file lacks is +// an error wrapping fs.ErrNotExist. +func keyed(path, key string) (uint64, error) { + b, err := os.ReadFile(path) + if err != nil { + return 0, err + } + for line := range bytes.Lines(b) { + f := bytes.Fields(line) + if len(f) == 2 && string(f[0]) == key { + return strconv.ParseUint(string(f[1]), 10, 64) + } + } + return 0, fmt.Errorf("no %s in %q: %w", key, path, fs.ErrNotExist) +} diff --git a/internal/ateworker/cgroup/memwatch_test.go b/internal/ateworker/cgroup/memwatch_test.go new file mode 100644 index 0000000000..e3252dca3b --- /dev/null +++ b/internal/ateworker/cgroup/memwatch_test.go @@ -0,0 +1,164 @@ +//go:build linux + +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package cgroup + +import ( + "fmt" + "os" + "path/filepath" + "testing" + "time" +) + +func writeCgroupFiles(t *testing.T, dir string, files map[string]string) { + t.Helper() + for name, content := range files { + if err := os.WriteFile(filepath.Join(dir, name), []byte(content), 0o600); err != nil { + t.Fatal(err) + } + } +} + +func TestMemoryWatch(t *testing.T) { + const limit = 256 << 20 + type step struct { + after time.Duration + current uint64 + cache uint64 + stallMs uint64 // memory.pressure some total so far + want bool + high uint64 // memory.events.local high, defaults to stallMs + } + for _, tc := range []struct { + name string + steps []step + }{ + { + name: "stuck at the limit", + steps: []step{ + {0, limit + 4<<20, 1 << 20, 0, false, 0}, + {500 * time.Millisecond, limit + 4<<20, 1 << 20, 400, false, 0}, + {time.Second, limit + 4<<20, 1 << 20, 900, true, 0}, + }, + }, + { + // reclaim pulls it under memory.high between allocations, which must + // not restart the clock + name: "stuck but dipping under the limit", + steps: []step{ + {0, limit + 4<<20, 0, 0, false, 0}, + {250 * time.Millisecond, limit - 1<<20, 0, 200, false, 0}, + {500 * time.Millisecond, limit + 4<<20, 0, 400, false, 0}, + {750 * time.Millisecond, limit - 1<<20, 0, 600, false, 0}, + {time.Second, limit + 4<<20, 0, 800, true, 0}, + }, + }, + { + name: "at the limit but making progress", + steps: []step{ + {0, limit, 0, 0, false, 0}, + {time.Second, limit, 0, 100, false, 0}, + {2 * time.Second, limit, 0, 300, false, 0}, + }, + }, + { + name: "over on page cache", + steps: []step{ + {0, limit + 4<<20, 64 << 20, 0, false, 0}, + {time.Second, limit + 4<<20, 64 << 20, 1000, false, 0}, + }, + }, + { + // node-wide reclaim stalls it without it hitting its own memory.high + name: "stalled without hitting its limit", + steps: []step{ + {0, limit, 0, 0, false, 1}, + {time.Second, limit, 0, 900, false, 1}, + }, + }, + { + name: "far over the limit is killed at once", + steps: []step{ + {0, limit + limit/16 + 1, 0, 0, true, 0}, + }, + }, + { + name: "stalls age out of the window", + steps: []step{ + {0, limit, 0, 0, false, 0}, + {time.Second, limit, 0, 900, true, 0}, + {3 * time.Second, limit, 0, 1100, false, 0}, + }, + }, + } { + t.Run(tc.name, func(t *testing.T) { + dir := t.TempDir() + start := time.Now() + var w MemoryWatch + for i, s := range tc.steps { + high := s.high + if high == 0 { + high = s.stallMs + } + writeCgroupFiles(t, dir, map[string]string{ + "memory.events.local": fmt.Sprintf("low 0\nhigh %d\nmax 0\noom 0\noom_kill 0\n", high), + "memory.current": fmt.Sprint(s.current), + "memory.stat": fmt.Sprintf("anon 1\ninactive_file %d\nactive_file 0\n", s.cache), + "memory.pressure": fmt.Sprintf("some avg10=0.00 avg60=0.00 avg300=0.00 total=%d\nfull avg10=0.00 avg60=0.00 avg300=0.00 total=0\n", s.stallMs*1000), + }) + got, current, err := w.Check(dir, limit, start.Add(s.after)) + if err != nil { + t.Fatal(err) + } + if got != s.want || current != s.current { + t.Errorf("step %d: Check() = %v, %d, want %v, %d", i, got, current, s.want, s.current) + } + } + }) + } +} + +// A big cgroup may go only memoryOvershootMax over its limit, not 1/16 of it. +func TestMemoryWatchCapsOvershoot(t *testing.T) { + const limit = 8 << 30 + for _, tc := range []struct { + current uint64 + want bool + }{ + {limit + memoryOvershootMax, false}, + {limit + memoryOvershootMax + 1, true}, + } { + dir := t.TempDir() + writeCgroupFiles(t, dir, map[string]string{ + "memory.current": fmt.Sprint(tc.current), + "memory.stat": "anon 1\ninactive_file 0\nactive_file 0\n", + "memory.pressure": "some avg10=0.00 avg60=0.00 avg300=0.00 total=0\nfull avg10=0.00 avg60=0.00 avg300=0.00 total=0\n", + "memory.events.local": "high 0\n", + }) + var w MemoryWatch + if got, _, err := w.Check(dir, limit, time.Now()); err != nil || got != tc.want { + t.Errorf("Check() at %d over = %v, %v, want %v", tc.current-limit, got, err, tc.want) + } + } +} + +func TestMemoryWatchMissingCgroup(t *testing.T) { + var w MemoryWatch + if _, _, err := w.Check(filepath.Join(t.TempDir(), "gone"), 1, time.Now()); err == nil { + t.Error("Check() on a missing cgroup succeeded") + } +} diff --git a/internal/ateworker/checkpoint.go b/internal/ateworker/checkpoint.go new file mode 100644 index 0000000000..98b4520e5a --- /dev/null +++ b/internal/ateworker/checkpoint.go @@ -0,0 +1,97 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package ateworker + +import ( + "encoding/json" + "os" + "path/filepath" + "slices" + + "go.ate.dev/substrate/internal/proto/ateworkerpb" +) + +// checkpointDoneFile is written into checkpointDir after all snapshot files are +// listed and the workload is torn down. It is wiped by os.RemoveAll(checkpointDir) +// on the next checkpoint or atelet's resetActorDirs. +const checkpointDoneFile = ".checkpoint-done.json" + +type checkpointRecord struct { + Fidelity ateworkerpb.SnapshotFidelity `json:"fidelity"` + SnapshotFiles []string `json:"snapshot_files,omitempty"` + DataSnapshotFiles []string `json:"data_snapshot_files,omitempty"` +} + +// RecordCheckpoint writes the completion marker into checkpointDir so a retry +// after a caller timeout or atelet restart can reuse the finished +// CheckpointWorkloadResponse. +func RecordCheckpoint(checkpointDir string, fidelity ateworkerpb.SnapshotFidelity, resp *ateworkerpb.CheckpointWorkloadResponse) error { + b, err := json.Marshal(checkpointRecord{ + Fidelity: fidelity, + SnapshotFiles: resp.GetSnapshotFiles(), + DataSnapshotFiles: resp.GetDataSnapshotFiles(), + }) + if err != nil { + return err + } + tmp := filepath.Join(checkpointDir, checkpointDoneFile+".tmp") + if err := os.WriteFile(tmp, b, 0o600); err != nil { + return err + } + return os.Rename(tmp, filepath.Join(checkpointDir, checkpointDoneFile)) +} + +// ReuseCheckpoint returns the recorded CheckpointWorkloadResponse from +// checkpointDir if the completion marker exists, its fidelity matches, and all +// listed snapshot files are still present on disk. +func ReuseCheckpoint(checkpointDir string, fidelity ateworkerpb.SnapshotFidelity) *ateworkerpb.CheckpointWorkloadResponse { + b, err := os.ReadFile(filepath.Join(checkpointDir, checkpointDoneFile)) + if err != nil { + return nil + } + var rec checkpointRecord + if err := json.Unmarshal(b, &rec); err != nil || rec.Fidelity != fidelity { + return nil + } + // DataSnapshotFiles is the VOLUMES subset of SnapshotFiles: the same file + // set on VOLUMES, and a strict subset on ROOTFS/MEMORY. + if fidelity == ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES { + if len(rec.SnapshotFiles) != len(rec.DataSnapshotFiles) { + return nil + } + for _, f := range rec.SnapshotFiles { + if !slices.Contains(rec.DataSnapshotFiles, f) { + return nil + } + } + } else if len(rec.SnapshotFiles) == 0 || len(rec.DataSnapshotFiles) >= len(rec.SnapshotFiles) { + return nil + } + for _, f := range rec.DataSnapshotFiles { + if !slices.Contains(rec.SnapshotFiles, f) { + return nil + } + } + for _, f := range rec.SnapshotFiles { + info, err := os.Stat(filepath.Join(checkpointDir, f)) + if err != nil || !info.Mode().IsRegular() { + return nil + } + } + return &ateworkerpb.CheckpointWorkloadResponse{ + SnapshotFiles: rec.SnapshotFiles, + DataSnapshotFiles: rec.DataSnapshotFiles, + } +} diff --git a/internal/ateworker/checkpoint_test.go b/internal/ateworker/checkpoint_test.go new file mode 100644 index 0000000000..8343de7a13 --- /dev/null +++ b/internal/ateworker/checkpoint_test.go @@ -0,0 +1,124 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package ateworker + +import ( + "os" + "path/filepath" + "slices" + "testing" + + "go.ate.dev/substrate/internal/proto/ateworkerpb" +) + +func TestRecordAndReuseCheckpoint(t *testing.T) { + checkpointDir := t.TempDir() + imgPath := filepath.Join(checkpointDir, "checkpoint.img") + if err := os.WriteFile(imgPath, []byte("snapshot-bytes"), 0o600); err != nil { + t.Fatal(err) + } + + // Without a completion marker (e.g. partial files from an interrupted attempt), + // ReuseCheckpoint returns nil. + if got := ReuseCheckpoint(checkpointDir, ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY); got != nil { + t.Fatalf("ReuseCheckpoint(before RecordCheckpoint) = %v, want nil", got) + } + + wantResp := &ateworkerpb.CheckpointWorkloadResponse{ + SnapshotFiles: []string{"checkpoint.img"}, + } + if err := RecordCheckpoint(checkpointDir, ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, wantResp); err != nil { + t.Fatalf("RecordCheckpoint: %v", err) + } + + // Reusing with matching fidelity and intact file succeeds. + got := ReuseCheckpoint(checkpointDir, ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY) + if got == nil || !slices.Equal(got.GetSnapshotFiles(), wantResp.GetSnapshotFiles()) { + t.Fatalf("ReuseCheckpoint() = %v, want %v", got, wantResp) + } + + // Mismatched fidelity is not reused. + if got := ReuseCheckpoint(checkpointDir, ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES); got != nil { + t.Fatalf("ReuseCheckpoint(different fidelity) = %v, want nil", got) + } + + // Once atelet moves/removes the file from checkpoint-state/, the marker is not reused. + if err := os.Remove(imgPath); err != nil { + t.Fatal(err) + } + if got := ReuseCheckpoint(checkpointDir, ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY); got != nil { + t.Fatalf("ReuseCheckpoint(after file removed) = %v, want nil", got) + } +} + +func TestReuseCheckpointDataSnapshotFilesByFidelity(t *testing.T) { + checkpointDir := t.TempDir() + fsDir := filepath.Join(checkpointDir, "fs") + if err := os.MkdirAll(fsDir, 0o700); err != nil { + t.Fatal(err) + } + for _, rel := range []string{"checkpoint.img", "fs/fscheckpoint.pb", "fs/multitar.img"} { + if err := os.WriteFile(filepath.Join(checkpointDir, rel), []byte("data"), 0o600); err != nil { + t.Fatal(err) + } + } + + // MEMORY fidelity with DataSnapshotFiles subset succeeds when all files exist. + memResp := &ateworkerpb.CheckpointWorkloadResponse{ + SnapshotFiles: []string{"checkpoint.img", "fs/fscheckpoint.pb", "fs/multitar.img"}, + DataSnapshotFiles: []string{"fs/fscheckpoint.pb", "fs/multitar.img"}, + } + if err := RecordCheckpoint(checkpointDir, ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, memResp); err != nil { + t.Fatalf("RecordCheckpoint(MEMORY): %v", err) + } + got := ReuseCheckpoint(checkpointDir, ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY) + if got == nil || !slices.Equal(got.GetSnapshotFiles(), memResp.GetSnapshotFiles()) || !slices.Equal(got.GetDataSnapshotFiles(), memResp.GetDataSnapshotFiles()) { + t.Fatalf("ReuseCheckpoint(MEMORY) = %v, want %v", got, memResp) + } + + // MEMORY fidelity rejects DataSnapshotFiles not present in SnapshotFiles. + badSubset := &ateworkerpb.CheckpointWorkloadResponse{ + SnapshotFiles: []string{"checkpoint.img"}, + DataSnapshotFiles: []string{"fs/fscheckpoint.pb"}, + } + if err := RecordCheckpoint(checkpointDir, ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, badSubset); err != nil { + t.Fatalf("RecordCheckpoint(badSubset): %v", err) + } + if got := ReuseCheckpoint(checkpointDir, ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY); got != nil { + t.Fatalf("ReuseCheckpoint(data file not in SnapshotFiles) = %v, want nil", got) + } + + // VOLUMES fidelity succeeds when SnapshotFiles and DataSnapshotFiles contain + // the same set of files, even in different order. + volResp := &ateworkerpb.CheckpointWorkloadResponse{ + SnapshotFiles: []string{"fs/multitar.img", "fs/fscheckpoint.pb"}, + DataSnapshotFiles: []string{"fs/fscheckpoint.pb", "fs/multitar.img"}, + } + if err := RecordCheckpoint(checkpointDir, ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, volResp); err != nil { + t.Fatalf("RecordCheckpoint(VOLUMES): %v", err) + } + got = ReuseCheckpoint(checkpointDir, ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES) + if got == nil || !slices.Equal(got.GetSnapshotFiles(), volResp.GetSnapshotFiles()) || !slices.Equal(got.GetDataSnapshotFiles(), volResp.GetDataSnapshotFiles()) { + t.Fatalf("ReuseCheckpoint(VOLUMES) = %v, want %v", got, volResp) + } + + // VOLUMES fidelity rejects non-data files in SnapshotFiles. + if err := RecordCheckpoint(checkpointDir, ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, memResp); err != nil { + t.Fatalf("RecordCheckpoint(VOLUMES with non-data files): %v", err) + } + if got := ReuseCheckpoint(checkpointDir, ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES); got != nil { + t.Fatalf("ReuseCheckpoint(VOLUMES with non-data files) = %v, want nil", got) + } +} diff --git a/internal/benchmarking/boomer/ateenv/lifecycle.go b/internal/benchmarking/boomer/ateenv/lifecycle.go index 3073a51058..af631c3063 100644 --- a/internal/benchmarking/boomer/ateenv/lifecycle.go +++ b/internal/benchmarking/boomer/ateenv/lifecycle.go @@ -44,7 +44,7 @@ import ( "go.opentelemetry.io/otel/attribute" "google.golang.org/grpc" "google.golang.org/grpc/codes" - "google.golang.org/grpc/credentials/insecure" + "google.golang.org/grpc/credentials" "google.golang.org/grpc/metadata" "google.golang.org/grpc/status" ) @@ -104,20 +104,20 @@ func init() { }) } -// routerGRPCTarget converts an HTTP/HTTPS router URL into a dns:/// gRPC dial target. +// routerGRPCTarget converts an HTTPS router URL into a dns:/// gRPC dial target. func routerGRPCTarget(rawURL string) (string, error) { if rawURL == "" { - rawURL = "http://atenet-router.ate-system.svc.cluster.local" + rawURL = boomerutil.DefaultRouterURL } if !strings.Contains(rawURL, "://") { - rawURL = "http://" + rawURL + rawURL = "https://" + rawURL } u, err := url.Parse(rawURL) if err != nil { return "", err } - if strings.EqualFold(u.Scheme, "https") { - return "", fmt.Errorf("router URL %q uses https, which is not supported by the plaintext gRPC dialer", rawURL) + if !strings.EqualFold(u.Scheme, "https") { + return "", fmt.Errorf("router URL %q uses %s, want https", rawURL, u.Scheme) } host := u.Hostname() if host == "" { @@ -125,7 +125,7 @@ func routerGRPCTarget(rawURL string) (string, error) { } port := u.Port() if port == "" { - port = "80" + port = "443" } return "dns:///" + net.JoinHostPort(host, port), nil } @@ -333,10 +333,14 @@ func (r *taskRuntime) startUser(ctx context.Context) (*ateEnvUser, error) { if err != nil { return nil, err } + tlsCfg, err := boomerutil.RouterTLSConfig() + if err != nil { + return nil, err + } dialFn = func() (ateenvv1alpha.ProcessServiceClient, io.Closer, error) { conn, err := grpc.NewClient( target, - grpc.WithTransportCredentials(insecure.NewCredentials()), + grpc.WithTransportCredentials(credentials.NewTLS(tlsCfg)), grpc.WithDisableServiceConfig(), ) if err != nil { diff --git a/internal/benchmarking/boomer/ateenv/lifecycle_test.go b/internal/benchmarking/boomer/ateenv/lifecycle_test.go index f544c42c61..201c6ba90a 100644 --- a/internal/benchmarking/boomer/ateenv/lifecycle_test.go +++ b/internal/benchmarking/boomer/ateenv/lifecycle_test.go @@ -336,15 +336,23 @@ func TestRouterGRPCTarget(t *testing.T) { }{ { input: "", - want: "dns:///atenet-router.ate-system.svc.cluster.local:80", + want: "dns:///atenet-router.ate-system.svc:443", }, { - input: "http://atenet-router.ate-system.svc.cluster.local", - want: "dns:///atenet-router.ate-system.svc.cluster.local:80", + input: "https://atenet-router.ate-system.svc", + want: "dns:///atenet-router.ate-system.svc:443", }, { - input: "http://atenet-router.ate-system.svc.cluster.local:8080", - want: "dns:///atenet-router.ate-system.svc.cluster.local:8080", + input: "https://atenet-router.ate-system.svc.cluster.local", + want: "dns:///atenet-router.ate-system.svc.cluster.local:443", + }, + { + input: "https://atenet-router.ate-system.svc.cluster.local:8443", + want: "dns:///atenet-router.ate-system.svc.cluster.local:8443", + }, + { + input: "atenet-router.ate-system.svc", + want: "dns:///atenet-router.ate-system.svc:443", }, { input: "router.example.internal:9000", @@ -364,12 +372,12 @@ func TestRouterGRPCTarget(t *testing.T) { }) } - if _, err := routerGRPCTarget("https://atenet-router.ate-system.svc.cluster.local"); err == nil { - t.Errorf("routerGRPCTarget(https) succeeded, want error") - } - if _, err := routerGRPCTarget("http:///missing-host"); err == nil { + if _, err := routerGRPCTarget("https:///missing-host"); err == nil { t.Errorf("routerGRPCTarget(empty host) succeeded, want error") } + if _, err := routerGRPCTarget("http://atenet-router.ate-system.svc.cluster.local"); err == nil { + t.Errorf("routerGRPCTarget(http) succeeded, want error") + } } func TestAteEnvIterate_SuspendMode(t *testing.T) { diff --git a/internal/benchmarking/boomer/boomerutil/client.go b/internal/benchmarking/boomer/boomerutil/client.go index 70e61a777c..8363645725 100644 --- a/internal/benchmarking/boomer/boomerutil/client.go +++ b/internal/benchmarking/boomer/boomerutil/client.go @@ -15,10 +15,14 @@ package boomerutil import ( + "crypto/tls" "fmt" + "os" + "sync" "go.ate.dev/substrate/api/ateapi/v1" "go.ate.dev/substrate/internal/ateapiauth" + "go.ate.dev/substrate/internal/credbundle" _ "go.ate.dev/substrate/internal/k8sresolver" "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc" "google.golang.org/grpc" @@ -28,17 +32,54 @@ import ( ) const ( - // ateapiCAFile verifies ateapi's servicedns serving cert; - // ateapiCredBundle is the benchmark's own pod certificate. Both are - // projected in by benchmarking/locust/manifests/locust.yaml. - ateapiCAFile = "/run/servicedns-ca/ca.crt" - ateapiCredBundle = "/run/podidentity.podcert.ate.dev/credential-bundle.pem" + // DefaultRouterURL is the default HTTPS base URL for atenet-router. + DefaultRouterURL = "https://atenet-router.ate-system.svc" + + // servicednsCAFile verifies ateapi's and atenet-router's servicedns serving + // certs; podIdentityCredBundle is the benchmark's own pod certificate. Both + // are projected in by benchmarking/locust/manifests/locust.yaml. + servicednsCAFile = "/run/servicedns-ca/ca.crt" + podIdentityCredBundle = "/run/podidentity.podcert.ate.dev/credential-bundle.pem" // ateapiServerName is the DNS SAN on the apiserver's serving cert, which // is shorter than the endpoint the benchmark dials. ateapiServerName = "api.ate-system.svc" + + // routerServerName is the 3-label DNS SAN on atenet-router's servicedns + // serving cert. + routerServerName = "atenet-router.ate-system.svc" ) +var routerTLSConfig = sync.OnceValues(func() (*tls.Config, error) { + return newRouterTLSConfig(servicednsCAFile, podIdentityCredBundle, routerServerName) +}) + +func newRouterTLSConfig(caFile, credBundle, serverName string) (*tls.Config, error) { + roots, err := credbundle.ParsePool(caFile) + if err != nil { + return nil, fmt.Errorf("loading router CA file: %w", err) + } + if _, err := os.Stat(credBundle); err != nil { + return nil, fmt.Errorf("checking router client credential bundle: %w", err) + } + return &tls.Config{ + MinVersion: tls.VersionTLS13, + RootCAs: roots, + ServerName: serverName, + GetClientCertificate: credbundle.ClientLoader(credBundle), + }, nil +} + +// RouterTLSConfig returns a mutual-TLS client configuration for dialing +// atenet-router. +func RouterTLSConfig() (*tls.Config, error) { + cfg, err := routerTLSConfig() + if err != nil { + return nil, err + } + return cfg.Clone(), nil +} + // DialControl opens an authenticated gRPC connection to ateapi. // // ateapi rejects calls that carry no credential, so present the pod @@ -62,8 +103,8 @@ func DialControl(endpoint string) (*grpc.ClientConn, ateapi.ControlClient, error dialOpts, err := ateapiauth.DialOptions(ateapiauth.ClientConfig{ K8sClient: k8sClient, - CAFile: ateapiCAFile, - ClientCredBundle: ateapiCredBundle, + CAFile: servicednsCAFile, + ClientCredBundle: podIdentityCredBundle, ServerName: ateapiServerName, }) if err != nil { diff --git a/internal/benchmarking/boomer/boomerutil/client_test.go b/internal/benchmarking/boomer/boomerutil/client_test.go new file mode 100644 index 0000000000..5b7352ce1b --- /dev/null +++ b/internal/benchmarking/boomer/boomerutil/client_test.go @@ -0,0 +1,192 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package boomerutil + +import ( + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/tls" + "crypto/x509" + "crypto/x509/pkix" + "encoding/pem" + "math/big" + "net/http" + "net/http/httptest" + "net/url" + "os" + "path/filepath" + "testing" + "time" +) + +func TestNewRouterTLSConfigMTLS(t *testing.T) { + caCert, caKey, caPEM := generateTestCA(t) + dir := t.TempDir() + + caFile := filepath.Join(dir, "ca.crt") + if err := os.WriteFile(caFile, caPEM, 0o600); err != nil { + t.Fatalf("write ca.crt: %v", err) + } + + credBundle := filepath.Join(dir, "credential-bundle.pem") + if err := os.WriteFile(credBundle, generateClientBundle(t, caCert, caKey), 0o600); err != nil { + t.Fatalf("write credential-bundle.pem: %v", err) + } + + serverCert := generateServerCert(t, caCert, caKey, routerServerName) + caPool := x509.NewCertPool() + caPool.AddCert(caCert) + + srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusOK) + })) + srv.TLS = &tls.Config{ + MinVersion: tls.VersionTLS13, + Certificates: []tls.Certificate{serverCert}, + ClientAuth: tls.RequireAndVerifyClientCert, + ClientCAs: caPool, + } + srv.StartTLS() + defer srv.Close() + + t.Run("matching ServerName succeeds", func(t *testing.T) { + tlsCfg, err := newRouterTLSConfig(caFile, credBundle, routerServerName) + if err != nil { + t.Fatalf("newRouterTLSConfig() error = %v", err) + } + client := &http.Client{ + Transport: &http.Transport{ + TLSClientConfig: tlsCfg, + ForceAttemptHTTP2: false, + }, + } + resp, err := client.Get(srv.URL) + if err != nil { + t.Fatalf("GET %s error = %v", srv.URL, err) + } + resp.Body.Close() + if resp.StatusCode != http.StatusOK { + t.Fatalf("status = %d, want 200", resp.StatusCode) + } + if resp.ProtoMajor != 1 { + t.Fatalf("proto = %s, want HTTP/1.1", resp.Proto) + } + }) + + t.Run("mismatched ServerName is rejected", func(t *testing.T) { + tlsCfg, err := newRouterTLSConfig(caFile, credBundle, "wrong.ate-system.svc") + if err != nil { + t.Fatalf("newRouterTLSConfig() error = %v", err) + } + client := &http.Client{ + Transport: &http.Transport{TLSClientConfig: tlsCfg}, + } + if _, err := client.Get(srv.URL); err == nil { + t.Fatal("GET succeeded with mismatched ServerName, want TLS verification error") + } + }) + + t.Run("missing files return error", func(t *testing.T) { + if _, err := newRouterTLSConfig(filepath.Join(dir, "missing-ca.crt"), credBundle, routerServerName); err == nil { + t.Fatal("expected error for missing CA file") + } + if _, err := newRouterTLSConfig(caFile, filepath.Join(dir, "missing-bundle.pem"), routerServerName); err == nil { + t.Fatal("expected error for missing credential bundle") + } + }) +} + +func generateTestCA(t *testing.T) (*x509.Certificate, *ecdsa.PrivateKey, []byte) { + t.Helper() + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + t.Fatalf("generate CA key: %v", err) + } + tmpl := &x509.Certificate{ + SerialNumber: big.NewInt(1), + Subject: pkix.Name{CommonName: "test-servicedns-ca"}, + NotBefore: time.Now().Add(-time.Hour), + NotAfter: time.Now().Add(time.Hour), + KeyUsage: x509.KeyUsageCertSign, + BasicConstraintsValid: true, + IsCA: true, + } + der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key) + if err != nil { + t.Fatalf("create CA cert: %v", err) + } + cert, err := x509.ParseCertificate(der) + if err != nil { + t.Fatalf("parse CA cert: %v", err) + } + return cert, key, pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}) +} + +func generateServerCert(t *testing.T, caCert *x509.Certificate, caKey *ecdsa.PrivateKey, dnsSAN string) tls.Certificate { + t.Helper() + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + t.Fatalf("generate server key: %v", err) + } + tmpl := &x509.Certificate{ + SerialNumber: big.NewInt(2), + DNSNames: []string{dnsSAN}, + NotBefore: time.Now().Add(-time.Hour), + NotAfter: time.Now().Add(time.Hour), + KeyUsage: x509.KeyUsageDigitalSignature, + ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}, + } + der, err := x509.CreateCertificate(rand.Reader, tmpl, caCert, &key.PublicKey, caKey) + if err != nil { + t.Fatalf("create server cert: %v", err) + } + return tls.Certificate{ + Certificate: [][]byte{der}, + PrivateKey: key, + } +} + +func generateClientBundle(t *testing.T, caCert *x509.Certificate, caKey *ecdsa.PrivateKey) []byte { + t.Helper() + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + t.Fatalf("generate client key: %v", err) + } + spiffeURI, err := url.Parse("spiffe://cluster.local/ns/benchmarking/sa/benchmark-runner") + if err != nil { + t.Fatalf("parse SPIFFE URI: %v", err) + } + tmpl := &x509.Certificate{ + SerialNumber: big.NewInt(3), + URIs: []*url.URL{spiffeURI}, + NotBefore: time.Now().Add(-time.Hour), + NotAfter: time.Now().Add(time.Hour), + KeyUsage: x509.KeyUsageDigitalSignature, + ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth}, + } + der, err := x509.CreateCertificate(rand.Reader, tmpl, caCert, &key.PublicKey, caKey) + if err != nil { + t.Fatalf("create client cert: %v", err) + } + pkcs8Key, err := x509.MarshalPKCS8PrivateKey(key) + if err != nil { + t.Fatalf("marshal client key: %v", err) + } + var bundle []byte + bundle = append(bundle, pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: pkcs8Key})...) + bundle = append(bundle, pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})...) + return bundle +} diff --git a/internal/e2e/README.md b/internal/e2e/README.md index 798adf5b3f..2e075470b5 100644 --- a/internal/e2e/README.md +++ b/internal/e2e/README.md @@ -27,7 +27,7 @@ what CI installs on the envoy lane; see `internal/e2e/suites/egresscredinject`. ## Sandbox classes The suites are runtime-agnostic: the same tests run against gVisor and against -the micro-VM (kata + cloud-hypervisor) sandbox class. `E2E_SANDBOX_CLASS` +the microVM (kata + cloud-hypervisor) sandbox class. `E2E_SANDBOX_CLASS` selects which, by repointing every fixture at its variant --- see `e2e.CounterFixture`, `e2e.EgressFixture` and `e2e.RenderFixtureManifest` in [sandbox.go](sandbox.go). Unset means gVisor. @@ -36,11 +36,11 @@ selects which, by repointing every fixture at its variant --- see # gVisor (the default), against the demos install-ate-kind.sh deploys $ hack/run-e2e-kind.sh -v -args --no-color -# micro-VM, against the counter-microvm and egress-microvm demos +# microVM, against the counter-microvm and egress-microvm demos $ E2E_SANDBOX_CLASS=microvm hack/run-e2e-kind.sh -v -args --no-color ``` -The micro-VM lane needs its fixtures installed first, which also needs a node +The microVM lane needs its fixtures installed first, which also needs a node with `/dev/kvm` (`hack/create-kind-cluster.sh` detects one and labels the node): ```shell @@ -53,7 +53,7 @@ installs the fixtures elsewhere: `E2E_SUBSTRATE_TEMPLATE_ATESPACE` / `E2E_SUBSTRATE_TEMPLATE_NAME` / `E2E_SUBSTRATE_POOL_NAMESPACE` / `E2E_SUBSTRATE_POOL_NAME` point the counter fixture somewhere else, and `E2E_TEMPLATE_READY_TIMEOUT` replaces the golden-snapshot budget (90s on -gVisor, 10m on micro-VM, where the golden is a cloud-hypervisor cold boot plus +gVisor, 10m on microVM, where the golden is a cloud-hypervisor cold boot plus a checkpoint). ## Settings @@ -97,7 +97,7 @@ misconfigured as S3 cannot pass by running nothing. A suite deletes the namespaces it created only when it passed. A failed run keeps them, because the failure is usually explained inside a worker pod (the -worker logs, and for a micro-VM worker the guest's console tail), and deleting +worker logs, and for a microVM worker the guest's console tail), and deleting the namespace takes those pods with it: ```shell diff --git a/internal/e2e/fixtures/testserver/kill_test.go b/internal/e2e/fixtures/testserver/kill_test.go index d2017e3b27..c9009031c8 100644 --- a/internal/e2e/fixtures/testserver/kill_test.go +++ b/internal/e2e/fixtures/testserver/kill_test.go @@ -21,6 +21,7 @@ import ( "os/exec" "path/filepath" "strconv" + "strings" "testing" "time" ) @@ -33,6 +34,18 @@ func TestKillMatching(t *testing.T) { exited := make(chan struct{}) go func() { _ = cmd.Wait(); close(exited) }() + // Start returns when exec closes the child's status pipe, before the + // kernel has set up the new argv, so /proc/<pid>/cmdline can read empty. + cmdline := filepath.Join("/proc", strconv.Itoa(cmd.Process.Pid), "cmdline") + for start := time.Now(); ; time.Sleep(time.Millisecond) { + if b, _ := os.ReadFile(cmdline); strings.Contains(string(b), "sleep\x0031337") { + break + } + if time.Since(start) > 5*time.Second { + t.Fatalf("%s still has no command line", cmdline) + } + } + n, err := killMatching("/proc", "sleep\x0031337", os.Getpid()) if err != nil { t.Fatal(err) diff --git a/internal/imagecache/README.md b/internal/imagecache/README.md index 672f7240be..7e3f2d198e 100644 --- a/internal/imagecache/README.md +++ b/internal/imagecache/README.md @@ -51,7 +51,7 @@ paths resolve in every pod) and a small per-bundle spec file. Because the consumer mounts the overlay **in its own mount namespace** — exactly where the workload resolves it (runsc's gofer for gVisor, virtiofsd -for the micro-VM) — no Kubernetes mount-propagation configuration is needed +for the microVM) — no Kubernetes mount-propagation configuration is needed anywhere. ## On-disk layout @@ -129,7 +129,7 @@ bundle-local `rootfs/`, `upper/`, and `work/` directories. ## Compose path (ateworker: `SetupBundleRootfs`) Called immediately before `runsc create`/`runsc restore` (gVisor) and before -staging the virtio-fs lower (micro-VM): +staging the virtio-fs lower (microVM): 1. **`FinalizeLayer`** for each referenced layer — materializes the recorded whiteouts as 0:0 char devices (`mknod`) and opaque dirs as @@ -170,7 +170,7 @@ rootfs with ExtraDirs and no mount. Actor semantics are unchanged from the untar era: the upper is wiped by atelet's `resetActorDirs` between runs, so every run still starts from a bit-exact, pristine image rootfs — it just costs a mount instead of an -extraction. The micro-VM path is nearly untouched: it bind-mounts the (now +extraction. The microVM path is nearly untouched: it bind-mounts the (now overlay-composed) bundle rootfs into virtiofsd's shared dir and the guest keeps building its own tmpfs upper, as before. diff --git a/internal/plugins/gcp-secret-manager/.ko.yaml b/internal/plugins/gcp-secret-manager/.ko.yaml index b1cf10e495..9d60cbabcb 100644 --- a/internal/plugins/gcp-secret-manager/.ko.yaml +++ b/internal/plugins/gcp-secret-manager/.ko.yaml @@ -14,7 +14,7 @@ # Substrate's base image, pinned by multi-platform index digest. Set # KO_DEFAULTPLATFORMS to override the platforms, e.g. host-only for kind. -defaultBaseImage: gcr.io/distroless/static-debian13:latest@sha256:f2ea2709ac8db56323cbd7d014277f32cb572d9ea124b0076f7aafe5980678fe +defaultBaseImage: gcr.io/distroless/static-debian13:latest@sha256:15f8637509792e05b97d83451b0b6ccfa4abd548f997e406ac1e72c87fe1fa78 defaultPlatforms: - linux/amd64 diff --git a/manifests/ate-install/ate-api-server.yaml b/manifests/ate-install/ate-api-server.yaml index 07b1f6ff3c..f74cedf468 100644 --- a/manifests/ate-install/ate-api-server.yaml +++ b/manifests/ate-install/ate-api-server.yaml @@ -122,6 +122,8 @@ spec: periodSeconds: 2 failureThreshold: 30 securityContext: + allowPrivilegeEscalation: false + runAsNonRoot: true capabilities: drop: - ALL @@ -198,6 +200,14 @@ spec: - --drain-timeout=15s - --objectstore-plugin-socket=/run/objectstore-plugin/control.sock - --snapshot-location=@env + securityContext: + allowPrivilegeEscalation: false + runAsNonRoot: true + capabilities: + add: + - NET_BIND_SERVICE + drop: + - ALL env: - name: POD_NAME valueFrom: diff --git a/manifests/ate-install/ate-controller.yaml b/manifests/ate-install/ate-controller.yaml index f5aeaba137..6974b92a3f 100644 --- a/manifests/ate-install/ate-controller.yaml +++ b/manifests/ate-install/ate-controller.yaml @@ -106,6 +106,13 @@ spec: - --ateapi-ca-file=/run/servicedns-ca/trust-bundle.pem - --ateapi-client-cert=/run/podidentity.podcert.ate.dev/credential-bundle.pem - --dns-egress-policy=enforce-in-atunnel + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + runAsNonRoot: true + capabilities: + drop: + - ALL env: - name: POD_NAME valueFrom: diff --git a/manifests/ate-install/ate-idp-server.yaml b/manifests/ate-install/ate-idp-server.yaml index a78641a2ba..4d07f29836 100644 --- a/manifests/ate-install/ate-idp-server.yaml +++ b/manifests/ate-install/ate-idp-server.yaml @@ -63,6 +63,8 @@ spec: scheme: HTTPS periodSeconds: 10 securityContext: + allowPrivilegeEscalation: false + runAsNonRoot: true capabilities: drop: - ALL diff --git a/manifests/ate-install/atelet.yaml b/manifests/ate-install/atelet.yaml index 8836ff8535..8b03be0cc8 100644 --- a/manifests/ate-install/atelet.yaml +++ b/manifests/ate-install/atelet.yaml @@ -183,6 +183,7 @@ spec: # Reads and writes snapshot and asset files under the /var/lib/ate # hostPath as root, like atelet; needs no capabilities. securityContext: + allowPrivilegeEscalation: false runAsUser: 0 runAsGroup: 0 capabilities: @@ -257,6 +258,7 @@ spec: # runAsUser is pinned explicitly rather than relying on the base image's # default user. securityContext: + allowPrivilegeEscalation: false runAsUser: 0 runAsGroup: 0 capabilities: diff --git a/manifests/ate-install/atenet-egress.yaml b/manifests/ate-install/atenet-egress.yaml index 994b2fae6f..f5d2f5ddc1 100644 --- a/manifests/ate-install/atenet-egress.yaml +++ b/manifests/ate-install/atenet-egress.yaml @@ -1398,6 +1398,11 @@ spec: # {"enabled":false}. Without a provider, a request matching a rule that # requires an injection is denied (see egress.applyEffects). #ATE_EGRESS_INJECT_FLAGS -- Don't modify this line. + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL env: - name: POD_NAME valueFrom: diff --git a/manifests/ate-install/atenet-router.yaml b/manifests/ate-install/atenet-router.yaml index 0b122cc46a..d9185f591c 100644 --- a/manifests/ate-install/atenet-router.yaml +++ b/manifests/ate-install/atenet-router.yaml @@ -251,6 +251,12 @@ spec: # generous route ceiling cannot stretch every rollout to the worst case. # If long turns must survive a shutdown, raise --drain-timeout and # terminationGracePeriodSeconds above together. + securityContext: + allowPrivilegeEscalation: false + runAsNonRoot: true + capabilities: + drop: + - ALL env: - name: POD_NAME valueFrom: @@ -317,6 +323,14 @@ spec: - "/usr/local/bin/envoy" - "-c" - "/etc/envoy/envoy.yaml" + # The Envoy image sets no USER, so runAsNonRoot needs a numeric UID. + securityContext: + allowPrivilegeEscalation: false + runAsNonRoot: true + runAsUser: 65532 + capabilities: + drop: + - ALL # Prevents Envoy from fast-exiting on SIGTERM before atenet-router finishes # its drain sequence. Polls for the drain-complete marker written by the # router on the shared emptyDir, terminating Envoy as soon as the drain diff --git a/manifests/ate-install/cloudsql/proxy-sidecar-patch.yaml b/manifests/ate-install/cloudsql/proxy-sidecar-patch.yaml index 2e41661bc0..5981617016 100644 --- a/manifests/ate-install/cloudsql/proxy-sidecar-patch.yaml +++ b/manifests/ate-install/cloudsql/proxy-sidecar-patch.yaml @@ -35,7 +35,7 @@ spec: # keeps running for the pod's lifetime. Requires Kubernetes 1.29+. initContainers: - name: cloud-sql-proxy - image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.25.2 + image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.26.3@sha256:f333a9c59890c3c67b8505166c5dbdcc0b29fca9173a2e09ca0446ab6997a736 restartPolicy: Always args: - "$(ATE_API_POSTGRES_CLOUDSQL_INSTANCE)" diff --git a/manifests/ate-install/generated/ate.dev_workerpools.yaml b/manifests/ate-install/generated/ate.dev_workerpools.yaml index bde0e0fc5b..e4506662eb 100644 --- a/manifests/ate-install/generated/ate.dev_workerpools.yaml +++ b/manifests/ate-install/generated/ate.dev_workerpools.yaml @@ -625,6 +625,10 @@ spec: required: - spec type: object + x-kubernetes-validations: + - message: metadata.name must be at most 63 characters, because worker pods + use it as a label value + rule: size(self.metadata.name) <= 63 served: true storage: true subresources: diff --git a/manifests/ate-install/kind/otel-collector.yaml b/manifests/ate-install/kind/otel-collector.yaml index 80cbd132b8..b8c83aeb81 100644 --- a/manifests/ate-install/kind/otel-collector.yaml +++ b/manifests/ate-install/kind/otel-collector.yaml @@ -176,7 +176,7 @@ spec: spec: containers: - name: otel-collector - image: otel/opentelemetry-collector-contrib:0.157.0@sha256:f2f01157055a9b2aab9df7118e1f1c9abf345e99b23bc7a2bc791db374a7d0f6 + image: otel/opentelemetry-collector-contrib:0.163.2@sha256:7f22d2a6748110ffb6a59bc0df448bccbf501cdb667d610ede4c61d3863d5849 args: - --config=/conf/otel-collector-config.yaml volumeMounts: @@ -241,7 +241,7 @@ spec: spec: containers: - name: jaeger - image: jaegertracing/all-in-one:1.55@sha256:f6b5d09073f14f76873d300f565a6691d815e81bea8e07e1dc3ff67e0596dd4e + image: jaegertracing/jaeger:2.22.0@sha256:836b967b5abcf1f7758066aa6b348619267a2f1459f72d472840570f9bd88a17 ports: - name: otlp-grpc containerPort: 4317 diff --git a/manifests/ate-install/kind/prometheus.yaml b/manifests/ate-install/kind/prometheus.yaml index 05b9eac578..20bbfab69c 100644 --- a/manifests/ate-install/kind/prometheus.yaml +++ b/manifests/ate-install/kind/prometheus.yaml @@ -172,7 +172,7 @@ spec: type: RuntimeDefault containers: - name: prometheus - image: prom/prometheus:v3.5.3@sha256:ddc2493835a1509976d5e4e0c94199c4f843ce1f42dd6bcfc8231ba734a93ff7 + image: prom/prometheus:v3.15.1@sha256:fa3302279710be2f21a8301ec17a8b4f2301fd829bd113839b3b6b02b6f191a2 args: - --config.file=/etc/prometheus/prometheus.yml - --storage.tsdb.path=/prometheus diff --git a/manifests/ate-install/kind/rustfs.yaml b/manifests/ate-install/kind/rustfs.yaml index 1cbe063255..b37f52b9ce 100644 --- a/manifests/ate-install/kind/rustfs.yaml +++ b/manifests/ate-install/kind/rustfs.yaml @@ -62,7 +62,7 @@ spec: fsGroup: 10001 containers: - name: rustfs - image: rustfs/rustfs:1.0.0-beta.3@sha256:378642b05b7dcb4849fb77ebe6aca4ced1c3f66e7e504247df95a5c9018d3358 + image: rustfs/rustfs:1.0.3@sha256:9ca9559533b537d10482fe923288b676ad08f92a7069d9a17327fbae3cab688b imagePullPolicy: IfNotPresent ports: - containerPort: 9000 @@ -102,7 +102,7 @@ spec: restartPolicy: OnFailure containers: - name: create-bucket - image: amazon/aws-cli:2.17.0@sha256:643507c10ada7964ca6157b3d799f030b90577643da9955d319a77399ed80d73 + image: amazon/aws-cli:2.38.5@sha256:c8156a0e74e13929a42d7e60f0d4e14e7d28d980cd21822505d4bff80e5460c1 env: - name: AWS_ACCESS_KEY_ID value: rustfsadmin diff --git a/manifests/ate-install/pod-certificate-controller.yaml b/manifests/ate-install/pod-certificate-controller.yaml index fab1f38686..cf48e459f9 100644 --- a/manifests/ate-install/pod-certificate-controller.yaml +++ b/manifests/ate-install/pod-certificate-controller.yaml @@ -191,6 +191,7 @@ spec: mountPath: "/run/ca-state" securityContext: allowPrivilegeEscalation: false + runAsNonRoot: true capabilities: add: - NET_BIND_SERVICE diff --git a/manifests/ate-install/postgres-size10/postgres-config-patch.yaml b/manifests/ate-install/postgres-size10/postgres-config-patch.yaml index 020494b6bc..ba73324efb 100644 --- a/manifests/ate-install/postgres-size10/postgres-config-patch.yaml +++ b/manifests/ate-install/postgres-size10/postgres-config-patch.yaml @@ -46,10 +46,10 @@ data: wal_buffers = 64MB huge_pages = try - # 2 ate-api-server replicas at pool_max_conns=64 + 3-conn watch pool + - # 1-conn readiness pool each = 136 steady-state; 300 leaves room to double - # replicas and for psql / monitoring / migrations. - max_connections = 300 + # Each ate-api-server replica holds pool_max_conns=64 plus 6 owner, watch + # and readiness connections, 70 in all. 500 fits 7 replicas with 10 to + # spare for psql / monitoring / migrations. + max_connections = 500 # 1 GB max_wal_size (the compiled default) forces a checkpoint every few # seconds under sustained writes and caps throughput well below what the diff --git a/manifests/ate-install/rustfs.yaml b/manifests/ate-install/rustfs.yaml index 0256197359..be80f29173 100644 --- a/manifests/ate-install/rustfs.yaml +++ b/manifests/ate-install/rustfs.yaml @@ -99,7 +99,7 @@ spec: fsGroup: 10001 containers: - name: rustfs - image: rustfs/rustfs:1.0.0-beta.3@sha256:378642b05b7dcb4849fb77ebe6aca4ced1c3f66e7e504247df95a5c9018d3358 + image: rustfs/rustfs:1.0.3@sha256:9ca9559533b537d10482fe923288b676ad08f92a7069d9a17327fbae3cab688b resources: limits: cpu: "1" @@ -147,7 +147,7 @@ spec: serviceAccountName: rustfs containers: - name: create-bucket - image: amazon/aws-cli:2.17.0@sha256:643507c10ada7964ca6157b3d799f030b90577643da9955d319a77399ed80d73 + image: amazon/aws-cli:2.38.5@sha256:c8156a0e74e13929a42d7e60f0d4e14e7d28d980cd21822505d4bff80e5460c1 resources: limits: cpu: 500m diff --git a/manifests/ate-install/sandboxconfig-gvisor.yaml b/manifests/ate-install/sandboxconfig-gvisor.yaml index 176c8b8fbd..44b81a697d 100644 --- a/manifests/ate-install/sandboxconfig-gvisor.yaml +++ b/manifests/ate-install/sandboxconfig-gvisor.yaml @@ -31,21 +31,21 @@ metadata: sandboxconfig.ate.dev/is-class-default: "true" spec: sandboxClass: gvisor - defaultVersion: v4 + defaultVersion: v5 versions: - # gVisor nightly 2026-10-17, the first to include google/gvisor@b79256080dec - # (save and restore multiple filesystem checkpoint bundles). ateworker-gvisor - # uses it to save container rootfs overlays (rootfs/) and durable-dir volumes - # (fs/) as separate bundles, and to run the sandbox with no root container - # (google/gvisor#15551). - - name: v4 + # gVisor release 20261019.0, the first tagged release with + # google/gvisor@b79256080dec (save and restore multiple filesystem checkpoint + # bundles). ateworker-gvisor uses it to save container rootfs overlays + # (rootfs/) and durable-dir volumes (fs/) as separate bundles, and to run the + # sandbox with no root container (google/gvisor#15551). + - name: v5 state: Enabled assets: amd64: gvisor: - url: "gs://gvisor/releases/nightly/2026-10-17/x86_64/gvisor.tar.zstd" - sha256: "fa72f0d45756c69a30beec21222c78241b0d330c36f4f2185648617cc23415c8" + url: "gs://gvisor/releases/release/20261019.0/x86_64/gvisor.tar.zstd" + sha256: "8637abfb50f81552af25ca08f7172589c09292a7c881efe8d9e09a9f3b6bec7b" arm64: gvisor: - url: "gs://gvisor/releases/nightly/2026-10-17/aarch64/gvisor.tar.zstd" - sha256: "238634978d99f0dd3e631c4f88eae7b19fb36937065428467f0b6d6b10697689" + url: "gs://gvisor/releases/release/20261019.0/aarch64/gvisor.tar.zstd" + sha256: "88556c21468a066d872a055425958c83fcbafc79c6477a195b18070c77f856d0" diff --git a/manifests/microvm/sandboxconfig-microvm.yaml.tmpl b/manifests/microvm/sandboxconfig-microvm.yaml.tmpl index d84c205ff8..19bd5b6fd3 100644 --- a/manifests/microvm/sandboxconfig-microvm.yaml.tmpl +++ b/manifests/microvm/sandboxconfig-microvm.yaml.tmpl @@ -43,9 +43,9 @@ metadata: sandboxconfig.ate.dev/is-class-default: "true" spec: sandboxClass: microvm - defaultVersion: v1 + defaultVersion: v2 versions: - - name: v1 + - name: v2 state: Enabled assets: arm64: @@ -55,14 +55,14 @@ spec: # virtiofsd serves the overlay RO lower (virtio-fs). virtiofsd: url: "${BUCKET_SCHEME}://${ASSET_BUCKET_NAME}/kata-assets/virtiofsd" - sha256: "3aa9240045d2261930ef04138fe05cf288f94c867e63e5d0845758bac5e3d458" + sha256: "4b43f4f6de6500203b01214a790314eea4182f5f110594ddb28ae2805622f06f" kata-kernel: url: "${BUCKET_SCHEME}://${ASSET_BUCKET_NAME}/kata-assets/vmlinux" - sha256: "4960f6cf781274eaa445fc9f88bb8c5ce951649278bc9caa13afb1e99cfe80e6" + sha256: "dece388830720fe8bddb41552fff5f486240aeece5acc0aebc1812761e3e3f66" kata-image: url: "${BUCKET_SCHEME}://${ASSET_BUCKET_NAME}/kata-assets/rootfs.img" - sha256: "7e3bdd500eb92de17c377bd84f8ed57e50d6bb68c7ec08c88f5e1916cd49f30d" - # amd64 assets are kata 4.1.0 (assemble.sh ARCH=amd64). + sha256: "69f9a76e5a2563b7c74d1f271f3a904240f50ef405465f832021525894519f97" + # amd64 assets are kata 4.3.1 (assemble.sh ARCH=amd64). amd64: cloud-hypervisor: url: "${BUCKET_SCHEME}://${ASSET_BUCKET_NAME}/kata-assets/cloud-hypervisor" @@ -72,7 +72,7 @@ spec: sha256: "15b2e72a78cc08a9bd8a6943e89fb69c88cb3cbeb63069efceade835342ac7d4" kata-kernel: url: "${BUCKET_SCHEME}://${ASSET_BUCKET_NAME}/kata-assets/vmlinux" - sha256: "8e9dbc3a6e4c26adb089d23d31201edebce905c7f3e31aa67b32bdd41df1b86f" + sha256: "f87d8359035cb7557b16a319e6431ea09b183e62268aef436bf8dc6530d786b6" kata-image: url: "${BUCKET_SCHEME}://${ASSET_BUCKET_NAME}/kata-assets/rootfs.img" - sha256: "96497f64da1de9c7473fef46c3d29ddd0805d334731cf9d903a21a5b2c33cefb" + sha256: "6770db2cca5710593f27c9ac5e1d9d7aab836422695a5e5c4807496a92a0f24d" diff --git a/tools/setup-gcp/README.md b/tools/setup-gcp/README.md index 981a1bd307..f0c685de76 100644 --- a/tools/setup-gcp/README.md +++ b/tools/setup-gcp/README.md @@ -127,7 +127,7 @@ go run ./tools/setup-gcp create cluster [flags] | `--network` | VPC network name. | `NETWORK` | `default` | | `--subnetwork` | VPC subnetwork name. | `SUBNETWORK` | `default` | | `--machine-type` | Machine type for the node pool. | `NODE_MACHINE_TYPE` | `c3-standard-4` | -| `--enable-nested-virtualization` | Create the node pool with nested virtualization, exposing `/dev/kvm` for micro-VM workers. | `ENABLE_NESTED_VIRTUALIZATION` | `true` | +| `--enable-nested-virtualization` | Create the node pool with nested virtualization, exposing `/dev/kvm` for microVM workers. | `ENABLE_NESTED_VIRTUALIZATION` | `true` | | `--boot-disk-size` | Boot disk size in GB for the node pool (0 = GKE default). | `BOOT_DISK_SIZE_GB` | None | | `--boot-disk-type` | Boot disk type for the node pool (empty = GKE default). | `BOOT_DISK_TYPE` | None | @@ -208,7 +208,7 @@ one of the options below. ### 3. Create Bucket Creates the GCS bucket for snapshots and the asset bucket, `NAME-assets`, for -micro-VM sandbox assets. +microVM sandbox assets. ```bash go run ./tools/setup-gcp create bucket [flags] @@ -217,7 +217,7 @@ go run ./tools/setup-gcp create bucket [flags] **Flags:** | Flag | Description | Default Env Var | Fallback Default | | :--- | :--- | :--- | :--- | -| `--name` | Name of the GCS snapshot bucket. The micro-VM asset bucket is created beside it as `NAME-assets`. | `BUCKET_NAME` | None (Required*) | +| `--name` | Name of the GCS snapshot bucket. The microVM asset bucket is created beside it as `NAME-assets`. | `BUCKET_NAME` | None (Required*) | *\*Note: Required unless the `BUCKET_NAME` environment variable is set.* @@ -244,7 +244,7 @@ go run ./tools/setup-gcp create iam [flags] #### What `create iam` actually grants On a fresh GCP project these bindings do not exist, and nothing else creates -them. Without them atelet cannot fetch micro-VM assets, ate-api cannot copy or +them. Without them atelet cannot fetch microVM assets, ate-api cannot copy or collect snapshots or issue per-actor tokens for them (`403` from GCS on the first suspend/resume), and nodes cannot pull images. `bootstrap` and `create iam` apply them for you. The table below is the reference for auditing them, or for @@ -271,7 +271,7 @@ have it enabled too, or their GCS calls fail with `401`. | snapshot transfer SA⁴ | `roles/storage.bucketViewer` | snapshot bucket | List/stat the snapshot bucket | | ate-api-server WI principal¹ | `roles/iam.serviceAccountTokenCreator` | snapshot transfer SA⁴ | The credential exchange impersonates it | | Cloud SQL GSA⁵ | `roles/iam.serviceAccountTokenCreator` | snapshot transfer SA⁴ | Granted by `create cloudsql`: with Cloud SQL, ate-api-server's pod runs as this account | -| atelet WI principal¹ | `roles/storage.objectViewer` | asset bucket (`BUCKET_NAME-assets`) | Read micro-VM sandbox assets | +| atelet WI principal¹ | `roles/storage.objectViewer` | asset bucket (`BUCKET_NAME-assets`) | Read microVM sandbox assets | | default compute SA³ | `roles/artifactregistry.reader` | project² | Nodes pull images from Artifact Registry, which also serves gcr.io | ¹ `principal://iam.googleapis.com/projects/PROJECT_NUMBER/locations/global/workloadIdentityPools/PROJECT_ID.svc.id.goog/subject/ns/ate-system/sa/SERVICE_ACCOUNT` — note it is keyed to the **project number**, not the ID. @@ -311,7 +311,7 @@ gcloud storage buckets add-iam-policy-binding "gs://${BUCKET_NAME}-assets" \ #### Upgrading an existing install Substrate does not support upgrading a cluster in place yet. A cluster with -micro-VM workers has to be recreated: each node indexes a sandbox runtime by +microVM workers has to be recreated: each node indexes a sandbox runtime by the digests of its assets, which do not change when the assets move to the asset bucket, so a node keeps fetching them from the snapshot bucket. The commands below still apply when the new cluster reuses the GCP project. @@ -329,7 +329,7 @@ bucket. Upgrade Substrate first, to a release whose node plugin gets snapshot credentials from ate-api-server's credential exchange. On an older release, removing these grants makes every suspend and resume fail with `403`. Remove those grants after running `create bucket` and `create iam`, and after -rerunning `hack/install-microvm-deps.sh --install` if you use micro-VM +rerunning `hack/install-microvm-deps.sh --install` if you use microVM workers, so the assets are in the asset bucket. `ATELET` is set as in the manual equivalent above: @@ -381,7 +381,7 @@ go run ./tools/setup-gcp bootstrap [flags] | `--network` | VPC network name. | `NETWORK` | `default` | | `--subnetwork` | VPC subnetwork name. | `SUBNETWORK` | `default` | | `--machine-type` | Machine type for the node pool. | `NODE_MACHINE_TYPE` | `c3-standard-4` | -| `--enable-nested-virtualization` | Create the node pool with nested virtualization, exposing `/dev/kvm` for micro-VM workers. | `ENABLE_NESTED_VIRTUALIZATION` | `true` | +| `--enable-nested-virtualization` | Create the node pool with nested virtualization, exposing `/dev/kvm` for microVM workers. | `ENABLE_NESTED_VIRTUALIZATION` | `true` | | `--boot-disk-size` | Boot disk size in GB for the node pool (0 = GKE default). | `BOOT_DISK_SIZE_GB` | None | | `--boot-disk-type` | Boot disk type for the node pool (empty = GKE default). | `BOOT_DISK_TYPE` | None | | `--bucket-name` | Name of the GCS bucket for snapshots. | `BUCKET_NAME` | None (Required*) | @@ -414,7 +414,7 @@ go run ./tools/setup-gcp bootstrap \ --machine-type="n2-standard-8" ``` -### Bootstrap a cluster whose nodes cannot run micro-VM workers +### Bootstrap a cluster whose nodes cannot run microVM workers Nested virtualization is on by default, so turning it off is the case that needs a flag: