ci(release): run Linux release canaries in QEMU VMs

.agents/skills/test-release-canary/SKILL.md

@@ -12,11 +12,14 @@ The Release Canary (`.github/workflows/release-canary.yml`) smoke-tests the arti
 | Job | Runner | Verifies |
 |---|---|---|
 | `macos` | `macos-latest-xlarge` | `install.sh` resolves the Homebrew formula, brew installs the cask, and `openshell status` reaches the brew-services–backed local gateway with the VM driver. |
-| `ubuntu` | `ubuntu-latest` | `install.sh` installs the Debian package, the post-install systemd user service starts, and `openshell status` reaches the local gateway with the Docker driver. |
-| `fedora` | `fedora:latest` container | `install.sh` installs the RPM packages, the local gateway starts under Podman, and `openshell status` succeeds. |
+| `linux-vm` (Ubuntu) | `ubuntu-24.04` / `ubuntu-24.04-arm` | The Nix VM harness boots Ubuntu 24.04 on AMD64 and ARM64, configures Docker, installs the rolling `dev` Debian packages, and verifies the local gateway. |
+| `linux-vm` (Fedora) | `ubuntu-24.04` / `ubuntu-24.04-arm` | The Nix VM harness boots Fedora 44 on AMD64 and ARM64, configures rootless Podman with enforcing SELinux, installs the rolling `dev` RPM packages, and verifies the local gateway. |
+| `ubuntu-snap` | `ubuntu-latest` | The Snap produced by the triggering Release Dev run installs with its required interfaces and reaches the Docker-backed gateway. |
 | `kubernetes` | `ubuntu-latest` + kind | `helm install oci://ghcr.io/nvidia/openshell/helm-chart --version 0.0.0-dev` succeeds in a kind cluster, the gateway pod becomes Ready, port-forward exposes 8080, and the released CLI registers the in-cluster gateway and runs `openshell status` against it. |
 
-`install.sh` defaults to the *latest tagged* release — the canary is therefore checking that the most recent public release still installs, not the just-published `dev` build. The `kubernetes` job is the exception: it pins to `0.0.0-dev` chart + `:dev` images.
+The Linux VM matrix sets `OPENSHELL_VERSION=dev`, so it tests the packages published by Release Dev. The Snap job downloads artifacts from the triggering run, and Kubernetes pins the `0.0.0-dev` chart and `:dev` images. The macOS job retains the installer's default latest-tagged release behavior.
+
+On Linux, the VM harness uses KVM when `/dev/kvm` is readable and writable. It falls back automatically to native-architecture QEMU TCG when KVM is unavailable, which is expected on public ARM64 runners. The ARM64 fallback switches from the default firmware to QEMU's bundled EDK2 firmware, which is validated under TCG. Normal canaries do not cross-emulate architectures.
 
 ## Trigger paths
 
@@ -31,7 +34,7 @@ on:
 ```
 
 - **Automatic.** Every successful `Release Dev` run (on `main` or a manual dispatch of Release Dev) fires the canary. Each job gates on `github.event.workflow_run.conclusion == 'success'` so a failed Release Dev does not run the canary.
-- **Manual.** `workflow_dispatch` lets you run the canary on demand against any branch's workflow definition.
+- **Manual.** `workflow_dispatch` lets you run the canary on demand against any branch's workflow definition. The Snap job is skipped because a manual run has no triggering Release Dev artifact.
 
 When dispatched manually, `github.event.workflow_run.head_sha` is empty and the workflow falls back to `github.sha` (the branch tip) for the `install.sh` URL.
 
@@ -59,9 +62,36 @@ gh run view <run-id> --log-failed
 
 ## Iterating on the canary itself
 
-When you change `release-canary.yml` on a branch, a manual dispatch on that branch tests *your branch's workflow logic* against *main's published artifacts* (`0.0.0-dev` chart, `:dev` images, latest tagged install.sh assets). This is what you want for iterating on the canary — you're validating that the canary still works against known-good artifacts.
+When you change `release-canary.yml` on a branch, a manual dispatch on that branch tests *your branch's workflow logic* against the floating published artifacts (`dev` Linux packages, `0.0.0-dev` chart, and `:dev` images). The macOS job continues to exercise the latest tagged Homebrew release.
+
+Note `install.sh` is pulled from `raw.githubusercontent.com/NVIDIA/OpenShell/${head_sha}/install.sh`, so changes to `install.sh` on your branch are exercised. The selected binary release follows the per-job behavior described above.
+
+## Local Linux VM reproduction
+
+Reproduce each native-architecture Linux canary with the same VM app:
+
+```shell
+nix run .#test-guest -- \
+  --distro ubuntu \
+  --with docker \
+  -- env \
+    OPENSHELL_CANARY_DRIVER=docker \
+    OPENSHELL_CANARY_INSTALL_SH_URL="https://raw.githubusercontent.com/NVIDIA/OpenShell/$(git rev-parse HEAD)/install.sh" \
+  bash -lc '
+    set -euo pipefail
+    mkdir -p "${HOME}/.config/openshell"
+    printf "OPENSHELL_DRIVERS=%s\n" "${OPENSHELL_CANARY_DRIVER}" \
+      > "${HOME}/.config/openshell/gateway.env"
+    curl -LsSf "${OPENSHELL_CANARY_INSTALL_SH_URL}" |
+      OPENSHELL_VERSION=dev sh
+    openshell --version
+    openshell status
+  '
+```
 
-Note `install.sh` is pulled from `raw.githubusercontent.com/NVIDIA/OpenShell/${head_sha}/install.sh`, so changes to `install.sh` on your branch *are* exercised even though the binaries it downloads are from the latest public tag.
+For the RPM path, replace `--distro ubuntu --with docker` with
+`--distro fedora --with podman --with selinux` and set
+`OPENSHELL_CANARY_DRIVER=podman`.
 
 ## Testing artifacts from a specific SHA
 
@@ -107,8 +137,9 @@ Loopback registration auto-derives the gateway name to `openshell` if `--name` i
 
 | Symptom | Likely cause | Where to look |
 |---|---|---|
-| `macos`/`ubuntu`/`fedora` job fails on `install.sh` | Latest tagged release missing an asset, checksum mismatch, or `install.sh` regression on this branch. | Job log around the `curl … install.sh \| sh` step. |
-| `macos`/`ubuntu`/`fedora` job fails on `openshell status` | Local gateway service did not start (systemd/brew/podman). Often a driver issue. | Service logs in the job log; `OPENSHELL_DRIVERS` env in the "Ensure …" step. |
+| `macos`/`linux-vm` job fails on `install.sh` | Published release asset missing, checksum mismatch, or `install.sh` regression on this branch. | Job log around the `curl … install.sh \| sh` step. |
+| `linux-vm` fails before SSH | Cloud-image download, QEMU startup, or guest boot failure. | The runner prints its QEMU and serial logs automatically on failure; check whether it selected KVM or TCG. |
+| `macos`/`linux-vm` job fails on `openshell status` | Local gateway service did not start (systemd/brew/Docker/Podman). Often a driver issue. | Installer diagnostics and the configured `OPENSHELL_DRIVERS` value in the job log. |
 | `kubernetes` job fails on `helm install --wait` | Chart did not deploy in 5 min — usually image pull failure or readiness probe failing. | "Diagnostics on failure" step dumps `helm status`, manifest, pod describe, pod logs. |
 | `kubernetes` job fails on `kubectl wait` | Gateway pod stuck `CrashLoopBackOff` or `ImagePullBackOff`. | Diagnostics dump; check `:dev` image existence at `ghcr.io/nvidia/openshell/gateway`. |
 | `kubernetes` job fails on `openshell gateway add` or `status` | Port-forward not reachable, or CLI/gateway proto mismatch. | `port-forward.log` and `openshell gateway list` in the diagnostics dump. |

.github/workflows/release-canary.yml

@@ -30,120 +30,81 @@ jobs:
           curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh | sh
           openshell status
 
-  ubuntu:
-    name: Ubuntu Docker
+  linux-vm:
+    name: Linux VM (${{ matrix.distro }}, ${{ matrix.architecture }})
     if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
-    runs-on: ubuntu-latest
-    timeout-minutes: 20
+    runs-on: ${{ matrix.runner }}
+    timeout-minutes: 90
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - runner: ubuntu-24.04
+            architecture: amd64
+            distro: ubuntu
+            driver: docker
+            configurations: docker
+          - runner: ubuntu-24.04-arm
+            architecture: arm64
+            distro: ubuntu
+            driver: docker
+            configurations: docker
+          - runner: ubuntu-24.04
+            architecture: amd64
+            distro: fedora
+            driver: podman
+            configurations: podman selinux
+          - runner: ubuntu-24.04-arm
+            architecture: arm64
+            distro: fedora
+            driver: podman
+            configurations: podman selinux
     steps:
-      - name: Ensure Docker
-        run: |
-          if ! command -v docker >/dev/null 2>&1; then
-            sudo apt-get update
-            sudo apt-get install -y docker.io
-          fi
-          sudo systemctl start docker || sudo service docker start
-          mkdir -p "${HOME}/.config/openshell"
-          printf 'OPENSHELL_DRIVERS=docker\n' > "${HOME}/.config/openshell/gateway.env"
-          docker info
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          ref: ${{ github.event.workflow_run.head_sha || github.sha }}
+          persist-credentials: false
 
-      - name: Install and check status
-        run: |
-          curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh | sh
-          openshell status
+      - name: Install Nix
+        uses: cachix/install-nix-action@630ae543ea3a38a9a4166f03376c02c50f408342 # v31.11.0
+        with:
+          enable_kvm: true
+          github_access_token: ${{ github.token }}
 
-  fedora:
-    name: Fedora RPM
-    if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
-    runs-on: linux-amd64-cpu8
-    timeout-minutes: 20
-    env:
-      FEDORA_CANARY_CONTAINER: openshell-fedora-canary-${{ github.run_id }}-${{ github.run_attempt }}
-    steps:
-      - name: Start Fedora systemd container and root user manager
+      - name: Install dev release and check status
+        env:
+          TEST_VM_CONFIGURATIONS: ${{ matrix.configurations }}
+          TEST_VM_DISTRO: ${{ matrix.distro }}
+          TEST_VM_DRIVER: ${{ matrix.driver }}
+          TEST_VM_INSTALL_SH_URL: https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh
         run: |
           set -euo pipefail
 
-          docker run --detach \
-            --name "${FEDORA_CANARY_CONTAINER}" \
-            --privileged \
-            --cgroupns=host \
-            --tmpfs /run \
-            --tmpfs /tmp \
-            --volume /sys/fs/cgroup:/sys/fs/cgroup:rw \
-            fedora:latest \
-            bash -lc 'dnf install -y curl dbus-daemon podman systemd && exec /usr/sbin/init'
-
-          for _ in $(seq 1 120); do
-            if docker exec "${FEDORA_CANARY_CONTAINER}" systemctl list-units --no-pager >/dev/null 2>&1; then
-              break
-            fi
-            if [ "$(docker inspect -f '{{.State.Running}}' "${FEDORA_CANARY_CONTAINER}")" != "true" ]; then
-              echo "::error::Fedora systemd container exited before systemd became reachable"
-              docker logs "${FEDORA_CANARY_CONTAINER}" >&2 || true
-              exit 1
-            fi
-            sleep 1
-          done
-
-          if ! docker exec "${FEDORA_CANARY_CONTAINER}" systemctl list-units --no-pager >/dev/null 2>&1; then
-            echo "::error::Fedora systemd container did not become reachable within 120s"
-            docker logs "${FEDORA_CANARY_CONTAINER}" >&2 || true
-            exit 1
-          fi
-
-          docker exec --interactive "${FEDORA_CANARY_CONTAINER}" env \
-            HOME=/root \
-            XDG_RUNTIME_DIR=/run/user/0 \
-            DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus \
-            bash -s <<'EOF'
-          set -euo pipefail
-          # install.sh manages the RPM gateway as a systemd user unit. This
-          # container is booted with systemd as PID 1, but it still has no
-          # login session. Start root's user manager explicitly so the
-          # installer can test service restart and gateway registration
-          # instead of its "restart later" fallback.
-          mkdir -p "${XDG_RUNTIME_DIR}"
-          chmod 700 "${XDG_RUNTIME_DIR}"
-          systemctl start user-runtime-dir@0.service || true
-          systemctl start user@0.service
-
-          for _ in $(seq 1 30); do
-            if systemctl --user daemon-reload; then
-              break
-            fi
-            sleep 1
+          read -r -a configurations <<< "${TEST_VM_CONFIGURATIONS}"
+          configuration_args=()
+          for configuration in "${configurations[@]}"; do
+            configuration_args+=(--with "${configuration}")
           done
-          if ! systemctl --user daemon-reload; then
-            systemctl status user@0.service --no-pager >&2 || true
-            journalctl -u user@0.service --no-pager -n 80 >&2 || true
-            systemctl --user status --no-pager >&2 || true
-            exit 1
-          fi
-          EOF
 
-      - name: Install and check status
-        run: |
-          set -euo pipefail
-
-          docker exec --interactive "${FEDORA_CANARY_CONTAINER}" env \
-            HOME=/root \
-            XDG_RUNTIME_DIR=/run/user/0 \
-            DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus \
-            INSTALL_SH_URL="https://raw.githubusercontent.com/NVIDIA/OpenShell/${{ github.event.workflow_run.head_sha || github.sha }}/install.sh" \
-            bash -s <<'EOF'
-          set -euo pipefail
-          mkdir -p "${HOME}/.config/openshell"
-          printf 'OPENSHELL_DRIVERS=podman\n' > "${HOME}/.config/openshell/gateway.env"
-          podman info
-          curl -LsSf "${INSTALL_SH_URL}" | sh
-          openshell status
-          EOF
-
-      - name: Stop Fedora systemd container
-        if: always()
-        run: |
-          docker rm -f "${FEDORA_CANARY_CONTAINER}" >/dev/null 2>&1 || true
+          # The variables in the single-quoted script expand inside the guest.
+          # shellcheck disable=SC2016
+          nix run .#test-guest -- \
+            --distro "${TEST_VM_DISTRO}" \
+            "${configuration_args[@]}" \
+            -- \
+            env \
+              "OPENSHELL_CANARY_DRIVER=${TEST_VM_DRIVER}" \
+              "OPENSHELL_CANARY_INSTALL_SH_URL=${TEST_VM_INSTALL_SH_URL}" \
+            bash -lc '
+              set -euo pipefail
+              mkdir -p "${HOME}/.config/openshell"
+              printf "OPENSHELL_DRIVERS=%s\n" "${OPENSHELL_CANARY_DRIVER}" \
+                > "${HOME}/.config/openshell/gateway.env"
+              curl -LsSf "${OPENSHELL_CANARY_INSTALL_SH_URL}" |
+                OPENSHELL_VERSION=dev sh
+              openshell --version
+              openshell status
+            '
 
   ubuntu-snap:
     name: Ubuntu Snap