test(e2e): add reusable QEMU infrastructure for E2E tests

[drew]

Summary

Add reusable Nix and QEMU infrastructure for running OpenShell tests in disposable Linux VMs. The harness provides pinned native-architecture cloud guests, composable runtime and security configurations, artifact installation, predictable cleanup, and retained diagnostics for failures.

This is the base infrastructure PR for the VM-backed release and E2E test integrations in the child PRs below.

Stack

This is the parent PR.

• ci(release): run Linux release canaries in QEMU VMs #2472 — run Linux release canaries in QEMU VMs
• test(e2e): add VM-backed E2E suite runner #2473 — run E2E suites through the QEMU VM infrastructure

Related Issue

• feat: E2E and release testing improvements #2527
• Prototype alternative to feat(e2e): use xtask for host Linux workflows #2254.

Changes

• Add pinned Ubuntu 24.04, Fedora 44, CentOS Stream 10, and Rocky Linux 9 cloud-image definitions for ARM64 and AMD64
• Add a nix run .#test-vm interface with disposable QCOW2 overlays, cloud-init SSH bootstrap, hardware acceleration where available, cleanup, and retained debug state
• Add composable Ansible configurations for Docker, rootless Podman, and SELinux enforcing mode
• Support repeatable package installation and direct executable copying into guests
• Add automatic KVM-to-TCG fallback for Linux hosts without accessible hardware virtualization
• Add dpkg to the Nix development shell and keep Debian package creation artifact-only
• Document requirements, supported combinations, package workflows, lifecycle, and current limitations

Testing

• bash -n nix/vm/run.sh tasks/scripts/package-deb.sh
• shellcheck nix/vm/run.sh tasks/scripts/package-deb.sh
• Parsed all Ansible playbooks as YAML
• nix fmt reports no changes
• Evaluated the flake app for aarch64-darwin, aarch64-linux, and x86_64-linux
• CentOS Stream 10: installed rootless Podman and verified SELinux enforcing
• Rocky Linux 9: installed Docker and verified SELinux enforcing
• Fedora 44 ARM64: installed the release RPMs and reached the Podman-backed gateway with SELinux enforcing
• Ubuntu 24.04 ARM64: installed the release Debian package and reached the Docker-backed gateway
• mise run pre-commit passes
• Unit tests added/updated (not applicable to this test infrastructure)
• Automated E2E integration is added in the stacked child PRs

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture and VM usage documentation updated

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[drew]

gator-agent

PR Review Status

Validation: This maintainer-authored PR is project-valid as reusable QEMU/Nix test infrastructure for the release and E2E stack.
Head SHA: 52cb5e1b90ababfdf89a4ee95a37bb111509263f

Review findings:

• Warning: the OCI cache path pulls and boots a full guest disk from a mutable repository reference. Metadata and disk checksums protect transfer integrity, but not provenance. Before wiring this into trusted CI, require an org-controlled immutable registry policy or artifact signing/verification; otherwise the cache repository becomes a supply-chain execution surface.
• Three line-specific warnings are inline.

Docs: Fern docs/navigation are not required for this contributor/test infrastructure change; nix/test-guest/README.md and architecture/build.md cover the new workflow.

Next state: gator:in-review

[drew]

gator-agent

PR Review Status

Validation: This maintainer-authored PR remains project-valid as reusable Nix/QEMU test guest infrastructure for release and E2E follow-on work.
Head SHA: 49ae902c28f3c514535434e65bc4dc2c640be5da

Review findings:

• Warning: package verification coverage regressed when the Debian package smoke-test block was removed before an equivalent VM-backed CI replacement exists.
• Warning: OCI prepared-disk cache pulls still verify integrity but not publisher provenance; trusted CI use needs an immutable org-controlled reference or signature/manifest verification.
• Suggestion: nix/test-guest/run.sh should use the same explicit option-value validation pattern as cache.sh for a cleaner CLI failure mode.

Docs: Fern docs/navigation are not required for this contributor/test infrastructure change; nix/test-guest/README.md and architecture/build.md cover the workflow.

Checks: DCO and gate-status publication passed; Branch Checks and Helm Lint are still pending on /ok to test mirror validation. I am not triggering CI while review warnings remain unresolved.

Next state: gator:in-review

[drew]

/ok to test 9f24004

[drew]

gator-agent

PR Review Status

Validation: This maintainer-authored PR remains project-valid as reusable Nix/QEMU test guest infrastructure for release and E2E follow-on work.
Head SHA: 9f240044655387561c88b4b305e8770f88042765

Review findings:

• Warning: the SSH control socket path can exceed Unix socket path limits during cache builds because TMPDIR may be a deep cache staging path.
• Warning: local prepared-disk cache hits still trust existing metadata without revalidating the disk checksum or QCOW2 structure.

Docs: Fern docs/navigation are not required for this contributor/test infrastructure change; nix/test-guest/README.md and architecture/build.md cover the workflow.

Checks: DCO and gate-status publication passed; Branch Checks and Helm Lint are still pending on /ok to test mirror validation. I am not triggering CI while review warnings remain unresolved.

Next state: gator:in-review

[drew]

/ok to test 86cf640

[github-actions]

Label test:e2e applied for 86cf640. Open Branch E2E Checks, find the run for commit 86cf640, and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[drew]

gator-agent

PR Review Status

Validation: This maintainer-authored PR remains project-valid as reusable Nix/QEMU test infrastructure for release and E2E work.
Head SHA: 3d48a73ab33a8f9554e13d8b320bfe79088d17ea

Review findings:

• Three line-specific warnings are inline: host-side scp option injection, Podman relay readiness, and mutable runtime-image inputs.
• Suggestion: add focused mocked shell tests for hostile argument shapes and cache metadata/checksum/digest rejection paths; keep full QEMU integration coverage in the stacked child PRs.

Durable feedback: Prior trusted maintainer resolutions and waivers remain honored. These findings are new or materially distinct: the scp concern is host-side option parsing rather than the resolved guest-destination quoting issue, and the two Podman findings arise from this head's new hook.

Docs: Fern docs/navigation are not required for this contributor/test infrastructure; nix/test-guest/README.md and architecture/build.md cover the workflow.

Checks: DCO and required-gate publication pass. Branch Checks, Helm Lint, and E2E remain pending, but author changes are needed before pipeline watch resumes.

Next state: gator:in-review

[drew]

gator-agent

PR Review Status

Validation: This maintainer-authored PR remains project-valid as reusable Nix/QEMU test infrastructure for release and E2E work.
Head SHA: 420ddce5e9777a3ac55c25d649852ec56f0f02ae

Review findings:

• One blocking QCOW2 validation control-flow defect and two cache safety warnings are inline.
• The blocking validator defect can accept prohibited disk structure when Bash suppresses errexit in conditional call contexts.

Durable feedback: Prior trusted maintainer resolutions and waivers remain honored. The findings here are distinct from the resolved checksum, existing-tag validation, and earlier runner/Podman findings.

Docs: Fern docs/navigation are not required for this contributor-facing test infrastructure; nix/test-guest/README.md and architecture/build.md cover the workflow.

Tests: No local tests were run. Please add focused shell coverage for validator failure propagation, the decompression bound, and post-push digest verification.

Checks: Required Branch Checks and Helm Lint are pending mirror validation, but author changes are needed before pipeline watch resumes.

Next state: gator:in-review

[elezar]

I think the minimal merge-ready scope for this parent PR is:

• Keep the generic VM provisioning and rootless Podman configuration here. A developer should still be able to run nix run .#test-guest -- --distro <distro> --with podman, get Podman installed, have the user socket enabled, and use the VM interactively.
• Move commit 3d48a73a (feat(test-guest): prepare Podman E2E artifacts) into test(e2e): add VM-backed E2E suite runner #2473. The checkout supervisor image and host/network relay are specific to wiring OpenShell E2E through these guests, so the Alpine pinning and relay-readiness findings can be resolved there without blocking adoption of the provisioning layer.
• Fix the core scp source-path issue in this PR by canonicalizing --install and --copy inputs before passing them to scp.

With that scope, I would not require this parent PR to add the QEMU CI matrix or complete Podman-backed OpenShell E2E integration; those belong in the stacked follow-up.

[drew]

I think the minimal merge-ready scope for this parent PR is:

• Keep the generic VM provisioning and rootless Podman configuration here. A developer should still be able to run nix run .#test-guest -- --distro <distro> --with podman, get Podman installed, have the user socket enabled, and use the VM interactively.
• Move commit 3d48a73a (feat(test-guest): prepare Podman E2E artifacts) into test(e2e): add VM-backed E2E suite runner #2473. The checkout supervisor image and host/network relay are specific to wiring OpenShell E2E through these guests, so the Alpine pinning and relay-readiness findings can be resolved there without blocking adoption of the provisioning layer.
• Fix the core scp source-path issue in this PR by canonicalizing --install and --copy inputs before passing them to scp.

With that scope, I would not require this parent PR to add the QEMU CI matrix or complete Podman-backed OpenShell E2E integration; those belong in the stacked follow-up.

Yea good call. Done.

[drew]

gator-agent

PR Review Status

Validation: This maintainer-authored PR remains project-valid as the common Nix/QEMU test-guest foundation tracked by #2527.
Head SHA: 2e3c89eaadfc4b5a4f67faddc2c841242ca8a22f

Review findings:

• One blocking QCOW2 validation defect, four cache safety/reliability warnings, and one Podman dependency suggestion are inline.
• General warning: add hermetic, no-QEMU shell regression tests with stubbed qemu-img, oras, and zstd for validation failure propagation, download/decompression bounds, publication races, stale-lock recovery, and leading-hyphen source canonicalization.

Durable feedback: Three previously raised cache findings remain actionable. The author marked those threads resolved, but there is no fixing reply or maintainer waiver and the current code still contains the defects; author-only thread resolution does not waive them.

Trusted maintainer feedback: Thanks @elezar, I checked the scope split you described. The Podman E2E artifact/relay work is removed from this parent, generic rootless Podman setup remains, and --install/--copy sources are canonicalized as requested.

Docs: Fern docs/navigation are not required for this internal contributor/test infrastructure; nix/test-guest/README.md and architecture/build.md cover the workflow.

Checks: DCO and required-gate publication pass. Branch Checks, Helm Lint, and E2E are pending, but author changes are needed before pipeline watch resumes.

Next state: gator:in-review

[elezar]

From my side the basic "give me a VM" workflow is already quite useful and there's more value in getting it in and addressing findings as we use it to unlock specific use cases than trying to solve for everything now.

Happy to approve pending the minor cleanups.

[drew]

/ok to test b75d508