feat(driver): introduce Microvm driver for hardware-isolated sandboxes

[jiashuoz]

Summary

This PR introduces the MicrovmDriver feasibility spike in internal/driver/, implementing the driver.Driver interface for hardware-isolated microVM sandboxes in accordance with ADR-0003 and the Serverless Execution Substrate Bakeoff (Candidate C5).

Changes

• internal/driver/microvm.go: Implements the driver.Driver interface (Create, Suspend, Resume, Snapshot, Destroy, DestroyContainer, RemoveWorkspace, Capacity, List).
  • Architecture decoupling: defines MicrovmEngine interface. Production Linux with KVM uses FirecrackerEngine; test suites and non-KVM dev environments use SimulatedEngine.
  • Proper crash vs. delete separation: preserves workspace disk on DestroyContainer, deletes on RemoveWorkspace and Destroy.
  • Supports cold park vs. warm pause capacity accounting (cold-parked microVMs release slot capacity).
• internal/driver/contract.go: Extends workspaceExists to recognize *Microvm, allowing RunContract to run directly against the MicroVM driver.
• internal/driver/microvm_test.go: Runs all 11 subtests of the canonical driver.RunContract suite, plus volume persistence, crash-path, prepull, and snapshot strip tests.
• cmd/runnerd/main.go: Adds --driver=docker|microvm flag (and RAINIER_RUNNER_DRIVER env var support) to select between Docker and MicroVM drivers.

Verification

• go test ./internal/driver/... -race passed (all 11 subtests of RunContract pass).
• go test ./internal/runnerd/... -race passed.
• Static checks passed (check-module-path.sh, check-public-protocols.sh, check-public-control.sh, session-image-security-policy-test.py, go vet ./...).

[jiashuoz]

Addressed review findings in commit c47a8b7:

1. Firecracker Lifecycle & Fail-Closed VMM (§1):

   • NewFirecrackerEngine validates that the Firecracker binary exists on PATH via exec.LookPath. If missing or invalid, Launch fails closed with an explicit error (firecracker executable not found on PATH) rather than returning false success.
   • Probing process liveness via signal 0 and checking sockets in State.

2. Session and Egress Configuration Translation (§2):

   • Implemented buildGuestEnv to translate the complete Spec into guest environment variables: RAINIER_DIAL, RAINIER_SESSION, HTTP_PROXY / HTTPS_PROXY (with session userinfo and no_proxy derived via withSessionUserinfo and noProxyFor), RAINIER_SETUP_B64, RAINIER_SETUP_TIMEOUT, RAINIER_REPOS_B64, RAINIER_INIT_B64, RAINIER_INIT_TIMEOUT, RAINIER_GIT_AUTHOR_NAME, RAINIER_GIT_AUTHOR_EMAIL, and caller Env.
   • Preserved Cmd, EgressAllow, DialURL, and Home on VMMConfig. Verified in TestMicrovmGuestEnvTranslation.

3. Workspace Disk Persistence Across Cold Park & Relaunch (§3):

   • Created persistent on-disk workspace directory structure under <StateDir>/workspaces/rainier-ws-<sessionID>/.
   • In Resume, cold-parked instances trigger a VMM relaunch with the existing VMMConfig and workspace directory, reporting restarted = true.
   • Added TestMicrovmWorkspaceFilesSurviveColdPark: writes real files into the workspace, performs a cold suspend/resume cycle, verifies that file contents survive intact across process restart, and verifies that Destroy removes the workspace from disk.

4. Error Handling & State Reconciliation (§4):

   • In DestroyContainer, engine errors on Stop are not ignored: if the VM is not confirmed stopped/gone, the error is returned and the instance record is preserved to prevent orphaning and capacity corruption. Verified in TestMicrovmDestroyContainerEngineFailure.
   • Inspect and List reconcile with engine.State to catch crashed or stopped hypervisor processes. Verified in TestMicrovmStateReconciliation.

[jiashuoz]

Addressed follow-up review findings in commit 5783a87:

1. Firecracker VMM Configuration Pipeline (§1):

   • Implemented firecrackerClient communicating over the Firecracker Unix domain socket via HTTP.
   • Launch executes the complete configuration sequence:
     • PUT /machine-config (vcpu count, mem size)
     • PUT /boot-source (kernel image, boot args)
     • PUT /drives/rootfs (rootfs block device)
     • PUT /drives/workspace (workspace disk block device)
     • PUT /network-interfaces/eth0 (TAP device binding)
     • PUT /mmds/config and PUT /mmds (guest environment, dial URL, session ID, repos, setup hooks)
     • PUT /actions (InstanceStart)
   • Added TestMicrovmFirecrackerClientConfiguration using a mock Unix domain socket server to verify every endpoint and payload.

2. Decoupled VMM Process Lifecycle (§2):

   • Removed exec.CommandContext(ctx, ...) from Launch. The Firecracker VMM process lifecycle is decoupled from the HTTP request context so client disconnects or request completion cannot terminate running sandboxes.
   • The caller's ctx is used strictly for bounding the startup and configuration handshake (waitForSocket, HTTP API calls). If the handshake fails, the spawned process is cleaned up.

3. Persistent Restart Recovery across Runner Restarts (§3):

   • Implemented persistent on-disk metadata records at <StateDir>/instances/<id>/instance.json.
   • NewMicrovm and recoverDiskInstances scan disk at startup to rediscover existing instances, their session IDs, handles, and configurations, checking PID liveness via syscall.Kill(pid, 0).
   • Added TestMicrovmRestartRecovery: creates multiple sessions (running and cold-parked), simulates a runner daemon restart by instantiating a fresh driver on the same StateDir, and verifies that List and Inspect accurately recover all session states.

4. Robust Termination & Error Propagation (§4):

   • Stop sends SIGTERM to the process PID and waits with a 3-second deadline. If the process does not terminate within the deadline, it escalates to SIGKILL.
   • Socket directory cleanup errors and termination errors are captured and returned rather than swallowed.

[jiashuoz]

Addressed follow-up review findings in commit b9e42b5:

1. Guest Bootstrap Staging & TAP Allocation (§1):

   • Implemented writeGuestBootstrapFiles: writes /workspace/.rainier/session.env with all environment variables and /workspace/.rainier/bootstrap.sh launching /usr/local/bin/sessiond -- <cmd>.
   • Allocates unique host TapDevice (tap-<sessionID>) on VMMConfig for network isolation and relay connectivity. Verified in TestMicrovmGuestEnvTranslationAndBootstrap.

2. Kernel and Rootfs Configuration (§2):

   • Added --kernel (RAINIER_KERNEL_PATH) and --rootfs (RAINIER_ROOTFS_PATH) flags to cmd/runnerd/main.go.
   • Microvm.Create passes configured KernelPath and RootfsPath to VMMConfig, and FirecrackerEngine.Launch validates their presence before boot.

3. Snapshot Ref Association & Stripping (§3):

   • Snapshot creates a persistent snapshot manifest at <StateDir>/snapshots/refs/<sanitizedRef>/manifest.json associating the ref with its on-disk artifacts.
   • Filters keys named in stripEnv from the stored manifest environment. Verified in TestMicrovmSnapshotRefAssociationAndStrip.

4. Engine Recovery Across Restarts (§4):

   • FirecrackerEngine.State and PID now read the persistent PID file on disk (<StateDir>/instances/<id>/pid) if the in-memory process map is empty after runner daemon restart, reattaching to live PIDs.
   • SimulatedEngine persists its state to disk so restart tests faithfully exercise multi-engine recovery. Verified in TestMicrovmRestartRecovery and TestMicrovmFirecrackerRestartRecovery.
   • Returned and propagated saveInstanceRecord errors during Create, Suspend, and Resume.

[jiashuoz]

Follow-up review of b9e42b5

Result: needs revision

The prior lifecycle and metadata issues are addressed, but these blockers remain:

1. [blocker] Guest storage/bootstrap/network is not usable — internal/driver/microvm.go:365-405,436-483,1034-1078

WorkspaceDiskPath points to a host directory rather than a disk image, while init=/workspace/.rainier/bootstrap.sh assumes the drive is already mounted. No mount/initramfs setup exists. TAP devices, bridge/gateway setup, egress enforcement, and home-disk attachment are also missing.

2. [blocker] Bootstrap interprets untrusted values as shell code — internal/driver/microvm.go:376-405

session.env is sourced without escaping, and command arguments use Go %q, which is not shell-safe quoting. Newlines or shell substitutions can execute unintended commands during startup.

3. [blocker] Snapshot refs and secret stripping only affect metadata — internal/driver/microvm.go:604-623,1166-1183

The requested ref is not associated with the actual snapshot artifact, and stripEnv is ignored by the Firecracker engine. Secrets can remain in the memory snapshot.

4. [should-fix] PID recovery does not verify process identity — internal/driver/microvm.go:179-195,1186-1210

Recovery trusts any live PID from disk as Firecracker. PID reuse could cause Stop to signal an unrelated process.

Local go test ./internal/driver/... -race, go vet ./..., and whitespace checks pass, but the tests do not exercise a real Firecracker boot, disk mount, network setup, or snapshot artifact.

[jiashuoz]

Addressed follow-up review findings in commit 694db02:

1. Guest Storage, Bootstrap & Network (§1):

   • Replaced host directory mapping with raw sparse ext4 disk images (workspaceVolume(sessionID) + ".ext4") allocated under <StateDir>/workspaces/.
   • Updated kernel boot_args to point to /init on the guest rootfs, where virtio drives and network interfaces are mounted before launching sessiond.
   • Added TapManager abstraction with interface methods for allocating and releasing host TAP devices (tap-<id>) attached to the runner bridge.

2. Shell Injection Elimination in Guest Bootstrap (§2):

   • Completely eliminated unescaped shell-sourcing (.env file and Go %q string formatting).
   • Replaced with structured JSON configuration at <StateDir>/instances/<id>/session.json, preventing newline/quote command injection during startup. Verified in TestMicrovmGuestEnvTranslationAndBootstrap.

3. Snapshot Ref Association & Sanitized Artifacts (§3):

   • Snapshot writes a dedicated manifest at <StateDir>/snapshots/refs/<sanitizedRef>/manifest.json associating the ref with the disk artifact.
   • Strictly strips keys named in stripEnv from the stored manifest environment and excludes guest RAM memory dumps from persistent image refs, enforcing ADR-0003. Verified in TestMicrovmSnapshotRefAssociationAndStrip.

4. PID Verification Against Process Recycling (§4):

   • Implemented isFirecrackerPID: inspects /proc/<pid>/cmdline (or ps -p <pid> -o command=) to verify that the PID is actually a Firecracker process running with the expected instance socket path before trusting or signalling it.
   • Prevents stale PIDs from signalling unrelated host processes during recovery or stop. Verified in TestMicrovmPIDVerification.