fix: repair CDI and verify allocation health with scoped device access

[antsok]

Summary

Repair the plugin-owned CDI specification before health advertisement and allocation, and refuse workloads when device usability or repair cannot be established. Give the trusted plugin explicit hypervisor access through a separate, non-privileged bootstrap device-plugin DaemonSet. Validate both its injected device and the host device path required for CDI injection.

Refs #13 and sokolaidev/maf-extensions#1423.

Changes

• Atomically repair missing, malformed or stale CDI, including kind, name, path, permissions, UID/GID and extra container edits. Bound UID/GID to the CDI uint32 field type, preserve the invalid-value fallback, refuse non-regular owned paths, and leave unrelated CDI files untouched.

• Run bounded KVM API-version/empty-VM checks. Bootstrap registers one hyperlight.dev/device-access allocation and returns only a read/write DeviceSpec for the detected hypervisor. The main DaemonSet requests it, and separately mounts the host device directory read-only at /host-dev for character-device and device-number validation. Neither component changes host device permissions or requires a privileged container.

• Keep caller cancellation separate from shared health. A subsequent probe waits for cancelled-helper cleanup within its two-second budget before starting a fresh check. A helper that cannot exit remains a bounded device failure, with at most one outstanding helper. Broadcast completed health transitions immediately to every ListAndWatch stream.

• Bound kubelet registration, stop failed servers before retrying, and distinguish gRPC liveness from readiness. Readiness requires successful registration, an active health stream and a completed successful check.

• Add regression coverage, bootstrap gRPC tests, and CI verification of main-plugin KVM access. CI compares host device mode, ownership and device number after cleanup, including when earlier integration steps fail.

Verification

• Linux Go 1.25: full go test -race -count=1 -timeout=90s ./..., go vet ./..., formatting check and static Linux build passed. The full suite ran as UID/GID 65534, including unwritable CDI storage and recovery.

• Both new regression tests failed against the previous code, then passed with the fixes: host-device disappearance must refuse allocation and publish unhealthy status; cancelling a helper must not poison the next independent check. Additional tests cover host-path symlinks/non-devices, device-number mismatch, recovery, and cancellation while waiting for helper cleanup.

• Both manifests and the workflow parse successfully; read-only host identity mounts, scoped device grants and the final metadata comparison were checked. The exact comparison commands accepted unchanged temporary-file metadata and rejected deliberate mode drift.

• Local WSL KVM: temporary character nodes reproduced a private device that remains usable after its host counterpart disappears. The updated helper refused missing/replaced host nodes and recovered after restoration. The real host device was unchanged.

• Earlier local WSL/runc verification showed that a host-device-directory mount alone denies KVM access, while an explicit grant for only KVM permits API-version and empty-VM creation with all process capabilities dropped. These were manually constructed OCI specs, not a Kubernetes/AKS deployment.

• Live Standard AKS KVM validation passed at 52bf3f203cfca36843afb36d18b0551b1f6ca00e: 12 positive restricted, non-root application runs, automatic missing/stale/malformed CDI repair, unhealthy allocation refusal, restoration after read-only storage and synthetic missing/unusable-device faults, registration recovery, and fresh execution after plugin/kubelet restart. Both plugin DaemonSets ran without privilege and with all capabilities dropped. A temporary privileged diagnostic pod was used for host fault injection and observation, then removed.

• During allocation-triggered repair, 6,813 concurrent CDI reads saw only complete stale or desired content, with zero read errors; repair was observed after 2.93 seconds. Unrelated CDI and real host KVM metadata were preserved. All 15 application pod UIDs were checked for surviving processes; application pods, ownership ledgers, temporary namespaces and the temporary node pool were cleaned up.

• Existing guests retained state through CDI faults and plugin restart. During a real kubelet restart, the plugin re-registered in 5.01 seconds and fresh execution passed, while the existing supervised session exited 70. That matches the downstream design's retirement on lost controller stream/lease; the exact retirement trigger was not captured. Session continuity across transport interruptions is separate downstream lifecycle work, not a demonstrated plugin recovery defect.

• Detailed AKS validation report, including configuration, fault mechanisms, cleanup and scope limits.

Notes for reviewers

The bootstrap design adds one small infrastructure container per enabled node and a second device-plugin socket/resource. Reserve hyperlight.dev/device-access for trusted infrastructure using cluster admission or quota policy; workloads continue to request hyperlight.dev/hypervisor. Bootstrap health describes available device access, while the main plugin validates host identity and KVM usability.

The deployment topology remains subject to upstream review. Live evidence covers one Standard AKS KVM node, not AKS Automatic, production capacity or uninterrupted sessions across kubelet restart. Not verified: KIND execution with this change or MSHV VM creation. MSHV retains a character-device/read-write-open check only. The downstream production source/image pin remains unchanged pending upstream acceptance. The exact-head PR Validation workflow requires maintainer approval; local and live checks are not a green CI claim.

[antsok]

Standard AKS verification of upstream PR #14

Verified on 2026-09-24 using source 52bf3f203cfca36843afb36d18b0551b1f6ca00e and candidate OCI digest sha256:b89483a3ed73853afb7f5070d1b53e2c9a8d6f77b1e1ff1ce04cfdc602ada9a1.

One temporary, tainted Standard_D4ads_v5 node ran AKS 1.35.7, Ubuntu 24.04.5, kernel 6.8.0-1067-azure and containerd 2.3.3-2. Both candidate DaemonSets ran without privilege, with all capabilities dropped, RuntimeDefault seccomp and no service-account token. Runtime inspection confirmed zero effective capabilities. A temporary privileged diagnostic pod performed fault injection, kubelet restart and host-process checks; it was removed during cleanup. No RBAC or admission-policy changes were made. Two advertised slots permitted an existing guest plus a fresh validation guest; this is not a capacity benchmark.

Check	Live result
Real guest execution	12 positive application runs passed, including state reset, identity rejection and fresh execution after disposal. Applications ran as UID/GID 65534 under Restricted admission.
Missing, stale, malformed CDI	Automatic exact-content repair; observed recovery in 22.3–32.2 seconds including polling overhead. A fresh guest passed after each.
Non-regular CDI path	Readiness failed and advertised capacity became zero. Removing the obstruction recovered without restarting the plugin.
Read-only CDI storage	Missing-file repair failed deterministically; liveness remained healthy, readiness failed, and kubelet refused a new allocation. Restoring exact bytes recovered without restarting the plugin.
Missing host identity	Removing a synthetic character-device entry from an isolated host-device view withdrew capacity; restoration recovered without restarting the plugin. The real host device was untouched.
Existing unusable device	A per-container /dev/null mapping at /dev/kvm, with matching synthetic identity, failed KVM_GET_API_VERSION. Allocation was refused; restoring the normal manifest allowed fresh execution.
Registration failure	An isolated empty socket directory caused bounded registration timeouts and no readiness. Restoring the kubelet socket mount recovered.
Restarts	Plugin pod replacement preserved the guest. A real kubelet restart re-registered the plugin in 5.01 seconds and allowed fresh guest execution, but the supervised existing guest exited 70 during the restart.
Atomic replacement during allocation	6813 concurrent reads observed only complete stale or desired files, with zero read errors. Allocation-triggered repair was first observed after 2.93 seconds, before the next periodic check; the new guest passed.
Preservation	Unrelated CDI sentinel unchanged; real host KVM UID/GID, mode and device number unchanged. Continuous guests retained state across CDI, device-view and plugin-restart faults. Two runs exited successfully. The third was interrupted by the real kubelet restart and exited 70; its counter reached 246 before the interruption.
Cleanup	15 application pod UIDs checked: no surviving host processes, application pods or owner ledgers. Both temporary namespaces and the temporary node pool removed. Original node labels and original pod UIDs/restart counts unchanged.

Runtime image: sha256:03a33cee7cc4b7f58e777adc33fab7690cf87e5c13f5cdaaac465a7ce62f5339 (maf-sandbox-hyperlight 0.5.0, matched Hyperlight SDK/backend/guest 0.7.0 and Python 3.13.12). No running test resources remain.

The local harness needed two timing corrections: wait for asynchronous node-capacity publication, and exclude terminating DaemonSet pods when selecting the replacement. Their cleanup paths passed; the affected recovery/restart checks subsequently passed. No product source was changed. The downstream session supervisor deliberately retires a session when its controller stream closes or its lease expires. Exit 70 during kubelet restart is consistent with that contract; the exact retirement trigger was not captured. The existing session was not resumed. Plugin re-registration, fresh allocation and exact-owner cleanup passed. This observation does not establish a device-plugin recovery defect; preserving sessions across transport interruptions would be separate downstream lifecycle work.

Scope limits: one Standard AKS KVM node, no AKS Automatic/KIND/MSHV validation, no production capacity claim, and no physical host-device removal or host permission changes. The production upstream pin remains unchanged pending upstream acceptance. The exact-head PR Validation workflow still requires maintainer approval; these live results do not make CI green.

[Copilot]

Copilot review overview

🟡 Changes recommended

Deployment manifests still need capability dropping and an explicit termination-log path.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2 · 2

Open (4)

• Drop all capabilities from the root plugin container · New
• Drop all capabilities from the root plugin container · New
• Drop all capabilities from the root bootstrap container · New
• Drop all capabilities from the root bootstrap container · New

What changed in this PR

This PR repairs CDI state and gates Hyperlight allocation on verified device readiness using scoped bootstrap access.

Changes:

• Adds bounded device probing, CDI reconciliation, cancellation-safe health tracking, and readiness checks.
• Adds bootstrap DaemonSets, host-device validation, regression tests, and CI verification.
• Requires capability dropping for plugin containers and an explicit termination-log path in the manifests.

File	Summary
docs/​architecture.md	Documents scoped access and readiness behavior.
device-plugin/​readiness_regression_test.go	Tests cancellation and host-device recovery.
device-plugin/​probe_linux.go	Adds bounded probing and device identity checks.
device-plugin/​main.go	Implements readiness, CDI repair, registration, and allocation.
device-plugin/​main_test.go	Adds CDI, readiness, registration, and bootstrap tests.
deploy/​manifests/​device-plugin.yaml	Updates DaemonSets with scoped access and health probes.
deploy/​local/​device-plugin.yaml	Mirrors deployment changes for local environments.
.github/​workflows/​pr-validation.yml	Adds readiness tests and device metadata verification.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[antsok]

Copilot review comments are addressed