multikernel: support isolated SR-IOV VF assignment - #4
Draft
nickolaev wants to merge 9 commits into
Draft
Conversation
nickolaev
force-pushed
the
sriov-vf-assignment
branch
2 times, most recently
from
July 22, 2026 19:44
b715339 to
0ff04f5
Compare
|
Hi @nickolaev Nice work, thanks for the PR! I finally have time to look into it. A few high-level comments:
|
Author
|
@congwang-mk thanks for the high-level review. Indeed, this is a naive implementation of "let's make ICMP pass"; It is very fragile as it is. The questions you asked are setting a proper direction. I have some ideas; let me try to clean things up and will ping you once I have a better patch series. |
nickolaev
force-pushed
the
sriov-vf-assignment
branch
2 times, most recently
from
July 30, 2026 16:41
8d8960b to
4e05ab7
Compare
Author
|
@congwang-mk I have updated the commit history with the revised work and the PR message reflects the new state. |
nickolaev
force-pushed
the
sriov-vf-assignment
branch
from
July 30, 2026 18:02
4e05ab7 to
f5c1493
Compare
A spawned kernel cannot safely rediscover the resource layout of a device that remains physically attached to the host. Make the instance description the owned source of PCI identity, BAR, and host-bridge metadata. Snapshot device resources and ECAM descriptors into the Multikernel device-tree transport. Parse, clone, and release that data with the instance lifecycle, and restore it through KHO without retaining live host PCI objects. This establishes the transport and ownership model only; later patches perform the exclusive VF lease. Signed-off-by: Nikolay Nikolaev <nicknickolaev@gmail.com>
PCI ECAM discovery and restoration are architecture policy. Keeping that policy in generic Multikernel or PCI probe code makes the transport depend on x86 implementation details. Provide a locked MMCONFIG region iterator, snapshot only windows that cover assigned devices, and restore those windows from the instance description during x86 platform setup. Select the restored ECAM operations before the spawned kernel scans PCI. Generic MMCONFIG code remains unaware of Multikernel instances. Signed-off-by: Nikolay Nikolaev <nicknickolaev@gmail.com>
A spawned kernel must not probe or mutate functions retained by the primary kernel. Filtering in the generic PCI probe path is too late and places Multikernel policy in shared PCI code. Wrap the x86 configuration-space operations so unassigned functions are rejected before hardware access. Present the assigned identity from instance metadata, retain only bridge traversal needed to reach an assignment, and restore the recorded BAR resources during early fixup. This keeps assignment policy in Multikernel and removes the special case from drivers/pci/probe.c. Signed-off-by: Nikolay Nikolaev <nicknickolaev@gmail.com>
Moving a PCI inventory entry between instances does not coordinate with the live host device or prevent two instances from requesting the same function. Introduce an assignment object for each leased device and serialize lease creation and release. Validate the root inventory against the live PCI function, accept only SR-IOV VFs, preserve the original host-driver state, move the inventory only after the lease commits, and use PCI bus notifications to fail an instance if its PF or VF disappears unexpectedly. The PF stays bound to its host driver. Only the VF changes ownership, and instance teardown returns it to the root inventory. Signed-off-by: Nikolay Nikolaev <nicknickolaev@gmail.com>
Instance creation reserves memory, CPUs, PCI devices, host-bridge metadata, and platform devices. Returning an error after any one of those transfers can otherwise expose a partially populated instance and leak resources from the root. Validate configuration counts and lists before transfer, acquire each resource class in a fixed order, and unwind every completed step in reverse order. Centralize release so create failure, instance deletion, and final reference teardown share the same all-or-nothing semantics. Balance references acquired for remote memory add and remove operations on every success and error path so resource hotplug cannot pin a deleted instance. Signed-off-by: Nikolay Nikolaev <nicknickolaev@gmail.com>
Passing physical host bridges to a spawned kernel leaves bridge configuration shared and mutable even when endpoint probing is filtered. The primary and secondary could then program the same routing state independently. Treat ECAM descriptors as discovery metadata only. Create a synthetic root for each required segment and bus range, scan assigned endpoints through the filtered configuration operations, and never enumerate the physical bridge functions in the spawned kernel. Fail the spawned-kernel PCI initialization when root metadata is missing, overlapping, or cannot be mapped so a partial topology is never exposed. Signed-off-by: Nikolay Nikolaev <nicknickolaev@gmail.com>
A config-space allowlist does not isolate DMA. Letting a leased VF retain the host default domain would allow it to reach memory outside the spawned instance, while disabling the IOMMU is not an acceptable assignment model. Require IOMMU support for VF leases and accept only singleton groups with isolated MSI delivery and compatible reserved regions. Build a host-owned paging domain, map only the instance memory regions with their allowed permissions, bind the VF to a driver-managed-DMA assignment driver, and attach the group before committing the lease. Any validation, mapping, binding, or attach failure unwinds the domain and leaves the VF with its host driver. Balance target-instance references when IOMMU state rejects memory changes. The spawned kernel receives no ownership of the host IOMMU programming. Signed-off-by: Nikolay Nikolaev <nicknickolaev@gmail.com>
Returning a VF while it can still issue DMA races the IOMMU teardown and host-driver reprobe. Unexpected PF or VF removal must also stop an active instance instead of silently losing its assigned device. Clear bus mastering, wait for pending transactions, and issue function-level reset while the assignment domain is still attached. Then detach and free the domain, restore the saved driver override and host driver, and only afterwards return the inventory to the root. Lease-loss notifications force an active instance to halt and mark it failed. Rollback entries that were prepared but never committed skip device quiesce and driver restoration, releasing only their prepared IOMMU resources. Signed-off-by: Nikolay Nikolaev <nicknickolaev@gmail.com>
Spawn kernels do not own the PIT, PIC, or IO-APIC resources. The X86_SUBARCH_MULTIKERNEL platform quirk therefore leaves timer_init and wallclock_init as no-ops so an instance never programs host timer hardware or requests IRQ0. setup_percpu_clockev() initializes the local APIC timer for instance ticks. Keeping global_clock_event unset bypasses LAPIC timer verification, whose fallback path requires the unavailable legacy IRQ0. Document that timer path next to the platform quirk. Signed-off-by: Nikolay Nikolaev <nicknickolaev@gmail.com>
nickolaev
force-pushed
the
sriov-vf-assignment
branch
from
July 30, 2026 22:05
f5c1493 to
b2d1b55
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Multikernel instances currently lack a complete PCI ownership model. A spawned
x86 kernel needs enough platform and device state to enumerate and operate its
assigned device without exposing or disturbing devices retained by the primary
kernel.
This series adds end-to-end assignment of an SR-IOV VF to a spawned kernel while
the primary retains the PF and enforces DMA isolation.
The series:
The resulting ownership model is:
Assignment requires an active hardware IOMMU, a singleton IOMMU group, isolated MSI delivery, and a DMA aperture
capable of covering every instance memory region. Instance memory cannot be changed while an IOMMU-backed VF lease is active.
This work can be demonstrated using the
mainbranch here: https://github.com/nickolaev/multikernelThe QEMU harness verifies VF enumeration and traffic in the spawned kernel, continued PF operation in the primary, IOMMU containment,
exclusive ownership, hostile lifecycle operations, repeated create/delete cycles, driver restoration, and fail-closed handling of unexpected VF loss.
It has been tested with
qemu-system-x86_64using Q35, IGB SR-IOV, Intel IOMMU, interrupt remapping, and strict IOMMU mode. It has not yet been tested on realSR-IOV hardware.