vm: cpuset fence — reserve host cores from the VM population (#184) - #185
Merged
Conversation
A saturated fleet leaves clone/wake execution, vk, and the OS nothing to run on: field data shows CH clone p95 24.9s / max 69.2s under load. cgroup_cpus writes a one-time cpuset fence on the parent so VM threads (including virtio/io-wq workers that CH's vCPU affinity cannot reach) physically cannot occupy reserved cores; --cpuset-cpus adds explicit per-VM placement. The kernel treats cpuset.cpus as a request and silently degrades ungrantable sets, so subset validation is cocoon's: fence against the ancestor effective set, placements against the parent's, and a fence shrink is refused while a live placement conflicts. A cleared config resets a stale fence once (read-gated, controller never disabled). Disk queue_affinity targets clamp to the effective set on every path that builds or patches disks — launch, restore, clone, DiskAttach.
…veCPUs single source placeScope enabled +cpuset on the parent alone — with no fence ever set the ancestor chain never delegated cpuset and a bare --cpuset-cpus failed; controller enablement now runs the same forEachLevel walk in ensureParent for fence and placement alike. Fence and placement writes compare parsed sets, not raw strings: the kernel echoes cpu lists canonicalized, so a non-canonical config string would re-run the scan and the serialized cpuset write on every launch. The placement-wins precedence lives once in cgroup.EffectiveCPUs; clone derives it once and threads it through resume opts.
…ent trims One forEachLevel walk enables cpu and cpuset together — half the subtree_control reads per fenced launch and one combined write (one cgroup_mutex acquisition) on first setup. patchDisks builds the affinity slice once per call instead of per disk. Six comments fail the deletion test: restated godoc, duplicated WHYs, narration.
41 files, net -166 lines; content unchanged, only 80-column wrapping merged to match the one-physical-line style the newer packages use.
Contributor
Author
Hardware round — 16-core testbed, binary pr185-01aad9f, regression against the merged #182 behaviorPositive matrix ran cocoon as root with a root-level parent (production shape). One structural note confirmed on the way: non-root + a root-level parent cannot work — clone3's common-ancestor check lands on the root cgroup — so non-root remains delegated-user-slice territory (as in the #183 round), where cpuset is typically not delegated; that path correctly fails preflight instead of degrading.
Environment fully torn down (scopes 0, test slice removed, isolated root deleted). |
The fence bounds the VMs; the caller's share is set on its own systemd unit — same cgroup files, plus the reserved-core-as-fast-lane guidance over pinning.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #184 (frozen after one review round).
What
Machine fence (
cgroup_cpus, e.g."0-14"): one read-gatedcpuset.cpuswrite on the #182 parent slice; every VM scope inherits, so the fleet — including the virtio/io-wq workers CH's vCPU affinity cannot reach — physically cannot occupy reserved cores. Overcommit happens inside the fence; reserved cores stay available to the host, vk, and the clone path (the motivating tail: CH clone p95 24.9s / max 69.2s under load).Per-VM placement (
--cpuset-cpuson run/clone): explicit opt-in pinning, snapshot-knobs-are-records semantics (clone from flags alone, restore keeps the VM's).Enforcement is cocoon's, not the kernel's
cpuset.cpusis a requested set — the kernel silently degrades ungrantable requests instead of failing. So: the fence validates against the ancestor's effective set, placements against the parent's, and a fence shrink is refused while a live VM's placement conflicts. Comparisons use parsed sets because the kernel echoes cpu lists canonicalized ("0-3,4-7"reads back"0-7") — raw-string comparison would defeat the read-gates and re-issue the globally-serialized cpuset write every launch.Convergence
Clearing
cgroup_cpusresets a stale fence once (read-gated; controller never disabled). Empty config on a fresh host writes nothing — byte-for-byte #182 behavior.queue_affinity clamp
Disk queue affinity used to pin queues to host CPUs 0..N-1 unconditionally; targets now clamp to the effective set (placement > fence > all) on every path that builds or patches disks: launch args, restore's config patch, clone's config patch + hot-added cidata/data disks, and runtime DiskAttach.
Not in this PR
Hardware acceptance (mpstat core-idle proof, clone-latency A/B under saturation, out-of-fence rejection, rollback convergence) runs as a follow-up round per the issue's list.