clone/restore execution pays the child VM's cgroup quota — provisioning work throttled under load

[CMGS]

Summary

Since #183 the VMM is spawned via CLONE_INTO_CGROUP directly into cocoon.slice/vm-<id>.scope, so everything the VMM does from its first instruction runs under the VM's Guaranteed-at-N policy (quota = N×period, burst 0) and inside the #185 fence. That includes the restore/clone-time memory loading:

• copy mode: the eager multi-core memory copy executes inside the child scope's N-core quota;
• mmap mode (default): the clone window itself is thin, but the deferred first-touch page faults pay inside the same quota after claim.

Restore/clone execution is operator/control-plane work — the per-VM policy is meant to bound the VM's workload, not the act of provisioning it. The docs already state this philosophy explicitly (docs/vm.md): "the fence bounds the VMs; the caller's own work (clone, restore, the API consumer) is deliberately not cocoon's to manage" — and warn that AllowedCPUs=15 "would confine clone's multi-core memory restore to one core". The implementation currently does exactly that confinement for the VMM-side half of restore, via the child's own scope.

Measured

16-core bare-metal host (AMD 9700X), CH lane, 1-vCPU/512M guests, 20-VM CPU-burn storm (each guest spinning its vCPU), sequential timed clones from one snapshot, n=12 per cell, quiet controls taken. A/B/C = pre-cgroup master (07f1d69) / 3542c8a scopes only / 3542c8a + cgroup_cpus=0-14:

clone p50/P90/max (ms)	07f1d69	scopes only	scopes + fence
--restore-mode copy	390 / 530 / 547	330 / 609 / 717	516 / 1032 / 1348
mmap (default)	98 / 169 / 216	155 / 334 / 463	114 / 198 / 230

Copy-mode P90 roughly doubles once the child scope + fence apply — the eager copy that wants N cores gets exactly 1 core of quota with burst 0, placed inside the saturated fenced set. Quiet-host clones are unaffected (quota not binding). For contrast, the same round measured the fence's intended win on control-plane ops: vm list under the same storm went 62/133/147 → 5/5/5 p50/P90/max — the mechanism works as designed for steady-state isolation; it is specifically the provisioning window that lands on the wrong side of the boundary.

Options

1. Spawn the VMM outside vm-<id>.scope (parent slice or a transient provisioning scope) and migrate it into its scope once restore completes / the VM reports ready — restore then runs as caller-side work, matching the documented contract. One controlled migration per boot; keeps CLONE_INTO_CGROUP semantics for the steady state.
2. Keep CLONE_INTO_CGROUP but hold cpu.max=max during the restore window and arm Guaranteed-at-N at ready. Simpler; restore still respects the fence's placement, only the quota is deferred.
3. At minimum, document the copy-mode × Guaranteed-at-N interaction as a known caveat for dense hosts.

Option 1 also covers mmap's deferred first-touch cost only if migration happens after warm-up, which it can't reasonably — so for mmap the residual is inherent and worth a doc note either way.

[CMGS]

Recommendation: Option 2, with the mmap residual documented per option 3. The case against option 1 is convergence, not preference:

Option 1 creates a new state that nothing today converges. "VMM running in the wrong cgroup" (migration failed, or crash before migration) does not exist in the #182/#185 lifecycle — GC and converge only reason about scope existence. Supporting it means a new reconcile path that compares /proc/<pid>/cgroup against the expected scope and re-migrates, and it reopens exactly the post-fork attach window CLONE_INTO_CGROUP was chosen to eliminate. io_uring workers (PF_IO_WORKER) also have a history of quirks around cgroup migration that birth-into-scope sidesteps.

Option 2's failure window is already covered by the restore interruption protocol. Shape:

• clone/restore launches prepare the scope with cpu.max = "max <period>" — weight, period, fence, and placement all apply from the first instruction, so the eager copy competes fairly inside the fence rather than running unbounded; only the ceiling is deferred;
• after vm.resume succeeds, before Finalize, one cpu.max write arms the Guaranteed-at-N quota (burst follows, trivially valid while quota was max);
• plain boot keeps the current arm-at-spawn path — no memory loading, no window needed.

Failure analysis: the arming write fails → restore fails → AbortLaunch kills the VMM, no residue. cocoon dies between resume and arming → the VMM runs uncapped-but-fenced-and-weighted — which is precisely the state the existing restore tombstone/quarantine already refuses to boot and converges by re-restore or rm, both of which kill that VMM. Zero new mechanism.

The mmap deferred-first-touch cost is inherent to both options (faults happen after claim, inside the guest's lifetime) and is mild in the numbers (169→198ms P90) — a doc caveat, while copy mode is where the P90 doubling lives.

Acceptance draft:

• same 20-VM storm + sequential clone A/B, --restore-mode copy: P90 returns to the ~600ms class (scopes-only level or better, since the fence keeps a quiet core for the caller side);
• immediately after ready, the cap is armed: a CPU burn inside the fresh clone pegs at exactly N cores with cpu.stat throttling — no leak of the provisioning ceiling into steady state;
• kill -9 of cocoon between resume and arming → re-restore converges to the correct quota;
• plain vm run/vm start behavior byte-identical to today.

[CMGS]

Reviewed against master 3542c8a. The measured problem is real, and Option 2 is the right direction: keep birth-in-final-scope semantics, leave weight/fence/placement active, and defer only the hard ceiling during snapshot memory loading. I would make one sequencing correction before freezing the design:

Arm the final quota before vm.resume, not after it. Both CH and FC finish vm.restore / snapshot/load while the VM is paused and then issue an explicit resume. If cocoon is killed after resume but before the proposed arming write, the daemonized VMM survives in cpu.max=max. The restore tombstone is not sufficient convergence for that state: .restore-dirty is checked by PrepareStart, but resident supervision does not consult that file when adopting a live VMM and can reconcile the record to Running. Clone has a different recovery authority — its Creating placeholder is reclaimed through stale-create cleanup, which kills the orphan VMM — so the two paths should not be described as sharing the restore tombstone protocol.

Minimal sequence:

1. Launch directly into the final scope with cpu.max = "max <period>"; weight, fence, placement, and burst=0 apply from the first instruction.
2. Complete memory load and any required paused-state device work.
3. Arm the configured finite cpu.max, then the configured burst.
4. Only after arming succeeds, call vm.resume; Finalize remains unchanged.

This preserves the intended provisioning exemption without creating an uncapped-running state. An arming failure aborts a still-paused VMM; a process death before arming leaves no running guest workload; after resume, Guaranteed-at-N is already active. The writes are naturally retryable: a failure after the finite cpu.max write but before burst is safe, and retry converges.

I would update the fault-injection acceptance to cover death after load/before arm and after arm/before resume, plus verify that the first post-resume CPU burn is capped. The mmap first-touch residual remains inherent and should be documented as proposed. With this sequencing change, I support freezing #186.

[CMGS]

Correction accepted — the arm-before-resume sequencing is right, and it invalidates a claim in my comment: I asserted the restore tombstone covers the die-after-resume window, but .restore-dirty is only consulted by PrepareStart; resident supervision adopting a live VMM does not read it and would reconcile the record to Running, leaving an uncapped VM that nothing converges. And clone's recovery authority is stale-create reclaim (which kills the orphan), not the restore tombstone — the two paths don't share a protocol as I implied.

Arm-before-resume dissolves the whole class instead of patching it: every failure before arming leaves a paused VMM with no running guest workload (abort kills it; a cocoon death leaves it for the existing orphan/stale-create machinery), and after resume Guaranteed-at-N is already live — there is no uncapped-running state to converge at all. The write sequence (finite cpu.max, then burst) is idempotent and retry-converges as you note.

Final shape as frozen:

1. clone/restore launch into the final scope with cpu.max="max <period>" — weight, fence, placement, burst=0 live from the first instruction;
2. memory load + paused-state device work complete;
3. arm finite cpu.max, then burst;
4. vm.resume; Finalize unchanged. Plain boot keeps arm-at-spawn.

Acceptance updated per your list: kill after load/before arm → paused VMM, existing reclaim converges, no guest ran uncapped; kill after arm/before resume → same, and the scope already shows the finite quota; first post-resume CPU burn pegs at exactly N with cpu.stat throttling; storm A/B copy-mode P90 back to the ~600ms class; plain run/start byte-identical; mmap first-touch residual documented as inherent.

Taking this as frozen; ready to implement on your go.