[mengmengmengqiang]

Ready for review — not a draft.

Base: layerfs_v2_0_0, not main. #540's affected code exists only on the v2.0.0 branch: api/crates/layerfs-control/src/daemon/pool.rs is absent from main, and this PR is a follow-up on the v2 daemon/control line (the same base as the other v2.0.0 PRs). A main-based PR would carry all 499 v2 commits instead of this fix.

Status on head 37cc7711a: GitHub reports the PR MERGEABLE / CLEAN, and no CI lane runs for this PR any more — ci.yml and format.yml skip a pull request whose base is layerfs_v2_0_0 (branches-ignore, owner decision 2026-11-07), so run 41800502973 on 4dbbcb7a3 is history, not a verdict on this head. The checks stated below were run locally on 37cc7711a (macOS; Linux-only legs marked NOT RUN). No human review has been submitted yet.

The head is 37cc7711a, 21 commits ahead of layerfs_v2_0_0 and 0 behind it (base head bdeb9e91f). The earlier run's one failure, sdk_blame_coarse, is fixed by #639, which is now in this branch. #68 (jiangkoumo) is merged into this branch: it restores sandbox.get's daemon_capacity and keeps a fenced control connection out of the pool.

Review focus

• A retained slot no longer owns capacity. Slots::finish now always leaves its slot Vacant and converts a failure that used to keep custody into a compact record in the bounded store; every such failure is also logged. Check that no result path can leave a slot out of Vacant once its worker has exited, and that a stale or duplicate SlotTicket finish changes nothing.
• The Capacity notice. When no slot is free the daemon writes one fixed advisory pre-handshake LAYERFS-CAPACITY frame and closes; the client maps it to Capacity { resource: "daemon_control_slot", retained, pool_vacant, reserved_vacant, retryable }. Legacy EOF still maps to Capacity with retained: null, a reset maps to Unavailable, and a random 24-byte record is never accepted. Check the notice is treated as advisory and unauthenticated, and that it admits nothing.
• The doctor query. layerfs doctor asks only Sandboxes whose record says they run and whose daemon endpoint is known for backend.daemon.connection_failures, inside a hard 2 s bound; an unreachable daemon is a warning with ok still true, and nothing is started or written.
• The 2 s close budget. All Workspace status probes inside one non-force sandbox.close share one total 2 s deadline (api/crates/layerfs-control/src/sandbox/serve.rs:43); unmount keeps 5 s, and force skips the probe. Check it is a close-phase budget, not a whole-close deadline.
• The test-coverage gap. cargo test -p layerfs-daemon --lib is 0 tests on this head: the inline daemon suites this fix needs were removed by the product-boundary guard, and nothing in-tree replaces them. The Docker E2E below proves slot reclamation, not the retained-record path. Review the diff for the invariants rather than treating green CI as coverage.

What changed in this revision

The initial reclaim fix is extended to the full slot-recovery design:

• Strict slot classes — Pool (8) plus Reserved (1, installer-only today: it is admitted only while the daemon is in InstallPending / Installing, and AdmissionClass::Lease is unused placeholder code). The reserved slot is not a recovery or lease slot, so a pool-exhausted sandbox.close cannot borrow it; giving it a recovery role is follow-up issue daemon: bound an admitted control slot's lifetime and keep recovery admissible when the pool is exhausted (#540 follow-up) #547. Admission is class-scoped, so ordinary connections cannot take it; each admission carries a generation-checked SlotTicket, and a stale or duplicate finish changes nothing.
• Panic and spawn safety — the worker runs under catch_unwind and owns a SlotGuard; exactly one finish runs on the normal, error, panic, dropped-thread and spawn-failure paths, recording a synthetic bounded cause.
• Richer retained records — class, generation, owner (instance:pid:thread), visible count, expires_in, recorded and the bounded failure list; the ring pre-reserves capacity (64 / 1 h) and is reaped lazily. A record is still never a slot.
• Precise Capacity notice — when no slot is free the daemon writes a fixed advisory pre-handshake LAYERFS-CAPACITY frame before closing. The client maps it to Capacity { resource: "daemon_control_slot", retained, pool_vacant, reserved_vacant, retryable: true }. Legacy EOF still maps to Capacity with retained: null; resets map to Unavailable; a random 24-byte record is never accepted. The notice is unauthenticated, advisory, and admits nothing.
• Host/operator query — backend.daemon.connection_failures is dispatched to the host (including forwarded Sandboxes) and sandbox.get exposes it best-effort as daemon_capacity with a bounded 2 s wait; the payload is nested admission (pool_vacant, pool_running, reserved_vacant, reserved_running) / retained (limit, ttl_ms, count, dropped, recorded, failures). The read was implemented in 7cfe4460e, 52f9c1666 and 31375c942 (then at api/crates/layerfs-control/src/sandbox/get.rs); the merge 82165e666, resolving the Split the control layer into one crate per concern (layerfs-project#101) #601 control split, rebuilt that file from the base copy and dropped this branch's hunk, so the new control/crates/layerfs-sandbox/src/get.rs had none of it. [v0.1.3] Optimize git_tool_workflow: cold reads, metadata traversal, and backing round trips #68 (88c88fbec) restores it on this head.

That precise Capacity is what the dial reports (Dial::open_within). At the product surface it is not a refusal: direct::unanswered re-codes any refusal that is not Busy/StaleSandbox to Unavailable and drops detail, direct::list reports Unavailable as Unreached, and sandbox.get shows it as down_reason — text only, with the notice's counts in the prose and daemon_capacity omitted. The capacity read cannot fill it either: it needs the same control slot the full pool has none of, so the notice is the only source. Measured on this head with a stand-in daemon that writes only the notice: sandbox.get returns after the 60 s admission window with state: running, workspaces: [], the counts inside down_reason, and no daemon_capacity.

• One close deadline — all Workspace status probes share one total 2 s close deadline (5 s for unmount), and force close skips the probe so it can stop a refusing daemon.

Tests are stated once, below: on this head cargo test -p layerfs-daemon --lib is 0 tests — the daemon-side invariants still have no in-tree test and are being restored in the stacked follow-up. What this head does carry from #68 is client-side coverage: the pre-Hello capacity notice and its counts (layerfs-sandbox/tests/pool_admission.rs), the bounded and pooled read, the fence and quarantine of a failed control connection (capacity_read.rs), and when sandbox.get shows or withholds daemon_capacity (layerfs-api/tests/sandbox_capacity.rs). Neither set asserts the daemon-side "a retained failure holds no slot" invariant.

Refs #540: do not close it on merge; issue #547 tracks the residual — an admitted in-flight worker still has no deadline or reclaim verb, and with the reserved slot installer-only a saturated control plane can still require a --force close and lose uncommitted Workspaces.

Doctor capacity wiring (delta)

On top of 52f9c1666 (commit c9457e3d4):

• Live, read-only query — layerfs doctor now asks each Sandbox whose record says it runs and whose daemon endpoint is known (a machine forward, else the endpoint the start recorded in control.json) for backend.daemon.connection_failures, the read model sandbox.get exposes as daemon_capacity. A Sandbox with no recorded endpoint is skipped; nothing is observed, started or written.
• Strictly bounded — Dial::open_within caps the connect, the Noise Hello and one exchange at 2 s, and a hard 2 s wall around the whole probe means an endpoint that accepts and then stalls returns a warning in about 2 s, not after a 10 s connect timeout or a 60 s admission wait.
• Warnings, not errors — an unreachable daemon is a warning naming its endpoint and the operation (backend.daemon.connection_failures), with ok still true; the doctor stays read-only and does not fail on an idle or stopped daemon. A capacity refusal is a warning naming Capacity on resource daemon_control_slot with its pool_vacant / reserved_vacant / retained counts.
• Capacity model — a successful answer adds a warning for retained connection failures (retained <count> of <limit> ... <dropped> dropped) and, when no pool slot is vacant, for the exhausted pool (pool_vacant / pool_running / reserved counts), both pointing at backend.daemon.connection_failures and sandbox.get's daemon_capacity.
• The report schema is unchanged: every new finding is a warning on the sandbox entity, so no schema redesign was needed.

Tests for this delta: the doctor capacity tests live in api/crates/layerfs-control/tests/doctor.rs (a fake authenticated Noise daemon answers Hello and backend.daemon.connection_failures); on the final head cargo test -p layerfs-control --test doctor → 9 passed (below), including a stalled endpoint bounded to ~2 s with no home writes and no query for a Sandbox without a recorded endpoint. cargo test -p layerfs-cli --test doctor → 2 passed, and cargo check -p layerfs-control -p layerfs-api -p layerfs-cli → clean.

Root cause

Application::serve gave each worker one of the daemon's fixed control slots. When a worker failed after a request had been decoded, the slot became Slot::Retained(cause) and stayed there:

• only Ok results, and Channel/Io failures with no request and no received bytes, released the slot;
• a failure after a decoded request (for example, the client disappeared before it could read the reply, so answer_call fails) was retained;
• take_connection_failure was the only path from Retained back to Vacant, and no product/operator call made it;
• the retained failure was never logged.

Each such client death permanently cost one of the 8 ordinary control slots. After 8, the accept loop closed every new socket and sandbox.get/close/workspace.* failed Busy after 60 s; only killing the daemon recovered. The client message attributed this to "other processes hold all 8 connections", which was not knowledge the client had.

Fix

1. Slots do not retain failures. Slots::finish always leaves the slot Vacant; a retained failure is converted to a compact record in a bounded RetainedFailures store. Admission capacity can no longer leak.
2. Every retained failure is logged to stderr / daemon.log in the same ConnectionFailure { ... } style as released failures. The line names the slot, received bytes, request kind, whether the reply was sent, the bounded cause, and fence/diagnostic errors.
3. The store is bounded: 64 records, 1 h TTL, oldest-first eviction with a dropped count. It is exposed through the existing daemon operator/control path as backend.daemon.connection_failures, returning admission { pool_vacant, pool_running, reserved_vacant, reserved_running }, retained { limit, ttl_ms, count, dropped, recorded, failures }, and each failure record.
4. The client refusal is precise. A daemon that closes before Hello now yields Capacity with detail.admission = "no-vacant-slot" (the accept loop's capacity refusal) or Unavailable with detail.admission = "connection-failed" for BrokenPipe/ConnectionReset, plus detail.observed with the I/O kind. It no longer claims other processes hold the connections.
5. sandbox.close phase stop is bounded. Its Workspace status refresh now uses a 2 s control-admission wait (Controls::lend_within) instead of the ordinary 60 s wait, so a refusing daemon returns an exact refusal promptly instead of holding the close's op intent.

Tests

What runs on this head

• cargo test -p layerfs-daemon --lib → 0 tests. The daemon crate has no inline #[cfg(test)] mod tests left: the product-boundary guard (tools/check_product_boundary.py) forbids test-only attributes in product src/, and the daemon tests that asserted this PR's invariants lived there. The crate compiles clean, but a clean build is not coverage.
• The in-tree suites that remain pass on 37cc7711a: cargo test -p layerfs-sandbox --test pool_admission (2 passed), --test capacity_read (6 passed), --test host_frame (5 passed), cargo test -p layerfs-api --test sandbox_capacity (5 passed); cargo fmt --all -- --check clean, check_product_boundary.py --skip-sdk PASS (1349 files), check_fuser_integrity.py exit 0, cargo clippy -p layerfs-sandbox -p layerfs-api --all-targets -- -D warnings clean. Previously on 4dbbcb7a3: cargo test -p layerfs-bridge --test admission --test capacity_model (4 passed), cargo test -p layerfs-bridge --test duplex (9 passed), cargo test -p layerfs-control --test doctor (9 passed), cargo test -p layerfs-api --test workspace_names (2 passed), cargo test -p layerfs-persistence --test sqlite_publication (13 passed).
• cargo +1.85.1 fmt --all -- --check → clean (pinned 1.85.1); cargo +1.85.1 clippy -p layerfs-control -p layerfs-daemon -p layerfs-bridge --all-targets -- -D warnings → clean, -p layerfs-api -p layerfs-persistence --all-targets → clean. The full workspace clippy was not run locally.
• python3.12 -B tools/check_product_boundary.py --skip-sdk → PASS: scanned 1238 production Rust/SQL/JavaScript files.

The coverage gap, and where it is being restored

The deleted inline tests were the only direct automated assertions of: a retained failure not consuming a slot (real Application + TCP + Noise, 12 failing requests), finish freeing a retained failure, stale/duplicate finish, guard drop and spawn failure finishing exactly once, the pre-Hello capacity notice, the 16-way concurrent failure storm, and the ring's limit/TTL/eviction/cause bounds. Nothing in-tree asserts those daemon-side invariants today; #68 (merged into this branch as 4f4906aef) adds client-side assertions for the pre-Hello notice and the connection fence, not for slot retention. The Docker E2E below exercises the released-after-logging class (retained.count stays 0), not the retained-record path, and the E2E scripts are not committed.

Issue #547 tracks the restoration of this coverage on the stacked branch fix/547-slot-regression-tests; its PR will be opened separately: package-level tests under engine/crates/layerfs-daemon/tests/ that drive a real daemon process and real TCP clients, so the guard is satisfied without test-only product surface or #[path] recompiles of private modules. This PR does not wait on that PR for the leak fix, but the gap is part of this PR's accounting, and green CI does not close it.

Risk

• Retained failures no longer keep an in-memory Box<ConnectionFailure>. The record keeps the deciding cause, request kind, reply state, and error text, but not the full decoded request/manifest or body bytes. The previous custody was never readable by any product path; the daemon log and operator query now are.
• Slot reuse is now immediate. A slot is only reused after its previous worker has exited (serve joins it before spawning the new worker), so no two workers share a slot.
• The new operator method is additive; existing control traffic is unchanged.
• The 2 s close-status wait is shorter than the ordinary admission wait. A healthy daemon accepts in milliseconds; a daemon that cannot admit now fails the close with its exact cause instead of stalling it.

Limitation / follow-up

• The store is in memory and bounded, so records are lost on daemon restart (as the old retained custody was). The daemon log is the durable evidence; the ring is the query cache. A persistent operator record would be a separate change.
• The client still cannot prove why a daemon closed a connection before Hello; it can only distinguish the clean-close/UnexpectedEof capacity signal from a transport failure. The daemon now exposes its slot/failure state, so an operator can confirm it.
• The admitted-slot lifetime/recovery-class residual is tracked by issue daemon: bound an admitted control slot's lifetime and keep recovery admissible when the pool is exhausted (#540 follow-up) #547: an already-admitted worker still has no deadline, and the reserved slot is installer-only, so the --force last resort remains until that follow-up lands. Do not close Daemon control slots leak: a connection that fails after a request stays Retained forever, and the daemon then refuses every connection #540 for it.

Real Docker E2E (both local engines)

Both runs use real daemons, real SIGKILLed clients, and the same 22-assertion probe. The invariant: a client that dies after its request was sent frees its control slot; a brand-new client is admitted; commit/unmount/close succeed; data survives; the 8 pool slots stay accounted for; one daemon.log line per failure.

OrbStack 29.8.2 — PASS 22/22, 9/9 rounds

• Sealed Linux bundle build-linux-4fa4864778644346b6aee8de91010a7e, tools/worktree.py run.
• Each round: fresh client sends workspace.commit (64 MiB change), gets SIGKILLed at +150 ms after the daemon admits it; commit completes, slot returns 295–766 ms later, exactly +1 ConnectionFailure line, a new client process answers sandbox.get in 1.9–6.8 ms.
• Slot table after every death: pool_vacant 6 + pool_running 2 = 8, reserved_vacant 1, reserved_running 0; no false "other processes hold all 8". The reserved count is the installer-only slot and stayed vacant throughout; it is not a recovery reserve and nothing borrowed it.
• commit/unmount/remount/close all succeed; 64 MiB cksum identical; close → NoSuchSandbox.

Docker Desktop 29.9.2 — PASS 22/22, 9/9 rounds

• Product macOS-client topology on the host, Docker Desktop worker via bridge/host.docker.internal; macOS bundle build-macos-adf82922369f41ffa2189792546da006.
• Container list during the run contains lfs-882b74ef-slots540 and lfs-882b74ef-docker; cleanup 204.
• Slot freed 722–1765 ms after the kill; +1 log line per round; fresh get 6.3–15.6 ms; same pool accounting, with the installer-only reserved slot still vacant; 64 MiB cksum preserved.

Harness caveat

The pinned Linux-container harness cannot drive machine docker on Docker Desktop: the v2 worker runs with --network host and Desktop does not expose random published loopback ports to that namespace (fixed ports and bridge + host.docker.internal work; minimal repro included). This is a harness limitation, not a #540 product failure. Docker Desktop evidence therefore uses the product's macOS-client topology.

Note on retained.count = 0

A SIGKILLed client whose request was decoded leaves a plain channel EOF: the daemon writes the reply to the dead socket, reads EOF, frees the slot, and logs one ConnectionFailure. That class is released-after-logging, not stored in the bounded RetainedFailures store; the store/retention path has no in-tree test on this head (see the coverage gap) and is the reason retained.count stays 0. The E2E proves slot reclamation, not a retained record.

Install-path retained record — not reachable in this build

We attempted the requested install-path retained-record E2E on both OrbStack and Docker Desktop. Both fail for a product-path reason, not nondeterminism:

• project.init / store.push write the project record with an installed manifest; sandbox::serve::start_claimed builds DaemonStore::Sqlite { install: Some(existing) }, and the launcher opens the already-pushed Store (setup.opens_existing()), skipping install.
• On both engines the target /layerfs-store/store.sqlite exists before the container is created (same inode as the worker's pushed file); no store.sqlite.installing transfer file appears; the SIGKILLed create leaves only a channel EOF (original_kind=None), and retained.count stays 0.
• The install protocol (InitialRecord::Install, install: None) is constructed only by daemon unit fixtures, never by a product launch. Install failures are held as startup custody (startup.install_failure, phase retained), not written to the bounded RetainedFailures store.

So the caveat cannot be removed in this build. Making it testable needs a product path that streams an install, or propagating install failures into the retained record path; both are out of scope for this PR.

CI fix (#543) — head 20d0d48a3

The fmt and guards and clippy -D warnings failures are fixed, and the read model is now one typed contract. All changes are on this Branch only (#544/#542 untouched).

Guard fmt and guards

python3 -B tools/check_product_boundary.py --skip-sdk reported 32 test-only attribute in product source lines from inline #[cfg(test)] mod tests in six product files. All inline test modules are gone from product src/:

• Kept and relocated to package tests/ (public APIs only): layerfs-bridge admission encoding/classification → engine/crates/layerfs-bridge/tests/admission.rs; the capacity DTO's round-trip/tolerance → engine/crates/layerfs-bridge/tests/capacity_model.rs; doctor capacity warnings → api/crates/layerfs-control/tests/doctor.rs (a fake authenticated Noise daemon answers Hello and backend.daemon.connection_failures, covering healthy, retained + exhausted pool, and a Capacity refusal through the public doctor() call). No test-only public API, no #[path] recompilation of private source, no #[allow].
• Deleted (private internals, impossible through a public API without new test-only surface): the pool admission unit tests in daemon/pool.rs, doctor/capacity.rs, and the daemon's application/{connection,owner,retained}.rs unit tests. Honest coverage loss: those deleted daemon tests were the only direct tests of retained-failure slot release, the dropped-SlotGuard path, the concurrent failure storm, the pre-Hello capacity notice, and the ring's limit/TTL/cause bounding. The daemon's own --lib target is now empty (0 tests); the specific invariants are covered by review and the real Docker E2E matrix below, not by an automated in-tree test, until the follow-up restores them. The pool's classification messages (EOF → Capacity with unknown retained, reset → Unavailable, notice → exact retained) also lose their unit tests; sandbox.get/doctor still cover the refusal shape indirectly.

Clippy -D warnings

• Removed the never-constructed NotAdmitted::Starting variant and its match arms (api/crates/layerfs-control/src/daemon/pool.rs); AdmissionReason::Starting (the advisory notice) is still produced and handled.
• Removing that error revealed a second, branch-introduced clippy::large_enum_variant on Finished::Retained(RetainedFailure) in layerfs-daemon/application/owner.rs (not present on the base). The record is now boxed, as clippy suggests.

Typed capacity model (clean-architecture delta)

• New engine/crates/layerfs-bridge/src/control_capacity.rs: CapacityModel (AdmissionModel + RetainedModel + typed failure records), parse(&SdkValue) and to_sdk(), plus the single CAPACITY_OPERATION constant. It is a plain DTO — no trait, repository, or test-only surface.
• The daemon producer (application/connection.rs) now builds that model instead of hand-assembling SdkValues; sandbox.get parses it once and re-publishes exactly the typed shape; doctor/capacity.rs parses once and reads typed fields. The duplicated "backend.daemon.connection_failures" literals in doctor, sandbox.get, sandbox.dispatch, daemon.operator, and the daemon producer now all use CAPACITY_OPERATION.
• RetainedModel::exhausted/AdmissionModel::exhausted keep the doctor logic small; no probe/findings file split was needed.

Verification on this head

• python3.12 -B tools/check_product_boundary.py --skip-sdk → PASS: scanned 1238 production Rust/SQL/JavaScript files.
• cargo +1.85.1 fmt --all -- --check → clean (pinned 1.85.1).
• cargo +1.85.1 clippy -p layerfs-control -p layerfs-daemon -p layerfs-bridge --all-targets -- -D warnings → clean; -p layerfs-api -p layerfs-persistence --all-targets → clean (the two edited test crates).
• cargo +1.85.1 test -p layerfs-bridge --test admission --test capacity_model → 4 passed; --test duplex → 9 passed.
• cargo +1.85.1 test -p layerfs-control --test doctor → 9 passed (6 existing + 3 new capacity tests).
• cargo +1.85.1 test -p layerfs-daemon --lib → 0 tests (inline unit tests removed; compiles clean).
• cargo +1.85.1 test -p layerfs-api --test workspace_names → 2 passed; cargo +1.85.1 test -p layerfs-persistence --test sqlite_publication → 13 passed.
• The full workspace Clippy was not run locally; the touched crates and both edited test crates were.
• The daemon E2E/FUSE matrix was not re-run (Linux-only; this environment is macOS).

Test-job baseline evidence (pristine layerfs_v2_0_0 @ d2c3f2c97)

Run from a clean git worktree add of the base commit, before any #543 edit:

• cargo test -p layerfs-api --test workspace_names → FAILED, 0 passed / 2 failed: both tests panicked on machine native's recorded daemon /nonexistent is not there (Unavailable where the test expected Exists / NoSuchSandbox).
• cargo test -p layerfs-persistence --test sqlite_publication → FAILED, 12 passed / 1 failed: wal_full_settings_and_unavailable_engine_are_explicit panicked at sqlite_publication.rs:63, left: None, right: Some(BackendUnavailable).

Both are stale test fixtures, not product regressions, and were fixed in a separate test-only commit 20d0d48a3 with no product behavior changed:

• workspace_names: the machine record now points at a placeholder daemon file (never launched, because the Sandbox has no daemon keys) instead of /nonexistent, so the call reaches the name/NoSuchSandbox assertions the test is about.
• sqlite_publication: removed the leftover BackendUnavailable assertion that selected the fake PostgreSQL backend removed in The connect layer: machines, Sandboxes and Stores as typed, extensible units #535; the test keeps asserting the real SQLite profile. It was renamed wal_full_settings_are_explicit.

After the fix both commands pass (2/2 and 13/13).

Diagrams

Architecture — final implementation, typed layerfs-bridge::ControlCapacity consumed by doctor and sandbox.get:

Recovery sequence — 8 failures, no daemon kill, slot freed, fresh client admitted:

[Mola-maker]

Hey @mengmengmengqiang, PR #543 is currently conflicting with recent layerfs_v2_0_0 updates. Could you rebase onto the latest v2 base when convenient? Thanks!

[mengmengmengqiang]

CI note

The single failing check on this PR is not caused by these changes:

layerfs-store-service::sdk_blame_coarse::lines_between_distant_edits_keep_their_origin

It is a pre-existing base regression from #517 (commit ca0d60a77), which fixed layerfs-content's diff but not layerfs-store-service's line_alignment::exact. No layerfs-store-service file is touched in this PR. Tracked in #638.

This PR's own checks pass: fmt, clippy -D warnings, product-boundary guard, fuser integrity, daemon/bridge/control-doctor tests.

[jiangkoumo]

@mengmengmengqiang 你好，我给这个 PR 补了几处缺口，开在你的分支上：tokenaissance#68

主要是：sandbox.get 的 daemon_capacity 读取在 82165e666 合并时遗落了（7cfe4460e 等提交里原本已实现），现按原设计恢复；另外修了隔离连接会被放回池的问题，并补了测试。Linux 全量 fast 1839/1840（唯一失败 sdk_blame_coarse 来自上游 #638，#639 已修）、slow 146/146、mount 45/0，真实 daemon 满池场景也验证过。方便时看一下，有问题我再改。

[mengmengmengqiang]

Problems this PR (and its follow-ups) fix

This PR is the base of the daemon-slot stack: #543 -> #650 -> #651 -> #652 -> #655. Together they close:

• Retained-slot leak (Daemon control slots leak: a connection that fails after a request stays Retained forever, and the daemon then refuses every connection #540). A connection that fails after a request stays Retained forever; eight of them make the daemon refuse every connection. Fixed here: one-finish protocol, Slots::finish always Vacant, bounded RetainedFailures store.
• sandbox.get's daemon_capacity regression. Implemented in 7cfe4460e / 52f9c1666 / 31375c942, dropped by the 82165e666 merge (the Split the control layer into one crate per concern (layerfs-project#101) #601 control split rebuilt the moved file from the base copy). Restored by [v0.1.3] Optimize git_tool_workflow: cold reads, metadata traversal, and backing round trips #68, included here.
• A fenced control connection leaking back into the pool. A fence can close the channel after a reply but before drop(fence); Control.failed was never set, so the next caller would borrow a dead connection. Fixed here.
• An admitted slot has no lifetime (daemon: bound an admitted control slot's lifetime and keep recovery admissible when the pool is exhausted (#540 follow-up) #547, follow-ups). After decode the socket timeouts are cleared and SlotState::Running has no deadline; a parked worker holds a slot forever. test(547): restore the daemon slot regression suite and structure slot custody (slices 1–2) #650 restores the deleted regression tests and refactors; feat(547): recovery-class reserved slot and idle reclamation (slices 3–4) #651 adds idle reclamation and the recovery class; feat(547): in-flight slot expiry, backend.daemon.reclaim and typed admission reasons (slices 5–6) #652 adds in-flight expiry and backend.daemon.reclaim.
• Recovery required destroying the data plane. With the pool full, a non-force sandbox.close failed and force lost uncommitted Workspaces. feat(547): recovery-class reserved slot and idle reclamation (slices 3–4) #651 admits one connection into the reserved slot for recovery methods only.
• The full-pool sandbox.get lost its structured counts (sandbox.get: daemon_capacity is absent exactly at a full daemon pool; counts only in down_reason #649). The counts existed only as prose in down_reason. fix(sandbox): carry a full-pool capacity notice into sandbox.get (#649) #655 adds a partial down: { code, resource, pool_vacant, reserved_vacant, retained } beside down_reason, without faking daemon_capacity.

Position in the stack: base. Merge order is #543 -> #650 -> #651 -> #652 -> #655; each follow-up is stacked on the previous one and retargeted to layerfs_v2_0_0 as the one below it merges.

Why split into separate PRs

• One invariant per PR, each with its own tests. The reviewer sees one delta and the behavior it changes; a regression can be reverted without unwinding the rest.
• Test debt first (test(547): restore the daemon slot regression suite and structure slot custody (slices 1–2) #650). The behavior changes in feat(547): recovery-class reserved slot and idle reclamation (slices 3–4) #651/feat(547): in-flight slot expiry, backend.daemon.reclaim and typed admission reasons (slices 5–6) #652 are guarded by an in-tree suite that previously ran zero tests; landing the tests before the behavior keeps them honest (they fail on the base).
• Stacked so the order is the argument. test(547): restore the daemon slot regression suite and structure slot custody (slices 1–2) #650 (structure) -> feat(547): recovery-class reserved slot and idle reclamation (slices 3–4) #651 (recovery class + idle) -> feat(547): in-flight slot expiry, backend.daemon.reclaim and typed admission reasons (slices 5–6) #652 (in-flight expiry + reclaim) -> fix(sandbox): carry a full-pool capacity notice into sandbox.get (#649) #655 (the full-pool visibility fix). Each builds on the previous contract, so each review only has to accept one step.
• sandbox.get: daemon_capacity is absent exactly at a full daemon pool; counts only in down_reason #649/fix(sandbox): carry a full-pool capacity notice into sandbox.get (#649) #655 is deliberately separate. It is a visibility fix on top of the slot-lifetime contract (surface the notice counts), not a redesign of fix(daemon): reclaim control slots after retained connection failure (#540) #543; coupling it would make an already-large review harder.