Repository navigation
fix(sandbox): 恢复 sandbox.get 的 daemon_capacity 读取，并修隔离连接回池问题（#543） - #68
Merged
mengmengmengqiang merged 6 commits intoNov 11, 2026
Merged
mengmengmengqiang merged 6 commits into
mengmengmengqiang merged 6 commits into
Conversation
…phemeral-AI-Lab#540) `sandbox.get` now carries `daemon_capacity`: the daemon's `backend.daemon.connection_failures` read model, the one `layerfs doctor` explains and its fix text already pointed at. The field is best-effort and bounded to 2 s: the query runs in its own thread behind a hard wall, so a daemon that accepts and stalls cannot hold the read, and a daemon that does not answer (stopped, gone, at no known endpoint, or holding no vacant control slot) leaves the field out instead of failing `sandbox.get`. The field is only added for a running Sandbox, and nothing is started to find out. The one query itself moved out of the doctor into `layerfs_sandbox::daemon::capacity` (`endpoint`, `read`, `OPERATION`, `WAIT`), so the doctor and the read model share one implementation: where the daemon is, how long an answer may take, and what an unanswered query is are decided once. Tests: layerfs-api --test sandbox_capacity 2 passed — a stand-in daemon's model reaches `sandbox.get` with its admission counts and retained failure record, and a daemon that reads the query and never answers leaves the field out after the bounded wait while the get still answers. layerfs-control --test doctor unchanged and passing (6 passed). Clippy -D warnings clean on layerfs-api, layerfs-control, layerfs-bridge and layerfs-sandbox; no test failures in those four or layerfs-daemon. Production LOC: 156991 -> 157011 (delta +20); api 4966 -> 4975 (+9), control +11. Counted with tools/production_loc.py over core/crates/*/src, engine/crates/*/src, control/crates/*/src and api/crates/*/src (tests, docs, manifests and generated artifacts excluded).
…ads (Ephemeral-AI-Lab#540) A daemon whose pool has no vacant slot answers the next connection with its advisory notice before any handshake. This pins what a client makes of it: `Dial::open_within` refuses `Capacity` on resource `daemon_control_slot` with the notice's vacant-slot and retained counts, retryable because nothing was attempted; a daemon that is still starting is `Unavailable` with admission `starting` instead, so a full pool and a starting one stay distinguishable. Measured against a real macOS daemon before this test existed (LP543-PATCH §4): with eight admitted clients holding every pool slot, the ninth raw connection read `notice version=1 reason=NoPoolSlot retained=0 pool_vacant=0 reserved_vacant=1`. Tests: layerfs-sandbox --test pool_admission 2 passed. Clippy -D warnings clean. Production LOC: 157011 -> 157011 (delta 0); a test-only commit.
…ral-AI-Lab#543) `sandbox.get` shows `daemon_capacity`, the daemon's `backend.daemon.connection_failures` read model, but only through the half of it that survived the `layerfs_v2_0_0` merge (82165e6): the API layer parsed an endpoint and dialed it itself. That half reaches neither a `docker` nor an `apple-container` Sandbox, whose daemon is behind a machine's worker; it dials a forwarded Sandbox's old port after its machine stopped or its forward exited because it skips `direct::check`; and it opened a second control connection per read, so one `sandbox.get` took two of the daemon's slots and, with seven slots held by other clients, reported no capacity at all - the field the doctor's own fix text tells an operator to read. This restores the operation author's design from 7cfe446, 52f9c16 and 31375c9 (all present before that merge, lost when it resolved Ephemeral-AI-Lab#540's conflicts): the read is a host operation again (`Call::Capacity`, `backend.daemon.connection_failures`), routed like the Workspace list it follows - `direct::capacity` for a forwarded Sandbox, its machine's worker for the rest - so `native`, `lima`, `docker` and `apple-container` all answer it, and `api/crates/layerfs-api/src/sandbox/get.rs` is a thin shell over the typed `Got` again. It is asked only for a `running` Sandbox whose machine and forward were reached, on the connection the Workspace list lent from the Sandbox's pool (one connection, one slot), within 2 s for admission and for the exchange, and fenced at that deadline (`daemon/fence.rs`, the pattern the doctor's bounded observation uses): a daemon that trickles its handshake or its reply is shut down rather than left holding a thread, a connection or a slot. Doctor behavior and wording are unchanged. Its own query now ends its control session instead of dropping the socket, which the daemon logged as an unexpected end. Docs: `docs/architecture/api/sdk/sandbox/get.md`, `docs/architecture/api/sdk/sandbox/README.md`, `docs/architecture/control/routing/daemon-channel.md`. Production LOC: 157011 -> 157130 (delta +119) Scope and method: first-party production Rust/SQL/JavaScript in core, engine, control and api (tests, fixtures, tools and docs excluded), counted with `tools/production_loc.py` at the parent commit 8978c2e and the committed tree. control 38406 -> 38524 (+118), api 4975 -> 4976 (+1); core 39749 and engine 73881 unchanged. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
…phemeral-AI-Lab#543) `sandbox.get`'s `daemon_capacity` (Ephemeral-AI-Lab#540) had one test of the shown answer and one of the 2 s bound, both on a forwarded Sandbox. This covers what the restored route decides: - `api/crates/layerfs-api/tests/sandbox_capacity.rs`: the Workspace list and the capacity read share one connection (the stand-in daemon counts them), a stopped machine with no worker record, an ended forward with a recorded process, and a Sandbox with no id are never dialed and answer in well under a second, and a daemon that reads the query and never answers still returns the get at its bound. - `control/crates/layerfs-sandbox/tests/capacity_read.rs`: a query ends its control session (`EndSession`) instead of dropping the socket, a trickling daemon is fenced at the deadline (its own write fails, so the connection was shut down), two pooled reads take the one connection the pool holds, and the host operation a machine's worker runs (`dispatch::call`, `Call::Capacity`) returns the model. - `control/crates/layerfs-sandbox/tests/host_frame.rs`: the `sandbox.host` frame carries `backend.daemon.connection_failures` and reads back as `Call::Capacity`. Production LOC: 157130 -> 157130 (delta 0) Scope and method: first-party production Rust/SQL/JavaScript in core, engine, control and api (tests, fixtures, tools and docs excluded), counted with `tools/production_loc.py` at the parent commit 88c88fb and the committed tree; this commit changes tests only, so the production total is unchanged. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
…ld table (Ephemeral-AI-Lab#543) The `daemon_capacity` row linked an issue URL that the guides do not have a record of; it cites the issue the way the surrounding documents do, `(Ephemeral-AI-Lab#540)`. Production LOC: 157130 -> 157130 (delta 0) Scope and method: first-party production Rust/SQL/JavaScript in core, engine, control and api (tests, fixtures, tools and docs excluded), counted with `tools/production_loc.py` at the parent commit 811f59e and the committed tree; this commit changes one documentation line, so the production total is unchanged. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
…eral-AI-Lab#543) `capacity::lent` (the `sandbox.get` capacity read, Ephemeral-AI-Lab#543) arms a fence on the pool connection it just borrowed, and `Lease::drop` decides reuse from `Control::is_failed()`. That answer was only `Control.failed`, which a fence's end never sets: `CloseHandle::close` (layerfs-bridge) sets the channel's own shared `failed` bit and shuts its socket down, and the `Control` above it learned of that only when a call it sent subsequently failed. The window, in order: 1. `trace::control_within` borrows a connection and `Fence::channel` arms it at the read's deadline. 2. `exchange` returns `Ok(answer)`: `Control.failed` is still false and the channel is healthy. 3. Before `drop(fence)` sends its stop, the fence thread's deadline expires and `CloseHandle::close()` quarantines the channel and shuts both directions down. `drop(fence)` joins that thread, so the close has completed before the lease is returned. 4. `Lease::drop` asks `Control::is_failed()`, hears false, and pushes the dead connection back into `pool.idle`. 5. The next call in the same long-lived process (a machine's worker, an SDK binding) pops it. `Sender::send` refuses at `shared.ready()` with `Quarantined` before writing a byte, but `Control::call_with` had already set `attempted = true`, so the caller is told `phase: Send, attempted: true` for an operation that never left. `capacity::lent` widened that window: its deadline was taken before `control_within(controls, WAIT)`, which re-reads the clock and may use the whole 2 s, so a read admitted at the boundary armed a fence already 0 ms from firing - a shutdown racing a reply that took microseconds. Fix, at the three points the ordering above crosses: - `layerfs-bridge`: `Connection::is_quarantined()` reads the one shared bit that every direction and close handle write. - `Control::is_failed()` is that bit or `Control.failed`, and `call_with` and `install_file` refuse on it before setting `attempted`. A quarantined connection now reports `phase: Encode, attempted: false` (nothing can have taken effect), and `Lease::drop` closes it instead of lending it again. - `capacity::lent` checks the window left after admission and refuses (`Unavailable`, nothing sent) instead of arming a spent deadline, which would shut a healthy connection down for nothing. Tests (`control/crates/layerfs-sandbox/tests/capacity_read.rs`): a fenced pool connection is not lent again (the next read opens the second connection and answers instead of reporting a quarantine), and a quarantined connection's call is `Encode` with `attempted: false`. With `is_failed` reverted to its previous form both fail; the first reproduces the reviewed symptom exactly (`phase: Send, attempted: true, cause: Channel(Quarantined)`). Docs: `docs/architecture/control/routing/daemon-channel.md` (the pool's drop line and the `Control` row) and `docs/architecture/api/sdk/sandbox/get.md` (what the fence and the spent admission window decide). Production LOC: 157130 -> 157143 (delta +13) Scope and method: first-party production Rust/SQL/JavaScript in core, engine, control and api (tests, fixtures, tools, docs, manifests and generated artifacts excluded), counted with `tools/production_loc.py` at the parent commit de009a8 and the committed tree. engine 73881 -> 73884 (+3, `duplex::Connection::is_quarantined`), control 38524 -> 38534 (+10, `daemon/capacity.rs` and `daemon/channel.rs`); core 39749 and api 4976 unchanged. Reference absent (retired R9c; last counted 65,417 at tree 498dd19).
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
mengmengmengqiang
merged commit Nov 11, 2026
4f4906a
into
tokenaissance:fix/540-daemon-slots
2 of 4 checks passed
Collaborator
|
谢谢，我把这个 PR 合进本地 先说一处更正 —— 之前有一版评审意见认为「全满时按上界返回 我用一个只写 真正值得改的是满池时计数全丢，链路我逐段确认过：
也就是说，恰恰在最需要这三个计数的时候，它们只剩一句人话。两个不改 SDK 形状的修法： 反过来不建议：让 另附一条实测现象（不是缺陷）：满池时 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
给 Ephemeral-AI-Lab#543 补几处缺口，目标分支是 Ephemeral-AI-Lab#543 本身（
fix/540-daemon-slots），合进来后 Ephemeral-AI-Lab#543 即可继续评审。改了什么
sandbox.get的daemon_capacity（88c88fbec）。这条读取路径作者在7cfe4460e、52f9c1666、31375c942已经实现，82165e666合并时遗落了，导致 PR 正文说有、源码里却没有。本提交按原设计恢复，不是新设计。811f59e02）；文档表格里直接引用 Daemon control slots leak: a connection that fails after a request stays Retained forever, and the daemon then refuses every connection Ephemeral-AI-Lab/layerfs#540（de009a8b8）。49e540386）：Fence 关闭时同步Control.failed；从池里借连接后再检查 deadline。补两条回归测试。验证
sdk_blame_coarse是上游 fix(content): align store-service line_alignment::exact with the #517 unique-line fallback (sdk_blame_coarse regression on v2.0.0) Ephemeral-AI-Lab/layerfs#638 的问题，已由 fix(content): align store-service line_alignment::exact with the #517 unique-line fallback Ephemeral-AI-Lab/layerfs#639 修复；本分支尚未合入新的 v2 头，合入后即消失。sandbox.get仍返回daemon_capacity（pool_vacant:0, pool_running:8）；释放一个槽立即恢复；全满时按上界返回down_reason，不挂起。说明
sandbox.close后消失，这是现设计。5002922ea、8978c2eb4）的提交消息未整理，按不改写历史的惯例保留。Refs Ephemeral-AI-Lab#540