Repository navigation
fix(daemon): retry detach from Retained custody and add sandbox.destroy (#538) - #640
Open
mengmengmengqiang wants to merge 4 commits into
Open
mengmengmengqiang wants to merge 4 commits into
mengmengmengqiang wants to merge 4 commits into
Conversation
mengmengmengqiang
added a commit
to tokenaissance/layerfs
that referenced
this pull request
Nov 12, 2026
…data plane (Ephemeral-AI-Lab#538) A forced unmount whose one plain detach the kernel refused `EBUSY` left a Retained(Detach) custody that was retryable (Ephemeral-AI-Lab#640), but `sandbox.close` still treated every Retained Workspace as "recoverable" and deleted the whole Sandbox data plane without re-attempting the detach. Close now re-attempts the clean detach of each retryable Retained Workspace first (one normal unmount on the daemon, which re-runs only the recorded detach and never repeats the abort write), and refuses `Busy` naming the held Workspaces while a holder is still alive. `sandbox.destroy` remains the explicit data-plane-only delete and is unchanged. - unmount_guard::check_all now also reports which Retained Workspaces are retryable, and retry_held re-attempts each one (treating an already-gone `Missing` reply as detached). - lifecycle::close and direct::held re-attempt before the MountsActive admission and refuse `Busy` while any holder persists.
…oy (Ephemeral-AI-Lab#538) When a forced unmount stops at Detach with the kernel refusing the one plain detach (EBUSY), the daemon kept Retained(Detach) custody and every later unmount/force re-reported it without re-attempting the detach, so a Workspace became unrecoverable once the holder exited. - Keep the forced teardown's Undrained connection as typed evidence (Evidence::Undrained) so a later terminal call can recover its session. - Add KernelSession/Undrained::retry_detach: re-attempt only the recorded EBUSY detach after a fresh local drain; the abort write is never repeated. - workspace.unmount and ForceUnmount now take a retryable retained Detach custody out (Service::take_retryable) and re-attempt the detach (Service::retry_detach), continuing Revoke/Close on success and re-keeping the exact custody with the new disposition when still held. - The Sandbox-side unmount guard records retryable custody (a forced detach refused EBUSY) and re-sends to the daemon instead of re-reporting it, so the retry actually reaches the daemon. - sandbox.destroy: an explicit operator/CLI verb that deletes only one Sandbox's data plane (runtime stop + delete, then its folder) without asking the daemon or settling commands, for an invisible namespace-copy holder (Ephemeral-AI-Lab#460) that exec records cannot see.
…nmounted (Ephemeral-AI-Lab#538) A later workspace.unmount on a Retained(Detach) custody re-attempts the detach and, once the holder is gone, must answer Reply::Unmounted — the control channel has no Unmount->ForceUnmounted correlation. retry_detach now returns the drained connection and the caller completes it: a force through force_completed (ForceUnmounted), a normal unmount through close (Unmounted). The FP-33 test now asserts the re-report while held and the Unmounted recovery once the holder exits.
mengmengmengqiang
force-pushed
the
fix/538-custody-retry
branch
from
November 12, 2026 04:12
e64ed57 to
4020e90
Compare
mengmengmengqiang
added a commit
to tokenaissance/layerfs
that referenced
this pull request
Nov 12, 2026
…data plane (Ephemeral-AI-Lab#538) A forced unmount whose one plain detach the kernel refused `EBUSY` left a Retained(Detach) custody that was retryable (Ephemeral-AI-Lab#640), but `sandbox.close` still treated every Retained Workspace as "recoverable" and deleted the whole Sandbox data plane without re-attempting the detach. Close now re-attempts the clean detach of each retryable Retained Workspace first (one normal unmount on the daemon, which re-runs only the recorded detach and never repeats the abort write), and refuses `Busy` naming the held Workspaces while a holder is still alive. `sandbox.destroy` remains the explicit data-plane-only delete and is unchanged. - unmount_guard::check_all now also reports which Retained Workspaces are retryable, and retry_held re-attempts each one (treating an already-gone `Missing` reply as detached). - lifecycle::close and direct::held re-attempt before the MountsActive admission and refuse `Busy` while any holder persists.
Contributor
Author
Problems this PR (and its follow-ups) fixThis PR is the base of the #538 custody stack: #640 -> #653 -> #654. Together they close:
Position in the stack: base. Merge order is #640 -> #653 -> #654; #653 targets Why split into separate PRs
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Closes the remaining gap of #538 on
layerfs_v2_0_0: aRetained(Detach)custody (a forced unmount whose one plain detach the kernel refusedEBUSY) was unrecoverable — every laterworkspace.unmount,forceandsandbox.closere-reported the stored custody without re-attempting the detach, even after the holder exited.Two pieces, matching the #541/#544 closing notes:
1. Typed-Evidence custody retry
Evidence::Undrained) instead of an erasedBox<dyn Debug>, so a later terminal call can recover itsKernelSession.Undrained::retry_detach(inlayerfs-fuse) re-attempts only the recordedEBUSYdetach after a fresh local drain; the abort write is never repeated.Service::take_retryablerecognises the retryable custody (Detach,abort: Written,detach: Busy) andService::retry_detachre-runs the detach. AForceUnmountcompletes it asForceUnmounted; a normalUnmountcompletes it asUnmounted(the control channel has noUnmount → ForceUnmountedcorrelation). A holder still present keeps the exact custody with the new detach disposition.2.
sandbox.destroyAn explicit operator/CLI verb that deletes only one Sandbox's data plane (runtime stop + delete, then its folder) without asking the daemon or settling commands — for an invisible #460 namespace-copy holder that exec records cannot see.
Tests
cargo check/clippy -D warningsforlayerfs-sandbox,layerfs-api,layerfs-cli,layerfs-node;cargo test -p layerfs-sandbox -p layerfs-api(incl. updatedhost_frameandcalls_table);tools/check_product_boundary.pyPASS.rust:1.85.1-bookworm, real/dev/fuse, root):clippy -D warningsforlayerfs-daemon+layerfs-fuse; the real FUSEforced_unmountFP-33 tests (cwd and descriptor) PASS with the new behaviour — re-report while held,Unmountedonce the holder exits.stalled_unmount7/7 PASS (run serially).Not run
The full Sandbox E2E (client SIGKILL during foreground exec → force
Busy→workspace.exec_cancel→Unmounted) still needs the sealed Linux bundle and itssandbox-image.tar, which this host does not have built; the daemon-level FUSE proof above covers the custody-retry half directly.Related: #538, #460, #541 (superseded), #544 (superseded).
Rebase and re-verification (2026-11-12)
Rebased from the fork point
833d04cf4ontolayerfs_v2_0_0@03883a63c(which now includes #639/#641/#621/#543/#547/#655). Four commits kept, authorship
unchanged. New head:
4020e9097.Conflicts and semantic merges:
control/crates/layerfs-sandbox/src/family.rs— the base'sgetre-exportgained
Downwhile this branch addsdestroy; kept both(
pub use super::get::{get, Down, ...}pluspub use super::destroy::{...}).control/crates/layerfs-sandbox/src/host/call.rs— the base addedOp::Capacityand this branch addsOp::SandboxDestroy, soOPShas 11entries. The auto-merge left the old length 10 and did not compile; the length
is fixed inside the first commit so every commit in the stack builds.
Verified on Dell (
aluminium, Linux x86_64, cargo 1.85.1) at the stack head995627e09(this branch plus #653/#654):cargo fmt --all -- --checkandcargo clippy --locked --workspace --exclude layerfs-node --all-targets -- -D warningsexit 0, and
cargo test -p layerfs-sandbox -p layerfs-control -p layerfs-apipasses 103 binaries / 297 tests / 0 failures / 4 ignored (the four FUSE mount
targets run as root under
flock /tmp/dell-fuse.lock;/dev/fuseleft at 666).The
clippy(u64::from(blocks_available())useless conversion incontrol/crates/layerfs-home/src/home/capacity.rs) andtest(
sdk_blame_coarse::lines_between_distant_edits_keep_their_origin) failuresseen on the stacked PRs came from this branch's old base and are fixed by
03883a63c.CI note: PRs into
layerfs_v2_0_0are exempt from theciworkflow(
branches-ignore), so this PR shows no checks by design; the same tree waschecked by the
ciruns on #653/#654 and by the Dell run above.