Skip to content

fix(daemon): retry detach from Retained custody and add sandbox.destroy (#538) - #640

Open
mengmengmengqiang wants to merge 4 commits into
Ephemeral-AI-Lab:layerfs_v2_0_0from
tokenaissance:fix/538-custody-retry
Open

mengmengmengqiang wants to merge 4 commits into
Ephemeral-AI-Lab:layerfs_v2_0_0from
tokenaissance:fix/538-custody-retry

Conversation

@mengmengmengqiang

@mengmengmengqiang mengmengmengqiang commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

What

Closes the remaining gap of #538 on layerfs_v2_0_0: a Retained(Detach) custody (a forced unmount whose one plain detach the kernel refused EBUSY) was unrecoverable — every later workspace.unmount, force and sandbox.close re-reported the stored custody without re-attempting the detach, even after the holder exited.

Two pieces, matching the #541/#544 closing notes:

1. Typed-Evidence custody retry

  • The forced teardown's undrained connection is now kept as typed evidence (Evidence::Undrained) instead of an erased Box<dyn Debug>, so a later terminal call can recover its KernelSession.
  • Undrained::retry_detach (in layerfs-fuse) re-attempts only the recorded EBUSY detach after a fresh local drain; the abort write is never repeated.
  • Service::take_retryable recognises the retryable custody (Detach, abort: Written, detach: Busy) and Service::retry_detach re-runs the detach. A ForceUnmount completes it as ForceUnmounted; a normal Unmount completes it as Unmounted (the control channel has no Unmount → ForceUnmounted correlation). A holder still present keeps the exact custody with the new detach disposition.
  • The Sandbox-side unmount guard records retryable custody and re-sends it to the daemon instead of re-reporting it, so the retry actually reaches the daemon.

2. sandbox.destroy

An explicit operator/CLI verb that deletes only one Sandbox's data plane (runtime stop + delete, then its folder) without asking the daemon or settling commands — for an invisible #460 namespace-copy holder that exec records cannot see.

Tests

  • macOS (host, pinned 1.85.1): cargo check/clippy -D warnings for layerfs-sandbox, layerfs-api, layerfs-cli, layerfs-node; cargo test -p layerfs-sandbox -p layerfs-api (incl. updated host_frame and calls_table); tools/check_product_boundary.py PASS.
  • Linux (Debian rust:1.85.1-bookworm, real /dev/fuse, root): clippy -D warnings for layerfs-daemon + layerfs-fuse; the real FUSE forced_unmount FP-33 tests (cwd and descriptor) PASS with the new behaviour — re-report while held, Unmounted once the holder exits. stalled_unmount 7/7 PASS (run serially).

Not run

The full Sandbox E2E (client SIGKILL during foreground exec → force Busy → workspace.exec_cancel → Unmounted) still needs the sealed Linux bundle and its sandbox-image.tar, which this host does not have built; the daemon-level FUSE proof above covers the custody-retry half directly.

Related: #538, #460, #541 (superseded), #544 (superseded).

Rebase and re-verification (2026-11-12)

Rebased from the fork point 833d04cf4 onto layerfs_v2_0_0 @ 03883a63c
(which now includes #639/#641/#621/#543/#547/#655). Four commits kept, authorship
unchanged. New head: 4020e9097.

Conflicts and semantic merges:

  • control/crates/layerfs-sandbox/src/family.rs — the base's get re-export
    gained Down while this branch adds destroy; kept both
    (pub use super::get::{get, Down, ...} plus pub use super::destroy::{...}).
  • control/crates/layerfs-sandbox/src/host/call.rs — the base added
    Op::Capacity and this branch adds Op::SandboxDestroy, so OPS has 11
    entries. The auto-merge left the old length 10 and did not compile; the length
    is fixed inside the first commit so every commit in the stack builds.

Verified on Dell (aluminium, Linux x86_64, cargo 1.85.1) at the stack head
995627e09 (this branch plus #653/#654): cargo fmt --all -- --check and
cargo clippy --locked --workspace --exclude layerfs-node --all-targets -- -D warnings
exit 0, and cargo test -p layerfs-sandbox -p layerfs-control -p layerfs-api
passes 103 binaries / 297 tests / 0 failures / 4 ignored (the four FUSE mount
targets run as root under flock /tmp/dell-fuse.lock; /dev/fuse left at 666).

The clippy (u64::from(blocks_available()) useless conversion in
control/crates/layerfs-home/src/home/capacity.rs) and test
(sdk_blame_coarse::lines_between_distant_edits_keep_their_origin) failures
seen on the stacked PRs came from this branch's old base and are fixed by
03883a63c.

CI note: PRs into layerfs_v2_0_0 are exempt from the ci workflow
(branches-ignore), so this PR shows no checks by design; the same tree was
checked by the ci runs on #653/#654 and by the Dell run above.

mengmengmengqiang added a commit to tokenaissance/layerfs that referenced this pull request Nov 12, 2026
…data plane (Ephemeral-AI-Lab#538)

A forced unmount whose one plain detach the kernel refused `EBUSY` left a
Retained(Detach) custody that was retryable (Ephemeral-AI-Lab#640), but `sandbox.close`
still treated every Retained Workspace as "recoverable" and deleted the
whole Sandbox data plane without re-attempting the detach. Close now
re-attempts the clean detach of each retryable Retained Workspace first
(one normal unmount on the daemon, which re-runs only the recorded detach
and never repeats the abort write), and refuses `Busy` naming the held
Workspaces while a holder is still alive. `sandbox.destroy` remains the
explicit data-plane-only delete and is unchanged.

- unmount_guard::check_all now also reports which Retained Workspaces are
  retryable, and retry_held re-attempts each one (treating an already-gone
  `Missing` reply as detached).
- lifecycle::close and direct::held re-attempt before the MountsActive
  admission and refuse `Busy` while any holder persists.
…oy (Ephemeral-AI-Lab#538)

When a forced unmount stops at Detach with the kernel refusing the one
plain detach (EBUSY), the daemon kept Retained(Detach) custody and every
later unmount/force re-reported it without re-attempting the detach, so a
Workspace became unrecoverable once the holder exited.

- Keep the forced teardown's Undrained connection as typed evidence
  (Evidence::Undrained) so a later terminal call can recover its session.
- Add KernelSession/Undrained::retry_detach: re-attempt only the recorded
  EBUSY detach after a fresh local drain; the abort write is never repeated.
- workspace.unmount and ForceUnmount now take a retryable retained Detach
  custody out (Service::take_retryable) and re-attempt the detach
  (Service::retry_detach), continuing Revoke/Close on success and re-keeping
  the exact custody with the new disposition when still held.
- The Sandbox-side unmount guard records retryable custody (a forced detach
  refused EBUSY) and re-sends to the daemon instead of re-reporting it, so
  the retry actually reaches the daemon.

- sandbox.destroy: an explicit operator/CLI verb that deletes only one
  Sandbox's data plane (runtime stop + delete, then its folder) without
  asking the daemon or settling commands, for an invisible namespace-copy
  holder (Ephemeral-AI-Lab#460) that exec records cannot see.
…nmounted (Ephemeral-AI-Lab#538)

A later workspace.unmount on a Retained(Detach) custody re-attempts the
detach and, once the holder is gone, must answer Reply::Unmounted — the
control channel has no Unmount->ForceUnmounted correlation. retry_detach
now returns the drained connection and the caller completes it: a force
through force_completed (ForceUnmounted), a normal unmount through close
(Unmounted). The FP-33 test now asserts the re-report while held and the
Unmounted recovery once the holder exits.
mengmengmengqiang added a commit to tokenaissance/layerfs that referenced this pull request Nov 12, 2026
…data plane (Ephemeral-AI-Lab#538)

A forced unmount whose one plain detach the kernel refused `EBUSY` left a
Retained(Detach) custody that was retryable (Ephemeral-AI-Lab#640), but `sandbox.close`
still treated every Retained Workspace as "recoverable" and deleted the
whole Sandbox data plane without re-attempting the detach. Close now
re-attempts the clean detach of each retryable Retained Workspace first
(one normal unmount on the daemon, which re-runs only the recorded detach
and never repeats the abort write), and refuses `Busy` naming the held
Workspaces while a holder is still alive. `sandbox.destroy` remains the
explicit data-plane-only delete and is unchanged.

- unmount_guard::check_all now also reports which Retained Workspaces are
  retryable, and retry_held re-attempts each one (treating an already-gone
  `Missing` reply as detached).
- lifecycle::close and direct::held re-attempt before the MountsActive
  admission and refuse `Busy` while any holder persists.
@mengmengmengqiang

Copy link
Copy Markdown
Contributor Author

Problems this PR (and its follow-ups) fix

This PR is the base of the #538 custody stack: #640 -> #653 -> #654. Together they close:

Position in the stack: base. Merge order is #640 -> #653 -> #654; #653 targets fix/538-custody-retry (#640's branch) and #654 targets fix/538-unmount-retry, so they rebase and retarget as the one below them merges.

Why split into separate PRs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant