Skip to content

fix(sandbox): retry retained detach on close instead of deleting the data plane (#538) - #654

Open
mengmengmengqiang wants to merge 1 commit into
Ephemeral-AI-Lab:fix/538-unmount-retryfrom
tokenaissance:fix/538-close-refuse
Open

mengmengmengqiang wants to merge 1 commit into
Ephemeral-AI-Lab:fix/538-unmount-retryfrom
tokenaissance:fix/538-close-refuse

Conversation

@mengmengmengqiang

@mengmengmengqiang mengmengmengqiang commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A forced unmount whose one plain detach the kernel refused EBUSY left a
Retained(Detach) custody that was retryable (#640), but sandbox.close still
treated every Retained Workspace as "recoverable" and deleted the whole
Sandbox data plane without re-attempting the detach.

Close now re-attempts the clean detach of each retryable Retained Workspace first
(one normal unmount on the daemon, which re-runs only the recorded detach and
never repeats the abort write), and refuses Busy naming the held Workspaces
while a holder is still alive. sandbox.destroy remains the explicit
data-plane-only delete and is unchanged.

What it fixes

  • unmount_guard::check_all reports, per Retained Workspace, whether its custody
    is a retryable forced detach (Retained { path, retryable }).
  • unmount_guard::retry_held re-attempts each retryable Retained detach,
    treating an already-gone Missing reply as detached.
  • lifecycle::close and direct::held re-attempt the detach before the
    MountsActive admission and refuse Busy (naming the Workspaces) while any
    holder persists, instead of deleting the data plane.

Files: control/crates/layerfs-sandbox/src/{direct,lifecycle,unmount_guard}.rs.

Conflict resolution (rebases)

Both branches added a path→token mapper to unmount_guard.rs: the unmount
branch as pub fn token(state, path) (reused by workspace/serve.rs), this
branch as an identical private fn token. The rebase keeps the public
function from fix/538-unmount-retry and deletes this branch's duplicate private
copy, while keeping every close-path change intact: Retained, check_all -> Vec<Retained>, absent, retry_held, and the Busy refusal in
lifecycle::close / direct::held. Nothing from either branch was dropped; the
files.rs token_of change was byte-identical on both sides, so it merged
cleanly.

The rebase onto 03883a63c also needed a semantic merge in lifecycle::close:
the new base probes the daemon only when !force (refresh_except_within(.., CLOSE_STATUS_WAIT), so a hung daemon cannot hold close(force)). retry_held
also asks the daemon, so it sits in the same if !force block. close(force)
keeps its documented recovery semantics: it skips the retry, stops the Sandbox
anyway, and reports the undrained Workspaces as unknown.

Stack

  1. fix(daemon): retry detach from Retained custody and add sandbox.destroy (#538) #640 fix/538-custody-retry @ 4020e9097 (base layerfs_v2_0_0 @ 03883a63c) — Retained custody and sandbox.destroy.
  2. fix/538-unmount-retry @ 46109ee56 → base fix/538-custody-retry — the unmount path.
  3. this PR fix/538-close-refuse @ 995627e09 → base fix/538-unmount-retry — the sandbox.close path.

Test evidence — Dell (aluminium), Linux x86_64, cargo 1.85.1

Rebased stack, verified at the stack head 995627e09 (Linux x86_64, 88 cores,
/dev/fuse present as crw-rw-rw-). Non-mount test binaries ran as the ordinary
user; the four Linux-only FUSE mount targets (the repo's own tools/ci_tests.py
mount lane) ran as root with /dev/fuse, under flock /tmp/dell-fuse.lock.
/dev/fuse was left at 666. Logs: /home/dell/538-538v2-*.log.

Gates (all exit 0):

Command Result
cargo +1.85.1 fmt --all -- --check exit 0
cargo +1.85.1 clippy --locked --workspace --exclude layerfs-node --all-targets -- -D warnings exit 0 — no exceptions
cargo +1.85.1 build --locked -p layerfs-daemon (mount-lane prerequisite) exit 0
cargo +1.85.1 test --locked -p layerfs-sandbox -p layerfs-control -p layerfs-api --no-run exit 0

Tests (cargo test -p layerfs-sandbox -p layerfs-control -p layerfs-api, each
built test binary run individually):

Package Binaries Passed Failed Ignored
layerfs-sandbox 9 27 0 0
layerfs-control 36 133 0 2
layerfs-api 58 137 0 2
total 103 297 0 4

FUSE mount targets run as root (12 tests, all pass): kernel_application 1,
kernel_mount_routes 2, observed_application 8, r5_1_actual_daemon_commit 1.

The 4 ignored tests are the repository's own #[ignore] fixtures, unchanged by
this branch: remote_store::minio_round_trip (needs MinIO),
store_family::a_killed_opener_leaves_its_log_frames,
doctor_sandbox::worker_fixture_server, store_doctor::worker_child.

Why the earlier CI on this PR was red

CI on the rebased head

ci run 43154298305
(head 995627e09): fmt and guards pass, clippy pass, test pass,
windows build (reports only) fail (pre-existing, non-gating).

NOT RUN

  • Whole-workspace suite (--workspace) — only the three packages named in
    the task were run.
  • layerfs-daemon package tests, including the Linux-only
    stalled_unmount mount target — outside the three requested packages; the
    daemon binary was built only as the mount-lane prerequisite.
  • Nightly/slow lane (tools/ci_tests.py nightly) — not requested.
  • macOS results are not evidence here: a local cargo check/fmt/clippy
    and one sandbox-package smoke run passed, reported as compile-only.

No merges; #538, #547 and #640 stay open.
Refs #538, #547, #640.

…data plane (Ephemeral-AI-Lab#538)

A forced unmount whose one plain detach the kernel refused `EBUSY` left a
Retained(Detach) custody that was retryable (Ephemeral-AI-Lab#640), but `sandbox.close`
still treated every Retained Workspace as "recoverable" and deleted the
whole Sandbox data plane without re-attempting the detach. Close now
re-attempts the clean detach of each retryable Retained Workspace first
(one normal unmount on the daemon, which re-runs only the recorded detach
and never repeats the abort write), and refuses `Busy` naming the held
Workspaces while a holder is still alive. `sandbox.destroy` remains the
explicit data-plane-only delete and is unchanged.

- unmount_guard::check_all now also reports which Retained Workspaces are
  retryable, and retry_held re-attempts each one (treating an already-gone
  `Missing` reply as detached).
- lifecycle::close and direct::held re-attempt before the MountsActive
  admission and refuse `Busy` while any holder persists.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant