Repository navigation
fix(sandbox): retry retained detach on close instead of deleting the data plane (#538) - #654
Open
mengmengmengqiang wants to merge 1 commit into
Conversation
…data plane (Ephemeral-AI-Lab#538) A forced unmount whose one plain detach the kernel refused `EBUSY` left a Retained(Detach) custody that was retryable (Ephemeral-AI-Lab#640), but `sandbox.close` still treated every Retained Workspace as "recoverable" and deleted the whole Sandbox data plane without re-attempting the detach. Close now re-attempts the clean detach of each retryable Retained Workspace first (one normal unmount on the daemon, which re-runs only the recorded detach and never repeats the abort write), and refuses `Busy` naming the held Workspaces while a holder is still alive. `sandbox.destroy` remains the explicit data-plane-only delete and is unchanged. - unmount_guard::check_all now also reports which Retained Workspaces are retryable, and retry_held re-attempts each one (treating an already-gone `Missing` reply as detached). - lifecycle::close and direct::held re-attempt before the MountsActive admission and refuse `Busy` while any holder persists.
mengmengmengqiang
force-pushed
the
fix/538-close-refuse
branch
from
November 12, 2026 04:13
61194d1 to
995627e
Compare
mengmengmengqiang
force-pushed
the
fix/538-unmount-retry
branch
from
November 12, 2026 04:13
0283524 to
46109ee
Compare
mengmengmengqiang
requested review from
jiangkoumo and
yifanxuaaa
and removed request for
yifanxuaaa
November 12, 2026 04:52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A forced unmount whose one plain detach the kernel refused
EBUSYleft aRetained(Detach)custody that was retryable (#640), butsandbox.closestilltreated every Retained Workspace as "recoverable" and deleted the whole
Sandbox data plane without re-attempting the detach.
Close now re-attempts the clean detach of each retryable Retained Workspace first
(one normal unmount on the daemon, which re-runs only the recorded detach and
never repeats the abort write), and refuses
Busynaming the held Workspaceswhile a holder is still alive.
sandbox.destroyremains the explicitdata-plane-only delete and is unchanged.
What it fixes
unmount_guard::check_allreports, per Retained Workspace, whether its custodyis a retryable forced detach (
Retained { path, retryable }).unmount_guard::retry_heldre-attempts each retryable Retained detach,treating an already-gone
Missingreply as detached.lifecycle::closeanddirect::heldre-attempt the detach before theMountsActiveadmission and refuseBusy(naming the Workspaces) while anyholder persists, instead of deleting the data plane.
Files:
control/crates/layerfs-sandbox/src/{direct,lifecycle,unmount_guard}.rs.Conflict resolution (rebases)
Both branches added a path→token mapper to
unmount_guard.rs: the unmountbranch as
pub fn token(state, path)(reused byworkspace/serve.rs), thisbranch as an identical private
fn token. The rebase keeps the publicfunction from
fix/538-unmount-retryand deletes this branch's duplicate privatecopy, while keeping every close-path change intact:
Retained,check_all -> Vec<Retained>,absent,retry_held, and theBusyrefusal inlifecycle::close/direct::held. Nothing from either branch was dropped; thefiles.rstoken_ofchange was byte-identical on both sides, so it mergedcleanly.
The rebase onto
03883a63calso needed a semantic merge inlifecycle::close:the new base probes the daemon only when
!force(refresh_except_within(.., CLOSE_STATUS_WAIT), so a hung daemon cannot holdclose(force)).retry_heldalso asks the daemon, so it sits in the same
if !forceblock.close(force)keeps its documented recovery semantics: it skips the retry, stops the Sandbox
anyway, and reports the undrained Workspaces as
unknown.Stack
fix/538-custody-retry@4020e9097(baselayerfs_v2_0_0@03883a63c) — Retained custody andsandbox.destroy.fix/538-unmount-retry@46109ee56→ basefix/538-custody-retry— the unmount path.fix/538-close-refuse@995627e09→ basefix/538-unmount-retry— thesandbox.closepath.Test evidence — Dell (aluminium), Linux x86_64, cargo 1.85.1
Rebased stack, verified at the stack head
995627e09(Linuxx86_64, 88 cores,/dev/fusepresent ascrw-rw-rw-). Non-mount test binaries ran as the ordinaryuser; the four Linux-only FUSE mount targets (the repo's own
tools/ci_tests.pymount lane) ran as root with
/dev/fuse, underflock /tmp/dell-fuse.lock./dev/fusewas left at666. Logs:/home/dell/538-538v2-*.log.Gates (all exit 0):
cargo +1.85.1 fmt --all -- --checkcargo +1.85.1 clippy --locked --workspace --exclude layerfs-node --all-targets -- -D warningscargo +1.85.1 build --locked -p layerfs-daemon(mount-lane prerequisite)cargo +1.85.1 test --locked -p layerfs-sandbox -p layerfs-control -p layerfs-api --no-runTests (
cargo test -p layerfs-sandbox -p layerfs-control -p layerfs-api, eachbuilt test binary run individually):
FUSE mount targets run as root (12 tests, all pass):
kernel_application1,kernel_mount_routes2,observed_application8,r5_1_actual_daemon_commit1.The 4 ignored tests are the repository's own
#[ignore]fixtures, unchanged bythis branch:
remote_store::minio_round_trip(needs MinIO),store_family::a_killed_opener_leaves_its_log_frames,doctor_sandbox::worker_fixture_server,store_doctor::worker_child.Why the earlier CI on this PR was red
clippy:control/crates/layerfs-home/src/home/capacity.rs:8hadu64::from(filesystem.blocks_available()), auseless_conversiononx86_64-linux. The old base
e64ed5743had it;03883a63cwidens throughu128(fix(daemon): reclaim control slots after retained connection failure (#540) #543), so the rebased stack is clean.test:layerfs-store-service::sdk_blame_coarse::lines_between_distant_edits_keep_their_originfailed on the old stack because it branched before fix(content): align store-service line_alignment::exact with the #517 unique-line fallback #639;
03883a63ccarriesfix(content): align store-service line_alignment::exact with the #517 unique-line fallback #639's fix, so it passes.
windows build (reports only)is a pre-existing Windows-only lint,non-gating (
continue-on-error) and unchanged by these branches.CI on the rebased head
cirun 43154298305(head
995627e09):fmt and guardspass,clippypass,testpass,windows build (reports only)fail (pre-existing, non-gating).NOT RUN
--workspace) — only the three packages named inthe task were run.
layerfs-daemonpackage tests, including the Linux-onlystalled_unmountmount target — outside the three requested packages; thedaemon binary was built only as the mount-lane prerequisite.
tools/ci_tests.py nightly) — not requested.cargo check/fmt/clippyand one sandbox-package smoke run passed, reported as compile-only.
No merges; #538, #547 and #640 stay open.
Refs #538, #547, #640.