[yifanxuaaa]

Tier: P1; gate: yes (risk:store, risk:format, Store-access gate by hand); owner approved with acceptance of A, N1 and 10 (see comments).
Docs: api/sdk/machine.md (machine.stop force), api/sdk/sandbox.md (forced native close), engine/store/sqlite.md (Last close), control/sandboxes/README.md (record fields), glossary.md.
Durable: NOT_RUN — disabled by owner until explicit reauthorization.

Update (head d303c7b)

Brought up to layerfs_v2_0_0 (27920d3) and reworked after an independent review (records in the comments): read-only Store closes take no lane; native force-stop refusals before a signal stay certain and the stop continues past them; forced native close is gated on a live native holder and records a stopped runtime; record format, Docker refusal and close rules documented.

Evidence at this code: Clippy -D warnings clean on all changed crates (macOS and Linux); macOS 57 tests and live machine_force_native passed; Linux 38 tests passed; Linux mount lane 45/45 binaries, 116 tests passed. Not run: live Docker/Lima providers.

Production LOC against layerfs_v2_0_0 at merge: 160621 -> 161187 (delta +566); core 39787 -> 39787, engine 74765 -> 74818, control 40093 -> 40602, api 5976 -> 5980 (tools/production_loc.py, blob 35e4a6d601fa).

---

Summary

Adds an explicit force option to machine.stop and makes sandbox.close --force work on native without a Store or daemon. Both stop the runtime through its provider or its own records, never through the machine's worker, and keep disks, Stores, saved history and runtime records.

The branch is three commits on the current layerfs_v2_0_0 tip (03883a63cf, #655):

1. 1b22005b40: port of the force-stop and native concurrency work (7b5f5a5630) onto v2.
   • machine.stop --force stops providers without a live worker. Sandboxes on the machine are recorded gone, or unresolved when the stop could not be confirmed, with their Workspace frontiers reported unknown.
   • Restores macOS writer and Store lifecycle admission, and treats a stale lease that disappears as already released.
2. 9d30a87ad2: merges fix(sandbox): carry a full-pool capacity notice into sandbox.get (#649) #655 (no conflicts).
3. eda9c04f12: forced sandbox.close on native from the Sandbox's own records: no Store open, daemon control call, active-Workspace refusal or retry. The forced native machine.stop uses the same host builder. This is the remaining part of c1cf6fd169.

Validation

macOS, registered private worktree (tools/worktree.py):

• Warning-denying all-targets Clippy:
  • layerfs-sandbox, layerfs-control, layerfs-api, layerfs-machine, layerfs-platform, layerfs-store, with postgresql,machine;
  • layerfs-cli with postgresql;
  • layerfs-daemon.
• Live machine_force_native with a matching private daemon: the forced native machine.stop and sandbox.close end the exact daemon, command and mount without opening the Store. Saved data stays readable.
• Passing suites:
  • layerfs-api machine_* / sandbox_*: 34
  • layerfs-platform machine_force / machine_lima: 17
  • layerfs-persistence writer_admission / sqlite_lock_custody: 12
  • layerfs-sandbox capacity_read / pool_admission: 8
  • layerfs-api sandbox_capacity: 8
  • layerfs-cli capabilities / stream_framed: 7
  • layerfs-sandbox-backend local_stop_mount: 1
  • layerfs-store lease_race: 1
• Discovery: layerfs help --json advertises force on machine.stop and sandbox.close.

Not run here: Linux/Docker E2E bundles.

Production LOC

• 1b22005b40: 158658 -> 159105 (delta +447)
• 9d30a87ad2 (first parent): 159105 -> 159152 (delta +47)
• eda9c04f12: 159152 -> 159204 (delta +52)

Scope: first-party core, engine, control and API product sources. Method: pinned tools/production_loc.py --root <git archive snapshot> --json.

🤖 Generated with Claude Code

[yifanxuaaa]

Review record

Reviewed head: 18b4df7 (integrate/v2-force-stop-latest), merge base 27920d3 with layerfs_v2_0_0.
Tier: P1; gate: yes (risk:store: layerfs-persistence sqlite session/seal/lifecycle, layerfs-store place.rs; risk:format: layerfs-sandbox record/sandbox.rs new fields; by hand: new machine not_held gates change who may use a Store on a machine).
Model: Fable 5.2 (claude-fable-5-2), independent context; diff and policy only, no PR body or commit messages.
Read: persistence sqlite lane/session/seal end to end and their callers (store-service bootstrap, read_service, read_handle, commit lane, catalog publish_lane; control store open.rs, place.rs); all control and api production hunks by diff; test hunks; docs hunks.
Run again by the reviewer: nothing (static review only).

Blocking

1. engine/crates/layerfs-persistence/src/backend/sqlite/session/lifecycle.rs:37-60: Drop for State takes a blocking directory flock for every session, read-only and diagnostic included. A quarantined writer keeps its writer_hold (writer_admission.rs finish_writer), so after one uncertain write in a daemon, the next reader close (read_service.rs close, under the reader-set mutex) blocks forever, and with it every later mount, unmount and session lookup; also open_diagnostic and control's idle-Store reaper for any Store in that directory. A read-only close cannot checkpoint or remove -wal/-shm, so the lane buys no custody there.
2. control/crates/layerfs-control/src/machine_force.rs:65,86-99 (with host_process/sandbox.rs:102-108 and the new not_held gates): native_attempt is set before native_stop runs, so a refusal issued before anything was signalled (e.g. recorded pid reused after reboot: "saved local daemon identity changed") records an uncertain machine.stop op on native. The new gates then refuse every sqlite Store, Sandbox and Workspace call on native, and no call clears it (machine remove native is refused for the built-in machine; repeating the force stop fails the same way; ordinary close is now gated).

Non-blocking
3. session/lifecycle.rs:45-53: a failed drop-time acquire leaks the connection until process exit, keeping any open uncertain transaction's write lock; unreported.
4. session/connection.rs:325-331: if Checkpointer::start fails, the raw connection drops with no lane (pre-existing); the new sqlite.md "Last close" row omits this exception.
5. layerfs-sandbox force.rs:59-68 with close.rs:147-152: a partly completed forced close (stop ok, delete or remove_dir_all fails) is recorded as a certain failure, leaving the record running with its runtime stopped (same shape as the ordinary close).
6. machine_force.rs:66-67,74,101-113: ? inside the error handler can replace the original error and drop stopped_sandboxes/unknown detail.
7. machine_force.rs:168-178: force_record admits an uncertain machine.stop op owned by a still-live process.
8. record/sandbox.rs:65-66,88-95: two new record fields without a version bump or format-page update; older builds would show a force-stopped Sandbox as running.
9. docker/power.rs:126-137: the non-forced docker machine stop also changes its refusals (non-worker role label refused; non-running containers skipped).
10. close.rs:107-112: forced native close skips exec settle, unmount guard and the containerGone reclaim branch, and is not subject to the new machine gate (documented in sandbox.md).

Risky-lines scan: no Durable, unsafe, DELETE FROM, rmtree, #[ignore], dependency hunk, busy handler or retry loop in production code; one production remove_dir_all (force.rs:68, mirrors the ordinary close); two asserts re-added unchanged with longer messages; one fixture value changed (stream_worker.rs image).

Not checked by the reviewer: builds and tests; PostgreSQL; Windows/WSL, apple-container and lima at runtime; groups::end, mounts::detach and namespaces reopen internals; the three new test files beyond a risky-lines scan; JS/Python binding signatures against docs.

Author evidence at this head (macOS and Linux, toolchain 1.85.1, registered worktree): Clippy -D warnings clean on all changed crates on both; boundary guard PASS; macOS 79 tests passed; Linux persistence writer_admission 14, sqlite_lock_custody 2, idle_checkpoint 4, store lease_race 1 and lease_stale 1, platform machine_force 9, api machine_force 1 and sandbox_lost 1 passed; api sandbox_family and sandbox_records each fail in the build container because base's kernel guard refuses a Store on its overlayfs /tmp (environment, unchanged tests). Linux mount lane (45 binaries) built, not run. machine_force_native not run.

Verdict: blocked by findings 1 and 2. Owner approval is required in any case (risk:store, risk:format).

[yifanxuaaa]

Review record

Reviewed head: d4d9df9 (integrate/v2-force-stop-latest). Covers the change since 18b4df7 (8 files, +133/-22) and updates the verdict for the whole PR; the 18b4df7 record stands for everything this push did not touch.
Tier: P1; gate: yes (risk:store: persistence sqlite session connection.rs and lifecycle.rs; risk:format: layerfs-sandbox record/sandbox.rs; by hand: the new machine not_held gates on Store and Sandbox calls).
Model: Fable 5.2 (claude-fable-5-2), independent context; diff and policy only, no PR body or commit messages.
Read: every line of the incremental diff (connection.rs, lifecycle.rs, machine_force.rs, reopen.rs, both doc hunks, both new tests); at head, machine_force.rs native loop and error handler, every State field reference in the sqlite backend, every caller of stop_now/for_stop, layerfs-control Cargo.toml.
Run again by the reviewer: nothing (static review only).

Earlier blocking findings

1. Unbounded lane wait in Drop for State: resolved. State keeps close_lane only for writable sessions; read-only and diagnostic closes (read_service close under the reader-set mutex, diagnostic open, other daemons' readers) take no lane. Seal unaffected. New test read_only_close_does_not_wait_for_a_held_writer_lane.
2. Native force-stop lockout: resolved. Recorded::for_stop reopens without signalling; native_attempt is set only after it succeeds, just before stop(0), and cleared after the Sandbox is marked. A pre-signal refusal clears the op on native and marks no Sandbox. New test native_force_stop_refused_before_any_signal_leaves_no_uncertain_op.
   Earlier non-blocking finding 4 (doc vs checkpoint-start failure path): resolved in sqlite.md.

New, non-blocking
A. lifecycle.rs:50-59: a writable close still waits without bound behind a writer that another session or process kept after an uncertain write (that session's writes already blocked forever before this PR). Owner to accept or ask for a non-waiting attempt.
B. machine_force.rs:93: error.code == "Unknown" is uncertain whatever the phase. No pre-signal producer of Unknown found (author checked reopen.rs local/namespaces paths: Failed/InvalidInput only).
C. machine_force.rs:64-74: a pre-signal refusal on one native Sandbox stops the loop; later Sandboxes are not stopped. Workaround: ordinary sandbox close.
D. The new api test covers the "no recorded runtime" refusal, not "identity changed".
E. Still open from the 18b4df7 record: 5, 6, 7, 8 (record format statement), 9 (ordinary docker stop changed), 10.

Author evidence at this head (toolchain 1.85.1, registered worktree): Clippy -D warnings clean on all changed crates, macOS and Linux; both new tests fail on 18b4df7's product code and pass at d4d9df9; macOS and Linux covering tests passed; Linux mount lane (tools/ci_tests.py mount-targets, 45 binaries, each in a fresh container with /dev/fuse, SYS_ADMIN, fresh ext4 /tmp, timeout 120, --test-threads=1): 45/45, 116 tests passed, 3 ignored by their own attributes; boundary guard PASS at 18b4df7 (the push changes no guarded path). Not run: machine_force_native, live Docker/Lima. Durable: NOT_RUN - disabled by owner until explicit reauthorization.

Verdict: mergeable after owner approval (gate: risk:store, risk:format, hand-added Store-access gate).

[yifanxuaaa]

Review record

Reviewed head: d303c7b (integrate/v2-force-stop-latest).
Records for intermediate heads: b4a0283 (fixes A, B, C, D, 3, 5-10; found N1-N6), 0fd2a99 (resolves N2, N3, N5, N6; D tightened; reopens A by design), d303c7b (resolves M1; comment only).
Tier: P1; gate: yes (risk:store: persistence session lifecycle and connection, layerfs-store place.rs and transfer comment; risk:format: layerfs-sandbox record/sandbox.rs machine_stopped and machine_stop_unknown; by hand: machine not_held gates on Store and Sandbox calls; contract doc control/sandboxes/README.md).
Model: Fable 5.2 (claude-fable-5-2), independent context; diff and policy only, no PR body or commit messages.
Run again by the reviewer: nothing (static review only).

Resolved across the review: blocking 1 (read-only/diagnostic closes take no lane) and 2 (native force-stop pre-signal refusal leaves no op); B, C, 3, 5, 6, 7, 8, 9, N2, N3, N5, N6, M1.
Open, non-blocking, for owner acceptance:

• A: a writable close waits without bound behind a writer another process kept after an uncertain write (every write in that state already waits; the non-waiting alternative left the WAL behind and broke bucket pull verify and restore's in-use check, N2).
• N1: after a failed lane wait (EINTR, directory cannot be opened) the close sets SQLITE_DBCONFIG_NO_CKPT_ON_CLOSE and closes, with no second attempt (alternative: leak the connection to process exit).
• 10: forced native sandbox.close skips per-command settle and the mount check (documented in api/sdk/sandbox.md).
• D: identity-changed refusal covered end to end only by the live macOS test (needs LAYERFS_DAEMON).
• M2: the replaced writable-close test proves the wait by a 200 ms window.

Author evidence at d303c7b's code (0fd2a99; d303c7b changes comments only), toolchain 1.85.1, registered worktree via tools/worktree.py:

• Clippy --all-targets -D warnings on all changed crates, macOS and Linux: clean.
• macOS: 57 tests passed, 0 failed; machine_force_native live with LAYERFS_DAEMON: 2 passed, no daemon left.
• Linux: 38 tests passed, 0 failed; mount lane (tools/ci_tests.py mount-targets, 45 binaries, each in a fresh container with /dev/fuse, SYS_ADMIN, fresh ext4 /tmp, timeout 120, --test-threads=1): 45/45, 116 passed, 0 failed, 3 ignored.
• At b4a0283, kernel_coherence size_never_shrinks_under_concurrent_append_and_refusals_stay_refused failed once (FUSE read racing appends; no dependency on layerfs-persistence) and passed on one diagnostic rerun; recorded as intermittent.
• Boundary guard PASS. Not run: live Docker/Lima providers. Durable: NOT_RUN - disabled by owner until explicit reauthorization.

Verdict: mergeable after owner approval and explicit acceptance of A, N1 and 10.

[yifanxuaaa]

Owner approval and acceptance

Head: d303c7b

The owner answered "Accept and merge" in the session after being shown the three remaining trade-offs from the review records above:

• A: a writable Store close waits without bound behind a writer another process kept after an uncertain write. Every write in that state already waits; the non-waiting alternative left the WAL behind and broke bucket pull's verify close and backup restore's in-use check (N2).
• N1: after a failed lane wait (interrupted, or the directory cannot be opened), the close sets SQLITE_DBCONFIG_NO_CKPT_ON_CLOSE and closes without touching the WAL, with no second attempt. The alternative is leaking the connection to process exit.
• 10: a forced native sandbox.close does not settle commands one by one or check mounts first; the runtime's stop ends them (documented in api/sdk/sandbox.md).

Gate: risk:store, risk:format and the hand-added Store-access gate are approved with this acceptance. Per the review policy, a contributor reads the risk:store and risk:format hunks within a week after merge and files an issue for each finding.