feat(ns-exec): add exec session supervisor skeleton (#538) by mengmengmengqiang · Pull Request #542 · Ephemeral-AI-Lab/layerfs

mengmengmengqiang · 2026-11-04T04:36:24Z

Draft / stacked on #499. The supervisor depends on #499's ns-exec seam. Because #499 is still a fork PR, this upstream PR is based on main; the diff temporarily includes #499's commits. Do not merge before #499. The supervisor commits are a20f5ae81 (skeleton), df60ad8ef (modular doc), cab2199e4 (presence/registry decisions).

Stacked on #499 (fork base feat/ns-exec-helper, head 386f1b22). This PR is the skeleton only; it is not wired to the Engine and does not change workspace.exec behavior.

Scope

A hidden ns-exec-supervisor mode that owns one command across the #499 mount-namespace seam, plus one hidden ns-exec-child mode on the same binary.

Supervisor. Receives the daemon's mount-namespace descriptor over the existing SCM_RIGHTS socket, spawns the child, and closes its own copy. It never calls setns and never chdirs into the Workspace, so it cannot become a FUSE mount holder.
Child. Reopens the inherited descriptor through /proc/self/fd (with the existing mnt: check), setns(CLONE_NEWNS), chdir(workspace), drops supplementary groups/gid/uid exactly as feat(sandbox): run workspace.exec in the daemon's mount namespace (ns-exec helper) #499's ns-exec does, then setsid() and PR_SET_PDEATHSIG(SIGKILL), re-checks getppid, and execs the command.
Control socket. 0600 Unix socket, four-byte big-endian length frames with a fixed small body: Started {pid, pgid}, Exited {code, signal, group_alive}, Status/StatusReply, Signal {signo}, and a thin Stop {grace_ms} SIGTERM wrapper. A late caller receives the current Started/Exited snapshot under the same lock that registers it. Signal/Stop use kill(-pgid, signo) only for the child's own group; the supervisor's own group and arbitrary pids are refused. Peer admission is the supervisor's effective uid; further authorization is deferred.
wait4. Reaps the direct child exactly once, reports code/signal, exits 128 + signal when signalled, and reports group_alive: true (via kill(-pgid, 0)) when grandchildren remain. Grandchildren are not killed in this slice.

Architecture modules

The design note is organized in layers (module map, interfaces, startup/signal/exit sequences, failure modes and invariants, implemented vs deferred, test matrix): EXEC-SUPERVISOR-SKELETON.md.

Supervisor process. Owns one session: receives the mount-namespace descriptor, binds the control socket, spawns the child, observes its session, serves callers, reaps once and exits with the child's status. Never setns/chdirs.
Child launcher. The only process that joins the namespace: reopens/validates the descriptor, setns, chdir, drops groups/gid/uid, setsid, PDEATHSIG, checks its parent and execs the command.
Control link and protocol. 0600 socket, four-byte length frames, peer-uid admission, Started/Exited/Status/Signal/Stop, and a snapshot for late callers under the same lock as registration.
Process table / session state. One in-memory slot (pid, pgid, started, exit, clients) behind one mutex; all state dies with the supervisor.
Launcher / orchestrator (future, ownership open). Starts one supervisor per command, hands it the descriptor and maps the session to the Engine. Not implemented; where it lives conflicts with the current "daemon has no command supervisor" rule.
Session registry / presence (future). Session ids, reconnect and caller-presence; required because the control socket lives only as long as the supervisor.

The note also records gaps that the skeleton does not close: starttime is not captured (identity is only (pid, pgid)), Stop is implemented but untested, a session-poll timeout yields Exited without Started, and group_alive is an observable-group probe rather than ownership.

Presence + registry decisions

The six decisions the design note left open are now locked — launcher ownership, lease granularity, daemon-restart semantics, lease authentication, background/disconnect policy and cancellation states. Full text and rationale: Presence + registry decisions (locked).

Launcher. Engine exec spawns the supervisor and owns stdio and the exit code; the daemon only receives the registration and owns registry, presence and policy. The daemon does not become a command I/O broker and does not spawn the supervisor in this phase.
Lease granularity. One dedicated lease connection per client process per Sandbox, multiplexing all of its sessions — not one per exec. Session ids/tokens identify sessions; the connection budget is one lease plus pooled RPC slots.
Daemon restart. P1 does not adopt old sessions. A restart loses the registry and lease; supervisors act on control EOF by foreground (cancel/cleanup) vs background (keep) policy, and only host records remain. This is lost-session semantics, not recovery.
Lease auth. SO_PEERCRED plus a daemon-issued per-session capability/token delivered over the authenticated control channel; never in argv, because same-uid processes can read /proc/<pid>/cmdline. Hostile same-Sandbox multi-tenancy additionally requires user/namespace isolation.
Background. Not one boolean: mode: foreground|background and a separate onDisconnect: cancel|keep|reconnectWindow. This phase ships background + cancel; reconnectWindow comes later. detachOnDisconnect is foreground + keep.
Cancelling. No reversible 500 ms Cancelling window. The reconnect window is the Orphaned phase (5 s default); once Cancelling starts it is one-way, and the grace period is only TERM → KILL cleanup time.

Phase impact: P1 registry + starttime; P2 presence + reaper; P3 reconnect/background; P4 Engine wiring.

Deliberately deferred

client-presence detection and reconnect;
the daemon session registry and session ids;
force-kill orchestration, kill policy/escalation;
close(force) integration;
reaping/killing grandchildren;
authorization beyond the socket mode and peer uid;
retaining session state after the supervisor exits (the control socket lives only as long as the supervisor).

Tests

engine/crates/layerfs-daemon/tests/exec_supervisor.rs (Linux-gated; runs in the mount lane with CAP_SYS_ADMIN, like ns_exec.rs). It builds a private mount namespace with unshare -m, serves its /proc/<pid>/ns/mnt descriptor over SCM_RIGHTS, and drives the real binary over the control socket:

test	result
join the served namespace, enter the Workspace directory, read a namespace-local marker, `Started` has `pgid == pid` and pid != supervisor, exit 7	pass
supervisor's own mount namespace and cwd stay outside the Workspace	pass
command runs as `65534:65534` when told to drop there	pass
`sleep 30` + `Signal(SIGTERM)` → `Exited {signal: 15}`, supervisor exit 143	pass
`Status` reports the running child; `Signal(SIGKILL)` → signal 9, exit 137	pass
`sh -c 'sleep 5 & sleep 1'` → `group_alive: true` instead of a clean exit	pass

Run in a privileged rust:1.85.1-bookworm container against a20f5ae81:

cargo fmt --all -- --check — pass
cargo check -p layerfs-daemon --all-targets — pass
cargo clippy -p layerfs-daemon --all-targets -- -D warnings — pass
cargo test -p layerfs-daemon --test exec_supervisor --no-run then exec_supervisor --test-threads=1 — 6 passed, 0 failed; repeated 10 consecutive runs, all green.

Docs

core/docs/issues/538/EXEC-SUPERVISOR-SKELETON.md is the modular architecture note: module map, interfaces (SCM_RIGHTS handoff, control frames, process identity, exit/group_alive semantics), startup/signal/exit sequences, failure modes and invariants, implemented-vs-deferred table and test matrix.

Dependency

Depends on #499 (feat/ns-exec-helper, 386f1b22). Merging this before #499 would not compile because it builds on #499's ns_fd/cli seam. No changes to #499 or upstream.

… an ns-exec helper `workspace.exec` is an Engine exec in the container's initial mount namespace; the daemon is not involved. When the daemon serves from a private mount namespace (the #3/Ephemeral-AI-Lab#499 prototype), a Workspace FUSE mount is absent from that namespace, so the Engine's `bash -c <command>` cannot see the Workspace. This adds a small setns helper so the command runs inside the daemon's namespace while the Engine keeps its Exec entry, streams, cancellation and exit code. - The hidden `layerfs-daemon ns-exec --pid-file <path> [--directory <path>] [--uid <uid> --gid <gid>] -- <command> [args...]` entry reads the daemon pid, opens /proc/<pid>/ns/mnt, joins it with setns(CLONE_NEWNS), re-enters the mount directory in the joined namespace, drops the root entry identity to the ordinary command identity, then execs the command. A missing pid file, an unreadable pid, a missing namespace, or a refused setns/chdir/drop prints a clear message and exits 125; the command is never run in the wrong namespace. - DaemonSetup gains an opt-in `private_mount_ns` flag (config record LFSD\x06). When set, the daemon entry takes unshare(CLONE_NEWNS) + MS_REC|MS_PRIVATE on `/` before the listener or any worker thread, then atomically writes `daemon.pid` into the local directory above the Overlay's directory (/layerfs-local/daemon.pid in a Sandbox). With the flag off no namespace is taken and no pid file is written. - layerfs-api builds the Engine argv: when the Sandbox's record carries the private-namespace pid file, the argv names the helper and the Engine starts it as root (`ExecRequest.root_launcher`), because the ordinary 501:20 command identity can neither open the daemon's namespace nor setns (verified: a non-root `nsenter` gets EPERM; root succeeds). The helper drops back to the command identity before exec. Without the pid file both the argv and the Engine user are unchanged. - Opt-in: `deploy.privateMountNs` or `LAYERFS_PRIVATE_MOUNT_NS=1` at Sandbox create; the pid file path is saved in and reopened from the sandbox record. Tests: - added engine/crates/layerfs-daemon/tests/ns_exec.rs: a real private namespace built with `unshare -m` and a tmpfs visible only there; ns-exec joins it (read and chdir), drops to 65534, refuses a missing/invalid pid file and a missing namespace without running the command, and the daemon writes its pid file only when the private namespace is asked for. - added unit tests: the wrapper argv decision (layerfs-api), the helper's argument parsing (layerfs-daemon) and the config flag/pid path (layerfs-bridge). - touched ExecRequest literals and the internal::exec call sites; the Engine protocol encoding tests still pass. - deleted: none. Not exercised: a real Sandbox end-to-end (a FUSE mount in the private namespace, then exec), and the daemon's full startup under the flag; the Sandbox container's /dev/fuse and Store are not part of this test set. Production LOC: 102763 -> 103008 (delta +245) core 38395 -> 38395 (0), engine 61709 -> 61954 (+245), api 2659 -> 2659 (0); reference absent (retired R9c; last counted 65,417 at tree 498dd19). Method: python3 tools/production_loc.py --root on a `git archive` extract of 582bc78 and on this staged tree. Refs #3, tokenaissance/layerfs#499.

…IGHTS The E2E showed the helper could not open the daemon's namespace: root in a Sandbox holds CAP_SYS_ADMIN but not CAP_SYS_PTRACE, and the daemon is not dumpable (`config.rs` prctl::set_dumpable(false)), so `/proc/<daemon>/ns/mnt` is EACCES for every Sandbox process. The same 0600 file opened fine from outside the container with CAP_SYS_PTRACE, so the kernel's dumpability check is the blocker, not the path. - The daemon, when `private_mount_ns` is on, binds `<local>/ns-exec.sock` (mode 0600, stale path removed, unlinked on drop) and serves exactly one SCM_RIGHTS message carrying the descriptor of its own `/proc/self/ns/mnt` to an admitted peer: uid 0 and the same pid namespace, read from SO_PEERCRED and `/proc/<pid>/ns/pid`. Nothing else is sent; a refused peer is closed without a message (`application/ns_fd.rs`). - `ns-exec` now requires `--socket`, receives that descriptor, joins the namespace, then keeps the existing `--directory` chdir, uid/gid drop and exec. The pid-file route is deleted: it can never work in a Sandbox, so the helper no longer names a path that only returns EACCES. - `DaemonSetup::check` requires the socket path for the private-namespace mode; the pid file stays as a diagnostic marker and is not written when absent. - The public `WorkspaceApi::exec` derives the namespace from the owner's `SandboxCreate.setup` instead of always passing none, so a private-ns Sandbox cannot run that path unwrapped in the initial namespace. - A live Sandbox whose record predates the socket (`nsExecPidFile` without `nsExecSocket`) refuses `workspace.exec` with `Unsupported` and an instruction to close and recreate it, rather than deriving a socket the old daemon never bound or repeating an opaque 125. - One `layerfs_sandbox::DAEMON_ENTRY` constant feeds the create request, the observed-topology check and the helper argv, so the entry path cannot drift. - `MountNamespace` carries a socket and has no empty state: `Some` always wraps and `None` never does. Tests: - ns-exec integration (11): join over a served fd (read and chdir), uid/gid drop, missing socket, a socket that sends nothing, a socket that sends a non-namespace file, no namespace source (integration 125), a peer in another pid namespace refused, socket mode 0600 with a stale path replaced and exactly one message (one byte, one fd, then EOF), a non-root peer refused, and the daemon serving socket and pid file only with the flag. - node unit (3): a saved socket wraps the command end to end (record -> mount_namespace() -> argv), a pid-only record requires recreation, and a plain Sandbox keeps its command. - bridge/api/daemon unit (2/2/7) and the 17 Engine protocol tests pass. - deleted: the pid-file exec tests and the pid-file helper route. Not exercised: a real Sandbox end-to-end; the host wiring and the same E2E rerun belong to the E2E agent. Production LOC: 103008 -> 103183 (delta +175) core 38395 -> 38395 (0), engine 61954 -> 62129 (+175), api 2659 -> 2659 (0); reference absent (retired R9c; last counted 65,417 at tree 498dd19). Method: python3 tools/production_loc.py --root on a `git archive` extract of 8545a28 and on this staged tree. Refs #3, tokenaissance/layerfs#499.

CI failed in "fmt and guards" and in clippy. Formatting itself was already clean; the guard step and the clippy build were not. - clippy: the engine_streams example still built an ExecRequest without the new `root_launcher` field, so `--all-targets` could not compile; the example keeps its previous non-root launch. `ns_exec::parse` returned a four-tuple that tripped `clippy::type_complexity`, so it now returns a named `Parsed` value with the same fields, refusals and order. - guard: product source may not carry inline tests, and a `lib.rs` carries no definitions. The `daemon_setup` unit tests moved unchanged to `tests/private_mount_ns.rs`; the API argv-wrapping proof now runs through the public `WorkspaceApi::exec` against a private-namespace Sandbox; the helper's argv grammar is proven through the real `ns-exec` subprocess (`tests/ns_exec.rs`), where the socket-message and peer-admission proofs already covered the `ns_fd` predicates it used to unit-test. `DAEMON_ENTRY` moved from `lib.rs` to `types/deployment.rs`, re-exported at the same path. No behaviour changes: same argv, same refusals, same launch identity, same configuration bytes and same public paths. Verified: `cargo fmt --all -- --check`, the boundary and fuser guards and their unit tests, `cargo clippy --workspace --exclude layerfs-node --all-targets -- -D warnings`, `cargo check` on the four touched crates, and the daemon/sandbox tests (`ns_exec` 12/12 with FUSE, `engine_protocol` 17/17, `private_mount_ns` 2/2) in the pinned rust:1.85.1 container.

…#538) One supervisor owns one command across the Ephemeral-AI-Lab#499 mount-namespace seam: - hidden `ns-exec-supervisor` receives the daemon's mount-namespace descriptor over SCM_RIGHTS, spawns `ns-exec-child`, and closes its own copy. The supervisor never setns()es and never chdir()es into the Workspace, so it cannot become a mount holder; only the child joins. - the child runs Ephemeral-AI-Lab#499's sequence (setns, chdir, drop uid/gid) and then takes a new session with setsid(); it sets PR_SET_PDEATHSIG and re-checks its parent so a lost supervisor does not leave an unsupervised command. - a 0600 Unix control socket frames Started/Exited/Status/Signal/Stop with a four-byte length and a small fixed body. Signal and Stop only target the child's own process group (kill(-pgid, signo)); the peer must have the supervisor's effective uid. - wait4 reaps exactly once, reports exit code or signal, and reports group_alive (kill(-pgid, 0)) when grandchildren remain instead of pretending a clean exit. Grandchildren are not killed in this slice. Deliberately deferred: client-presence detection, the daemon session registry, force-kill orchestration and close(force) integration. Tests (Linux, mount lane): join the namespace served over SCM_RIGHTS and report exit 7; pgid == pid and different from the supervisor; supervisor stays outside the Workspace namespace/cwd; identity drop to 65534; sleep + SIGTERM -> 143; Status + SIGKILL -> 137; group_alive for a background sleep. Design note in core/docs/issues/538/EXEC-SUPERVISOR-SKELETON.md.

…al-AI-Lab#538)

jiangkoumo · 2026-11-08T11:24:11Z

Superseded by 2.0's exec design: commands run via layerfs-sandbox-helper entering the Sandbox's namespaces, and workspace.exec_cancel stops a command through its own cgroup (control/crates/layerfs-sandbox/src/exec/). The #499 ns-exec seam this stacks on was closed as superseded and is not part of 2.0. Closing; happy to revisit against layerfs_v2_0_0.

(Reviewed against layerfs_v2_0_0 @ 716197c during the Staging cleanup.)

mengmengmengqiang added 6 commits October 29, 2026 03:05

docs(ns-exec): modularize the supervisor architecture design (Ephemer…

df60ad8

…al-AI-Lab#538)

docs(ns-exec): lock presence + registry decisions (Ephemeral-AI-Lab#538)

cab2199

This was referenced Nov 4, 2026

fix(workspace): force-kill only the target Workspace on v2 (#538) #544

Closed

fix(workspace): force-kill only the target Workspace (#538) #541

Closed

fix(daemon): reclaim control slots after retained connection failure (#540) #543

Merged

mengmengmengqiang self-assigned this Nov 6, 2026

yhcedpn added the enhancement label Nov 6, 2026

yhcedpn added this to the LayerFS v2.0.0 milestone Nov 6, 2026

jiangkoumo closed this Nov 8, 2026

yifanxuaaa added the new feature A new user-facing capability label Nov 13, 2026

Navigation Menu

Repository navigation

feat(ns-exec): add exec session supervisor skeleton (#538) - #542

feat(ns-exec): add exec session supervisor skeleton (#538)#542
mengmengmengqiang wants to merge 6 commits into
Ephemeral-AI-Lab:mainfrom
tokenaissance:feat/exec-supervisor-skeleton

mengmengmengqiang commented Nov 4, 2026

Uh oh!

jiangkoumo commented Nov 8, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

mengmengmengqiang commented Nov 4, 2026

Scope

Architecture modules

Presence + registry decisions

Deliberately deferred

Tests

Docs

Dependency

Uh oh!

jiangkoumo commented Nov 8, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants