[serxa]

Problem

When a Codex workflow run ends (completed, cancelled, budget-stopped, or the daemon crashed), some descendant processes survive as orphans. Codex runs sandboxed commands under codex-linux-sandbox, which setsids into its own process group. CodexAppServerClient._signal_process_tree signals one process group (os.killpg), so these descendants are never reached. They stay parented to PID 1 after the run is done.

Fix

Opt-in codex.lifecycle.mode: strict launches a workflow run's codex app-server inside a transient systemd user scope (systemd-run --user --scope, KillMode=control-group). Every process the run forks stays in that cgroup. CodexAppServerClient.close() stops the scope after it kills the process group, so every way a client is closed (run end, kill, budget stop, transport-retry replacement, daemon shutdown) also kills the escaped descendants. Cleanup is confirmed by the scope's recursive cgroup.events showing populated=0, or by the unit unloading.

Each contained run writes <runs_dir>/<run>/lifecycle.json with the unit name, InvocationID, boot_id and cgroup. Two backstops read that record: the workflow service's client teardown, and a startup pass that cleans up after a crash. A stop is sent only to a unit whose InvocationID matches the record. A failed systemctl query is retried at the next startup and is never treated as "gone". If the engine recreates a crashed client for the same run, the previous scope is stopped before the new launch.

Scope

• Off by default. In disabled mode the launch is unchanged. The startup pass still cleans up scopes recorded by earlier strict runs.
• Applies to Codex workflow runs only. Interactive and cron sessions are unaffected.
• Needs Linux, cgroup v2 and a systemd user manager the daemon can reach. Docker and the system-level unit usually have none. There strict fails every Codex workflow run before exec rather than launch it uncontained. Documented in docs/config.md.

Testing

tests/test_codex_lifecycle.py runs real systemd user scopes with a setsid + double-fork escaper fixture: contain and reap, nested child cgroups, refusing to stop a replaced unit, relaunch, startup reconcile, close() stopping its own scope, and the fail-before-exec and record-validation paths. An integration test drives the real AgentEngine, WorkflowRunService and CodexBackend against a stalling fake app-server. It stops a run before any content arrives via kill_run, the budget kill and engine.stop_session, then checks that only one scope was launched and no descendant survived. These cases fail without the fix. No model or network. Existing Codex and workflow suites pass.

[alex-clickhouse]

P1: Prevent transport retries from creating a scope after terminal cleanup

At service.py:721–725, _finalize_terminal starts the scope reaper before the executor and stop sequence have finished. If the reaper kills the app-server before response content arrives, the engine's transport-recovery path treats the EOF as a crash and launches a replacement client. prepare_launch allows this once the original scope has a complete receipt. The stop sequence then disconnects the replacement client with killpg, but its setsid descendants survive. No further terminal reap is scheduled for the replacement scope.

I reproduced this at cd298ee5 using the real AgentEngine, WorkflowRunService, CodexBackend, and systemd user scopes, with an offline fake app-server that spawns a setsid/double-fork descendant and takes longer than the five-second interrupt grace to finish. Stopping before any response content arrives produced the same result through all three entry points:

• kill_run: two scope launches, status killed, one surviving descendant.
• _budget_kill: two scope launches, status budget_exhausted, one surviving descendant.
• engine.stop_session: two scope launches, status killed, one surviving descendant.

In each case, the replacement scope's lifecycle record had no cleanup receipt. A normal-completion control created one scope and left no survivors. The reproduction makes no model API calls and does not mock lifecycle operations.

Please prevent retries once the run is terminal and perform the final scope reap after the executor and client teardown can no longer create another scope. Add an integration test for stop/budget enforcement racing with transport recovery; testing launch and reap separately does not cover this ordering.

[serxa]

TBH I'm not sure if the approach with cgroup is the right thing here. It might be an overkill for this issue. Anyway, this is what my nerve installation came up with when faced the problem on my Linux environment.

If you think direction is not correct feel free to close the PR, otherwise I can address the feedback.

[nerveai]

Nerve PR review · ✅ Ready to merge

Updated for ba5a09e: both earlier blocking points and alex-clickhouse's P1 are resolved.

CodexAppServerClient.close() now stops its own scope after the killpg (codex/appserver.py:313-327). A kill, a budget stop, a retry replacement and engine.shutdown() therefore all take the setsid'd descendants with them. The terminal reap moved out of _finalize_terminal to after client teardown (workflows/service.py:1026-1028), which removes the mid-turn kill that triggered the relaunch. The unit= guard (lifecycle.py:363) stops a stale client from stopping a newer scope. The result is opt-in containment that closes a real escape route for sandboxed Codex workflow commands, and the default path is unchanged.

Completeness: in CI, all 22 lifecycle tests ran against real scopes and passed, none skipped. That includes the new stop-before-content integration test over kill_run, _budget_kill and stop_session. codex.lifecycle.mode is documented with its systemd-user-manager prerequisite and the Docker / system-unit caveat (docs/config.md:1048-1064). It reloads hot through the live codex reference, which the reload table already covers for codex.*.

_{Checked: full suite in an offline sandbox at ba5a09e, in 3 chunks: 6630 passed, 48 skipped (the systemd tests skip offline). CI at the same head: 6654 passed. Up to date with main 1695d9d. Read: lifecycle.py, launch/close in appserver.py, the teardown, stop and startup paths in workflows/service.py, and the engine's crash-retry and shutdown · automated review by nerveai — mention @nerveai to ask a follow-up.}

[serxa]

Addressed in ba5a09e, covering the P1 above and the review below.

P1 (relaunch after terminal reap): CodexAppServerClient.close() now stops the run's scope after it kills the process group. Every client close stops its own scope: run end, kill, budget stop, a transport-retry replacement, and daemon shutdown. The terminal reap no longer runs from _finalize_terminal, where it could kill the app-server mid-turn and trigger the retry. It now runs after client teardown, as a backstop next to the startup pass. After a stop, the reap also accepts the unit unloading as proof of an empty scope. That covers a stop that lands while the client is still registering its scope.

Integration test: test_workflow_stop_before_content_reaps_and_never_relaunches uses the real AgentEngine, WorkflowRunService and CodexBackend with a fake app-server. The fake spawns a setsid + double-fork descendant and stalls without content, ignoring interrupt. The test stops the run via kill_run, _budget_kill and engine.stop_session and checks three things: one scope launch, no surviving descendant, and a complete receipt on the record. All three cases fail at cd298ee5.

Review: close() now owns the scope, as suggested. codex.lifecycle.mode is documented in docs/config.md with the systemd user manager prerequisite and the Docker / system-unit caveat. The descendant process-group walk is left out of this PR: it is a separate, weaker mechanism (no double-fork or crash coverage).