Skip to content

fix(broker): stop tearing down a live broker that misses the readiness probe - #768

Open
mzl9039 wants to merge 1 commit into
openai:mainfrom
mzl9039:fix/broker-readiness-race
Open

mzl9039 wants to merge 1 commit into
openai:mainfrom
mzl9039:fix/broker-readiness-race

Conversation

@mzl9039

@mzl9039 mzl9039 commented Sep 18, 2026

Copy link
Copy Markdown

Problem

ensureBrokerSession() probes the persisted broker with a 150 ms budget:

// plugins/codex/scripts/lib/broker-lifecycle.mjs
return await waitForBrokerEndpoint(endpoint, 150);

A broker that is currently serving a turn keeps its event loop busy, so it can easily
fail to accept the probe connection inside that window. The failed probe is then treated
as "the broker is gone" and the session is torn down.

Today that teardown mostly orphans the broker (this is #753). But once killProcess is
wired up — which is exactly what #762 does — the same path kills a perfectly healthy
broker
, taking down its app-server and the in-flight turn with it. The caller sees:

codex app-server exited before the turn completed.

A second, independent race makes it easier to hit: ensureBrokerSession() has no mutual
exclusion, so concurrent clients can all observe loadBrokerSession() === null, each spawn
a broker, and let the last writer win — orphaning the other brokers along with the
app-servers and turns they own.

Reproduction

Submit two background tasks in the same workspace a few seconds apart:

node plugins/codex/scripts/codex-companion.mjs task "<long read-only task>" --background
sleep 3
node plugins/codex/scripts/codex-companion.mjs task "<another long read-only task>" --background
before after
first task failedcodex app-server exited before the turn completed. completed
second task runningcompleted completed
broker processes replaced each other, then none one, reused
app-servers killed mid-turn yes none

Fix

  1. Raise the readiness probe budget to 3 s, and never tear down a broker whose process
    is still alive
    . A failed probe only proves the broker did not answer in time; only
    ESRCH from process.kill(pid, 0) proves it is gone (EPERM and PID reuse are
    treated conservatively as alive).
  2. Serialize the check-then-create window with a stale-aware lock file, so concurrent
    callers share one broker instead of racing to create several.

If the lock cannot be acquired within its budget the code falls through to the previous
behaviour rather than failing the caller outright.

Tests

tests/broker-lifecycle.test.mjs adds two cases. Both fail on main with the
relevant assertion and pass with this change:

  • a live broker must never be killed after a failed probe
  • concurrent callers must share one broker

Full suite: 93 passing (91 before + 2 new), no regressions.

Note: running the suite inside a live Claude Code session needs
env -u CLAUDE_PLUGIN_DATA -u CODEX_COMPANION_SESSION_ID, otherwise
state.test.mjs picks up the ambient plugin data dir — that is #456, not this change.

Relationship to existing work

…s probe

`ensureBrokerSession()` probes the persisted broker with a 150 ms budget:

    return await waitForBrokerEndpoint(endpoint, 150);

A broker that is serving a turn keeps its event loop busy, so it can easily miss
that window. The probe failure is then treated as "the broker is gone" and the
session is torn down — which, once `killProcess` is wired up, kills a perfectly
healthy broker together with the app-server and the in-flight turn it owns.

Two changes:

1. Raise the readiness probe budget to 3 s, and never tear down a broker whose
   process is still alive. A failed probe only proves the broker did not answer
   in time; only `ESRCH` from `process.kill(pid, 0)` proves it is gone.
2. Serialize the check-then-create window with a stale-aware lock file.
   Concurrent clients could all observe `loadBrokerSession() === null`, each
   spawn a broker, and let the last writer win — orphaning the other brokers
   along with their app-servers and in-flight turns.

Both new tests fail on main with the relevant assertion and pass with the fix.
Full suite: 93 passing (91 before + 2 new).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant