Conversation
…s probe
`ensureBrokerSession()` probes the persisted broker with a 150 ms budget:
return await waitForBrokerEndpoint(endpoint, 150);
A broker that is serving a turn keeps its event loop busy, so it can easily miss
that window. The probe failure is then treated as "the broker is gone" and the
session is torn down — which, once `killProcess` is wired up, kills a perfectly
healthy broker together with the app-server and the in-flight turn it owns.
Two changes:
1. Raise the readiness probe budget to 3 s, and never tear down a broker whose
process is still alive. A failed probe only proves the broker did not answer
in time; only `ESRCH` from `process.kill(pid, 0)` proves it is gone.
2. Serialize the check-then-create window with a stale-aware lock file.
Concurrent clients could all observe `loadBrokerSession() === null`, each
spawn a broker, and let the last writer win — orphaning the other brokers
along with their app-servers and in-flight turns.
Both new tests fail on main with the relevant assertion and pass with the fix.
Full suite: 93 passing (91 before + 2 new).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ensureBrokerSession()probes the persisted broker with a 150 ms budget:A broker that is currently serving a turn keeps its event loop busy, so it can easily
fail to accept the probe connection inside that window. The failed probe is then treated
as "the broker is gone" and the session is torn down.
Today that teardown mostly orphans the broker (this is #753). But once
killProcessiswired up — which is exactly what #762 does — the same path kills a perfectly healthy
broker, taking down its app-server and the in-flight turn with it. The caller sees:
A second, independent race makes it easier to hit:
ensureBrokerSession()has no mutualexclusion, so concurrent clients can all observe
loadBrokerSession() === null, each spawna broker, and let the last writer win — orphaning the other brokers along with the
app-servers and turns they own.
Reproduction
Submit two background tasks in the same workspace a few seconds apart:
failed—codex app-server exited before the turn completed.completedrunning→completedcompletedFix
is still alive. A failed probe only proves the broker did not answer in time; only
ESRCHfromprocess.kill(pid, 0)proves it is gone (EPERMand PID reuse aretreated conservatively as alive).
callers share one broker instead of racing to create several.
If the lock cannot be acquired within its budget the code falls through to the previous
behaviour rather than failing the caller outright.
Tests
tests/broker-lifecycle.test.mjsadds two cases. Both fail onmainwith therelevant assertion and pass with this change:
a live broker must never be killed after a failed probeconcurrent callers must share one brokerFull suite: 93 passing (91 before + 2 new), no regressions.
Relationship to existing work
killProcessfor that teardown. The two are complementary, but mergingfix: terminate broker process when ensureBrokerSession tears down (fixes #753) #762 without this change converts an orphaned-broker leak into an actively killed
in-flight turn.
same class of failure.