Skip to content

🤖 fix: wake stream-ends no longer falsely settle active workspace turns - #3949

Open
coadler wants to merge 4 commits into
mainfrom
fix/false-terminal-uncorrelated-wake-stream-end
Open

🤖 fix: wake stream-ends no longer falsely settle active workspace turns#3949
coadler wants to merge 4 commits into
mainfrom
fix/false-terminal-uncorrelated-wake-stream-end

Conversation

@coadler

@coadler coadler commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

A sub-agent report, bash-monitor wake, or family message can end its stream inside a child workspace while a delegated workspace turn still runs. That stream-end carries no turn correlation metadata. The uncorrelated-supersede path then settled the active handle as interrupted with "Workspace turn superseded by an uncorrelated workspace stream-end". The owner saw a false terminal. The child kept working and later pushed commits and opened PRs that no wake-up ever reported.

Background

Three production instances (workspaces 441bbc205d, 1fb57a782d, 0b726f0717) showed the same shape. Each child spawned explore sub-agents or armed bash monitors. Timing matched wake delivery: one false terminal arrived minutes after spawn, before any push.

The existing guards only covered two cases: compaction streams (agentId === "compact") and uncorrelated ends that predate the turn prompt in history. Mid-turn synthetic wake streams fell through to the interrupt path.

Implementation

interruptWorkspaceTurnFromUncorrelatedStreamEnd now scans child history between the turn anchor and the uncorrelated stream-end, with three behaviors:

  • Manual supersession preserved. Manual user input (metadata.synthetic !== true) in the window proves the workspace was redirected away from the delegated work. The handle settles interrupted, as before.
  • Live continuations defer. Synthetic wake rows continue the delegated work instead of redirecting it. When continuation evidence is live (queued/preparing input, auto-retry, pending monitor wakes, correlated workspace-turn continuations, another active stream), settlement defers so the real correlated terminal settles the handle later.
  • Idle wake ends settle from the event. When nothing else is queued or streaming, the uncorrelated wake end IS the delegated turn's last activity. The handle settles from that event (completed/error by finish reason) instead of stranding as running until restart recovery. A later correlated end can still correct it via allowTerminalResettle.

Two additional fixes from review:

  • Auto-compaction hides the original correlated prompt behind a summary boundary. The history scan now also anchors on compaction-summary.pendingFollowUp.workspaceTurnMetadata (mirroring AgentSession.inheritOpenWorkspaceTurnMetadata), so post-compaction wakes stay guarded.
  • The old isStreamEndBeforeWorkspaceTurnPrompt helper merged into the handler because both checks share one history scan.

Validation

  • Regression test reproduces the exact production signature before the fix: status: "interrupted" with the supersede error string, mid-work, while the child keeps streaming.
  • Companion tests pin each branch: live continuation evidence keeps the handle running; an idle wake end completes the handle with the wake output as the report; manual input still supersedes; compaction-preserved correlation still anchors after the boundary.
  • Full taskService.test.ts: 410 pass. Adjacent suites: agentSession.workspaceTurnInheritance, agentSession.queueDispatch, acp.promptCorrelation all pass.
  • make typecheck and make lint pass.

Risks

Low-to-moderate. The change narrows the false-interrupt path but adds a new settlement route (idle wake ends settle completed). Two mitigations bound the risk: settlement only fires when no continuation evidence is live anywhere in the child, and allowTerminalResettle: true lets any later correlated end correct a premature completion. Residual worst case is a report whose text comes from a wake response rather than the turn's true final message — degraded fidelity, not lost work.


Generated with xum • Model: openrouter:stealth/ox-alpha • Thinking: high • Cost: $0.01

Sub-agent progress reports, bash-monitor wakes, and family messages
dispatch synthetic user turns inside a child workspace. Their streams
carry no workspace-turn correlation metadata, so their stream-end hit
the uncorrelated-supersede path and settled the still-running delegated
turn as interrupted. The owner saw a false terminal while the child
kept working.

The supersede interrupt now scans child history between the turn prompt
and the uncorrelated stream-end. Only manual (non-synthetic) user input
in that window proves the workspace was redirected; synthetic wake
continuations keep the handle active so the real terminal report can
settle it later.
@coadler

coadler commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

This comment has been minimized.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 65cec367c1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/taskService.ts
Comment thread src/node/services/taskService.ts Outdated
@coadler

coadler commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 65cec367c1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/taskService.ts Outdated
@chatgpt-codex-connector

This comment has been minimized.

…d turns

Codex review found two gaps in the uncorrelated wake guard:

- An unconditional ignore could strand the handle as running when the
  synthetic wake stream was the delegated turn's last activity. The
  guard now defers only while continuation evidence is live (queued
  input, auto-retry, monitor wakes, correlated continuations, another
  active stream) and otherwise settles the handle from the wake event
  itself. A later correlated end still corrects it via
  allowTerminalResettle.
- After auto-compaction, getHistoryFromLatestBoundary no longer shows
  the correlated prompt, so promptIndex stayed -1 and post-compaction
  wakes fell through to the supersede path. The history scan now also
  anchors on compaction-summary.pendingFollowUp.workspaceTurnMetadata,
  mirroring AgentSession.inheritOpenWorkspaceTurnMetadata.
@coadler

coadler commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Addressed both P1 threads (PRRT_kwDOPxxmWM6bz40N, PRRT_kwDOPxxmWM6bz40Q) in 2314bfe:

  • Compacted turns: history scan now anchors on compaction-summary.pendingFollowUp.workspaceTurnMetadata.
  • Idle wake ends: defer only with live continuation evidence; otherwise settle the handle from the wake event with allowTerminalResettle.

@chatgpt-codex-connector

This comment has been minimized.

Second Codex round: returning early on getHistoryFromLatestBoundary
failure left the active handle running, and for a normal created
workspace the owner's waiter then never settles because this stream-end
was the only settlement signal. Restore the pre-existing supersede
settlement as the terminal fallback and pin it with a test that injects
the failure for every history read.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2314bfedb6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/node/services/taskService.ts
Comment thread src/node/services/taskService.ts
Comment thread src/node/services/taskService.ts Outdated
@coadler

coadler commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Addressed PRRT_kwDOPxxmWM6bz84z in 06a524b: a failed getHistoryFromLatestBoundary now falls back to the supersede settlement instead of returning unsettled, since that stream-end may be the only settlement signal for the waiter. Test injects the failure across all history reads.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: 06a524b266

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

This comment has been minimized.

Third Codex round, three gaps:

- Nested-work blindness: continuation evidence now includes
  hasActiveWorkspaceTurnDeferredBlockers (descendant tasks, workflows,
  nested turns), which a continuing reported-agent workspace skips on
  the parent side of handleStreamEnd.
- Immutable completion: uncorrelated wake settlements are stamped
  provisionalOutcome; settleWorkspaceTurn lets a later strictly-
  correlated stream-end replace them even when completed.
- Vanished continuations: deferral persists the wake end as a deferred
  message ID (new persistDeferredUncorrelatedWakeEnd, correlation-free
  sibling of markWorkspaceTurnStreamEndDeferred). Stale recovery reads
  deferred uncorrelated ends back via buildDeferredWakeEndEventFromHistory,
  so a continuation that dies without another stream-end settles
  deterministically. isLiveWorkspaceTurn now counts auto-retry and
  monitor-wake continuations as runtime activity.
@coadler

coadler commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

Addressed all three round-3 threads in 69541ae:

  • PRRT_kwDOPxxmWM6b0Mpp: hasLiveUncorrelatedWakeContinuationEvidence now includes hasActiveWorkspaceTurnDeferredBlockers (descendants, workflows, nested turns).
  • PRRT_kwDOPxxmWM6b0Mpv: wake settlements stamp provisionalOutcome; settleWorkspaceTurn permits a correlated end to replace them even from completed.
  • PRRT_kwDOPxxmWM6b0Mp0: deferral persists the end via persistDeferredUncorrelatedWakeEnd; stale recovery settles deferred uncorrelated ends once blockers clear; isLiveWorkspaceTurn counts auto-retry and monitor-wake continuations.

@chatgpt-codex-connector

This comment has been minimized.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 69541ae996

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +11101 to +11103
const event =
this.buildWorkspaceTurnStreamEndEventFromHistory(record, message) ??
(isDeferred ? this.buildDeferredWakeEndEventFromHistory(record, message) : null);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Mark recovered uncorrelated wake outcomes as provisional

When blockers disappear, a deferred uncorrelated wake is rebuilt here through buildTerminalWorkspaceTurnRecordFromEvent, which explicitly deletes provisionalOutcome. If snapshot/stale recovery wins a race with a correlated stream-end callback, the recovered completed record is therefore immutable and the later real terminal is ignored. Preserve the provisional marker for this deferred uncorrelated path just as the immediate wake-settlement path does.

Useful? React with 👍 / 👎.

.passthrough()
.optional(),
deferredMessageIds: z.array(z.string().min(1)).optional(),
provisionalOutcome: z.boolean().optional(),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep provisional handle records readable after downgrade

After this version writes provisionalOutcome, downgrading to the parent version makes its .strict() handle schema reject the unknown persisted key; readWorkspaceTurnFile then returns null, so task_await reports the handle as missing and startup reconciliation skips it. Store this state in a backward-readable representation or otherwise avoid adding an unknown top-level field to these persistent records.

AGENTS.md reference: AGENTS.md:L10-L11

Useful? React with 👍 / 👎.

Comment on lines +11363 to +11367
await this.taskHandleStore.upsertWorkspaceTurn({
...current,
updatedAt: getIsoNow(),
deferredMessageIds: [...(current.deferredMessageIds ?? []), event.messageId],
});

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve deferred wake outcomes across history boundaries

Persisting only the message ID is insufficient when the continuation performs compaction and then fails to dispatch: the deferred wake precedes the new durable boundary, so getHistoryFromLatestBoundary() can no longer return that message. Once the continuation signals clear, recovery finds no terminal event, and settleStaleWorkspaceTurn deliberately keeps any same-process record with deferredMessageIds active, leaving the owner and parallel-task slot blocked indefinitely until restart. Persist the deferred event's terminal data on the handle or search older history epochs during recovery.

Useful? React with 👍 / 👎.

Comment on lines +1179 to +1180
function isManualChildWorkspaceInput(message: MuxMessage): boolean {
return message.role === "user" && message.metadata?.synthetic !== true;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Treat user-triggered auto-compaction as manual supersession

When an uncorrelated wake end is processed after on-send auto-compaction for a newly queued user prompt has been persisted, that compaction request is marked synthetic: true, so this predicate does not recognize the redirect. If the compaction and queue are no longer reported as live, the old handle is completed from the wake output instead of interrupted as superseded. The existing isManualUserSupersessionMessage already distinguishes these user-triggered auto-compactions from internal-resume compactions; reuse that predicate rather than introducing a less complete duplicate.

AGENTS.md reference: AGENTS.md:L200-L202

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant