Skip to content

fix(codex): record provider error notifications on the turn - #203

Merged
tmatup merged 1 commit into
mainfrom
fix/codex-error-notifications
Sep 28, 2026
Merged

tmatup merged 1 commit into
mainfrom
fix/codex-error-notifications

Conversation

@tmatup

@tmatup tmatup commented Sep 28, 2026

Copy link
Copy Markdown
Member

Summary

Short version: codex already tells us when the model provider stalls or fails and it is retrying. We threw that message away. This PR keeps it in task.json and task.log.

  • Codex's app-server sends an error notification ({error: TurnError, willRetry}) for every provider stream error. That includes the ones it retries internally (for example, the default 300 s stream idle timeout).
  • _CodexTurnState.dispatch routed only 5 methods, so error fell through. A provider stall read as "the model was slow".
  • Now dispatch routes error to on_error. It logs a WARNING and records a ProviderError row: at, message, kind (the codexErrorInfo category), http_status, will_retry, details.
  • The rows ride AgentEndEvent, then land in TurnRecord.provider_errors, so crashed partial turns keep them too.

Example of a new task.log line:

[codex] provider error (will_retry=True, kind=responseStreamDisconnected, http_status=None): stream disconnected before completion: idle timeout waiting for SSE

Why now

Run adhoc-2026-09-28_16-14-30, arm v2, test skill-flow-slack-channel-description (gpt-5.6-luna, Azure custom provider) ended in TIMEOUT. It spent 553 s + 572 s on two model calls for a ~12.5K-token request, before the agent ran its first command. The other ~36 tasks on the same endpoint in the same 20 minutes had no model call over 60 s. The run record holds only two long "thinking" messages, so we cannot say whether codex hit its idle timeout, got a 5xx, or waited on headers. This PR makes that visible next time.

Part 1 of #151. Part 2 (per-request time-to-first-token) stays open.

Not in this PR: changing the codex provider retry settings (stream_idle_timeout_ms 300 s × stream_max_retries 5 is larger than the 1200 s task budget). That is a behaviour change for every eval, so it is proposed separately.

Test plan

  • New TestProviderErrorNotifications (3 cases), built from real openai_codex ErrorNotification payloads: retried stream error with kind + HTTP status, a bare category, missing info, and a clean turn (empty list).
  • Golden snapshots regenerated (GOLDEN_REGEN=1). The only diff is "provider_errors": [] in all 34.
  • make verify: 6104 passed, 3 skipped.

🤖 Generated with Claude Code

https://claude.ai/code/session_014Tw2Pqkugyik4uVJHo2Ref

Codex retries a stalled or failed provider stream itself and reports each
attempt only through its app-server `error` notification (`willRetry`).
`_CodexTurnState.dispatch` dropped that method, so a provider stall left
no trace in task.json or task.log: adhoc-2026-09-28_16-14-30 v2
skill-flow-slack-channel-description lost 1,125 s of its 1,200 s budget
to two stalls on a ~12.5K-token request before the first tool call, and
the record shows only two long "thinking" messages.

- dispatch routes `error` to a new `on_error`, which logs a WARNING and
  appends a `ProviderError` (at, message, kind, http_status, will_retry,
  details) to the turn state.
- `AgentEndEvent` / `TurnRecord` carry `provider_errors`; EventCollector
  passes it through, so crashed partial turns keep it too.
- REPORT_SCHEMA documents the field; golden snapshots gain the empty list.

Closes part 1 of #151 (part 2, per-request TTFT timing, stays open).

🤖 Generated with Claude Code
Co-Authored-By: [Claude](mailto:noreply@anthropic.com)
Claude-Session: https://claude.ai/code/session_014Tw2Pqkugyik4uVJHo2Ref
@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

Claude finished @tmatup's task in 1m 35s —— View job


Code Review in Progress

Todo List:

  • Read .github/code_review.md for review guidelines
  • Read CLAUDE.md for project conventions
  • Review the full diff (git diff origin/main...HEAD)
  • Read complete changed files for context
  • Perform cross-file consistency checks
  • Analyze what might be missing
  • Write comprehensive review feedback

Starting code review...

@bai-uipath bai-uipath left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve: makes sense, small and safe. It only adds visibility: how the run behaves and how it's scored stay the same.

Before After
Codex retries a stalled stream Same Same
Task outcome TIMEOUT, scored as a miss TIMEOUT, scored as a miss
task.log Nothing One WARNING per provider error
task.json Long "thinking" messages, no errors One provider_errors row per error (kind, HTTP status, will_retry)

Worth fixing, not blockers:

  • The WARNING line leaves out details. Codex can put the underlying cause there instead of in message. Fix: include it, so task.log shows the cause whichever field it lands in.
  • An unparsed payload writes a misleading row. If the SDK can't validate the notification, it arrives as UnknownNotification and still reaches the error handler, which records an empty message and will_retry=False. Fix: read the raw params, or skip the row.

Follow-up: a provider-caused timeout still counts against the model, and a will_retry=False error still ends the turn as COMPLETED. Reclassifying those as infra errors is the payoff of this data.

Minor: the test class docstring describes the old behavior, and the clean-turn test is already covered by the golden snapshots.

@tmatup
tmatup merged commit 894f742 into main Sep 28, 2026
19 checks passed
@tmatup
tmatup deleted the fix/codex-error-notifications branch September 28, 2026 23:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants