Skip to content

Codex: count usage per notification and report the last call's context - #473

Open
alex-clickhouse wants to merge 1 commit into
mainfrom
alex/codex-usage-followup
Open

alex-clickhouse wants to merge 1 commit into
mainfrom
alex/codex-usage-followup

Conversation

@alex-clickhouse

Copy link
Copy Markdown
Collaborator

This fixes four problems in the Codex turn usage and cleans up the tests. The codex-rs links are for 0.153.3, the version that nerve accepts.

Changes

  • Baseline after resume. After thread/resume and thread/fork, app-server sends the restored usage of the thread with the id of an earlier turn (thread_processor.rs#L4031-L4042). start_turn drained that notification and receive_turn dropped it. So the first turn on a new client used total - last of its first notification. When that notification resent the previous counts (a retry after a failed request, or a usage limit), the turn counted the previous turn's last response again. The client now takes the baseline from the replay. It uses total - last only when there is no replay.
  • Reset in the middle of a turn. The usage of a turn is now the sum of the increases in total for each notification. A drop in total is a reset to 0 on ContextWindowExceeded (protocol.rs#L2291-L2304). The responses before the reset stay in the usage of the turn. On main, that turn records 0.
  • num_turns. A notification counts as a model response only when it increases total. codex also sends a notification after a failed request, on a usage limit and after a compaction (session/mod.rs#L4412-L4459).
  • Context bar. The turn's input divided by num_turns is the mean input per call. The context grows during a tool loop, so the mean is too low. For example, calls from 100k to 260k in a 272k window show 180k (66%), not 260k (96%). The Codex backend now reports the input tokens of the last response as context_tokens. The engine sends it in the done message and stores it in last_usage, and ContextBar uses it when it is present. The Claude backend does not set it, so its context bar does not change.

Tests

  • The five new backend tests fail on main and pass here.
  • A new resume_usage mode in the fake app-server sends the replay after thread/resume, then a resend of the restored counts, then one response. On main the turn records 5,000 fresh, 261,000 cached and 90 output tokens with num_turns=2. The correct values are 3,000, 131,000 and 50 with num_turns=1.
  • A new ContextBar test covers both ways to get the context size.
  • Removed test_dropped_mid_turn_notification_self_heals, which tests a notification drop that receive_turn cannot cause, and test_reset_turn_accounting_clears_state, which only checks private attributes. Corrected the comment of test_second_turn_deltas_off_first_turn_end.
  • The backend suite, the web build and the web tests pass.

🤖 Generated with Claude Code

- Take the usage baseline from the restored usage that app-server sends
  after thread/resume and thread/fork. start_turn drained it and
  receive_turn dropped it, so the first turn on a new client used
  total - last of its first notification. When that notification resent
  the previous counts, the turn counted the previous turn's last
  response again.
- Add up the increase in total for each notification. A drop in total
  is a reset to 0 (ContextWindowExceeded), so the responses before the
  reset stay in the usage of the turn.
- Count a notification in num_turns only when it increases total.
  Resends, usage limits and compactions are not model responses.
- Report the input tokens of the last response as context_tokens. The
  context bar uses it instead of the mean input per call, which is too
  low when the context grows during the turn.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nerveai

nerveai commented Oct 21, 2026

Copy link
Copy Markdown

Nerve PR review · ✅ Ready to merge

This fixes the Codex per-turn token counts that per-turn cost is computed from. On main, the first turn after each resume (every daemon restart or client rebuild) counts the previous turn's last response a second time. A mid-turn ContextWindowExceeded records 0, and resends and compactions inflate num_turns. The context bar also shows the mean input per call, not the last call's. Summing the increases of total, with the baseline taken from the resume replay, is the right model. I checked it against codex-rs 0.153.3. The replay goes out right after the thread/resume/thread/fork response, with an earlier turn's id, whenever turns are included, and Nerve never sets excludeTurns. Compaction and rate-limit updates keep total and set last input to 0. The engine's crash retry always builds a new client, so a per-client baseline loses nothing new. Claude leaves context_tokens unset and keeps today's estimate.

Completeness: backend tests plus a fake app-server resume mode, and a ContextBar test. No engine test checks that context_tokens reaches last_usage/done (num_turns has none either). The done example in docs/api.md does not list the new field. The branch merges cleanly with current main and with #449, which also touches codex/backend.py.

Checked: full suite in an offline sandbox at 54895f2, 3 chunks: 3902 passed; CI green (backend, frontend build); read the Codex client's connect/start_turn/receive_turn, the engine retry loop and _persist_turn, and codex-rs' replay and recompute_token_usage; frontend tests not run locally · automated review by nerveai — mention @nerveai to ask a follow-up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants