Skip to content

Measure time the proxy spends backing off from the provider - #26

Merged
patsypppe merged 5 commits into
mainfrom
feat/upstream-wait-accounting
Oct 25, 2026
Merged

patsypppe merged 5 commits into
mainfrom
feat/upstream-wait-accounting

Conversation

@patsypppe

Copy link
Copy Markdown
Owner

Summary

In the first clean run of the Harbor slice, three tasks timed out at their 120 s deadline in all 9 trials while others passed. A plausible cause is provider rate limiting: the proxy backs off and retries, and that waiting counts against the trial's wall clock. That is unverified, and fixing it would change what a timeout verdict means, so this PR measures it first.

  • The proxy records retries and milliseconds waited per trial (even when the call ultimately fails).
  • Efficiency gains upstream_wait_ms; trials record how much of their duration was provider throttling.
  • The Harbor benchmark digest reports wait_s_mean and dur_s_mean per task.

No verdict logic changes.

Test plan

  • make check (501 unit tests)
  • CI
  • Harbor slice re-run from this branch (3 timing-out tasks, concurrency 1) to read the numbers

Generated by Claude Code

The proxy records retries and milliseconds waited per trial; the trial's
efficiency record gains upstream_wait_ms and the Harbor benchmark digest reports
it. Groundwork for deciding whether provider throttling should count against a
trial's deadline, measured instead of assumed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017urKavHf7CSyMqnzeCCyoy
@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Meridian gate

✅ PASS — suite pass^k moved -0.000 (0.486 → 0.486), within the 0.030 tolerance

Failing trials in this run
  • expired-coupon — SELECT status FROM orders WHERE id = 1042 returned [['pending']], expected [['invoiced']]
  • expired-on-bulk-order — SELECT status FROM orders WHERE id = 1045 returned [['pending']], expected [['invoiced']]
  • expiry-boundary — SELECT status FROM orders WHERE id = 1046 returned [['pending']], expected [['invoiced']]
  • happy-path — SELECT status FROM orders WHERE id = 1041 returned [['pending']], expected [['invoiced']]
  • stale-coupon — SELECT status FROM orders WHERE id = 1044 returned [['pending']], expected [['invoiced']]
  • unknown-coupon — SELECT status FROM orders WHERE id = 1047 returned [['pending']], expected [['invoiced']]
task baseline pass^3 head pass^3
expired-coupon 0.400 0.400
expired-on-bulk-order 0.400 0.400
expiry-boundary 0.400 0.400
happy-path 0.400 0.400
missing-field 1.000 1.000
stale-coupon 0.400 0.400
unknown-coupon 0.400 0.400
baseline head
suite pass^3 0.486 0.486

Sensitivity. Not estimated: all 7 tasks moved by the same amount, so there is no spread to estimate sensitivity from. Treat the verdict as directional.

n=5 · k=3 · status complete · harness errors 0.0% · cost 51c

head run run-1792903612-bdce95 · baseline run run-1792903573-1f667b · manifest sha256:73f9a19222707979… · reproduce with meridian replay run-1792903612-bdce95

… last activity

The agent appends each model request/reply and command to an append-only
transcript as it goes (flushed per line, so it survives a kill). A timeout
verdict now carries the tail of it instead of just 'exceeded the deadline'.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017urKavHf7CSyMqnzeCCyoy
… ends

A call killed mid-backoff at the trial deadline never reached the recording, so
exactly the trials that lost time to throttling reported none.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017urKavHf7CSyMqnzeCCyoy
…adline

The deadline now moves with the time the proxy spent backing off from the
provider on this trial's behalf, capped at twice the base deadline so a provider
that never recovers still ends the trial. A timeout that used an extension says
so. Found by transcript: agents that finished their work in the first reply were
scored as timeouts while their next model request sat in a rate-limit backoff.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017urKavHf7CSyMqnzeCCyoy
…ing per run

A 429 says which limit was hit and when to come back. The proxy keeps that text
(clipped) and the run prints one line of how much throttling it absorbed, so
'the run was throttled' becomes something that can be acted on.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017urKavHf7CSyMqnzeCCyoy
@patsypppe
patsypppe merged commit ac0ed57 into main Oct 25, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants