Repository navigation
Measure time the proxy spends backing off from the provider - #26
Merged
Merged
Conversation
The proxy records retries and milliseconds waited per trial; the trial's efficiency record gains upstream_wait_ms and the Harbor benchmark digest reports it. Groundwork for deciding whether provider throttling should count against a trial's deadline, measured instead of assumed. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017urKavHf7CSyMqnzeCCyoy
Meridian gate✅ PASS — suite pass^k moved -0.000 (0.486 → 0.486), within the 0.030 tolerance Failing trials in this run
Sensitivity. Not estimated: all 7 tasks moved by the same amount, so there is no spread to estimate sensitivity from. Treat the verdict as directional.
head run |
… last activity The agent appends each model request/reply and command to an append-only transcript as it goes (flushed per line, so it survives a kill). A timeout verdict now carries the tail of it instead of just 'exceeded the deadline'. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017urKavHf7CSyMqnzeCCyoy
… ends A call killed mid-backoff at the trial deadline never reached the recording, so exactly the trials that lost time to throttling reported none. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017urKavHf7CSyMqnzeCCyoy
…adline The deadline now moves with the time the proxy spent backing off from the provider on this trial's behalf, capped at twice the base deadline so a provider that never recovers still ends the trial. A timeout that used an extension says so. Found by transcript: agents that finished their work in the first reply were scored as timeouts while their next model request sat in a rate-limit backoff. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017urKavHf7CSyMqnzeCCyoy
…ing per run A 429 says which limit was hit and when to come back. The proxy keeps that text (clipped) and the run prints one line of how much throttling it absorbed, so 'the run was throttled' becomes something that can be acted on. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017urKavHf7CSyMqnzeCCyoy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
In the first clean run of the Harbor slice, three tasks timed out at their 120 s deadline in all 9 trials while others passed. A plausible cause is provider rate limiting: the proxy backs off and retries, and that waiting counts against the trial's wall clock. That is unverified, and fixing it would change what a
timeoutverdict means, so this PR measures it first.Efficiencygainsupstream_wait_ms; trials record how much of their duration was provider throttling.wait_s_meananddur_s_meanper task.No verdict logic changes.
Test plan
make check(501 unit tests)Generated by Claude Code