gateway_proxy: shorten upstream read timeout and surface stream interruptions - #352
Open
rohita5l wants to merge 1 commit into
Open
gateway_proxy: shorten upstream read timeout and surface stream interruptions#352rohita5l wants to merge 1 commit into
rohita5l wants to merge 1 commit into
Conversation
…ruptions Lower the upstream `read` timeout from 600s to 120s. On the relayed path the pings the proxy sees are AIGW's keep-alive frames (~10s) covering only healthy streams, and AIGW aborts a genuine stall at ~60s and closes cleanly. The proxy read timeout then only needs to catch a dark upstream (pod crash / network partition, where neither tokens nor pings arrive) — 120s catches that in ~2 min while staying well above the 10s keep-alive so slow-but-healthy streams never trip it. connect/write/pool unchanged. When the upstream drops mid-stream after the head is sent, emit a terminal Anthropic `event: error` frame so Claude Code retries cleanly instead of rendering a silent truncation as "incomplete". Gate this on the body being uncompressed text/event-stream — this path also relays non-streaming JSON and possibly-gzipped bodies byte-for-byte, and injecting plaintext SSE framing into either would corrupt it (strictly worse than a clean truncation). Lead with a double CRLF to force an SSE event boundary in case the drop landed mid-line. Note: B1/B2 are necessary but not sufficient — a keep-alive-fed stall is still bounded only by AIGW's ~600s webClientRequestTimeoutMs until #2390858 lands. Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two changes to the ucode loopback refresh proxy (
src/ucode/gateway_proxy.py) so a stalled/interrupted upstream stream degrades cleanly instead of hanging for 10 minutes or rendering as a silent truncation.B1 — lower the upstream
readtimeout 600s → 120s. On the relayed path, the pings the proxy sees are AIGW's keep-alive frames (~10s) that cover only healthy streams; AIGW aborts a genuine stall at ~60s and closes cleanly (proxy sees EOF and relays it). The proxy read timeout then only needs to catch a dark upstream — pod crash / network partition, where neither tokens nor pings arrive — and 120s catches that in ~2 min instead of 10. It stays well above the 10s keep-alive interval so slow-but-healthy streams never trip it.connect/write/poolunchanged.B2 — emit a terminal SSE error frame on mid-stream drop. When
iter_rawraiseshttpx.HTTPErrorafter the head is already sent, write a well-formed Anthropicevent: errorframe before closing so Claude Code retries cleanly rather than rendering "incomplete".B3 — corrected the timeout-rationale comment to reflect the relayed path (AIGW keep-alive frames, not Anthropic's own pings).
Correctness guard on B2
_relay_responsehandles every response and is byte-transparent overContent-Encoding. So the error frame is gated on the body being uncompressedtext/event-stream— injecting plaintext SSE framing into a partial gzip body or a non-streaming JSON body would corrupt it, which is strictly worse than a clean truncation. The frame also leads with a double CRLF to force an SSE event boundary in case the drop landed mid-line.Caveat — this does not fully fix the customer hang
With AIGW keep-alive on, each
: pingresets httpx's read timer, so B1 only shortens the dark-upstream case. A keep-alive-fed stall is still bounded only by AIGW's ~600swebClientRequestTimeoutMsuntil #2390858 (A1) lands. B1/B2 are necessary but not sufficient; A1 remains the primary lever.Testing
uv run ruff check .— cleanuv run ruff format --check— cleanuv run pytest tests/test_gateway_proxy.py— 21 passedThis pull request and its description were written by Isaac.