Skip to content

gateway_proxy: shorten upstream read timeout and surface stream interruptions - #352

Open
rohita5l wants to merge 1 commit into
mainfrom
gateway-proxy-stream-timeout
Open

gateway_proxy: shorten upstream read timeout and surface stream interruptions#352
rohita5l wants to merge 1 commit into
mainfrom
gateway-proxy-stream-timeout

Conversation

@rohita5l

Copy link
Copy Markdown
Collaborator

What

Two changes to the ucode loopback refresh proxy (src/ucode/gateway_proxy.py) so a stalled/interrupted upstream stream degrades cleanly instead of hanging for 10 minutes or rendering as a silent truncation.

B1 — lower the upstream read timeout 600s → 120s. On the relayed path, the pings the proxy sees are AIGW's keep-alive frames (~10s) that cover only healthy streams; AIGW aborts a genuine stall at ~60s and closes cleanly (proxy sees EOF and relays it). The proxy read timeout then only needs to catch a dark upstream — pod crash / network partition, where neither tokens nor pings arrive — and 120s catches that in ~2 min instead of 10. It stays well above the 10s keep-alive interval so slow-but-healthy streams never trip it. connect/write/pool unchanged.

B2 — emit a terminal SSE error frame on mid-stream drop. When iter_raw raises httpx.HTTPError after the head is already sent, write a well-formed Anthropic event: error frame before closing so Claude Code retries cleanly rather than rendering "incomplete".

B3 — corrected the timeout-rationale comment to reflect the relayed path (AIGW keep-alive frames, not Anthropic's own pings).

Correctness guard on B2

_relay_response handles every response and is byte-transparent over Content-Encoding. So the error frame is gated on the body being uncompressed text/event-stream — injecting plaintext SSE framing into a partial gzip body or a non-streaming JSON body would corrupt it, which is strictly worse than a clean truncation. The frame also leads with a double CRLF to force an SSE event boundary in case the drop landed mid-line.

Caveat — this does not fully fix the customer hang

With AIGW keep-alive on, each : ping resets httpx's read timer, so B1 only shortens the dark-upstream case. A keep-alive-fed stall is still bounded only by AIGW's ~600s webClientRequestTimeoutMs until #2390858 (A1) lands. B1/B2 are necessary but not sufficient; A1 remains the primary lever.

Testing

  • uv run ruff check . — clean
  • uv run ruff format --check — clean
  • uv run pytest tests/test_gateway_proxy.py — 21 passed

This pull request and its description were written by Isaac.

…ruptions

Lower the upstream `read` timeout from 600s to 120s. On the relayed path
the pings the proxy sees are AIGW's keep-alive frames (~10s) covering only
healthy streams, and AIGW aborts a genuine stall at ~60s and closes cleanly.
The proxy read timeout then only needs to catch a dark upstream (pod crash /
network partition, where neither tokens nor pings arrive) — 120s catches that
in ~2 min while staying well above the 10s keep-alive so slow-but-healthy
streams never trip it. connect/write/pool unchanged.

When the upstream drops mid-stream after the head is sent, emit a terminal
Anthropic `event: error` frame so Claude Code retries cleanly instead of
rendering a silent truncation as "incomplete". Gate this on the body being
uncompressed text/event-stream — this path also relays non-streaming JSON and
possibly-gzipped bodies byte-for-byte, and injecting plaintext SSE framing
into either would corrupt it (strictly worse than a clean truncation). Lead
with a double CRLF to force an SSE event boundary in case the drop landed
mid-line.

Note: B1/B2 are necessary but not sufficient — a keep-alive-fed stall is still
bounded only by AIGW's ~600s webClientRequestTimeoutMs until #2390858 lands.

Co-authored-by: Isaac
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant