Skip to content

Docker proxies and imports: an upstream lost mid-blob answers 503, not 500 - #869

Merged
schmitch merged 2 commits into
mainfrom
claude/689-docker-upstream-mid-blob-503
Nov 3, 2026
Merged

schmitch merged 2 commits into
mainfrom
claude/689-docker-upstream-mid-blob-503

Conversation

@schmitch

@schmitch schmitch commented Nov 3, 2026

Copy link
Copy Markdown
Member

Fixes #689

What

When the upstream connection failed while Forge staged a blob (a reset, the idle read_timeout, the download deadline), the request answered 500 internal error and the log blamed storage. It now answers like the other upstream failures.

  • images/mod.rs: a new marker error, UpstreamBodyError, travels inside the io::Error, the same way push.rs tags request bodies with RequestBodyError. upstream_body_reader(response) wraps the upstream stream with it. In impl From<CasError> for Failure, a CasError::Io that carries the marker maps to upstream_unreachable: 503 the upstream registry is unreachable, logged as the warning registry proxy: upstream request failed. Every other I/O error from staging, such as a disk error, still answers 500 storage error.
  • Registry::proxied_blob (images/proxy.rs) stages through the new reader. This covers blobs of at most 1 MiB, such as configs, and layers that come without a Content-Length.
  • The image import (images/import.rs) stages through it too. A failed item now says the source registry is unreachable: reading <digest> failed instead of internal error.

Tests

  • Unit test: images::tests::an_unreadable_upstream_body_is_the_upstreams_error sits beside an_unreadable_request_body_is_the_clients_error. A real reqwest response is cut short against its Content-Length; the test checks it maps to the unreachable 503, and that a plain I/O error still maps to 500.
  • Integration test: registry.rs::an_upstream_cut_short_mid_blob_is_unreachable. MockRegistry gets a cut_blobs switch that sends half of a small blob and then drops the connection. The pull answers 503 with the unreachable message and caches nothing. Once the upstream recovers, the pull answers 200.

Run locally:

  • cargo fmt --all --check passes.
  • cargo clippy -p forge-server --all-targets -- -D warnings passes.
  • cargo xtask changelog --check passes.
  • The new unit and integration tests pass, along with the nearby proxy tests in registry.rs.

The changelog fragment is docs/changes/2.2.0/689.fixed.md.


Generated by Claude Code

…t 500

Staging an upstream blob tags the body's read errors with an
UpstreamBodyError inside the I/O error, as push.rs tags request bodies,
and From<CasError> for Failure maps it to the upstream-unreachable
failure (503, logged as a warning). An import's item names the source
registry as unreachable. Staging's own I/O errors stay 500.

Fixes #689

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012e6DJZDiJztZTi98zcU8Mj
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012e6DJZDiJztZTi98zcU8Mj
@schmitch
schmitch enabled auto-merge November 3, 2026 21:38
@schmitch
schmitch added this pull request to the merge queue Nov 3, 2026
Merged via the queue into main with commit b019900 Nov 3, 2026
19 checks passed
@schmitch
schmitch deleted the claude/689-docker-upstream-mid-blob-503 branch November 3, 2026 21:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Docker proxies and imports: an upstream connection lost mid-blob answers 500 "storage error" instead of 503

2 participants