fix(#4060): extend backoff gate to remote MCP HTTP errors

docs/tools/lsp/index.md

@@ -197,7 +197,7 @@ Available Capabilities:
 
 LSP toolsets are managed by the same supervisor as MCP toolsets, so a crashed `gopls` (or any other language server) is reconnected automatically with exponential backoff. Use the [`lifecycle`](../../configuration/tools/index.md#toolset-lifecycle) block to tune the policy per toolset — for example, mark `gopls` as `strict` if your CI flow requires it to be available, or use `/toolset-restart gopls` from the TUI to force a reconnect when the server gets stuck.
 
-**Startup failure behaviour:** local LSP server startup failures (missing binary, server-unavailable) fail fast — each turn retries immediately with no artificial delay. The rate-limit backoff gate applies only to model-provider embedding calls (see [Indexing failures, retries and backoff](../rag/index.md#indexing-failures-retries-and-backoff)); it does not apply to LSP server startup.
+**Startup failure behaviour:** missing-binary and bad-config failures fail fast — each turn retries immediately with no artificial delay. A language server that crash-loops is not currently paced by the backoff gate; the supervisor's own reconnect policy (controlled by the `lifecycle` block) is the primary throttle for crash recovery.
 
 ```yaml
 toolsets:

docs/tools/mcp/index.md

@@ -258,7 +258,7 @@ toolsets:
 
 See [Toolset Lifecycle](../../configuration/tools/index.md#toolset-lifecycle) for all profiles and tuning knobs, and [`/toolset-restart`](../../features/tui/index.md) to force a reconnect from the TUI.
 
-**Startup failure behaviour:** local MCP startup failures (missing binary, connection refused, authentication error) fail fast — each turn retries immediately with no artificial delay. The rate-limit backoff gate applies only to model-provider embedding calls (see [Indexing failures, retries and backoff](../rag/index.md#indexing-failures-retries-and-backoff)); it does not apply to MCP server startup.
+**Startup failure behaviour:** local MCP failures (missing binary, connection refused, bad auth) fail fast — each turn retries immediately with no artificial delay. Remote MCP servers (Streamable HTTP / SSE) that respond with one of a fixed set of retryable HTTP statuses — 429 Too Many Requests, 408 Request Timeout, 500/502/503/504, or 529 (Anthropic-style "overloaded") — are paced by the same [bounded exponential backoff gate](../rag/index.md#indexing-failures-retries-and-backoff) that RAG embedding calls use, so a temporarily-overloaded remote MCP server does not trigger a new connect attempt on every agent turn. This is a fixed enumeration, not a full 5xx range: less-common codes such as 501, 505, or the Cloudflare 520–527 family do not arm the gate. Note MCP's trigger set is broader than RAG's current 429-only pacing (see the linked page and [#4097](https://github.com/docker/docker-agent/issues/4097)). Not yet applied when toolsets are wrapped in code mode — see [#4067](https://github.com/docker/docker-agent/issues/4067).
 
 ## Combined Example
 

docs/tools/rag/index.md

@@ -183,10 +183,12 @@ every agent turn.
 
 ### What triggers backoff
 
-Backoff applies only to **HTTP 429 rate-limit** responses from the embedding or
-model provider — the one signal that reliably reaches the toolset gate. Other
-errors (5xx, 408) are handled per-file within the indexing run and do not arm
-the gate. These are the current gate triggers:
+For **RAG indexing**, backoff applies only to **HTTP 429 rate-limit** responses
+from the embedding or model provider — the one signal that reliably reaches the
+toolset gate. Other errors (5xx, 408) are handled per-file within the indexing
+run and do not arm the gate (tracked as a gap in
+[#4097](https://github.com/docker/docker-agent/issues/4097)). These are the
+current gate triggers for RAG indexing specifically:
 
 | Failure kind | Behaviour |
 |---|---|
@@ -198,6 +200,11 @@ the gate. These are the current gate triggers:
 > 5xx and 408 errors from the embedding provider are retried per-file and do not
 > propagate to the toolset gate. Only 429 (rate-limit) terminates the indexing run
 > early and surfaces the gate so Docker Agent can pace the next attempt.
+>
+> This 429-only trigger set is specific to the RAG/embedding path. Other toolset
+> types have their own trigger sets against the same gate — for example, remote
+> MCP toolsets also pace on 408 and a fixed set of 5xx-family statuses (see
+> [MCP startup failure behaviour](../mcp/index.md#lifecycle-auto-restart-profiles)).
 
 ### Retry policy and parameters
 

pkg/tools/mcp/oauth.go

@@ -500,6 +500,12 @@ type oauthTransport struct {
 	// swallows in favor of a bare http.StatusText.
 	lastErrStatus int
 	lastErrBody   []byte
+	// lastErrRetryAfter captures the raw Retry-After header value (if any) of
+	// the most recent non-2xx response, so enrichConnectError can forward it
+	// to modelerrors.WrapHTTPError and have the StartableToolSet backoff gate
+	// honor a server-supplied retry hint instead of falling back to the
+	// generic computed delay.
+	lastErrRetryAfter string
 	// lastAuthRequired records when the transport short-circuited an
 	// interactive OAuth flow because the request context disallowed
 	// prompts (see WithoutInteractivePrompts). The MCP SDK wraps transport
@@ -874,6 +880,7 @@ func (t *oauthTransport) logErrorResponse(req *http.Request, resp *http.Response
 	t.mu.Lock()
 	t.lastErrStatus = resp.StatusCode
 	t.lastErrBody = body
+	t.lastErrRetryAfter = resp.Header.Get("Retry-After")
 	t.mu.Unlock()
 
 	slog.Warn("Authenticated MCP request was rejected by the server",
@@ -885,23 +892,33 @@ func (t *oauthTransport) logErrorResponse(req *http.Request, resp *http.Response
 	)
 }
 
-// lastServerError returns the status code and a short, human-readable
-// explanation drawn from the most recent non-2xx response seen by this
-// transport. The string is empty when no such response has been captured
-// or when the body yielded no useful text.
+// lastServerErrorSnapshot returns the status code, a short human-readable
+// explanation, and the raw Retry-After header value, all captured together
+// under a single lock from the most recent non-2xx response seen by this
+// transport. status is 0 when no such response has been captured; msg and
+// retryAfter are "" when the body yielded no useful text / no header was
+// present, respectively.
+//
+// The three fields are read under one lock (rather than via separate
+// accessors) so a caller building a combined error never pairs a status
+// captured from one response with a Retry-After header captured from a
+// different, concurrent one: this transport's RoundTrip can be invoked
+// concurrently for a single logical connect attempt (e.g. a standalone SSE
+// probe alongside the initialize call).
 //
 // This is how the transport surfaces provider-specific errors (e.g. Slack's
 // "App is not enabled for Slack MCP server access") that would otherwise
 // be hidden behind the MCP SDK's generic http.StatusText-derived messages.
-func (t *oauthTransport) lastServerError() (int, string) {
+func (t *oauthTransport) lastServerErrorSnapshot() (status int, msg, retryAfter string) {
 	t.mu.Lock()
-	status := t.lastErrStatus
+	status = t.lastErrStatus
 	body := t.lastErrBody
+	retryAfter = t.lastErrRetryAfter
 	t.mu.Unlock()
 	if status == 0 {
-		return 0, ""
+		return 0, "", ""
 	}
-	return status, extractServerMessage(body)
+	return status, extractServerMessage(body), retryAfter
 }
 
 // authorizationRequired reports whether the transport short-circuited an

pkg/tools/mcp/remote.go

@@ -16,6 +16,7 @@ import (
 	"github.com/docker/docker-agent/pkg/environment"
 	"github.com/docker/docker-agent/pkg/httpclient"
 	"github.com/docker/docker-agent/pkg/js"
+	"github.com/docker/docker-agent/pkg/modelerrors"
 	"github.com/docker/docker-agent/pkg/upstream"
 )
 
@@ -211,8 +212,32 @@ func enrichConnectError(err error, t *oauthTransport) error {
 	if t.authorizationRequired() {
 		return &AuthorizationRequiredError{URL: t.baseURL}
 	}
-	if status, msg := t.lastServerError(); status != 0 && msg != "" {
-		return fmt.Errorf("failed to connect to MCP server: %w (server responded %d: %s)", err, status, msg)
+	// Wrap on status alone: many rate-limit / load-balancer 429s and 503s
+	// carry an empty body, so gating on msg != "" (as an earlier version of
+	// this code did) silently dropped the *modelerrors.StatusError wrap —
+	// and with it, the StartableToolSet backoff gate never armed.
+	//
+	// status, msg and retryAfter are read together as a single snapshot
+	// (rather than via two separately-locked accessor calls) so they can
+	// never be pieced together from two different concurrent responses on
+	// this transport (e.g. a standalone SSE probe racing the initialize call).
+	if status, msg, retryAfter := t.lastServerErrorSnapshot(); status != 0 {
+		var enriched error
+		if msg != "" {
+			enriched = fmt.Errorf("failed to connect to MCP server: %w (server responded %d: %s)", err, status, msg)
+		} else {
+			// No status text extracted from the body: modelerrors.StatusError.Error()
+			// already prefixes "HTTP %d: ", so repeating the code here would read as
+			// "HTTP 503: ... (server responded 503)".
+			enriched = fmt.Errorf("failed to connect to MCP server: %w", err)
+		}
+		// Forward the server's Retry-After hint (if any) so the backoff gate
+		// honors it instead of falling back to the generic computed delay.
+		resp := &http.Response{Header: http.Header{}}
+		if retryAfter != "" {
+			resp.Header.Set("Retry-After", retryAfter)
+		}
+		return modelerrors.WrapHTTPError(status, resp, enriched)
 	}
 	return fmt.Errorf("failed to connect to MCP server: %w", err)
 }
@@ -249,8 +274,9 @@ func (c *remoteMCPClient) SetUnmanagedOAuthRedirectURI(uri string) {
 // values never go stale on a long-lived connection.
 //
 // The oauthTransport is returned alongside the client so callers can inspect
-// the most recent server-side failure (via lastServerError) when Connect()
-// returns a bare HTTP-status error and we need to surface the actual cause.
+// the most recent server-side failure (via lastServerErrorSnapshot) when
+// Connect() returns a bare HTTP-status error and we need to surface the
+// actual cause.
 //
 // The transport chain wraps `httpclient.WrapWithOTel` outermost so every
 // outbound MCP request injects W3C `traceparent` (and creates an HTTP