Skip to content

docs: fix Nginx reverse-proxy connect timeout for fast failover - #186

Merged
majinghe merged 1 commit into
rustfs:mainfrom
majinghe:docs/nginx-proxy-connect-timeout
Oct 22, 2026
Merged

majinghe merged 1 commit into
rustfs:mainfrom
majinghe:docs/nginx-proxy-connect-timeout

Conversation

@majinghe

Copy link
Copy Markdown
Collaborator

Summary

Fixes rustfs/rustfs#8762.

The Nginx reverse-proxy guide set proxy_connect_timeout 300s in both the S3 API and Console server blocks. With a multi-node upstream, every request that lands on a failed node stalls for minutes while Nginx waits for the TCP handshake, and clients time out long before Nginx tries the next node. The 300 s value was clearly meant for data transfer, not for connection establishment.

Following the suggestion in the issue, the timeouts are now split: a short connect timeout so a dead node fails over within seconds, and generous read/send timeouts so large S3 transfers are not cut off. The reasoning is documented right below the config example and in the multi-node upstreams section, since the mistake is easy to reintroduce.

Changes

In content/{en,zh,de,fr,ja}/developer/integration/reverse-proxy/nginx.md:

  • S3 and Console location / blocks: proxy_connect_timeout 300s; → proxy_connect_timeout 5s; + proxy_read_timeout 300s; + proxy_send_timeout 300s;
  • Added a sentence after the config example explaining the short-connect / long-transfer split.
  • Added a sentence in "Multi-node upstreams" explaining what a long connect timeout costs during failover.

For reference, the other reverse-proxy guides already use short connect timeouts (HAProxy timeout connect 10s, Envoy connect_timeout: 5s); this brings the Nginx guide in line.

Verification

Reproduced against the issue's scenario before changing the docs. Environment: Azure VM (Ubuntu 24.04), RustFS 1.0.3 in Docker (the version named in the issue), Nginx 1.25.5 (nginx:1.25-alpine, the image used by the guide). A two-node upstream (one live RustFS node + one dead node simulated by silently dropping SYN packets, i.e. the TCP behavior of a powered-off server) with ip_hash forcing one request onto the dead node:

Request proxy_connect_timeout Client-visible stall on dead node Outcome
GET /health/ready 300 s (old docs value) 135.6 s fails over to live node, HTTP 200
PUT (S3 upload) 300 s (old docs value) 135.6 s fails over to live node, HTTP 200
GET /health/ready 5 s (this fix) 5.0 s fails over to live node, HTTP 200
PUT (S3 upload) 5 s (this fix) 5.0 s fails over to live node, HTTP 200
control requests hitting the live node both 1.3–1.6 ms HTTP 200

Notes on the numbers:

  • The 300 s stall capped at ~136 s on this host because the Linux kernel's SYN retries (net.ipv4.tcp_syn_retries=6, ≈127 s) abort the connect before Nginx's 300 s limit. Where the connect stays pending longer, the old value blocks for the full 5 minutes as reported in the issue. Either way the client waits minutes instead of seconds — the reporter's clients timed out before the failover ever happened.
  • Failover itself works for both GET and PUT once the connect attempt gives up (the request was never sent upstream), so the short connect timeout is all that is needed; proxy_next_upstream defaults are fine.

Regression check with the fixed timeouts (connect 5 s / read 300 s / send 300 s): a 256 MB object uploaded (2.8 s) and downloaded (0.9 s) through the proxy both returned HTTP 200 with matching MD5 — long transfers are unaffected.

Locale coverage

en, zh, de, fr, ja — all five copies of developer/integration/reverse-proxy/nginx.md updated in lockstep. Code blocks are identical across locales (enforced by npm run docs:check); prose is written in each page's language.

Testing

  • npm run docs:check — all checks pass (including "Translated technical code blocks match English")
  • npm run build — static site builds successfully

The example set proxy_connect_timeout 300s, which stalls every request
landing on an unreachable node for minutes while nginx waits for the TCP
handshake, so clients time out before nginx tries the next node in a
multi-node upstream. Split the timeouts as suggested in rustfs/rustfs#8762:
proxy_connect_timeout 5s for fast failure/failover, proxy_read_timeout and
proxy_send_timeout 300s for large object transfers. Verified against
RustFS 1.0.3 with nginx 1.25: dead-node stall drops from ~136s to 5s, and a
256MB transfer passes unchanged. Applied to en, zh, de, fr, ja.
@vercel

vercel Bot commented Oct 21, 2026

Copy link
Copy Markdown

@majinghe is attempting to deploy a commit to the overtrue's projects Team on Vercel.

A member of the Team first needs to authorize it.

@majinghe
majinghe merged commit d03e833 into rustfs:main Oct 22, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Documentation: Nginx proxy_connect_timeout wrong?

1 participant