Repository navigation
docs: fix Nginx reverse-proxy connect timeout for fast failover - #186
Merged
Merged
Conversation
The example set proxy_connect_timeout 300s, which stalls every request landing on an unreachable node for minutes while nginx waits for the TCP handshake, so clients time out before nginx tries the next node in a multi-node upstream. Split the timeouts as suggested in rustfs/rustfs#8762: proxy_connect_timeout 5s for fast failure/failover, proxy_read_timeout and proxy_send_timeout 300s for large object transfers. Verified against RustFS 1.0.3 with nginx 1.25: dead-node stall drops from ~136s to 5s, and a 256MB transfer passes unchanged. Applied to en, zh, de, fr, ja.
|
@majinghe is attempting to deploy a commit to the overtrue's projects Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes rustfs/rustfs#8762.
The Nginx reverse-proxy guide set
proxy_connect_timeout 300sin both the S3 API and Console server blocks. With a multi-node upstream, every request that lands on a failed node stalls for minutes while Nginx waits for the TCP handshake, and clients time out long before Nginx tries the next node. The 300 s value was clearly meant for data transfer, not for connection establishment.Following the suggestion in the issue, the timeouts are now split: a short connect timeout so a dead node fails over within seconds, and generous read/send timeouts so large S3 transfers are not cut off. The reasoning is documented right below the config example and in the multi-node upstreams section, since the mistake is easy to reintroduce.
Changes
In
content/{en,zh,de,fr,ja}/developer/integration/reverse-proxy/nginx.md:location /blocks:proxy_connect_timeout 300s;→proxy_connect_timeout 5s;+proxy_read_timeout 300s;+proxy_send_timeout 300s;For reference, the other reverse-proxy guides already use short connect timeouts (HAProxy
timeout connect 10s, Envoyconnect_timeout: 5s); this brings the Nginx guide in line.Verification
Reproduced against the issue's scenario before changing the docs. Environment: Azure VM (Ubuntu 24.04), RustFS 1.0.3 in Docker (the version named in the issue), Nginx 1.25.5 (
nginx:1.25-alpine, the image used by the guide). A two-node upstream (one live RustFS node + one dead node simulated by silently dropping SYN packets, i.e. the TCP behavior of a powered-off server) withip_hashforcing one request onto the dead node:proxy_connect_timeoutGET /health/readyPUT(S3 upload)GET /health/readyPUT(S3 upload)Notes on the numbers:
net.ipv4.tcp_syn_retries=6, ≈127 s) abort the connect before Nginx's 300 s limit. Where the connect stays pending longer, the old value blocks for the full 5 minutes as reported in the issue. Either way the client waits minutes instead of seconds — the reporter's clients timed out before the failover ever happened.GETandPUTonce the connect attempt gives up (the request was never sent upstream), so the short connect timeout is all that is needed;proxy_next_upstreamdefaults are fine.Regression check with the fixed timeouts (connect 5 s / read 300 s / send 300 s): a 256 MB object uploaded (2.8 s) and downloaded (0.9 s) through the proxy both returned HTTP 200 with matching MD5 — long transfers are unaffected.
Locale coverage
en,zh,de,fr,ja— all five copies ofdeveloper/integration/reverse-proxy/nginx.mdupdated in lockstep. Code blocks are identical across locales (enforced bynpm run docs:check); prose is written in each page's language.Testing
npm run docs:check— all checks pass (including "Translated technical code blocks match English")npm run build— static site builds successfully