atenet-router: raise actor-cluster circuit breakers - #806
Open
Chuang Wang (chuangw6) wants to merge 1 commit into
Open
atenet-router: raise actor-cluster circuit breakers#806Chuang Wang (chuangw6) wants to merge 1 commit into
Chuang Wang (chuangw6) wants to merge 1 commit into
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
Chuang Wang (chuangw6)
force-pushed
the
router-limits
branch
from
August 8, 2026 00:47
d98eeca to
5cfc492
Compare
Chuang Wang (chuangw6)
changed the title
Chuang Wang (chuangw6)
force-pushed
the
router-limits
branch
5 times, most recently
from
August 8, 2026 01:12
3c308a7 to
d9139a5
Compare
Bowei Du (bowei)
left a comment
Bowei Du (bowei)
left a comment
Collaborator
There was a problem hiding this comment.
I see you added new limits and constants -- but how do we expect it to behave for people who install it on a smaller machine than you are giving space for?
Collaborator
|
Collaborator
|
Ok, let's remove the config that won't be needed in version 37 and we should be good with this PR |
Chuang Wang (chuangw6)
force-pushed
the
router-limits
branch
2 times, most recently
from
August 12, 2026 19:01
2a93a40 to
363c405
Compare
Bowei Du (bowei)
previously approved these changes
Aug 12, 2026
Chuang Wang (chuangw6)
force-pushed
the
router-limits
branch
from
August 12, 2026 19:05
363c405 to
f70a5d1
Compare
Bowei Du (bowei)
previously approved these changes
Aug 12, 2026
| t.Errorf("max_requests after SetExtProcMaxRequests(0) = %d, want default %d", got, defaultExtProcMaxRequests) | ||
| } | ||
| }) | ||
|
|
Collaborator
There was a problem hiding this comment.
remove these test -- this is overkill
Collaborator
Author
There was a problem hiding this comment.
Done — removed both added tests; the change ships with no test diff.
Chuang Wang (chuangw6)
force-pushed
the
router-limits
branch
2 times, most recently
from
August 17, 2026 18:48
f70a5d1 to
31a6e53
Compare
Chuang Wang (chuangw6)
force-pushed
the
router-limits
branch
from
August 17, 2026 18:51
31a6e53 to
4681572
Compare
The actor cluster configured no breakers, leaving Envoy's 1,024 default, which caps concurrent requests through the router far below what it can carry. Connections, pending and requests rise to 20,000, kept under the ~28k source-port budget so overload trips a counted breaker instead of the kernel's opaque EADDRNOTAVAIL; max_retries keeps its default since nothing here retries. The ext_proc breaker now applies its ceiling to max_pending_requests as well as max_requests.
Chuang Wang (chuangw6)
force-pushed
the
router-limits
branch
from
August 17, 2026 18:57
4681572 to
a7b1ade
Compare
Collaborator
|
Get the tests to pass |
Bowei Du (bowei)
approved these changes
Aug 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The actor cluster configured no circuit breakers, leaving Envoy's 1,024 default — a low ceiling for the one proxy carrying every actor's ingress. Under overload it surfaces as storms of instant 503s once concurrency crosses 1,024.
max_connections,max_pending_requestsandmax_requestsrise to 20,000, kept under the ~28k ephemeral-port budget so overload trips a counted breaker instead of the kernel's opaqueEADDRNOTAVAIL.max_retrieskeeps Envoy's default — nothing on this cluster retries.--extproc-max-requestsceiling now coversmax_pending_requestsas well asmax_requests.Tested with
go test ./cmd/atenet/internal/router/.