Spread egress-lane VMs over several CNI networks - #58
Merged
Conversation
CMGS
force-pushed
the
pr2-none-lane-egress
branch
from
July 29, 2026 16:33
7e6e17d to
296acbc
Compare
A Linux bridge holds at most BR_MAX_PORTS ports — 1024, a compile-time kernel constant with no sysctl — and answers EXFULL for the next one. One bridge per node therefore caps egress density near a thousand VMs however much CPU and memory the node has, and a fleet run reached exactly that: nineteen of twenty nodes wedged between 1004 and 1014 ports. Measured on one node afterwards, the cost is not only the ceiling. Filling a single bridge to 1000 took 86s and the per-second rate collapsed from 105 to single digits as it filled, because br_add_if walks the bridge's port list while holding the global rtnl lock. The same node filled 1000 none-lane VMs, which take no bridge port at all, in 5s at a flat 200/s. Accept a list of conflists and hash the VM name to pick one, so N bridges give N×1024 and each stays in the range where that walk is short. Hashing rather than a counter keeps the choice free of process state: a VM resolves to the same conflist whoever asks and whenever, which matters because its record persists the network it was built on and restore has to agree. The attachment key is now `networks`, a list; a one-entry list is byte-for-byte the old argv. The scalar `network` key is retired — strict config decoding fails the load loudly instead of silently dropping the egress lane.
asl findings in pool (checkpoint/claim/pool/telemetry/pool_test) and store/s3; layout only, no behavior change.
- e2e: shared harness.Claim replaces the per-tool connect/claim prologue (7 tools; the tools with per-step timing or per-node error context keep their own) - egress: one relay() carries the forward path shared by the plain proxy and the intercept handler - mcp: parseAndBox generic replaces the per-handler parse-then-resolve prologue (10 handlers) - types: TTLField embed dedups the wire TTL conversion (4 request bodies) - config, store/peer: cmp.Or for two hand-rolled defaults
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A Linux bridge holds 1024 ports:
BR_MAX_PORTSis a compile-time constant, andfind_portno()scans linearly for a free one while holding the global rtnlmutex. So a single bridge caps the egress lane at 1024 VMs per node, and long
before that cap the scan itself becomes the bottleneck — measured on one node,
egress-lane creation peaked at 105/s and collapsed to single digits within five
seconds as the bridge filled, with load average reaching 287 while the CPU sat
idle. The none lane, which attaches nothing, held a flat ~200/s throughout.
A node may now name several CNI networks instead of one. Each VM is assigned by
hashing its own name, so VMs spread evenly across the configured networks and
the assignment is stable for a given VM without any per-node state. Four bridges
move the ceiling to ~4096 per node and cut the average port-scan length by the
same factor.
This does not remove the rtnl serialization, only divides the work that happens
under it. The lane that avoids the lock entirely is
net=none(#57), which iswhy that one is the default.
🤖 Generated with Claude Code