Shard egress-lane VMs over several raw bridges - #59
Merged
Conversation
CMGS
force-pushed
the
feat/bridges-shards
branch
3 times, most recently
from
July 30, 2026 07:26
94247d2 to
3ecee7d
Compare
The networks list shards over CNI conflists, and the CNI attachment is what it prices in: a per-VM network namespace, a CNI plugin execution, a veth pair and a TC redirect on every clone. Measured on a 384-core host filling 1000 egress VMs (quiet console, udev exec queue stopped), the conflist shards ran at 25.8/s where a single raw bridge ran 48/s — the sharding win was smaller than the machinery cost. bridges is the same shard list over host bridge devices with the raw TAP-on-bridge attachment kept: one tap in the root netns, no netns, no plugin, no veth. The same fill runs at 83.3/s — 3.2x the conflist shards, level with four separate daemons sharding one bridge each without the fourfold goldens — and the ceiling story is unchanged: each entry is one Linux bridge, so N entries give N x BR_MAX_PORTS. Because the taps stay in the root netns, guarded egress works over bridges, which the conflist form cannot offer. The shard pick is the same VM-name hash networks uses, so a record's attachment stays reproducible without process state. The scalar bridge key is retired like network before it — a one-entry bridges list is the same argv, and strict config decoding fails the old key loudly.
CMGS
force-pushed
the
feat/bridges-shards
branch
from
July 30, 2026 07:33
3ecee7d to
9059b83
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#58 gave the egress lane its shard list, but over CNI conflists — and the CNI attachment is what it prices in: every clone pays a network namespace, a CNI plugin execution, a veth pair and a TC redirect. Measured on a 384-core host filling 1000 egress VMs to exec-probed warm (quiet console, udev exec queue stopped,
refill_concurrency: 64, same harness for every row):networks×4 (#58, CNI)bridge×1 (raw tap)bridges×4 (this PR, raw tap)What it adds
bridges— a list of host bridge devices, sharded by the same stable VM-name hashnetworksuses, with the raw TAP-on-bridge attachment kept: one tap in the root netns, one RTM_SETLINK, no netns, no plugin process, no veth. Each entry is one Linux bridge, so N entries give N×BR_MAX_PORTSexactly like the conflist form.Two consequences fall out of the taps staying in the root netns:
bridges. The nft NIC lock needs the tap reachable from the root netns, which is why guarded egress rejectsnetworksat load; abridgeslist satisfies it, so the guarded lane is no longer capped at one bridge.bridgeargv.Shape
bridgekey is retired the same waynetworkwas in Spread egress-lane VMs over several CNI networks #58: strict config decoding rejects it loudly, and the deploy docs carry the one-line rename ("bridge": "br0"→"bridges": ["br0"]).bridgesandnetworksare mutually exclusive, with the same empty-name/duplicate validation.engine:networkForgeneralises toshardOf(shards, name)— same fnv hash, so a persisted record's attachment stays reproducible without process state, for either form.lockEgressarms for any bridge form.Verification
make go-lint(golangci, GOOS linux+darwin) 0 issues;go test -racegreen on config/engine/pool and e2e; gofmt clean. On hardware: the 1000-VM fill above, hash spread within 9% of even across four bridges, and the pool drained clean afterwards. Onlyrefill_concurrency: 64was measured for the new form.