Shard egress-lane VMs over several raw bridges

docs/cluster.md

@@ -179,7 +179,7 @@ fully from it:
 
 | state | source of truth | survives restart |
 |---|---|---|
-| operator config (`tenants`, `secrets`, egress policies, `bridge`/`networks`, `mesh`, `preview_secret`, `egress_ca`) | `config.json` (human/deploy-tool owned) | re-read at boot |
+| operator config (`tenants`, `secrets`, egress policies, `bridges`/`networks`, `mesh`, `preview_secret`, `egress_ca`) | `config.json` (human/deploy-tool owned) | re-read at boot |
 | API-applied pool targets (`PUT /v1/pools`) | `<data_dir>/pools.json` (machine owned) | yes |
 | claims | the claims journal + `Reconcile` | yes |
 | placement hints (warm counts, template sets) | gossip | rebuilt |

docs/deploy.md

@@ -32,6 +32,11 @@ This CH-only release does not convert existing VM or snapshot state. Drain old
 claims and use fresh `data_dir` and `checkpoint_dir` locations when upgrading;
 older checkpoints and promoted templates must not be reused.
 
+The scalar egress-attachment keys are retired: rename `"bridge": "br0"` to
+`"bridges": ["br0"]` and `"network": "cni"` to `"networks": ["cni"]` before
+starting the new binary — config loading rejects the old spellings loudly
+rather than silently dropping the egress lane.
+
 ## Configuration
 
 sandboxd reads one JSON file (`-config`, default
@@ -45,7 +50,7 @@ sandboxd reads one JSON file (`-config`, default
   "restore_mode": "mmap",
   "no_direct_io": true,
   "advertise_addr": "10.0.0.5:7777",
-  "bridge": "br0",
+  "bridges": ["br0"],
   "api_token": "…",
   "mesh": {
     "node_id": "node-a",
@@ -68,7 +73,7 @@ sandboxd reads one JSON file (`-config`, default
 | `restore_mode` | unset | clone and wake-restore memory mode: `copy`, `ondemand`, or `mmap`; use `mmap` for dense pools |
 | `no_direct_io` | false | use buffered writable disks for Cloud Hypervisor cold boots and clones; recommended for dense ephemeral pools to avoid direct-I/O CoW journal contention |
 | `advertise_addr` | = `listen` | the host:port clients reach this node at; returned as a claim's owner address and gossiped to peers. Must be routable when `listen` is a wildcard |
-| `bridge` / `networks` | unset | egress-lane attachment: a host bridge device, or a list of CNI conflist names. Mutually exclusive; with neither set the node serves only the no-network lane. Each conflist attaches its own Linux bridge and a bridge holds at most 1024 ports (kernel `BR_MAX_PORTS`), so N conflists raise the node's egress ceiling to N×1024 — VMs spread over them by a stable hash of the VM name, so size the list with headroom (the spread is statistical, not exact). [Guarded egress](egress.md) needs the bridge form and rejects a CNI network at load |
+| `bridges` / `networks` | unset | egress-lane attachment: a list of host bridge devices, or a list of CNI conflist names. Mutually exclusive; with neither set the node serves only the no-network lane. A Linux bridge holds at most 1024 ports (kernel `BR_MAX_PORTS`), so an N-entry list raises the node's egress ceiling to N×1024 — VMs spread over the list by a stable hash of the VM name, so size it with headroom (the spread is statistical, not exact). `bridges` keeps the raw TAP-on-bridge attachment (taps in the root netns, no per-VM network namespace or CNI plugin execution); `networks` runs the CNI chain per VM. [Guarded egress](egress.md) needs `bridges` and rejects a CNI network at load |
 | `egress_ca` | unset | [HTTPS-interception](egress.md#https-interception) PKI: `root_cert` (the cluster root baked into intercepted guests; may bundle old+new roots during rotation) plus this node's `intermediate_cert`/`intermediate_key` from `sandboxd ca issue-intermediate`. Required when any pool rule sets `intercept` |
 | `api_token` | unset | the operator (root) credential: when set, guards the node-level endpoints (Bearer) with full access, including release-by-id cleanup. Per-sandbox tokens guard ordinary sandbox-scoped calls |
 | `tenants` | unset | multi-tenant tokens next to `api_token`: `[{"name": "acme", "token": "…", "max_claims": 50}]`. A tenant token reaches the resource-creating verbs (claim, fork, promote, checkpoint, preview) and everything it creates is stamped with the tenant name; operator surfaces (`GET /v1/sandboxes` and the per-id reads under it, `GET /v1/info`, `PUT /v1/pools`, `POST/DELETE /v1/drain`, `/metrics`) answer it 403. `max_claims` (0 = unlimited) caps that tenant's live claims next to the node-wide cap. Requires `api_token` set (operator surfaces need it). Names and tokens must be unique, tokens distinct from `api_token`. On a cluster all nodes must carry the same tenants set (the SDK replays a tenant token across a redirect; a peer missing that tenant answers 401), and per-node caps mean a tenant's effective cluster limit is `max_claims` × nodes. Empty = exactly the single-token behavior |
@@ -112,7 +117,7 @@ here validates on load:
   "listen": ":7777",
   "data_dir": "/var/lib/sandboxd",
   "advertise_addr": "10.0.0.5:7777",
-  "bridge": "br0",
+  "bridges": ["br0"],
   "restore_mode": "mmap",
   "no_direct_io": true,
 

docs/egress.md

@@ -83,10 +83,10 @@ domain policy first; the allow-list widens the IP gate only.
   reach the local L2 segment (never routed off-link). Give egress-lane VMs a
   bridge they do not share with an untrusted listener.
 - **Bridge lane only (egress lane).** A CNI network's tap lives in the VM netns,
-  out of reach of the root-netns lock, so a guarded egress *lane* needs a bridge
-  and is rejected on CNI `networks`. None-lane policies ride the proxy and work
-  on either. A bridge egress lane locks every NIC default-deny, even with no
-  policy configured.
+  out of reach of the root-netns lock, so a guarded egress *lane* needs the
+  `bridges` form (those taps stay in the root netns) and is rejected on CNI
+  `networks`. None-lane policies ride the proxy and work on either. A bridge
+  egress lane locks every NIC default-deny, even with no policy configured.
 - **No custom NAT64/DNS64 prefix routed to the host.** The SSRF guard folds the
   standard NAT64 forms (RFC 6052 well-known `64:ff9b::/96`, RFC 8215 local-use
   `64:ff9b:1::/48`), but an operator-specific network-specific prefix (RFC 6052
@@ -103,7 +103,7 @@ the environment, never the config file.
 
 ```jsonc
 {
-  "bridge": "sbxbr0",                       // egress-lane pools only; none-lane needs no attachment
+  "bridges": ["sbxbr0"],                    // egress-lane pools only; none-lane needs no attachment
   "secrets": [
     { "name": "gh", "header": "Authorization", "value_env": "GH_TOKEN" }
   ],

e2e/fakeengine_test.go

@@ -31,7 +31,7 @@ type fakeEngine struct {
 
 func newFakeEngine(dir string) *fakeEngine {
 	return &fakeEngine{
-		real:      engine.New("cocoon", "", nil, false, ""),
+		real:      engine.New("cocoon", nil, nil, false, ""),
 		dir:       dir,
 		listeners: map[string]io.Closer{},
 		socks:     map[string]string{},

sandboxd/config/config.go

@@ -170,10 +170,11 @@ type Config struct {
 	// to Listen, which is correct when Listen is a routable host:port.
 	AdvertiseAddr string `json:"advertise_addr,omitempty"`
 
-	// Bridge and Networks pick the egress-lane attachment (TAP-on-bridge vs
-	// CNI conflists); mutually exclusive. With neither set the node serves
-	// only the no-network lane.
-	Bridge string `json:"bridge,omitempty"`
+	// Bridges shards egress-lane VMs over several host bridge devices with
+	// the raw TAP-on-bridge attachment: taps stay in the root netns, so
+	// guarded egress can lock them. Mutually exclusive with Networks; with
+	// neither set the node serves only the no-network lane.
+	Bridges []string `json:"bridges,omitempty"`
 
 	// Networks shards egress-lane VMs over several CNI conflists, one Linux
 	// bridge each: a bridge holds at most 1024 ports (BR_MAX_PORTS, no
@@ -276,7 +277,7 @@ type Config struct {
 
 // HasEgress reports whether the node can attach egress-lane VMs.
 func (c *Config) HasEgress() bool {
-	return c.Bridge != "" || len(c.Networks) > 0
+	return len(c.Bridges) > 0 || len(c.Networks) > 0
 }
 
 // ClusterDigest fingerprints the must-match config so a divergent node is
@@ -336,7 +337,7 @@ func autoRefillConcurrency(cpus int) int {
 }
 
 func (c *Config) validate() error {
-	if err := c.validateNetworks(); err != nil {
+	if err := c.validateAttachment(); err != nil {
 		return err
 	}
 	// A CNI network's tap lives in the VM netns, unreachable from the root-netns
@@ -459,19 +460,26 @@ func (c *Config) validateSecrets() (map[string]struct{}, error) {
 	return names, nil
 }
 
-// validateNetworks checks the egress-lane attachment; a repeated conflist
+// validateAttachment checks the egress-lane attachment; a repeated shard
 // would report N shards while filling one bridge.
-func (c *Config) validateNetworks() error {
-	if c.Bridge != "" && len(c.Networks) > 0 {
-		return fmt.Errorf("bridge and networks are mutually exclusive")
+func (c *Config) validateAttachment() error {
+	if len(c.Bridges) > 0 && len(c.Networks) > 0 {
+		return fmt.Errorf("bridges and networks are mutually exclusive")
 	}
-	seen := make(map[string]struct{}, len(c.Networks))
-	for _, n := range c.Networks {
+	if err := validateShards(c.Bridges, "bridges", "bridge device"); err != nil {
+		return err
+	}
+	return validateShards(c.Networks, "networks", "conflist")
+}
+
+func validateShards(names []string, field, kind string) error {
+	seen := make(map[string]struct{}, len(names))
+	for _, n := range names {
 		if n == "" {
-			return fmt.Errorf("networks must not contain an empty conflist name")
+			return fmt.Errorf("%s must not contain an empty %s name", field, kind)
 		}
 		if _, ok := seen[n]; ok {
-			return fmt.Errorf("duplicate conflist %q", n)
+			return fmt.Errorf("duplicate %s %q", kind, n)
 		}
 		seen[n] = struct{}{}
 	}

sandboxd/config/config_test.go

@@ -85,7 +85,6 @@ func TestLoadRejectsInvalid(t *testing.T) {
 		name, body, want string
 	}{
 		{"bad json", `{`, "config"},
-		{"bridge and networks", `{"bridge":"br0","networks":["cni"],"pools":[]}`, "mutually exclusive"},
 		{"bad fork count", `{"max_fork_count":-1,"pools":[]}`, "max_fork_count"},
 		{"negative refill concurrency", `{"refill_concurrency":-1,"pools":[]}`, "refill_concurrency"},
 		{"bad restore mode", `{"restore_mode":"Mmap","pools":[]}`, "restore_mode"},
@@ -218,14 +217,14 @@ func TestHasEgress(t *testing.T) {
 	if (&Config{}).HasEgress() {
 		t.Error("no attachment must mean no egress")
 	}
-	if !(&Config{Bridge: "br0"}).HasEgress() || !(&Config{Networks: []string{"cni"}}).HasEgress() {
-		t.Error("bridge or network must enable egress")
+	if !(&Config{Bridges: []string{"br0", "br1"}}).HasEgress() || !(&Config{Networks: []string{"cni"}}).HasEgress() {
+		t.Error("bridges or networks must enable egress")
 	}
 }
 
 func TestLoadKeepsExplicitValues(t *testing.T) {
 	path := writeConfig(t, `{"listen":"0.0.0.0:9999","advertise_addr":"10.0.0.5:9999","max_fork_count":4,
-		"refill_concurrency":8,"no_direct_io":true,"bridge":"br0","pools":[{"template":"rt:24.04","net":"egress","size":"small","warm":3}]}`)
+		"refill_concurrency":8,"no_direct_io":true,"bridges":["br0"],"pools":[{"template":"rt:24.04","net":"egress","size":"small","warm":3}]}`)
 	cfg, err := Load(path)
 	if err != nil {
 		t.Fatalf("Load: %v", err)
@@ -240,10 +239,13 @@ func TestLoadKeepsExplicitValues(t *testing.T) {
 
 func TestLoadRejectsUnusableNetworkLists(t *testing.T) {
 	for name, body := range map[string]string{
-		"bridge and networks together": `{"bridge":"br0","networks":["a"],"pools":[]}`,
-		"retired scalar network key":   `{"network":"a","pools":[]}`,
-		"empty conflist name":          `{"networks":["a",""],"pools":[]}`,
-		"repeated conflist":            `{"networks":["a","b","a"],"pools":[]}`,
+		"bridges and networks together": `{"bridges":["br0"],"networks":["a"],"pools":[]}`,
+		"retired scalar network key":    `{"network":"a","pools":[]}`,
+		"retired scalar bridge key":     `{"bridge":"br0","pools":[]}`,
+		"empty conflist name":           `{"networks":["a",""],"pools":[]}`,
+		"empty bridge name":             `{"bridges":["br0",""],"pools":[]}`,
+		"repeated conflist":             `{"networks":["a","b","a"],"pools":[]}`,
+		"repeated bridge":               `{"bridges":["br0","br1","br0"],"pools":[]}`,
 	} {
 		t.Run(name, func(t *testing.T) {
 			if _, err := Load(writeConfig(t, body)); err == nil {
@@ -253,6 +255,13 @@ func TestLoadRejectsUnusableNetworkLists(t *testing.T) {
 	}
 }
 
+func TestLoadAcceptsGuardedEgressOnABridgesList(t *testing.T) {
+	path := writeConfig(t, `{"bridges":["sbx0","sbx1"],"pools":[{"template":"rt:24.04","net":"egress","size":"small","egress":{"allow":[{"host":"x"}]}}]}`)
+	if _, err := Load(path); err != nil {
+		t.Fatalf("guarded egress on a bridges list must load (taps stay in the root netns): %v", err)
+	}
+}
+
 func TestLoadAcceptsAShardedNetworkList(t *testing.T) {
 	path := writeConfig(t, `{"networks":["cocoon-sbx0","cocoon-sbx1","cocoon-sbx2","cocoon-sbx3"],"pools":[]}`)
 	cfg, err := Load(path)

sandboxd/engine/cloneargs_test.go

@@ -23,7 +23,7 @@ func TestCloneArgsRestoreMode(t *testing.T) {
 	}
 	for _, tc := range cases {
 		t.Run(tc.name, func(t *testing.T) {
-			e := New("cocoon", "br0", nil, false, tc.mode)
+			e := New("cocoon", []string{"br0"}, nil, false, tc.mode)
 			for _, args := range [][]string{
 				e.cloneArgs("/goldens/g1", "sbx-1", tc.key),
 				e.cloneSnapArgs("ck_1", "sbx-1", tc.key),
@@ -45,7 +45,7 @@ func TestLifecycleArgsApplyDirectIOPolicy(t *testing.T) {
 	key := types.PoolKey{Template: "rt:24.04", Net: types.NetNone, Size: types.SizeMedium}
 	for _, noDirectIO := range []bool{false, true} {
 		t.Run(strconv.FormatBool(noDirectIO), func(t *testing.T) {
-			e := New("cocoon", "", nil, noDirectIO, "")
+			e := New("cocoon", nil, nil, noDirectIO, "")
 			want := "--no-direct-io=" + strconv.FormatBool(noDirectIO)
 			cold := e.runColdArgs("sbx-1", key)
 			for _, args := range [][]string{
@@ -64,38 +64,46 @@ func TestLifecycleArgsApplyDirectIOPolicy(t *testing.T) {
 	}
 }
 
-// TestEgressVMsSpreadOverEveryConfiguredNetwork pins the point of the list:
-// a shard the hash never picks is bridge capacity that does not exist.
-func TestEgressVMsSpreadOverEveryConfiguredNetwork(t *testing.T) {
-	nets := []string{"cocoon-sbx0", "cocoon-sbx1", "cocoon-sbx2", "cocoon-sbx3"}
-	e := New("cocoon", "", nets, false, "")
+// TestEgressVMsSpreadOverEveryConfiguredShard pins the point of both shard
+// lists: a shard the hash never picks is bridge capacity that does not exist.
+func TestEgressVMsSpreadOverEveryConfiguredShard(t *testing.T) {
+	shards := []string{"sbx0", "sbx1", "sbx2", "sbx3"}
 	key := types.PoolKey{Template: "rt:24.04", Net: types.NetEgress, Size: types.SizeMedium}
-
-	counts := map[string]int{}
-	for i := range 4000 {
-		args := e.cloneArgs("/goldens/g1", "sbx-pool1-"+strconv.Itoa(i), key)
-		j := slices.Index(args, "--network")
-		if j < 0 {
-			t.Fatalf("args %v carry no --network", args)
-		}
-		counts[args[j+1]]++
-	}
-	for _, n := range nets {
-		// The bound catches a starved shard, not non-uniformity.
-		if counts[n] < 4000/len(nets)/2 {
-			t.Errorf("shard %s got %d of 4000, want a fair share: %v", n, counts[n], counts)
-		}
+	for name, tc := range map[string]struct {
+		e    *Engine
+		flag string
+	}{
+		"networks": {New("cocoon", nil, shards, false, ""), "--network"},
+		"bridges":  {New("cocoon", shards, nil, false, ""), "--bridge"},
+	} {
+		t.Run(name, func(t *testing.T) {
+			counts := map[string]int{}
+			for i := range 4000 {
+				args := tc.e.cloneArgs("/goldens/g1", "sbx-pool1-"+strconv.Itoa(i), key)
+				j := slices.Index(args, tc.flag)
+				if j < 0 {
+					t.Fatalf("args %v carry no %s", args, tc.flag)
+				}
+				counts[args[j+1]]++
+			}
+			for _, s := range shards {
+				// The bound catches a starved shard, not non-uniformity.
+				if counts[s] < 4000/len(shards)/2 {
+					t.Errorf("shard %s got %d of 4000, want a fair share: %v", s, counts[s], counts)
+				}
+			}
+		})
 	}
 }
 
 // TestNetworkChoiceIsStableForAName guards restore and teardown: the record
 // persists the network a VM was built on, so the choice must not move.
 func TestNetworkChoiceIsStableForAName(t *testing.T) {
-	e := New("cocoon", "", []string{"a", "b", "c"}, false, "")
-	first := e.networkFor("sbx-pool1-42")
+	shards := []string{"a", "b", "c"}
+	first := shardOf(shards, "sbx-pool1-42")
 	for range 100 {
-		if got := e.networkFor("sbx-pool1-42"); got != first {
-			t.Fatalf("networkFor returned %q then %q for one name", first, got)
+		if got := shardOf(shards, "sbx-pool1-42"); got != first {
+			t.Fatalf("shardOf returned %q then %q for one name", first, got)
 		}
 	}
 }
@@ -107,13 +115,13 @@ func TestNetArgsHonorsTheLaneAndTheAttachment(t *testing.T) {
 	none := types.PoolKey{Template: "rt:24.04", Net: types.NetNone, Size: types.SizeMedium}
 	egress := types.PoolKey{Template: "rt:24.04", Net: types.NetEgress, Size: types.SizeMedium}
 
-	if args := New("cocoon", "", []string{"cni"}, false, "").netArgs("sbx-1", none, false); len(args) != 0 {
+	if args := New("cocoon", nil, []string{"cni"}, false, "").netArgs("sbx-1", none, false); len(args) != 0 {
 		t.Errorf("none lane took an attachment: %v", args)
 	}
-	if args := New("cocoon", "br0", nil, false, "").netArgs("sbx-1", egress, false); !slices.Equal(args, []string{"--bridge", "br0"}) {
+	if args := New("cocoon", []string{"br0"}, nil, false, "").netArgs("sbx-1", egress, false); !slices.Equal(args, []string{"--bridge", "br0"}) {
 		t.Errorf("bridge lane args = %v", args)
 	}
-	if args := New("cocoon", "", []string{"cocoon-dhcp"}, false, "").netArgs("sbx-1", egress, false); !slices.Equal(args, []string{"--network", "cocoon-dhcp"}) {
+	if args := New("cocoon", nil, []string{"cocoon-dhcp"}, false, "").netArgs("sbx-1", egress, false); !slices.Equal(args, []string{"--network", "cocoon-dhcp"}) {
 		t.Errorf("single-network args = %v", args)
 	}
 }