Skip to content

Carry a VPC pod to the edge only when served - #445

Open
scotwells wants to merge 1 commit into
feat/edge-networkservice-behaviorfrom
feat/edgereachability-e2e
Open

Carry a VPC pod to the edge only when served#445
scotwells wants to merge 1 commit into
feat/edge-networkservice-behaviorfrom
feat/edgereachability-e2e

Conversation

@scotwells

Copy link
Copy Markdown
Contributor

Every VPC pod's endpoint slice is federated to every edge cluster, so a cell publishes one tenant's pod addresses and segment identifiers onto edges that serve other tenants and other locations, and nothing filters them because the facts that decide whether an edge needs a pod live in a project control plane a cell cannot read. This adds a hub-only record, written per project namespace by the control plane, of the workload addresses behind a proxy, and the cell's write-back carries a slice only when that record names one of its addresses. Silence keeps a pod published on purpose, since a route withdrawn under a pod that is still serving black-holes live traffic while a route left up for an idle pod costs a table entry. This is the last of six stacked pull requests splitting the network services prototype, and it adds the end-to-end scenario that proves a member serves a real request through the edge on its allocated address.

Test plan

  • Unit tests cover resolving the record from proxies, services, and the interfaces they select
  • Unit tests cover the write-back carrying a slice only when named, withdrawing one that stops serving, and holding on silence
  • Build, vet, and the controller tests pass under envtest
  • End-to-end scenario passes against the two-cluster environment in CI

Every VPC pod's EndpointSlice is federated to every edge cluster, so a
cell publishes one tenant's pod addresses and SRv6 SIDs onto edges that
serve other tenants and other locations. Nothing filters them, because
the two facts that decide whether an edge needs a pod, the
NetworkService selecting it and the HTTPProxy naming that service, live
in a project control plane a cell cannot read.

EdgeReachability records the answer where both planes meet. The control
plane resolves, per project namespace, the workload addresses behind a
proxy and writes them to the federation hub. The write-back reads the
record from the hub it already publishes into and carries a slice only
when the record names one of its addresses. Silence keeps a pod
published, since a withdrawn route under a serving pod black-holes live
traffic while a route left up costs a table entry.

Key changes:
- Add EdgeReachability, a hub-only record of a project's served addresses
- Add a controller resolving proxies, services and the interfaces they
  select in each project control plane
- Filter the VPC EndpointSlice write-back on the record and resync every
  minute, since nothing in a cell watches the hub
- Add the end-to-end scenario proving a member serves through the edge
@ecv

ecv commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Read through the reachability record and the gating it drives. The withhold-but-never-black-hole default and the resync ticker on the write-back side both look right, and the envtest cases around them are convincing.

One thing I want to ask before approving. publishedAddresses reads an EndpointSlice for every Instance backend a proxy names, but the controller watches only HTTPProxy, NetworkService and NetworkInterface, and it neither requeues nor runs a periodic pass. If an Instance backend's slice changes addresses with no event on those three types, is there anything that recomputes the record? From here it looks like it stays stale until the next unrelated proxy or interface change, and the write-back then gates VPC pod copies on an address set that no longer matches the workload.

Is the missing EndpointSlice watch deliberate, on the grounds that a slice change always comes with an interface change in practice? If it is, a comment saying so would help; if not, a watch or a resync mirroring vpcEndpointSliceResyncInterval would close it.

Smaller and not blocking: the record carries every family interfaceBackhaulAddresses returns, while the proxy path programs only the family addressOfType picks, so a dual-stack member advertises an address nothing routes to. Over-inclusive rather than wrong, but it costs some of the precision the record exists to give.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants