atenet: bind the Envoy sockets and Services dual-stack - #911
atenet: bind the Envoy sockets and Services dual-stack#911Yuan Gao (ygao-g) wants to merge 2 commits into
Conversation
|
Bowei Du (@bowei) mind taking a look? |
5308c09 to
3162d63
Compare
| Address: &corev3.Address_SocketAddress{ | ||
| SocketAddress: &corev3.SocketAddress{ | ||
| Address: "::", | ||
| Ipv4Compat: false, |
There was a problem hiding this comment.
Do we want to sync up the configuration in the YAML with the configuration here?
There was a problem hiding this comment.
Actually, the comment here might have confused me. This is talking about two different sockets? If it is, can you remove the comment above.
There was a problem hiding this comment.
Yes, two different sockets. And I have removed the confusing comments.
This is the router's ingress data-plane listener; the ipv4_compat: true you saw in the YAML is the Envoy admin socket on 9901. I've cut the comment down to the one line explaining why Ipv4Compat is false here — the cross-reference to the admin socket was the confusing part and it's gone.
On the original question: the admin sockets can't take this shape anyway, since bootstrap.v3.Admin has a single address field and no additional_addresses — one :: socket with ipv4_compat is the only way to serve both families there. The egress :443 listener is a real Listener, so that one could match the ingress shape if you'd rather have a single idiom across listeners. Happy to do it; it's a few lines plus the test that asserts the current shape.
There was a problem hiding this comment.
Happy to switch if you'd rather have it — but I think the two shapes are equivalent here, so it's a style call rather than a correctness one.
The reason to keep an IPv4 primary is to stop v4 peers arriving as ::ffff:-mapped addresses. Nothing on either path reads the downstream peer: ingress ext_proc requests one attribute, the authority filter-state (xds.go:1016), and egress requests one, xds.filter_chain_name (atenet-egress.yaml:134-135). There's no use_remote_address, no XFF config and no DOWNSTREAM_REMOTE_ADDRESS in cmd/atenet or either manifest, and the original_dst on ingress is the destination ext_proc supplies in dynamic metadata, not the source.
I kept egress as one socket to match the admin listener in the same file, which is :: + ipv4_compat: true and where the flag is load-bearing — the drain sequence dials 127.0.0.1:15000 (atenet-egress.yaml:310), as the comment at line 40 already notes. The symmetric move doesn't exist: ingress can't take ipv4_compat while keeping its 0.0.0.0 primary, because both would bind IPv4:port — that's what "collide with that primary" means at xds.go:1113.
So it's one idiom within the file against one across the component, and I don't have a strong preference. Say which you'd rather see and I'll match it — a few lines plus the test.
5257d6b to
580afef
Compare
378c535 to
5c35bdc
Compare
24055c6 to
6c1910d
Compare
6c1910d to
cff4f76
Compare
be492e5 to
e48bdf3
Compare
The router and egress manifests bind the IPv4 wildcard on every Envoy socket, and neither gateway's Service asks for a second IP family. On a dual-stack cluster the router answers on its IPv4 ClusterIP and on nothing at all for IPv6; on an IPv6-primary cluster the kubelet cannot probe the pod on its only address, so atenet-egress crashloops while Envoy itself starts fine. Both gateways now bind `::` as well and ask for PreferDualStack. That makes them accept IPv6, not reach it: the egress dns_lookup_family, the DNS AAAA path, and atunnel's original-destination lookup stay IPv4. The experimental sdsmint egress variant is untouched.
Dropping ipv4_compat from a gateway's Envoy admin socket fails silently: the drain sequence reads the refused IPv4 loopback dial as "Envoy already exited" and reports a drain it never performed. No Go test reads these manifests. Each admin block is bounded by the first line that dedents back to its own key and judged on its own, with comments stripped, so that neither a block that outgrows a fixed context window nor a second compliant block elsewhere in the file can mask a regression.
e48bdf3 to
cf406b1
Compare
Nothing checked that the router's dataplane listeners bind more than an IPv4 socket, and nothing reached an actor over the router's IPv6 ClusterIP. Every other path a test has into the router -- a port-forward, the pods/proxy and services/proxy subresources -- is mediated by the API server, which picks the family, so no existing test could have caught a listener that lost its IPv6 socket. Reads the bound addresses from Envoy's own admin /listeners, and drives an in-cluster probe pod at the router over each ClusterIP in turn. Red until agent-substrate#911 binds those sockets, so this stays a draft until then. The per-family probe skips on a single-stack cluster. Part of agent-substrate#246.
Nothing checked that the router's dataplane listeners bind more than an IPv4 socket, and nothing reached an actor over the router's IPv6 ClusterIP. Every other path a test has into the router -- a port-forward, the pods/proxy and services/proxy subresources -- is mediated by the API server, which picks the family, so no existing test could have caught a listener that lost its IPv6 socket. Reads the bound addresses from Envoy's own admin /listeners, and drives an in-cluster probe pod at the router over each ClusterIP in turn. Red until agent-substrate#911 binds those sockets, so this stays a draft until then. The per-family probe skips on a single-stack cluster. Part of agent-substrate#246.
Fixes #910
Part of #246
The router and egress manifests bind the IPv4 wildcard on every Envoy socket, and neither gateway's Service asks for a second IP family. On a dual-stack cluster the router answers on its IPv4 ClusterIP and on nothing at all for IPv6; on an IPv6-primary cluster the kubelet cannot probe the pod on its only address, so
atenet-egresscrashloops while Envoy itself starts fine.Both gateways now bind
::as well, and both Services ask forPreferDualStack. A newhack/verifycheck guards theipv4_compatflag that the in-pod loopback callers depend on, since nothing else in the tree reads these manifests. The experimentalatenet-egress-with-sdsmint.yamlvariant is deliberately out of scope.This makes both gateways accept IPv6; it does not make them reach IPv6 destinations. The egress
dns_lookup_family, the DNS AAAA path, and atunnel's original-destination lookup stay IPv4, and are tracked in #246 and #686.Testing
CI has no IP-family matrix, so the green e2e run here is the IPv4 regression check and nothing more. The other two families were verified by hand on kind, using
IP_FAMILYfrom #877.[::]:443socket. Reverting only the two bind lines turns those assertions red, so they are not passing vacuously.ipv4_compatwas read from/config_dump—/listenersnever emits it.atenet-egressgoes from1/2 CrashLoopBackOff(64 restarts, startup probe refused) to2/2 Running, andTestActorDirectAccesspasses, including through the router's ingress.TestActorEgressstill fails inside the actor's netns, which is ateomnet: enable IPv6 forwarding in worker pod netns #979's scope.🤖 Generated with Claude Code