Skip to content

atenet: bind the Envoy sockets and Services dual-stack - #911

Open
Yuan Gao (ygao-g) wants to merge 2 commits into
agent-substrate:mainfrom
ygao-g:atenet-envoy-dualstack
Open

atenet: bind the Envoy sockets and Services dual-stack#911
Yuan Gao (ygao-g) wants to merge 2 commits into
agent-substrate:mainfrom
ygao-g:atenet-envoy-dualstack

Conversation

@ygao-g

@ygao-g Yuan Gao (ygao-g) commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Fixes #910
Part of #246

It's a good idea to open an issue first for discussion.

  • Tests pass
  • Appropriate changes to documentation are included in the PR (none needed — no flags, APIs, or user-facing behaviour change)

The router and egress manifests bind the IPv4 wildcard on every Envoy socket, and neither gateway's Service asks for a second IP family. On a dual-stack cluster the router answers on its IPv4 ClusterIP and on nothing at all for IPv6; on an IPv6-primary cluster the kubelet cannot probe the pod on its only address, so atenet-egress crashloops while Envoy itself starts fine.

Both gateways now bind :: as well, and both Services ask for PreferDualStack. A new hack/verify check guards the ipv4_compat flag that the in-pod loopback callers depend on, since nothing else in the tree reads these manifests. The experimental atenet-egress-with-sdsmint.yaml variant is deliberately out of scope.

This makes both gateways accept IPv6; it does not make them reach IPv6 destinations. The egress dns_lookup_family, the DNS AAAA path, and atunnel's original-destination lookup stay IPv4, and are tracked in #246 and #686.

Testing

CI has no IP-family matrix, so the green e2e run here is the IPv4 regression check and nothing more. The other two families were verified by hand on kind, using IP_FAMILY from #877.

  • Dual-stack. The networking suite passes with zero skips: an actor is reached over each of the router's ClusterIPs, and an IPv4 actor transits the egress [::]:443 socket. Reverting only the two bind lines turns those assertions red, so they are not passing vacuously. ipv4_compat was read from /config_dump/listeners never emits it.
  • IPv6-only, with the kind CoreDNS fix from hack: fix DNS on IPv6-only kind clusters #958 underneath. atenet-egress goes from 1/2 CrashLoopBackOff (64 restarts, startup probe refused) to 2/2 Running, and TestActorDirectAccess passes, including through the router's ingress. TestActorEgress still fails inside the actor's netns, which is ateomnet: enable IPv6 forwarding in worker pod netns #979's scope.

🤖 Generated with Claude Code

@ygao-g

Copy link
Copy Markdown
Collaborator Author

Bowei Du (@bowei) mind taking a look?

Address: &corev3.Address_SocketAddress{
SocketAddress: &corev3.SocketAddress{
Address: "::",
Ipv4Compat: false,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to sync up the configuration in the YAML with the configuration here?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, the comment here might have confused me. This is talking about two different sockets? If it is, can you remove the comment above.

@ygao-g Yuan Gao (ygao-g) Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, two different sockets. And I have removed the confusing comments.

This is the router's ingress data-plane listener; the ipv4_compat: true you saw in the YAML is the Envoy admin socket on 9901. I've cut the comment down to the one line explaining why Ipv4Compat is false here — the cross-reference to the admin socket was the confusing part and it's gone.

On the original question: the admin sockets can't take this shape anyway, since bootstrap.v3.Admin has a single address field and no additional_addresses — one :: socket with ipv4_compat is the only way to serve both families there. The egress :443 listener is a real Listener, so that one could match the ingress shape if you'd rather have a single idiom across listeners. Happy to do it; it's a few lines plus the test that asserts the current shape.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to switch if you'd rather have it — but I think the two shapes are equivalent here, so it's a style call rather than a correctness one.

The reason to keep an IPv4 primary is to stop v4 peers arriving as ::ffff:-mapped addresses. Nothing on either path reads the downstream peer: ingress ext_proc requests one attribute, the authority filter-state (xds.go:1016), and egress requests one, xds.filter_chain_name (atenet-egress.yaml:134-135). There's no use_remote_address, no XFF config and no DOWNSTREAM_REMOTE_ADDRESS in cmd/atenet or either manifest, and the original_dst on ingress is the destination ext_proc supplies in dynamic metadata, not the source.

I kept egress as one socket to match the admin listener in the same file, which is :: + ipv4_compat: true and where the flag is load-bearing — the drain sequence dials 127.0.0.1:15000 (atenet-egress.yaml:310), as the comment at line 40 already notes. The symmetric move doesn't exist: ingress can't take ipv4_compat while keeping its 0.0.0.0 primary, because both would bind IPv4:port — that's what "collide with that primary" means at xds.go:1113.

So it's one idiom within the file against one across the component, and I don't have a strong preference. Say which you'd rather see and I'll match it — a few lines plus the test.

@ygao-g
Yuan Gao (ygao-g) force-pushed the atenet-envoy-dualstack branch 2 times, most recently from 5257d6b to 580afef Compare August 18, 2026 14:59
@ygao-g Yuan Gao (ygao-g) changed the title atenet: bind the Envoy listeners, admin sockets, and Services dual-stack atenet: bind the Envoy sockets and Services dual-stack Aug 18, 2026
@ygao-g Yuan Gao (ygao-g) added kind/bug Something isn't working / bugfixes area/network labels Aug 18, 2026
@ygao-g
Yuan Gao (ygao-g) force-pushed the atenet-envoy-dualstack branch 3 times, most recently from 378c535 to 5c35bdc Compare August 19, 2026 03:40
@ygao-g
Yuan Gao (ygao-g) force-pushed the atenet-envoy-dualstack branch 3 times, most recently from 24055c6 to 6c1910d Compare August 19, 2026 22:58
Comment thread cmd/atenet/internal/router/dataplane.go Outdated
Comment thread cmd/atenet/internal/router/xds.go Outdated
Comment thread manifests/ate-install/atenet-egress.yaml Outdated
The router and egress manifests bind the IPv4 wildcard on every Envoy
socket, and neither gateway's Service asks for a second IP family. On a
dual-stack cluster the router answers on its IPv4 ClusterIP and on
nothing at all for IPv6; on an IPv6-primary cluster the kubelet cannot
probe the pod on its only address, so atenet-egress crashloops while
Envoy itself starts fine.

Both gateways now bind `::` as well and ask for PreferDualStack. That
makes them accept IPv6, not reach it: the egress dns_lookup_family, the
DNS AAAA path, and atunnel's original-destination lookup stay IPv4. The
experimental sdsmint egress variant is untouched.
Dropping ipv4_compat from a gateway's Envoy admin socket fails silently:
the drain sequence reads the refused IPv4 loopback dial as "Envoy
already exited" and reports a drain it never performed. No Go test reads
these manifests.

Each admin block is bounded by the first line that dedents back to its
own key and judged on its own, with comments stripped, so that neither a
block that outgrows a fixed context window nor a second compliant block
elsewhere in the file can mask a regression.
Yuan Gao (ygao-g) added a commit to ygao-g/substrate that referenced this pull request Aug 20, 2026
Nothing checked that the router's dataplane listeners bind more than an
IPv4 socket, and nothing reached an actor over the router's IPv6
ClusterIP. Every other path a test has into the router -- a
port-forward, the pods/proxy and services/proxy subresources -- is
mediated by the API server, which picks the family, so no existing test
could have caught a listener that lost its IPv6 socket. Reads the bound
addresses from Envoy's own admin /listeners, and drives an in-cluster
probe pod at the router over each ClusterIP in turn.

Red until agent-substrate#911 binds those sockets, so this stays a draft until then.
The per-family probe skips on a single-stack cluster. Part of agent-substrate#246.
Yuan Gao (ygao-g) added a commit to ygao-g/substrate that referenced this pull request Aug 20, 2026
Nothing checked that the router's dataplane listeners bind more than an
IPv4 socket, and nothing reached an actor over the router's IPv6
ClusterIP. Every other path a test has into the router -- a
port-forward, the pods/proxy and services/proxy subresources -- is
mediated by the API server, which picks the family, so no existing test
could have caught a listener that lost its IPv6 socket. Reads the bound
addresses from Envoy's own admin /listeners, and drives an in-cluster
probe pod at the router over each ClusterIP in turn.

Red until agent-substrate#911 binds those sockets, so this stays a draft until then.
The per-family probe skips on a single-stack cluster. Part of agent-substrate#246.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/network kind/bug Something isn't working / bugfixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

atenet Envoy sockets and Services lack IPv6 support (hardcoded to 0.0.0.0)

2 participants