ci: run the IPv6-only e2e job against the full IPv6 stack [DO NOT MERGE — signal only] - #1065
Draft
Yuan Gao (ygao-g) wants to merge 14 commits into
Draft
ci: run the IPv6-only e2e job against the full IPv6 stack [DO NOT MERGE — signal only]#1065Yuan Gao (ygao-g) wants to merge 14 commits into
Yuan Gao (ygao-g) wants to merge 14 commits into
Conversation
On a fresh IP_FAMILY=ipv6 cluster nothing resolves from inside a pod and no actor boots: CoreDNS inherits the node's IPv4 resolver, which a v6-only pod cannot reach, and "kind-registry" NXDOMAINs in atelet's own netns. Point the forward at an IPv6 upstream, overridable with IPV6_DNS_UPSTREAM, and give the registry its own server block, so it is asked for nothing but its own name. IPv4 and dual-stack clusters are unchanged, and atenet-egress still crashloops on v6-only for an unrelated Envoy bind bug. Asking once was not enough to prove that: about half of fresh clusters do not answer the first query, and a pod that goes unanswered stays unanswered, so the check re-asks with a new pod and prints what the pod saw when it gives up. It lives in hack/verify-ipv6-dns.sh rather than inline, because the registry block records an address the registry can move off and there was no way to re-check a cluster without rebuilding it.
The HTTP and HTTPS ingress listeners bound 0.0.0.0 only, so on a dual-stack cluster Envoy answered on the router Service's IPv4 ClusterIP and on nothing at all for IPv6. Each primary socket now carries an additional "::" address on the same port. Ipv4Compat stays false on the additional address: clearing IPV6_V6ONLY would collide with the primary already bound to that port. Leaving the primary alone is what keeps an IPv4-only cluster unchanged -- with the caveat that a node lacking AF_INET6 entirely could not bind "::" and the listener would not come up. First of three commits binding atenet's gateways dual-stack. (cherry picked from commit 501991d285a1efeb9a146a029136e2eb140001f4)
The Envoy admin socket bound 0.0.0.0, and the atenet-router Service carried no ipFamilyPolicy -- which the API server defaults to SingleStack, one IPv4 ClusterIP and nothing else. Between them the router had no IPv6 address to answer on. The socket now binds "::" with ipv4_compat, one socket for both families, and the Service asks for PreferDualStack. bootstrap.v3.Admin takes a single address and has no additional_addresses, so the ingress listeners' shape is not available here; ipv4_compat is what makes the one socket serve both families. It is load-bearing: dataplane.go health-checks the admin listener over http://127.0.0.1:9901/ready, so a bare "::" would report the dataplane component of /statusz unhealthy. Prefer, not Require, keeps the Service valid on a single-stack cluster; spec.ipFamilies is left alone because the primary family is immutable and the API server appends the secondary itself. (cherry picked from commit 2a21292a3262d4a642c723550881d253737e410d)
The gateway's admin and :443 sockets bound 0.0.0.0, so on an IPv6-primary cluster the kubelet probed the pod on its only address and atenet-egress crashlooped -- Envoy started fine and logged "admin address: 0.0.0.0:15000" -- while an actor's CONNECT had no v6 path in. Both sockets now bind "::" with ipv4_compat, and the Service asks for PreferDualStack so a dual-stack cluster hands out an IPv6 ClusterIP to reach them on. One socket here rather than the ingress listeners' pair: IPv4 peers then arrive as ::ffff: addresses, and nothing on this path reads the peer -- actor identity comes from the client certificate and the access log records the cert SAN. ipv4_compat also has to stay on the admin socket, because the ext-proc sidecar's drainer dials 127.0.0.1:15000 and envoydrain.go reads a refusal there as "Envoy already exited", skipping the drain silently. Last of three. (cherry picked from commit 2549657bc13650207b28ae49a80b7e2e5e790c5e)
Both gateway admin sockets bind "::" with ipv4_compat, and the flag is what keeps their in-pod callers working: dataplane.go health-checks the router's over IPv4 loopback, and envoydrain.go dials the egress one the same way and reads a refusal as "Envoy already exited", skipping the drain without reporting an error. No Go test, golden file, or verify script read either manifest, so dropping the flag would have failed silently. make verify now rejects an admin socket that binds "::" without it. (cherry picked from commit 4b478a0)
The CONNECT-terminating listeners landed after the first commit of this series, so they kept a bare 0.0.0.0 socket while ingress HTTP and HTTPS gained their "::" pair. Give them the same additional address, so all four of the router's socket listeners answer on both families. Both are port-gated and no e2e suite configures them yet, which is why nothing caught this; the internal main_internal listener has no socket and needs nothing. (cherry picked from commit 54c727d22843556dfcbd06a4910087efe2114ba2)
TCPOriginalDestination read only the IPv4 SOL_IP/SO_ORIGINAL_DST, so an actor's IPv6 connection redirected into the transparent egress listener had no destination to dial and the proxy failed it. Read IP6T_SO_ORIGINAL_DST too, falling back to it only when the IPv4 lookup returns ENOENT, so unrelated IPv4 failures keep their own error. One step towards dual-stack actor networking; the actor veth and its nftables rules are still IPv4-only. Co-authored-by: Yuan Gao <ypgao@google.com> (cherry picked from commit d8527b5e8ea24e588f694da726ee76784d11c41b)
Both ateom herders defaulted the actor ingress flags to "0.0.0.0:443" and "0.0.0.0:444", which reads as IPv4-only. It never was: Go treats an unspecified address as a wildcard and binds it dual-stack, so the sockets already served both families. Spell the defaults ":443" and ":444" so the flag says what it does, and note why in a comment. Part of the dual-stack actor networking series; no behavior change. (cherry picked from commit 51b2cbef37e1e2c6699fb7d8eee64e968f222be9)
The actor veth and pod eth0 sit in the same netns, and only net.ipv4.ip_forward was enabled there, so ip6_forward() dropped every actor IPv6 packet, DNS queries included, on a dual-stack or IPv6-only cluster. Write net.ipv6.conf.all.forwarding as well, treating a missing path as nothing to enable so a netns with IPv6 compiled out still comes up. Part of agent-substrate#945; the actor veth itself is still IPv4-only, so nothing generates that traffic yet. Co-authored-by: Yuan Gao <ypgao@google.com> (cherry picked from commit 29fe4bc)
Yuan Gao (ygao-g)
force-pushed
the
ipv6-ci-integration
branch
from
August 19, 2026 18:34
95624a2 to
bbc44ca
Compare
Yuan Gao (ygao-g)
force-pushed
the
ipv6-ci-integration
branch
from
August 20, 2026 15:50
db4701d to
efaf5ee
Compare
The egress Envoy pinned dns_lookup_family to V4_ONLY, so it asked only for A records. On an IPv6-only cluster no upstream name resolves and no actor can reach the internet. AUTO tries AAAA and falls back to A, so IPv4-only clusters behave as before. One step of the IPv6 egress work, and not the one that unblocks it -- actor egress still stops earlier, in atunnel's original-destination lookup. (cherry picked from commit de81578)
Runs the full install plus the demo and networking e2e suites against a single-stack IPv6-only kind cluster, and asserts the cluster really is v6-only so a green run cannot quietly become a second IPv4 run. It stays out of the e2e-test merge gate, so it reports IPv6 status without being able to block a PR, and it runs on every PR for now so the results are visible; the TODO on the trigger records the intended ci/ipv6 label gate. ubuntu-latest has no IPv6 egress, so the job stands up tayga for NAT64 and points CoreDNS at an upstream resolver through the well-known prefix. DNS64 is scoped to a catch-all server block: synthesizing AAAA over the cluster zones destroys the v6-only ClusterIP answers and the control plane never comes up. (cherry picked from commit 748e841)
The probe attached to the pod to collect its markers, and an attach can end before the last write arrives. A CI run lost the registry marker that way, so the check reported a registry it could not reach -- and then refused to re-probe, because only the resolve leg was treated as a settling race. Wait for the pod to terminate and read its log instead, and close the probe with a PROBE_DONE marker so a short read is re-probed rather than read as a failed fetch. A registry that really is down still fails on the first attempt.
The actor's NAT and filter rules lived in an ip table, which can only ever carry IPv4. They are now in an inet table, so one table can hold both address families when the actor veth becomes dual-stack. A bare payload match is ambiguous in an inet table, so every match now opens with an NFPROTO comparison and behaves exactly as it did before. NAT in the inet family needs Linux 4.18 or later. Teardown sweeps ip as well as inet. A table name is unique per family, so the ip table an earlier ateom left behind is invisible to an inet-only cleanup: the dump comes back empty, the "already clean" path reports success, and the stale table keeps redirecting alongside the new one. Part of agent-substrate#246
Actor networking was IPv4-only, so an actor on a dual-stack worker pod could not reach an IPv6-only destination at all. SetupActorNetwork now assigns the fd00:169:254::/126 counterparts of the existing point-to-point pair to both ends of the actor veth, installs an IPv6 default route in the interior netns, and adds the matching rules to the inet-family actor table. Whether the actor gets IPv6 is decided once in the worker pod netns and carried into the interior one, which is created fresh and so always reports IPv6 available whatever the cluster's families are. Both halves have to hold: the pod needs a global IPv6 address of its own, and the veth has to accept an IPv6 address -- IPv4-only GKE sets disable_ipv6 and netlink then rejects the assignment with EPERM. Addresses carry IFA_F_NODAD rather than the accept_dad sysctl, which the unprivileged ateom container cannot write. Part of agent-substrate#246
Yuan Gao (ygao-g)
force-pushed
the
ipv6-ci-integration
branch
from
August 21, 2026 01:00
efaf5ee to
4da3798
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #246.
The IPv6-only kind e2e job gets a cluster up and passes the demo suite on
main, but TestActorEgress and TestActorEgressHTTPS fail. This branch stacks
the fourteen commits that close that gap and runs the job against them, so
the result is visible before any of them merge.
Where each commit comes from, in branch order:
5a9f010274ad449f9f6bcdf1908499f8ac9ee2dcf047e7315ad28ad01edbafcc62183e7b022e1080516483f13f07c43a555a4d344da37980Every row is that PR's commit unchanged except one: SURAJ KUMAR (@krsnaSuraj)'s forwarding
sysctl from #979 is reworked, so EnableIPv4Forwarding becomes EnableForwarding
and both sysctls are written through one loop. Bingtan Lu (@lubingtan)'s #753 is verbatim.
Both keep their original author, and neither changes behavior or tests from the
PR it came from.
The last two rows replace the three #1057 commits this branch used to carry.
#1057 has been restructured into #1116, which moves the actor nftables table
from the
ipfamily toinetwith no behaviour change on IPv4, plus a singlecommit for the IPv6 address and the family gate. That gate is what keeps the
IPv4
e2e-testgreen — without it the actor gets an IPv6 address on a clusterwith no route for it and the two egress tests fail. The restructure also brings
a teardown fix this branch was missing: a table name is unique per family, so
an inet-only cleanup leaves a stale
iptable redirecting alongside the newone.
Signal only; it will never be merged. The last fully green run was
efaf5eea,before the restructure — this tip is a fresh run. What is left to file is the
takeover of #753 and #979.
🤖 Generated with Claude Code