ci: IPv6-only kind e2e plus the actor veth [DO NOT MERGE — signal only] - #1059
Closed
Yuan Gao (ygao-g) wants to merge 10 commits into
Closed
ci: IPv6-only kind e2e plus the actor veth [DO NOT MERGE — signal only]#1059Yuan Gao (ygao-g) wants to merge 10 commits into
Yuan Gao (ygao-g) wants to merge 10 commits into
Conversation
On a fresh IP_FAMILY=ipv6 cluster nothing resolves from inside a pod and no actor boots: CoreDNS inherits the node's IPv4 resolver, which a v6-only pod cannot reach. Two Corefile clauses fix it -- a hosts entry mapping kind-registry to its IPv6 address, and a forward to an IPv6 upstream, overridable with IPV6_DNS_UPSTREAM. IPv4 and dual-stack clusters are unchanged, and atenet-egress still crashloops on v6-only for an unrelated Envoy bind bug. (cherry picked from commit e336c22)
The probe that checks a pod can resolve an external name asked once, right after the CoreDNS rollout reported complete, and a brand-new cluster is not ready by then -- it failed the create on two of three fresh clusters here while the cluster was merely still settling. Retrying inside the pod does not rescue it, because a pod that fails keeps failing for the next 36 seconds while a fresh pod ten seconds later resolves on its first try, so the whole probe pod now runs up to four times. Five consecutive fresh clusters came up clean. (cherry picked from commit 533811a)
The HTTP and HTTPS ingress listeners bound 0.0.0.0 only, so on a dual-stack cluster Envoy answered on the router Service's IPv4 ClusterIP and on nothing at all for IPv6. Each primary socket now carries an additional "::" address on the same port. Ipv4Compat stays false on it: setting it would clear IPV6_V6ONLY and collide with the primary already bound to that port, and Envoy rejects the whole listener when an additional address fails to bind, taking down all ingress rather than the IPv6 half. IPv4-only clusters are unaffected -- the primary is untouched, and a host without IPv6 simply has no second socket to bind. First of three commits binding atenet's gateways dual-stack. (cherry picked from commit ed822b5)
The Envoy admin socket bound 0.0.0.0, and the atenet-router Service carried no ipFamilyPolicy -- which the API server defaults to SingleStack, one IPv4 ClusterIP and nothing else. Between them the router had no IPv6 address to answer on. The socket now binds "::" with ipv4_compat, one socket for both families, and the Service asks for PreferDualStack. ipv4_compat is load-bearing rather than incidental: dataplane.go health-checks the admin listener over http://127.0.0.1:9901/ready, so a bare "::" would report the dataplane component of /statusz unhealthy. Prefer, not Require, keeps the Service valid on a single-stack cluster, where it is a no-op; spec.ipFamilies is left alone because the primary family is immutable and the API server appends the secondary itself. (cherry picked from commit d5edac3)
The gateway's admin and :443 sockets bound 0.0.0.0, so on an IPv6-primary cluster the kubelet probed the pod on its only address and atenet-egress crashlooped -- Envoy started fine and logged "admin address: 0.0.0.0:15000" -- while an actor's CONNECT had no v6 path in. Both sockets now bind "::" with ipv4_compat, and the Service asks for PreferDualStack so a dual-stack cluster hands out an IPv6 ClusterIP to reach them on. ipv4_compat matters on the admin socket in particular: the ext-proc sidecar's drainer dials 127.0.0.1:15000, and envoydrain.go reads a refusal there as "Envoy already exited" and skips the drain silently. (cherry picked from commit d480c6f)
EnableIPv4Forwarding now also writes /proc/sys/net/ipv6/conf/all/forwarding so actor IPv6 traffic (including DNS queries) is routed between the actor veth and pod eth0 instead of being dropped by ip6_forward() on dual-stack / IPv6-only clusters. Factor the sysctl write into writeSysctlIfUnset preserving the original read-only remount/restore behavior, and add unit coverage for its fast paths. Fixes: agent-substrate#945 (cherry picked from commit 14e98ae1ff4c29c514e924f0a76a2f27f8e9d11b)
IPv6 sysctls are absent on kernels with IPv6 disabled (e.g. some containers set net.ipv6.conf.* only when IPv6 is enabled). Treat a missing path as 'nothing to enable' instead of forcing a remount and failing, matching the documented behavior. (cherry picked from commit b01a8b4b3c62614841e38fa14dfa8b0ddeb68731)
… rules Add an fd00:169:254::/126 point-to-point pair alongside the existing IPv4 addresses, with a matching ::/0 default route, and move the actor nftables table from ip to inet so a single table carries both families. Each payload match now guards on NFPROTO to stay off the other family's packets, and teardown lists the inet family too: naming the wrong family there dumps empty, takes the "already clean" path, and silently leaks the table. Assign the IPv6 addresses with IFA_F_NODAD instead of writing the accept_dad sysctl. The ateom container is unprivileged, so containerd mounts /proc/sys read-only and the write failed with EROFS, taking SetupActorNetwork and every actor start down with it on both sandbox classes. A root-gated assertion pins the flag; the existing tests run as real root, where the sysctl is writable and the bug is invisible. (cherry picked from commit 96374378f6730e050e270dca74952a4c0816f89d)
…bled An IPv4-only cluster leaves net.ipv6.conf.all.disable_ipv6=1 in the worker pod netns, which is the default on IPv4-only GKE, and netlink there rejects the veth's IPv6 address with EPERM. The assignment sits on the path of every SetupActorNetwork call site, so actor startup went from working to failing outright and the actor never left ResumeGoldenActor. Gate the IPv6 address and default route on a per-link disable_ipv6 read, leaving the interior IPv4-only on those clusters instead of failing. A root-gated test covers a netns with IPv6 disabled; reverting the gate reproduces the EPERM against it. (cherry picked from commit 209b7ddc501ab6188b98b5f32b22415bfaf60685)
Runs the full install plus the demo and networking e2e suites against a single-stack IPv6-only kind cluster, and asserts the cluster really is v6-only so a green run cannot quietly become a second IPv4 run. It stays out of the e2e-test merge gate, so it reports IPv6 status without being able to block a PR, and it runs on every PR for now so the results are visible; the TODO on the trigger records the intended ci/ipv6 label gate. ubuntu-latest has no IPv6 egress, so the job stands up tayga for NAT64 and points CoreDNS at an upstream resolver through the well-known prefix. DNS64 is scoped to a catch-all server block: synthesizing AAAA over the cluster zones destroys the v6-only ClusterIP answers and the control plane never comes up. (cherry picked from commit ef02488)
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Not for merging. Same job as #939, one layer further down the stack. #939 shows the IPv6-only e2e failing at
TestActorEgress/TestActorEgressHTTPSbecause an actor has no IPv6 stack at all; this draft adds the actor-veth work on top so we can see what the next failure actually is, rather than predicting it. The run results are the deliverable, not the branch — nothing here should land in this shape.#1065 supersedes this run with the full stack on a newer base.
Where each commit comes from. This branch is based on the #993 merge, so several are earlier revisions of what their source PR carries today:
09196821e4ca0fcf6d94c78b301bf6f7f804208ac91689fa3055db5e1ea34bc4aa8e395ae5703c4f#877, #958, #911 and #979 all ride along because GitHub will not take a base branch that lives only in a fork; each drops out as it merges.
A failure here is still the expected outcome, one layer lower. The prediction to check the run against: with the veth dual-stacked, egress should get past
network is unreachableand fail instead insideatunnel, which reads the redirected destination with the IPv4-onlySO_ORIGINAL_DST. Behind that sits a third layer,dns_lookup_family: V4_ONLYin the egress Envoy config. Locally verified on this exact stack: builds and vets clean forGOOS=linux, and the five root-gatedTestSetupActorNetwork*tests pass undersudoin a Linux VM, including the IPv4-only-netns case. The IPv6 CI job itself has never been run locally —ubuntu-latesthas no IPv6 egress so the job stands up tayga/NAT64 that a local cluster does not need, which is precisely why this needs to run in CI.Coverage and gating are unchanged from #939: 2 of 7 suites, gVisor only,
certauth only, single node, no CSI, and the job stays out of thee2e-testmerge gate so it can never block a PR. Part of #246.🤖 Generated with Claude Code