atunnel: support IPv6 original destination lookup - #753
atunnel: support IPv6 original destination lookup#753Bingtan Lu (lubingtan) wants to merge 1 commit into
Conversation
…o ipv6-integration
|
Deep-reviewed this against current Linux master kernel sources (net/netfilter/nf_conntrack_proto.c, net/ipv4/netfilter/ip_tables.c, net/ipv6/netfilter/ip6_tables.c, net/ipv6/af_inet6.c, include/uapi/linux/netfilter_ipv6/ip6_tables.h) and ran a live veth/netns/nftables repro. The core approach is correct; a few issues below. Verified correct
FindingsMAJOR-1: Error masking on ordinary IPv4 sockets
Fix: only attempt the v6 fallback when the socket is actually v6 ( MAJOR-2: Tests hard-fail (not skip) on hosts with a default-deny input firewall
Fix: have the test install its own filter-chain accept rule for the redirect port (own table, priority filter, before ufw), matching production wiring — or probe the connect and MINOR-3: Production IPv6 path is still dead code
NIT-4: ENOENT mechanism undocumentedAdd the why: pure-v6 sockets leave NIT-5:
|
|
Two asks; the rest of the PR I would take as is. 1. Gate the IPv6 fallback on the connection's family. krsnaSuraj's MAJOR-1, taking the first of the two options they offered — diff --git a/internal/atunnel/original_dst_linux.go b/internal/atunnel/original_dst_linux.go
index 8b430911..bd8b7646 100644
--- a/internal/atunnel/original_dst_linux.go
+++ b/internal/atunnel/original_dst_linux.go
@@ -43,14 +43,23 @@ func TCPOriginalDestination(conn net.Conn) (string, error) {
return "", fmt.Errorf("atunnel: acquiring TCP syscall connection: %w", err)
}
+ // The IPv6 option is only meaningful on an AF_INET6 socket: on AF_INET the
+ // kernel returns EOPNOTSUPP, which would mask the real IPv4 error. A
+ // v4-mapped local address still means an IPv4 flow, so To4 is the test.
+ local, ok := tcpConn.LocalAddr().(*net.TCPAddr)
+ if !ok {
+ return "", fmt.Errorf("atunnel: original destination requires a TCP local address, got %T", tcpConn.LocalAddr())
+ }
+ isIPv6 := local.IP.To4() == nil
+
var sockoptErr error
var destination string
if err := rawConn.Control(func(fd uintptr) {
destination, sockoptErr = originalIPv4Destination(fd)
- // Linux returns ENOENT when the IPv4 original-destination option is
- // queried on a redirected IPv6 connection. Only then try the IPv6
- // equivalent, so unrelated IPv4 failures retain their original error.
- if errors.Is(sockoptErr, unix.ENOENT) {
+ // A pure-IPv6 socket leaves the inet addresses zeroed, so the IPv4
+ // conntrack lookup always misses with ENOENT. That is the redirected
+ // IPv6 connection, and the only case worth retrying.
+ if isIPv6 && errors.Is(sockoptErr, unix.ENOENT) {
destination, sockoptErr = originalIPv6Destination(fd)
}
}); err != nil {2. A regression test. The non-obvious part is that it has to run in a fresh netns. Conntrack tracks loopback in any namespace carrying nftables rules, including the one Docker runs in, so in the host namespace Test diffdiff --git a/internal/atunnel/original_dst_linux_test.go b/internal/atunnel/original_dst_linux_test.go
index 4f26d980..36122c5d 100644
--- a/internal/atunnel/original_dst_linux_test.go
+++ b/internal/atunnel/original_dst_linux_test.go
@@ -22,6 +22,7 @@ import (
"fmt"
"net"
"os"
+ "runtime"
"strings"
"testing"
"time"
@@ -37,6 +38,63 @@ import (
"github.com/agent-substrate/substrate/internal/roottest"
)
+// TestTCPOriginalDestinationPreservesErrno covers the failure path on an
+// ordinary connection that no REDIRECT rule touched. The IPv4 lookup misses
+// and reports ENOENT; that error must reach the caller. Retrying the IPv6
+// option on an AF_INET socket would replace it with EOPNOTSUPP, which says
+// nothing about why the lookup failed.
+//
+// It runs in a fresh namespace because conntrack tracks loopback in any
+// namespace that has nftables rules — including the one docker runs in — and a
+// tracked connection returns its real destination instead of missing.
+func TestTCPOriginalDestinationPreservesErrno(t *testing.T) {
+ roottest.Require(t, "CAP_SYS_ADMIN for a network namespace with no conntrack hooks")
+
+ ns := newTestNetNS(t)
+ runtime.LockOSThread()
+ defer runtime.UnlockOSThread()
+ host, err := netns.Get()
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer func() {
+ if err := netns.Set(host); err != nil {
+ t.Errorf("restoring host network namespace: %v", err)
+ }
+ _ = host.Close()
+ }()
+ if err := netns.Set(ns); err != nil {
+ t.Fatal(err)
+ }
+ loopback, err := netlink.LinkByName("lo")
+ if err != nil {
+ t.Fatal(err)
+ }
+ if err := netlink.LinkSetUp(loopback); err != nil {
+ t.Fatal(err)
+ }
+
+ listener, err := net.Listen("tcp4", "127.0.0.1:0")
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer listener.Close()
+ client, err := net.Dial("tcp4", listener.Addr().String())
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer client.Close()
+ server, err := listener.Accept()
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer server.Close()
+
+ if _, err := TCPOriginalDestination(server); !errors.Is(err, unix.ENOENT) {
+ t.Fatalf("want the IPv4 lookup's ENOENT, got %v", err)
+ }
+}
+
func TestTCPOriginalDestination(t *testing.T) {
roottest.Require(t, "CAP_NET_ADMIN + CAP_SYS_ADMIN for an actor-like network namespace and nftables REDIRECT rule")On MAJOR-1's severity. I probed the errnos directly:
Row 3 is your design, working as intended. Row 2 is the bug — but it needs an IPv4 connection with no conntrack entry, and the one consumer in Your tests have run in CI, just not here. This PR's workflow has been queued in Diffs are illustration, not a patch to apply verbatim. |
3 participants
Fixes #686
Solution
TCPOriginalDestinationfirst queries the existing IPv4SOL_IP/SO_ORIGINAL_DSTsocket option. Linux returnsENOENTwhen that option is queried for a redirected IPv6 connection, so only in that case it falls back toSOL_IPV6/IP6T_SO_ORIGINAL_DST(value 80 fromlinux/netfilter_ipv6/ip6_tables.h). Other IPv4 errors are returned unchanged.The IPv6 lookup decodes
RawSockaddrInet6and formats the result withnet.JoinHostPort, preserving the required bracketed address form.Tests
The new root-gated integration tests model the production egress path for both address families: an actor-like network namespace sends TCP through a veth, an nftables
PREROUTINGrule redirects it to a local listener, and the listener verifies thatTCPOriginalDestinationreturns the actor's pre-redirect target. The IPv6 test also disables DAD for the test-only veth addresses so listeners can bind deterministically.Validation
go test ./internal/atunnel(privileged IPv4 and IPv6 redirect tests)NO_COLOR= GOCACHE=/tmp/substrate-go-build-user make verify