design-proposal: tenant external network connectivity via a gateway VM (IPsec/WireGuard)#30
design-proposal: tenant external network connectivity via a gateway VM (IPsec/WireGuard)#30myasnikovdaniil wants to merge 2 commits into
Conversation
Adds a design proposal for tenant-managed site-to-site connectivity (IPsec or WireGuard) terminated in a KubeVirt VM that NAT-bridges an external site to the tenant's managed-app ClusterIPs, both directions, without granting tenants privileged host-cluster pods. Includes the end-to-end prototype validation results and the relationship to the ClusterMesh/Kilo proposal (#7). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughAdds a design proposal for tenant-managed external network connectivity via a gateway VM, covering traffic flow design, deployment semantics, security boundaries, validation findings, rollout, and alternatives. ChangesTenant Site Gateway Design Proposal
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a design proposal for tenant-managed external network connectivity via a gateway VM (IPsec / WireGuard) in Cozystack, which terminates VPN tunnels inside a KubeVirt VM to isolate privileged networking. The review feedback highlights several key technical refinements for the proposal: ensuring the guest OS is configured to own virtual service-exposure addresses for inbound traffic, addressing potential port collisions for outbound traffic to multiple remote targets by mapping them to unique targetPorts, providing concrete KubeVirt API examples for the block size override, and noting that standard VRRP for high availability may be blocked by CNI-level IP/MAC spoofing prevention, requiring alternative failover mechanisms.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
|
|
||
| ### Inbound path (external site → any tenant app) | ||
|
|
||
| The remote peer targets a virtual "service-exposure" address that the gateway owns and maps to an app `ClusterIP`. |
There was a problem hiding this comment.
For the inbound path, the remote peer targets a virtual "service-exposure" address. To ensure the gateway VM can receive and DNAT this traffic, the guest OS needs to configure these virtual IPs (e.g., as alias IPs or on a dummy interface) so that the guest kernel accepts packets destined for them.
| The remote peer targets a virtual "service-exposure" address that the gateway owns and maps to an app `ClusterIP`. | |
| The remote peer targets a virtual "service-exposure" address that the gateway VM owns (configured as an alias or dummy interface IP inside the guest OS via cloud-init) and maps to an app ClusterIP. |
There was a problem hiding this comment.
This isn't needed for the validated design: the tunnel is route-based (VTI), so the decrypted packet arrives on the tunnel interface and is DNAT'd in PREROUTING — which runs before the kernel's local-delivery check — so the gateway need not own the virtual service-exposure address on an interface. The dev-cluster prototype confirmed this: nat destination rule inbound-interface vti0 destination <vip> → <ClusterIP> worked with no alias/dummy IP assigned. Keeping the text as-is to avoid documenting a requirement that doesn't apply to the route-based path.
| 1. The app connects to a local Service name the chart creates (e.g. `remote-db.<tenant>.svc:port`) — the only app-side change is the target hostname. | ||
| 2. The node routes the `ClusterIP` to the gateway VM. | ||
| 3. **DNAT**: the local listener → the real remote `IP:port` behind the tunnel; the VM encrypts and sends it over the tunnel; SNAT into the tunnel's inner subnet lets replies return. |
There was a problem hiding this comment.
When using a local ClusterIP Service to route outbound traffic to the gateway VM, the CNI/kube-proxy on the client node will perform DNAT, translating the Service's ClusterIP to the gateway VM's pod IP before the packet reaches the VM.\n\nIf a tenant has multiple remote targets that use the same destination port (e.g., two different remote databases on port 5432), the gateway VM will receive packets for both targets with the same destination IP (the VM's pod IP) and destination port (5432). It will have no way to distinguish which remote target was originally intended.\n\nTo resolve this, the design should explicitly specify that each remote target must map to a unique targetPort (local listener port) on the gateway VM. The local Services can still expose the standard port (e.g., 5432), but they must map to these unique targetPorts on the gateway VM (e.g., 5432 and 5433), which the gateway VM then DNATs to the respective remote IP:port.
| 1. The app connects to a local Service name the chart creates (e.g. `remote-db.<tenant>.svc:port`) — the only app-side change is the target hostname. | |
| 2. The node routes the `ClusterIP` to the gateway VM. | |
| 3. **DNAT**: the local listener → the real remote `IP:port` behind the tunnel; the VM encrypts and sends it over the tunnel; SNAT into the tunnel's inner subnet lets replies return. | |
| 1. The app connects to a local Service name the chart creates (e.g. remote-db.<tenant>.svc:port) — the only app-side change is the target hostname.\n2. The node translates the ClusterIP to the gateway VM's pod IP and a unique targetPort (local listener port) allocated on the VM. This unique port is required so the VM can distinguish between multiple remote targets that use the same destination port.\n3. **DNAT**: the unique local listener port on the VM -> the real remote IP:port behind the tunnel; the VM encrypts and sends it over the tunnel; SNAT into the tunnel's inner subnet lets replies return. |
There was a problem hiding this comment.
Good catch — addressed in 72db47e. The outbound path now states each remote target maps to a per-target-unique listener port on the gateway (the local Service still exposes the standard port, e.g. 5432), so the VM can disambiguate multiple targets sharing a destination port. The values schema's remoteTargets[].localPort already carries this.
| - **Native ESP dropped by the CNI overlay (IPsec).** On Cilium/kube-ovn, native ESP (IP proto 50) does not traverse the overlay even pod-to-pod; ESP-in-UDP (forced UDP encapsulation) is required unconditionally (validated — see Testing). WireGuard, being UDP-native, is unaffected. | ||
| - **DNAT must target stable `ClusterIP`s**, never ephemeral pod IPs. | ||
| - **Source IP is lost inbound** (SNAT) — only relevant for apps with source-IP ACLs. | ||
| - **VM image must be bootable and 512-native.** DRBD-backed (4K-sector) StorageClasses cannot boot a 512-native GPT image; the disk must sit on a 512-native StorageClass or use a KubeVirt `blockSize` override. The image must ship a real bootloader (validated the hard way — see Testing). |
There was a problem hiding this comment.
To make the KubeVirt blockSize override actionable and clear for the implementer, it would be beneficial to include a brief example of the exact KubeVirt API fields used to configure it.
| - **VM image must be bootable and 512-native.** DRBD-backed (4K-sector) StorageClasses cannot boot a 512-native GPT image; the disk must sit on a 512-native StorageClass or use a KubeVirt `blockSize` override. The image must ship a real bootloader (validated the hard way — see Testing). | |
| - **VM image must be bootable and 512-native.** DRBD-backed (4K-sector) StorageClasses cannot boot a 512-native GPT image; the disk must sit on a 512-native StorageClass or use a KubeVirt blockSize override (e.g., setting blockSize.custom.logical to 512 and blockSize.custom.physical to 4096 in the disk spec). The image must ship a real bootloader (validated the hard way — see Testing). |
There was a problem hiding this comment.
Done in 72db47e — added the concrete example (blockSize.custom.logical: 512 / physical: 4096 on the disk spec).
|
|
||
| - **Secret handling.** Best shape for mounting PSK/cert/WireGuard-key material into the guest without persisting it in plaintext cloud-init at rest. | ||
| - **VIP allocation.** One LoadBalancer VIP per gateway vs. sharing; interaction with the tenant's LB address pool and quotas. | ||
| - **HA.** Is active/standby (e.g. VRRP + shared VIP) in scope for a later iteration? A single VM is a per-tenant SPOF. |
There was a problem hiding this comment.
In standard Kubernetes CNIs (including Cilium and kube-ovn), IP/MAC spoofing prevention is typically enabled by default. This means standard VRRP (keepalived) running inside guest VMs on the default pod network may fail because the CNI will block gratuitous ARPs or refuse to route the shared VIP to the standby VM.\n\nTo support HA, the platform might need to integrate with CNI-specific virtual IP features or use a Kubernetes Service/EndpointSlice-based failover mechanism instead of pure guest-level VRRP.
| - **HA.** Is active/standby (e.g. VRRP + shared VIP) in scope for a later iteration? A single VM is a per-tenant SPOF. | |
| - **HA.** Is active/standby in scope for a later iteration? Note that standard VRRP (e.g., keepalived with a shared VIP) inside guest VMs on the default pod network may fail due to CNI-level IP/MAC spoofing prevention (which blocks gratuitous ARPs and unauthorized IPs). The HA design will likely need to integrate with CNI-specific virtual IP features or use a Kubernetes Service/EndpointSlice-based failover mechanism. |
There was a problem hiding this comment.
Agreed that guest-level VRRP with a shared VIP is problematic — the new High Availability section covers it (72db47e). One correction from validation: the blocker isn't IP/MAC anti-spoofing (that's solvable with a scoped kube-ovn allowed-address-pair — validated: the VIP moves through OVN with port-security kept on). The real blocker is that VRRP advertisements (IP proto 112) are dropped pod-to-pod by Cilium's conntrack (non-TCP/UDP/ICMP/SCTP; see Cilium CFP #39601, closed not-planned), so keepalived can't elect over the pod network. HA therefore uses Service/endpoint failover or a side-channel for the election, as documented.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@design-proposals/tenant-site-gateway/README.md`:
- Around line 157-158: The tenant-secret flow is still underspecified in the
README: define the concrete delivery path for PSKs/certs/WireGuard keys into the
guest without putting them in the VM spec or cloud-init, and document the
post-reconciliation read-access model clearly. Update the tenant-supplied
secrets section to name the component or mechanism that performs delivery and
the expected access boundaries so the security story is explicit.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: f6ce1e74-edfe-4394-9c0e-b857bcb66a81
📒 Files selected for processing (1)
design-proposals/tenant-site-gateway/README.md
Adds a High availability section (KubeVirt live-migration for planned maintenance; Service-fronted active/passive and kube-ovn allowed-address-pairs shared-VIP for unplanned failure), with the shared-VIP mechanism validated on a development cluster. Attributes the pod-to-pod drop of non-TCP/UDP/ICMP/SCTP IP protocols (VRRP proto 112, ESP proto 50) to Cilium's conntrack rather than the geneve tunnel, unifying it with the native-ESP finding. Also addresses review feedback: per-target-unique outbound listener ports, a concrete blockSize example, a note that SNAT keeps the gateway anti-spoofing clean, and a stronger secret-handling open question. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
myasnikovdaniil
requested review from
Andrei Kvapil (kvaps) and
Timofei Larkin (lllamnyp)
1 participant
Summary
Adds a design proposal for tenant-managed external network connectivity:
terminating a site-to-site VPN (IPsec or WireGuard) inside a KubeVirt VM that
NAT-bridges an external site to the tenant's managed apps, in both directions,
for any L4 protocol — without granting tenants privileged host-cluster pods.
The proposal covers:
terminated in a VM guest (contained privilege)
packages/apps/site-gatewaycatalog app with a values-schema sketchfills the tenant-namespace-apps + NAT-egress space design-proposal: cross-cluster mesh for tenant access to host services #7 defers
The design was prototyped and validated end-to-end (two gateway VMs + a real
managed Postgres); results are included, notably the finding that native ESP is
dropped by the Cilium/kube-ovn overlay (forced UDP encapsulation required for
IPsec; WireGuard sidesteps it).
Test plan
Design proposal; no code. Implementation testing (helm-unittest across backends,
e2e of the two-VM topology asserting inbound/outbound + SNAT-required + MSS
behavior) is scoped in the proposal and will follow in implementation PRs.
🤖 Generated with Claude Code
Summary by CodeRabbit