Integration with SandD

.env.example

@@ -1,11 +1,17 @@
-# Nebula provider credentials — SECRETS ONLY.
+# Nebula provider credentials — SECRETS (plus one per-cluster endpoint).
 #
 # Copy to .env and fill in real values. .env is gitignored — never commit tokens.
-# Consumed by hack/deploy.sh (make deploy-all), which turns these into one
-# Kubernetes Secret PER PROVIDER. Non-secret config (image, namespace, Kind
+# Consumed by hack/deploy.sh (make deploy-all), which turns the credentials into one
+# Kubernetes Secret PER PROVIDER. Static non-secret config (image, namespace, Kind
 # cluster) is NOT here — pass it as make variables, e.g.
 #   make deploy-all IMG=myrepo/nebula:v1 DEPLOY_KIND_CLUSTER=nebula-test-e2e
 #
+# The lone non-secret that DOES belong here is SANDD_TUNNEL_SERVER (bottom): it is a
+# per-cluster value discovered at deploy time (the headscale NLB hostname), so keeping
+# it beside the deploy that consumes it — rather than a hand-passed make flag re-typed
+# every run — is the whole point. It rides the SAME parsed-not-sourced path as the
+# creds, so it can't leak into kubectl either.
+#
 #   cp .env.example .env
 #   # edit .env
 #   make deploy-all
@@ -17,13 +23,43 @@ MODAL_TOKEN_ID=
 MODAL_TOKEN_SECRET=
 
 # --- AWS provider ----------------------------------------------------------
-# SECRETS ONLY. In production prefer IRSA / instance role and leave these blank
-# (the SDK's default credential chain finds the role) — the AWS secret is then
-# skipped, which is fine. Set them only for local/dev without a role. Both keys
-# are required together; leave both blank to skip.
+# SECRETS ONLY — the PROVISIONING identity (the AWS account that launches GPU
+# instances). This is a SEPARATE identity from the one that talks to the cluster
+# you deploy into (e.g. an EKS control plane in a different account).
+#
+# These use the standard AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY names — the
+# same names `aws eks get-token` (kubectl's EKS auth plugin) reads — yet they do
+# NOT collide with cluster auth: deploy.sh PARSES this file into a private array
+# rather than sourcing/exporting it, so these values reach the provider Secret but
+# never enter the environment of kubectl or any other child process. Your own
+# ambient AWS credentials keep talking to the cluster.
+#
+# In production prefer IRSA / instance role and leave these blank (the SDK's
+# default credential chain finds the role) — the AWS secret is then skipped, which
+# is fine. Set them only for local/dev without a role. Both required together;
+# leave both blank to skip.
 AWS_ACCESS_KEY_ID=
 AWS_SECRET_ACCESS_KEY=
 
 # --- Additional providers (add as adapters land) ---------------------------
 # Each provider gets its OWN secret (see hack/deploy.sh PROVIDER_SECRETS), e.g.:
 # RUNPOD_API_KEY=
+
+# --- SandD mesh endpoint (NON-SECRET, per-cluster) -------------------------
+# The internet-facing headscale NLB hostname the SandD mesh dials, as a bare
+# http://<host> (no port — the Service listens on 80; see config/sandd). It is only
+# known AFTER config/samples/headscale-service.yaml provisions and changes on every
+# recreate, so it can't live in the tracked manifests. deploy-all reads it from here
+# and `make deploy` substitutes the __SANDD_TUNNEL_SERVER__ token into headscale's
+# server_url and the nebula-sandd-config ConfigMap (byte-identical, as headscale
+# requires) — no re-typing across files, no manager restart.
+#
+# Read it back after applying the Service (README config/sandd step 1):
+#   kubectl apply -f config/samples/headscale-service.yaml
+#   HS=$(kubectl -n nebula-system get svc nebula-headscale \
+#     -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
+#   echo "SANDD_TUNNEL_SERVER=http://$HS"   # paste the result below, then make deploy-all
+#
+# Leave blank to skip SandD wiring — the token is then left in place (an obviously
+# broken render, surfaced by a deploy.sh warning), not a silent misconfig.
+SANDD_TUNNEL_SERVER=

Dockerfile.keybroker

@@ -0,0 +1,37 @@
+# keybroker — the headscale pre-auth key broker (cmd/keybroker), packaged as a
+# SIDECAR for the headscale pod. It shells out to the `headscale` CLI over the
+# shared unix socket, so the final image must carry BOTH our Go binary AND the
+# headscale CLI — hence the `FROM headscale/headscale` base rather than distroless.
+#
+# Build (from the repo root):
+#   docker build -f Dockerfile.keybroker -t inftyai/nebula-keybroker:latest .
+#
+# Keep the headscale tag here IN SYNC with config/sandd/headscale.yaml so the CLI
+# in this sidecar speaks the same wire/flag version as the headscale it dials.
+
+# --- build the broker binary ---------------------------------------------------
+FROM golang:1.24 AS builder
+ARG TARGETOS
+ARG TARGETARCH
+
+WORKDIR /workspace
+COPY go.mod go.mod
+COPY go.sum go.sum
+RUN go mod download
+
+# Only the broker's own sources are needed (it is stdlib-only), but copy the
+# module dirs it might import so the build is self-contained.
+COPY cmd/keybroker/ cmd/keybroker/
+
+# CGO off => a static binary that runs on the headscale base image regardless of
+# its libc.
+RUN CGO_ENABLED=0 GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH} \
+    go build -a -o keybroker ./cmd/keybroker
+
+# --- final image: headscale CLI + our broker -----------------------------------
+# FROM headscale so `headscale preauthkeys create` is on PATH. The broker calls it
+# over the local unix socket shared with the real headscale container in the pod.
+FROM headscale/headscale:0.23
+COPY --from=builder /workspace/keybroker /usr/local/bin/keybroker
+# Override headscale's own entrypoint: this container runs the BROKER, not headscale.
+ENTRYPOINT ["/usr/local/bin/keybroker"]

Makefile

@@ -1,6 +1,12 @@
 # Image URL to use all building/pushing image targets
 IMG ?= inftyai/nebula-controller:latest
 
+# KEYBROKER_IMG is the SandD key-broker sidecar image (cmd/keybroker,
+# Dockerfile.keybroker) that runs alongside headscale — see config/sandd. Built and
+# pushed separately from the manager IMG because it is a distinct, optional image
+# with its own base (FROM headscale, so it carries the headscale CLI).
+KEYBROKER_IMG ?= inftyai/nebula-keybroker:latest
+
 # NAMESPACE is where the manager runs (must match config/manager). Consumed by
 # hack/deploy.sh via the deploy-all target.
 NAMESPACE ?= nebula-system
@@ -155,13 +161,21 @@ docker-build: ## Build docker image with the manager.
 docker-push: ## Push docker image with the manager.
 	$(CONTAINER_TOOL) push ${IMG}
 
+.PHONY: docker-build-keybroker
+docker-build-keybroker: ## Build docker image for the SandD key-broker sidecar.
+	$(CONTAINER_TOOL) build -f Dockerfile.keybroker -t ${KEYBROKER_IMG} .
+
+.PHONY: docker-push-keybroker
+docker-push-keybroker: ## Push the SandD key-broker sidecar image.
+	$(CONTAINER_TOOL) push ${KEYBROKER_IMG}
+
 # PLATFORMS defines the target platforms for the manager image be built to provide support to multiple
 # architectures. (i.e. make docker-buildx IMG=myregistry/mypoperator:0.0.1). To use this option you need to:
 # - be able to use docker buildx. More info: https://docs.docker.com/build/buildx/
 # - have enabled BuildKit. More info: https://docs.docker.com/develop/develop-images/build_enhancements/
 # - be able to push the image to your registry (i.e. if you do not set a valid value via IMG=<myregistry/image:<tag>> then the export will fail)
 # To adequately provide solutions that are compatible with multiple platforms, you should consider using this option.
-PLATFORMS ?= linux/arm64,linux/amd64,linux/s390x,linux/ppc64le
+PLATFORMS ?= linux/arm64,linux/amd64
 .PHONY: docker-buildx
 docker-buildx: ## Build and push docker image for the manager for cross-platform support
 	# copy existing Dockerfile and insert --platform=${BUILDPLATFORM} into Dockerfile.cross, and preserve the original Dockerfile
@@ -195,7 +209,17 @@ uninstall: manifests kustomize ## Uninstall CRDs from the K8s cluster specified
 .PHONY: deploy
 deploy: manifests kustomize ## Deploy controller to the K8s cluster specified in ~/.kube/config.
 	cd config/manager && $(KUSTOMIZE) edit set image controller=${IMG}
-	$(KUSTOMIZE) build $(KUSTOMIZE_BUILD_FLAGS) config/default | $(KUBECTL) apply -f -
+	# SANDD_TUNNEL_SERVER is the internet-facing headscale NLB hostname, only known
+	# after that Service provisions and different on every recreate — so it can't be
+	# baked into the tracked manifests. The config/sandd overlay carries the literal
+	# token __SANDD_TUNNEL_SERVER__ in headscale's server_url and the nebula-sandd-config
+	# ConfigMap; substitute it here so both render correct on the FIRST apply (no
+	# manager/headscale restart). Driven from .env via deploy-all (make deploy-all),
+	# or pass SANDD_TUNNEL_SERVER=... directly. When unset the token is left as-is (the
+	# render is then obviously broken rather than silently pointing nowhere).
+	$(KUSTOMIZE) build $(KUSTOMIZE_BUILD_FLAGS) config/default | \
+		$(if $(SANDD_TUNNEL_SERVER),sed 's|__SANDD_TUNNEL_SERVER__|$(SANDD_TUNNEL_SERVER)|g',cat) | \
+		$(KUBECTL) apply -f -
 
 .PHONY: deploy-e2e
 deploy-e2e: manifests kustomize ## Deploy for e2e: config/default plus the fake-provider env var (baked in at deploy time, not via a post-deploy rollout).

README.md

@@ -96,3 +96,9 @@ placement controller owns those.
 - See [config/samples](config/samples) for example NodePools and a runnable workload.
 - See [docs/add-a-provider.md](docs/add-a-provider.md) to add a provider backend.
 - See [docs/architecture.md](docs/architecture.md) for design details.
+
+## License
+
+Apache-2.0 — see [LICENSE](LICENSE). Third-party components (e.g. Tailscale, used by
+the optional SandD channel) are listed in
+[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).

THIRD_PARTY_NOTICES.md

@@ -0,0 +1,15 @@
+# Third-party notices
+
+Nebula is licensed under Apache-2.0 (see [LICENSE](LICENSE)). It relies on the
+following third-party software, which is licensed separately.
+
+## Tailscale
+
+The optional [SandD](https://github.com/InftyAI/SandD) access channel uses the
+[Tailscale](https://github.com/tailscale/tailscale) client (`tailscale`/`tailscaled`),
+© Tailscale Inc., licensed under
+[BSD-3-Clause](https://github.com/tailscale/tailscale/blob/main/LICENSE).
+
+Nebula does not link or vendor it; the provider integration fetches the official
+binary onto the GPU host at provision time. If you build and redistribute an image
+with Tailscale baked in, retain its copyright notice per BSD-3-Clause.

api/v1alpha1/nodepool_types.go

@@ -136,9 +136,9 @@ const (
 type FailoverPolicy struct {
 	// BlocklistTTL is the BASE duration a failed placement is excluded before the
 	// provider becomes a candidate for it again. The controller adds a random jitter
-	// (up to a minute) on top so Pods that failed for the same reason do not all
-	// retry the just-freed candidate in lockstep, so the effective exclusion is this
-	// value plus that jitter.
+	// (up to 30s) on top so Pods that failed for the same reason do not all retry the
+	// just-freed candidate in lockstep, so the effective exclusion is this value plus
+	// that jitter.
 	// +kubebuilder:default="30s"
 	BlocklistTTL metav1.Duration `json:"blocklistTTL,omitempty"`
 }