From c19d8a83d26fbed6af818c201d58dc18b0a54117 Mon Sep 17 00:00:00 2001 From: Jonas Thelemann Date: Mon, 7 Sep 2026 13:52:30 +0200 Subject: [PATCH] docs(swarm): correct manager address rebinding --- docs/multi-node.md | 72 ++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 66 insertions(+), 6 deletions(-) diff --git a/docs/multi-node.md b/docs/multi-node.md index 2bc5186b..b5c30996 100644 --- a/docs/multi-node.md +++ b/docs/multi-node.md @@ -69,13 +69,59 @@ None of those ports belong on the public interface. ### Moving an existing single-node swarm onto the private network -A node's data plane address is fixed when it joins, and there is no command to change it in place. -Rebinding an existing manager means `docker swarm init --force-new-cluster`, which keeps every service, secret, config and volume. +A node has two addresses, and they move independently. +The data plane address is the endpoint the overlay's VXLAN tunnels terminate on, and it surfaces as `Status.Addr`. +The control plane address is what other nodes are told to use to reach the manager, and it surfaces as `ManagerStatus.Addr` and in `/var/lib/docker/swarm/state.json`. -Get the flags right the first time. -The daemon stops the running swarm node before it validates the request, so a rejected argument leaves the node stopped and no longer a manager, and retrying then fails because the command requires a manager to begin with. -Both flags take a bare address: `10.99.1.1`, never `10.99.1.1/32`. -Recovering from that state means restarting the Docker daemon, which rebuilds the swarm node from `/var/lib/docker/swarm/`. +`docker swarm init --force-new-cluster` rebinds the data plane address but leaves the control plane address untouched. +It restores the raft state from the existing snapshot, and the manager's own address is part of that snapshot. +Restarting the Docker daemon does not move it either, even though `/var/lib/docker/swarm/docker-state.json` records the requested address by then. + +That distinction matters because a worker does not keep the address it joined on. +It replaces its stored remote with whatever the manager advertises, and libnetwork seeds the overlay's gossip cluster from that list on port `7946`. +A manager advertising a public address therefore breaks the overlay on every worker, whichever address the worker was joined with. + +The failure is quiet. +The worker's dispatcher session stays up on the address it originally dialled, so `docker node ls` reports the node `Ready` while no gossip ever reaches it. +Service names then resolve only to the tasks on the local node, and containers on different nodes cannot reach each other by name. + +Changing the control plane address means a genuine reinitialisation. + +```sh +docker swarm leave --force +docker swarm init --advertise-addr --data-path-addr --listen-addr +``` + +Every address flag takes a bare address: `10.99.1.1`, never `10.99.1.1/32`. + +That destroys the raft store, and with it every service, secret and config, along with the storage label. +Volumes are left alone, so no data is lost, but the stack has to be redeployed afterwards and the label reapplied. + +#### Preserving the secrets + +Production secrets are external, so `dargstack deploy` does not recreate them and no copy exists on the host. +The raft store is their only home, and `docker secret inspect` does not return values. +Read them out of a container before leaving the swarm, and stage them somewhere that never touches the disk. + +```sh +docker service create --name secret-dump --constraint node.hostname== --restart-condition none \ + $(docker secret ls --format '{{.Name}}' | sed 's/^/--secret /' | tr '\n' ' ') busybox:latest sleep 3600 +mkdir -p /dev/shm/secrets +docker exec "$(docker ps -q -f name=secret-dump)" tar cf - -C /run/secrets . | tar xf - -C /dev/shm/secrets +docker service rm secret-dump +``` + +Confirm the dump holds every secret and that none of the files came out empty, then recreate them once the new swarm is up. + +```sh +cd /dev/shm/secrets && for f in *; do docker secret create "$f" "$f"; done +``` + +Remove the staging directory afterwards with `rm -rf /dev/shm/secrets`. +It lives in `tmpfs`, so its contents never reach persistent storage, but they do not survive a reboot either, and between leaving the old swarm and recreating the secrets it holds the only copy. + +Registry credentials need the same treatment. +Swarm embeds them into the service spec at deploy time rather than storing them centrally, so a rebuild leaves private images unpullable until `docker login` runs again and the affected services are updated with `--with-registry-auth`. ### Joining a node @@ -91,6 +137,19 @@ Confirm the address took before deploying anything, since a node that joined on docker node inspect --format '{{.Status.Addr}}' ``` +Check the manager's control plane address as well, since the two are set separately and only one of them is visible from the joining node. + +```sh +docker node inspect --format '{{.ManagerStatus.Addr}}' +``` + +Neither reading proves the overlay works. +That takes a name resolving across nodes: with a global service running everywhere, its `tasks.` name has to return one address per node rather than only the local one. + +```sh +docker exec nslookup tasks. +``` + ### Network MTU Hetzner Cloud's private networks run at an MTU of 1450 rather than the 1500 the overlay driver assumes, so `src/production/compose.yaml` sizes the stack's network down to 1400 to leave room for the VXLAN header. @@ -99,3 +158,4 @@ Docker never updates an existing network's options on deploy, so changing that v Getting it wrong does not look like a network fault. Health checks and small requests keep succeeding while large payloads stall, which reads as an application bug for as long as anyone is willing to chase it. +A multi-megabyte transfer between containers on different nodes settles it, where a health check would not.