Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 66 additions & 6 deletions docs/multi-node.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,13 +69,59 @@ None of those ports belong on the public interface.

### Moving an existing single-node swarm onto the private network

A node's data plane address is fixed when it joins, and there is no command to change it in place.
Rebinding an existing manager means `docker swarm init --force-new-cluster`, which keeps every service, secret, config and volume.
A node has two addresses, and they move independently.
The data plane address is the endpoint the overlay's VXLAN tunnels terminate on, and it surfaces as `Status.Addr`.
The control plane address is what other nodes are told to use to reach the manager, and it surfaces as `ManagerStatus.Addr` and in `/var/lib/docker/swarm/state.json`.

Get the flags right the first time.
The daemon stops the running swarm node before it validates the request, so a rejected argument leaves the node stopped and no longer a manager, and retrying then fails because the command requires a manager to begin with.
Both flags take a bare address: `10.99.1.1`, never `10.99.1.1/32`.
Recovering from that state means restarting the Docker daemon, which rebuilds the swarm node from `/var/lib/docker/swarm/`.
`docker swarm init --force-new-cluster` rebinds the data plane address but leaves the control plane address untouched.
It restores the raft state from the existing snapshot, and the manager's own address is part of that snapshot.
Restarting the Docker daemon does not move it either, even though `/var/lib/docker/swarm/docker-state.json` records the requested address by then.

That distinction matters because a worker does not keep the address it joined on.
It replaces its stored remote with whatever the manager advertises, and libnetwork seeds the overlay's gossip cluster from that list on port `7946`.
A manager advertising a public address therefore breaks the overlay on every worker, whichever address the worker was joined with.

The failure is quiet.
The worker's dispatcher session stays up on the address it originally dialled, so `docker node ls` reports the node `Ready` while no gossip ever reaches it.
Service names then resolve only to the tasks on the local node, and containers on different nodes cannot reach each other by name.

Changing the control plane address means a genuine reinitialisation.

```sh
docker swarm leave --force
docker swarm init --advertise-addr <node-private> --data-path-addr <node-private> --listen-addr <node-private>
```

Every address flag takes a bare address: `10.99.1.1`, never `10.99.1.1/32`.

That destroys the raft store, and with it every service, secret and config, along with the storage label.
Volumes are left alone, so no data is lost, but the stack has to be redeployed afterwards and the label reapplied.

#### Preserving the secrets

Production secrets are external, so `dargstack deploy` does not recreate them and no copy exists on the host.
The raft store is their only home, and `docker secret inspect` does not return values.
Read them out of a container before leaving the swarm, and stage them somewhere that never touches the disk.

```sh
docker service create --name secret-dump --constraint node.hostname==<manager> --restart-condition none \
$(docker secret ls --format '{{.Name}}' | sed 's/^/--secret /' | tr '\n' ' ') busybox:latest sleep 3600
mkdir -p /dev/shm/secrets
docker exec "$(docker ps -q -f name=secret-dump)" tar cf - -C /run/secrets . | tar xf - -C /dev/shm/secrets
docker service rm secret-dump
```

Confirm the dump holds every secret and that none of the files came out empty, then recreate them once the new swarm is up.

```sh
cd /dev/shm/secrets && for f in *; do docker secret create "$f" "$f"; done
```

Remove the staging directory afterwards with `rm -rf /dev/shm/secrets`.
It lives in `tmpfs`, so its contents never reach persistent storage, but they do not survive a reboot either, and between leaving the old swarm and recreating the secrets it holds the only copy.

Registry credentials need the same treatment.
Swarm embeds them into the service spec at deploy time rather than storing them centrally, so a rebuild leaves private images unpullable until `docker login` runs again and the affected services are updated with `--with-registry-auth`.

### Joining a node

Expand All @@ -91,6 +137,19 @@ Confirm the address took before deploying anything, since a node that joined on
docker node inspect <node> --format '{{.Status.Addr}}'
```

Check the manager's control plane address as well, since the two are set separately and only one of them is visible from the joining node.

```sh
docker node inspect <manager> --format '{{.ManagerStatus.Addr}}'
```

Neither reading proves the overlay works.
That takes a name resolving across nodes: with a global service running everywhere, its `tasks.` name has to return one address per node rather than only the local one.

```sh
docker exec <container> nslookup tasks.<service>
```

### Network MTU

Hetzner Cloud's private networks run at an MTU of 1450 rather than the 1500 the overlay driver assumes, so `src/production/compose.yaml` sizes the stack's network down to 1400 to leave room for the VXLAN header.
Expand All @@ -99,3 +158,4 @@ Docker never updates an existing network's options on deploy, so changing that v

Getting it wrong does not look like a network fault.
Health checks and small requests keep succeeding while large payloads stall, which reads as an application bug for as long as anyone is willing to chase it.
A multi-megabyte transfer between containers on different nodes settles it, where a health check would not.
Loading