Skip to content

fix(postgres): probe over TCP so dependents aren't released early - #152

Merged
slayerjain merged 1 commit into
mainfrom
fix/pg-healthcheck-tcp
Sep 4, 2026
Merged

fix(postgres): probe over TCP so dependents aren't released early#152
slayerjain merged 1 commit into
mainfrom
fix/pg-healthcheck-tcp

Conversation

@slayerjain

Copy link
Copy Markdown
Member

The bug

postgres' docker-entrypoint runs a temporary server reachable only over the Unix socket (listen_addresses='') once initdb has finished, so it can create the database and run docker-entrypoint-initdb.d.

A pg_isready healthcheck without -h talks to that socket. So it reports "accepting connections" while no TCP listener exists at all — compose marks the service healthy, depends_on: condition: service_healthy releases the dependent, and the dependent connects straight into ECONNREFUSED.

It only bites on a fresh volume, because a populated one skips initdb and the temporary server entirely. That is why it surfaces as a rare flake rather than a consistent failure.

Evidence

From the CI failure this was diagnosed from (enterprise umami-linux, a matrix cell that creates its volume):

event time
temp server "ready to accept connections" (socket only) 08:54:29.422 — false-healthy begins
fast shutdown request 08:54:30.668 — window 1.25s
shutdown checkpoint 08:54:30.78008:54:33.421 (2.7s)
real TCP listener after 08:54:33.583

compose logged Container umami_db WaitingContainer umami_app Started, then the app died with Can't reach database server at 172.78.0.10:5432, while the db showed Up 11 seconds (healthy).

Verification

Reproduced and mutation-tested locally with a compose mirroring the depends_on: service_healthy shape:

healthcheck dependent
pg_isready -U ... -d ... APP_FAILED_TO_REACH_DB, exit 1
pg_isready -h 127.0.0.1 -U ... -d ... APP_OK, exit 0

Directly measured the false-healthy window against the patched file: the old check reports healthy from t=2s while TCP is refused until t=8s; the patched check holds starting and only goes healthy once TCP is genuinely up. Reproduced on postgres 13.3, 15 and 16.

Why -h 127.0.0.1

The stock images ship listen_addresses = '*', so the real server binds 0.0.0.0 — one bind() covers loopback and the container IP, and the static ipv4_address is configured before PID 1 runs, so there is no phase where loopback works and the network IP does not. During startup/recovery pg_isready reports rejecting connections, so the probe correctly keeps failing until the server can actually serve. localhost would be the wrong literal (it may resolve ::1 first).

Note on pg_isready semantics

pg_isready is a liveness probe: it exits 0 even for a nonexistent role/database, and even when pg_hba.conf rejects the connection. The -U/-d flags here are effectively decorative. That is fine for this fix — the real listener only starts after initdb created the role and database — but it does mean the probe is not a substitute for an application-level readiness check.

postgres' docker-entrypoint runs a temporary server reachable only over the
Unix socket (listen_addresses='') once initdb has finished, so it can create
the database and run docker-entrypoint-initdb.d. A pg_isready probe without
-h talks to that socket, so it answers "accepting connections" while no TCP
listener exists at all. compose then marks the service healthy and
depends_on: service_healthy releases the dependent straight into
ECONNREFUSED.

Measured at ~235ms with no initdb.d scripts, and at 1.2s of false-healthy
followed by a 2.7s shutdown checkpoint in the CI failure this was diagnosed
from. Reproduced on postgres 13.3, 15 and 16. Only bites on a fresh volume,
since a populated one skips the temporary server entirely -- which is why it
surfaced as a rare flake rather than a consistent failure.

Passing -h 127.0.0.1 forces a TCP probe, which tracks the real listener in
every phase: the stock images ship listen_addresses = '*', and during startup
or recovery pg_isready reports "rejecting connections", so the probe
correctly keeps failing until the server can actually serve.

The k8s manifest gets the same change on its readinessProbe only. The
livenessProbe deliberately keeps the socket check: liveness asks whether the
process is alive and a failure restarts the pod, so a TCP probe there would
restart-loop a pod whose initdb outruns
initialDelaySeconds + failureThreshold * periodSeconds.

Signed-off-by: slayerjain <shubhamkjain@outlook.com>
@slayerjain
slayerjain merged commit dc61839 into main Sep 4, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant