fix(postgres): probe over TCP so dependents aren't released early - #152
Merged
Conversation
postgres' docker-entrypoint runs a temporary server reachable only over the Unix socket (listen_addresses='') once initdb has finished, so it can create the database and run docker-entrypoint-initdb.d. A pg_isready probe without -h talks to that socket, so it answers "accepting connections" while no TCP listener exists at all. compose then marks the service healthy and depends_on: service_healthy releases the dependent straight into ECONNREFUSED. Measured at ~235ms with no initdb.d scripts, and at 1.2s of false-healthy followed by a 2.7s shutdown checkpoint in the CI failure this was diagnosed from. Reproduced on postgres 13.3, 15 and 16. Only bites on a fresh volume, since a populated one skips the temporary server entirely -- which is why it surfaced as a rare flake rather than a consistent failure. Passing -h 127.0.0.1 forces a TCP probe, which tracks the real listener in every phase: the stock images ship listen_addresses = '*', and during startup or recovery pg_isready reports "rejecting connections", so the probe correctly keeps failing until the server can actually serve. The k8s manifest gets the same change on its readinessProbe only. The livenessProbe deliberately keeps the socket check: liveness asks whether the process is alive and a failure restarts the pod, so a TCP probe there would restart-loop a pod whose initdb outruns initialDelaySeconds + failureThreshold * periodSeconds. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
postgres'docker-entrypointruns a temporary server reachable only over the Unix socket (listen_addresses='') onceinitdbhas finished, so it can create the database and rundocker-entrypoint-initdb.d.A
pg_isreadyhealthcheck without-htalks to that socket. So it reports "accepting connections" while no TCP listener exists at all — compose marks the service healthy,depends_on: condition: service_healthyreleases the dependent, and the dependent connects straight intoECONNREFUSED.It only bites on a fresh volume, because a populated one skips
initdband the temporary server entirely. That is why it surfaces as a rare flake rather than a consistent failure.Evidence
From the CI failure this was diagnosed from (enterprise
umami-linux, a matrix cell that creates its volume):08:54:29.422— false-healthy begins08:54:30.668— window 1.25s08:54:30.780→08:54:33.421(2.7s)08:54:33.583compose logged
Container umami_db Waiting→Container umami_app Started, then the app died withCan't reach database server at 172.78.0.10:5432, while the db showedUp 11 seconds (healthy).Verification
Reproduced and mutation-tested locally with a compose mirroring the
depends_on: service_healthyshape:pg_isready -U ... -d ...APP_FAILED_TO_REACH_DB, exit 1pg_isready -h 127.0.0.1 -U ... -d ...APP_OK, exit 0Directly measured the false-healthy window against the patched file: the old check reports healthy from t=2s while TCP is refused until t=8s; the patched check holds
startingand only goeshealthyonce TCP is genuinely up. Reproduced on postgres 13.3, 15 and 16.Why
-h 127.0.0.1The stock images ship
listen_addresses = '*', so the real server binds0.0.0.0— onebind()covers loopback and the container IP, and the staticipv4_addressis configured before PID 1 runs, so there is no phase where loopback works and the network IP does not. During startup/recoverypg_isreadyreports rejecting connections, so the probe correctly keeps failing until the server can actually serve.localhostwould be the wrong literal (it may resolve::1first).Note on
pg_isreadysemanticspg_isreadyis a liveness probe: it exits 0 even for a nonexistent role/database, and even whenpg_hba.confrejects the connection. The-U/-dflags here are effectively decorative. That is fine for this fix — the real listener only starts afterinitdbcreated the role and database — but it does mean the probe is not a substitute for an application-level readiness check.