feat(shutdown): drain WebSocket clients on SIGTERM

[Ferryx349]

Description

• On SIGTERM, /readyz returns 503 so load balancers stop routing new traffic
• Rejects new WebSocket connections and drains existing clients with Nostr CLOSED and graceful WS close
• Bounded by WS_DRAIN_TIMEOUT_MS (default 30s); primary forwards SIGTERM to cluster workers and waits for exit

Related Issue

Fixes :- #772

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

• Non-functional change (docs, style, minor refactor)
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my code changes.
• I added a changeset, or this is docs-only and I added an empty changeset.
• All new and existing tests passed.

[changeset-bot]

🦋 Changeset detected

Latest commit: d3993fa

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
nostream	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[coveralls]

coverage: 72.608% (+0.2%) from 72.408% — feat/ws-drain into main

[phoenix-server]

Review — drain WebSocket clients on SIGTERM

The shape is right and the mechanics are sound: the worker owns both the HTTP routes and the WS server, so the draining flag genuinely reaches /readyz; the ordering (flag → telemetry → close → drain) is correct; the shuttingDown early-return in onClusterExit is what actually makes a cluster shutdown possible instead of respawning workers as they drain; and the post-timeout terminate() sweep re-reads the live client set, so connections that arrive mid-drain are still caught. Comments below, worst first.

1. The 30 s drain ceiling cannot be honored in the reference deployment

The root docker-compose.yml sets no stop_grace_period; only docker-compose.tor.yml has one (10m30s). So the deployment that will actually run this gets Docker's 10 s default, and the entire cgroup is SIGKILLed mid-drain. Two consequences: the WS_DRAIN_TIMEOUT_MS ceiling is unenforceable (the process is gone long before 30 s), and App.onExit's wait-for-all-workers can never run to completion.

Practical impact today is small — the CLOSED frames and the close handshake go out immediately, and the timeout is only a ceiling — but the knob should mean something. Either set/document a stop_grace_period longer than WS_DRAIN_TIMEOUT_MS in deploy/README.md and the compose file, or lower the default to something under 10 s.

Related, in the same method: the primary's once('exit') wait is unbounded. One worker that hangs in its own close path (a DB pool that will not drain, say) holds the primary until SIGKILL. A Promise.race against a deadline would make the primary's exit as bounded as the worker's drain.

2. The /readyz 503 is effectively unreachable

WebSocketServerAdapter.close() calls super.close() first, and WebServerAdapter.close() calls this.webServer.close() immediately — the HTTP listener stops accepting new connections before drainClients() runs. A load balancer's check during shutdown therefore gets ECONNREFUSED rather than 503. HAProxy marks the backend down either way, so the outcome is fine, but the changeset and deploy/README.md both advertise the 503 as the mechanism, and it is not what a checker observes.

Either drain (or set the draining flag and wait a beat) before closing the listener, so the 503 is what the LB actually sees, or correct those two sentences to describe connection refusal.

3. The drain response reports a fabricated dependency outage

getReadyzRequestHandler answers with database: {ok: false}, redis: {ok: false} while draining. The real reason is "draining"; anything monitoring dependency health reads a false Postgres/Redis outage for the whole drain window. A distinct status (draining), or a real snapshot with only the status overridden, is more honest and no harder.

4. The primary's new shutdown path has no test

test/unit/app/ covers the worker types but there is no app.spec.ts, so the riskiest lines in this PR are uncovered: SIGTERM forwarded to every worker, waiting for all exits, and — most importantly — the shuttingDown early-return that suppresses respawn. That early return is load-bearing: without it the cluster restarts each worker as it exits, and the drain never completes. It deserves a test the way the drain timeout and the reject-while-draining paths got theirs.

Nits

• worker.process.kill('SIGTERM') where the cluster's own worker.kill() is the idiomatic call.
• resetDrainingState() ships in src/utils/ and is used only by tests.
• The CLOSED reason closed: relay shutting down reads as a machine-readable prefix, but closed: is not one; the repo already uses error: (error: unable to count events) and auth-required:. error: relay shutting down, or plain prose, would match.
• PR body: Fixes :- #772 has a stray :-, so the issue will not auto-close on merge. Checklist is also unchecked and "How Has This Been Tested?" is empty.

Checked and deliberately not flagged

• Reading process.env.WS_DRAIN_TIMEOUT_MS directly in a util module is consistent with the existing ADMIN_DEPENDENCY_PING_TIMEOUT_MS / ADMIN_METRICS_SSE_INTERVAL_MS precedent in src/utils/ — no change needed.
• Promise.race against the timeout leaves the per-client once('close') listeners attached, but the follow-up terminate() fires close on exactly those sockets, so there is no leak to chase.
• Closing the HTTP listener first means new WS upgrades stop before onConnection can reject them; the isDraining() guard remains correct for the upgrade race it does cover.

Mergeable after 1 and 2; 3 and 4 are cheap and worth taking in the same pass.

[Ferryx349]

Addressed the comments.