Recover PRISM watchdog restarts in ~1s instead of 60s+

[Anatolie]

Incident being solved

Mainnet, 2026-08-11 01:55 UTC. The PRISM coordinator's liveness watchdog fired and hard-exited:

prism coordinator: liveness watchdog firing; unresponsive subsystems=['block_submitter']
timeout=300s. Exiting non-zero so the restart policy recovers the process.

Docker restarted the container in ~130 ms, but the replacement process could not open its Stratum listeners for ~60 seconds, because the dead process's Postgres ledger-writer lease was still held. Startup logged this on a 15 s poll until TTL expiry:

prism ledger writer lease held until 2026-08-11 01:56:18.896325+00; waiting 15s before retry
(holder writer=<writer-id> epoch=1 session=<session-token>)

That 60 s window was a paging production outage: public Stratum subscribe timeouts plus sidecar readyz 503s. Runtime config at the time: PRISM_LEDGER_LEASE_TTL_SECONDS=60, PRISM_LEDGER_WRITER_EPOCH=1, PRISM_WRITER_QUIESCENCE_TIMEOUT_SECONDS=15.

Root causes

1. The watchdog never released the lease. watchdog_loop had three os._exit(1) call sites (liveness overdue-heartbeats, publication-progress coordination, and publication). All three bypassed the graceful lease-release machinery that already existed for the SIGTERM path (claim_lease_release / finish_lease_release / lease_release_withheld).
2. Startup waited out the TTL even against its own corpse. The lease-wait path polled every 15 s until TTL expiry even when the observed holder was provably a previous incarnation of this same process — same writer id, same epoch, different session token.

Net effect of this PR: watchdog-exit downtime goes from ~TTL (60 s) + poll jitter (up to 15 s) down to container restart + O(1 s).

What changed

1. Bounded lease release on watchdog exit

All three os._exit(1) sites now route through PrismCoordinator._watchdog_hard_exit(reason), which spawns a fresh daemon thread to attempt release under a hard ~5 s wall-clock deadline (DEFAULT_PRISM_WATCHDOG_LEASE_RELEASE_TIMEOUT_SECONDS) and then os._exit(1)s from a finally block regardless of outcome. Nothing — timeout logging, thread-start failure, a blocked log pipe — can extend or suppress the terminal action.

The watchdog fires precisely because other threads are wedged, so the release path is careful about what it touches:

• It does not call shutdown(), which takes self.lock — potentially held by the very subsystem that stopped making progress. It closes writer admission via the shutdown controller and uses the controller's tracked-writer barrier instead, which preserves the graceful path's lease_release_withheld invariant (never release a lease while writer operations are still in flight, or a successor could interleave with an unfinished write) without that lock.
• It uses release_writer_lease_fresh_connection(), a one-shot psql fork, rather than the shared native connection pool, which may be exhausted or wedged.
• wait_for_writer_quiescence() now accepts an explicit timeout so the watchdog can bound it by its own deadline rather than the 15 s config value.
• Diagnostics moved after the safety-critical work (_watchdog_exit_diagnostic), so a full container log pipe cannot park the process before the release attempt.

2. Same-identity lease adoption at startup

When the observed holder has our writer id and our epoch but a different session token, the replacement fences and adopts immediately instead of waiting out the TTL. The safety analysis is the bulk of the work here, because the dangerous case is a live twin (old container still running through a compose recreate) which will keep renewing.

The chosen invariant is a three-layer proof, documented in comments at _writer_lease_adoption_wait_seconds and _try_adopt_writer_lease:

1. Postgres session advisory lock. Each coordinator holds pg_try_advisory_lock(hash(writer_id, epoch)) on a dedicated, never-transparently-replaced connection (_NativePostgresLeaseGuard). A live twin still holds that lock, so a replacement blocks at guard acquisition and never even reaches lease polling. Losing the connection loses the lock, which is what makes it a real liveness signal rather than a timestamp guess.
2. Silence proof. Even after acquiring the guard, Postgres must report the exact same session token unchanged for lease_adoption_silence_seconds (default 1.0 s) before any takeover — giving a holder whose guard connection dropped time to notice and self-exit.
3. Exact-session CAS. _try_adopt_writer_lease updates only where (writer_id, writer_epoch, writer_session_token, updated_at) all match what was observed. Postgres row-lock ordering makes this the final fence: a predecessor renewal that wins first changes updated_at and the CAS affects zero rows; if the CAS wins first, every later predecessor mutation is fenced out. A CAS loss re-observes the owner and requires a fresh full silence interval rather than permanently blacklisting the token (permanent refusal would recreate the original TTL outage if that renewal was the holder's dying act).

Different writer id or different epoch is never adopted — those keep ordinary TTL fencing.

Fast adoption is opt-in per session via the heartbeat-v1: session-token prefix, which only the coordinator assigns. psql-only deployments cannot hold a session advisory lock; they downgrade the token before publishing it, log prism ledger writer fast adoption disabled: ..., and retain TTL fencing. Other ledger users are never treated as fast-adoptable merely because they share an identity.

3. Lease heartbeat + liveness monitor

For the silence proof to be sound, a healthy coordinator must be visibly live. ledger_lease_heartbeat_loop renews on the guard's isolated session every 0.25 s — four renewals inside the 1 s silence window. ledger_lease_heartbeat_monitor_loop polls every 0.05 s and hard-exits if the last success is older than 0.75 s, so a coordinator whose heartbeat wedges is gone before Postgres could ever observe a full second of silence. Startup blocks until the first heartbeat lands, and hard-exits if it doesn't arrive inside the same budget.

4. Per-RPC exact-session fence before external side effects

The periodic heartbeat is an early process-liveness detector, not an authorization oracle: CLOCK_MONOTONIC may not advance across host suspend, and a Postgres connection object doesn't necessarily know its server session died until the next I/O. So _require_fresh_ledger_lease_for_external_side_effect(component) performs a synchronous, bounded (0.5 s) exact-session renewal on the guard connection before every external mutation — submitblock, and the CTV broadcaster's getnewaddress / signrawtransactionwithwallet / submitpackage / sendrawtransaction (wired through the new before_external_side_effect hook on CtvFanoutBroadcaster). It runs on a daemon worker so a dead network path is bounded even when the driver's server-side statement timeout can't help. Failure or timeout hard-exits and raises ShutdownInProgress — fail closed.

Relatedly, JsonRpc._call no longer applies its transparent one-shot transport retry to mutating methods (_QBIT_RPC_NO_TRANSPORT_RETRY_METHODS). The lease fence applies to the outer call, so an invisible second POST could otherwise land after the lease was lost. Read-only calls keep the retry. Durable block/CTV workflows retry later as a new, fully fenced operation.

5. Exit-code discrepancy (forensics)

The incident report showed Docker recording ExitCode=0 despite os._exit(1). Investigation found no code swallowing the status: the PRISM image has no entrypoint or init wrapper, its exec-form CMD starts python3 -m lab.prism.prism_coordinator as container PID 1, and neither compose file overrides it — so status 1 reaches Docker unchanged.

The ExitCode=0 reading was an artifact of method: docker inspect reports the container's current execution state, not an invocation history, and shows 0 once the replacement is already running. docs/mainnet-deployment.md now documents this, along with the correct docker events --filter event=die incantation to capture the failing invocation's real exitCode, and a Dockerfile comment warning that any future wrapper must exec the coordinator or explicitly propagate its status.

Two additional fixes found in review

Serve-path AttributeError (commit 4f695ad). _serve_with_listener_stack calls _start_ledger_lease_heartbeat() as its first statement, and that method dereferenced self.ledger directly. Coordinators built by test fixtures via PrismCoordinator.__new__ (bypassing __init__, which is what sets self.ledger) entered serve() without the attribute and crashed with AttributeError: 'PrismCoordinator' object has no attribute 'ledger' — breaking tests/test_prism_stratum_restart_bind.py and therefore CI's python -m unittest discover. Fixed in production code, not the test: a missing ledger is now treated exactly like a non-fast-adoption-capable ledger (getattr(self, "ledger", None)), returning None and starting no threads. The same defensive pattern was applied to the other fixture-reachable paths added by this branch (ledger_lease_heartbeat_loop, _require_fresh_ledger_lease_for_external_side_effect, release_ledger_lease); paths unreachable without a ledger were left alone. No behavior change when self.ledger is present.

Freshness-stamp regression (commit 9f4c710). Two paths wrote _ledger_lease_heartbeat_last_success_monotonic with plain assignment of their own call-start time: the heartbeat thread, and the per-RPC fence (which writes on the calling thread after joining a worker with up to a 0.5 s budget). Neither guarded against regression, so a slower writer could overwrite a fresher stamp with an older one. Concrete failure: fence starts at T0, its worker renew succeeds at T0+0.45, the heartbeat stamps T0+0.25 and T0+0.5, then the fence caller is descheduled (GIL convoy, throttled cgroup) and at T0+0.75 writes T0 — the 0.05 s monitor sees age ≥ 0.75 s and hard-exits a coordinator whose heartbeat and advisory guard are perfectly healthy. Availability-only (the stamp is a liveness signal, not an authorization check; both writes are legitimate successful renews — only the ordering was wrong).

Fixed by funneling both writers through _record_ledger_lease_heartbeat_success(), which takes a dedicated lock (deliberately not _ledger_lease_heartbeat_failure_lock) and applies a monotonic max(). The initial arm-time assignment in _start_ledger_lease_heartbeat is left as-is. Conservative call-start stamping is preserved at both sites — a delayed response must never look fresher than what the database actually proved. Fencing decisions and hard-exit-on-failure semantics are otherwise unchanged.

Testing

python3 -m unittest tests.test_prism_coordinator_shutdown tests.test_prism_share_ledger tests.test_prism_stratum_restart_bind tests.test_prism_coordinator_vardiff tests.test_prism_progress_health tests.test_ctv_broadcaster tests.test_ctv_broadcaster_daemon → 594 tests, OK.

New coverage (~1000 lines across 5 test files) includes:

• Watchdog exit performs a release attempt that succeeds within the deadline in the happy path, gives up at the deadline and still _exits when the DB hangs (blocking fake), and never runs on the watchdog thread itself.
• Startup adoption takes over a stale same-identity lease immediately, refuses takeover when the holder session renews concurrently, and never adopts a different writer id or epoch.
• _start_ledger_lease_heartbeat() returns None and starts no threads on a coordinator with no ledger attribute.
• The freshness stamp does not regress when a fresher value lands between the fence worker's success and the fence caller's final write (deterministic interleaving, no sleeps). This test fails on the unfixed code — verified by temporarily reverting.

python3 -m unittest discover -s tests -p 'test_*.py' shows no new failures. The pre-existing macOS/Python 3.14 failures (5 splice/spool tests in test_prism_coordinator_job_cache) reproduce identically on origin/1.x.x and are out of scope.

Open risks

• Residual preflight window. The exact-session fence is a preflight check, not an atomic transaction across Postgres and qbitd — two independent systems. A process paused after verification and resumed after its database session was lost could still reach the RPC target. Broadcast and block-submit RPCs deduplicate identical payloads, which limits the blast radius, but separately built CTV fee children can still conflict. Strictly eliminating this requires qbitd/wallet RPCs to validate a coordinator-supplied fencing generation. Documented in docs/mainnet-deployment.md.
• Operational constraint. Operators must not manually start a replacement with the same writer identity while predecessor termination is uncertain — particularly during a Postgres restart or network partition.
• Deployment prerequisite. Fast adoption requires the native Postgres client (PRISM_POSTGRES_NATIVE_CLIENT=auto or 1). psql-only deployments log the downgrade and keep TTL fencing — correct, just not fast.

Commits

619a732	Fence and adopt stale PRISM writer sessions
5f07ce1	Release PRISM lease before watchdog hard exit
fcaa24b	Document PRISM watchdog exit status forensics
efe51c9	Harden PRISM lease adoption against live twins
4f695ad	Guard PRISM lease helpers without ledger
9f4c710	Prevent PRISM lease freshness regression

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

---

Note

High Risk
Touches security-critical writer-lease fencing, dual-writer prevention, and mutating block/CTV RPC paths. A fencing bug could allow concurrent writers or block external side effects incorrectly.

Overview
Cuts PRISM watchdog-restart downtime from ~lease TTL (60s+) to ~1s by releasing the writer lease on hard exit and allowing safe same-identity takeover at startup.

Watchdog lease release. All watchdog os._exit(1) paths now go through _watchdog_hard_exit, which best-effort releases the lease on a fresh daemon thread under a ~5s deadline, then exits unconditionally. Release uses a one-shot psql connection and the shutdown controller's writer barrier (not shutdown() / self.lock) so a wedged subsystem cannot block termination.

Fast same-identity adoption. Replacements can adopt a silent same-writer/same-epoch predecessor after acquiring a dedicated PostgreSQL advisory guard, observing ~1s of silence, and winning an exact-session CAS. Live twins keep the guard and cannot be fenced out. psql-only deployments downgrade to TTL fencing.

Heartbeat + external fences. A dedicated heartbeat/monitor keeps guarded sessions visibly live and hard-exits before silence can be observed. Mutating RPCs (submitblock, CTV wallet/broadcast calls) now do a bounded exact-session renewal first, and JsonRpc no longer transparently retries those methods.

Also documents watchdog exit-code forensics and the residual preflight window between Postgres verification and qbitd/wallet RPC.

^{Reviewed by Cursor Bugbot for commit 5ef0b15. Bugbot is set up for automated code reviews on this repo. Configure here.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9f4c710a0c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 9f4c710. Configure here.}