Submit found blocks to the node before accounting and keep the submitter alive under DB saturation (1.x.x)

[Anatolie]

Summary

Found blocks were being lost because the PRISM coordinator did durable Postgres accounting before offering the block to qbitd. When production Postgres saturated, the submitblock RPC sat behind a database write long enough for the tip to move, and the coordinator then abandoned a real block as stale-job. The same unbounded database wait wedged the block_submitter thread entirely, tripping the liveness watchdog and restarting the process.

This branch makes the node offer the fast lane — it happens before any accounting — bounds every blocking call the submitter can make, moves accounting onto its own thread so it can never convoy later block submissions, and adds per-phase heartbeats so a future watchdog firing names the exact call that stopped progressing.

Incident that motivated this (production, 2026-08-11)

• The pool found 5+ real blocks between 00:44 and 01:43 UTC.
• Three candidates were abandoned in two hours, e.g. 01:50:52 block candidate abandoned reason=stale-job: tip moved before submit — the tip had moved ~80s earlier, meaning the submit had been stuck in the pipeline that entire time.
• The submitter then stopped entirely: last heartbeat ~01:50:19, and at 01:55:18 the watchdog fired with unresponsive subsystems=['block_submitter'] timeout=300s, restarting the process and causing a paging outage.
• Postgres was saturated at the time (a separate fix); qbitd itself was healthy throughout — tips flowing, CreateNewBlock instant.
• The log gave no indication of which call had wedged for 300 seconds, only the thread name.

Root cause: _submit_next_block_candidate_writer called _mark_block_candidate_attempted (a durable Postgres write) before submit_block_candidate (the submitblock RPC), and nothing on that path — lock acquisition, Postgres statements, or the RPC — had a timeout. A delayed submit is a lost block; a duplicate submit is harmless, since the node rejects blocks it already knows.

What changed

1. Submit-first ordering (c18a661)

_submit_block_candidate_to_node now offers the durable candidate to qbitd before the attempt-marker write, writer admission, audit construction, or payout publication. The node result plus the same-hash disposition lease then transfer to the accounting tail.

Crash-point analysis for the new ordering (duplicate submits are acceptable, double accounting is not — everything is deduped by block hash):

Crash point	Durable state	Replay behaviour
Before submitblock	outbox row pending	Replay resubmits the same bytes; nothing was accounted.
After submitblock, before attempt marker	outbox row pending	Replay resubmits; qbitd's accepted-duplicate response is treated as a successful landing signal. Block-hash-keyed ledger persistence keeps accounting exactly once.
After attempt marker, before finalize	outbox row pending + attempt marker	The existing finalize-only retry registry completes the terminal update; the accounting tail is not re-run.
After finalize	submitted / abandoned	An exact resubmission observes the terminal state inside the same durable pre-submit transaction — submitted coalesces to success, abandoned stays rejected — before any new node offer is made.

The at-most-once terminal-accounting guarantee that the old mark-before-submit ordering provided is preserved via the finalize-retry registry rather than by delaying the submit. Exact share replays now return the original row as not newly inserted (BlockCandidateIntentPersistResult), so process-local worker and vardiff counters are not credited twice.

A short in-memory prospective-payout barrier is installed before qbitd can observe a candidate, so startup prewarm cannot issue child work from the stale balance base — without falsely claiming the block landed.

2. Bounded blocking calls and phase heartbeats (c18a661)

• Postgres: every statement the submitter touches (outbox read/replay, mark-attempted, finalize) gets a fresh deadline via new SingleWriterShareLedger.operation_timeout / statement_timeout scopes, which set both the client-side budget and SET LOCAL statement_timeout. Expiry raises LedgerOperationTimeout; the row stays pending and enters the ordinary candidate backoff. Direct outbox reads/mutations additionally go through a single-flight wrapper so a driver that ignores its deadline cannot accumulate retry threads.
• RPC: submitblock runs under an explicit hard deadline.
• Locks: bare with self.lock on this path is replaced by a bounded acquire-loop that heartbeats in slices and periodically logs which lock is contended and the current phase.
• Heartbeats: _record_block_submitter_phase tags liveness with the active phase — replay-outbox-query, submitblock-rpc, fast-lane-admission, accounting-queue, and so on. A watchdog firing now reports block_submitter:replay-outbox-query instead of just the thread name, which is exactly the diagnostic the 2026-08-11 log was missing.
• Poison detection: at most two timeout-ignoring RPC workers and two timeout-ignoring ledger workers may stay detached. If either bounded pool remains exhausted for PRISM_BLOCK_SUBMIT_STUCK_CALL_EXIT_SECONDS, the coordinator requests shutdown and exits nonzero so the supervisor replaces the poisoned process; durable outbox rows remain pending for replay.

3. Priority isolation: node offers vs. accounting (bf7bf61)

Adversarial review of the first commit found that a stalled accounting tail could still convoy later block submissions. Accounting now runs on its own block_accounting thread behind a height-prioritized queue:

• The submitter hands off (node result, lease) and immediately returns to the next candidate.
• A full primary handoff spills into a result-preserving overflow queue — an already-offered block is never converted back into a raw-submit retry. Once spillover starts, later handoffs queue behind it so older spill entries can't starve.
• block_submitter and block_accounting expose independent phase heartbeats, so slow accounting neither delays node offers nor disguises which phase stopped.
• Recovery restores pending outbox rows in batches into a separate, lower-priority replay queue with no per-row database accounting, so live discoveries always outrank restart work while an older stalled replay cannot hide newer durable rows.
• Fast-lane capacity reservations (_reserve_block_fast_lane_slot) keep PRISM_MAX_BLOCKS / stop-after-block semantics correct now that admission and accounting are decoupled.
• Abandonment is counted only after any prepared payout state is rejected and the false disposition is fixed; if cleanup fails the candidate stays pending and can still converge to submitted on later chain evidence.

4. Startup replay no longer crash-loops on a slow ledger (e082971)

Review caught a P0 regression introduced by the two commits above: routing the startup outbox replay through the new 1s database deadline meant a slow-but-healthy Postgres raised BlockSubmitterDatabaseTimeout / LedgerOperationTimeout out of serve() → main(), exiting nonzero ~1s after boot and crash-looping — in precisely the conditions this branch exists to survive, including after the branch's own fail-stop exits.

_run_startup_block_candidate_replay now catches TimeoutError (covering both subclasses) at the startup boundary, logs the phase and configured budget, and continues boot so listeners open and the submitter/accounting threads start. This is safe because block_submit_loop already calls replay_pending_block_candidates() every iteration with paced backoff, and every durable candidate remains pending in qbit_block_candidate_outbox; the pre-listener pass only gives durable candidates a head start over fresh miner wakeups.

Deliberate scope limits in that fix:

• Non-timeout database errors at startup replay stay fatal, matching base behaviour — a hard error is a real failure, not backpressure.
• ShutdownInProgress handling and the drain_threads clean-stop path are unchanged.
• replay_recovered_shares runs on the main thread with no timeout scope armed, so its behaviour is untouched.
• The single-flight wrapper deliberately leaves a timed-out call registered under its key, so the loop's next retry reuses the still-running worker; the startup fix preserves that reuse.

Configuration

New knobs, all with production-safe defaults wired through .env.example and compose.yaml:

Variable	Default	Purpose
PRISM_BLOCK_SUBMIT_RPC_TIMEOUT_SECONDS	1	Hard deadline on the fast-lane submitblock RPC.
PRISM_BLOCK_SUBMIT_DB_TIMEOUT_SECONDS	1	Fresh deadline for each submitter Postgres statement and local ledger gate.
PRISM_BLOCK_SUBMIT_LOCK_WAIT_LOG_SECONDS	5	Cadence for logging the contended lock and current phase.
PRISM_BLOCK_SUBMIT_STUCK_CALL_EXIT_SECONDS	30	Exit nonzero when a timeout-ignoring RPC/DB worker pool stays exhausted this long.

docs/prism-ledger-ops.md is updated to describe the fast lane, the accounting handoff, best-effort startup replay, the phase-tagged heartbeats, and the revised exactly-once accounting story.

Testing

python3 -m unittest tests.test_prism_coordinator_vardiff tests.test_prism_share_ledger — 487 tests, OK (run locally on this branch). The 5 pre-existing failures in tests.test_prism_coordinator_job_cache are unchanged from base 6c00d6e and are untouched here.

Coverage added for each acceptance property:

• test_submitter_offers_block_before_writer_admission_with_rpc_deadline — the RPC precedes writer admission and carries a deadline.
• test_sixty_second_attempt_mark_stall_does_not_delay_rpc_or_heartbeat — with a database that stalls 60s, the candidate still reaches the node RPC promptly and the submitter keeps heartbeating.
• test_crash_after_submit_before_attempt_mark_replays_duplicate_once — crash between submit and mark-attempted; replay resubmits and accounting lands exactly once.
• test_accounting_saturation_does_not_convoy_node_offers, test_replay_batch_reaches_node_while_oldest_accounting_stalls — accounting stalls do not block later node offers.
• test_overdue_submitter_heartbeat_names_the_stuck_phase — watchdog output includes the phase label.
• test_startup_block_replay_timeout_starts_submitter_and_converges, test_startup_block_replay_catches_psql_server_timeout, test_startup_block_replay_keeps_hard_database_errors_fatal, test_startup_block_replay_preserves_shutdown_stop — the boot regression and its intentional limits.
• test_stuck_rpc_worker_pool_requests_nonzero_restart, test_stuck_ledger_worker_pool_requests_nonzero_restart, test_one_stuck_ledger_call_does_not_restart_with_spare_capacity — poisoned-pool fail-stop only fires with no spare capacity.
• test_synchronous_waiter_joins_finalize_only_registry, test_restart_resubmit_coalesces_terminal_submitted_outbox, test_restart_resubmit_honors_terminal_abandoned_outbox, test_exact_share_replay_does_not_repeat_process_credit — exactly-once accounting across replay and same-hash races.
• Ledger-level: test_operation_timeout_bounds_local_writer_lock_admission, test_statement_timeout_refreshes_for_each_database_step, test_subprocess_operation_timeout_sets_client_and_server_deadlines, test_native_operation_timeout_is_transaction_local, and the matching server-deadline/hard-error cases.

Risks and rollout notes

• Duplicate submits are now expected on the crash/replay path. This is intended — qbitd rejects blocks it already knows, and accepted-duplicate is treated as a landing signal. Accounting is deduped by block hash.
• The 1s database default is aggressive by design. Under sustained saturation the submitter will time out and back off rather than block; rows stay pending and converge on retry. If a deployment has a genuinely slower ledger, raise PRISM_BLOCK_SUBMIT_DB_TIMEOUT_SECONDS — note this budget also covers TCP connect + auth on a cold connection.
• New fail-stop path: an exhausted, timeout-ignoring worker pool now exits nonzero on purpose. That requires a supervisor that restarts the process; durable rows are safe across that restart.
• The accounting overflow queue is unbounded process-local, bounded in practice by max-block admission on unresolved real offers.
• Postgres saturation itself is being addressed separately; this branch only makes the block-submit path immune to it.

🤖 Generated with Claude Code

---

Note

High Risk
Changes found-block submission ordering, crash/replay semantics, and fail-stop behavior on the critical path that lands pool blocks and drives payout accounting.

Overview
Found blocks are now offered to qbitd before any durable accounting, so a saturated Postgres can no longer delay submitblock long enough to lose the tip race or wedge block_submitter.

The node result hands off to an independent height-prioritized block_accounting lane; a full primary queue spills into a result-preserving overflow instead of turning an already-offered block back into a raw retry. Live discoveries outrank restart replay, and startup outbox recovery is best-effort so a slow ledger no longer crash-loops boot.

Every submit-path RPC, DB statement, and lock wait is freshly deadline-bounded, with phase-tagged heartbeats and a poisoned-worker fail-stop. New knobs (PRISM_BLOCK_SUBMIT_*) are wired through .env.example / compose.yaml, and docs/prism-ledger-ops.md documents the fast lane, accounting isolation, and revised exactly-once replay story.

^{Reviewed by Cursor Bugbot for commit 223d2a0. Bugbot is set up for automated code reviews on this repo. Configure here.}

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 5a563ca. Configure here.}