Add hard-failure health checks with a dashboard banner

[rsantacroce]

Why

Every problem this pool has had was found by someone deciding to run a query — duplicate shares, a fee that never reached the accrual path, an audit that compared two numbers the proxy itself wrote, templates carrying no sidechain commitments, and most recently accepted shares destroyed by a momentarily locked DB. None of them announced themselves.

What

Seven checks, each binary and actionable, on every page as a banner and at GET /health (503 when failing, so an uptime checker can watch it):

check	catches
events_lost	accepted work that never reached the DB
duplicate_shares	the extranonce1 collision class
ledger	credit ≠ difficulty × rate_used, rate not derivable from its template, credit at an unpublished rate
margin	owed more than mined
payout_ambiguous	in-flight rows with no txid — the one state needing a human
payout_stalled	a batch unconfirmed for over an hour
template_commitments	blocks valid and miners paid, but no sidechain can merge-mine

Deliberately excluded: BMM capture, settlement latency, backlog size, reserve headroom. Worth watching, but they fluctuate — a banner that goes red for a slow afternoon is a banner nobody reads. A payout settling normally has an explicit test asserting it stays quiet.

Two things that shaped the design

events_lost had nowhere to live. It was process-local and readable only at shutdown, so no query could see it. pool_meta now mirrors it, written on the template path — a different connection state from the batch commit that failed.

The duplicate check is slow. Measured against the live DB:

COUNT(block_hash) - COUNT(DISTINCT block_hash)   2.04s   (337k shares)
ledger arithmetic                                0.06s
margin                                           0.04s

better-sqlite3 is synchronous, so running that per request with a 15s auto-refresh would stall the dashboard. It runs on a 5-minute timer and pages read the snapshot. A windowed check would have been cheap, but it would go green on a duplicate from two days ago that nobody noticed — the timer keeps full-history coverage.

Not-a-pass semantics

A check that cannot run reports unavailable, never ok, so a DB predating a table doesn't read as healthy. Before the first pass, /health returns 503 checking rather than a green light.

Tests

18 new tests. Each failure mode gets a DB built in that state; healthy and normal-operation cases assert silence. Includes a regression test for a bug the check would otherwise have had — COUNT(*) counts NULL hashes that COUNT(DISTINCT ...) ignores, which would report every legacy row as a duplicate.

64/64 dashboard tests pass; make clean under -Werror; C suite passes.