Stop discarding accepted shares when the DB is momentarily locked

[rsantacroce]

The bug

writer_main() dequeues a batch and then opens the transaction:

s->ring_count -= take;              // events are out of the ring
...
if (sqlite3_exec(s->db, "BEGIN IMMEDIATE", ...) != SQLITE_OK) {
    LOG_ERROR("store: BEGIN failed: %s", ...);
    atomic_fetch_add(&s->pg_errors, 1);   // ...and that is all
}

On a failed BEGIN the events are gone and are never retried. Shares the miner was already told were accepted never reach the ledger.

Compounding it, the connection set no busy_timeout, so SQLite returned SQLITE_BUSY the instant any other connection held the write lock — no waiting at all. One concurrent writer was enough to trigger it: a manual sqlite3 session, a backup, a maintenance script.

How it surfaced

Running a one-off maintenance script against the live forknet DB, mid-transaction:

10:23:24 ERROR store: BEGIN failed: database is locked
10:23:24 ERROR store: BEGIN failed: database is locked

Two batches at ~18 shares/sec over a 100 ms window — roughly 3–4 shares, ~250k sats of miner credit, gone. Small, but it was silent, it favoured the pool over the miners, and nothing in the ledger audits would ever have shown it: the shares were never written, so every consistency check still passes.

The fix

• PRAGMA busy_timeout = 5000 — wait for the lock instead of failing instantly. This alone would have made the incident a non-event.
• Retry a failed batch up to 3 times with linear backoff (25/50/75 ms) in a new commit_batch(). Counters advance only on the attempt that commits, so a retried batch is counted once. After a failed COMMIT the transaction is rolled back, so replaying the batch is safe.
• When every attempt fails the batch is still lost — nothing can put it back in the ring — but it now says so in those terms and lands in a dedicated events_lost counter rather than a generic error tally.

events_lost must be 0. It means accepted work that will never be credited, which is a different failure from shares_dropped (enqueue-side overflow, already counted and visible).

Worst case is now a longer stall rather than lost data: 3 attempts × 5 s. That is the right trade — overflow during a stall is counted in shares_dropped, whereas a dropped batch here is credited work vanishing.

Test

test_commit_survives_a_locked_db takes the write lock on a second connection, holds it across several commit windows while 40 shares are enqueued, then releases. Asserts all 40 land and events_lost == 0.

Verified it catches the original bug — with busy_timeout removed and attempts forced to 1:

Assertion failed: (st.events_lost == 0), function test_commit_survives_a_locked_db

make clean under -Werror; full C suite and all 47 dashboard tests pass.