Stop processes and supervisors whose timers stop returning

[julik]

(Written by Claude on @julik's behalf)

Speculative, for #808. #778 stops a process whose heartbeats keep failing, but both of its paths hang off a raise, and a heartbeat can also never return at all: Process#heartbeat does a real round-trip, and on a half-open socket - a pooler that stopped serving while its kernel keeps the connection - the thread blocks in the driver indefinitely, so presumed_dead? is never reached. Concurrent::TimerTask does not help either, since it reschedules the task and notifies its observers only after the task returns; the heartbeat thread just goes quiet forever, having neither succeeded nor failed. timeout_interval is a no-op that warns it was never implementable, so there is nothing to configure.

Both commits have the same shape: record when the watched call last returned, in an ensure so success and failure both count, then watch that from a second TimerTask that reads nothing but memory and therefore cannot block the way the thing it watches can. Recording in the ensure is what keeps this orthogonal to #778 - that change owns "keeps failing", this one owns "stopped coming back", and they do not double-fire.

They differ in what they do about it, and the second is not a tidier version of the first:

• A process calls stop_to_be_replaced, as Stop processes whose heartbeats keep failing past the alive threshold #778 does.
• A supervisor just stops, because it cannot replace itself. It needs its own fix even though Processes::Base includes Registrable: stop_to_be_replaced only unregisters and wakes the run loop, and unlike Runnable#shutting_down?, Supervisor#supervise breaks on stopped? without ever checking registered?. Pruning blocking is also its own failure - it is how supervisors notice dead processes, so it fails at exactly the moment they do.

The supervisor threshold is MAINTENANCE_STALL_FACTOR (2) times the alive threshold rather than a bare comparison, because the maintenance task's own execution_interval is the alive threshold - one threshold of gap between healthy returns is normal, so a bare comparison would fire every cycle. Both tests redefine the blocking method by hand and restore it in ensure, since mocha has raises but no blocking equivalent.

What this does not cover, and the reason it is a draft: a process whose run loop is itself blocked will not exit even once unregistered, and in fork mode a child is only replaced when it exits. Closing that needs something harsher than stop - exit! after a grace period, roughly how Puma culls a hung worker - and I would rather hear what you think of the shape before writing it. Pruning deleting rows without signalling anything is related and also untouched here.

Ran under sqlite: both new tests fail without their lib change and pass with it, each commit is green standing alone, and async_supervisor_test + worker_test + process_recovery_test come out at 30 runs / 104 assertions / 0 failures.

Watchdogs watching watchdogs, but at least this one never talks to the database.