Skip to content

electrum: delta-driven subscription updates (O(touched) periodic ticks)#237

Draft
DeviaVir wants to merge 1 commit into
new-indexfrom
delta-driven-subscription-updates
Draft

electrum: delta-driven subscription updates (O(touched) periodic ticks)#237
DeviaVir wants to merge 1 commit into
new-indexfrom
delta-driven-subscription-updates

Conversation

@DeviaVir

Copy link
Copy Markdown

Problem

The 5s periodic subscription loop calls get_history for every subscribed scripthash on every tick — O(subscribed) even when nothing changed. On a production replica holding ~60k subscriptions the periodic_update histogram shows most ticks cheap but a fat tail of >250ms ticks, and the loop owns most of the replica's remaining steady-state CPU after the status-hash caching work.

Approach

The mempool already knows exactly which scripthashes each tx touches (tx_scripthashes, added with the eviction-ownership fix). This PR:

  • tracks a touched-epoch per scripthash on every mempool add/removal batch
  • each connection snapshots the epoch per tick and only re-checks subscriptions touched since its previous tick — O(touched), typically hundreds, not 60k

Correctness fallbacks (full scan preserved)

  • first tick of a connection (no snapshot yet)
  • any tick where the chain tip moved — the chain index has no touched-set (yet); blocks are rare relative to 5s ticks so this keeps today's behavior exactly where it's hardest to prove equivalence
  • any tick whose snapshot predates the tracking map's pruning floor (map bounded at 200k entries / ~720 epochs ≈ 1h)

Lock discipline: touched_since is &self on the mempool read guard — per-connection ticks never take the write lock; pruning happens inside add() which already holds it.

Status / next steps

Draft pending:

  • profiling pass on prod metrics to quantify the win (expectation: ~5-10× off the remaining subscription CPU, >250ms tail eliminated)
  • regression test: subscription notified on mempool touch + NOT notified/missed across epoch pruning and tip changes
  • consider a block-level touched-set as a follow-up so tip ticks also go delta-driven

Builds clean, 21/21 lib tests pass.

The periodic (5s) subscription update loop re-checked every subscribed
scripthash via get_history on every tick - O(subscribed) work per cycle
even when nothing changed. With ~60k subscriptions on a production
replica this kept the periodic_update latency histogram fat-tailed
(>250ms ticks) and burned steady CPU.

The mempool already records which scripthashes every tx touches
(tx_scripthashes, from the eviction-ownership fix). Track a touched-epoch
per scripthash on mempool add/remove; connections snapshot the epoch each
tick and only re-check subscriptions whose scripthash was actually
touched since their previous tick: O(touched) instead of O(subscribed).

Correctness fallbacks keep the full scan for: the first tick of a
connection, any tick where the chain tip moved (the chain index has no
equivalent touched-set, and blocks are rare relative to ticks), and any
tick whose epoch snapshot predates the tracking map's pruning floor
(bounded at 200k entries / ~720 epochs).

touched_since takes &self on the read guard so per-connection ticks never
contend on the mempool write lock; pruning happens inside add().
@DeviaVir DeviaVir self-assigned this Jul 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant