electrum: delta-driven subscription updates (O(touched) periodic ticks)#237
Draft
DeviaVir wants to merge 1 commit into
Draft
electrum: delta-driven subscription updates (O(touched) periodic ticks)#237DeviaVir wants to merge 1 commit into
DeviaVir wants to merge 1 commit into
Conversation
The periodic (5s) subscription update loop re-checked every subscribed scripthash via get_history on every tick - O(subscribed) work per cycle even when nothing changed. With ~60k subscriptions on a production replica this kept the periodic_update latency histogram fat-tailed (>250ms ticks) and burned steady CPU. The mempool already records which scripthashes every tx touches (tx_scripthashes, from the eviction-ownership fix). Track a touched-epoch per scripthash on mempool add/remove; connections snapshot the epoch each tick and only re-check subscriptions whose scripthash was actually touched since their previous tick: O(touched) instead of O(subscribed). Correctness fallbacks keep the full scan for: the first tick of a connection, any tick where the chain tip moved (the chain index has no equivalent touched-set, and blocks are rare relative to ticks), and any tick whose epoch snapshot predates the tracking map's pruning floor (bounded at 200k entries / ~720 epochs). touched_since takes &self on the read guard so per-connection ticks never contend on the mempool write lock; pruning happens inside add().
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The 5s periodic subscription loop calls
get_historyfor every subscribed scripthash on every tick — O(subscribed) even when nothing changed. On a production replica holding ~60k subscriptions theperiodic_updatehistogram shows most ticks cheap but a fat tail of >250ms ticks, and the loop owns most of the replica's remaining steady-state CPU after the status-hash caching work.Approach
The mempool already knows exactly which scripthashes each tx touches (
tx_scripthashes, added with the eviction-ownership fix). This PR:Correctness fallbacks (full scan preserved)
Lock discipline:
touched_sinceis&selfon the mempool read guard — per-connection ticks never take the write lock; pruning happens insideadd()which already holds it.Status / next steps
Draft pending:
Builds clean, 21/21 lib tests pass.