Conversation
Wallet transaction streams backed by external providers could hang indefinitely: page requests used the axios default of no timeout, so a stalled provider response never errored and never ended the stream, and wallet-scoped queries set no result limit, letting cursor pagination walk an address's entire history. Both were observed hanging ETH wallet scans in prod. Give every page request a 30s timeout (overridable via args.timeout), matching the point-lookup Moralis client. Cap pagination at 1000 pages when the caller provides no limit or paging bound, and log a warning when this safety net truncates a stream. Also stop issuing one extra page request after an explicit paging cap ended the stream (push(null) fell through without returning).
Clamp negative ages from future last-activity dates so clock skew or a bad provider timestamp counts a wallet as just-active instead of being misclassified by chance. Pin the inclusive window boundaries with tests.
Introduce the service shell that will drive weekly wallet-stats snapshots. The schedule math lives in pure, injectable-dep functions: snapshotDateIfDue resolves the current week's snapshot date and gates on a durable watermark, and missedSnapshotDates enumerates weeks skipped between the watermark and now for later backfill. tick and the collection run are stubbed for subsequent tasks.
Sum spendable balance per wallet in a single coins aggregation so a snapshot needn't query wallet-by-wallet. The match mirrors the unspent, valid-mint predicate CoinModel.getWalletBalance uses, and unwinding the wallets array credits coins shared across wallets to each holder. Balances are carried as BigInt to avoid float rounding on large sats.
Derive each wallet's last-activity date for the snapshot. Coins store block heights rather than times, so resolve since -> sinceHeight, take each wallet's max mint-or-spend height over coins active since then, and resolve those heights back to block times in one lookup. Unconfirmed activity sits at negative sentinel heights with no block time; those wallets are omitted so activity counts only once confirmed.
Roll the collected balances and activity into a snapshot document and its per-wallet fact rows with one pure pass. Counters accumulate as BigInt to stay exact on sats-scale sums. Duplicate wallets still yield a fact row for later inspection but are kept out of every counter so they can't inflate totals; the bitcore/imported split derives each wallet's creation time from its ObjectID against the snapshot date.
Hint the balance aggregation onto the partial wallets index the planner
would otherwise miss, as getBalanceAtTime does; the match carries the
index's partialFilterExpression so the hint is valid. Resolve activity
heights with a bounded height>=sinceHeight range scan instead of a
height:{$in:[...]} doc that could hold hundreds of thousands of values,
and skip the scan when nothing confirmed needs a date. Route the test
block stubs by query shape and pin the predicate, hint, and short-circuit
so a mangled query can't slip through unit tests.
EVM chains have no coin rows to aggregate, so read each wallet's balance and nonce from the chain-state provider directly. Balances are requested in hex and summed as BigInt so wei values above 2^53 keep full precision rather than round through a float. Activity is inferred: a balance or nonce change since the prior snapshot dates the wallet to asOf, an unchanged wallet carries its prior date forward, and a wallet that looks active but lacks a date (or has never been snapshotted) falls back to a 12-month token-transfer lookup. Provider calls retry through 429s with capped exponential backoff and bail out promptly on stop.
Duplicate detection is expensive and stable, so run it only the first time a wallet is seen: any wallet with a prior fact keeps that fact's isDup verdict and skips the address scan. Never-snapshotted wallets get the maintenance scripts' check, where a first address shared by more than one wallet flags the whole cluster. detectDups itself writes nothing; the verdict is persisted when buildSnapshot stamps isDup onto the facts, so the two stay in step without a second write path.
The prior-facts aggregate sorted by snapshotDate to take each wallet's latest verdict, but at production scale that $sort becomes a blocking in-memory sort over all historical facts and can exceed the 100MB aggregation sort limit. A duplicate verdict never flips once set, so $max over the boolean isDup returns the same latest answer without any sort stage.
Prepare the collection helpers for the weekly tick. Anchor missed-date enumeration to the schedule day by walking back from the current scheduled date instead of forward from the watermark, so an off-schedule watermark no longer drifts the reported gaps. Validate snapshotDayUTC and snapshotHourUTC when read, warning once and falling back to Monday 02:00 rather than letting a bad config skew or crash a run. Let buildSnapshot stamp EVM nonces so both chain types share one assembly path. Cap the rate-limit retry at a total elapsed time (default 10min) and re-check the stop flag after each backoff sleep, so one throttled wallet can't stall the tick and shutdown waits at most one interval.
Wire the collected pieces into a scheduler pass. tick walks the target chain/networks, and for each one whose weekly snapshot is due it lists the wallets, runs the UTXO aggregation path or the per-wallet EVM path by chain type, assembles the snapshot, records any skipped schedule dates in meta.gaps, bulk-upserts the facts, and upserts the snapshot last so re-runs stay idempotent. Resilience is built in: re-entrant ticks are dropped, the whole body is guarded so neither the interval nor --EXIT path leaks a rejection, a bad chain is isolated from the rest, per-wallet failures are counted and skipped rather than aborting the run, the EVM loop throttles and honors stop between wallets, and a stop mid-collection abandons the snapshot for that chain. EVM prior facts are read by snapshotDate equality to avoid a latest-per-wallet sort. The default token-activity probe is a lazy, swappable, limit-1 Moralis check that degrades to null on any error.
A configured non-UTXO, non-EVM chain (XRP, SOL) fell through to the EVM path, where every wallet errored on getAccountNonce and a zeroed snapshot with erroredWalletCnt=N still persisted and advanced the watermark every week. Skip such chains before any DB work, warning once per chain, so no junk snapshot is written. Also note the equality-read trade-off: a wallet that errored last run has no watermark fact, so its activity carry-forward is lost and re-derived next run.
The default token-activity probe lazy-imported a MoralisClient from an untracked clients/ directory (a WIP client extraction), so a fresh checkout of this branch failed to compile. Rebuild the probe on tracked surfaces only: a direct axios GET to Moralis with an explicit 30s timeout, the chain id formatted via the tracked adapters util, and the apiKey read from config (no key => no request). Behavior is unchanged (errors still degrade to null) and it's now self-contained enough to unit test. Migrate onto MoralisClient once that extraction lands.
Give the service a standalone worker mirroring the pruning worker and an npm script to run it. Unlike pruning this is a long-lived scheduler, so the script omits the --EXIT default (--EXIT stays available for cron-style one-shot runs). Register WalletStats in the clustered primary process alongside P2P; it self-gates on isDisabled and ships disabled, so it is inert until a deployment opts in.
Slow provider pages under load can legitimately exceed 30 seconds, which would surface as spurious stream errors. 90 seconds still bounds a stalled response while leaving headroom for large result pages.
The token-activity lookup only ran for wallets with a nonzero native balance or nonce, or on their first-ever snapshot. Wallets that only move ERC-20 tokens keep both native signals at zero, so one that began transferring tokens after its first snapshot could never be dated and stayed out of every activity window permanently. Probe any wallet with no known activity date instead; cost stays bounded at one capped lookup per undated wallet per run.
This was referenced Sep 17, 2026
Contributor
Author
|
Next phase preview: the authenticated read API for these snapshots is up as a stacked preview PR on my fork showing just its own commits leolambo#23. It'll open here against master once this merges. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a
WalletStatsServicethat records weekly wallet statistics per chain and network: wallet counts, balances, activity windows, and duplicate detection, stored in two new collections (walletstats,walletstatswallets). These numbers are currently produced by long-running operational scripts that iterate every wallet through the chain state providers; a full run takes days and the EVM pass is prone to hanging on stalled provider requests. With snapshots accumulating on a schedule, wallet statistics become a database query.Changelog
ExternalApiStreampage requests now carry a 90s timeout and a default page cap. A stalled or endlessly-paginating external provider surfaces as a stream error within the timeout. This applies to all wallet tx streaming, and fixes the hang the EVM stats script kept hitting in prod.WalletStatsServicesnapshots wallet counts, balances, activity windows (14d/30d/90d/6m/12m) and duplicate-wallet counts weekly. UTXO chains come from indexed aggregations overcoins. EVM chains use per-address balance and nonce reads plus one bounded, spam-filtered Moralis check that catches wallets whose only activity is receiving tokens.meta.gaps, re-runs upsert on a unique{chain, network, date}key, and chains that are neither UTXO nor EVM (XRP, SOL) are skipped with a warning.walletstatswalletsso cohort-style questions can be answered from stored data.npm run walletstats, with--EXITfor cron use) or inside the main process. Disabled by default; merging this changes nothing untilservices.walletStats.disabled: falseis set.Example config
snapshotDayUTC/snapshotHourUTCset the weekly collection time (default Monday 02:00 UTC).sleepMs/everythrottle the EVM per-wallet loop. Everything is optional exceptdisabled: false.Testing Notes
On a node with synced data (regtest works fine):
bitcore.config.jsonunderbitcoreNode.services:"walletStats": { "disabled": false }node build/src/workers/walletStats.js --CHAIN BTC --NETWORK regtest --EXIT truedb.walletstats.find().pretty()should show one snapshot for the current week with counts that match your wallet book, anddb.walletstatswalletsshould have one fact doc per wallet.With
walletStatsleft out of config (the default), nothing starts and there's no behavior change.Checklist
BWCif modifying the bitcore-wallet-client package,CLIif modifying the bitcore-cli package, etc.)