fix(p2p): gossip tx validation stalls behind tx pool finalization (A-1656) - #25148
Open
spalladino wants to merge 6 commits into
Open
fix(p2p): gossip tx validation stalls behind tx pool finalization (A-1656)#25148spalladino wants to merge 6 commits into
spalladino wants to merge 6 commits into
Conversation
Gossip tx validation waits on the tx pool serial queue (canAddPendingTx / addPendingTxs). handleFinalizedBlock occupied that queue for the entire finalization of an epoch's worth of mined txs (hydrate + deserialize + archive + delete), stalling validation for 10-40s on mainnet nodes. - Split finalization into chunk-sized serial-queue items so gossip pool operations interleave with finalization. - Archive raw proof-stripped buffers instead of deserializing and re-serializing every tx. - Add per-stage timing instrumentation to gossiped tx validation, queue wait/execution metrics to the tx pool serial queue, and IVC metrics to the peer BatchChonkVerifier. - Scrape and log the new metrics in the spartan TPS benchmarks.
… cursor leak Finalization now runs as chunk-sized serial-queue items (prepare, archive, delete, complete) so gossip-driven pool operations interleave instead of waiting for an entire epoch's worth of mined txs. The archive copies raw proof-stripped buffers instead of deserializing and re-serializing each tx. Also fixes a latent deadlock in TxArchive: getHeadIndex/getTailIndex called .next() once on entriesAsync and abandoned the generator. Inside a write transaction the committed-state iterator is unbounded, so the abandoned generator never sent CLOSE_CURSOR, permanently leaking one of the store's cursor semaphore slots per call and deadlocking the store after 8 archive transactions.
…6 repro Revert this commit before merging.
…hive - deleteFinalizedTxs re-checks each tx is still mined at or before the cutoff before deleting, since other pool operations may interleave between the finalization plan being computed and the delete chunk. - archiveTxBuffers skips txs already archived, so a retried or crashed finalization cannot append a duplicate FIFO index entry whose eviction would delete the stored value out from under the newer entry. - handleFinalizedBlock chains concurrent finalizations so their chunked queue items never interleave with each other.
PR #25032 added a top-level import of standard-contracts/artifacts-historical/HandshakeRegistry-5.0.1.json, but the release image dockerignore only whitelists artifacts/, so every node started from a v5-line release image crash-loops at ESM link time with ERR_MODULE_NOT_FOUND. This broke all spartan network deploys from v5-next since 2026-07-29.
…or A-1656 repro" This reverts commit 4815488.
spalladino
added a commit
that referenced
this pull request
Aug 10, 2026
…#25155) Since #25032, `@aztec/standard-contracts` has a top-level import of historical artifact JSONs from `standard-contracts/artifacts-historical/`. The release image dockerignore ignores everything and then whitelists specific paths, and it only whitelisted `standard-contracts/artifacts/` — so the historical artifacts never made it into the image. - Every node built from a v5-next release image crash-loops on startup at ESM link time with `ERR_MODULE_NOT_FOUND` for `HandshakeRegistry-5.0.1.json`. - This has broken all spartan deploys from this line since 2026-07-29. - The fix is a single whitelist entry: `!/yarn-project/standard-contracts/artifacts-historical/`. Split out from #25148 so the deploy fix can land independently of that PR's review. Once this merges, #25148 will be rebased and its duplicate commit will drop out.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes A-1656 (mainnet nodes logging `Gossip validation for tx took 20000+ms, approaching mcache eviction window of 8400ms`).
Root cause
Gossip tx validation waits on the tx pool's single serial queue (`canAddPendingTx` / `addPendingTxs`). `handleFinalizedBlock` occupied that queue as one item for the entire finalization of an epoch's worth of mined txs — hydrating each tx from LMDB, deserializing it, re-serializing it into the archive, and deleting it. Mainnet log analysis shows the slow-validation spikes cluster exactly on the once-per-epoch finalization tick (every 38.4 min, ~100–200 txs per tick); the worst spike (43s) coincides with the snapshot pause of archiver/world-state sync.
Reproduced in a unit test (
tx_pool_v2_finalization.test.ts): finalizing 128 mined txs of ~1.1MB each stalls a concurrentcanAddPendingTxfor 3.9s on a local NVMe machine with archiving enabled — on 0.5–2 vCPU cloud nodes with network disks the observed 10–40s is consistent.Chonk verification is ruled out: the batch verifier shows no pile-up (queue wait avg 6ms, max 172ms on next-net), and a
prove-n-tps-realbench run from this PR pushed 891 real-proven txs at ~1 TPS through real node-side chonk verification with zero slow-validation warnings (tx validation ~170–440ms throughout).Fixes
handleFinalizedBlocknow runs as chunk-sized serial-queue items (prepare → archive per chunk → delete per chunk → complete), so gossip pool operations interleave with finalization. Post-fix, the repro scenario stallscanAddPendingTxfor 1ms (was 3.9s). Hardened per review: delete chunks re-check each tx is still mined at or before the cutoff, archiving is idempotent (a retried finalization can no longer create a duplicate FIFO index entry whose eviction corrupts the newer one), and concurrent finalizations are chained.Tx.fromBuffer→withoutProof()→toBuffer()per tx. Total finalization time drops 3.9s → 395ms in the repro.TxArchive.getHeadIndex/getTailIndexcalled.next()once onentriesAsyncand abandoned the generator. Inside a write transaction the committed-state iterator is unbounded (WriteTransaction#iteratedoes not pass the limit down), so the abandoned generator never sentCLOSE_CURSOR, permanently leaking one of the store'smaxReaders - 1cursor-semaphore slots per call — deadlocking the whole store (and with it the pool serial queue) after 8 archive transactions. Now consumed viafor await.standard-contracts/artifacts-historical/was missing from the release image dockerignore whitelist, so every node started from a v5-line release image since fix(pxe): support contracts built with the v5.0.1 handshake registry #25032 (Jul 29) crash-looped withERR_MODULE_NOT_FOUND— this broke all spartan network deploys from v5-next.Instrumentation
aztec.p2p.gossip.tx_validation_stage_duration.aztec.mempool.tx_pool_v2.queue_*).BatchChonkVerifiernow records the standard IVC verifier metrics (it previously recorded none).n_tps,n_tps_prove) scrape and log all of the above at the end of a run.Network-level validation (
ci-network-benchruns on this PR)With the fix, finalization queue items execute in ~10ms and no slow validations cluster on finalize ticks;
canAddPendingTx/addPendingTxswait ~10–19ms P95 across all TPS scenarios. The remaining serializer surfaced by the new metrics ishandlePrunedBlocks(~57s on the queue during a reorg, 24 slow validations) — tracked separately as A-1668.