Skip to content

fix: keep proven headers in one archive and let the proof chains recover from bad batches - #536

Open
KSlashh wants to merge 3 commits into
devfrom
audit/16-1-proof-chain-recovery
Open

KSlashh wants to merge 3 commits into
devfrom
audit/16-1-proof-chain-recovery

Conversation

@KSlashh

@KSlashh KSlashh commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator

Audit items M33, M37, M38, with L56 and I08. Open against dev.

One header archive (M33). Every header-chain proof was published with a .blocks sidecar holding every header from Genesis to its last block, and the sidecars were never removed. The chain gets one proof per Bitcoin block, and on mainnet each sidecar is about 74 MB, so the disk grew by about 10 GB a day. Once it was full the header chain stopped, and the operator and watchtower proofs with it.

Proven headers are now kept once, in headers.blocks next to the header-chain proofs:

  • Saving a proof: the batch's headers are appended to the archive and synced before the proof is published. The append checks that the archive and the batch agree where they overlap, and that each new header extends the previous one. A tail torn by a crash is dropped first.
  • Reading: operator and watchtower proofs read only the headers the selected proof covers. The count and the last block hash come from the proof's own public values, and the last archived header must be that block. The Genesis seed proof still reads its .blocks sidecar.
  • Old sidecars: on the first save after a start, every old .blocks sidecar in the directory whose last header matches the archive at the same height is deleted. 0-* seed sidecars are kept, and nothing else ending in .blocks is touched.
  • Upgrade: a node needs no separate step. The first proof saved after the upgrade fills the archive from the header cache, which already holds every header from Genesis.
  • The helpers live in proof_builder::headers, a host-only crate. The guest crates are untouched, so the program ids do not change.

Header continuity (L56). A newly fetched header was not checked against the one before it. Now:

  • If a header does not extend a header fetched in the same batch (a reorg inside the batch), the batch's headers are dropped and fetched again, at most 3 times.
  • If it does not extend the last proven header, the fetch fails. The failed headers are removed from the cache, which is left at the batch start.

Commit-chain restart (M37). With init_input = true, which is what the example config sets, every restart proved the commit chain from Genesis again and deleted every stored commit-chain proof. The loop now resumes from stored progress first, as the header chain already did. init_input only applies while nothing is stored. Restarting the chain on purpose now requires clearing its rows.

State-chain batch reset (M38). Before, a batch whose proof failed fatally was retried forever with the same frozen .args/.ctx snapshot. Wrong block data from the configured Goat RPC therefore stopped the state chain for good, and once the backlog filled up, operator tasks needing a higher state-chain height could not be prepared. Now:

  • After 3 fatal failures, the batch and every later unproven batch are deleted along with their snapshots. The ctx builder then prepares them again from the last proven batch with fresh data. The reset and the ctx builder take the same lock, so no batch row is created in between and no gap is left.
  • After 3 resets, the prover stops retrying the batch and only alerts until it is restarted. It does not exit, because an exit restarts it in Fleet mode and the retries would never end.
  • Resets and the stall are counted in proof_attempts{type="state-chain"} with outcome="batch_reset" and outcome="stalled".
  • Transient and prover-unavailable failures do not count toward the 3.

I08. Not changed. The stale migration names are only in a comment of an applied migration. Editing that file changes its checksum, which sqlx refuses on every existing database.

Review notes

  • headers.blocks must be in the header-chain proof directory. The shipped configs already keep each chain in its own directory.
  • The header cache (block_headers) still exists. The archive differs from it in holding only proven headers, so a proof's prefix never changes.

Tests: the archive grows by batch, survives a torn tail and refuses a changed or broken chain; a proof reads its own prefix or its seed sidecar and refuses another chain's archive; only covered sidecars are deleted; a header that does not extend the chain is refetched inside the batch and refused at the batch start; a state-chain reset drops the unproven batches and their snapshots and the builder restarts after the last proven batch; the store's reset delete only touches unproven rows of its own chain.

The test coverage and validation notes above are from the zkm PR notes. Branch scope and diff formatting were checked locally; builds and tests were not rerun during publication.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant