From a325edd3849aa66aebd0a0d78d826d5fcc4b5c3f Mon Sep 17 00:00:00 2001 From: eigmax Date: Sat, 7 Nov 2026 04:19:00 +0000 Subject: [PATCH 1/2] docs: remove the node README RPC API section --- node/README.md | 33 --------------------------------- 1 file changed, 33 deletions(-) diff --git a/node/README.md b/node/README.md index 4353b452..88be4160 100644 --- a/node/README.md +++ b/node/README.md @@ -10,7 +10,6 @@ transaction graph setup and signing, P2P communication, and proof coordination. - [Execution and storage](#execution-and-storage) - [Build & Run](#build--run) - [Configuration](#configuration) -- [RPC API](#rpc-api) - [Related documentation](#related-documentation) ## Bitcoin transaction graph @@ -195,38 +194,6 @@ export GOAT_PRIVATE_KEY='your-private-key' # ... other required variables ``` -## RPC API - -Prometheus metrics are not served by the business RPC listener. Set -`--metrics-addr` to a private, process-unique address and scrape -`http:///metrics`; `GET /metrics` on `--rpc-addr` returns `404`. - -### Endpoints - -| Endpoint | Method | Description | -|----------|--------|-------------| -| `/v1/health` | GET | Readiness and health of this node (a local database ping, nothing else is called): readiness bits, each backend's availability with its last error, event watcher state and lag, SPV lag, uptime; `200` when ready, `503` otherwise | -| `/v1/nodes`, `/v1/nodes/:id`, `/v1/nodes/overview` | GET | Known nodes, one node, node statistics | -| `/v1/instances`, `/v1/instances/:id`, `/v1/instances/overview`, `/v1/instances/settings` | GET | Peg-in instances, one instance, instance statistics, peg-in settings | -| `/v1/instances/:id/unsigned-pegin-txn` | GET | Unsigned Peg-In-Prepare transaction of an instance | -| `/v1/graphs`, `/v1/graphs/:id` | GET | List all graphs, get graph details | -| `/v1/graphs/:id/txn?cursor=0` | GET | Get graph transaction list | -| `/v1/graphs/:id/tx?tx_name=cur-pre-kickoff.hex` | GET | Get specific transaction hex | -| `/v1/graphs/:id/neighbor-ids` | GET | Neighbouring graphs of the same instance | -| `/v1/graphs/ready-to-kickoff` | GET | Get graphs ready for kickoff | -| `/v1/graphs/pegout` | POST | Initiate a peg-out (signed request) | -| `/v1/swaps`, `/v1/swaps/:escrow_hash` | GET | Swap bridge-out escrows, one escrow | -| `/v1/proofs/chain_proofs_desc`, `/v1/proofs/operator_proofs_desc` | GET | Proof descriptors, forwarded to `GOAT_PROOF_BUILD_URL` | -| `/metrics` | GET | Prometheus metrics (dedicated metrics listener only) | -| `/v1/debug/status`, `/v1/debug/engine` | GET | Node status and the execution engine snapshot (rows by state, waiting and poisoned aggregates); `rpc-debug-endpoints` feature only | -| `/v1/debug/engine/clear-poison`, `/v1/graphs/:id/send-challenge`, `/v1/graphs/:id/send-verifier-challenge` | POST | Release poisoned actions, send a challenge / verifier challenge (signed requests); `rpc-debug-endpoints` feature only | - -### Operations - -Use `/v1/health` for readiness and backend status, and the dedicated metrics -listener for Prometheus. The [operator tools](../scripts/ops/README.md) provide -health triage, structured log queries, and instance or graph timelines. - ## Related documentation - [Project README](../README.md) — project layout and transaction graph checks From ba587e5308da479d4274fbd2bdc0aca8ad1e91a0 Mon Sep 17 00:00:00 2001 From: eigmax Date: Sat, 7 Nov 2026 08:28:09 +0000 Subject: [PATCH 2/2] docs: keep project READMEs focused on protocol and usage --- README.md | 86 ++---- circuits/README.md | 283 +++--------------- circuits/crates/state-chain/samples/README.md | 17 +- deployment/README.md | 115 ++----- localnet/README.md | 170 ++--------- localnet/e2e/README.md | 94 +++--- node/README.md | 198 ++---------- proof-builder-rpc/README.md | 211 +++---------- scripts/ops/README.md | 72 ++--- 9 files changed, 256 insertions(+), 990 deletions(-) diff --git a/README.md b/README.md index 59fb0710..3aa2ac7f 100644 --- a/README.md +++ b/README.md @@ -1,81 +1,47 @@ # GOAT BitVM Node -GOAT Network's BitVM bridge implementation. See [GOAT BitVM Whitepaper](https://www.goat.network/bitvm2-whitepaper) for more details. - -## Layout - -- `node/`: Main node implementation, including P2P, RPC, and scheduled tasks -- `circuits/`: Circuits and proof generation logic -- `proof-builder-rpc/`: RPC server for offloading proof generation to a separate process -- `crates/`: Shared Rust crates for common types and utilities -- `deployment`: Deployment scripts and documentation -- `scripts/ops/`: Read-only operator helpers for a running node (health triage, log queries, instance / graph traces) -- `localnet/`: clone-and-run mock testnet (five `mock-gc` nodes on regtest + anvil + graph-node) and the on-demand e2e suite, see [localnet/README.md](localnet/README.md); `make localnet-reset`, `make e2e-smoke` - +GOAT Network's BitVM bridge implementation connects Bitcoin and GOAT L2 through +presigned transaction graphs, optimistic verification, and proof-based disputes. +See the [GOAT BitVM Whitepaper](https://www.goat.network/bitvm2-whitepaper) for +protocol background. ## Transaction graph [![BitVM Bitcoin transaction graph with per-input sighash labels](transaction-graph.svg)](transaction-graph.svg) -Open the [full-size transaction graph](transaction-graph.svg) to inspect the -connectors, competing spends, CSV paths, and per-input signature types. Blue -paths show operator and claim transactions; red paths show challenges and -timeouts. Dots mark connected branches; crossings without dots are not connected. +Open the [full-size graph](transaction-graph.svg) for peg-in, Operator claims, +Watchtower and Verifier challenges, timeouts, and PreKickoff paths. Blue lines +show Operator and claim transactions; red lines show challenges and timeouts. +Dots join branches; crossings without dots do not. -The graph includes peg-in funding and refunds, the Take-1 and Take-2 claim -paths, Watchtower and Verifier branches, and PreKickoff slots. See the -[node transaction guide](node/README.md#bitcoin-transaction-graph) for the -connector and signature notation. +## Guides -## Transaction graph checks +- [Node](node/README.md) — roles and execution +- [Circuits](circuits/README.md) — proof composition and compatibility +- [Proof Builder](proof-builder-rpc/README.md) — proof coordination and recovery +- [Deployment](deployment/README.md) — running bridge participants +- [Localnet](localnet/README.md) — local testing +- [End-to-end checks](localnet/e2e/README.md) — protocol and fault scenarios +- [Operator tools](scripts/ops/README.md) — read-only diagnosis -The checks cover critical connector relationships, default network timing, and -core dispute safety for Bitcoin, Testnet4, Signet and Regtest: - -- Rust tests construct the real transaction graph and check parent outpoints, - competing spends, and the CSV values in transaction inputs and scripts. -- `audit/tla/TransactionGraph.tla` checks the Take-1 budget, ACK budget, - timeout order, conditional Disprove maturity, and Take-2's two CSV clocks. -- `audit/tla/BridgeSafety.tla` checks actual dispute transitions against two - requirements: an invalid proof must never reach Take-2, and Take-2 must have - committed pubin. It models the current Commit-Timeout and verifier gates, - including an operator withholding evidence and a possibly offline watchtower. - An honest verifier's enabled defences have an explicit bounded response time. -- `crates/bitvm-gc/tests/tla_shipped_values.rs` checks that the TLA values match Rust. A timing - change requires regenerating the values and rerunning the checks. -- Every other `audit/tla/.tla` with a `.cfg` is checked as well; - its header states what it models and which code it follows. +## Transaction graph checks -The timing model is finite, init-only arithmetic. Its Disprove comparison assumes -VerifierAssert confirms within `action_window` of OperatorAssert; earlier maturity -does not guarantee earlier confirmation. The safety model explores transitions -after WCI confirms, with OperatorAssert confirming in the same or any later -block (including after Connector-F has matured), abstracts proof validity -as a boolean, and assumes published pubin is consistent with the Assert. It does -not assume pubin will be published or a watchtower branch will resolve. -These checks do not establish end-to-end bridge safety or liveness and omit -cryptographic verification, fees, relay policy, reorgs, executor capacity and -contract settlement. -Implementation changes outside the tested structure and generated values require -manual review of the model's assumptions. +Rust tests check connector wiring, competing spends, and CSV values. TLA+ models +check timing and core dispute safety, including invalid assertions, unpublished +public inputs, and late assertions after Connector F has matured. ```bash -# JRE 11+ is needed for TLC. make tla-setup cargo test -p bitvm-gc --test transaction_graph_invariants --test tla_shipped_values make tla -# After changing Rust's timing constants: -make tla-values -make tla ``` -Every TLC failure fails CI. There are no expected-failure configurations or -stored traces. Temporary TLC files are removed on exit. `TLA2TOOLS` overrides -the tool jar path; `TLC_OPTS` supplies optional whitespace-separated TLC flags. -Commit-Timeout no longer waits for unresolved watchtower branches, and an -invalid Assert is challenged without waiting for Connector E, so both safety -requirements hold. They must not be weakened to make CI green. +After changing timing constants, run `make tla-values` before checking the models. +Every model-checking failure fails CI. These checks rely on bounded response +assumptions and do not establish end-to-end bridge safety or liveness; fees, +relay policy, reorgs, cryptography, and contract settlement need separate review. ## Contributing -Contributions are welcome! Please open an issue or submit a pull request for any improvements or bug fixes. +Open an issue or pull request with a concrete problem, proposed change, and +relevant validation. diff --git a/circuits/README.md b/circuits/README.md index 84bbc0de..b527c99a 100644 --- a/circuits/README.md +++ b/circuits/README.md @@ -1,262 +1,53 @@ # BitVM Circuits -## Overview +The proof system connects Bitcoin chain evidence, sequencer commitments, and +GOAT execution to the bridge's claims and disputes. The initial trust anchor is +a GOAT snapshot containing a sequencer set and an EVM block hash. -The initial trust anchor is a GOAT snapshot containing the sequencer set and EVM block hash. -For Bitcoin transaction wiring and signature types, see the [transaction graph](../transaction-graph.svg). +See the [transaction graph](../transaction-graph.svg) for where proofs and +public-input commitments enter the Bitcoin protocol. -Main components: +## Proof composition -* BTC Header Chain: the bitcoin header chain -* Commit Chain: the sequencer set commitment, where each commitment creates a block -* State Chain: the EVM state transition, each block contains a EVM block, the withdrawals and a cosmos light block +| Proof | Purpose | +|-------|---------| +| Header chain | Establish Bitcoin headers and accumulated work | +| Commit chain | Establish sequencer-set commitments and publisher transitions | +| State chain | Verify GOAT execution, withdrawal state, and sequencer agreement | +| Watchtower | Authenticate a chain claim using the three chain proofs | +| Operator | Verify the withdrawal claim and evaluate indexed Watchtower evidence | -Circuits: +A Watchtower proof binds its sequencer commitment to Bitcoin inclusion and the +corresponding sequencer and state history. The Operator proof checks the graph's +Watchtower evidence and withdrawal state. For a valid claim from the same +genesis, the Operator's chain must have at least the claimed work and commitment +inclusion height. -* Watchtower proof(latest_sequencer_commit_txid, genesis_sequencer_commit_txid, header_chain_proof, commit_chain_proof, state_chain_proof, SPV) +The Operator proof is used by the assertion flow. Its static public input is +bound during graph setup; its dynamic public input is committed after the +Watchtower challenge set is known. -> * Constraints: ->> ->> * The latest_sequencer_commit_txid is included in the header chain proof ->> * The latest_sequencer_commit_txid is the latest commit txn in the commit chain ->> * The state chain's latest block(EVM block) is signed by the sequencer set in the latest block of commit chain ->> -> * Outputs: total_work, the bitcoin block height that includes the latest sequencer set commitment +## Build and inspect -* Operator proof: - -> * Constraints: - ->> * The latest_sequencer_commit_txid is included in the header chain proof ->> * The latest_sequencer_commit_txid is the latest commit txn in the commit chain ->> * The state chain's latest block(EVM block) is signed by the sequencer set in the latest block of commit chain ->> * For each included Watchtower proof, - ->>> * The watchtower challenge transaction is valid ->>> * The proof is valid ->>> * The graph is as same as the operator's graph id ->>> * For claims from the same genesis, the operator's total work is at least the Watchtower's ->>> * The operator's commitment inclusion height is at least the watchtower's for a claim from the same genesis - ->> * Verify that the withdrawal state change in the Gateway contract is correct - -> * Outputs: latest block hash of the header chain, hash(graph_id, operator's genesis sequencer commit transaction id, included_watchtowers) - -## Circuit Upgrade - -Given that the interval between Peg-in and Peg-out may extend over several months, two types of upgrades require separate consideration: - -1. Proof system upgrades (Ziren): Result in a different recursion verification key. - -2. Modifications of guest program: Arise from protocol changes in Bitcoin, Cosmos, or Geth, result in different ELF and guest verification key. - -For case 1, Ziren's proof network can keep multiple recursion verification keys, and load the correct key with respect to the version in the proof. - -For case 2, - -> * If there is no Ziren upgrade during this interval, and the inputs of the proof aggregation are compressed proofs, but with different `start_pc`, `pc`, etc. -> * If there is some Ziren upgrades during this interval, once we support the multiple verification keys in Ziren, this problem can be reduced to the former one. - -With multiple proof recursions, we generate a Groth16 proof, and verify with `Groth16Verifier::verify_by_imm_groth16_vk(proof, zkm_public_values, zkm_vk_hash, groth16_vk, part_stark_vk)`. - -Ziren 2.0 (circuit version `v2.0.0`) changed the wrap circuit itself: its public inputs are -`[vkey_hash, committed_values_digest, vk_root]`, where `vk_root` is the root of the recursion verifying-key -allowlist, so the immutable Groth16 vk differs from the 1.2.x one. A 1.2.x Groth16 proof does not verify -under 2.0, so the header, commit and state chain proofs restart from genesis at the 2.0 cut-over. The -BABE garbled-circuit verifier treats `vkey_hash` and `vk_root` as static inputs and `committed_values_digest` -as the dynamic one. - -## Build inputs - -Build all five guest programs with the same `BITCOIN_NETWORK` and -`ZKM_IMM_WRAP_VK` settings used by the nodes and Proof Builder. Program or proof -system changes require matching verification keys and proof artifacts. - -## Preparation - -Run the proof examples from `circuits/`. The host binaries load `.env` there; -export the same configuration before running the helper scripts. - -```bash -mkdir -p data/header-chain -mkdir -p data/commit-chain -mkdir -p data/state-chain -mkdir -p data/watchtower -mkdir -p data/operator -``` - -if `Network Prover` is used, see [this](https://docs.zkm.io/dev/prover.html#network-prover) for more details. - -Launch the Regtest. - -```bash -(cd ../scripts && docker compose up -d) -``` - -## Bitcoin Header Chain - -``` -# regtest -# export BITCOIN_NETWORK=regtest - -# testnet -# export BITCOIN_NETWORK=testnet4 -# export ESPLORA_URL=https://mempool.space/testnet4/api - - -bash cron-header-chain-proof.sh $start $batch -``` - -## Sequencer Set Commit Chain - -* Publish sequencer set commitment with correct envs `BITCOIN_NETWORK` and `ESPLORA_URL`. - -Example: - -```bash -cd ../node -GOAT_BLOCK_NUMBER=9511050 bash -x ssp-ci.sh -cd ../circuits -``` - -This publication script is a Regtest fixture with hardcoded publishers. In the `ssp-ci.sh`, we simulate 2-round publisher rotations. -`GOAT_BLOCK_NUMBER` is the GOAT's current block number, which is used as the key to fetch sequencer set commitment. - -Deploy the SequencerSetPublisher contract before publishing the sequencer set. - -Then setup the correct contract address in your `.env`. - -``` -GOAT_SEQUENCER_SET_PUBLISHER_CONTRACT_ADDRESS=0x... -ENV_GOAT_SEQUENCER_SET_MULTI_SIG_VERIFIER_ADDRESS=0x... -``` - -After publishing, a `commit_info.json` of format as below will be generated. - -``` -[ - { - "txid": "dcadccc909994689e9f3a36c9d349e89f0cb96764f6d8f4d9632e0f76b0ec84e", - "threshold": 4, - "publisher_public_keys": [ - "031b84c5567b126440995d3ed5aaba0565d71e1834604819ff9c17f5e9d5dd078f", - "024d4b6cd1361032ca9bd2aeb9d900aa4d45d9ead80ac9423374c451a7254d0766", - "02531fe6068134503d2723133227c867ac8fa6c83c537e9a44c3c5bdbdcb1fe337", - "03462779ad4aad39514614751a71085f2f10e1c7a593e4e030efb5b8721ce55b0b", - "0362c0a046dacce86ddd0343c6d3c7c79c2208ba0d9c9cf24a6d046d21d21f90f7" - ], - "sequencers": [...] - } -] -``` - -* txid: the publisher's commitment transaction of Cosmos sequencer set. -* threshold: the number of publisher's signature -* publisher_public_keys: the publisher's compressed public keys -* sequencers: sequencer's public keys, obtained from cosmos's `/validators`. - -Generate the proof: - -``` -# Genesis -RUST_LOG=info cargo run --package commit-chain-proof --bin commit-chain-proof -r -- --init-input --output-proof "data/commit-chain/0-1.bin" --commit-info ./data/commit-chain/commit_info.json.0 --commits data/commit-chain/0-1.bin.commits - -# Regular proof -RUST_LOG=info cargo run --package commit-chain-proof --bin commit-chain-proof -r -- --input-proof "data/commit-chain/0-1.bin" --output-proof "data/commit-chain/1-1.bin" --commit-info ./data/commit-chain/commit_info.json.1 --commits data/commit-chain/1-1.bin.commits - -RUST_LOG=info cargo run --package commit-chain-proof --bin commit-chain-proof -r -- --input-proof "data/commit-chain/1-1.bin" --output-proof "data/commit-chain/2-1.bin" --commit-info ./data/commit-chain/commit_info.json.2 --commits data/commit-chain/2-1.bin.commits -``` - -## State Chain - -State Chain represents the L2's state transition, which checks the EVM's execution, withdrawal transaction inclusion and sequencers' agreement. - -We generate `state-chain-proof` periodically, like by 5 GOAT EVM blocks. Optionally, the block may contain a `proceedWithdraw` transaction. - -* Submit the `proceedWithdraw` transaction on GOAT Network. -* Generate state-chain proof. - -``` -export BATCH_SIZE=10 -export L2_CONTRACT_ADDRESSES="$GOAT_GATEWAY_CONTRACT_ADDRESS" - -export EL_START_BLOCK_NUMBER=10346748 -export start=$EL_START_BLOCK_NUMBER -bash cron-state-chain-proof.sh $start $BATCH_SIZE -``` - -## Watchtower proof - -When a challenge occurs, each Watchtower can publish a `watchtower-challenge-tx` -containing its proof commitment. - -* Generate proofs - -``` -export BITCOIN_NETWORK=regtest -export GENESIS_SEQUENCER_COMMIT_TXID=$(cat ./data/commit-chain/commit_info.json.0 | jq -r .genesis_txid) -export LATEST_SEQUENCER_COMMIT_TXID=$(cat ./data/commit-chain/commit_info.json.0 | jq -r .txid) -export HEADER_CHAIN_INPUT_PROOF="data/header-chain/0-116000.bin" -export COMMIT_CHAIN_INPUT_PROOF="data/commit-chain/0-1.bin" -export STATE_CHAIN_INPUT_PROOF="data/state-chain/9511050-10.bin" - -RUST_LOG=info cargo run --package watchtower-proof --bin watchtower-proof -r -- --output "data/watchtower/output.bin" - -export LATEST_SEQUENCER_COMMIT_TXID=$(cat ./data/commit-chain/commit_info.json.1 | jq -r .txid) -export COMMIT_CHAIN_INPUT_PROOF="data/commit-chain/1-1.bin" -RUST_LOG=info cargo run --package watchtower-proof --bin watchtower-proof -r -- --output "data/watchtower/output2.bin" - -export LATEST_SEQUENCER_COMMIT_TXID=$(cat ./data/commit-chain/commit_info.json.2 | jq -r .txid) -export COMMIT_CHAIN_INPUT_PROOF="data/commit-chain/2-1.bin" -RUST_LOG=info cargo run --package watchtower-proof --bin watchtower-proof -r -- --output "data/watchtower/output3.bin" -``` - -* latest-sequencer-commit-txid: the latest publisher's commitment Bitcoin transaction id -* header-chain-input-proof: the header chain's proof, input and vk. -* commit-chain-input-proof: the commit chain's proof, input and vk. - -## Operator proof - -* Simulate a withdraw challenge +Run from the repository root: ```bash -(cd .. && cargo test -r -p bitvm-gc --test regtest regtest_take2_flow -- --ignored) +cargo build -r -p header-chain-proof -p commit-chain-proof -p state-chain-proof \ + -p watchtower-proof -p operator-proof +cargo run -r -p operator-proof --bin operator-proof -- --help ``` -Fund the Operator on Regtest and obtain the graph ID, Watchtower-Challenge-Init -transaction ID, and indexed Watchtower transactions and public keys from that run. - -* Generate proofs +For continuous proof generation and task recovery, use the +[Proof Builder](../proof-builder-rpc/README.md). For complete local bridge +scenarios, use the [end-to-end suite](../localnet/e2e/README.md). -After `proceedWithdraw`, generate the Operator proof for the corresponding graph -and withdrawal block. Use values from the same graph and proof-chain history. - -```bash -export BITCOIN_NETWORK=regtest -export HEADER_CHAIN_INPUT_PROOF="data/header-chain/header.bin" -export COMMIT_CHAIN_INPUT_PROOF="data/commit-chain/commit.bin" -export STATE_CHAIN_INPUT_PROOF="data/state-chain/state.bin" - -export GENESIS_SEQUENCER_COMMIT_TXID="${GENESIS_SEQUENCER_COMMIT_TXID:?Set the genesis commitment transaction ID}" -export LATEST_SEQUENCER_COMMIT_TXID="${LATEST_SEQUENCER_COMMIT_TXID:?Set the latest commitment transaction ID}" -export OPERATOR_COMMITTED_BLOCKHASH="${OPERATOR_COMMITTED_BLOCKHASH:?Set the committed Bitcoin block hash}" -export GRAPH_ID="${GRAPH_ID:?Set the graph ID as 32 hexadecimal characters}" -export EXECUTION_LAYER_BLOCK_NUMBER="${EXECUTION_LAYER_BLOCK_NUMBER:?Set the withdrawal block height}" -export INCLUDED_WATCHTOWERS="${INCLUDED_WATCHTOWERS:?Set the Watchtower inclusion bitmask}" -export WATCHTOWER_PUBLIC_KEYS="${WATCHTOWER_PUBLIC_KEYS:?Set the indexed public-key list}" -export WATCHTOWER_CHALLENGE_TXIDS="${WATCHTOWER_CHALLENGE_TXIDS:?Set the indexed transaction-ID list}" -export WATCHTOWER_CHALLENGE_INIT_TXID="${WATCHTOWER_CHALLENGE_INIT_TXID:?Set the challenge-init transaction ID}" - -RUST_LOG=info cargo run --package operator-proof --bin operator-proof -r -- --output "data/operator/output.bin" -``` +## Proof compatibility -The operator proof is consumed directly by the Assert flow. BABE setup binds its static public -input, while the dynamic public input is committed after the watchtower challenge set is known. +Program changes and proof-system upgrades can change verification keys. Nodes +and proof workers must agree on the programs, network, and verification keys +used for a graph. Rebuild chain proofs when an upgrade cannot verify the previous +proof artifacts. -* latest-sequencer-commit-txid: the latest publisher's commitment Bitcoin transaction id -* header-chain-input-proof: the header chain's proof, input and vk. -* commit-chain-input-proof: the commit chain's proof, input and vk. -* included-watchtower: a 256-bit bitmask; each bit flags a valid watchtower inside operator proof. -* execution-layer-block-number: the block number that including `proceedWithdraw`(Peg-out) transaction of GOAT Network's execution layer(Geth). -* watchtower-challenge-txids and watchtower-public-keys: comma-separated lists in graph slot order. The inclusion bitmask uses the same indices. -* watchtower-challenge-init-txid: the watchtower challenge init transaction id in GOAT's BitVM graph. +A graph's committed public inputs and presignatures cannot be changed by updating +its stored parameters. Treat compatibility of existing graphs separately from +admission of newly created graphs. diff --git a/circuits/crates/state-chain/samples/README.md b/circuits/crates/state-chain/samples/README.md index cf52341e..c963d3a9 100644 --- a/circuits/crates/state-chain/samples/README.md +++ b/circuits/crates/state-chain/samples/README.md @@ -1,13 +1,8 @@ -# Samples +# State-chain samples -* signed_header +These samples contain signed headers and current and next validator sets for +two adjacent GOAT consensus heights. They provide concrete inputs for checking +consensus transitions in the state-chain proof. -http://127.0.0.1:26657/commit?height=5756785 - -* validator_set and next_validator_set - -"http://127.0.0.1:26657/validators?height=5756784" - -* [optional]blocks - -http://127.0.0.1:26657/block?height=5756785 \ No newline at end of file +See the [circuits guide](../../../README.md) for how state-chain proofs fit into +the bridge's proof composition. diff --git a/deployment/README.md b/deployment/README.md index c217c8c4..779ad15b 100644 --- a/deployment/README.md +++ b/deployment/README.md @@ -1,10 +1,9 @@ # Deployment Guide -Deploy Committee, Operator, Verifier and Watchtower nodes with `bitvm-noded`. -Relayer is a Committee mode, enabled with `ENABLE_RELAYER=true`. See the -[node guide](../node/README.md) for configuration and RPC details, and the -[transaction graph](../transaction-graph.svg) for Bitcoin claim and dispute paths. -For a local test environment, use the [localnet guide](../localnet/README.md). +A deployment consists of Committee, Operator, Verifier and Watchtower nodes, +with Committee nodes optionally acting as Relayers. The [node guide](../node/README.md) +describes their roles, and the [transaction graph](../transaction-graph.svg) +shows their Bitcoin claim and dispute paths. ## Build @@ -13,97 +12,29 @@ Run from the repository root: ```bash source ~/.zkm-toolchain/env cargo build -r -p bitvm-noded --bin bitvm-noded +./target/release/bitvm-noded --help ``` -The binary is `target/release/bitvm-noded`. +## Prepare participants -## Configure a node +1. Select the intended Bitcoin and GOAT networks and deployed bridge contracts. +2. Register each participant's identity for its role and fund the required + Bitcoin and GOAT accounts. +3. Choose the proof service and peer network shared by the participants. +4. Give each node isolated persistent state and distinct listening addresses. +5. Start the nodes and confirm they are ready before initiating bridge activity. -The node loads `.env` from its working directory. Use a separate directory and -SQLite database for each node. +All participants must agree on the graph parameters, Verifier slot count, and +proof programs used for new graphs. Existing finalized graphs retain the values +committed when they were created. -```dotenv -RUST_LOG=info -ACTOR=Verifier -BITCOIN_NETWORK=testnet4 -GOAT_NETWORK=test -GOAT_CHAIN_URL=https://rpc.testnet3.goat.network -GOAT_GATEWAY_CONTRACT_ADDRESS= -GOAT_GATEWAY_EVENT_THE_GRAPH_URL= -GOAT_GATEWAY_EVENT_FILTER_FROM= -GOAT_PROOF_BUILD_URL=http://127.0.0.1:7777 -PEER_KEY= -BITVM_SECRET= -GOAT_ADDRESS= -``` - -Replace the placeholders with values for your deployment. Contract addresses, -subgraph URLs and starting blocks must describe the same deployment. -Generate peer and funding keys with the commands in the -[node guide](../node/README.md#generate-node-keys). - -| Role | Configuration | -|------|---------------| -| Committee | `ACTOR=Committee`; set `GOAT_PRIVATE_KEY` | -| Operator | `ACTOR=Operator`; set `GOAT_ADDRESS` or `GOAT_PRIVATE_KEY`, and fund the Bitcoin signing key | -| Verifier | `ACTOR=Verifier`; set `GOAT_ADDRESS` or `GOAT_PRIVATE_KEY`, and fund the Bitcoin signing key | -| Watchtower | `ACTOR=Watchtower`; configure its registered Bitcoin signing key | -| Relayer | `ACTOR=Committee`, `ENABLE_RELAYER=true`; set `GOAT_PRIVATE_KEY` | - -Optional settings include: - -| Variable | Purpose | -|----------|---------| -| `BTC_CHAIN_URL` | Bitcoin Esplora endpoint | -| `PROTO_NAME` | P2P protocol namespace; use the same value for participating nodes | -| `NODE_NAME` | Display name | -| `EXTERNAL_SOCKET_ADDR` | Advertised node RPC socket address; bypasses automatic discovery | -| `GOAT_SEQUENCER_SET_PUBLISHER_CONTRACT_ADDRESS` | Sequencer-set publisher contract | -| `ENV_GOAT_SEQUENCER_SET_MULTI_SIG_VERIFIER_ADDRESS` | Sequencer-set multisignature verifier contract | -| `GOAT_SWAP_CONTRACT_ADDRESS` | Swap contract for swap-enabled services | -| `GOAT_SWAP_EVENT_THE_GRAPH_URL` / `GOAT_SWAP_EVENT_FILTER_FROM` | Swap subgraph and starting block | -| `MIN_REQUIRED_VERIFIER` | Verifier slots for new graphs; default `1` | - -Set the same positive `MIN_REQUIRED_VERIFIER` on all participating nodes before -creating graphs. Existing finalized graphs retain their original Verifier slots. -The [configuration reference](../node/README.md#configuration) covers engine, -network limits, timeouts and key-retention settings. - -## Run - -```bash -./target/release/bitvm-noded \ - --rpc-addr 0.0.0.0:9100 \ - --metrics-addr 127.0.0.1:9108 \ - --p2p-port 8443 \ - --db-path 'sqlite:./node.db?mode=rwc' -``` - -Use `--bootnodes ''` to join existing peers. The Docker entrypoint -also accepts `BOOTNODES` and passes it to this CLI option. Give each process -its own database and listening ports. - -## Health and metrics - -`GET /v1/health` on the business RPC listener returns `200` when ready and `503` -otherwise, with readiness and backend status. The node retries an unavailable -GOAT RPC during startup before exiting if initialization still cannot complete. - -Prometheus uses the dedicated `--metrics-addr` listener and `/metrics` path. -`GET /metrics` on the business RPC listener returns `404`. Bind metrics to a -private interface and use a distinct port per process. - -The [Docker Compose configuration](docker-compose.yml) publishes metrics on -host loopback: +## Operate -| Service | Host metrics endpoint | -|---------|-----------------------| -| `committee_0` | `http://127.0.0.1:9900/metrics` | -| `committee_1` | `http://127.0.0.1:9901/metrics` | -| `verifier_0` | `http://127.0.0.1:9902/metrics` | -| `operator_0` | `http://127.0.0.1:9903/metrics` | -| `watchtower_0` | `http://127.0.0.1:9904/metrics` | +Run nodes under a supervisor so interrupted processes restart. Retain their +persistent state across restarts. Monitor readiness, chain progress, peers, and +failed work with the [operator tools](../scripts/ops/README.md). -Use the [operator tools](../scripts/ops/README.md) for health triage, log queries -and instance or graph timelines. Proof service configuration and recovery are -covered in the [Proof Builder RPC guide](../proof-builder-rpc/README.md). +Keep operational monitoring on a private network. Follow the +[Proof Builder guide](../proof-builder-rpc/README.md) for proof coordination and +recovery. Use the [localnet](../localnet/README.md) to test changes before a live +deployment. diff --git a/localnet/README.md b/localnet/README.md index 965a7364..42859017 100644 --- a/localnet/README.md +++ b/localnet/README.md @@ -1,157 +1,51 @@ -# localnet — a clone-and-run mock testnet for the node +# Localnet -Five `mock-gc` nodes (committee_0, committee_1, operator_0, verifier_0, watchtower_0) on bitcoin -regtest + a GOAT chain (a local anvil, or the public GOAT testnet3, see "GOAT backend") + a local -graph-node, every endpoint on 127.0.0.1. The garbled circuits, -soldering and proofs are deterministic mocks with configurable delays; everything on chain is -real, so peg-in, peg-out, challenge/assert/take-2 and disprove run the genuine transaction graph -(see the [node guide](../node/README.md#local-testnets) and [transaction graph](../transaction-graph.svg)). The on-demand regression suite lives in -[e2e/](e2e/README.md). CI runs the whole suite after pushes to main / dev, see "CI" below. +Localnet runs two Committee nodes, an Operator, a Verifier and a Watchtower on +Bitcoin Regtest and a GOAT chain. Proof generation and verification use +deterministic mocks, while Bitcoin transactions follow the real +[transaction graph](../transaction-graph.svg). -## Host requirements - -docker (with compose v2), jq, sqlite3, python3, curl, and the Rust/zkm toolchain to build the -node (`source ~/.zkm-toolchain/env`). forge runs in the pinned foundry container -(`tools/forge.sh`), graph-cli in the pinned node container (`subgraph/deploy-local.sh`); a host -`cast` is used when present and otherwise the container one (`tools/cast.sh`, slower per call). -The p2p-isolation e2e case additionally needs passwordless `sudo iptables`. +The default GOAT backend is a private Anvil chain. The testnet3 backend uses GOAT +testnet3 with a separately deployed bridge. The test environment binds its +services to loopback. ## Quick start +Run from the repository root: + ```bash -git clone --recurse-submodules https://github.com/GOATNetwork/bitvm-node.git && cd bitvm-node # or: git submodule update --init --recursive -make localnet-reset # builds the binaries if needed, then wipes+creates the whole localnet (2 min; 7 min when keys are generated; + the build) +make localnet-reset make localnet-status -make e2e-smoke # pegin_happy pegout_happy challenge_take2 (~23 min) -make localnet-down # stop nodes/miner/proof builder; `localnet/scripts/down.sh infra` also stops the containers +make e2e-smoke +make localnet-down ``` -`make localnet-up` adopts an already created localnet (containers, chains, contracts, node DBs) -without rebuilding anything; it refuses if the data dir has no keys/contracts yet (run the reset). - -## Layout: code vs state - -| | where | what | -|---|---|---| -| code | `localnet/` (this directory) | `scripts/` lifecycle + ops, `tools/` fault proxy / key generation / container wrappers, `e2e/` the suite, `infra/docker-compose.yml`, `contracts/` (git submodule, upstream contracts), `contracts-local/` (GatewayLocal + mocks + deploy), `subgraph/` (gateway subgraph sources + deploy) | -| state | `$LOCALNET_DATA` (default `localnet/.local`, git-ignored) | `bin/` installed binaries, `state/` keys, contracts.json, subgraph.json, cli envs, `nodes//` .env + sqlite DB, `data/` chain data of the containers, `logs/`, `run/` pids/overrides/tuning, `results/` e2e reports, `forge/` + `subgraph/` build artifacts | - -`scripts/localnet.env` is the only config file. Everything that must differ between two localnets -on one host derives from three variables, all taken from the environment: - -| variable | default | effect | -|---|---|---| -| `LOCALNET_DATA` | `localnet/.local` | state directory | -| `PORT_BASE` | `22000` | every port is `PORT_BASE + offset`: esplora +2, bitcoind RPC +443, ipfs +501, GOAT RPC (anvil or the testnet3 forwarder) +545, graph query/admin/index/metrics +800/+820/+830/+840, node rpc/p2p/metrics +900/+910/+920 (+ role index), proof builder +950, fault proxies +960..+979, chaos control +989 | -| `COMPOSE_PROJECT` | `bvlocal` | docker project, container names `-`, network `-net`, iptables chain `_CHAOS` | +`localnet-reset` rebuilds the test environment and clears its chain and node +state. `localnet-up` restarts an initialized environment without resetting it. -So a second localnet next to the first one is -`LOCALNET_DATA=/srv/bv2 PORT_BASE=23000 COMPOSE_PROJECT=bvlocal2 make localnet-reset`. -Per-data-dir overrides go in `$LOCALNET_DATA/localnet.override.env` (sourced last), e.g. -`export ANVIL_CHAIN_ID=48815` to adopt a localnet whose anvil was created before the default -moved to 48816. Runtime knobs the e2e suite flips (proof verdict, solder delay) are written to -`$LOCALNET_DATA/run/tuning.env` and picked up by `scripts/nodes.sh` at (re)start; the tracked -`localnet.env` is never edited by a script. +## Test scenarios -### GOAT backend +The [end-to-end suite](e2e/README.md) covers peg-in, peg-out, challenges, +disproofs, restarts, and backend outages. Run the smoke suite for a small change +and the full suite for broader protocol or scheduling changes: -`GOAT_BACKEND=anvil` (default) runs a private anvil. `GOAT_BACKEND=testnet3` runs no chain at all: -the `goat-rpc` container (nginx, ~20 MiB) forwards plain HTTP on port +545 to -`GOAT_UPSTREAM_RPC` (default `https://rpc.testnet3.goat.network`, chain id 48816 like the anvil), so -nodes, the fault proxy and graph-node are unchanged. Every reset still deploys a fresh contract stack -of our own there and the subgraph starts at its creation block. Gas comes from `GOAT_FUNDER_KEY`, or -from the file `GOAT_FUNDER_KEY_FILE` (default `$LOCALNET_DATA/secrets/goat-funder.key`, chmod 600, -never in the repo): a funded testnet3 account, one per data dir, since two localnets sharing a funder -race on its nonce. The role top-ups default to 0.0005 below 0.0002 (anvil: 100 below 10); the reset -keeps `state/keys.json`, so the roles keep their balances between runs. Set the backend per data dir -in `localnet.override.env`, then reset; `chaos.sh infra pause goat` pauses whichever GOAT container -runs. The public chain brings its limits: an RPC outage fails the run, and anyone can call the -deployed contracts. - -## Scripts - -``` -scripts/reset.sh --yes DESTRUCTIVE: build if needed, wipe, recreate infra, keys, 101 blocks, contracts, subgraph, fund, start -scripts/up.sh adopt + start (idempotent); fails with a hint when a node DB does not fit the binary -scripts/down.sh [infra] stop nodes, proof builder, miner, fault proxy [+ docker compose stop] -scripts/status.sh containers, chain heights, subgraph sync, support processes, per-node readiness -scripts/nodes.sh start|stop|restart|status [role...] -scripts/miner.sh start|stop|status|once [n] regtest miner, 1 block / MINE_INTERVAL_SECS -scripts/proof-builder.sh start|stop|status mock-proof-builder -scripts/fund.sh BTC + gas top-ups (idempotent) -scripts/gen-node-envs.sh nodes//.env + state/cli/{user,operator,verifier}.env from state/*.json -scripts/install-bins.sh copy the release binaries into $LOCALNET_DATA/bin -scripts/chaos.sh fault injection (proxy faults, p2p cuts, freeze, container pause); `status`, `reset` -scripts/triage.sh [role] scripts/logq.sh ... scripts/trace.sh read-only diagnosis helpers (JSON logs under logs/nodes//) -build.sh cargo build -r -p bitvm-noded --features "rpc-debug-endpoints mock-gc" --bins (repo target) +```bash +make e2e ``` -Tuning: edit `scripts/localnet.env` (MOCK_GC_*_DELAY_SECS, MOCK_PROOF_*, MINE_INTERVAL_SECS, ...), -then `scripts/nodes.sh restart` (GC knobs) or `scripts/proof-builder.sh stop && start` (proof knobs). -The e2e runner applies its own faster tier (2 s GC / 5 s proof) through `run/tuning.env` for the duration of a run. -Keep challenge/eval and proof delays far below the block cadence or the honest operator loses the -WronglyChallenged race. - -## Contracts +Mock proof verdicts exercise protocol branches; they do not test cryptographic +soundness. Keep proof and challenge processing fast enough to complete before +the simulated Bitcoin deadlines. Mock builds are not for live deployments. -`contracts/` contains the L2 contracts used by the local deployment. The -Gateway must recognize Guardian output index 3 in Kickoff. The local GOAT chain -uses chain ID 48816 so it accepts the node's Regtest transaction metadata. +## Diagnosis -`contracts-local/` is a second foundry project that imports the submodule (`bitvm-l2/` remapping): - -- `GatewayLocal.sol` = upstream `GatewayUpgradeable` + two committee-only knobs: `debugMintPegBTC` - (pegBTC faucet for the peg-out cases) and `debugSetResponseWindowBlocks` (the initializer fixes - 200 goat blocks; the deploy lowers it to `RESPONSE_WINDOW_BLOCKS`, default 50). Runtime size - 23,930 B, ~650 B under the EIP-170 limit: keep additions tiny. -- `MockBitcoinSPV.sol`, `MockWGBTC.sol`: the SPV relay and the stake token, our own instances on either backend. -- `script/DeployLocal.sol`: one broadcast deploys mocks + GatewayLocal/CommitteeManagement/ - StakeManagement proxies (same order and console labels as upstream `DeployGateway.sol`). -- `deploy-local.sh`: gas funding, the script, the committee/verifier/operator registrations and - the operator stake, then writes `state/contracts.json`. - -## Subgraph - -`subgraph/gateway/` holds only sources (schema, manifest with a zero placeholder address, mapping, -ABI regenerated from the submodule build, `package-lock.json`). `subgraph/deploy-local.sh` stages -them under `$LOCALNET_DATA/subgraph/gateway`, patches address + start block from -`state/contracts.json`, and runs `npm ci` / `graph codegen` / `graph build` / `graph create` / -`graph deploy` inside the node container (node_modules is cached per data dir). - -## Images - -bitcoin/bitcoin:29, ipfs/kubo:v0.17.0, postgres:16 by tag; mempool/electrs, graph-node, foundry -(anvil + forge + cast) and node:22-alpine by digest, see `scripts/localnet.env`. To move a digest, -change it there and `scripts/reset.sh --yes`. - -## Observability - -- RPC per node (`rpc-debug-endpoints` on): `GET /v1/debug/status`, `/v1/debug/engine`, `/v1/instances`, `/v1/graphs`. -- Metrics `http://127.0.0.1:/metrics`: `bitvm_node_ready`, p2p peers/topics, `bitvm_node_engine_rows{table,state}`. -- Logs: one JSON file per boot under `logs/nodes//`, `current` points at the newest; `scripts/logq.sh`, `scripts/trace.sh`. -- DBs: `nodes//bitvm-node.db` (`sqlite3 -readonly`; UUIDs are 16-byte BLOBs, use `hex(col)`). +Check readiness and chain progress before starting a scenario. Use the +[operator tools](../scripts/ops/README.md) to inspect stalled instances or graphs. +Clear fault injection and restore mining before rerunning a failed scenario. ## CI -`.github/workflows/e2e_localnet.yml` runs on the self-hosted `bitvm2-node` runner on every push to -`main` / `dev` and on manual dispatch, with `GOAT_BACKEND=testnet3` (Actions → "Localnet e2e" → Run workflow); the same triggers -as `ci_time_consuming.yml`. It is informational, not a required check. One run is build → -`make localnet-reset` → `make e2e` → stop, about two hours; runs queue (`concurrency: -localnet-e2e`) and a running one is never cancelled. - -The CI localnet is a third, private one: `LOCALNET_DATA=/data/actions-runner-bitvm2-node/cache/localnet-e2e/data`, -`PORT_BASE=25000`, `COMPOSE_PROJECT=bvci`, `CARGO_TARGET_DIR=.../localnet-e2e/target`, and the -funder key in `.../localnet-e2e/goat-funder.key` on the runner host (outside the data dir). The data dir -is kept between runs (keys, solc and node_modules caches; the reset wipes chains and node DBs), and -the last 10 result dirs stay in it. The job summary shows `report.md`; the run's result dir -(`report.json`, per-case logs, `logs.tar.gz`) is uploaded as the `localnet-e2e-` artifact for -14 days. The runner user needs docker, passwordless `sudo` for the p2p case (otherwise it is -reported as SKIP), the zkm toolchain and a free 25000–25999 port range. To reproduce a CI failure -locally, run the same three make targets with the same variables (any free `PORT_BASE`, and a -funder key of your own). - -## Known limits - -- Mocked verdicts: proof-validity branches test the mock policy (`MOCK_GC_PROOF_VERDICT`), not cryptography. -- A node whose DB predates a migration the binary cannot apply dies at start; `up.sh` reports it, the fix is a reset. -- All ports bind loopback; the p2p listener too (`P2P_LISTEN_IP=127.0.0.1`). +The [Localnet e2e workflow](../.github/workflows/e2e_localnet.yml) runs on pushes +to `main` and `dev`, and on manual dispatch. It builds the mock nodes, resets a +private test environment, runs the suite, and reports results. Runs are queued +to avoid concurrent changes to the same test environment. diff --git a/localnet/e2e/README.md b/localnet/e2e/README.md index 23d8f29f..e76be559 100644 --- a/localnet/e2e/README.md +++ b/localnet/e2e/README.md @@ -1,56 +1,48 @@ -# e2e — on-demand end-to-end regression of the node +# End-to-end checks -Run this suite manually after a change, or through the [Localnet e2e workflow](../../.github/workflows/e2e_localnet.yml) on pushes to `main` / `dev` and manual dispatch. It uses the [localnet](../README.md) (five mocked nodes on regtest, a GOAT chain and a local graph). `LOCALNET_DATA`, `PORT_BASE` and `COMPOSE_PROJECT` select the environment. +The suite exercises bridge protocol and recovery behavior on the +[localnet](../README.md). It runs manually and through the +[Localnet e2e workflow](../../.github/workflows/e2e_localnet.yml). +## Run + +From the repository root: + +```bash +make localnet-status +make e2e-smoke +make e2e ``` -localnet/scripts/status.sh # infra, chains, nodes all up -localnet/e2e/run.sh --list # the cases, in run order -localnet/e2e/run.sh # all of them (95–110 min), report in $LOCALNET_DATA/results/latest/report.md -localnet/e2e/run.sh pegout_happy challenge_take2 # a subset; make e2e-smoke = pegin_happy pegout_happy challenge_take2 -localnet/e2e/run.sh --stop-on-fail # stop at the first failing case + +List scenarios or run a subset: + +```bash +localnet/e2e/run.sh --list +localnet/e2e/run.sh pegout_happy challenge_take2 +localnet/e2e/run.sh --stop-on-fail ``` -Knobs: `E2E_CASE_TIMEOUT` (seconds per case, default 1200; an expired case is killed with its whole process tree, -chaos is reset and every node restarted before the post-check), `E2E_DELAY_TIER=fast|default` (default `fast`: the run -writes 2 s GC / 5 s proof delays to `$LOCALNET_DATA/run/tuning.env`, restarts nodes + proof builder, and restores the -`localnet.env` values (15/30 s, the manual-debugging tier) at the end), `E2E_POLL_SECS` (default 2), -`E2E_STOP_AFTER` (default 1: when the run exits, for any reason, `scripts/down.sh infra` stops nodes, support -processes and containers, data kept; `make localnet-up` brings it back. 0 leaves everything running for debugging). -Per run `$LOCALNET_DATA/results//` holds `.log`, `report.md`, `report.json` (same table as structured -data: result, duration, rc, post-check rc, notes) and `logs.tar.gz` (every node / miner / proof-builder / fault-proxy -log written during the run). Exit code 0 only when nothing failed; `p2p_isolate_operator` is reported as SKIP (not a -failure) when the host has no passwordless `sudo` for iptables. - -To test a new node commit: `localnet/build.sh`, `localnet/scripts/install-bins.sh`, `localnet/scripts/nodes.sh restart`, then `localnet/e2e/run.sh`. - -| case | what it drives | pass = | -|---|---|---| -| `pegin_happy` | request → answers → Prepare → ceremony → Peg-In-Confirm → L2 mint → graph posted | instance RelayerL2Minted, graph L2Recorded on all nodes | -| `pegout_happy` | initWithdraw → kickoff → proceedWithdraw → Take-1 → finishWithdrawHappyPath | OperatorTake1 on all nodes, finish sent | -| `challenge_take2` | peg-out challenged by the verifier, assert found valid → Take-2 | OperatorTake2, finishWithdrawUnhappyPath | -| `dispute_disprove` | same with `MOCK_GC_PROOF_VERDICT=always-invalid` on operator + verifier → ChallengeAssert → Disprove | Disprove on all nodes, finishWithdrawDisproved; verdict restored to `valid` afterwards | -| `backtoback` | two peg-ins with a 120 s verifier solder delay; B is prepared while A's ceremony runs, and the verifier queues B's setup behind A's solder (one GC job per node at a time, `ENGINE_CPU_POOL_SLOTS` default 1) | A still in its ceremony on every node when B's Prepare confirms; B's setup job created before A's solder finished; both L2Recorded; consecutive slots | -| `pegin_restart_*` | peg-in with committee_0 / operator_0 / verifier_0 restarted the moment one of the instance's jobs is Running on it (verifier: 20 s solder for the case) | a job was caught running (verifier: its solder); it ends Done, re-run by the new process or finished during the old one's graceful shutdown; peg-in completes; one graph and one Peg-In-Confirm txid | -| `goat_timeout_at_kickoff` | Goat RPC answers 25 s slower than the 20 s timeout for 60 s at the kickoff block | peg-out completes after the window | -| `subgraph_latency` | 8 s on every subgraph query for a whole peg-in | this case's peg-in completes | -| `proof_builder_blackhole` | proof-builder requests accepted but never answered for 240 s during Prover-Assert | at least one operator request hit the black hole; Assert then Take-2 | -| `p2p_isolate_operator` | operator_0's p2p links cut for 90 s during the ceremony (iptables, needs sudo; SKIP without) | 0 peers half way through; both committee peers back within 60 s of the heal; peg-in completes | -| `boot_with_goat_down` | watchtower_0 restarted with its Goat / esplora / subgraph proxy down | the same process survives each outage and is ready within 240 s after it; a death fails the case (the node is started again for the later cases) | - -Every case starts with an empty scratch dir (`$LOCALNET_DATA/run/e2e`, nothing carries over from an earlier case) -and a pre-check: a snapshot of every node's Failed job ids, nodes ready, no chaos injection left, and the operator's -locked stake at or above the gateway minimum (a Disprove slashes it; an operator under the minimum is silently -ignored by every peer, so the pre-check tops it up with the mock WGBTC faucet). Every case ends with a post-check: -`/v1/debug/engine` has no Failed / Poisoned inbox rows, no waiting / poisoned aggregates and no Running / Poisoned -jobs on any node, no Failed job exists that was not in the snapshot (and the debug endpoint reports no more Failed -jobs than the snapshot holds; a failed snapshot or DB query fails the check instead of counting as "none"), `scripts/chaos.sh status` is empty, the miner is -restarted. `run/tuning.env` is saved before each case and restored after it whatever happened (pass, fail, timeout, -Ctrl-C); when the restore had to change something the nodes are restarted and the report notes it. Scenario -scripts exit with their own verdict and the runner uses it; an unknown case name is an error (exit 2). The fault cases attach every node to `tools/faultproxy.py` the first time they run -(`scripts/chaos.sh attach all`, a restart of every node) and leave them attached; `scripts/chaos.sh detach all` undoes it. - -Layout: `lib.sh` helpers (status polling, log waits, DB queries, chaos helpers), `flows/` the protocol drives, -`scenarios/` the fault injections around a flow; `$LOCALNET_DATA/results//` one log per case plus `report.md`, -`results/latest`. CLI identities come from `$LOCALNET_DATA/state/cli/{user,operator,verifier}.env` (test-only keys, -generated by `scripts/gen-node-envs.sh`); peg-ins consume one wallet UTXO each. Cases that flip a mock knob -(`dispute_disprove`, `backtoback`) do it through `$LOCALNET_DATA/run/tuning.env` and restore it afterwards. +## Coverage + +| Scenario | Expected behavior | +|----------|-------------------| +| Peg-in | Committee signing completes and the graph is posted | +| Peg-out | An unchallenged claim completes through Take-1 | +| Valid challenged claim | The Operator completes through Take-2 | +| Invalid assertion | Verifier-Assert and Disprove prevent the claim | +| Back-to-back peg-ins | Setup work queues without losing either instance | +| Node restart | Interrupted work resumes without duplicate graph creation | +| Slow or unavailable backends | Pending work recovers after the outage | +| P2P isolation | Peers reconnect and the protocol resumes | + +The [transaction graph](../../transaction-graph.svg) shows the Bitcoin paths. +Proof verdicts are mocked; these scenarios do not establish cryptographic +soundness. + +## Results + +Each scenario checks readiness and failed work before it runs, then checks for +new failures, stuck work, and leftover fault injection afterward. The suite +reports individual results and exits unsuccessfully if any scenario fails. +The P2P isolation scenario is skipped when the host cannot apply its network +isolation rules. diff --git a/node/README.md b/node/README.md index 88be4160..3a8fdabe 100644 --- a/node/README.md +++ b/node/README.md @@ -1,16 +1,8 @@ # BitVM Node -`bitvm-noded` runs the Bitcoin and GOAT L2 bridge protocol: chain monitoring, -transaction graph setup and signing, P2P communication, and proof coordination. - -## Contents - -- [Transaction graph](#bitcoin-transaction-graph) -- [Roles](#roles) -- [Execution and storage](#execution-and-storage) -- [Build & Run](#build--run) -- [Configuration](#configuration) -- [Related documentation](#related-documentation) +`bitvm-noded` coordinates bridge participants on Bitcoin and GOAT L2. It monitors +both chains, builds and signs transaction graphs, exchanges protocol messages, +and coordinates proof generation and disputes. ## Bitcoin transaction graph @@ -18,187 +10,57 @@ transaction graph setup and signing, P2P communication, and proof coordination. Open the [full-size SVG](../transaction-graph.svg) for connector wiring, CSV paths, competing spends, PreKickoff slots, and per-input authorization types. -Blue paths show operator and claim transactions; red paths show challenges and -timeouts. Dots join branches; crossings without dots do not. +The legend explains the signature notation. -- **Peg-in:** PegIn-Deposit funds Z. PegIn-Confirm creates C0; PegIn-Refund is - the competing refund path. +- **Peg-in:** the deposit funds Z; confirmation creates C0, while refund is the + competing path. - **Claims:** Take-1 reimburses the Operator after A's CSV delay. Take-2 requires C0, D, F and G and must satisfy both D's and F's CSV delays. -- **Disputes:** Watchtower and Verifier branches can prevent a claim by spending - its required outputs. Verifier-Assert is also called ChallengeAssert; +- **Disputes:** Watchtower and Verifier branches prevent claims by spending + required outputs. Verifier-Assert is also called ChallengeAssert; Pubin-Disprove spends D directly. -The SVG's legend explains `ALL`, `NONE`, `SINGLE`, `ACP`, and inputs authorized -without a transaction signature. Fee inputs and GOAT settlement are omitted. - ## Roles | Role | Responsibility | |------|----------------| | Committee | Presign transactions and endorse peg-ins and graphs | -| Operator | Build graphs, publish assertions, and execute Bitcoin claims | +| Operator | Build graphs, publish assertions, and execute claims | | Verifier | Prepare circuits, verify assertions, and submit disproofs | | Watchtower | Monitor Bitcoin and submit Watchtower challenges | | Relayer | Committee mode that posts data to GOAT and serves graph requests | -Set `ACTOR` to the node's role. Committee nodes enable relaying with -`ENABLE_RELAYER=true`. - -## Execution and storage - -Chain observations, admitted P2P messages, and job results enter a durable inbox. -Instance and Graph aggregates apply events in SQLite transactions and produce -outbox messages or leased jobs. Heavy setup, signing, proof work, and network -calls run in jobs; their results return through the inbox. - -The Instance aggregate handles peg-in validation and signing. The Graph aggregate -handles graph setup, presigning, Bitcoin claims, and disputes. Chain scans supply -the facts used to advance their state. - -SQLite stores graph metadata in `graph`, serialized definitions in -`graph_raw_data`, and execution state in the engine tables. Missing graph -definitions are fetched directly with `GetGraph` from relayers, committee -members, or the Operator and authenticated before use. Gossip carries setup -messages and acknowledgements. - -## Build & Run - -### Build Commands - -```bash -# Build for regtest -BITCOIN_NETWORK=regtest cargo build -r - -# Build for testnet4 -BITCOIN_NETWORK=testnet4 cargo build -r - -# Build only the node binary -cargo build -r -p bitvm-noded -``` - -### Local testnets - -```bash -cargo build -r -p bitvm-noded --features "rpc-debug-endpoints mock-gc" --bins -``` - -`mock-gc` replaces proof machinery with deterministic mocks for local testing. -Use it only on local testnets. See the [localnet guide](../localnet/README.md) -for setup and end-to-end scenarios. - -### Generate Node Keys - -```bash -# Generate P2P peer key -bitvm-noded key peer -# Output: -# PEER_KEY= -# PEER_ID= - -# Generate funding address (for Operator/Verifier) -bitvm-noded key funding-address -# Output: -# Funding P2WSH address: bc1q... -``` +## Execution -### Start Node +Chain observations, admitted messages, and job results enter a durable inbox. +Instance and Graph aggregates apply events and schedule work. Heavy setup, +signing, proof generation, and network calls run as jobs; results return through +the inbox. Persistent state lets interrupted work resume after restart. -```bash -bitvm-noded \ - --rpc-addr 0.0.0.0:8080 \ - --metrics-addr 127.0.0.1:9108 \ - --db-path ./node.db \ - --p2p-port 4001 \ - --bootnodes '/ip4/x.x.x.x/tcp/4001/p2p/' -``` +The Instance aggregate handles peg-in validation and signing. The Graph +aggregate handles setup, presigning, claims, and disputes. Missing graph +definitions are retrieved from peers and authenticated before use. -### Start Local Mock RPC +## Build and run -Use this when you only need to test HTTP interfaces. It starts the RPC routes without -P2P, chain watchers, or maintenance tasks, and seeds a local SQLite database with -mock nodes, instances, graphs, and overview data. +Run from the repository root: ```bash -cargo run -p bitvm-noded --bin mock-rpc -- --rpc-addr 127.0.0.1:18080 +cargo build -r -p bitvm-noded --bin bitvm-noded +./target/release/bitvm-noded --help ``` -Useful test calls: - -```bash -curl http://127.0.0.1:18080/v1/nodes/overview -curl http://127.0.0.1:18080/v1/instances -curl http://127.0.0.1:18080/v1/swaps -curl http://127.0.0.1:18080/v1/graphs -``` +See the [deployment guide](../deployment/README.md) for running participants and +the [operator tools](../scripts/ops/README.md) for diagnosis. -The mock binary prints seeded instance and graph IDs on startup. Endpoints that -require real `graph_raw_data`, such as graph transaction hex export, still need -real graph raw data in the database. - -### CLI Options - -| Option | Description | Default | -|--------|-------------|---------| -| `--rpc-addr` | RPC service bind address | `0.0.0.0:8080` | -| `--db-path` | SQLite database path | `sqlite:/tmp/bitvm-node.db` | -| `--p2p-port` | P2P listen port | `0` (random) | -| `--bootnodes` | Bootstrap node addresses | - | -| `--metrics-addr` | Dedicated Prometheus listener address | Disabled | -| `--metrics-path` | Path served by the dedicated metrics listener | `/metrics` | -| `--enable-kademlia` | Enable Kademlia DHT | `true` | - -## Configuration - -### Environment Variables - -| Variable | Required | Description | Default | -|----------|----------|-------------|---------| -| `ACTOR` | Yes | Node role: `Committee`, `Operator`, `Verifier`, `Watchtower` | `Verifier` | -| `BITCOIN_NETWORK` | Yes | Bitcoin network: `bitcoin`, `testnet4`, `signet`, `regtest` | `testnet4` | -| `GOAT_NETWORK` | Yes | GOAT network: `main`, `test` | `test` | -| `GOAT_CHAIN_URL` | Yes | GOAT L2 RPC endpoint | - | -| `GOAT_GATEWAY_CONTRACT_ADDRESS` | Yes | Gateway contract address | - | -| `BITVM_SECRET` | Yes | Node private key or seed (`seed:xxx` format) | - | -| `PEER_KEY` | Yes | libp2p node key (Base64 encoded) | - | -| `GOAT_PRIVATE_KEY` | Conditional | GOAT chain private key (required for Committee) | - | -| `GOAT_ADDRESS` | Conditional | GOAT address (required for Operator/Verifier) | - | -| `ENABLE_RELAYER` | No | Enable relayer mode for Committee nodes | `false` | -| `ENABLE_BABE_SETUP_STATE_CLEANUP` | No | Enable scheduled BABE setup state cleanup for Operator/Verifier nodes | `false` | -| `BTC_CHAIN_URL` | No | Bitcoin Esplora API endpoint | Public Esplora | -| `NODE_NAME` | No | Node display name | `ZKM` | -| `OPERATOR_NODE_SERVICE_FEE` | No | Operator service fee rate | `0.001` | -| `ENABLE_COMMITTEE_INSTANCE_KEY_DELETE` | No | Delete the committee instance key envelope when the instance fails (except `Timeout`) or 32 blocks after the Peg-In-Confirm confirmed; off, the intent is only logged | `false` | -| `COMMITTEE_INSTANCE_KEY_DELETE_TIMELOCK_BLOCKS` | No | Number of BTC blocks to wait after pegin-confirm confirmation before deleting key envelope; deletion also waits until Prepare height + T0 | `32` | -| `PEGIN_ABANDON_BUFFER_BLOCKS` | No | Blocks after `Prepare height + connector_z` before an instance whose Peg-In-Confirm is neither confirmed nor in the mempool is abandoned | `18` / `14` / `3` / `6` by network | -| `OPERATOR_GRAPH_START_CUTOFF_BLOCKS` | No | Operators start no new graph for an instance past `Prepare height + cutoff` | `⌊2·connector_z/3⌋` | -| `ENGINE_WORKER_COUNT` / `ENGINE_CPU_POOL_SLOTS` / `ENGINE_PEGOUT_POOL_SLOTS` / `ENGINE_NETWORK_POOL_SLOTS` | No | Execution engine: aggregate workers, concurrent CPU-bound setup jobs, concurrent CPU-bound peg-out jobs (own slots, so a graph build never delays a challenge), concurrent network jobs | `1` / `1` / `1` / `4` | -| `ENGINE_APPLY_TIMEOUT_SECS` / `ENGINE_RECHECK_INTERVAL_SECS` | No | Execution engine: apply budget (also the SQLite busy timeout) and the re-check of waiting events | `120` / `60` | -| `P2P_MAX_JSON_MESSAGE_BYTES` | No | Largest JSON gossip message accepted | `2 MiB` | -| `P2P_INBOX_MAX_QUEUED_BYTES` | No | Payload bytes of admitted network events the engine inbox may hold (the committee reserve and per-peer quotas derive from it; unregistered authors are relayed, never stored) | `2 GiB` | -| `P2P_MAX_INCOMING_CONNECTIONS` / `P2P_INBOUND_REGISTERED_RESERVE` | No | Inbound connection ceiling and the slots reserved for registered peers | `512` / `128` | -| `P2P_RESERVED_PEERS` / `P2P_TRUSTED_OPERATOR_BINDINGS` | No | Peers that always get a connection slot; configured operator key bindings | empty | -| `P2P_LISTEN_IP` | No | Interface the p2p listener binds to; `127.0.0.1` keeps a node on a shared host loopback-only | `0.0.0.0` | -| `BTC_REQUEST_TIMEOUT_SECS` / `GOAT_RPC_TIMEOUT_SECS` | No | Per-request timeout of the Esplora and GOAT RPC clients, seconds (clamped to 1–120) | `15` / `20` | -| `HTTP_REQUEST_TIMEOUT_SECS` | No | Whole-request timeout of the proof-builder HTTP client, seconds (clamped to 1–600); the connect timeout is 10 s | `60` | -| `MAINTENANCE_SUBTASK_TIMEOUT_SECS` | No | Timeout for each maintenance subtask; timed-out subtasks record `timed_out` and later subtasks still run. BTC scans use a separate serial loop. Operator balances refresh 32 verified bindings per pass, plus the local operator; pending peg-ins use `INSTANCE_MAINTENANCE_BATCH_SIZE`. Both advance their cursor before each item, and give each item the BTC and GOAT client timeouts of its backend calls plus 5 s | `60` | - -### Relayer Configuration - -To run a Committee node as a graph data relayer: +### Local testnets -```bash -export ACTOR=Committee -export ENABLE_RELAYER=true -export GOAT_PRIVATE_KEY='your-private-key' -# ... other required variables -``` +Use the [localnet guide](../localnet/README.md) to run the bridge with mocked +proof machinery. Mock builds are for local testing only. ## Related documentation -- [Project README](../README.md) — project layout and transaction graph checks -- [Localnet guide](../localnet/README.md) — local testing -- [Proof Builder RPC](../proof-builder-rpc/README.md) — proof service -- [Circuits README](../circuits/README.md) — proof generation -- [Deployment guide](../deployment/README.md) — deployment -- [Incentives](../docs/incentives.md) — incentives +- [Project overview and graph checks](../README.md) +- [Circuits](../circuits/README.md) +- [Proof Builder](../proof-builder-rpc/README.md) +- [Incentives](../docs/incentives.md) diff --git a/proof-builder-rpc/README.md b/proof-builder-rpc/README.md index 24816c2e..7cb13033 100644 --- a/proof-builder-rpc/README.md +++ b/proof-builder-rpc/README.md @@ -1,190 +1,53 @@ -# Proof Builder RPC +# Proof Builder -## Build +The Proof Builder coordinates header, commit and state-chain proofs, then +produces Operator and Watchtower proofs on demand. See the +[circuits guide](../circuits/README.md) for proof composition and the +[transaction graph](../transaction-graph.svg) for their role in claims and disputes. -From the repository root: +## Build and inspect -``` -BITCOIN_NETWORK=testnet4 cargo build -r -p proof-builder-rpc -``` - -## Deployment - -Run the service from `proof-builder-rpc/` so the example configuration's relative -proof paths resolve correctly. Copy `proof-builder.toml.example` to -`proof-builder.toml` and configure the chain endpoints and proof inputs before -startup. - -Initial parameters are read from `proof-builder.toml` (see -[proof-builder.toml.example](./proof-builder.toml.example)). After that, they will be loaded from -the database. - -Proof inputs are described in the [circuits guide](../circuits/README.md). The -[transaction graph](../transaction-graph.svg) shows where the on-demand proofs -are used. - -To reuse local Bitcoin headers for the first batch built from Genesis, set -`prefer_local_headers_on_init = true` in `[header_chain]`, together with `init_input = true` -and `start = 0`. The option defaults to `false` for existing configurations. Headers are read -from `block_headers`, which contains consecutive 80-byte headers starting at height 0. -If that working cache is missing or incomplete, the builder also checks -`0-.bin.blocks` in the same directory. A sidecar containing more complete headers -is copied into the working cache, up to the requested batch size; the source sidecar is unchanged. -For example, a Genesis batch of 157450 headers can reuse `0-157450.bin.blocks` without changing -the configured `block_headers` path. Logs identify both the sidecar source and working cache. -Choose a distinct Genesis `output_proof`, such as `genesis-157450.bin`; the following batch is -written to `157450-1.bin` in the same output directory. -For `batch_size = 10000`, the first batch uses heights 0 through 9999. A complete local batch -requires no Bitcoin RPC calls and leaves any later cached headers intact. If local files are absent -or incomplete, only missing headers are fetched through Esplora; an incomplete trailing header -is fetched again. This preference overrides `force_fetch` for the Genesis batch only. -After the first proof succeeds, subsequent batches contain one block and use the existing -fetching behaviour. Restarting with database progress also resumes with one block per batch. -When this option is enabled, surplus cached headers are replaced as later batches are fetched, -including when `force_fetch = false`. +Run from the repository root: -The four Operator/Watchtower task endpoints require signed requests and live GOAT contract -authorization. Configure the read-only GOAT connection before startup: - -```dotenv -GOAT_NETWORK=test -GOAT_CHAIN_URL=https://rpc.testnet3.goat.network -GOAT_GATEWAY_CONTRACT_ADDRESS=, +```bash +cargo build -r -p proof-builder-rpc +cargo run -r -p proof-builder-rpc -- --help ``` -Proof Builder accepts one or more comma-separated Gateway addresses. All configured Gateways use -the same GOAT network and RPC. It discovers the CommitteeManagement contract through each Gateway -and exits if any configured contract set cannot be initialized. Operator requests require graph -ownership and a matching instance/graph relation. Operator registration and stake are enforced -when graphs are admitted and are not rechecked for existing proof tasks. -Watchtower requests require membership in the current global Watchtower registry and the request -public key must match the authenticated signer. Operator graph authorization is queried for each request. Watchtower membership -uses the short-lived registry cache described below. Failed authorization queries -return HTTP 503. - -Each authenticated request selects a configured deployment with the signed `gateway_address` -field. For staged upgrades, the field may be omitted while Proof Builder has exactly one Gateway; -it is required when multiple Gateways are configured. Nodes continue to use one -`GOAT_GATEWAY_CONTRACT_ADDRESS` value and add it to every Operator/Watchtower proof request. - -Nodes sign requests with their existing `BITVM_SECRET`; no GOAT or Bitcoin private key is -configured on the Proof Builder. Deploy signing-capable nodes before enabling the authenticated -Proof Builder so that in-flight proof polling is not rejected. - -Authenticated requests carry `x-proof-auth-timestamp`, `x-proof-auth-nonce`, -`x-proof-auth-public-key`, and `x-proof-auth-signature`. The signature binds the caller role, -HTTP method, route, timestamp, nonce, and canonical JSON body. Rust callers should use -`proof_builder::api_auth::sign_proof_builder_request`; each retry must generate a new nonce. -A nonce is remembered only after the signer is authorized for the graph, and the service -serves at most 64 requests at a time; requests beyond that, or arriving while the replay cache -is full, get `503` and should be retried. A request, body upload included, that takes longer -than 30 seconds is cancelled with `408`. The Watchtower registry is cached for 5 seconds. - -To enable multiple Gateways without interrupting proof polling, first upgrade Proof Builder with a -single Gateway, then upgrade all Nodes to send `gateway_address`, and finally configure the -comma-separated Gateway list. +## Authorization -## Fleet app worker mode +Nodes authenticate proof requests with their bridge signing identities. The +service checks Operator ownership of a graph or current Watchtower membership +before accepting work. It supports multiple Gateway deployments on the same +GOAT network, with each request selecting its intended deployment. -Proof Builder can run as a zkm-fleet integrated app worker. Fleet starts a dedicated Ziren GPU -server for the app and launches `proof-builder-rpc` beside it. The -worker proves directly against that loopback server instead of submitting through the SDK -network prover and the Fleet gateway. The HTTP API and node configuration are unchanged. +The service does not need the nodes' private keys. Replay protection, bounded +concurrency, and request deadlines limit repeated or stalled work. -Fleet mode starts when Fleet supplies its worker environment. Any partial Fleet environment is a -startup error. Without these variables the SDK network prover (`ZKM_PROVER=network`) is used as -before, which is how operators fall back explicitly. +## Scheduling and recovery -| Variable | Source | Meaning | -|---|---|---| -| `FLEET_GPU_ENDPOINT` | Fleet | Loopback URL of the replica's GPU server | -| `FLEET_APP_STORAGE_ROOT` | Fleet | Durable app storage (`proof_storage_root`) | -| `FLEET_APP_ELF_ID` | Fleet | App id: the SHA-256 of the program bundle manifest | -| `FLEET_APP_NAMESPACE`, `FLEET_APP_TENANT` | Fleet | App identity | -| `FLEET_EXIT_WITH_PARENT` | Fleet | Exit with the Fleet actor (Linux `PR_SET_PDEATHSIG`) | -| `PROOF_BUILDER_PROGRAMS_FILE` | worker env file | `programs.json` from the package | +Long-running chain proofs persist their progress. Operator and Watchtower tasks +with unavailable inputs stay pending while later work is checked. Independent +forward and bounded retry scans revisit older tasks under continuing load. -The app id covers all five guests. Each line is ` `, sorted by name -(commit-chain, header-chain, operator, state-chain, watchtower), and the id is the SHA-256 of -those lines. `proof-builder-rpc --print-programs` sets every program up on the CPU prover and -prints the ELF digests, program IDs, verifying-key hashes, wrap mode and app id as JSON; package -that output as `programs.json`. At startup the worker checks that the embedded programs, the -programs file and `ZKM_IMM_WRAP_VK` match the Fleet app id. After each GPU setup it also checks -the verifying key. +A timeout prevents a subsequently completed proof from overwriting a task that +is no longer pending. Restarting resumes from saved progress; restore proof +artifacts together with the matching service state. -Startup also requires the following: +## Fleet workers -- The working directory (`worker_workdir`) and the sqlite database must be inside the storage - root, and the worker takes an exclusive lock on `/proof-builder.lock`. -- Use `/run` as the working directory so that relative `../circuits/data/...` paths - resolve to `/circuits/data/...`. -- Groth16 wrapping runs on the worker's CPU. Pre-install the Groth16 circuit artifacts for the - Fleet execution user before enabling the app. +Fleet can run the service beside a dedicated GPU server. Proving is serialized +through a shared gate, and completed artifacts are published before progress is +committed. The worker checks program identities and verification keys at startup. -Proving behaviour in Fleet mode: +Transient proving failures leave work pending. Invalid inputs have a bounded +attempt count. On shutdown, the worker stops accepting new proving work and +allows in-flight work a bounded time to finish; unfinished work can resume after +restart. -- The GPU server runs one proof at a time. Builder setup and every proving unit (prove, publish - and database commit) pass through a FIFO gate. -- An operator or watchtower task stays `New` while it queues and proves. Its result is written - only while it is still `New`, so a timeout request that arrives in the meantime wins. -- If the GPU server stops reporting ready, or a proof cannot be published, the task stays - pending and the worker exits. Fleet then restarts the GPU server and the worker. -- Otherwise a failed prover call is replayed by executing its guest on the CPU. GPU RPC - errors do not distinguish transport failures from rejected inputs. If the guest executes, - the prover or its transport failed; the proof is retried in place and the failure does not - count as an attempt. -- A proof that fails because the guest rejects its input, or because the result does not - verify, marks the task `Failed` after `--max-proof-attempts` consecutive failures - (default 2). -- On SIGTERM or a task failure the worker starts no new proof. It waits up to - `--drain-timeout-secs` for the proof in progress to be published and committed, then exits. - A proof still running at the timeout does not delay the exit; it is retried after restart. - Keep that value below the service's `worker_stop_timeout_s`. Exit codes: 0 after a clean stop, - 1 after a task failure or an abandoned proof, 101 after a panic. -- Proofs and their sidecars are published through synced temporary files and atomic renames. - Sidecars are published before the proof file, and the database is updated after all files - and the directory are synced. Sidecars include the header chain's `.blocks`, which operator - and watchtower proofs read for SPV. State-chain `.args`/`.ctx` snapshots are written atomically - as well. -- The state-chain context builder prepares at most `--state-chain-max-pending-batches` - unproved batches (default 2; 0 means unlimited). - -## Metrics - -Prometheus metrics use a dedicated listener and are disabled unless `--metrics-addr` is set. The -business RPC listener does not serve metrics, so `GET /metrics` on `--rpc-addr` returns `404`. - -```bash -proof-builder-rpc \ - --rpc-addr 0.0.0.0:7777 \ - --metrics-addr 10.42.0.8:9109 \ - --config proof-builder.toml -``` - -The dedicated listener serves only `/metrics`. Bind it to a private interface and allow only the -Prometheus source network to reach it. The listener does not provide TLS or authentication and -should not be exposed directly to the public internet. An unavailable metrics port causes startup -to fail. - -## Progress and recovery - -Long-running header, commit and state-chain tasks persist their progress in -SQLite. Restarting the service resumes from that progress; retain the matching -proof files and sidecars when restoring a database backup. - -Operator and Watchtower tasks with unavailable inputs stay pending while later -tasks are checked. Forward and bounded retry scans alternate, so older tasks are -revisited even while new work arrives. Timeout requests prevent a subsequently -completed proof from overwriting a task that is no longer pending. - -Sequencer-set publication is a separate operation. Inspect the publisher CLI -before advancing or rotating the publisher set: - -```bash -cargo run -r -p bitvm-noded --bin sequencer-set-publish -- --help -``` +## Operations -The [Regtest publication fixture](../node/ssp-ci.sh) demonstrates the sequence -with test publishers. For state-chain initialization and proof inputs, see the -[circuits guide](../circuits/README.md#state-chain) and the -[configuration example](proof-builder.toml.example). +Monitor chain-proof progress and pending on-demand tasks alongside node health. +Sequencer-set publication is a separate operation from proof generation. For +bridge deployment and local regression scenarios, see the +[deployment guide](../deployment/README.md) and [localnet](../localnet/README.md). diff --git a/scripts/ops/README.md b/scripts/ops/README.md index 045b3f89..9198dca1 100644 --- a/scripts/ops/README.md +++ b/scripts/ops/README.md @@ -1,64 +1,36 @@ -# Node ops scripts +# Operator tools -Three read-only helpers for one running `bitvm-noded`. They need `bash`, `jq`, `curl`; `trace.sh` also needs -`sqlite3`. Defaults come from the environment, flags override them: +Three read-only helpers inspect a running node. They query health and progress, +read structured logs, and trace an instance or graph without changing its state. -| Variable | Meaning | Default | -|---|---|---| -| `NODE_RPC` | the node's RPC base URL (`--rpc-addr`) | `http://127.0.0.1:8080` | -| `NODE_METRICS` | the metrics listener base URL (`--metrics-addr`) | `http://127.0.0.1:9090` | -| `NODE_LOG` | the node's JSON log file (stdout of `bitvm-noded`) | none | -| `NODE_DB` | the node's SQLite file (the `--db-path` without `sqlite:`) | none | -| `NODE_PID` | the node's pid, for process stats in `triage.sh` | none | +## Health overview -## `triage.sh` — one screen: is this node healthy, and if not, where - -``` -scripts/ops/triage.sh [-r RPC] [-m METRICS] [-f LOG] [-p PID] [-s 10m] +```bash +scripts/ops/triage.sh ``` -Prints `GET /v1/health` (readiness bits, each backend's availability with its last error, event watcher state), -the key gauges from `/metrics` (peers, lags, balances, backend probe counters, tasks whose last success is older -than 90 s, alert counters), `/v1/debug/status` and the parked / poisoned engine rows when the node was built with -`rpc-debug-endpoints`, and the WARN/ERROR lines of the last window grouped by event and outcome when a log file is -given. First thing to run when a node looks stuck. +Start here when a node appears stalled. The overview combines readiness, +backend availability, peer and chain progress, and recent warnings. -## `logq.sh` — query the JSON log +## Log queries -``` -scripts/ops/logq.sh [-f LOG] [-s 10m|2h|1d|2026-10-15T08:00:00] [-l INFO|WARN|ERROR] [-e EVENT] [-g SUBSTRING] [-n N] [-j] +```bash +scripts/ops/logq.sh -s 30m -l WARN ``` -One compact line per record: `time level role event-or-target outcome message extra=fields`. `-l WARN` includes -ERROR; `-e` matches the structured `event` field exactly (`maintenance_tick_result`, `engine_jobs`, -`backend_availability`, ...); `-g` is a plain substring filter applied before parsing, so an instance id, graph id -or txid is cheap even on a large file; `-j` prints the raw JSON lines. Reads stdin when no file is given, so it -composes with `journalctl -o cat` or `docker logs`. +The log query helper reads standard input and filters by time, severity, event, +or a plain-text match. It skips non-JSON lines and can emit raw matching records. -Useful queries: +## Instance and graph tracing +```bash +scripts/ops/trace.sh 'instance-or-graph-id' ``` -logq.sh -f node.log -s 30m -l WARN # everything that went wrong in the last half hour -logq.sh -f node.log -s 1h -e backend_availability # when did a backend go down / come back -logq.sh -f node.log -s 2h -e engine_jobs -l WARN # job attempts that failed on an error path -logq.sh -f node.log -s 1d -g 475a4574 # every line naming an instance / graph / tx -``` - -## `trace.sh` — timeline of one peg-in instance or graph - -``` -scripts/ops/trace.sh [-r RPC] [-d DB] [-f LOG] [-s 6h] [-n 300] -``` - -Resolves the related ids (the graphs of an instance, the instance of a graph), prints the RPC status, the -database rows that drive the flow (instance, graphs, `pending_graph_init`, the engine aggregate, every job with its -state / attempts / lease / last error, parked inbox events, poison rows) and the log lines naming any of the ids, -oldest first. Give it the same node's database and log; for several nodes run it once per node and compare the -`graph` and `job` lines. -## Notes +Replace the example identifier with the instance or graph being investigated. +The trace combines related protocol state, work progress, and a chronological +view of relevant events. Inspect each participant separately when comparing +behavior across nodes. -- All three only read: no RPC that changes state, the database is opened read-only. -- The log format is the node's JSON output (`tracing_subscriber` JSON with top-level `event` / `outcome` fields). -- `/v1/health` returns 503 while the node is not ready; `triage.sh` still prints its body. -- `logq.sh` skips lines that are not JSON (panic messages, stray stderr output) instead of stopping there. +See the [node guide](../../node/README.md) for roles and execution, and the +[transaction graph](../../transaction-graph.svg) for Bitcoin claim and dispute paths.