diff --git a/bin/ethlambda/src/benchmark/mod.rs b/bin/ethlambda/src/benchmark/mod.rs index 70e5d2fc..869ddcce 100644 --- a/bin/ethlambda/src/benchmark/mod.rs +++ b/bin/ethlambda/src/benchmark/mod.rs @@ -6,8 +6,8 @@ //! and the slot-alignment sleep are outside the measured span, matching the //! node's own `lean_block_building_time_seconds` boundary. //! -//! See docs/plans/block-building-benchmark.md for the design and roadmap -//! (real-crypto pools and replay-from-datadir land in later milestones). +//! See docs/benchmarking.md for what is and is not measured, how to read a +//! report, and the current limitations. mod corpus; mod report; diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 46e8041d..6b6f9c50 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -23,4 +23,5 @@ # Development +- [Benchmarking](./benchmarking.md) - [Spec Deviations](./spec_deviations.md) diff --git a/docs/benchmarking.md b/docs/benchmarking.md new file mode 100644 index 00000000..924437b7 --- /dev/null +++ b/docs/benchmarking.md @@ -0,0 +1,137 @@ +# Benchmarking block building + +`ethlambda benchmark` measures block building the way the node performs it when +it proposes, against a reproducible synthetic workload, with no devnet running. + +Block building is otherwise only observable through the Prometheus histograms a +live node exports. Those are noisy, depend on whatever the network happened to +be doing, and cannot be diffed against a baseline — which makes them a poor +instrument for tracking performance. The benchmark +trades network realism for repeatability: the same parameters produce the same +blocks every run, so two reports differ only where the code differs. + +## Running it + +```bash +make bench # defaults, mock crypto +BENCH_ARGS="synthetic --iterations 50" make bench +``` + +`make bench` is a thin wrapper. The binary takes the same arguments directly: + +```bash +ethlambda benchmark synthetic --mock-crypto --num-validators 8 --iterations 10 +``` + +A default mock run finishes in well under a second, which is why CI can afford +to run one on every pull request. + +| Flag | Default | Meaning | +| --- | --- | --- | +| `--num-validators` | `8` | Validators in the synthetic genesis | +| `--warmup-slots` | `8` | Unmeasured slots built first, so measured builds run on a state with realistic historical roots and justifications | +| `--iterations` | `10` | Measured builds, one block each | +| `--proofs-per-data` | `1` | Aggregates seeded per `AttestationData`, mimicking committee aggregators over disjoint validator subsets | +| `--seed` | `42` | Seed for the validator set; fixes the whole run | +| `--mock-crypto` | off | Placeholder proofs instead of real XMSS/leanVM signatures. **Currently required** — see [Limitations](#limitations) | +| `--enable-proposer-aggregation` | off | Mirrors the node flag: collapse same-data proofs via recursive leanVM aggregation | +| `--max-attestations-per-block` | `3` | Mirrors the node flag: distinct `AttestationData` per block | +| `--format` | `human` | `human` or `json` | +| `--output ` | — | Also write the JSON report to a file | + +Logs go to stderr and the report to stdout, so `--format json` pipes straight +into `jq`. + +## What it measures + +Each iteration enters `produce_block_with_signatures` — the same function +`BlockChainServer::propose_block` calls — and the harness reports the phases +inside it: + +| Phase | Work | +| --- | --- | +| `select_payloads` | Choosing which attestations go in the block | +| `compact` | Collapsing or picking among proofs for the same data | +| `stf_simulate` | The state transition that seals `state_root` | +| `overhead` | The rest of the measured span: tick processing, attestation promotion, fork-choice head, pool clone | +| `wall` | The whole span | + +`overhead` is `wall` minus the sum of the phases, so the columns add up by +construction. + +Deliberately **outside** the measured span, matching the boundary of the node's +own `lean_block_building_time_seconds` metric: gossip publish, the +slot-alignment sleep, and importing the block that was just built. The import +still happens between iterations — otherwise every iteration would build on the +same head and `process_slots` would get more expensive as the run went on. + +Phase times come from the sample sums of the existing +`lean_block_proposal_attestation_build_phase_seconds` histogram, read before and +after each build. Histogram sums accumulate raw f64 seconds, so the difference +between two readings is the elapsed phase time and bucket boundaries play no +part. Nothing is added to the hot path for the benchmark's benefit. The harness +asserts each phase was observed exactly once per build and fails the run +otherwise, because a mis-attributed report is worse than no report. + +## Reading a report + +``` +Block-building benchmark — synthetic workload (mock crypto) + validators=8 warmup_slots=8 iterations=10 proofs_per_data=1 seed=42 + enable_proposer_aggregation=false max_attestations_per_block=3 + ethlambda/v0.1.0/aarch64-apple-darwin/rustc-v1.97.1 leansig=15cbdd43 leanvm=e2592df4 os=macos arch=aarch64 threads=14 + + iter compact select_payloads stf_simulate overhead wall root + 1 0.000ms 0.002ms 0.015ms 0.068ms 0.085ms 0x7282cc99 + ... + + phase count min mean p50 p90 max + select_payloads 10 0.002ms 0.002ms 0.002ms 0.003ms 0.003ms + ... +``` + +Every measured iteration gets its own row, and the summary follows below it. +Outliers are never discarded: XMSS signing and OTS window advancement produce +legitimate heavy tails, and hiding them would misrepresent the thing being +measured. A coefficient of variation above 10% is flagged so a noisy run is not +mistaken for a result. + +Percentiles are nearest-rank, without interpolation. Sample counts here are +small, so an actual observed value is more informative than a blend of two +neighbours. + +The `root` column is the block root of each built block. It is what makes a +before/after comparison trustworthy: if an optimization leaves the root +sequence unchanged, it changed only speed and not which attestations were +selected. If the roots move, the change altered block contents and the timing +comparison means something different than intended. + +## Comparing two runs + +Same seed and same parameters produce identical root sequences, so a baseline +and a candidate can be diffed directly. The header line exists to tell you when +they *cannot* be compared: + +- `leansig` and `leanvm` are the resolved revisions the binary was built + against, read from `Cargo.lock` at build time. leanSig tracks a moving branch + and leanVM performs the signature aggregation, so either one moving changes + the measured crypto. +- `os`, `arch` and `threads` change results across machines. + +Two reports that disagree on any of those are not measuring the same thing. + +## Limitations + +- **`--mock-crypto` is required.** Real XMSS/leanVM pools are not wired up yet, + so the run measures selection, compaction and the state transition — not + signing or aggregation. +- **The seal phase is not measured.** Signing, type-1 wrapping and type-2 + merging happen after the measured span and are not reported. +- **Synthetic workloads only.** Replaying a real datadir is not implemented, so + results reflect a synthetic chain rather than a deep production state. + +## In CI + +The Test job runs a short mock benchmark and asserts the JSON report's shape +(`schema_version`, one sample per iteration). It costs seconds, and it means a +change to the report contract cannot land unnoticed.