From d1d984f036382581321a1f3fa0041bdecd55505b Mon Sep 17 00:00:00 2001 From: Pablo Deymonnaz Date: Mon, 31 Aug 2026 14:33:38 -0300 Subject: [PATCH 1/2] docs: document the block-building benchmark MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review feedback on the first version of this PR: a `docs/plans/` file written as "this is how the benchmark was originally designed" goes stale the moment the benchmark changes, and then actively misleads. This documents what the benchmark does instead, in the mdbook alongside the other operational pages. Covers what to run and with which flags, which phases are measured and what is deliberately outside the span, how phase times are derived from the existing histogram, how to read a report, and what the block-root column is for — a root sequence that survives an optimization is the evidence that only speed changed. Two things the plan file never said, both of which a reader needs: when two reports may not be compared at all (the leanSig and leanVM revisions, and the machine fields), and what is not supported yet — real crypto, the seal phase, and replay from a datadir. Those are stated as current limitations rather than as milestones, so the page describes the tool rather than a schedule for it. The module doc in benchmark/mod.rs points at the new page. --- bin/ethlambda/src/benchmark/mod.rs | 4 +- docs/SUMMARY.md | 1 + docs/benchmarking.md | 138 +++++++++++++++++++++++++++++ 3 files changed, 141 insertions(+), 2 deletions(-) create mode 100644 docs/benchmarking.md diff --git a/bin/ethlambda/src/benchmark/mod.rs b/bin/ethlambda/src/benchmark/mod.rs index 70e5d2fc..869ddcce 100644 --- a/bin/ethlambda/src/benchmark/mod.rs +++ b/bin/ethlambda/src/benchmark/mod.rs @@ -6,8 +6,8 @@ //! and the slot-alignment sleep are outside the measured span, matching the //! node's own `lean_block_building_time_seconds` boundary. //! -//! See docs/plans/block-building-benchmark.md for the design and roadmap -//! (real-crypto pools and replay-from-datadir land in later milestones). +//! See docs/benchmarking.md for what is and is not measured, how to read a +//! report, and the current limitations. mod corpus; mod report; diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 46e8041d..6b6f9c50 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -23,4 +23,5 @@ # Development +- [Benchmarking](./benchmarking.md) - [Spec Deviations](./spec_deviations.md) diff --git a/docs/benchmarking.md b/docs/benchmarking.md new file mode 100644 index 00000000..451b9d89 --- /dev/null +++ b/docs/benchmarking.md @@ -0,0 +1,138 @@ +# Benchmarking block building + +`ethlambda benchmark` measures block building the way the node performs it when +it proposes, against a reproducible synthetic workload, with no devnet running. + +Block building is otherwise only observable through the Prometheus histograms a +live node exports. Those are noisy, depend on whatever the network happened to +be doing, and cannot be diffed against a baseline — which makes them a poor +instrument for the work tracked in +[#465](https://github.com/lambdaclass/ethlambda/issues/465). The benchmark +trades network realism for repeatability: the same parameters produce the same +blocks every run, so two reports differ only where the code differs. + +## Running it + +```bash +make bench # defaults, mock crypto +BENCH_ARGS="synthetic --iterations 50" make bench +``` + +`make bench` is a thin wrapper. The binary takes the same arguments directly: + +```bash +ethlambda benchmark synthetic --mock-crypto --num-validators 8 --iterations 10 +``` + +A default mock run finishes in well under a second, which is why CI can afford +to run one on every pull request. + +| Flag | Default | Meaning | +| --- | --- | --- | +| `--num-validators` | `8` | Validators in the synthetic genesis | +| `--warmup-slots` | `8` | Unmeasured slots built first, so measured builds run on a state with realistic historical roots and justifications | +| `--iterations` | `10` | Measured builds, one block each | +| `--proofs-per-data` | `1` | Aggregates seeded per `AttestationData`, mimicking committee aggregators over disjoint validator subsets | +| `--seed` | `42` | Seed for the validator set; fixes the whole run | +| `--mock-crypto` | off | Placeholder proofs instead of real XMSS/leanVM signatures. **Currently required** — see [Limitations](#limitations) | +| `--enable-proposer-aggregation` | off | Mirrors the node flag: collapse same-data proofs via recursive leanVM aggregation | +| `--max-attestations-per-block` | `3` | Mirrors the node flag: distinct `AttestationData` per block | +| `--format` | `human` | `human` or `json` | +| `--output ` | — | Also write the JSON report to a file | + +Logs go to stderr and the report to stdout, so `--format json` pipes straight +into `jq`. + +## What it measures + +Each iteration enters `produce_block_with_signatures` — the same function +`BlockChainServer::propose_block` calls — and the harness reports the phases +inside it: + +| Phase | Work | +| --- | --- | +| `select_payloads` | Choosing which attestations go in the block | +| `compact` | Collapsing or picking among proofs for the same data | +| `stf_simulate` | The state transition that seals `state_root` | +| `overhead` | The rest of the measured span: tick processing, attestation promotion, fork-choice head, pool clone | +| `wall` | The whole span | + +`overhead` is `wall` minus the sum of the phases, so the columns add up by +construction. + +Deliberately **outside** the measured span, matching the boundary of the node's +own `lean_block_building_time_seconds` metric: gossip publish, the +slot-alignment sleep, and importing the block that was just built. The import +still happens between iterations — otherwise every iteration would build on the +same head and `process_slots` would get more expensive as the run went on. + +Phase times come from the sample sums of the existing +`lean_block_proposal_attestation_build_phase_seconds` histogram, read before and +after each build. Histogram sums accumulate raw f64 seconds, so the difference +between two readings is the elapsed phase time and bucket boundaries play no +part. Nothing is added to the hot path for the benchmark's benefit. The harness +asserts each phase was observed exactly once per build and fails the run +otherwise, because a mis-attributed report is worse than no report. + +## Reading a report + +``` +Block-building benchmark — synthetic workload (mock crypto) + validators=8 warmup_slots=8 iterations=10 proofs_per_data=1 seed=42 + enable_proposer_aggregation=false max_attestations_per_block=3 + ethlambda/v0.1.0/aarch64-apple-darwin/rustc-v1.97.1 leansig=15cbdd43 leanvm=e2592df4 os=macos arch=aarch64 threads=14 + + iter compact select_payloads stf_simulate overhead wall root + 1 0.000ms 0.002ms 0.015ms 0.068ms 0.085ms 0x7282cc99 + ... + + phase count min mean p50 p90 max + select_payloads 10 0.002ms 0.002ms 0.002ms 0.003ms 0.003ms + ... +``` + +Every measured iteration gets its own row, and the summary follows below it. +Outliers are never discarded: XMSS signing and OTS window advancement produce +legitimate heavy tails, and hiding them would misrepresent the thing being +measured. A coefficient of variation above 10% is flagged so a noisy run is not +mistaken for a result. + +Percentiles are nearest-rank, without interpolation. Sample counts here are +small, so an actual observed value is more informative than a blend of two +neighbours. + +The `root` column is the block root of each built block. It is what makes a +before/after comparison trustworthy: if an optimization leaves the root +sequence unchanged, it changed only speed and not which attestations were +selected. If the roots move, the change altered block contents and the timing +comparison means something different than intended. + +## Comparing two runs + +Same seed and same parameters produce identical root sequences, so a baseline +and a candidate can be diffed directly. The header line exists to tell you when +they *cannot* be compared: + +- `leansig` and `leanvm` are the resolved revisions the binary was built + against, read from `Cargo.lock` at build time. leanSig tracks a moving branch + and leanVM performs the signature aggregation, so either one moving changes + the measured crypto. +- `os`, `arch` and `threads` change results across machines. + +Two reports that disagree on any of those are not measuring the same thing. + +## Limitations + +- **`--mock-crypto` is required.** Real XMSS/leanVM pools are not wired up yet, + so the run measures selection, compaction and the state transition — not + signing or aggregation. +- **The seal phase is not measured.** Signing, type-1 wrapping and type-2 + merging happen after the measured span and are not reported. +- **Synthetic workloads only.** Replaying a real datadir is not implemented, so + results reflect a synthetic chain rather than a deep production state. + +## In CI + +The Test job runs a short mock benchmark and asserts the JSON report's shape +(`schema_version`, one sample per iteration). It costs seconds, and it means a +change to the report contract cannot land unnoticed. From 3f9f63eaf014c02fe4d92a4fe249565f9ea545ef Mon Sep 17 00:00:00 2001 From: Pablo Deymonnaz Date: Tue, 1 Sep 2026 12:20:27 -0300 Subject: [PATCH 2/2] Update docs/benchmarking.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Tomás Grüner <47506558+MegaRedHand@users.noreply.github.com> --- docs/benchmarking.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/benchmarking.md b/docs/benchmarking.md index 451b9d89..924437b7 100644 --- a/docs/benchmarking.md +++ b/docs/benchmarking.md @@ -6,8 +6,7 @@ it proposes, against a reproducible synthetic workload, with no devnet running. Block building is otherwise only observable through the Prometheus histograms a live node exports. Those are noisy, depend on whatever the network happened to be doing, and cannot be diffed against a baseline — which makes them a poor -instrument for the work tracked in -[#465](https://github.com/lambdaclass/ethlambda/issues/465). The benchmark +instrument for tracking performance. The benchmark trades network realism for repeatability: the same parameters produce the same blocks every run, so two reports differ only where the code differs.