feat(cli): add offline block-building benchmark sub-command

[pablodeymo]

🗒️ Description / Motivation

Adds ethlambda benchmark synthetic — an offline harness that measures block building
exactly as executed when the node proposes, against a reproducible synthetic
workload, with no devnet required.

"Optimize block building" (#465) is the top roadmap item, but the only observability
today is Prometheus histograms on a live devnet: noisy, not reproducible, and unable to
compare an optimization against a baseline. This is milestone M1 (mock-crypto mode) of
the design in docs/plans/block-building-benchmark.md; M2 adds real XMSS/leanVM pools
and post-build seal-phase measurement, M3 adds replay-from-datadir.

Supersedes #497. Same harness, but the CLI plumbing it needed is now split out:
the node/benchmark token dispatch is #591, which this PR stacks on, and the
non-determinism this harness uncovered in extend_proofs_greedily merged separately
as #590. What is left here is the benchmark itself. Review #591 first; the base
moves to main once it lands.

$ make bench
Block-building benchmark — synthetic workload (mock crypto)
  validators=8 warmup_slots=8 iterations=10 proofs_per_data=1 seed=42
  ...
  phase              count        min       mean        p50        p90        max
  compact               10    0.000ms    0.000ms    0.000ms    0.000ms    0.000ms
  select_payloads       10    0.001ms    0.001ms    0.001ms    0.001ms    0.001ms
  stf_simulate          10    0.011ms    0.011ms    0.011ms    0.011ms    0.011ms
  overhead              10    0.047ms    0.048ms    0.048ms    0.049ms    0.049ms
  wall                  10    0.060ms    0.061ms    0.061ms    0.062ms    0.062ms

What Changed

File	Change
bin/ethlambda/src/benchmark/{mod,corpus,report}.rs	New harness: seeded synthetic chain over InMemoryBackend, per-slot pool seeding into the pending pool (promoted by the proposal tick, as in production), iteration loop driving produce_block_with_signatures + block import; stats + human/JSON report
bin/ethlambda/src/command.rs	benchmark joins node as a dispatched token, parsed by a clap::Parser of its own so harness arguments never enter CliOptions; argv[0] is rewritten to ethlambda benchmark so usage lines name the sub-command that owns them
bin/ethlambda/src/main.rs	main is synchronous: it dispatches, and only the node path enters the tokio runtime (run_node carries the #[tokio::main] attributes). Benchmark logs go to stderr at WARN so the report owns stdout
crates/storage/{lib,store}.rs	Export NEW_PAYLOAD_CAP so the harness rejects --proofs-per-data batches the pending pool would evict whole
bin/ethlambda/build.rs	Embed the resolved leanSig and leanVM revisions from Cargo.lock into reports (leansig tracks the moving devnet4 branch; leanVM does the aggregation, so its pinned rev moves the measured crypto too). The per-[[package]] parse collects name and source before extracting the rev, so it does not depend on TOML field order
Makefile, .github/workflows/ci.yml	make bench; seconds-fast mock smoke step in the Test job validating the JSON contract
docs/plans/block-building-benchmark.md	Design doc and milestone roadmap

Correctness / Behavior Guarantees

• cli.rs is not touched, and the node runtime is unchanged. The seven node-required
  arguments stay plain PathBuf/String, so clap keeps emitting its own
  missing-argument errors and there is no Option<T> to unwrap on the node path. That
  replaces feat(cli): add offline block-building benchmark subcommand #497's subcommand_negates_reqs approach, which needed both.
• Determinism: same seed + params → identical per-iteration block roots (recorded in
  the JSON, so a baseline-vs-optimized diff proves an optimization changed only speed,
  not attestation selection). Verified across repeated runs. The harness never reads the
  wall clock into results.
• Exact phase attribution with zero hot-path changes: per-iteration
  select_payloads/compact/stf_simulate come from the sample sums of the existing
  lean_block_proposal_attestation_build_phase_seconds histogram, deltaed between
  iterations; overhead is the clamped remainder of wall minus the phases.
• The benchmark never starts the tokio runtime, so it cannot park a worker thread for the
  duration of a CPU-bound run.

Tests Added / Run

• command.rs unit tests cover the benchmark token: it parses with no node argument, it
  rejects node flags, and its usage line names the sub-command.
• Verified by hand on this branch: make bench; the CI smoke assertion
  (jq -e '.schema_version == 1 and (.samples | length == 3)'); identical block-root
  sequences across two runs at the same seed; and ethlambda --genesis config.yaml still
  failing with clap's own missing-argument list.
• make fmt, make lint, make test (582 tests, 30 suites) — all clean.

Related Issues / PRs

• Supersedes feat(cli): add offline block-building benchmark subcommand #497
• Stacked on feat(cli): accept a node sub-command for running the node #591
• Related to Block building optimizations #465, fix(blockchain): make proof selection tie-breaking deterministic #590

✅ Verification Checklist

• Ran make fmt — clean
• Ran make lint (clippy with -D warnings) — clean
• Ran make test (cargo test --workspace --profile release-fast) — all passing

[github-actions]

🤖 Kimi Code Review

Overall Assessment: Well-structured, secure benchmark harness with proper isolation between node and benchmark code paths. No critical consensus bugs or security vulnerabilities. Minor maintainability issues noted below.

---

build.rs (bin/ethlambda)

Issue: Fragile manual TOML parsing of Cargo.lock.

• Line 63-78: Parsing relies on split("[[package]]") and line prefixes. This breaks if Cargo changes lockfile formatting (e.g., adds inline tables, different quoting) or if package names contain escaped characters.
• Line 43: Path construction join("../../Cargo.lock") assumes the crate is always two levels deep from workspace root. If the crate moves, builds fail or embed "unknown" silently.

Recommendation: Use the cargo-lock crate (already in ecosystem) or toml crate to parse the lockfile robustly. If avoiding deps, add a build.rs test that fails CI if ETHLAMBDA_LEANSIG_REV is "unknown" when it shouldn't be.

---

benchmark/corpus.rs

Issue: Hardcoded pubkey size may drift from type definition.

• Line 88: let mut bytes = [0u8; 52]; assumes ValidatorPubkeyBytes is 52 bytes. If the type changes (e.g., XMSS parameter upgrade), this compiles but fails at SSZ serialization boundaries or crypto ops.

Recommendation: Use ValidatorPubkeyBytes::size() or std::mem::size_of::<ValidatorPubkeyBytes>() to keep the mock data aligned with the real type.

---

benchmark/mod.rs

Issue: Hardcoded metric name coupling.

• Line 226: const PHASE_HISTOGRAM: &str = "lean_block_proposal_attestation_build_phase_seconds";
  If the metric name changes in ethlambda-blockchain, the benchmark compiles but reports all phases as zero/missing at runtime.

Recommendation: Export the metric name constant from ethlambda-blockchain (e.g., pub const BLOCK_PROPOSAL_PHASE_HISTOGRAM: &str = ...) and import it here.

Issue: Potential silent precision loss in timing.

• Line 197: overhead_seconds clamps negative values to 0.0 due to floating-point rounding. While commented, if overhead consistently measures as 0.0 due to phase sums exceeding wall time, it masks measurement errors.

Recommendation: Log a warning if overhead_seconds == 0.0 and wall time > 1ms, or if the unattributed time exceeds 5% of wall time (stricter than the 2% comment suggests).

---

benchmark/report.rs

Issue: Population vs. sample standard deviation.

• Line 277: Variance divides by count (population). For benchmark samples, sample standard deviation (divide by count - 1) is statistically more appropriate for small iteration counts, though the difference is negligible for N=10.

Note: Document that CV uses population stddev if this is intentional.

---

command.rs

Code Quality: Excellent backward compatibility handling. The args.drain(..2) manipulation for benchmark subcommand argv[0] rewriting is correct and preserves clap's usage strings properly.

---

main.rs

Security/Isolation: Good separation of concerns.

• Line 73-82: Benchmark runs synchronously on main thread without Tokio runtime, eliminating scheduling noise for CPU-bound measurements. Node path retains async runtime.
• Line 93-102: Benchmark logging directed to stderr with WARN level keeps stdout pipe-clean for JSON reports. Correct use of tracing_subscriber.

---

Consensus & State Transition Correctness

Validations:

1. Attestation partitioning: participant_groups (corpus.rs:98) correctly creates disjoint validator sets using modulo indexing, accurately simulating committee aggregation.
2. Proposer rotation: slot % options.num_validators (mod.rs:164) matches the round-robin proposer selection in is_proposer.
3. State advancement: Importing built blocks via on_block_without_verification (mod.rs:188) ensures process_slots costs remain constant across iterations rather than growing linearly with slot number.
4. Pool capacity guard: The check at mod.rs:135 prevents silent eviction of seeded attestations when proofs_per_data > NEW_PAYLOAD_CAP.

Note on on_block_without_verification: Appropriate for this context since the block was produced by the same process immediately before import; no untrusted input is processed.

---

Security

• No unsafe code introduced in diff.
• Mock crypto isolation: The mock_crypto flag correctly conflicts with enable_proposer_aggregation (mod.rs:74) to prevent undefined behavior from empty proof aggregation.
• Determinism: splitmix64 PRNG (corpus.rs:108) is cryptographically insecure but appropriate for deterministic test data generation. Seeded state ensures reproducible benchmarks.

---

CI Integration (ci.yml)

Line 78-84: The smoke test validates JSON schema version and sample count. Consider adding a determinism check:

# Run twice with same seed and compare block roots
 cargo run ... --format json > run1.json
 cargo run ... --format json > run2.json
 jq -e '.samples[].block_root' run


---
*Automated review by Kimi (Moonshot AI) · kimi-k2.5 · custom prompt*

[github-actions]

🤖 Codex Code Review

1. The new benchmark’s determinism guarantee is not actually enforced. SyntheticOptions promises identical block roots for the same seed, and seed_pool says insertion order pins proof choice, but extend_proofs_greedily still keeps candidate indices in a HashSet and selects with max_by_key on coverage only. When two proofs add the same marginal coverage, tie-breaking depends on randomized hash iteration order, so repeated runs can produce different selected proofs and different block_roots. That breaks the benchmark’s core regression signal. Add an explicit stable tie-break, e.g. lowest original index.

2. The reported percentiles are off by one for many sample sizes. percentile is documented as “nearest-rank”, but round((len - 1) * q) is not nearest-rank. For example, with 10 samples, p50 becomes the 6th sample instead of the 5th; with 4 samples, the current test explicitly locks in the upper median behavior (report.rs). Since these summaries are the main benchmark output, this will mislead performance comparisons. Use a true nearest-rank formula such as ceil(len * q).max(1) - 1, or rename/document the current convention if intentional.

Aside from those benchmark/reporting issues, I did not see a new consensus-rule regression in fork choice, attestation validation, STF, XMSS, or SSZ handling from this PR’s direct code changes.

I could not run cargo test end-to-end here because the environment cannot fetch the repo’s git dependencies and the default cargo cache is read-only.

---

Automated review by OpenAI Codex · gpt-5.4 · custom prompt

[pablodeymo]

Split into three smaller PRs, each with a goal of its own:

• docs: document the block-building benchmark #594 — the design plan, on its own so the measurement boundary, the phase-attribution approach and the milestones can be argued before there is code to argue around. Targets main directly, so it does not deepen the stack.
• feat(cli): add offline block-building benchmark sub-command #595 — the harness: the real proposer path driven deterministically, reporting one row per measured iteration. Stacked on feat(cli): accept a node sub-command for running the node #591.
• feat(cli): make benchmark reports comparable across runs and machines #596 — comparable reports: aggregate statistics, the leansig/leanVM provenance that says whether two reports may be compared at all, and JSON + the CI contract. Additive on top of feat(cli): add offline block-building benchmark sub-command #595.

#595 + #596 reconstruct what was reviewed here, plus a per-iteration table (#595 would otherwise collect the phase breakdown without ever showing it). Each of the three was verified independently: make fmt, make lint, make test, and the benchmark run end-to-end.

Closing in favour of that stack.