Skip to content

chore(benchmarks): regenerate public node/bun baseline at b77aba6343 - #11073

Closed
proggeramlug wants to merge 1 commit into
mainfrom
chore/public-baseline-b77aba6
Closed

proggeramlug wants to merge 1 commit into
mainfrom
chore/public-baseline-b77aba6

Conversation

@proggeramlug

@proggeramlug proggeramlug commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Regenerate the public Perry/Node/Bun baseline at b77aba63433b100b9482d47181b5ccc5e724ff81 (v0.5.1639) so the lint public-baseline freshness step has conforming evidence for the root manifest after #10354 (turnloop). #10977 fingerprints the root manifest except its workspace release version; a real manifest change requires measurement.

This PR contains only the freshly generated benchmarks/results/public-node-bun-v1.json, the generated README section, and benchmarks/suite/results/RESULTS.md. No timings, hashes, benchmark sources, correctness oracles, sample counts, or thresholds were edited. The three files match the mini's generated files byte for byte. README content outside the generated markers is unchanged.

Measurement:

  • Host: perry-macos.local (perry-macos.fritz.box), Apple M1 / arm64, 8 CPUs, 8 GiB RAM, macOS 26.5.1 / Darwin 25.5.0, AC power, initially 32 GiB disk free.
  • Fresh remote clone at the exact measured commit, with no existing target directory. Original ./benchmarks/run_public_baseline.sh ran detached and uninterrupted, exiting 0. It started at 2026-09-22T23:55:39Z; artifact assembly completed at 2026-09-23T01:34:16Z. Fresh local clone and PR branch also start at that pinned commit.
  • Exact tools: Node v22.23.1, Bun 1.3.14, esbuild 0.28.1, Zig 0.15.2. Node/esbuild/Zig were selected from the existing side-by-side installations; ~/nodebin was untouched. Both polyglot banners confirmed untimed esbuild precompilation to .mjs; all nine compute-polyglot .mjs files and required Perry executables were freshly generated. Rust/Cargo: nightly 1.100.0, toolchain dated 2026-08-20. No inherited CARGO_PROFILE_*, RUSTFLAGS, encoded Rust flags, CARGO_TARGET_DIR, or PERRY_* overrides.
  • Quiet preflight: 13 readings across 60 seconds, CPU active 1.37%–1.81%. Every component retained the original CPU active ≤25% for 60 consecutive seconds gate. Confirmation readings: suite 1.4%, polyglot 1.5%, JSON polyglot 1.5%, app patterns 1.7%, honest-bench 1.5%. macOS uses the harness's taskpolicy scheduler hint, not hard CPU affinity.

All 50 workloads / 1,443 positive raw timing samples passed correctness and validation, with all components naming the measured commit:

Component Workloads Samples per peer/workload Total public samples
Suite 24 5 360
Polyglot 9 11 297
JSON polyglot 2 11 66
App patterns 12 15 540
Honest-bench 3 20 180

Honest-bench additionally measured Rust and Zig: all 300 total rows passed the strict Bun-reference check. App patterns retained 3 warmups and honest-bench retained 5 warmups.

Verification:

  • python3 benchmarks/ci_public_baseline_check.py — pass (the original artifact failed with benchmark inputs changed before regeneration).
  • python3 benchmarks/public_baseline.py check — pass, including generated documentation.
  • PYTHONPATH=. python3 tests/test_public_baseline.py — 15 tests pass.
  • Independent read-only audit — exact workload sets, pins, host, component commits, positive raw samples, fingerprints, and three-file scope all pass.
  • git diff --check and bash scripts/check_file_size.sh — pass.
  • Source fingerprint: 9c87723d7cedca511b1dba158fdd505bdbabd667367ee41152f82370c9ceeae5; harness fingerprint: 513dba8ff9eaf931edc8a5fc01a0093a1156c31b0b6c9cc1218f710405e315e0.
  • Artifact SHA-256: 311cd8efa1865e8bc700cd476843f0535f2c76d42f0fe19d60bb9886e8ad78c7.

Performance notes retained without suppressing losing rows:

  • Suite historical --warn-only regressions: loop overhead +28.9% (RAM −3.8%, noise allowance 1.0 ms), object creation +300.0% (RAM +34.7%, noise 1.0 ms), binary trees +333.3% (RAM +33.9%, noise 4.4 ms). The harness explicitly reported the comparison failure without failing the build. These are historical comparisons, not freshness failures or a controlled cross-host performance claim.
  • App-pattern harness slow rows versus Bun: JSON stringify 2.08×, promise chains 4.13×, regex replacement 15.22×. All pass correctness and remain in the public artifact.

Observed anomalies and extra generated files:

  • The artifact actually present at pinned main b77aba6343 was the September 1 Ubuntu artifact measured at 827a92bad5a5, not the September 22 mini artifact present on prior branch commit 6d9286d3. This PR replaces it with the new measurement; it does not copy that earlier branch's evidence.
  • Optional auxiliary Go produced no measurements in either polyglot report (no compute Go executable; all compute Go cells were missing despite a go: done progress line). Go is outside the exported public Perry/Node/Bun evidence. Hermes was explicitly skipped because it was absent. No optional-language results are claimed as passing.
  • Fresh compilation emitted one dead-code warning for relevant_box_roots; compilation succeeded. Auto-optimized runtimes, including the regex-specific feature set, were rebuilt within this fresh clone.
  • Remote git status also listed benchmarks/honest_bench/REPORT.md, benchmarks/honest_bench/results/metadata.json, benchmarks/honest_bench/results/results.json, benchmarks/honest_bench/results/summary.txt, benchmarks/json_polyglot/RESULTS.md, benchmarks/polyglot/RESULTS_AUTO.md, and untracked benchmarks/app-patterns/results/matrix-20260922-182300.md. These seven extra generated files were not copied or committed. No other tracked source/input modifications were reported.
  • Main advanced during measurement to 990b3eeada01406e40a9d99ecdf87f9f91d39db3 (v0.5.1640). Across the fingerprinted input sets, the only change is the exempt workspace release version. This PR remains based on requested b77aba6343, and the artifact accurately names that measured compiler, not the later main compiler.

CI at head 21a210735df50bfe469f589225dcc95a3e2fc05f: lint job is still type-checking Windows runtime/stdlib; the public-baseline freshness step is pending. Local freshness and all 15 tests pass; CI success is not yet claimed.

https://claude.ai/code/session_01EQdCw7BN4AAnn2hAbNXg33

Measure b77aba6 (v0.5.1639) in a fresh
clone on perry-macos.local: Apple M1, arm64, 8 CPUs, 8 GiB RAM,
macOS 26.5.1 / Darwin 25.5.0, AC power.

Use Node v22.23.1, Bun 1.3.14, esbuild 0.28.1 and Zig 0.15.2.
Run the original public-baseline entry point uninterrupted, without
build/profile overrides, and retain all raw measurements and losing rows.
Preflight: 13 CPU-active readings over 60 seconds, 1.37%-1.81%.
Per-component quiet confirmations: 1.4%, 1.5%, 1.5%, 1.7%, 1.5%;
each requires CPU active <=25% for 60 consecutive seconds.

Validate 50 workloads (24+9+2+12+3), 1443 positive raw timing samples
across Perry/Node/Bun, and 5/11/11/15/20 samples per workload/peer.
All correctness checks pass; both freshness checkers and all 15 public
baseline tests pass. Assembly completed 2026-09-23T01:34:16Z.

Preserve warn-only historical regressions: loop overhead +28.9%,
object creation +300.0%, binary trees +333.3%. Optional auxiliary Go
produced no measurements; it is not part of the public artifact.
Commit only the artifact, generated README section, and suite RESULTS.

Claude-Session: https://claude.ai/code/session_01EQdCw7BN4AAnn2hAbNXg33
@coderabbitai

coderabbitai Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 60082ef7-b2bb-4d3c-a90d-335090e12475

📥 Commits

Reviewing files that changed from the base of the PR and between 990b3ee and 21a2107.

📒 Files selected for processing (3)
  • README.md
  • benchmarks/results/public-node-bun-v1.json
  • benchmarks/suite/results/RESULTS.md

Included review availability: Your plan provides up to 8 included reviews per hour; 6 remain after this review.


📝 Walkthrough

Walkthrough

The pull request refreshes benchmark measurements in the README and results document. It updates the evidence commit, Perry version, median timings, result classifications, and summary counts.

Changes

Benchmark results refresh

Layer / File(s) Summary
Refresh published benchmark reports
README.md, benchmarks/suite/results/RESULTS.md
The benchmark reports now use the newer Perry commit and version. They include refreshed Perry, Node.js, and Bun medians, revised win/loss/mixed classifications, and updated summary counts.

Priority: ⬇️ Low

Estimated code review effort: 1 (Trivial) | ~3 minutes

Change: Other

Merge Risk: ⚪ Minimal · up to 21a21

No actionable merge-blocking risk was identified in the benchmark refresh.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly and concisely identifies the benchmark regeneration and the measured baseline commit.
Description check ✅ Passed The description provides a detailed summary, precise scope, related issue references, measurement method, verification results, and generated-file details. It does not use every template heading or in…
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@proggeramlug

Copy link
Copy Markdown
Contributor Author

Superseded by #11075, which landed the same artifact rebuilt on current main (v0.5.1640) so it validates on the tree it merges into — your branch predated train 257. Your measurement run is credited in the commit message with its provenance: one uninterrupted run at b77aba6 on the quiet mini, node v22.23.1 / bun 1.3.14 exact, esbuild 0.28.1, Zig 0.15.2, quiet gate re-checked per component, 50 workloads and 1,443 samples, all correctness passed. lint's public-baseline step is now GREEN for the first time since 2026-09-01.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant