Skip to content

ci: move walltime benchmarks to the x64 macro runner - #613

Open
not-matthias wants to merge 2 commits into
mainfrom
cod-4140-migrate-gen2-macro-runners
Open

not-matthias wants to merge 2 commits into
mainfrom
cod-4140-migrate-gen2-macro-runners

Conversation

@not-matthias

@not-matthias not-matthias commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Move the walltime benchmarks to codspeed-macro-x64-ryzen-9950x-ubuntu-24-04:

  • benchmarks: the walltime matrix entry
  • cargo-codspeed is now installed by downloading its release binary, replacing taiki-e/install-action, whose script needs /dev/fd.
  • memtrack-benchmarks now runs on ubuntu-24.04, because it loads eBPF programs from memtrack built at this revision.

Benchmark history starts a new baseline on the new runners.

Refs COD-4140

@not-matthias
not-matthias force-pushed the cod-4140-migrate-gen2-macro-runners branch from 92bf990 to 82f5534 Compare November 5, 2026 11:46
@not-matthias not-matthias changed the title ci: move walltime benchmarks to gen2 macro runners ci: move walltime benchmarks to the x64 macro runner Nov 5, 2026
@not-matthias
not-matthias force-pushed the cod-4140-migrate-gen2-macro-runners branch from 82f5534 to 107715e Compare November 5, 2026 11:48
Switch the walltime and memtrack jobs from codspeed-macro to
codspeed-macro-x64-ryzen-9950x-ubuntu-24-04.
@not-matthias
not-matthias force-pushed the cod-4140-migrate-gen2-macro-runners branch from 107715e to 7241b22 Compare November 5, 2026 11:57
@not-matthias
not-matthias marked this pull request as ready for review November 5, 2026 11:58
@greptile-apps

greptile-apps Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Low impact] The PR appears safe to merge; no actionable issues were found in the runner-label changes.

Summary

Moves both walltime benchmark jobs to codspeed-macro-x64-ryzen-9950x-ubuntu-24-04.

  • Walltime benchmarks now run on the x64 Ryzen 9950X runner.

Reviews (1) · Last reviewed commit: "ci: move walltime benchmarks to the x64 ..." · Reviewed by Greptile

@codspeed

codspeed Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by ×2.9

4 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 8 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ 18 benchmarks with a significant change ran on different hardware

Their base and head results were measured on machines with a different CPU, core count or memory, so the changes might not come from the code. Run base and head on the same runner type.

Learn more about Macro Runners

See affected benchmarks

⚠️ 18 benchmarks with a significant change ran in a different software environment

Their base and head results were measured with a different OS or toolchain, such as compiler, runtime or library versions, so the changes might come from this difference rather than from the code. Open an affected benchmark to see what changed.

See affected benchmarks

⚡ 18 improved benchmarks
✅ 23 untouched benchmarks
⏩ 6 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Walltime write_stack_events 167.9 ms 30.5 ms ×5.5
⚡ Walltime encode_events_realistic[1] 579.6 ms 142 ms ×4.1
⚡ Walltime find_module_events[1000000] 170.6 ms 41.9 ms ×4.1
⚡ Walltime encode_events_realistic[2] 296.8 ms 72.9 ms ×4.1
⚡ Walltime find_module_events_with_stacks[1000000] 178.3 ms 44 ms ×4.1
⚡ Walltime encode_events_realistic[4] 161.2 ms 39.8 ms ×4.1
⚡ Walltime write_events[100000] 71.5 ms 17.7 ms ×4
⚡ Walltime write_events[10000] 6.9 ms 1.7 ms ×4
⚡ Walltime encode_events_realistic[8] 98.5 ms 27.7 ms ×3.6
⚡ Walltime encode_events_realistic[16] 82.9 ms 27.9 ms ×3
⚡ Walltime find_module_events_with_stacks[100000000] 10.1 s 4.2 s ×2.4
⚡ Walltime find_module_events_with_stacks[10000000] 1,007.6 ms 423.6 ms ×2.4
⚡ Walltime find_module_events[10000000] 958.6 ms 404 ms ×2.4
⚡ Walltime find_module_events[100000000] 9.5 s 4 s ×2.4
⚡ Walltime memtrack track dd 1,177.4 ms 725.8 ms +62.22%
⚡ Walltime memtrack track tar (with physical) 9.4 s 6 s +55.66%
⚡ Walltime memtrack track tar 9.3 s 6 s +53.81%
⚡ Walltime memtrack track dd (with physical) 1.3 s 1 s +23.98%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing cod-4140-migrate-gen2-macro-runners (692f674) with main (229413f)2

Open in CodSpeed

Footnotes

  1. 6 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on main (8b62e4a) during the generation of this report, so 229413f was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

…GitHub runners

Download the cargo-codspeed release binary instead of using
taiki-e/install-action, which needs /dev/fd. Run the memtrack benchmarks
on ubuntu-24.04: they load eBPF from a memtrack built at this revision.
@not-matthias
not-matthias force-pushed the cod-4140-migrate-gen2-macro-runners branch from bacddc9 to 692f674 Compare November 5, 2026 13:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant