Skip to content

ci: add memtrack walltime benchmarks to CI - #537

Merged
not-matthias merged 3 commits into
mainfrom
cod-3093-memtrack-data-format-improvements
Sep 18, 2026
Merged

not-matthias merged 3 commits into
mainfrom
cod-3093-memtrack-data-format-improvements

Conversation

@not-matthias

Copy link
Copy Markdown
Member

Adds a memtrack-benchmarks CI job that runs codspeed-memtrack track against a few representative workloads (ls, dd, tar), each with RSS-only and RSS+rmap variants, to catch regressions in memtrack's own overhead.

Runs both walltime (execution time) and memory (memtrack's own peak RSS/allocations while tracking each workload) modes against the same crates/memtrack/codspeed.yml config.

@codspeed

codspeed Bot commented Sep 16, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 17 untouched benchmarks
🆕 6 new benchmarks

Performance Changes

Mode Benchmark BASE HEAD Efficiency
🆕 WallTime memtrack track tar N/A 8.2 s N/A
🆕 WallTime memtrack track tar (with physical) N/A 9.7 s N/A
🆕 WallTime memtrack track dd N/A 1.3 s N/A
🆕 WallTime memtrack track dd (with physical) N/A 1.4 s N/A
🆕 WallTime memtrack track ls N/A 1.1 s N/A
🆕 WallTime memtrack track ls (with physical) N/A 1.2 s N/A

Comparing cod-3093-memtrack-data-format-improvements (6c953d6) with main (5af022c)

Open in CodSpeed

@not-matthias
not-matthias force-pushed the cod-3093-memtrack-data-format-improvements branch 2 times, most recently from 596e932 to 5f67554 Compare September 16, 2026 10:29
@not-matthias
not-matthias changed the base branch from main to cod-3492-memtrack-classic-tracepoints-cannot-attach-inside-the September 16, 2026 10:29
Base automatically changed from cod-3492-memtrack-classic-tracepoints-cannot-attach-inside-the to main September 16, 2026 10:49
@not-matthias
not-matthias force-pushed the cod-3093-memtrack-data-format-improvements branch 2 times, most recently from b7adccd to 3efb21e Compare September 16, 2026 14:49
@not-matthias
not-matthias marked this pull request as ready for review September 16, 2026 14:51
@greptile-apps

greptile-apps Bot commented Sep 16, 2026

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

The pull request appears safe to merge, with no outstanding actionable defects.

Summary

Adds dedicated CI benchmarks that measure the walltime overhead of running memtrack against representative ls, dd, and tar workloads, with and without physical-memory tracking.

  • Runs the benchmarks on the dedicated codspeed-macro runner.
  • Installs and configures the matching memtrack and CodSpeed runner revisions.
  • Batches ring-buffer events to reduce channel-allocation overhead.
  • Flushes partial event batches on timeout, explicit drain, and shutdown.
  • Updates all internal event consumers to flatten the new batches.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[CI workload matrix: ls, dd, tar] --> B[CodSpeed walltime runner]
  B --> C[codspeed-memtrack track]
  C --> D[Tracked workload]
  D --> E[eBPF ring buffer]
  E --> F[Batch up to 1024 events]
  F --> G[Flush on capacity, timeout, drain, or shutdown]
  G --> H[Flatten event batches]
  H --> I[Encode memtrack artifact]
Loading

Reviews (5) · Last reviewed commit: "ci: cache runner builds across memtrack ..."

Comment thread .github/workflows/ci.yml Outdated
Comment thread crates/memtrack/src/ebpf/poller.rs
@not-matthias
not-matthias force-pushed the cod-3093-memtrack-data-format-improvements branch from 21ce920 to 829da2f Compare September 16, 2026 16:17

@GuillaumeLagrange GuillaumeLagrange left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

olgtm, good idea!

Comment thread .github/workflows/ci.yml
Comment thread crates/memtrack/codspeed.yml Outdated
Comment thread crates/runner-shared/benches/memtrack_writer.rs Outdated

Copy link
Copy Markdown
Contributor

Memtrack walltime benchmarks took 19m to run, maybe we can shard something before introducing a 20m delay in the CI pipeline? 😇

@not-matthias
not-matthias force-pushed the cod-3093-memtrack-data-format-improvements branch 5 times, most recently from 2c36a70 to bcade9d Compare September 18, 2026 14:45
Comment thread .github/workflows/ci.yml
The poll thread sent one item per channel message, and std's mpsc allocates a
31-slot block per 31 messages, so a run that captured 704k events also
allocated 22.7k blocks purely to hand them over.

The callback now fills a shared 1024-item buffer and sends it whole. Partial
batches are flushed after every poll, after the drain-path consume before its
ack, and on the shutdown consume, so `drain()` keeps promising that all
pending entries sit in the channel once it returns.

Consumers take `Vec<T>` and flatten: the encode pipeline keeps its
`IntoIterator<Item = MemtrackEvent>` contract unchanged.
Install codspeed-runner and memtrack once via baptiste0928/cargo-install, pointed at a shared cached target directory, and let the runner action handle instrument caching and benchmark execution instead of a separate debug build.
@not-matthias
not-matthias force-pushed the cod-3093-memtrack-data-format-improvements branch from bcade9d to 6c953d6 Compare September 18, 2026 15:04
@not-matthias
not-matthias merged commit 6c953d6 into main Sep 18, 2026
51 checks passed
@not-matthias
not-matthias deleted the cod-3093-memtrack-data-format-improvements branch September 18, 2026 15:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants