Skip to content

test(fspy): benchmark tracking overhead#566

Draft
wan9chi wants to merge 5 commits into
mainfrom
agent/benchmark-fspy-overhead
Draft

test(fspy): benchmark tracking overhead#566
wan9chi wants to merge 5 commits into
mainfrom
agent/benchmark-fspy-overhead

Conversation

@wan9chi

@wan9chi wan9chi commented Jul 23, 2026

Copy link
Copy Markdown
Member

Motivation

fspy does not have a stable cross-platform benchmark that isolates the wall-clock cost it adds to tracked child processes. A small synthetic workload makes regressions visible without introducing real-world repository variance.

Changes

  • Add one workload: four threads each open and immediately close 2,048 pre-created files.
  • Measure the same dynamically linked target untracked and tracked on Linux, macOS, and Windows Namespace runners.
  • Add a statically linked Linux variant to exercise seccomp user notification as well as LD_PRELOAD.
  • Normalize each result as tracked/untracked overhead, then compare with the latest result from the same PR or fall back to main.
  • Persist results as workflow artifacts and update a sticky PR comment, including fork PRs through a trusted workflow_run reporter.

Add a single multithreaded open-and-close workload, a Linux static variant, and three-platform Namespace reporting with PR/main baselines.

Co-authored-by: GPT-5.6 Codex <codex@openai.com>

wan9chi commented Jul 23, 2026

Copy link
Copy Markdown
Member Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

@socket-security

socket-security Bot commented Jul 23, 2026

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Updatedcargo/​serde_json@​1.0.149 ⏵ 1.0.1518210093100100
Addedcargo/​criterion2@​3.0.49810098100100

View full report

wan9chi and others added 4 commits July 24, 2026 00:57
Make the reporter shell explicit and document why its text-only workflow_run boundary is intentionally safe.

Co-authored-by: GPT-5.6 Codex <codex@openai.com>
Allow upload-artifact to include the dot-prefixed benchmark results directory.

Co-authored-by: GPT-5.6 Codex <codex@openai.com>
Use a shorter process-level sampling window and inherit child stderr so runtime failures remain diagnosable.

Co-authored-by: GPT-5.6 Codex <codex@openai.com>
Use pull-request write permission for the sticky benchmark report endpoint.

Co-authored-by: GPT-5.6 Codex <codex@openai.com>
@github-actions

Copy link
Copy Markdown

fspy benchmark

Results for commit ecf9e9af19c581f156188679cb2a9e8d458c9ba8. Each platform uses its latest result from this PR, falling back to main.

Linux

Compared with previous PR run at 594f3ce2.

Target Untracked Tracked fspy overhead Normalized change
dynamic 15.832 ms 22.848 ms +44.31% -7.09%
static 19.230 ms 63.804 ms +231.79% -10.95%

Workload: 4 threads, 2,048 files per thread, 8,192 total open-and-close operations.

X64 · AMD EPYC · run 30028036239

Macos

Compared with previous PR run at 594f3ce2.

Target Untracked Tracked fspy overhead Normalized change
dynamic 27.331 ms 31.002 ms +13.43% -6.92%

Workload: 4 threads, 2,048 files per thread, 8,192 total open-and-close operations.

ARM64 · Apple M4 Pro (Virtual) · run 30028036239

Windows

Compared with previous PR run at 594f3ce2.

Target Untracked Tracked fspy overhead Normalized change
dynamic 75.048 ms 66.611 ms -11.24% -5.15%

Workload: 4 threads, 2,048 files per thread, 8,192 total open-and-close operations.

X64 · AMD EPYC-v4 Processor · run 30028036239

This comment is updated by the Fspy Benchmark workflow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant