Skip to content

Clean up temporary benchmark output on errors - #2350

Open
jabrailkhalil wants to merge 1 commit into
google:mainfrom
jabrailkhalil:fix/gbench-temporary-output-cleanup
Open

jabrailkhalil wants to merge 1 commit into
google:mainfrom
jabrailkhalil:fix/gbench-temporary-output-cleanup

Conversation

@jabrailkhalil

Copy link
Copy Markdown

Failed comparisons can leave automatically created result files in the temporary directory. Cleanup now runs after both benchmark execution and result loading, while explicitly requested output files remain untouched. An already-removed temporary file is tolerated; other cleanup errors still surface.

The regression tests use real temporary files and child Python processes. They cover success, nonzero exit, launch failure, invalid JSON, unsupported schema, explicit output on success/failure, and a child removing its temporary output. Four regressions fail on the old implementation; all eight pass with the fix.

Validation:

  • All 40 Python tooling tests pass on Windows / Python 3.12 and Linux / Python 3.12.14, using the repository's pinned NumPy/SciPy requirements.
  • Actual compare.py CLI checks reproduce the leak before the fix and verify cleanup afterward, with original exit codes and explicit output preserved.
  • Ruff 0.14.0 check/format, mypy 1.18.2 and git diff --check pass for the changed Python files.

This changes Python tooling only. The C++ library and compiled bindings were not rebuilt. AUTHORS/CONTRIBUTORS are updated as requested by CONTRIBUTING; the Google CLA already passes on my existing Google contributions.

Fixes #2349.

I used AI assistance for analysis, implementation, tests, validation, review and drafting this contribution.

Remove only tool-owned output when a benchmark fails to launch, exits with an error, or returns unusable results. Preserve explicit output files and tolerate a child removing its temporary output.

AI assistance: analysis, implementation, tests, validation, review, and drafting.
@jabrailkhalil

Copy link
Copy Markdown
Author

The Google CLA check has passed. The workflows currently show action_required; could a maintainer approve their run? The Python tooling suite passes all 40 tests on both Windows and Linux.

AI assistance: checking CI status and drafting this update.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

compare.py leaves temporary output files behind on benchmark failures

1 participant