Repository navigation
Restore validation image cleanup - #1323
burtenshaw wants to merge 3 commits into
Conversation
* feat: clean up owned validation images * docs: clarify runtime validation coverage * fix: verify image cleanup by inventory * fix: scope cleanup inventory to owned images
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
Alignment Review Report
Automated Checks
- Lint: PASS —
usort check/ruff format --check/ruff checkall pass on every file this PR touches (src/openenv/validation/providers/docker.py,src/openenv/validation/runner.py,src/openenv/cli/commands/validate.py, and the 4 touched test files).bash .claude/hooks/lint.shitself exit-1's, but only on ~58 pre-existing files elsewhere in the tree (e.g.wildfire_envexamples,harness/README.md) plus the two long-standingusort-only files called out inAGENTS.md(tests/envs/test_grid_world.py,tests/envs/test_julia_env.py) — none of those are in this diff. CI's ownlintjob on this PR is green, confirming the same. - Debug code: CLEAN —
check-debug.shlists only pre-existingprint()/TODO hits elsewhere insrc/(e.g.containers/test_local_docker_provider.py,cli/__main__.py); none fall on a line this PR changed. The twoprint(...)matches inside this diff are both inside test-fixture strings (tests/test_cli/test_validate.py:450,tests/test_validation/integration/test_docker_lifecycle.py:93/101), not executable debug statements. - Tests: ran the new/changed suites (
tests/test_validation/test_docker_provider.py,tests/test_validation/test_runtime_execution.py,tests/test_cli/test_validate.py) — 137 passed. Fullpytest tests/(excluding the pre-existingsmolagents-missing collection error intest_coding_codeact_env.py) — 3816 passed, 179 skipped, 2 failed; the 2 failures (tests/envs/test_websockets.py::TestProtocolWebSocketClient::test_protocol_client_state/_multiple_episodes) reproduce identically on the base commit (cad68dd3, pre-this-PR) in an A/B worktree check, so they're pre-existing and unrelated to this change. CI'stest (3.11)/test (3.12)jobs on this PR are both green; the Docker-backedRuntime validation / Linux Dockerworkflow (which exercises the new--cleanup-imagesintegration tests directly against the Docker daemon) was still pending at review time.
Open RFCs Context
- RFC 008 — Environment Auto-Validation (In Review, @zkwentz) is the only open RFC touching this area (
src/openenv/validation/**,openenv validate). Its "Level 2 execution amendment" explicitly requires the Docker provider to own "idempotent cleanup on success, failure, timeout and cancellation" and that "cleanup evidence must establish that no run-owned subject remains." This PR extends that exact idiom from containers to the optional build image — opt-in via--cleanup-images, ownership-tagged, verified by inventory rather than trusting a command's exit code, and reported incleanup.json. No other open/In-Review/Draft RFC (000–006, 010–012) mentions Docker image lifecycle, so there's no cross-RFC conflict.
Tier 1: Fixes Required
None. All PR-authored files are lint-clean, debug-clean, and fully covered by passing tests (including deliberately adversarial cases: idempotent re-cleanup, shared-tag preservation, ownership/identity mismatches, inventory-overflow/engine-failure handling, cleanup-during-interrupt/teardown-failure, and rejection of non-Docker providers).
Tier 2: Alignment Discussion
Principle Conflicts
None identified. The change stays entirely inside the validation subsystem's local/author-side lane (no hub/runner/queue machinery added, per RFC 008's non-goals); it manages Docker images via plain subprocess argv lists (no shell, no injection surface — image_owner is always an internally generated "validation-" + uuid4().hex, never user input); and on cleanup failure it only surfaces the exception type name in evidence/report output (never str(exc)), consistent with the "no credential/detail exposure" posture elsewhere in this module (verified by test_image_cleanup_error_preserves_primary_failure, which asserts the raw exception text never reaches cleanup.json or the report).
RFC Conflicts
None identified. This is a faithful, incremental extension of RFC 008's Level-2 Docker-provider cleanup contract (containers → images), not a divergence from it. One non-blocking sequencing note worth surfacing to the RFC author: this is a re-application of previously-reviewed and already-merged-to-main content (#1294, merged at 782f0717) onto a review/discovery branch, after an intervening full-stack revert (#1319) and rebuild (#1322). The PR description already states this plainly ("exactly match #1294's final squash"), and git diff against the real #1294 squash confirms the file list and line counts line up — so this is bookkeeping/re-sequencing within the RFC-008 stack rather than a new design decision, and nothing here needs re-litigating on its merits.
ALIGNMENT FLAG: Re-landing previously-merged RFC-008 code on a review-stack branch
- Principle/RFC at stake: RFC 008 (In Review, @zkwentz) — general stack-sequencing hygiene, not a design conflict.
- The concern: Because this duplicates already-shipped history on a different branch line (
ben/rfc008-discovery-review→ben/rfc008-cleanup-review), it's worth a human confirming the intended merge target/order for this stack before any further PRs build on top ofben/rfc008-cleanup-review, so the RFC-008 branch graph doesn't end up with two independent copies of the same commit content landing onmainthrough different paths. - Suggested reviewer: @zkwentz (RFC 008 author) and @burtenshaw (author of this PR and of the original #1294/#1319/#1322 stack).
Summary
- 0 mechanical issues to fix
- 1 alignment point for human review (non-blocking sequencing note)
- 0 RFC conflicts to discuss (RFC 008 note above is sequencing, not a conflict)
Sent by Cursor Automation: Pre-review
There was a problem hiding this comment.
Reviewed at 7d84e20b and ran the pinned lab on Docker 29.7.2 (linux/amd64). fast, protocol and docker (30/30) pass. Image ownership and removal are checked by listing rather than error text, consistent with #1259; cleanup is opt-in, and --no-prune keeps shared layers. No issues.
Preserve opt-in owned-image cleanup alongside parent SIGTERM handling, unsupported-discovery skips and network acceptance inventory.
|
Closing this optional image-tag cleanup follow-up as part of the validation landing cleanup requested by the author. It adds no missing runtime check and is not required for the retained container teardown checks. The necessary telemetry, repeatability and discovery work remains in #1320–#1322; the branch is retained for any future cleanup-only proposal. |


Summary
This PR restores #1294's optional cleanup of run-owned image tags after validation and replay, preserving shared images and build caches. Recreated after #1319; follow-up to #1177, building on #1322.
Evidence
The source tree and incremental patch exactly match #1294's final squash, including its latest inventory/ownership fixes. On this new head: 791 fast and 41 protocol tests pass in the pinned lab; artifact hashes and touched-file lint pass. Fresh GitHub CI is separate.
Merge Danger
Two-way; affects run-owned Docker image cleanup. Depends on #1322. Leave open for human review; do not merge.