Skip to content

test(storage): property-test the prune retention invariants - #815

Draft
rominf wants to merge 1 commit into
mainfrom
test/proptest-storage-retention
Draft

rominf wants to merge 1 commit into
mainfrom
test/proptest-storage-retention

Conversation

@rominf

@rominf rominf commented Oct 30, 2026

Copy link
Copy Markdown
Collaborator

Why

rocm storage remove-old-installs deletes multi-gigabyte install folders from a set the user never listed one by one, so a retention bug destroys data the user did not choose to lose. The example tests in storage::tests pin specific scenarios, but the dangerous inputs are combinations: pointers spelled in a different case, two installs whose keys differ only in case, a whole family sharing one runtime id, tied install times, a pointer naming nothing, --keep 0. Property tests over those combinations are what found the case-twin runtime-key defect fixed in #497; this change lands the suite itself so it keeps guarding the policy.

What changes

Property tests for the pure retention policy (select_runtimes_to_remove), 2048 cases each:

  • the install in use, the rollback target, anything the active-runtime marker names and anything ROCm CLI does not own are never selected, including at --keep 0;
  • each retention group keeps exactly its --keep most recent installs that nothing else holds (ties broken by key) and removes the rest; raising --keep never removes more;
  • every install is kept or removed exactly once, and each kept install is reported with the first hold reason that applies, in the order HoldReason declares them;
  • every selected key resolves, through the real select_runtime_manifest, to that same install rather than to a case twin of it.

The expected groups and holds are computed in the test from the --keep help and the matching rules the policy implements, not by calling the policy's helpers, so a defect in one of those helpers cannot hide itself. Generators draw from small, weighted alphabets so the rare inputs above actually occur, plus toolchain records whose devel flag disagrees with their recorded package specs; generator_reach_report is a deterministic measurement that fails if the generators stop reaching them.

A second property (256 cases) applies whole generated plans to real registry files and install folders through the real removal call and checks that the install in use survives.

The checked-in regression seed is a case-twin input that fails when the #497 selector fix is reverted. docs/testing.md gains a "Removing Old Installs" section with the commands.

Verification

  • cargo fmt --all --check and cargo clippy --workspace --all-targets -- -D warnings are clean; cargo test -p rocm --bin rocm storage:: passes 31/31.
  • Mutation evidence — each deliberate defect below makes at least one new property fail:
Deliberate defect in production code Caught by
select_runtime_manifest resolves newest case-insensitive match first (the defect #497 fixed), seed file removed the_deleter_resolves_each_selected_key_to_the_install_the_policy_cleared (fresh run)
retention group ignores the toolchain axis, or reads only the devel flag each_group_keeps_its_most_recent_installs_and_removes_the_rest
keep everything / keep one too few / tie-break reversed each_group_keeps_…
default id holds an install when it names several every_hold_reason_is_the_first_that_applies, each_group_keeps_…
hold reasons checked in a different order (3 swaps) every_hold_reason_is_the_first_that_applies
active pointer compared case-sensitively; read-only install not held the matching never-selected property
active and default holds dropped the_install_in_use_is_never_selected_for_removal, applying_a_whole_plan_never_removes_the_install_in_use
devel extra matched by contains, without trim, or under any package's [ each_group_keeps_…, every_hold_reason_is_…

Not covered: RetentionInputs::from_config (how config and the marker file become pointers) — the properties build RetentionInputs directly. Extras spelled Devel and requirements on other rocm-* packages are not generated.

Scope

Test and docs only; no production code changes. No e2e scenario is added because no user-observable behaviour changes. The retention properties in therock::version_proptests (order independence, recency over version) are left where they are; the new module cross-references them.

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. — N/A, test-only.
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md. — N/A.
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state. — N/A, no user-facing messages changed.

`rocm storage remove-old-installs` deletes multi-gigabyte install folders
from a set the user never listed, so its safety rules are worth testing
over generated input rather than examples alone.

Property tests for the pure policy (`select_runtimes_to_remove`), 2048
cases each:

- the install in use, the rollback target, anything the active-runtime
  marker names and anything ROCm CLI does not own are never selected,
  including at `--keep 0`;
- each retention group keeps exactly its `--keep` most recent installs
  that nothing else holds (ties broken by key) and removes the rest, and
  raising `--keep` never removes more;
- every install is kept or removed exactly once, and each kept install is
  reported with the first hold reason that applies;
- every selected key resolves, through the real `select_runtime_manifest`,
  to that same install rather than to a case twin of it.

The expected groups and holds are computed in the test from the `--keep`
help and the matching rules the policy implements, not with the policy's
own helpers. Generators draw from small, weighted alphabets so case-twin
keys, shared runtime ids, tied install times, flag/specs disagreement on
the toolchain choice and groups with more candidates than `--keep`
actually occur; a deterministic reach report fails if they stop.

A second property, 256 cases, applies whole plans to real registry files
and install folders through the real removal call and checks the install
in use survives.

The checked-in regression seed is a case-twin input that fails when the
runtime selector fix from #497 is reverted. docs/testing.md gains a
"Removing Old Installs" section listing the commands.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant