Repository navigation
test(proc-lifecycle): property-test the process-tree walk - #800
Conversation
A tree stop signals every PID that `collect_process_tree` returns for the root: `terminate_verified` walks once to snapshot the tree it will wait on and again to pick the signal targets. Since the managed-service stop in `rocmd` was routed through `terminate_verified` (#499), that walk decides the kill set for both `rocm services stop` and `rocmd`. A wrong walk either leaves a GPU worker running or signals a process that is not the service's. The walk is now a separate `walk_process_tree` over a parent-to-children table, so it can run against synthetic tables instead of the live `/proc`; `collect_process_tree` still builds the table from `/proc` and hands it over, with no change in behaviour. The properties generate tables of one to six PIDs over a six-PID alphabet, so shared parents, cycles and self-parents come up often. A consistent process tree has no cycle, but the table is built from one `stat` read per PID at different moments, so a cycle from a PID reused mid-scan is rare but cannot be ruled out, and the walk must terminate on it. The properties check the walk against an independent breadth-first oracle (exact reachable set, root first, no PID listed twice) and that every member after the root comes after its parent, as the walk now documents. Each walk runs under a time bound, so a walk that spins on a cycle fails with its input instead of hanging the job. A companion test asserts the generator keeps reaching each of those shapes, including a root that lies on a cycle of two or more PIDs, so the suite cannot go vacuous. 4096 cases by default; `PROPTEST_CASES` overrides that. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review (Claude Code) · d140541
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — no blocking findings
Full review of the whole change.
No ticket named (the PR references #499 only as background context for why the walk matters, not as a dependency or a "fixes" relationship).
Blocking
None
Non-blocking
walk_bounded'srecv_timeoutconflates "timed out" with "sender disconnected" —crates/rocm-core/src/process_tree_properties.rs:62-70
mpsc::Receiver::recv_timeoutreturnsErrboth on a genuine 5s timeout and immediately onRecvTimeoutError::Disconnected(e.g. if the spawned thread panicked instead of hanging). Both cases take the same branch and print "walk_process_tree did not terminate within 5s", which would misattribute a future panic as a hang. Currently dormant:walk_process_tree's body (HashMap insert/push/extend only) has no path that can panic today, and Rust's default panic hook would still print the real panic to stderr first, so the true cause wouldn't be fully hidden — just awkwardly double-reported if the walk ever grows a fallible operation.
Confidence 72 · mechanical · Fix: match on theRecvTimeoutErrorvariant and report "panicked" vs. "timed out" distinctly, or let the panic propagate via.join()instead of a channel.
Decisions for the author
read_parent_pidand the/proctable-assembly side ofcollect_process_treeremain untested by this PR — non-blocking-improvement
The property tests cover the traversal (walk_process_tree) thoroughly but not the parsing that builds its input table from/proc/<pid>/stat(handling ofcommnames containing spaces or)). The PR's own scope section already names this as a deliberate follow-up rather than an oversight, and the traversal is the half carrying the cycle/correctness risk this PR set out to close, so not fixing it now is a reasonable choice — raised here so the author can confirm a follow-up is still intended.
Positive signals
- The independent oracle (
reachable, a BFS overVecDeque) is a genuinely different algorithm from the DFS-with-stack implementation under test, so a bug shared between "how I check" and "how I compute" is unlikely to slip through both — verified by hand-tracing a 2-cycle, a self-parent, and a shared-parent diamond against both. generator_reaches_the_interesting_shapesguards the suite against going vacuous (a generator that silently stopped producing cycles/self-parents would otherwise pass every property while testing nothing interesting) — a good instance of a pattern already established in this crate (examine_driver_proptests.rs'sinputs_reach_every_region) and applied correctly here with its own calibrated floors.- The bounded-thread-plus-
process::exit(101)harness turns a would-be CI hang (on a walk that spins forever on a cycle) into an immediate, diagnosable failure that prints the offending table — thoughtful test-infrastructure design for a traversal that must terminate. - The refactor pulling
walk_process_treeout ofcollect_process_treeis byte-identical in the moved logic (sameseen-before-push order, same stack semantics) — a genuinely behavior-preserving extraction, confirmed by diffing the moved body directly rather than trusting the PR's own "no behaviour change" claim.
Deployment notes
None — test-only change (plus a behavior-preserving internal refactor), Unix/Linux-gated (#[cfg(not(windows))]), no CLI-observable, config, or dependency changes. proptest was already a rocm-core dev-dependency before this PR.
What this covered
Read both changed files completely at the PR head (crates/rocm-core/src/process.rs's full ~2045 lines and the new 223-line process_tree_properties.rs), plus the surrounding callers (signal_process_scope, process_tree_pids, read_parent_pid, the existing mod tests block) and the two precedent files this PR explicitly imitates (examine_driver_proptests.rs, archive_props.rs), in full. Checked history of process.rs and of both precedent proptest files for prior mistakes or caveats (none found relevant), and checked review feedback on PR #499 (the PR that made this walk load-bearing for rocmd) for carryover relevance — that feedback concerned service-record/identity-verification logic, not process-tree traversal, so none of it applies here. Checked the repo's root AGENTS.md rubric in full against the change (license headers, platform-gating conventions, coverage-floor interaction, the Gherkin-scenario requirement, the subprocess/FFI checklist) and found full compliance. CI checked live via the GitHub API: all required checks on d140541 are green (clippy, fmt/prek, the affected-crates test job, Windows build-and-test, Commit signatures + sign-off, CodeQL, etc.); two Strix Halo Ubuntu E2E hardware lanes (nightly/release) show "cancelled" rather than failed, matching a previously-documented benign hardware-lane-scheduling pattern on this repo rather than a regression from this change.
Reviewed via two independent fanned-out passes (a code-coupling pass over both files together, and an agent-instruction-adherence pass against AGENTS.md) plus a design-review pass reasoned through directly. No local cargo/rustc toolchain was available in this environment, so no test was executed directly — findings and verifications are from reading the code and hand-tracing concrete example tables (a 2-cycle, a self-parent, a shared-parent diamond) against both the implementation and its oracle, not from a run; CI's own green Test (affected crates) run is relied on for the executed result. The history/prior-changes pass (Step 4b) was run by the orchestrating pass directly rather than as a separate independent fanned-out worker, because a one-time subagent concurrency limit (shared with unrelated sessions on this host) prevented a third worker from launching; it is therefore not fully independent of the rest of this review, though it used the same blind, evidence-only method. Not reconciled against any discussion on this PR — it currently has zero existing reviews.
🤖 Generated with Claude Code
jussielo-amd
left a comment
There was a problem hiding this comment.
Looks good, thanks Romain. No blockers.
- The
walk_process_treeextraction is a behaviour-preserving move (body matches the old tail ofcollect_process_tree; all callers still go through it). - The BFS oracle is independent of the DFS under test, and the generator guard keeps the suite from going vacuous. Windows gating is complete.
Non-blocking
- Without the
seencheck, a cyclic root makes the spawned thread groworderfor the full 5 s beforeexit(101); that could be a lot of memory and an OOM-kill could pre-empt the diagnostic. Only reachable when the suite is already red, but capping output inside the thread would make failures cleaner. process::exit(101)ends the whole binary under plaincargo test(one test under nextest). The doc comment says so; just confirming it's a deliberate trade.
Heads-up: main has moved process.rs, but merge-tree is clean.
🤖 by agenthub on AMD AgentHub
tests/e2e-cucumber/expectations.tomlfor the fixed ticket ID and removed/narrowed any now-stale xfail rows. — N/A, test-only.docs/architecture.md. — N/A.Why
A tree stop signals every PID that
collect_process_treereturns for the service's root PID:terminate_verifiedwalks once to snapshot the tree it will wait on and again to pick the signal targets. Since #499 routedrocmd's managed-service stop throughterminate_verified, that one walk decides the kill set for bothrocm services stopandrocmd. A wrong walk either leaves a GPU worker running or signals a process that is not the service's, and until now the walk had no test of its own: the existing tree tests only use small real process trees, where cycles and other odd tables never occur.What changes
collect_process_treeintowalk_process_tree, which takes an already-built parent-to-children table.collect_process_treestill reads/procand hands the table over. The walk's doc now states its ordering: root first, every other PID after its parent, no PID twice.crates/rocm-core/src/process_tree_properties.rs, Unix-only, mounted as a child module ofprocess.rslikeexamine_driver_proptests.rs):ppid == 0shape occurs), with unconstrained parents, so shared parents, cycles and self-parents come up in a large share of cases. A consistent process tree has no cycle, but the table is built from onestatread per PID at different moments, so a cycle from a PID reused mid-scan is rare but cannot be ruled out, and the walk must terminate on it. A self-parent cannot come from/procand is included as the one-node case of the same cycle.walk_matches_reachability_oracle: the walk's PID set equals an independent breadth-first reachability oracle; the root leads; no PID is listed twice.parents_are_listed_before_their_children: every member after the root comes after its parent, as the walk documents.generator_reaches_the_interesting_shapesguards against a vacuous suite. Over 1024 deterministic draws the generator reaches a shared parent in 567, a self-parent in 443, a cycle of two or more PIDs in 238, and a root lying on such a cycle (the shape where the walk'sseencheck matters) in 116; the floors are about half of each.therock_version_proptests.rsuses for cheap pure functions;PROPTEST_CASESoverrides it, as inexamine_driver_proptests.rs.No new dependencies:
proptestis already a rocm-core dev-dependency, soCargo.lock,MANIFEST.mdand the third-party notices are untouched.Can the properties fail?
Each mutant was applied to the walk (or the generator), the suite run, and the original restored:
seencheck (a PID signalled twice)root + 1(a stranger adopted)seencheckppid >= pid(no cycles)cycle of two or more PIDs: 0/1024, floor 120The walk itself passes every property on
main; no defect was found in it.Scope
/procside of the walk (read_parent_pidand the table assembly incollect_process_tree); the test'schildren_ofmirrors that table builder rather than exercising it. A parse property forread_parent_pid(comm names containing)or spaces) would be a natural follow-up.rocmd's formerps-based descendant walk is not tested here because fix(rocmd): verify process identity before stopping a managed service #499 removed it; fix(rocmd): verify process identity before stopping a managed service #499 also ships the tests that a recorded PID whose start time no longer matches is refused, and that its subtree survives (stop_managed_service_refuses_a_pid_whose_recorded_identity_no_longer_matches,stop_managed_service_leaves_the_subtree_of_a_refuted_pid_running).AGENTS.md §3
Test-only apart from a private refactor: no CLI output, exit code, file layout or runtime selection changes, so no Gherkin scenario is needed.
Verification
On Linux (WSL2):
cargo test -p e2e-cucumber -p rocm -p rocm-core -p rocm-dash-daemon -p rocm-engine-aim -p rocm-engine-lemonade -p rocm-engine-protocol -p rocm-engine-vllm -p rocmd -p xtask(the setcargo xtask affectedselects): exit 0, 3501 passed.cargo clippy --locked --workspace --all-targets -- -D warnings: exit 0.cargo fmt --all -- --check: exit 0.prek run --from-ref origin/main --to-ref HEAD: all hooks pass except the hawkeye license-header hook, which could not run here because the locally installed hawkeye is older than the versionlicenserc.tomltargets; the new file carries the same SPDX header as its siblings, and theLicense header check (hawkeye)CI job uses the pinned version.Not verified locally: Windows. The new code is
cfg(not(windows))throughout, so a Windows build does not compile it.