ci(unit-tests): capture stacks and database activity when the rest framework job stalls - #16099
Merged
Merged
Conversation
…amework job stalls The Locations-on rest framework job intermittently stops printing near the end of its parallel phase and sits there until the 25 minute step timeout. It has happened at least a dozen times since August on both amd64 and arm64. When a --parallel worker never returns its subsuite, the main process blocks in multiprocessing's pool iterator with no output, so the log cannot say whether a test was stuck, and on what, or a worker died. Run a watchdog on the runner beside `docker compose up`. It only reads from the containers and changes nothing about how the tests run. After 5 minutes with no uwsgi output, and again at 20 minutes, it records the container's processes and memory, a py-spy dump of every python process (taken from the host by host pid, so the container needs no ptrace capability; py-spy is installed only when a dump is needed), and pg_stat_activity with blocking pids plus pg_locks. A new always() step prints the result, so it survives the step being killed by its timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
blakeaowens
approved these changes
Sep 26, 2026
devGregA
approved these changes
Sep 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[sc-15986]
Description
The Locations-on variant of the rest framework unit test job (
test-rest-framework (linux/<arch>, true)) sometimes stops printing near the end of its parallel phase and waits until the 25 minute step timeout kills it. Since late August it has happened at least 12 times, on amd64 and arm64, and only in thetruejobs. For example: jobs 97868263864, 103479249654, 104441871990, 107526847727, 108235271203 and 108259660801. It predates #16091 and #16092.The log can't show why. When a
--parallelworker never returns its subsuite, the main process blocks inmultiprocessing/pool.py(next, called fromdjango/test/runner.py) and prints nothing. That can be reproduced locally by killing a worker. Sixteen local full parallel runs never hung, so the next CI occurrence is the best chance to see it.This PR adds diagnostics for that occurrence. It changes no test behavior and touches only CI.
.github/scripts/unit-tests-hang-watchdog.shruns on the runner next todocker compose up. It only reads from the containers.py-spy dumpof every python process in the container, taken from the host by host PID so the container needs no ptrace capabilitypg_stat_activitywithpg_blocking_pids()pg_locksfor waiting locks and whatever holds thempy-spy(pinned to 0.4.2) is installed only when a dump is needed, so a healthy run pays nothing.Unit testsstep starts the watchdog in the background and kills it when the tests finish. It keeps the exit code fromdocker compose up.Hang diagnosticsstep (if: always()) prints the output file. It still runs when the test step is killed by its timeout, and prints "No stall detected." otherwise.Test results
shellcheckis clean on the new script, with all checks enabled and with the repo's-e SC1091 -e SC2086.actionlinton the workflow reports only the two existing SC2086 notes in theSet-platformstep.postgresand auwsgicontainer that prints and then goes quiet. It dumped on the stall trigger and again on the deadline trigger. Both SQL queries returned rows. It exited quietly when the container stopped.sudo,pipxandpy-spywere stubbed there, because host PIDs on macOS belong to the Docker VM. The realpy-spyattach from a Linux runner is not verified yet; this PR's own rest framework jobs only prove that the healthy path is unchanged.Documentation
CI only, no user-facing change.
🤖 Generated with Claude Code