Skip to content

ci(unit-tests): capture stacks and database activity when the rest framework job stalls - #16099

Merged
Maffooch merged 1 commit into
bugfixfrom
ci/rest-framework-hang-diagnostics
Sep 26, 2026
Merged

Maffooch merged 1 commit into
bugfixfrom
ci/rest-framework-hang-diagnostics

Conversation

@Maffooch

@Maffooch Maffooch commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

[sc-15986]

Description

The Locations-on variant of the rest framework unit test job (test-rest-framework (linux/<arch>, true)) sometimes stops printing near the end of its parallel phase and waits until the 25 minute step timeout kills it. Since late August it has happened at least 12 times, on amd64 and arm64, and only in the true jobs. For example: jobs 97868263864, 103479249654, 104441871990, 107526847727, 108235271203 and 108259660801. It predates #16091 and #16092.

The log can't show why. When a --parallel worker never returns its subsuite, the main process blocks in multiprocessing/pool.py (next, called from django/test/runner.py) and prints nothing. That can be reproduced locally by killing a worker. Sixteen local full parallel runs never hung, so the next CI occurrence is the best chance to see it.

This PR adds diagnostics for that occurrence. It changes no test behavior and touches only CI.

  • .github/scripts/unit-tests-hang-watchdog.sh runs on the runner next to docker compose up. It only reads from the containers.
  • It dumps after 5 minutes with no output from the uwsgi container. It dumps again at 20 minutes, before the timeout. It stops after 3 dumps. Each dump contains:
    • the container's processes and memory
    • a py-spy dump of every python process in the container, taken from the host by host PID so the container needs no ptrace capability
    • pg_stat_activity with pg_blocking_pids()
    • pg_locks for waiting locks and whatever holds them
  • py-spy (pinned to 0.4.2) is installed only when a dump is needed, so a healthy run pays nothing.
  • The Unit tests step starts the watchdog in the background and kills it when the tests finish. It keeps the exit code from docker compose up.
  • A new Hang diagnostics step (if: always()) prints the output file. It still runs when the test step is killed by its timeout, and prints "No stall detected." otherwise.

Test results

  • shellcheck is clean on the new script, with all checks enabled and with the repo's -e SC1091 -e SC2086.
  • actionlint on the workflow reports only the two existing SC2086 notes in the Set-platform step.
  • I ran the script locally against a small compose project with postgres and a uwsgi container that prints and then goes quiet. It dumped on the stall trigger and again on the deadline trigger. Both SQL queries returned rows. It exited quietly when the container stopped. sudo, pipx and py-spy were stubbed there, because host PIDs on macOS belong to the Docker VM. The real py-spy attach from a Linux runner is not verified yet; this PR's own rest framework jobs only prove that the healthy path is unchanged.

Documentation

CI only, no user-facing change.

🤖 Generated with Claude Code

…amework job stalls

The Locations-on rest framework job intermittently stops printing near the
end of its parallel phase and sits there until the 25 minute step timeout.
It has happened at least a dozen times since August on both amd64 and arm64.
When a --parallel worker never returns its subsuite, the main process blocks
in multiprocessing's pool iterator with no output, so the log cannot say
whether a test was stuck, and on what, or a worker died.

Run a watchdog on the runner beside `docker compose up`. It only reads from
the containers and changes nothing about how the tests run. After 5 minutes
with no uwsgi output, and again at 20 minutes, it records the container's
processes and memory, a py-spy dump of every python process (taken from the
host by host pid, so the container needs no ptrace capability; py-spy is
installed only when a dump is needed), and pg_stat_activity with blocking
pids plus pg_locks. A new always() step prints the result, so it survives
the step being killed by its timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Maffooch Maffooch added this to the 3.3.300 milestone Sep 25, 2026
@Maffooch
Maffooch enabled auto-merge September 26, 2026 00:15
@Maffooch
Maffooch added this pull request to the merge queue Sep 26, 2026
Merged via the queue into bugfix with commit c511c2e Sep 26, 2026
49 checks passed
@Maffooch
Maffooch deleted the ci/rest-framework-hang-diagnostics branch September 26, 2026 00:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants