[hua7450]

What and why

PR CI's longest path is the Rest Python tests, which run on two chained runners. On run 40014592006 (#10416), Full Suite - Rest (Python, shard-1 + variables) took 51 minutes, then Full Suite - Rest (Python + variables) took 81 and finished 132 minutes in. Baseline → Contrib had finished at 68. The second runner's 81 minutes sit close to its 90-minute timeout. The core group alone takes 50 minutes; it took 12 on October 8. 27 of those minutes are one file, core/test_behavioral_response_measurement_copies.py (#9913).

With the Enterprise runner capacity, this PR runs the Rest Python tests on parallel shards that balance themselves:

• RestPlan (new, about a minute) downloads the JUnit and GNU time reports of the newest successful release run on main and runs policyengine_us/tests/rest_python_shards.py plan. The planner reads the groups from the Makefile's dry run and pytest's collection, splits any group longer than 25 minutes into time-balanced chunks, and deals processes longest-first over 4–12 shards, using the fewest that keep the longest near 25 minutes or the longest single process.
• RestPython is a matrix of those shards (Full Suite - Rest (Python shard N)). Each runs its processes one after another, each with its own JUnit and GNU time report, and the variables YAML runs on the lightest shard.
• Rest (Check Rest Python shards, renamed from Full Suite - Rest (Python + variables), which suggested it ran tests) runs no tests: it fails unless the plan and every shard succeeded.

What stays the same:

• Memory isolation: a process runs files of one Makefile group only, in that group's order, so no process holds more than its group did.
• Coverage: each file runs exactly once. test_rest_python_groups.py checks that for plans made with no, measured, partly measured, skewed, tiny and huge times.
• Local runs: make test-other-python-rest runs every group in order, as before.
• Other CI: Baseline-then-Contrib ordering, the Microsimulation job and push.yaml parity, with Publish now needing all three Rest jobs.

As tests are added, each release run's reports rebalance the next plan. Long groups split on their own, and the shard count grows up to 12, so CI has no shard lists to maintain. Only successful release runs count, because a failed one can lack a lost runner's reports and skew the estimates. A file two reports cover (a re-run's leftover) counts once. Without reports (a fork, or expired artifacts), every file counts the same and the plan still runs every file once. A Makefile group option that a shard could not pass on to pytest, such as -p or --deselect, now fails the plan instead of being dropped.

Expected effect

Plans from real reports:

Reports from	Shards	Longest shard (estimate)
Release run 40016344220	5	26.4 min
PR run 40014592006	5	27.8 min
PR run 40012270922	5	29.5 min

This PR's first run, 40038526568, planned 5 shards from release run 40016344220:

Shard	Processes	Planned	Actual job time
1	core-2 (the 27-minute file)	26.4	24.8
2	remaining, variables YAML	23.6	24.6
3	core-1, policy-parameters	24.6	27.9
4	cached-arrays, heavy-a, policy	23.6	19.2
5	heavy-b, core-heavy, remaining-heavy	23.8	22.1

Every Rest Python test finished 29 minutes after the plan started, instead of about 130 on two chained runners, which leaves Baseline → Contrib (about 70 minutes) as a PR's longest path. The shards used 119 runner-minutes, against 132 before. The floor is the 27-minute file, which gets a shard to itself; splitting it would be a separate change.

Runners per PR go from 27 to 27 plus the shard count (32 with today's 5 shards), at most 39. Runner-minutes stay about the same: the tests are unchanged, and each extra runner or process adds about 30 seconds of setup and 1.5 minutes of interpreter and model start.

Test plan

• For policy changes: not applicable (CI only).
• Source conflicts: not applicable.
• make format
• Changelog fragment in changelog.d/
• Fork probe run 40035075843 of the GitHub Actions behaviour this relies on:
  • a matrix fed by a skipped job's output skips cleanly (non-release pushes to main);
  • process lists containing quotes, $() and backticks reach the shard intact;
  • "Re-run failed jobs" keeps the plan and reruns only the failed shard and the summary.
• Local:
  • the code-health guards (test_rest_python_groups.py, test_pr_runner_budget.py, test_make_test_matches_ci.py, test_pr_workflow_skips_drafts.py): 72 passed;
  • 9 planner mutants each fail a guard: a shard count that never grows, always maxes out or ignores the longest process; variables on the heaviest shard; ignored failures; summed duplicate times; dropped group options; a shell-joined command; a chunk that drops a file;
  • the timing download step works against this repository;
  • the planner gives the plans above;
  • a planned file list collects the same tests as its group's command;
  • a real two-process shard runs through rest_python_shards.py run.
• This PR's CI (run 40038526568): every shard passed, and its JUnit reports match main's file for file:
  • the groups main's first runner covers: 99 files, 2,824 tests, against 2,792 in the release run for the same base commit; the only difference is the +32 tests in the rewritten test_rest_python_groups.py;
  • the groups main's second runner covers: 148 files, 2,887 tests, against 2,887 in release run 40016344220;
  • the variables YAML: 69 tests, against 69;
  • results: 5,700 passed, 11 skipped, 0 failed;
  • test_batched.py (the YAML batch runner, collected for its name) has no tests in either layout.

#10400, #10401 and #10058 also edit the Makefile's Rest comments or test_rest_python_groups.py. Their changes work with the planner, which reads whatever groups and ignores the Makefile defines. Whichever merges second has a small text conflict to resolve.

axiom: n/a: CI scheduling and workflow guard changes only.

🤖 Generated with Claude Code