Skip to content

Run the certified aggregate tests in one CI shard - #2893

Draft
MaxGhenis wants to merge 1 commit into
mainfrom
certified-aggregate-one-shard
Draft

MaxGhenis wants to merge 1 commit into
mainfrom
certified-aggregate-one-shard

Conversation

@MaxGhenis

Copy link
Copy Markdown
Collaborator

Every test in test_aggregate_impact_certified.py now runs in one CI shard, so the other six shards stop building that file's simulations.

The file's module fixture builds them, and each shard that holds one of its tests pays. In merge queue run 42212587066, the fixture held one worker for 8 to 9 minutes at the start of five shards. Those shards peaked at 14.2 to 15.3 GB on 16 GB runners, and the two without it at 10.9 and 12.6 GB.

The duration table spreads these tests because each long entry for this file measured the same fixture, and the deal sends long entries to different shards. Main's table puts them on five shards and #2887's puts them on all seven. That draft lost eight of its 14 shard jobs at 15.4 to 15.7 GB about four minutes in (Test, Release test), and no test failed.

placement_unit now names the file as the thing to place, in the deal and in the hash, and the file weighs as much as its longest entry. It lands on shard 6 under main's table, under #2887's, and under those of #2007, #2010, #2131, #2543 and #2830. Dealing it as one re-deals the rest once: 491 of the 573 listed tests change shard, and the gap between shards' listed loads stays at about 20 seconds. This leaves the table itself alone.

Invariants, tested in test_ci_test_shards.py:

  • The shards partition any collection, and a test's shard depends only on its node id and the table. A test outside SHARED_FIXTURE_FILES keeps the same deal rule and the same hash.
  • Every test in a shared file, listed or not, lands in one shard at any number of shards, with or without the table.
  • No shard's listed load exceeds another's by more than the longest placement unit, whatever order the table lists them in.

Shard 6/7 keeps a peak near 15 GB, so its margin stays where shard 4/7's sits today. This head's run should show that shard there and the other six at about 13 GB or less.

Reproducer: policyengine_uk/tests/code_health/test_ci_test_shards.py::test_the_certified_aggregate_tests_share_one_shard

🤖 Generated with Claude Code

The module fixture in test_aggregate_impact_certified.py builds its own
simulations, and each shard that holds one of the file's tests pays for
them. The duration table deals the file's long entries apart, so main
spreads its nine tests over five shards. Deal and hash the file as one
unit, so one shard pays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@juaristi22 juaristi22 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 94a43ce01ed17357b179fc657e1966fe0e853a98: COMPLETE, no findings or material evidence gaps.

The placement-unit change and its collection/deal/hash integration preserve the tested partition, shared-file co-location and load-bound contracts. All 55 focused sharding/CI-health tests passed in one campaign. No model calculations, certified aggregate fixtures or private data were run.

This validation covers shard placement and the associated collection behavior. Historical memory figures and expected memory savings remain attributed to the author's CI evidence; they were not independently measured here.

The PR remains Draft with CI pending, so this is a completed review comment, not an approval.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants