Skip to content

Give each Python test shard 75 minutes - #2626

Merged
MaxGhenis merged 1 commit into
mainfrom
ci-shard-timeout-75
Oct 24, 2026
Merged

MaxGhenis merged 1 commit into
mainfrom
ci-shard-timeout-75

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

The Python test shards get 75 minutes instead of 60.

Why

With five shards, the Python jobs took 45 to 59 minutes on 23 October (the table rounds each down to whole minutes):

Run Event Shard minutes
40027855419 merge_group 46, 53, 53, 54, 52
40027962439 merge_group 50, 58, 50, 58, 45
40029283977 pull_request 52, 58, 53, 54, 54
40032765600 pull_request 47, 57, 56, 55, 54
40034704845 merge_group 47, 57, 51, 50, 52

In the merge-queue run for #2213 (40034703741), shard 4/5's tests all passed in 59m32s (1756 passed, 11 skipped). The 60-minute limit then cancelled the job 9 seconds into its Codecov upload. Result's own audit found that every collected test ran once and passed or had an allowlisted outcome, but Test / Result failed on Python: cancelled, so the queue removed #2213 (reason failed_checks) and rebuilt every group behind it. With the queue 44 deep and the shards a few minutes under the limit, any slow runner costs a pull request its place and the queue an hour.

What changes

  • .github/workflows/test-suite.yaml: the Python job's timeout-minutes goes from 60 to 75, and its comment records the measurements above. A hung shard is still stopped.
  • changelog.d/ci-shard-timeout-75.changed.md.

The required checks (Test / Result, Lint) and the shard count are unchanged.

Alternative not taken

A sixth shard would cut each shard to about 40–50 minutes and shorten every merge group. It would also add one runner to every run, and this organisation has had runs queued for hours for lack of a runner (#2099). That trade belongs in its own pull request.

Follow-up

The Codecov step fails with Token required because branch is protected in merge-group runs and in at least some pull-request runs (shard 1/5 of runs 40034704845, 40027855419 and 40032765600). fail_ci_if_error: false hides the failure, but the step still counts against the job's limit. Fixing or skipping it is a separate change.

Validation

Workflow-only change; no model code or tests change. The YAML parses, and git diff --check is clean. A merge group's workflow comes from the group's own merge commit, so the new limit applies to the first group that contains this pull request and to every one after it.

🤖 Generated with Claude Code

With five shards the Python test jobs took 45 to 59 minutes on 23 October.
In the merge-queue run for #2213 (run 40034703741), shard 4/5's tests all
passed in 59m32s, and the 60-minute limit then cancelled the job during its
Codecov upload. The queue took #2213 out and rebuilt every group behind it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis force-pushed the ci-shard-timeout-75 branch from c9de5d5 to 75bdc6b Compare October 23, 2026 23:55
@MaxGhenis
MaxGhenis added this pull request to the merge queue Oct 24, 2026
@MaxGhenis
MaxGhenis removed this pull request from the merge queue due to a manual request Oct 24, 2026
@MaxGhenis
MaxGhenis added this pull request to the merge queue Oct 24, 2026
Merged via the queue into main with commit 579e06e Oct 24, 2026
33 of 34 checks passed
@MaxGhenis
MaxGhenis deleted the ci-shard-timeout-75 branch October 24, 2026 02:39
@MaxGhenis

Copy link
Copy Markdown
Collaborator Author

Merged by the UK hub (Claude session 8e19f9d5) at head 75bdc6bf2, through the merge queue with the head pinned.

  • What it does: it gives each Python test shard 75 minutes instead of 60. With five shards the jobs took 45 to 59 minutes on 23 October. In merge-queue run 40034703741, shard 4/5's tests all passed in 59m32s, and the 60-minute limit then cancelled the job during its Codecov upload. That ejected Add historical Housing Benefit non-dependant deduction rates #2213 from the queue and rebuilt every group behind it.
  • Review: independent Opus 5.5 review (the Codex lanes were closed), in two rounds.
    • Round 1 found the change right but blocked on wording: the comment said a test was still running, and the log showed every test had passed.
    • The hub amended the comment, commit message, changelog and body.
    • Round 2 APPROVE at this exact head:
      • it checked all 25 shard timings against the run data;
      • it confirmed the 59m32s and Codecov cancellation, and the queue removal for failed_checks;
      • nothing else depends on 60 minutes: the queue's response timeout is 360, and Result needs only the job results.
  • CI: every check passes at this head, including the Release test suite that versioning.yaml runs for workflow changes.
  • Impact: CI only. No model code, parameters or tests change. The new limit applies to every merge group that contains this pull request, and to every one after it.
  • Follow-up: the Codecov step's protected-branch token failure, which is hidden by fail_ci_if_error: false but counts against the limit, is a separate task.
  • Why now: the merge queue is 44 deep before the Budget, with shards a few minutes under the limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant