Skip to content

Add CI job timeouts and build the JOSS draft on main pushes only - #559

Merged
MaxGhenis merged 2 commits into
mainfrom
ci-capacity/ci-hygiene
Oct 8, 2026
Merged

MaxGhenis merged 2 commits into
mainfrom
ci-capacity/ci-hygiene

Conversation

@MaxGhenis

Copy link
Copy Markdown
Contributor

What changes

Two commits, both in .github/workflows, plus a changelog fragment.

  1. Job timeouts instead of GitHub's 6-hour default:

    workflow job timeout slowest observed, 2026-10-06 to 10-08
    pr_code_changes.yaml Test (4-Python matrix) 60 min 27.9 min
    pr_code_changes.yaml BundleVerification (Verify release inputs and bundle packages) 30 0.6
    pr_code_changes.yaml Mypy 20 0.4
    pr_code_changes.yaml Install + smoke-import 15 0.7
    pr_code_changes.yaml Lint, Check changelog fragment 10 0.2
    push.yaml Test 60 25.5
    push.yaml Lint 10 0.1
    pr_docs_changes.yaml Test documentation builds 30 2.0
    draft-pdf.yml Draft PDF 15 no run in the window

    The release jobs in push.yaml (Versioning, Publish, NotifyConsumers) are unchanged.

  2. draft-pdf.yml push trigger limited to main. The JOSS draft built on pushes to every branch and on pull requests. A paper change on a PR branch therefore built the PDF twice: once for the branch push and once for the PR. Pull requests still build it, and so do pushes to main.

  3. Changelog fragment changelog.d/ci-capacity-ci-hygiene.changed.md.

Why

PolicyEngine is on GitHub Team, which runs at most 60 standard GitHub-hosted jobs at once across all the org's repos (limits). That pool was saturated from 2026-10-06 to 10-08, mostly by policyengine-us PR CI. At 2026-10-08 16:28Z all 60 slots were busy and 227 jobs were queued. This repo's jobs waited a median of 34 minutes for a runner in the PR workflow (p90 63).

No CI job here had a timeout, so a hung test job would hold a shared runner for 6 hours. PR runs already cancel superseded runs, so this PR adds no concurrency groups.

How this was measured

Data: every workflow run created in the PolicyEngine org from 2026-10-06 00:00Z to a snapshot at 2026-10-08 16:28:45Z (2.687 days), with all of each run's jobs, fetched from the GitHub REST API.

  • Job duration: completed_at minus started_at, for jobs that got a runner.
  • Queue wait: started_at minus created_at.
  • Draft PDF frequency: git log origin/main --since=2026-07-01 -- paper.md paper.bib architecture.png, which shows one commit, on 2026-08-14.

Expected saving

About 0 runner-min/day today. No job hung in the window and draft-pdf.yml did not run, so this is insurance. A hung Test job is now cut off at 60 minutes instead of 360, and a paper edit on a PR branch builds the draft once instead of twice.

Risks and how they are bounded

  • A slower future suite could trip the 60-minute Test timeout. That is about 2.2× the slowest observed run (27.9 min). Raising it is a one-line change.
  • Pushes to non-main branches with no PR no longer build the JOSS draft. Opening a PR against the branch builds it. The existing draft-pdf.yml concurrency group and paths filter are unchanged.
  • Release jobs keep the default limit, so no publish step can be cut off mid-upload.
  • Release. Merging the changelog fragment cuts a patch release, as with other CI-only changes.

Verification

Run on the branch after git fetch origin and git rebase origin/main (already on the tip, 9663cf26):

  • actionlint 1.7.12 on origin/main's workflows and on the branch's: the same single finding on both, and no new ones. It is a pre-existing note that actions/configure-pages@v3 in push.yaml is too old.
  • The tests that parse these workflow files: python -m pytest -q tests/test_release_tro_generation.py::test_release_workflows_gate_complete_inputs_and_lock tests/test_spm_bundle_bootstrap.py::test_release_checks_published_spm_before_publication_and_after_pypi_visibility. 2 passed.
  • ruff check . (all checks passed) and ruff format --check . (266 files already formatted). No Python changed.
  • uvx towncrier check --compare-with origin/main, which is what .github/check-changelog.sh runs: finds the new fragment.

🤖 Generated with Claude Code

MaxGhenis and others added 2 commits October 8, 2026 13:06
No PR or push CI job had a timeout, so a hung job would hold one of the
org's shared runners for the 6-hour default. Time out the test matrix at
60 minutes (slowest observed from 2026-10-06 to 10-08: 28) and the lint,
changelog, mypy, smoke-import, bundle-verification, docs and paper jobs
at 10 to 30 minutes. Release jobs (Versioning, Publish, NotifyConsumers)
are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
draft-pdf.yml ran on pushes to every branch and on pull requests, so a
paper change on a PR branch built the draft twice. Limit the push
trigger to main; pull requests still build it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Landing on the merge gates. Audit at head 8a02987e348b360123fb979a52f5e6a0a42343b1:

  • gh pr checks exits 0 (13 passed, 0 skipped by design); mergeable is MERGEABLE; not a draft; no changes-requested review.
  • Independent review 1: an adversarial reviewer agent in the authoring session (local_78df3fa0) approved this head. Verdict file: ~/reviews/pe-actions-concurrency-2026-10-08/reviews-from-78df3fa0/policyengine.py-559.md.
  • Independent review 2: the session landing this (local_cfdb0a5b, which did not write the change) read the full diff at this head and agrees. The change touches only CI workflow configuration.
  • Context: the PolicyEngine org is at GitHub's limit of 60 concurrent standard-runner jobs for the Team plan, so superseded and hung jobs delay every repo's checks.

@MaxGhenis
MaxGhenis merged commit 166ef6c into main Oct 8, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant