ci: shard the push-path contract corpus across four test-linux legs

[kyle-sexton]

Closes #3705

Summary

On a push to main, test-linux used to run the whole plugin contract corpus in one job, which took about 550 s. This PR splits the corpus across four test-linux legs on every event, the same way the affected selection is already split on a pull request. The job key does not change, so the six-job layout, ci-status.needs and lane coverage all stay the same.

Two metrics move: how long the test-linux lane takes on push, and how long a push waits for ci-status to go green. The whole push run does not get shorter. test-windows is not in ci-status's needs, and it took 713 s on run 34815969269, 603 s of that in the apply-rename.test.sh step. The run's total time stays tied to test-windows until the separate PR that cuts that suite lands.

Fix

• scripts/run-plugin-tests.sh has a new --shard I/N option. It splits the sorted list of discovered suites by index modulo N, which is how affected-tests.sh --shard already works:
  • No suite lands on two legs, and the legs together cover the whole corpus.
  • A leg with no suites exits 0.
  • The check for stale serial-allowlist entries still compares against every discovered suite, not just the leg's.
  • Each leg runs its own serial suites first.
• ci.yml test-linux:
  • The matrix is now leg: [0, 1, 2, 3] on every event. LEG and LEGS are set once at job level.
  • On push, the contract step runs scripts/run-plugin-tests.sh --jobs 3 --shard "$LEG/$LEGS".
  • On a PR, the UNMAPPED fallback uses the same shard on every leg. It no longer runs only on leg 0.
  • Every step after the contract step now runs on leg 0 only, on both events. That covers the scripts/ suites, manifest validation, the sync --check/--check-bump steps, the Node sub-projects and disk-hygiene. Before, they ran on all four legs of a PR and once on the single push leg.
  • The leg check is a first line [ "$LEG" = 0 ] || exit 0, not a step if:. scripts/check-docs-only-gate.sh accepts exactly one condition form on a gated step. A step that exits 0 reports success, never skipped, and ci-status fails on skipped.
  • The comment above the matrix is rewritten to match.

Verification

Local checks:

• bash scripts/run-plugin-tests.test.sh: PASS=67 FAIL=0. The new cases check that:
  • the legs do not overlap and together equal the unsharded run, including when an allowlisted suite sits on another leg
  • a leg's serial subset runs
  • an empty leg exits 0
  • the bad specs 2/2, x, 1, 1/0 and '' exit 2
• bash scripts/affected-tests.test.sh: PASS=76 FAIL=0. It includes the live check that ci.yml passes the leg into --shard.
• scripts/check-docs-only-gate.sh --check passes. scripts/check-lane-coverage.sh --check passes. Their test suites pass 64/0 and 40/0.
• actionlint, zizmor --offline, shellcheck, shfmt -d and check-shell-portability.sh --paths all pass.

Live measurement: the probe commit b0c544f, reverted in 8e98db1, forced this PR's contract step onto the push branch. Run 34884762048, all four legs green:

leg	job wall	contract step	suites (serial)
0	192 s	79 s	87 (3)
1	205 s	146 s	87 (4)
2	156 s	110 s	87 (3)
3	158 s	120 s	86 (8)

The legs sum to 347 suites, which is the whole corpus. The serial counts sum to 18, the whole allowlist. Leg 0 also carries the roughly 80 s of steps that run only once. Leg 1 is the slowest at 205 s and runs none of them. Moving those steps to any other leg would push that leg above 205 s, so leg 0 is the best place for them.

Before, on push:

• Run 34815969269: test-linux 556 s, of which the contract step took 444 s. ci-status finished 583 s into the run.
• Run 34750258935: test-linux 549 s. ci-status finished 584 s into the run.

After:

• The test-linux lane takes as long as its slowest leg, 205 s, down from 549 to 556 s.
• The wait for ci-status on push is now set by lint, which finished 516 s and 512 s into those same two runs. I expect ci-status to finish around 515 s into a push run instead of 583 s, about 65 to 70 s sooner. This is an estimate: nothing has run on a push to main yet. Making lint faster is out of scope here.

The probe run's lint failed only on check-stale-base-overlap, because #4166 landed on main and also touched ci.yml. Merging main in fixed it.

Related

• Refs melodic-software/github-iac#378
• perf(ci): collapse ci.yml into six jobs and stop the AI review lanes re-running on every push #3696 (six-job consolidation), perf(ci): parallelise the plugin contract suites and defer the hook-utils clock benchmarks #3693 (--jobs 3), tool-honesty and jira conformance suites fail under run-plugin-tests --jobs 4, pass serially #3694

🤖 Generated with Claude Code

https://claude.ai/code/session_019mo9xaZ53JHHy4aBjgd8wK

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-14T19:08:24.499493Z	b0c544f	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[claude]

Claude encountered an error after 1s —— View job

---

I'll analyze this and get back to you.

[github-actions]

Warning

Automated review did not complete — this is an infrastructure failure, not a review verdict.

Treat any Claude comment on this PR (including a placeholder like "I'll analyze this and get back to you") as incomplete, not "no findings."

• Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/34884762904
• Failure class: rate-limit
• Last SDK result: {"subtype":"success","is_error":true,"num_turns":1,"duration_ms":695,"total_cost_usd":0,"api_error_status":429,"error_type":null,"class":"rate-limit"}

Re-run the job, or workflow_dispatch this workflow with the PR number, to retry the review. A new push re-triggers this lane only if the caller's pull_request triggers include synchronize (the canonical caller omits it).
An automatic retry may already have run — it is skipped when a partial review could duplicate comments, or when the failure class needs an operator (auth).

[claude]

Claude encountered an error after 0s —— View job

---

I'll analyze this and get back to you.

[github-actions]

Warning

Automated security review did not complete — this is an infrastructure failure, not a review verdict.

Treat any Claude comment on this PR (including a placeholder like "I'll analyze this and get back to you") as incomplete, not "no findings."

• Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/34884763204
• Failure class: rate-limit
• Last SDK result: {"subtype":"success","is_error":true,"num_turns":1,"duration_ms":468,"total_cost_usd":0,"api_error_status":429,"error_type":null,"class":"rate-limit"}

The check is green on purpose, and it is not evidence. It certifies that a security pass ran, and this one did not complete — but the cause is outside this PR's control, so merging is deliberately left unblocked rather than locking every merge for the length of the outage. Nothing was reviewed at this head. Where this check is required, it is satisfied without that evidence; a human should review security-sensitive changes here before merging.

Re-run the job, or workflow_dispatch this workflow with the PR number, to retry the review (ci-workflows#227). A new push also retries it only if the caller's pull_request triggers include synchronize (the canonical security caller keeps it). An automatic retry may already have run — it is skipped when a partial review could duplicate comments, or when the failure class needs an operator.

Re-running does NOT help for every class:

• rate-limit that persists across re-runs, or auth — the credential or usage budget needs an operator; retrying will not clear it.
• a run that exhausted its turn budget ("subtype":"error_max_turns" above) will exhaust it again. As the PR author, split the change into smaller PRs; raising --max-turns is a change to the caller workflow, not something you can set on this PR.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b0c544f030

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".