Speed up integration test CI by packing the matrix and trimming per-job setup - #4118
Speed up integration test CI by packing the matrix and trimming per-job setup#4118devin-ai-integration[bot] wants to merge 11 commits into
Conversation
…ixed cost Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
PR SummaryMedium Risk Overview
Branch protection note: merged matrix rows rename several Reviewed by Cursor Bugbot for commit 4482014. Bugbot is set up for automated code reviews on this repo. Configure here. |
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4118 +/- ##
==========================================
- Coverage 61.25% 60.25% -1.01%
==========================================
Files 2195 2078 -117
Lines 192497 179013 -13484
==========================================
- Hits 117916 107866 -10050
+ Misses 63092 60812 -2280
+ Partials 11489 10335 -1154
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
… rows to ~2 suites Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…x wasmvm archives, unpack the two longest matrix rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 3c55534. Configure here.

Integration test CI is slower and pricier than it needs to be. Turns out on
ubuntu-largeevery matrix job starts within ~1s of each other, so there's no batching: wall-clock is justprepare-cluster + slowest row, and job count only buys runner-minutes. So the trick is to cut per-job fixed cost, shortenprepare-cluster, and pack rows without making any of them long.Before → after (main vs this branch, all jobs green):
jqdump of every(env, script)pair in the matrix is byte-for-byte the same on main and here.What changed:
ci-free-disk.shskips the ~3.5 minrm -rfwhen/already has ≥CI_FREE_DISK_MIN_GIBfree (default 40,prepare-clusterasks for 60). Runners boot with ~85 GiB free, so we were burning 3.5 min per job freeing space we never touched. This also reachesexecution-specs.ymlandrelease-upgrade-test.yml(allubuntu-large); they're well within budget and the check self-heals if the runner image ever shrinks.rpcnodeimage gets its ownprepare-rpcnodejob onubuntu-latest, in parallel withprepare-clusterinstead of on its critical path.prepare-cluster7m20s → 4m38s.libwasmvm*archives (~500 MB of a 950 MB tree) that a linux/amd64 build never links. Checkout 50s → 28s per job.waited separately so a failed pull fails at the pull, not atdocker tag),::group::per script so failures point at the right suite, npm cache, 2s boot polling, skipapt-get jqwhen present.Packing policy now lives in a comment above
set-matrixin the workflow (not in a row_commentthat vanishes with its row): keep rows under ~8 min, suites asserting absolute chain state go first and say so in_comment, destructive rows (kill/wipe/state-sync a validator, disable wasm, upgrades) stay alone. Known couplings are recorded:TestSeiDBStateStoreasserts absolute wasm code counts,TestMintModuleand the Gov burn case assert absoluteuseisupply. Both hold today (fees aren't burned, tokenfactory doesn't burnusei).Two things worth knowing:
JOB_TIMEOUT(28m) is pergo testinvocation, and packed rows have up to three. Rows are 4–7 min so a hang late in a row getting a GitHub cancel instead of a goroutine dump is remote; noted next to the setting.Integration Test (...)check contexts got renamed. If branch protection requires any of them individually rather than theIntegration Test Checkumbrella, the ruleset needs a tweak.