Skip to content

Speed up integration test CI by packing the matrix and trimming per-job setup - #4118

Closed
devin-ai-integration[bot] wants to merge 11 commits into
mainfrom
devin/1788988106-pack-integration-matrix
Closed

Speed up integration test CI by packing the matrix and trimming per-job setup#4118
devin-ai-integration[bot] wants to merge 11 commits into
mainfrom
devin/1788988106-pack-integration-matrix

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Integration test CI is slower and pricier than it needs to be. Turns out on ubuntu-large every matrix job starts within ~1s of each other, so there's no batching: wall-clock is just prepare-cluster + slowest row, and job count only buys runner-minutes. So the trick is to cut per-job fixed cost, shorten prepare-cluster, and pack rows without making any of them long.

Before → after (main vs this branch, all jobs green):

  • Wall-clock: 19m01s → 12m45s (−33%)
  • Runner time: 227 → 144 runner-minutes (−37%)
  • Coverage: identical. A jq dump of every (env, script) pair in the matrix is byte-for-byte the same on main and here.

What changed:

  • ci-free-disk.sh skips the ~3.5 min rm -rf when / already has ≥ CI_FREE_DISK_MIN_GIB free (default 40, prepare-cluster asks for 60). Runners boot with ~85 GiB free, so we were burning 3.5 min per job freeing space we never touched. This also reaches execution-specs.yml and release-upgrade-test.yml (all ubuntu-large); they're well within budget and the check self-heals if the runner image ever shrinks.
  • rpcnode image gets its own prepare-rpcnode job on ubuntu-latest, in parallel with prepare-cluster instead of on its critical path. prepare-cluster 7m20s → 4m38s.
  • Sparse checkout everywhere: drops the darwin/aarch64 libwasmvm* archives (~500 MB of a 950 MB tree) that a linux/amd64 build never links. Checkout 50s → 28s per job.
  • Matrix rows sharing a cluster env now run their suites back-to-back in one job: 35 → 26 jobs. Merged: Compat + RPC .io/.iox, Precompile/Endpoints + Precompiles, Autobahn Compat + Autobahn GIGA, SeiDB SS + Mint/Staking/Bank + Distribution, Gov/Oracle/Authz + Wasm, and the three Autobahn Cosmos rows. Everything else keeps its own row.
  • Smaller stuff: parallel image pulls (each waited separately so a failed pull fails at the pull, not at docker tag), ::group:: per script so failures point at the right suite, npm cache, 2s boot polling, skip apt-get jq when present.

Packing policy now lives in a comment above set-matrix in the workflow (not in a row _comment that vanishes with its row): keep rows under ~8 min, suites asserting absolute chain state go first and say so in _comment, destructive rows (kill/wipe/state-sync a validator, disable wasm, upgrades) stay alone. Known couplings are recorded: TestSeiDBStateStore asserts absolute wasm code counts, TestMintModule and the Gov burn case assert absolute usei supply. Both hold today (fees aren't burned, tokenfactory doesn't burn usei).

Two things worth knowing:

  • JOB_TIMEOUT (28m) is per go test invocation, and packed rows have up to three. Rows are 4–7 min so a hang late in a row getting a GitHub cancel instead of a goroutine dump is remote; noted next to the setting.
  • 14 Integration Test (...) check contexts got renamed. If branch protection requires any of them individually rather than the Integration Test Check umbrella, the ruleset needs a tweak.

…ixed cost

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@cursor

cursor Bot commented Sep 9, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Matrix packing changes job boundaries and check context names while relying on documented ordering constraints; mis-packed rows could cause flaky or order-dependent failures without dropping coverage.

Overview
Speeds up integration CI without removing tests: the matrix is packed from 35 jobs to 26 by running compatible suites back-to-back on one cluster boot (same env), with ordering/isolation rules documented in integration-test.yml (absolute-state assertions first, destructive/upgrade/disable-wasm rows stay alone).

prepare-rpcnode builds and pushes the rpcnode image in parallel with prepare-cluster (which no longer builds rpcnode); build-seid-in-localnode-ci only requires the localnode image via ensure-integration-ci-localnode-image.

ci-free-disk.sh skips the ~3.5 min rm -rf when root has at least CI_FREE_DISK_MIN_GIB free (40 GiB default; 60 for prepare-cluster). Jobs use sparse checkout (exclude darwin/aarch64 wasmvm ~500 MB), parallel GHCR pulls, npm cache, faster cluster polling, ::group:: per matrix script, and conditional jq install.

Branch protection note: merged matrix rows rename several Integration Test (...) check contexts; update rules if individual rows are required gates.

Reviewed by Cursor Bugbot for commit 4482014. Bugbot is set up for automated code reviews on this repo. Configure here.

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The automated review did not complete; see the failing AI Review check for details.

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedSep 10, 2026, 9:55 AM

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread .github/workflows/integration-test-matrix.json
Comment thread .github/workflows/integration-test.yml
@codecov

codecov Bot commented Sep 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.25%. Comparing base (e43f64e) to head (4482014).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #4118      +/-   ##
==========================================
- Coverage   61.25%   60.25%   -1.01%     
==========================================
  Files        2195     2078     -117     
  Lines      192497   179013   -13484     
==========================================
- Hits       117916   107866   -10050     
+ Misses      63092    60812    -2280     
+ Partials    11489    10335    -1154     
Flag Coverage Δ
sei-db 69.80% <ø> (ø)
sei-db-state-db ?

Flags with carried forward coverage won't be shown. Click here to find out more.
see 117 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

… rows to ~2 suites

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration devin-ai-integration Bot changed the title ci: pack integration-test matrix by cluster config, trim per-job fixed cost ci: skip needless disk reclaim, pack integration-test matrix by cluster config Sep 9, 2026
masih and others added 2 commits September 9, 2026 22:04
…x wasmvm archives, unpack the two longest matrix rows

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration devin-ai-integration Bot changed the title ci: skip needless disk reclaim, pack integration-test matrix by cluster config ci: cut integration-test wall-clock 33% and runner-minutes 37% without dropping coverage Sep 9, 2026
devin-ai-integration Bot and others added 5 commits September 9, 2026 22:46
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 3c55534. Configure here.

Comment thread .github/workflows/integration-test.yml Outdated
@devin-ai-integration devin-ai-integration Bot changed the title ci: cut integration-test wall-clock 33% and runner-minutes 37% without dropping coverage Cut integration-test wall-clock 33% and runner-minutes 37% without dropping coverage Sep 10, 2026
@devin-ai-integration devin-ai-integration Bot changed the title Cut integration-test wall-clock 33% and runner-minutes 37% without dropping coverage Speed up integration test CI by packing the matrix and trimming per-job setup Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants