Skip to content

ci: drop apt and sccache from the setup action, and key the native cache on the image - #82

Merged
rubas-agent[bot] merged 3 commits into
mainfrom
ci-cold-build
Oct 23, 2026
Merged

rubas-agent[bot] merged 3 commits into
mainfrom
ci-cold-build

Conversation

@rubas-agent

@rubas-agent rubas-agent Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Before. PR #81 changes only Cargo.lock, and its job Elixir + Rust verification hit the 20 minute timeout twice. Most of the time went to apt-get update and apt-get install in the setup action:

Run Image apt step Job
main, 2026-10-20 (39442726587) ubuntu-24.04 13 s 1 min
PR #81, attempt 1 (39934643971) ubuntu-26.04 17 min 13 s cancelled at 20 min
PR #81, attempt 2 (39934643971) ubuntu-26.04 19 min 30 s cancelled at 20 min

When attempt 1 was cancelled, task check had run for only 2 min 20 s.

Problem. ubuntu-latest moved to ubuntu-26.04 on 2026-10-19, and its apt mirror took 17 to 20 minutes. The apt step was not necessary, because the runner image already has cmake, gcc, make, pkg-config, and libclang. The move to the new image also showed two more faults:

  • The native target cache from ubuntu-24.04 is not valid on ubuntu-26.04. bindgen tells cargo to rebuild whisper-rs-sys when a system header changes, and the old target names headers such as /usr/lib/llvm-18/... that do not exist on 26.04. So each run built whisper.cpp three times (release, clippy, test). The key matched exactly, so actions/cache never saved a new target.
  • sccache keeps its objects in the GitHub Actions cache. On one run, that cache returned 503 Egress is over the account limit, and sccache stopped mix compile at once. sccache gave 0 hits on main and 3 hits from 90 requests on this PR.

After. The setup action has no apt step and no sccache, and the native target cache key starts with the runner image (ubuntu26-native-...).

Run on this PR Native cache Setup step task check Job
Cold, no target cache (39941498567) miss 20 s 3 min 50 s 4 min 18 s
Warm, re-run (39941498567) exact hit 21 s 8 s 34 s

A change to Cargo.lock now costs at most the cold run, about 4 1/2 minutes, so the 20 minute timeout stays as it is. The first run on main after the merge is cold and saves the ubuntu26 target for the next runs.

Cause, design, and tests

The cause is in .github/actions/setup/action.yml, in the step Install system packages. The packages that it installed were already on the image: cmake 4.4.4 in /usr/local/bin comes before the apt cmake in /usr/bin, so the build never used the apt cmake. libclang1-21 was already on the image too, because the apt log selected only libclang-21-dev and libclang-dev.

The native cache key had runner.os (Linux), which is the same for every Ubuntu release. The whisper-rs-sys build script calls bindgen::CargoCallbacks, which writes rerun-if-changed for every header that bindgen reads, also for the system headers. A new step reads $ImageOS (set by the GitHub-hosted runner, for example ubuntu26) and puts it first in the key and in the restore key. A target from another release is thus never restored. The mix and cargo registry caches do not depend on the image, so they keep runner.os.

I removed sccache instead of making it optional. The cargo target cache already gives the warm build, and sccache only added a second dependency on the cache service.

whisper_cpp is public, so GitHub-hosted minutes cost nothing. Our self-hosted incus runners have no scale set for whisper_cpp, and on a public repo a fork PR could run its code on them. So the job stays on ubuntu-latest.

Tests: the cold and the warm run above, both on ubuntu-26.04. Both ran the full task check: format, compile with --warnings-as-errors, credo, clippy, mix test, cargo test, and zizmor.

@rubas-agent rubas-agent Bot changed the title ci: drop the apt step that stalled CI for 17 to 20 minutes ci: drop apt and sccache from the setup action, and key the native cache on the image Oct 23, 2026
@rubas-agent
rubas-agent Bot merged commit 6e5bddd into main Oct 23, 2026
2 checks passed
@rubas-agent
rubas-agent Bot deleted the ci-cold-build branch October 23, 2026 10:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant