CI: raise ROCm ccache to 2G, drop 1d eviction - #6
Conversation
The Windows ROCm leg is getting a 4.13% ccache hit rate (16 of 387), which is the same signature we already fixed in llama.cpp. Two causes, both here: ccache-action defaults max-size to 500M when the input is unset. That truncates the cache at save, and a truncated cache is worse than none: it restores fine, then misses on everything that got dropped, so the hit rate collapses with nothing failing. Linux measured 0.4 GB of 0.5 and 73.65% hits; Windows 0.3 of 0.5 and 4.13%. evict-old-files: 1d discards objects by age on top of that. Age eviction at a sub-daily build cadence throws away objects that are still being reused, and a local replay showed it costs hit rate rather than saving space. ccache's own LRU under a 2G ceiling covers the same ground. Both blocks match upstream leejet byte for byte, so this is a deliberate divergence and will show up on the next sync.
My first pass had the Windows diagnosis wrong and the eviction change backwards. Four consecutive Windows runs disprove the truncation story: 92812296767 0.2 / 0.5 (42.26%) 13 / 385 (3.38%) 93021695452 0.2 / 0.5 (42.54%) 17 / 387 (4.39%) 93045420552 0.3 / 0.5 (64.40%) 16 / 387 (4.13%) 93056114309 0.3 / 0.5 (65.52%) 16 / 387 (4.13%) The cache never reached the 500M cap, so it was never truncated, and the hit rate is flat whether it is 42% or 65% full. Two runs restoring different caches produced identical hit counts, which is a restored cache contributing nothing. The real cause: every Windows run logs "Cache not found for input keys: rocm-wheels-7.14.0-Windows" and re-expands the devel tree, so clang gets a new mtime, and ccache's default compiler_check=mtime hashes mtime and size. Every object is keyed to a compiler hash that never recurs. Ubuntu never expands a devel tree and sits at 73.65%. Set CCACHE_COMPILERCHECK=content on the Windows job, which the ccache manual recommends for exactly this bootstrapping case. Restored evict-old-files: 1d on both. The manual defines --evict-older-than as removing files "used less recently than AGE" and notes ccache refreshes mtime on every hit, so it never discards objects still being reused. On Windows it is the only thing reaping objects that can never be hit again, and removing it alongside a 2G cap would have parked ~2 GB of dead objects per ref. max-size: 2G stays on Ubuntu only, where it is earned: that cache plateaus at the default (88.19% full, 142 misses per run). Windows keeps the default until the compiler hash is stable, which also avoids multiplying a repo already at 10.39 GB of cache.
|
Correcting this PR: my Windows diagnosis was wrong, and the eviction change was backwards. Pushed a rewrite in The truncation story does not survive the logsI claimed the 500M default truncates the Windows cache at save. Four consecutive Windows ROCm runs say otherwise:
The cache has never reached the cap, so it was never truncated, and the hit rate is flat whether it is 42% or 65% full. Two runs that restored different caches produced identical hit counts, which is what a restored cache contributing exactly zero hits looks like. Raising the cap could not have fixed this. The actual causeEvery Windows run logs both of these: The Fixed at the cause: Removing
|
max-size is an upper bound, not a reservation: ccache stores only what it compiles, and GitHub bills actual bytes, so a cap the build never approaches costs nothing. Measured across llama.cpp's 38 build legs at 2G, total usage is 10.69 GB, mean 0.28 GB, and no leg exceeds 0.9 GB. Windows was left at the default in the previous commit because it was not cap-bound and its objects were unusable. With CCACHE_COMPILERCHECK=content the objects become reusable, so the cache is worth letting grow, and evict-old-files: 1d still reaps anything untouched for a day. Uniform settings across both legs also mean one less thing to re-tune when a build grows.
1 participant
The Windows ROCm leg has a 4.13% ccache hit rate and the Ubuntu one is pressed against its cache cap. This fixes both, at their actual causes.
An earlier revision of this PR treated the two as the same problem and got Windows wrong. See the review comments below for the correction; this description reflects the current diff.
Windows: the compiler hash never recurs
Four consecutive Windows ROCm runs:
0.2 / 0.5 (42.26%)13 / 385 (3.38%)0.2 / 0.5 (42.54%)17 / 387 (4.39%)0.3 / 0.5 (64.40%)16 / 387 (4.13%)0.3 / 0.5 (65.52%)16 / 387 (4.13%)The cache never reaches the 500M cap, and the hit rate is flat regardless of how full it is. Two runs restoring different caches produced identical hit counts, which is what a restored cache contributing nothing looks like.
Every Windows run logs
Cache not found for input keys: rocm-wheels-7.14.0-Windowsand then re-expands the devel tree, soclang.exegets a fresh mtime each run. ccache runs withcompiler_check = mtime(the default), which hashes the compiler mtime and size, so every object is keyed to a hash that never comes back. Ubuntu never expands a devel tree and sits at 73.65%.Fix:
CCACHE_COMPILERCHECK: contenton the Windows job. The ccache manual recommendscontentfor exactly this case, noting it "makes ccache very slightly slower compared to mtime, but makes it cope better with compiler upgrades during a build bootstrapping process".Ubuntu: genuinely cap-bound
Job 93056114302 reports
Cache size (GB): 0.4 / 0.5 (88.19%)with 142 misses per run, and the Ubuntu entries plateau at the cap (0.236, 0.239, 0.433, 0.437 GB). That is the LRU-cleanup regime, somax-size: 2G.Windows gets the same 2G. It is not cap-bound today, but
max-sizeis an upper bound rather than a reservation: ccache stores only what it compiles and GitHub bills actual bytes, so a cap a build never approaches costs nothing. For scale, llama.cpp's 38 build legs all run at 2G and total 10.69 GB, mean 0.28 GB, with no single leg above 0.9 GB. Uniform settings also mean one less thing to re-tune when a build grows.evict-old-files: 1dstays on bothAn earlier revision removed it on the theory that it discards objects still in use. That is wrong: the ccache manual defines
--evict-older-than AGEas removing files "used less recently than AGE", and ccache refreshes mtime on every cache hit. It only reaps objects untouched for 24 hours, and on Windows it is the only thing reaping objects that can never be hit again.Scope
Six lines: one job-level env var,
max-sizeon the Ubuntu ccache step, andevict-old-filesleft as upstream has it. Parsing the workflow before and after and comparing with the ccachewith:blocks masked confirms nothing else moved. The Windows ccache block now matches upstream leejet byte for byte again, so only the Ubuntumax-sizeline is a divergence to carry through the next sync.Measuring it
The first Windows run after merge is not a clean signal:
rocm-wheels-7.14.0-Windowswas only created at2026-08-08T04:34:22, after all four runs above, so a jump could partly be that cache landing. Watch the second run.Not in this PR
Cache usage here is 10.39 GB (
active_caches_size_in_bytes: 10390575412), so there is no janitor in this change. Two things for later: the timestamped ccache keys accumulate generations where only the newest is prefix-reachable, androcm-wheels-7.14.0-Windowsexists four times at 1.31 GB under one key, which is ref-scoped copies from PR branches rather than generations.