From a8108b78cd2ef7a7c1f8210afe2b559b6b51c7c6 Mon Sep 17 00:00:00 2001 From: danielhanchen Date: Sat, 8 Aug 2026 09:54:43 +0000 Subject: [PATCH 1/3] CI: raise ROCm ccache to 2G, drop 1d eviction The Windows ROCm leg is getting a 4.13% ccache hit rate (16 of 387), which is the same signature we already fixed in llama.cpp. Two causes, both here: ccache-action defaults max-size to 500M when the input is unset. That truncates the cache at save, and a truncated cache is worse than none: it restores fine, then misses on everything that got dropped, so the hit rate collapses with nothing failing. Linux measured 0.4 GB of 0.5 and 73.65% hits; Windows 0.3 of 0.5 and 4.13%. evict-old-files: 1d discards objects by age on top of that. Age eviction at a sub-daily build cadence throws away objects that are still being reused, and a local replay showed it costs hit rate rather than saving space. ccache's own LRU under a 2G ceiling covers the same ground. Both blocks match upstream leejet byte for byte, so this is a deliberate divergence and will show up on the next sync. --- .github/workflows/build.yml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 71401593c..192eb1309 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -478,7 +478,9 @@ jobs: uses: ggml-org/ccache-action@v1.2.16 with: key: windows-rocm-${{ env.ROCM_VERSION }}-x64 - evict-old-files: 1d + # 500M (the action default) truncates this build's cache at save, and + # a truncated cache still restores then misses: 4.13% hits measured. + max-size: 2G - name: Install ROCm with Wheels if: steps.cache-rocm.outputs.cache-hit != 'true' @@ -595,7 +597,7 @@ jobs: uses: ggml-org/ccache-action@v1.2.16 with: key: ubuntu-rocm-cmake-${{ matrix.ROCM_VERSION }}-${{ matrix.build }} - evict-old-files: 1d + max-size: 2G - name: Dependencies id: depends From b143f91836c9989dcb3caf0775753c012c37b171 Mon Sep 17 00:00:00 2001 From: danielhanchen Date: Sat, 8 Aug 2026 10:14:09 +0000 Subject: [PATCH 2/3] Fix the Windows ROCm cache miss at its actual cause My first pass had the Windows diagnosis wrong and the eviction change backwards. Four consecutive Windows runs disprove the truncation story: 92812296767 0.2 / 0.5 (42.26%) 13 / 385 (3.38%) 93021695452 0.2 / 0.5 (42.54%) 17 / 387 (4.39%) 93045420552 0.3 / 0.5 (64.40%) 16 / 387 (4.13%) 93056114309 0.3 / 0.5 (65.52%) 16 / 387 (4.13%) The cache never reached the 500M cap, so it was never truncated, and the hit rate is flat whether it is 42% or 65% full. Two runs restoring different caches produced identical hit counts, which is a restored cache contributing nothing. The real cause: every Windows run logs "Cache not found for input keys: rocm-wheels-7.14.0-Windows" and re-expands the devel tree, so clang gets a new mtime, and ccache's default compiler_check=mtime hashes mtime and size. Every object is keyed to a compiler hash that never recurs. Ubuntu never expands a devel tree and sits at 73.65%. Set CCACHE_COMPILERCHECK=content on the Windows job, which the ccache manual recommends for exactly this bootstrapping case. Restored evict-old-files: 1d on both. The manual defines --evict-older-than as removing files "used less recently than AGE" and notes ccache refreshes mtime on every hit, so it never discards objects still being reused. On Windows it is the only thing reaping objects that can never be hit again, and removing it alongside a 2G cap would have parked ~2 GB of dead objects per ref. max-size: 2G stays on Ubuntu only, where it is earned: that cache plateaus at the default (88.19% full, 142 misses per run). Windows keeps the default until the compiler hash is stable, which also avoids multiplying a repo already at 10.39 GB of cache. --- .github/workflows/build.yml | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 192eb1309..4d519c555 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -451,6 +451,9 @@ jobs: env: ROCM_VERSION: "7.14.0" GPU_TARGETS: "gfx1010;gfx1011;gfx1012;gfx1030;gfx1031;gfx1032;gfx1033;gfx1034;gfx1035;gfx1036;gfx1100;gfx1101;gfx1102;gfx1103;gfx1150;gfx1151;gfx1152;gfx1153;gfx1200;gfx1201" + # rocm-sdk re-expands clang from the wheel every run, so its mtime always + # differs and the default compiler_check=mtime misses every object. + CCACHE_COMPILERCHECK: content steps: - uses: actions/checkout@v3 @@ -478,9 +481,7 @@ jobs: uses: ggml-org/ccache-action@v1.2.16 with: key: windows-rocm-${{ env.ROCM_VERSION }}-x64 - # 500M (the action default) truncates this build's cache at save, and - # a truncated cache still restores then misses: 4.13% hits measured. - max-size: 2G + evict-old-files: 1d - name: Install ROCm with Wheels if: steps.cache-rocm.outputs.cache-hit != 'true' @@ -597,6 +598,8 @@ jobs: uses: ggml-org/ccache-action@v1.2.16 with: key: ubuntu-rocm-cmake-${{ matrix.ROCM_VERSION }}-${{ matrix.build }} + evict-old-files: 1d + # Plateaus at the 500M default (88.19% full, 142 misses/run). max-size: 2G - name: Dependencies From b20c4df33f545214a7c2170b318a3acca8d6f047 Mon Sep 17 00:00:00 2001 From: danielhanchen Date: Sat, 8 Aug 2026 10:34:53 +0000 Subject: [PATCH 3/3] Apply the 2G cap to the Windows ROCm ccache too max-size is an upper bound, not a reservation: ccache stores only what it compiles, and GitHub bills actual bytes, so a cap the build never approaches costs nothing. Measured across llama.cpp's 38 build legs at 2G, total usage is 10.69 GB, mean 0.28 GB, and no leg exceeds 0.9 GB. Windows was left at the default in the previous commit because it was not cap-bound and its objects were unusable. With CCACHE_COMPILERCHECK=content the objects become reusable, so the cache is worth letting grow, and evict-old-files: 1d still reaps anything untouched for a day. Uniform settings across both legs also mean one less thing to re-tune when a build grows. --- .github/workflows/build.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 4d519c555..de178d929 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -482,6 +482,8 @@ jobs: with: key: windows-rocm-${{ env.ROCM_VERSION }}-x64 evict-old-files: 1d + # Upper bound, not a reservation: ccache stores only what it compiles. + max-size: 2G - name: Install ROCm with Wheels if: steps.cache-rocm.outputs.cache-hit != 'true'