Skip to content

ci: unfreeze the build caches and build the bindings centrally - #619

Merged
andiwand merged 3 commits into
mainfrom
ci-consolidate-builds
Jul 26, 2026
Merged

ci: unfreeze the build caches and build the bindings centrally#619
andiwand merged 3 commits into
mainfrom
ci-consolidate-builds

Conversation

@andiwand

@andiwand andiwand commented Jul 26, 2026

Copy link
Copy Markdown
Member

🤖 Generated with Claude Code

Stacked on #618.

Two changes, independent of each other.

1. Stop freezing the conan and ccache caches

actions/cache never overwrites an existing key, so a key that never varies is written exactly once and then frozen — every later run restores that first snapshot and rebuilds whatever is missing from source.

Measured on run 30175816272, the ubuntu-clang build spends 13 of its 22 minutes in conan install, rebuilding 14 dependencies from source (pdf2htmlex, poppler, cairo, fontforge, wvware, fontconfig, libgsf, …) — on every single run. Every job in that run logs Cache restored from key: …; none logs a cache save. The control case is already in the repo: the jni workflow's key was young enough to hold a complete cache, and it did the same conan install in under a minute.

Two jobs sharing one key made it worse — build and build-test-downstream both wrote conan-ubuntu-24.04-clang-18-r1 with different dependency sets, so whichever landed first left the other rebuilding forever.

  • conan is now keyed on what determines its content (odr-index revision + conanfile.py + conan config + profile), so it is rewritten precisely when the dependencies change.
  • ccache gets a per-run key restored by prefix so it can actually grow, and is written only on the default branch — branch-scoped entries are invisible to other branches and would only churn the 10 GB repo quota, which is already full (10.74 GB across 24 entries).
  • CACHE_FLAVOR separates jobs that resolve a different dependency set for the same profile.

CCACHE_KEY_SUFFIX/CONAN_KEY_SUFFIX stay as the manual "throw it all away" knob.

2. Build the bindings in the central build job

jni.yml and python.yml's build-test each reconfigured and rebuilt the core from scratch for ubuntu-clang and macos-26 — the two profiles build_test already covers. Per push: three independent resolutions of the same dependency graph, three compiles of the same library, three sets of caches competing for quota.

ODR_CLI/ODR_TEST/ODR_JNI/ODR_PYTHON are independent add_subdirectory guards and with_jni/with_python only widen the conan graph, so a single configure produces all of it. The matrix gains a bindings flag; both suites run in-job via ctest since they take seconds and an artifact round-trip would cost more than it saves.

jni.yml is removed and python.yml keeps format/wheels/sdist/publish/pypi.

What deliberately did not move

  • wheels builds the reduced dependency set pyproject.toml pins (no pdf2htmlEX/wvWare/libmagic — their runtime data cannot ship inside a wheel), so its binary cannot be the one build_test produces. It is also already cheap at 4–5 min.
  • maven is untouched: the jar is classes-only and needs no native artifact. It only becomes a consumer of this job if the jar ever bundles per-platform libodr_jni.
  • windows / android stay without bindings.

Trade-off

Binding compile time now sits on the critical path ahead of docker and the reference-output test job, and a binding failure blocks them. Limited to the two profiles that need it.

Verification

actionlint clean apart from pre-existing warnings. ctest --test-dir build/python is a new invocation (the old workflow called pytest directly) — verified locally against a build with ODR_JNI/ODR_PYTHON/ODR_TEST all on: 32 passed. The JNI ctest path is unchanged from jni.yml.

Base automatically changed from maven-dist to main July 26, 2026 07:23
andiwand and others added 3 commits July 26, 2026 09:29
`actions/cache` never overwrites an existing key, so a key that never
varies is written once and then frozen: every later run restores that
first snapshot and rebuilds whatever it is missing from source.

Measured on run 30175816272, the ubuntu-clang build spends 13 of its 22
minutes in `conan install`, rebuilding 14 dependencies (pdf2htmlex,
poppler, cairo, fontforge, wvware, ...) from source — on every single
run. No job in that run logs a cache save. The jni workflow, whose key
happens to be young enough to hold a complete cache, does the same
`conan install` in under a minute.

Two jobs sharing one key made it worse: `build` and
`build-test-downstream` both wrote `conan-ubuntu-24.04-clang-18-r1` with
different dependency sets, so whichever landed first left the other
rebuilding forever. `CACHE_FLAVOR` now separates them.

Key the conan cache on what determines its content (odr-index revision +
conanfile + profile) so it is rewritten exactly when dependencies change,
and give ccache a per-run key restored by prefix so it can grow. ccache
is saved only on the default branch — branch-scoped entries are invisible
to other branches and would only churn the 10 GB repo quota, which is
already full (10.74 GB across 24 entries).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LDqWDNJQNroJPH9j5BfaPv
The JNI and Python bindings each had their own workflow that reconfigured
and rebuilt the core from scratch for ubuntu-clang and macos-26 — the two
profiles `build_test` already covers. Per push that was three independent
resolutions of the same dependency graph and three compiles of the same
library, plus three sets of caches competing for the repo quota.

`ODR_CLI`/`ODR_TEST`/`ODR_JNI`/`ODR_PYTHON` are independent subdirectory
guards and `with_jni`/`with_python` only widen the conan graph, so one
configure produces all of it. The matrix gains a `bindings` flag; the two
suites run in-job via ctest because they take seconds and an artifact
round-trip would cost more than it saves.

`wheels` stays where it is: it deliberately builds the reduced dependency
set that `pyproject.toml` pins (no pdf2htmlEX/wvWare/libmagic, whose
runtime data cannot ship inside a wheel), so its binary cannot be the one
`build_test` produces. `maven` is unaffected — the jar is classes-only and
needs no native artifact.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LDqWDNJQNroJPH9j5BfaPv
That job's matrix only ever contains ubuntu-24.04-clang-18, so the step's
`startsWith(matrix.host_profile, 'android')` guard never fires and the
`matrix.ndk_version` it interpolates is not defined anywhere — actionlint
flags the reference as an error.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LDqWDNJQNroJPH9j5BfaPv
@andiwand
andiwand force-pushed the ci-consolidate-builds branch from 62aa7af to ad2a2cc Compare July 26, 2026 07:29
@andiwand
andiwand enabled auto-merge (squash) July 26, 2026 08:09
@andiwand
andiwand disabled auto-merge July 26, 2026 08:29
@andiwand
andiwand merged commit 78b9036 into main Jul 26, 2026
25 of 27 checks passed
@andiwand
andiwand deleted the ci-consolidate-builds branch July 26, 2026 08:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant