Reuse CI switches across compatible native library changes

.agents/skills/journal-build/references/cache-and-ci.md

@@ -11,13 +11,17 @@ The Apple builder consumes a prepared switch; it neither provisions it nor manag
 
 CI dependency caching belongs to `.github/workflows/ci.yml` and `.agents/skills/journal-build/scripts/prepare_ci_dependencies.py`. The workflow first installs the native Homebrew inputs (GMP, SQLite, zlib, zstd and pkg-config), selects their explicit pkg-config directories, and resolves locked Git pins once. `--cache-identity` then records `.ci-reports/opam-cache.json` **before setup creates a compiler or switch**. No installed compiler or opam command is needed for this lookup.
 
-The exact `journal-opam-v3-` key binds root opam/lock bytes and resolved Git SHAs, absolute workspace/switch/root paths, native OS/architecture/build, runner image/version, selected Xcode/clang/macOS SDK, library/header bytes and flags, and compiler environment. A fixed opam-repository commit binds compiler source checksums, patches, options/build commands and package recipes; a pinned setup action and expected opam version bind provisioning behavior. These inputs live in the workflow and must be updated deliberately with toolchain changes. Missing inputs fail closed. The shared Apple builder's actual compiler binary hash remains unchanged for native provenance; generated compiler bytes are recorded in the saved-state receipt instead of used as a pre-restore lookup input.
+The `journal-opam-v4-<OS>-<arch>-<family>-<native>` key separates compatibility from native-provider observations. The family binds exact OCaml package and pinned opam-repository recipes, expected opam version and pinned setup action, lock/manifests and resolved Git SHAs, absolute workspace/switch/root prefixes, native OS major/architecture, selected Xcode/SDK/clang, effective build-tool paths/bytes and compilation flags. A changed effective `cc` still changes the family; an extra empty PATH directory does not. Environment values are hashed in reports. Runner ImageVersion, OS patch/build, full PATH digest and unrelated workflow/helper comment bytes do not determine compatibility. Increment schema_version deliberately when the bootstrap contract changes. `CI_OPAM_VERSION` is an expected version check, not a downloader selector; the pinned setup action controls its release selection, and an unexpected opam version fails sealing. The Apple builder's compiler binary hash and native artifact provenance remain strict and unchanged.
 
-Restore both the actual `$GITHUB_WORKSPACE/_opam` switch and `~/.opam` metadata before setup-ocaml, with its own overlapping caches disabled. Only an exact Actions hit with a matching `.journal-ci-cache.json` receipt is accepted. The receipt checks input identity, compiler binary bytes and critical switch/root metadata without running the restored compiler. A miss, partial restoration, absent receipt or corrupt state is recorded in `.ci-reports/cache-restore.json`; only the disposable GitHub-hosted runner's exact switch/root paths are moved to `RUNNER_TEMP/journal-rejected-cache-*` before clean setup. Local/foreign paths and symlinks are refused. setup-ocaml initializes or reuses this switch and does not compile a second compiler for an accepted restore.
+The native suffix records current GMP, SQLite, zlib and zstd versions, pkg-config flags, selected headers and libraries. Request an exact variant first, then allow prefix restoration **only within the same family**. Restore the complete `$GITHUB_WORKSPACE/_opam` and `~/.opam` before setup-ocaml, with setup's overlapping caches disabled. A matching switch receipt and compiler/switch metadata hashes are required before the restored compiler is run. Native and bytecode compiler smoke builds must pass before setup reuses the switch. A missing archive, invalid receipt or unusable compiler selects a clean setup; only exact disposable hosted-runner paths may be quarantined. No compiler/SDK/lock/path fallback is allowed.
 
-On every run, setup re-pins local project packages, the preparer re-pins exact resolved checkouts, and `opam install . --deps-only --with-test --locked --ignore-pin-depends --depext-only --yes` verifies system requirements. The ordinary locked install, `dune build @all app/native_embed.exe.o` and `dune runtest --force` remain unconditional. After success, `--seal-cache` rechecks native inputs and actual compiler configuration/version/architecture, absolute prefix and opam version, then writes `.ci-reports/cache-receipt.json` and the switch receipt. The setup action's expected opam executable PATH prefix is normalized for this final comparison; actual selected toolchain/library inputs and other compiler flags are reobserved. A changed final input is an error and cannot be saved. Save a new cache only after all checks pass.
+A different SQLite version or archive hash does not itself reinstall anything. After setup, exact re-pinning and depext checks, `--check-native-cache` examines installed C stub undefined symbols and actual Mach-O install-name providers, asks dyld to resolve required symbols in both current and linked providers, and compiles/runs the existing SQLite binding in native and bytecode modes (WAL, foreign keys, prepared statements and blobs). Zarith also runs big-integer arithmetic through both modes. Compatible bindings remain untouched. Failed binding/link probes select only their installed owner packages (for example sqlite3), and `opam reinstall --locked --ignore-pin-depends --yes ROOTS` computes their reverse dependency closure. Empty roots never invoke reinstall. Unidentifiable owners or failed post-repair probes fail the job. SQLite 3.x is not a blanket compatibility promise: a downgrade missing a required symbol is incompatible. Runtime/link checks cover the installed dependencies and full source regression covers application behavior.
 
-Caches are immutable. If an exact key contains corrupt state, the run records/rejects it and safely cold-builds, but cannot replace that entry. Bump the namespace on an authorized CI change (or delete only that identified cache with explicit authorization), then rerun. Never change another owner's switch, relax ABI identity, rewrite locks, or skip final checks to obtain a hit. GitHub PR merge-ref caches are scoped to that PR: sibling PR misses are normal until a matching default-branch cache exists. Image/SDK/library/repository/lock or resolved moving-ref changes also correctly select a cold cache.
+The helper test step clears its ambient `PKG_CONFIG_PATH` because Apple CLI fixtures require explicit tool selection; the CI identity fixture supplies its own native providers, and other workflow steps retain the selected CI providers. The ordinary locked install, helper boundary tests, `dune build @all app/native_embed.exe.o` and full `dune runtest --force` stay unconditional. Sealing reobserves family/native inputs, repeats native probes, and checks actual compiler configuration/version/architecture, absolute prefix and opam version. Changed final inputs cannot be saved. A validated fallback is recorded as `source: compatible`, not an exact Actions hit; a successful different native variant is saved under its new immutable key. Corrupt exact entries cannot be overwritten; bump the namespace for an authorized contract repair or request deletion of only that identified cache. Never clear shared caches or alter another owner's switch/pins.
+
+`cache-restore.json` and CI logs distinguish no accessible archive, family input mismatch, receipt/hash rejection, compiler probe failure, compatible native changes and selective repair. Each known difference reports its field, cached/current values and decision. Audit-only changes are explicitly ignored. A separate tiny data-only `journal-opam-input-v4-` record may restore across families to explain misses; it never authorizes executable restoration. If no readable prior manifest exists, logs show current inputs and `comparison_available: false`, without guessing a cause. Redact personal home names and hash environment values. GitHub default/base/current-ref visibility and immutable archive versions still apply; sibling PR merge-ref caches are isolated.
+
+The workflow cancels an older run only when a newer `pull_request.synchronize` head enters the same workflow/PR concurrency group. Main and controlled comparison runs have unique groups. No manual cancellation or repository settings change is needed.
 
 ### Measure cold and hot CI runs
 
@@ -39,4 +43,8 @@ gh run download "$COLD_RUN_ID" --repo logseq/logseq_journal \
   --name journal-ci-evidence --dir "$EVIDENCE_DIR/hot-artifact"
 ```
 
-Save each attempt's `journal-ci-evidence` artifact immediately after it completes (including the cold artifact before rerunning); it contains `dependencies.json`, `opam-cache.json`, `cache-restore.json`, `cache-receipt.json` and `packages.txt`. Select the real attempt numbers if not 1/2. Compare the identity JSON/key, resolved SHAs, image/version, compiler, absolute prefix and installed package lists. A changed identity or absent exact hit is a miss, not a hot sample. Record setup/bootstrap, system-dependency preparation, restore, re-pin, locked-install elapsed time and installed/recompiled package counts, final build/test time, cache save time and total job duration. Count setup/compiler cost as well as dependency-install cost; strict matching may trade hit rate for correctness. Keep full build/tests in both attempts. Report measured seconds and counts; do not infer a saving percentage from the earlier two runs, which both installed 170 packages despite a root-cache hit.
+Save each attempt's `journal-ci-evidence` artifact immediately after it completes (including the cold artifact before rerunning); it contains `dependencies.json`, `opam-cache.json`, `cache-restore.json`, `cache-receipt.json` and `packages.txt`. Select the real attempt numbers if not 1/2. Compare the identity JSON/key, resolved SHAs, image/version, compiler, absolute prefix and installed package lists. A changed family or absent restored archive is a cold sample. For a compatible native variant, record the matched/requested keys, `source: compatible`, actual provider/runtime versions, required-symbol results and unchanged compiler/stub hashes; do not label it an exact Actions hit. Record setup/bootstrap, system-dependency preparation, restore, re-pin, locked-install elapsed time and installed/recompiled package counts, final build/test time, cache save time and total job duration. Count setup/compiler cost as well as dependency-install cost; strict matching may trade hit rate for correctness. Keep full build/tests in both attempts. Report measured seconds and counts; do not infer a saving percentage from the earlier two runs, which both installed 170 packages despite a root-cache hit.
+
+### Controlled native-provider comparison
+
+When explicitly authorized, a temporary push workflow on the owned branch can call this same workflow twice sequentially with `workflow_call` input `sqlite-profile: '3.50.4'` then `'3.49.2'`. `prepare_ci_sqlite.py` downloads checksum-pinned official SQLite amalgamations and compiles them into the same disposable `RUNNER_TEMP/journal-ci-sqlite` prefix/install name. The second profile also prepends an empty PATH directory. It never installs into Homebrew or a user switch. The binding runtime must report the requested provider version in these fixture runs, so a pkg-config-only change cannot pass as a real experiment. Keep both full builds/tests, compare actual compiler/stub bytes and all opam installation actions, then remove the temporary caller and require green normal PR CI on the final head. The reusable workflow remains the single build/test implementation. Evidence belongs outside Git.