Restore CI compiler and dependencies before setup with stable inputs

.agents/skills/journal-build/references/cache-and-ci.md

@@ -9,11 +9,15 @@ For failure, read `failed_phase`, `error` and that phase's log. Correct target/e
 
 The Apple builder consumes a prepared switch; it neither provisions it nor manages GitHub caches. Reuse its existing `run-*/manifest.json`: `inputs.toolchain.opam_prefix` is the canonical absolute switch prefix, `ocaml`/`compiler_sha256` identify its compiler, `inputs.dependencies` records checked lock versions, resolved pins and native library hashes, and `cache_key` selects the incremental Apple workspace. Re-run preflight against the same explicit inputs before building. A matching directory name or a previous `cache_reused: true` does not replace those checks. Do not relocate a switch, copy large caches between worktrees, change global pins, or mix macOS/Simulator/device objects to obtain a hit. Retain final input verification, full build/tests and the target acceptance limits above.
 
-CI dependency caching belongs to `.github/workflows/ci.yml` and `.agents/skills/journal-build/scripts/prepare_ci_dependencies.py`. After setup, resolution and a fresh `--depext-only` system-dependency preparation, `python3 .agents/skills/journal-build/scripts/prepare_ci_dependencies.py --cache-identity` records `.ci-reports/opam-cache.json` without installing or pinning. It reuses the builder's compiler identity helper; this report describes compatibility, not installed dependency validation. The workflow caches the actual `opam var prefix` (normally `$GITHUB_WORKSPACE/_opam`), not just `~/.opam`. The setup action's overlapping caches are disabled. Only exact keys are restored; there are no broad `restore-keys`.
+CI dependency caching belongs to `.github/workflows/ci.yml` and `.agents/skills/journal-build/scripts/prepare_ci_dependencies.py`. The workflow first installs the native Homebrew inputs (GMP, SQLite, zlib, zstd and pkg-config), selects their explicit pkg-config directories, and resolves locked Git pins once. `--cache-identity` then records `.ci-reports/opam-cache.json` **before setup creates a compiler or switch**. No installed compiler or opam command is needed for this lookup.
 
-The CI key binds all root opam/lock bytes and resolved Git SHAs, compiler configuration/binary, opam version/root, OS/architecture, runner image/version, selected Xcode/clang/macOS SDK, GMP/SQLite/zlib versions, flags and available library/header bytes, relevant compiler environment, absolute workspace/prefix and the provisioning recipe. Missing or incompatible identity is an error. For a miss or a recoverable restore failure, the fresh setup switch plus unconditional re-pin and `opam install . --deps-only --with-test --locked --ignore-pin-depends --yes` remains the rebuild path. A cache hit never skips those commands, `dune build @all app/native_embed.exe.o`, or `dune runtest --force`.
+The exact `journal-opam-v3-` key binds root opam/lock bytes and resolved Git SHAs, absolute workspace/switch/root paths, native OS/architecture/build, runner image/version, selected Xcode/clang/macOS SDK, library/header bytes and flags, and compiler environment. A fixed opam-repository commit binds compiler source checksums, patches, options/build commands and package recipes; a pinned setup action and expected opam version bind provisioning behavior. These inputs live in the workflow and must be updated deliberately with toolchain changes. Missing inputs fail closed. The shared Apple builder's actual compiler binary hash remains unchanged for native provenance; generated compiler bytes are recorded in the saved-state receipt instead of used as a pre-restore lookup input.
 
-If the restored switch is corrupt or pin/lock checks fail, preserve the identity, action logs and package list; do not accept the old state, rewrite locks or force a different-platform cache. On an authorized subsequent CI change, bump the workflow's `journal-opam-v2-` namespace (or remove only the identified corrupt Actions cache with explicit authorization) and rerun on a fresh hosted runner. Setup and locked installation then reconstruct it. Locally, use a separate explicitly authorized prepared switch at its own absolute prefix; the Apple builder reports mismatches and never repairs an existing owner's switch. A busy switch/cache lock still requires coordination.
+Restore both the actual `$GITHUB_WORKSPACE/_opam` switch and `~/.opam` metadata before setup-ocaml, with its own overlapping caches disabled. Only an exact Actions hit with a matching `.journal-ci-cache.json` receipt is accepted. The receipt checks input identity, compiler binary bytes and critical switch/root metadata without running the restored compiler. A miss, partial restoration, absent receipt or corrupt state is recorded in `.ci-reports/cache-restore.json`; only the disposable GitHub-hosted runner's exact switch/root paths are moved to `RUNNER_TEMP/journal-rejected-cache-*` before clean setup. Local/foreign paths and symlinks are refused. setup-ocaml initializes or reuses this switch and does not compile a second compiler for an accepted restore.
+
+On every run, setup re-pins local project packages, the preparer re-pins exact resolved checkouts, and `opam install . --deps-only --with-test --locked --ignore-pin-depends --depext-only --yes` verifies system requirements. The ordinary locked install, `dune build @all app/native_embed.exe.o` and `dune runtest --force` remain unconditional. After success, `--seal-cache` rechecks native inputs and actual compiler configuration/version/architecture, absolute prefix and opam version, then writes `.ci-reports/cache-receipt.json` and the switch receipt. The setup action's expected opam executable PATH prefix is normalized for this final comparison; actual selected toolchain/library inputs and other compiler flags are reobserved. A changed final input is an error and cannot be saved. Save a new cache only after all checks pass.
+
+Caches are immutable. If an exact key contains corrupt state, the run records/rejects it and safely cold-builds, but cannot replace that entry. Bump the namespace on an authorized CI change (or delete only that identified cache with explicit authorization), then rerun. Never change another owner's switch, relax ABI identity, rewrite locks, or skip final checks to obtain a hit. GitHub PR merge-ref caches are scoped to that PR: sibling PR misses are normal until a matching default-branch cache exists. Image/SDK/library/repository/lock or resolved moving-ref changes also correctly select a cold cache.
 
 ### Measure cold and hot CI runs
 
@@ -35,4 +39,4 @@ gh run download "$COLD_RUN_ID" --repo logseq/logseq_journal \
   --name journal-ci-evidence --dir "$EVIDENCE_DIR/hot-artifact"
 ```
 
-Save each attempt's `journal-ci-evidence` artifact immediately after it completes (including the cold artifact before rerunning); it contains `dependencies.json`, `opam-cache.json` and `packages.txt`. Select the real attempt numbers if not 1/2. Compare the identity JSON/key, resolved SHAs, image/version, compiler, absolute prefix and installed package lists. A changed identity or absent exact hit is a miss, not a hot sample. Record setup/bootstrap, system-dependency preparation, restore, re-pin, locked-install elapsed time and installed/recompiled package counts, final build/test time, cache save time and total job duration. Count setup/compiler cost as well as dependency-install cost; strict matching may trade hit rate for correctness. Keep full build/tests in both attempts. Report measured seconds and counts; do not infer a saving percentage from the earlier two runs, which both installed 170 packages despite a root-cache hit.
+Save each attempt's `journal-ci-evidence` artifact immediately after it completes (including the cold artifact before rerunning); it contains `dependencies.json`, `opam-cache.json`, `cache-restore.json`, `cache-receipt.json` and `packages.txt`. Select the real attempt numbers if not 1/2. Compare the identity JSON/key, resolved SHAs, image/version, compiler, absolute prefix and installed package lists. A changed identity or absent exact hit is a miss, not a hot sample. Record setup/bootstrap, system-dependency preparation, restore, re-pin, locked-install elapsed time and installed/recompiled package counts, final build/test time, cache save time and total job duration. Count setup/compiler cost as well as dependency-install cost; strict matching may trade hit rate for correctness. Keep full build/tests in both attempts. Report measured seconds and counts; do not infer a saving percentage from the earlier two runs, which both installed 170 packages despite a root-cache hit.

.agents/skills/journal-build/scripts/prepare_ci_dependencies.py

@@ -8,15 +8,17 @@
 from pathlib import Path
 import re
 import subprocess
+import shutil
+import tempfile
 
 from journal_build import fingerprint, opam_identity, sdk_identity, sha
 
 def run(*args):
     print("+", " ".join(args), flush=True)
     subprocess.run(args, check=True)
 
-def cache_identity(root):
-    """Describe a native CI dependency switch; never restore or install anything."""
+def cache_identity(root, lookup_path=None):
+    """Describe build inputs before any switch/compiler exists; never provision."""
     root = root.resolve()
 
     def capture(*args):
@@ -27,9 +29,17 @@ def capture(*args):
     image = {name: os.environ.get(name) for name in ("ImageOS", "ImageVersion")}
     if not all(image.values()):
         raise ValueError("runner ImageOS and ImageVersion are required for cache identity")
-    identity = opam_identity(lambda *args: capture("opam", *args))
-    if "architecture: arm64" not in identity["ocaml"] or "system: macosx" not in identity["ocaml"]:
-        raise ValueError("CI dependency cache requires a native arm64 macosx OCaml compiler")
+    compiler = os.environ.get("CI_COMPILER", "")
+    repository = os.environ.get("CI_OPAM_REPOSITORY", "")
+    if not re.fullmatch(r"ocaml-base-compiler\.[0-9]+\.[0-9]+\.[0-9]+", compiler):
+        raise ValueError("an exact vanilla CI_COMPILER is required")
+    if not re.fullmatch(r"git\+https://github.com/ocaml/opam-repository.git#[0-9a-f]{40}", repository):
+        raise ValueError("an exact compiler/package recipe repository commit is required")
+    opam_version = os.environ.get("CI_OPAM_VERSION", "")
+    if not re.fullmatch(r"[0-9]+\.[0-9]+\.[0-9]+", opam_version):
+        raise ValueError("an exact CI_OPAM_VERSION is required")
+    identity = dict(opam_prefix=str(root / "_opam"), compiler=compiler,
+                    compiler_repository=repository, opam_version=opam_version)
     manifests = sorted(root.glob("*.opam*"))
     if not any(p.name.endswith(".opam.locked") for p in manifests):
         raise ValueError("locked dependency manifests are required for cache identity")
@@ -40,23 +50,25 @@ def capture(*args):
     clang = capture("xcrun", "--sdk", "macosx", "--find", "clang")
     gmp = Path(capture("pkg-config", "--variable=libdir", "gmp")) / "libgmp.a"
     system_libraries = {}
-    for name in ("gmp", "sqlite3", "zlib"):
+    for name in ("gmp", "sqlite3", "zlib", "libzstd"):
         directory = Path(capture("pkg-config", "--variable=libdir", name))
-        library = "z" if name == "zlib" else name
+        library = {"zlib": "z", "libzstd": "zstd"}.get(name, name)
         headers = Path(capture("pkg-config", "--variable=includedir", name))
         header_names = {"gmp": ("gmp.h",), "sqlite3": ("sqlite3.h", "sqlite3ext.h"),
-                        "zlib": ("zlib.h", "zconf.h")}[name]
+                        "zlib": ("zlib.h", "zconf.h"), "libzstd": ("zstd.h",)}[name]
         inputs = [p for p in directory.glob("lib" + library + "*")
                   if p.suffix in {".a", ".dylib", ".tbd"} and p.is_file()]
-        inputs += [headers / h for h in header_names if (headers / h).is_file()]
+        if not inputs or any(not (headers / h).is_file() for h in header_names):
+            raise ValueError(f"missing native library/header inputs: {name}")
+        inputs += [headers / h for h in header_names]
         system_libraries[name] = {"version": capture("pkg-config", "--modversion", name),
             "flags": capture("pkg-config", "--cflags", "--libs", name),
             "files": {str(p): sha(p) for p in sorted(inputs)}}
     recipe = [root / name for name in (".github/workflows/ci.yml", ".agents/skills/journal-build/scripts/prepare_ci_dependencies.py", ".agents/skills/journal-build/scripts/journal_build.py")]
-    identity.update(schema_version=2, host=dict(system=platform.system(), architecture=platform.machine(),
-                    release=platform.release(), image=image),
-                    workspace=str(root), opam_root=str(Path(capture("opam", "var", "root")).resolve()),
-                    opam_version=capture("opam", "--version"), resolved=resolved,
+    identity.update(schema_version=3, host=dict(system=platform.system(), architecture=platform.machine(),
+                    release=platform.release(), build=capture("sw_vers", "-buildVersion"), image=image),
+                    workspace=str(root), opam_root=str(Path(os.environ.get("OPAMROOT", Path.home() / ".opam")).resolve()),
+                    resolved=resolved,
                     manifests={p.name: sha(p) for p in manifests},
                     recipe={str(p.relative_to(root)): sha(p) for p in recipe},
                     xcode=capture("xcodebuild", "-version"), sdk_path=sdk,
@@ -65,16 +77,96 @@ def capture(*args):
                     gmp={"path": str(gmp), "sha256": sha(gmp)}, system_libraries=system_libraries,
                     environment={name: os.environ.get(name) for name in
                         ("PATH", "DEVELOPER_DIR", "SDKROOT", "CC", "CXX", "CFLAGS", "CXXFLAGS", "CPPFLAGS",
-                         "LDFLAGS", "PKG_CONFIG_PATH", "OCAMLPARAM", "MACOSX_DEPLOYMENT_TARGET")})
+                         "LDFLAGS", "PKG_CONFIG_PATH", "PKG_CONFIG_LIBDIR", "OCAMLPARAM", "MACOSX_DEPLOYMENT_TARGET",
+                         "OCAMLLIB", "LIBRARY_PATH", "CPATH", "C_INCLUDE_PATH", "CPLUS_INCLUDE_PATH",
+                         "CAML_LD_LIBRARY_PATH", "ARCHFLAGS",
+                         "OPAMJOBS", "OPAMEXTERNALSOLVER", "OPAMPRECISETRACKING", "OPAMCRITERIA",
+                         "OCAML_CONFIGURE_OPTIONS")})
+    if lookup_path is not None:
+        # setup-ocaml only prepends its opam executable to PATH. All actual
+        # compiler/SDK/library selections above are freshly observed.
+        identity["environment"]["PATH"] = lookup_path
     return {"key": fingerprint(identity), "identity": identity}
 
 
+def cache_files(report):
+    """Critical restored state checked without executing the cached compiler."""
+    identity = report["identity"]
+    prefix, opam_root = Path(identity["opam_prefix"]), Path(identity["opam_root"])
+    required = [prefix / "bin/ocamlopt.opt", prefix / ".opam-switch/switch-config",
+                prefix / ".opam-switch/switch-state", opam_root / "config"]
+    optional = list((prefix / "bin").glob("ocaml*"))
+    optional += list((prefix / ".opam-switch/packages").glob("ocaml*/opam"))
+    return {str(p): sha(p) for p in sorted(set(required + [p for p in optional if p.is_file()]))}
+
+
+def cache_receipt(report, opam):
+    identity = report["identity"]
+    observed = opam_identity(opam)
+    version = identity["compiler"].removeprefix("ocaml-base-compiler.")
+    if (observed["opam_prefix"] != identity["opam_prefix"] or
+        not all(field in observed["ocaml"].splitlines() for field in
+                ("version: " + version, "architecture: arm64", "system: macosx")) or
+        opam("--version") != identity["opam_version"]):
+        raise ValueError("installed compiler/opam differs from requested CI inputs")
+    return dict(inputs=report, compiler=observed, files=cache_files(report))
+
+
+def validate_restored_cache(report, exact_hit):
+    try:
+        if not exact_hit:
+            raise ValueError("no exact Actions cache hit")
+        receipt = json.loads((Path(report["identity"]["opam_prefix"]) / ".journal-ci-cache.json").read_text())
+        if receipt["inputs"] != report or receipt["files"] != cache_files(report):
+            raise ValueError("restored input identity or compiler/switch metadata differs")
+        return dict(status="accepted", compiler=receipt["compiler"])
+    except (OSError, ValueError, KeyError, TypeError) as error:
+        return dict(status="rejected", reason=str(error))
+
+
+def quarantine_cache(report):
+    """Move rejected state only on the workflow's disposable hosted runner."""
+    identity = report["identity"]
+    workspace = Path(os.environ.get("GITHUB_WORKSPACE", "/")).resolve()
+    prefix, opam_root = Path(identity["opam_prefix"]), Path(identity["opam_root"])
+    if (os.environ.get("GITHUB_ACTIONS") != "true" or
+        os.environ.get("RUNNER_ENVIRONMENT") != "github-hosted" or
+        prefix != workspace / "_opam" or opam_root != Path.home() / ".opam" or
+        prefix.is_symlink() or opam_root.is_symlink()):
+        raise ValueError("refusing to move non-owned CI cache paths")
+    destination = Path(tempfile.mkdtemp(prefix="journal-rejected-cache-", dir=os.environ["RUNNER_TEMP"]))
+    for name, path in (("switch", prefix), ("root", opam_root)):
+        if path.exists():
+            shutil.move(str(path), str(destination / name))
+    return str(destination)
+
+
 def main():
     parser = argparse.ArgumentParser(description=__doc__)
     mode = parser.add_mutually_exclusive_group()
     mode.add_argument("--pin", action="store_true", help="pin previously resolved checkouts")
-    mode.add_argument("--cache-identity", action="store_true", help="record native CI switch identity without installing")
+    mode.add_argument("--cache-identity", action="store_true", help="record CI build-input identity before compiler setup")
+    mode.add_argument("--validate-cache", action="store_true", help="validate exact restored state or quarantine for cold setup")
+    mode.add_argument("--seal-cache", action="store_true", help="verify final native inputs and record restored-state receipt")
     args = parser.parse_args()
+    if args.validate_cache or args.seal_cache:
+        report = json.loads(Path(".ci-reports/opam-cache.json").read_text())
+        if args.validate_cache:
+            result = validate_restored_cache(report, os.environ.get("CI_CACHE_HIT") == "true")
+            if result["status"] != "accepted":
+                result["quarantine"] = quarantine_cache(report)
+            Path(".ci-reports/cache-restore.json").write_text(json.dumps(result, indent=2) + "\n")
+            print(json.dumps(result, indent=2))
+        else:
+            if cache_identity(Path.cwd(), lookup_path=report["identity"]["environment"]["PATH"]) != report:
+                raise ValueError("native CI inputs changed after lookup; refusing to save switch")
+            def opam(*argv):
+                return subprocess.check_output(["opam", *argv], text=True).strip()
+            receipt = cache_receipt(report, opam)
+            path = Path(report["identity"]["opam_prefix"]) / ".journal-ci-cache.json"
+            path.write_text(json.dumps(receipt, indent=2) + "\n")
+            Path(".ci-reports/cache-receipt.json").write_bytes(path.read_bytes())
+        return
     if args.cache_identity:
         report = Path(".ci-reports/opam-cache.json")
         report.write_text(json.dumps(cache_identity(Path.cwd()), indent=2) + "\n")