From 22cb26610650d3ae9d0b0aad01c27b34a81ff389 Mon Sep 17 00:00:00 2001 From: rcmerci Date: Fri, 16 Oct 2026 16:54:51 +0800 Subject: [PATCH] Restore CI compiler and switch from stable build inputs --- .../journal-build/references/cache-and-ci.md | 12 +- .../scripts/prepare_ci_dependencies.py | 122 +++++++++++++++--- .../scripts/test_journal_build.py | 87 +++++++++++-- .github/workflows/ci.yml | 60 +++++++-- 4 files changed, 240 insertions(+), 41 deletions(-) diff --git a/.agents/skills/journal-build/references/cache-and-ci.md b/.agents/skills/journal-build/references/cache-and-ci.md index 605dd85..ca7ebd5 100644 --- a/.agents/skills/journal-build/references/cache-and-ci.md +++ b/.agents/skills/journal-build/references/cache-and-ci.md @@ -9,11 +9,15 @@ For failure, read `failed_phase`, `error` and that phase's log. Correct target/e The Apple builder consumes a prepared switch; it neither provisions it nor manages GitHub caches. Reuse its existing `run-*/manifest.json`: `inputs.toolchain.opam_prefix` is the canonical absolute switch prefix, `ocaml`/`compiler_sha256` identify its compiler, `inputs.dependencies` records checked lock versions, resolved pins and native library hashes, and `cache_key` selects the incremental Apple workspace. Re-run preflight against the same explicit inputs before building. A matching directory name or a previous `cache_reused: true` does not replace those checks. Do not relocate a switch, copy large caches between worktrees, change global pins, or mix macOS/Simulator/device objects to obtain a hit. Retain final input verification, full build/tests and the target acceptance limits above. -CI dependency caching belongs to `.github/workflows/ci.yml` and `.agents/skills/journal-build/scripts/prepare_ci_dependencies.py`. After setup, resolution and a fresh `--depext-only` system-dependency preparation, `python3 .agents/skills/journal-build/scripts/prepare_ci_dependencies.py --cache-identity` records `.ci-reports/opam-cache.json` without installing or pinning. It reuses the builder's compiler identity helper; this report describes compatibility, not installed dependency validation. The workflow caches the actual `opam var prefix` (normally `$GITHUB_WORKSPACE/_opam`), not just `~/.opam`. The setup action's overlapping caches are disabled. Only exact keys are restored; there are no broad `restore-keys`. +CI dependency caching belongs to `.github/workflows/ci.yml` and `.agents/skills/journal-build/scripts/prepare_ci_dependencies.py`. The workflow first installs the native Homebrew inputs (GMP, SQLite, zlib, zstd and pkg-config), selects their explicit pkg-config directories, and resolves locked Git pins once. `--cache-identity` then records `.ci-reports/opam-cache.json` **before setup creates a compiler or switch**. No installed compiler or opam command is needed for this lookup. -The CI key binds all root opam/lock bytes and resolved Git SHAs, compiler configuration/binary, opam version/root, OS/architecture, runner image/version, selected Xcode/clang/macOS SDK, GMP/SQLite/zlib versions, flags and available library/header bytes, relevant compiler environment, absolute workspace/prefix and the provisioning recipe. Missing or incompatible identity is an error. For a miss or a recoverable restore failure, the fresh setup switch plus unconditional re-pin and `opam install . --deps-only --with-test --locked --ignore-pin-depends --yes` remains the rebuild path. A cache hit never skips those commands, `dune build @all app/native_embed.exe.o`, or `dune runtest --force`. +The exact `journal-opam-v3-` key binds root opam/lock bytes and resolved Git SHAs, absolute workspace/switch/root paths, native OS/architecture/build, runner image/version, selected Xcode/clang/macOS SDK, library/header bytes and flags, and compiler environment. A fixed opam-repository commit binds compiler source checksums, patches, options/build commands and package recipes; a pinned setup action and expected opam version bind provisioning behavior. These inputs live in the workflow and must be updated deliberately with toolchain changes. Missing inputs fail closed. The shared Apple builder's actual compiler binary hash remains unchanged for native provenance; generated compiler bytes are recorded in the saved-state receipt instead of used as a pre-restore lookup input. -If the restored switch is corrupt or pin/lock checks fail, preserve the identity, action logs and package list; do not accept the old state, rewrite locks or force a different-platform cache. On an authorized subsequent CI change, bump the workflow's `journal-opam-v2-` namespace (or remove only the identified corrupt Actions cache with explicit authorization) and rerun on a fresh hosted runner. Setup and locked installation then reconstruct it. Locally, use a separate explicitly authorized prepared switch at its own absolute prefix; the Apple builder reports mismatches and never repairs an existing owner's switch. A busy switch/cache lock still requires coordination. +Restore both the actual `$GITHUB_WORKSPACE/_opam` switch and `~/.opam` metadata before setup-ocaml, with its own overlapping caches disabled. Only an exact Actions hit with a matching `.journal-ci-cache.json` receipt is accepted. The receipt checks input identity, compiler binary bytes and critical switch/root metadata without running the restored compiler. A miss, partial restoration, absent receipt or corrupt state is recorded in `.ci-reports/cache-restore.json`; only the disposable GitHub-hosted runner's exact switch/root paths are moved to `RUNNER_TEMP/journal-rejected-cache-*` before clean setup. Local/foreign paths and symlinks are refused. setup-ocaml initializes or reuses this switch and does not compile a second compiler for an accepted restore. + +On every run, setup re-pins local project packages, the preparer re-pins exact resolved checkouts, and `opam install . --deps-only --with-test --locked --ignore-pin-depends --depext-only --yes` verifies system requirements. The ordinary locked install, `dune build @all app/native_embed.exe.o` and `dune runtest --force` remain unconditional. After success, `--seal-cache` rechecks native inputs and actual compiler configuration/version/architecture, absolute prefix and opam version, then writes `.ci-reports/cache-receipt.json` and the switch receipt. The setup action's expected opam executable PATH prefix is normalized for this final comparison; actual selected toolchain/library inputs and other compiler flags are reobserved. A changed final input is an error and cannot be saved. Save a new cache only after all checks pass. + +Caches are immutable. If an exact key contains corrupt state, the run records/rejects it and safely cold-builds, but cannot replace that entry. Bump the namespace on an authorized CI change (or delete only that identified cache with explicit authorization), then rerun. Never change another owner's switch, relax ABI identity, rewrite locks, or skip final checks to obtain a hit. GitHub PR merge-ref caches are scoped to that PR: sibling PR misses are normal until a matching default-branch cache exists. Image/SDK/library/repository/lock or resolved moving-ref changes also correctly select a cold cache. ### Measure cold and hot CI runs @@ -35,4 +39,4 @@ gh run download "$COLD_RUN_ID" --repo logseq/logseq_journal \ --name journal-ci-evidence --dir "$EVIDENCE_DIR/hot-artifact" ``` -Save each attempt's `journal-ci-evidence` artifact immediately after it completes (including the cold artifact before rerunning); it contains `dependencies.json`, `opam-cache.json` and `packages.txt`. Select the real attempt numbers if not 1/2. Compare the identity JSON/key, resolved SHAs, image/version, compiler, absolute prefix and installed package lists. A changed identity or absent exact hit is a miss, not a hot sample. Record setup/bootstrap, system-dependency preparation, restore, re-pin, locked-install elapsed time and installed/recompiled package counts, final build/test time, cache save time and total job duration. Count setup/compiler cost as well as dependency-install cost; strict matching may trade hit rate for correctness. Keep full build/tests in both attempts. Report measured seconds and counts; do not infer a saving percentage from the earlier two runs, which both installed 170 packages despite a root-cache hit. +Save each attempt's `journal-ci-evidence` artifact immediately after it completes (including the cold artifact before rerunning); it contains `dependencies.json`, `opam-cache.json`, `cache-restore.json`, `cache-receipt.json` and `packages.txt`. Select the real attempt numbers if not 1/2. Compare the identity JSON/key, resolved SHAs, image/version, compiler, absolute prefix and installed package lists. A changed identity or absent exact hit is a miss, not a hot sample. Record setup/bootstrap, system-dependency preparation, restore, re-pin, locked-install elapsed time and installed/recompiled package counts, final build/test time, cache save time and total job duration. Count setup/compiler cost as well as dependency-install cost; strict matching may trade hit rate for correctness. Keep full build/tests in both attempts. Report measured seconds and counts; do not infer a saving percentage from the earlier two runs, which both installed 170 packages despite a root-cache hit. diff --git a/.agents/skills/journal-build/scripts/prepare_ci_dependencies.py b/.agents/skills/journal-build/scripts/prepare_ci_dependencies.py index 8272a69..2a9b5e0 100644 --- a/.agents/skills/journal-build/scripts/prepare_ci_dependencies.py +++ b/.agents/skills/journal-build/scripts/prepare_ci_dependencies.py @@ -8,6 +8,8 @@ from pathlib import Path import re import subprocess +import shutil +import tempfile from journal_build import fingerprint, opam_identity, sdk_identity, sha @@ -15,8 +17,8 @@ def run(*args): print("+", " ".join(args), flush=True) subprocess.run(args, check=True) -def cache_identity(root): - """Describe a native CI dependency switch; never restore or install anything.""" +def cache_identity(root, lookup_path=None): + """Describe build inputs before any switch/compiler exists; never provision.""" root = root.resolve() def capture(*args): @@ -27,9 +29,17 @@ def capture(*args): image = {name: os.environ.get(name) for name in ("ImageOS", "ImageVersion")} if not all(image.values()): raise ValueError("runner ImageOS and ImageVersion are required for cache identity") - identity = opam_identity(lambda *args: capture("opam", *args)) - if "architecture: arm64" not in identity["ocaml"] or "system: macosx" not in identity["ocaml"]: - raise ValueError("CI dependency cache requires a native arm64 macosx OCaml compiler") + compiler = os.environ.get("CI_COMPILER", "") + repository = os.environ.get("CI_OPAM_REPOSITORY", "") + if not re.fullmatch(r"ocaml-base-compiler\.[0-9]+\.[0-9]+\.[0-9]+", compiler): + raise ValueError("an exact vanilla CI_COMPILER is required") + if not re.fullmatch(r"git\+https://github.com/ocaml/opam-repository.git#[0-9a-f]{40}", repository): + raise ValueError("an exact compiler/package recipe repository commit is required") + opam_version = os.environ.get("CI_OPAM_VERSION", "") + if not re.fullmatch(r"[0-9]+\.[0-9]+\.[0-9]+", opam_version): + raise ValueError("an exact CI_OPAM_VERSION is required") + identity = dict(opam_prefix=str(root / "_opam"), compiler=compiler, + compiler_repository=repository, opam_version=opam_version) manifests = sorted(root.glob("*.opam*")) if not any(p.name.endswith(".opam.locked") for p in manifests): raise ValueError("locked dependency manifests are required for cache identity") @@ -40,23 +50,25 @@ def capture(*args): clang = capture("xcrun", "--sdk", "macosx", "--find", "clang") gmp = Path(capture("pkg-config", "--variable=libdir", "gmp")) / "libgmp.a" system_libraries = {} - for name in ("gmp", "sqlite3", "zlib"): + for name in ("gmp", "sqlite3", "zlib", "libzstd"): directory = Path(capture("pkg-config", "--variable=libdir", name)) - library = "z" if name == "zlib" else name + library = {"zlib": "z", "libzstd": "zstd"}.get(name, name) headers = Path(capture("pkg-config", "--variable=includedir", name)) header_names = {"gmp": ("gmp.h",), "sqlite3": ("sqlite3.h", "sqlite3ext.h"), - "zlib": ("zlib.h", "zconf.h")}[name] + "zlib": ("zlib.h", "zconf.h"), "libzstd": ("zstd.h",)}[name] inputs = [p for p in directory.glob("lib" + library + "*") if p.suffix in {".a", ".dylib", ".tbd"} and p.is_file()] - inputs += [headers / h for h in header_names if (headers / h).is_file()] + if not inputs or any(not (headers / h).is_file() for h in header_names): + raise ValueError(f"missing native library/header inputs: {name}") + inputs += [headers / h for h in header_names] system_libraries[name] = {"version": capture("pkg-config", "--modversion", name), "flags": capture("pkg-config", "--cflags", "--libs", name), "files": {str(p): sha(p) for p in sorted(inputs)}} recipe = [root / name for name in (".github/workflows/ci.yml", ".agents/skills/journal-build/scripts/prepare_ci_dependencies.py", ".agents/skills/journal-build/scripts/journal_build.py")] - identity.update(schema_version=2, host=dict(system=platform.system(), architecture=platform.machine(), - release=platform.release(), image=image), - workspace=str(root), opam_root=str(Path(capture("opam", "var", "root")).resolve()), - opam_version=capture("opam", "--version"), resolved=resolved, + identity.update(schema_version=3, host=dict(system=platform.system(), architecture=platform.machine(), + release=platform.release(), build=capture("sw_vers", "-buildVersion"), image=image), + workspace=str(root), opam_root=str(Path(os.environ.get("OPAMROOT", Path.home() / ".opam")).resolve()), + resolved=resolved, manifests={p.name: sha(p) for p in manifests}, recipe={str(p.relative_to(root)): sha(p) for p in recipe}, xcode=capture("xcodebuild", "-version"), sdk_path=sdk, @@ -65,16 +77,96 @@ def capture(*args): gmp={"path": str(gmp), "sha256": sha(gmp)}, system_libraries=system_libraries, environment={name: os.environ.get(name) for name in ("PATH", "DEVELOPER_DIR", "SDKROOT", "CC", "CXX", "CFLAGS", "CXXFLAGS", "CPPFLAGS", - "LDFLAGS", "PKG_CONFIG_PATH", "OCAMLPARAM", "MACOSX_DEPLOYMENT_TARGET")}) + "LDFLAGS", "PKG_CONFIG_PATH", "PKG_CONFIG_LIBDIR", "OCAMLPARAM", "MACOSX_DEPLOYMENT_TARGET", + "OCAMLLIB", "LIBRARY_PATH", "CPATH", "C_INCLUDE_PATH", "CPLUS_INCLUDE_PATH", + "CAML_LD_LIBRARY_PATH", "ARCHFLAGS", + "OPAMJOBS", "OPAMEXTERNALSOLVER", "OPAMPRECISETRACKING", "OPAMCRITERIA", + "OCAML_CONFIGURE_OPTIONS")}) + if lookup_path is not None: + # setup-ocaml only prepends its opam executable to PATH. All actual + # compiler/SDK/library selections above are freshly observed. + identity["environment"]["PATH"] = lookup_path return {"key": fingerprint(identity), "identity": identity} +def cache_files(report): + """Critical restored state checked without executing the cached compiler.""" + identity = report["identity"] + prefix, opam_root = Path(identity["opam_prefix"]), Path(identity["opam_root"]) + required = [prefix / "bin/ocamlopt.opt", prefix / ".opam-switch/switch-config", + prefix / ".opam-switch/switch-state", opam_root / "config"] + optional = list((prefix / "bin").glob("ocaml*")) + optional += list((prefix / ".opam-switch/packages").glob("ocaml*/opam")) + return {str(p): sha(p) for p in sorted(set(required + [p for p in optional if p.is_file()]))} + + +def cache_receipt(report, opam): + identity = report["identity"] + observed = opam_identity(opam) + version = identity["compiler"].removeprefix("ocaml-base-compiler.") + if (observed["opam_prefix"] != identity["opam_prefix"] or + not all(field in observed["ocaml"].splitlines() for field in + ("version: " + version, "architecture: arm64", "system: macosx")) or + opam("--version") != identity["opam_version"]): + raise ValueError("installed compiler/opam differs from requested CI inputs") + return dict(inputs=report, compiler=observed, files=cache_files(report)) + + +def validate_restored_cache(report, exact_hit): + try: + if not exact_hit: + raise ValueError("no exact Actions cache hit") + receipt = json.loads((Path(report["identity"]["opam_prefix"]) / ".journal-ci-cache.json").read_text()) + if receipt["inputs"] != report or receipt["files"] != cache_files(report): + raise ValueError("restored input identity or compiler/switch metadata differs") + return dict(status="accepted", compiler=receipt["compiler"]) + except (OSError, ValueError, KeyError, TypeError) as error: + return dict(status="rejected", reason=str(error)) + + +def quarantine_cache(report): + """Move rejected state only on the workflow's disposable hosted runner.""" + identity = report["identity"] + workspace = Path(os.environ.get("GITHUB_WORKSPACE", "/")).resolve() + prefix, opam_root = Path(identity["opam_prefix"]), Path(identity["opam_root"]) + if (os.environ.get("GITHUB_ACTIONS") != "true" or + os.environ.get("RUNNER_ENVIRONMENT") != "github-hosted" or + prefix != workspace / "_opam" or opam_root != Path.home() / ".opam" or + prefix.is_symlink() or opam_root.is_symlink()): + raise ValueError("refusing to move non-owned CI cache paths") + destination = Path(tempfile.mkdtemp(prefix="journal-rejected-cache-", dir=os.environ["RUNNER_TEMP"])) + for name, path in (("switch", prefix), ("root", opam_root)): + if path.exists(): + shutil.move(str(path), str(destination / name)) + return str(destination) + + def main(): parser = argparse.ArgumentParser(description=__doc__) mode = parser.add_mutually_exclusive_group() mode.add_argument("--pin", action="store_true", help="pin previously resolved checkouts") - mode.add_argument("--cache-identity", action="store_true", help="record native CI switch identity without installing") + mode.add_argument("--cache-identity", action="store_true", help="record CI build-input identity before compiler setup") + mode.add_argument("--validate-cache", action="store_true", help="validate exact restored state or quarantine for cold setup") + mode.add_argument("--seal-cache", action="store_true", help="verify final native inputs and record restored-state receipt") args = parser.parse_args() + if args.validate_cache or args.seal_cache: + report = json.loads(Path(".ci-reports/opam-cache.json").read_text()) + if args.validate_cache: + result = validate_restored_cache(report, os.environ.get("CI_CACHE_HIT") == "true") + if result["status"] != "accepted": + result["quarantine"] = quarantine_cache(report) + Path(".ci-reports/cache-restore.json").write_text(json.dumps(result, indent=2) + "\n") + print(json.dumps(result, indent=2)) + else: + if cache_identity(Path.cwd(), lookup_path=report["identity"]["environment"]["PATH"]) != report: + raise ValueError("native CI inputs changed after lookup; refusing to save switch") + def opam(*argv): + return subprocess.check_output(["opam", *argv], text=True).strip() + receipt = cache_receipt(report, opam) + path = Path(report["identity"]["opam_prefix"]) / ".journal-ci-cache.json" + path.write_text(json.dumps(receipt, indent=2) + "\n") + Path(".ci-reports/cache-receipt.json").write_bytes(path.read_bytes()) + return if args.cache_identity: report = Path(".ci-reports/opam-cache.json") report.write_text(json.dumps(cache_identity(Path.cwd()), indent=2) + "\n") diff --git a/.agents/skills/journal-build/scripts/test_journal_build.py b/.agents/skills/journal-build/scripts/test_journal_build.py index 79c2dad..af1f195 100644 --- a/.agents/skills/journal-build/scripts/test_journal_build.py +++ b/.agents/skills/journal-build/scripts/test_journal_build.py @@ -194,7 +194,7 @@ def test_active_switch_pin_does_not_require_raw_metadata_url(self): def compiler_fixture(self, prefix): compiler = prefix / 'bin/ocamlopt.opt' compiler.parent.mkdir(exist_ok=True) - compiler.write_text('#!/bin/sh\nprintf "architecture: arm64\\nsystem: macosx\\n"\n') + compiler.write_text('#!/bin/sh\nprintf "version: 5.5.0\\narchitecture: arm64\\nsystem: macosx\\n"\n') compiler.chmod(0o755) return compiler @@ -242,15 +242,23 @@ def test_ci_identity_is_stable_and_invalidates_required_inputs(self): workflow.parent.mkdir(parents=True) workflow.write_bytes((ROOT / '.github/workflows/ci.yml').read_bytes()) env = dict(f.build.env, OPAMSWITCH='fixture', RUNNER_OS='macOS', RUNNER_ARCH='ARM64', - ImageOS='macos26-arm64', ImageVersion='fixture-image') + ImageOS='macos26-arm64', ImageVersion='fixture-image', + CI_COMPILER='ocaml-base-compiler.5.5.0', CI_OPAM_VERSION='2.6.1', + PKG_CONFIG_PATH=':'.join(str(Path(subprocess.check_output( + ['brew', '--prefix', name], text=True).strip()) / 'lib/pkgconfig') + for name in ('gmp', 'sqlite', 'zlib', 'zstd')), + CI_OPAM_REPOSITORY='git+https://github.com/ocaml/opam-repository.git#' + 'a' * 40) with patch.dict(os.environ, env, clear=True): baseline = ci.cache_identity(f.journal) self.assertEqual(baseline, ci.cache_identity(f.journal)) - self.assertEqual(baseline['identity']['opam_prefix'], str(f.build.prefix)) + self.assertEqual(baseline['identity']['opam_prefix'], str(f.journal / '_opam')) + # Lookup must work before setup creates a compiler or a switch. + compiler = f.build.prefix / 'bin/ocamlopt.opt' + compiler.unlink() + self.assertEqual(baseline, ci.cache_identity(f.journal)) inputs = [f.lock, f.journal / 'journal.opam', f.journal / '.ci-reports/dependencies.json', workflow, - f.journal / '.agents/skills/journal-build/scripts/prepare_ci_dependencies.py', f.journal / '.agents/skills/journal-build/scripts/journal_build.py', - f.build.prefix / 'bin/ocamlopt.opt'] + f.journal / '.agents/skills/journal-build/scripts/prepare_ci_dependencies.py', f.journal / '.agents/skills/journal-build/scripts/journal_build.py'] for path in inputs: with self.subTest(path=path): data = path.read_bytes() @@ -259,11 +267,11 @@ def test_ci_identity_is_stable_and_invalidates_required_inputs(self): path.write_bytes(changed) self.assertNotEqual(baseline['key'], ci.cache_identity(f.journal)['key']) path.write_bytes(data) - for name in ('ImageVersion', 'CFLAGS', 'DEVELOPER_DIR'): + for name in ('ImageVersion', 'CFLAGS', 'DEVELOPER_DIR', 'OPAMROOT', 'CI_OPAM_VERSION', 'CI_COMPILER', 'CI_OPAM_REPOSITORY'): with self.subTest(environment=name), patch.dict(os.environ, {name: 'changed'}): - if name == 'DEVELOPER_DIR': + if name in ('DEVELOPER_DIR', 'CI_COMPILER', 'CI_OPAM_REPOSITORY', 'CI_OPAM_VERSION'): # An unavailable selected toolchain must fail, not restore an old key. - with self.assertRaises(subprocess.CalledProcessError): + with self.assertRaises((subprocess.CalledProcessError, ValueError)): ci.cache_identity(f.journal) else: self.assertNotEqual(baseline['key'], ci.cache_identity(f.journal)['key']) @@ -281,6 +289,69 @@ def test_ci_identity_is_stable_and_invalidates_required_inputs(self): with self.assertRaises(FileNotFoundError): ci.cache_identity(f.journal) + def test_ci_cache_receipt_accepts_exact_state_and_rejects_mismatches(self): + import prepare_ci_dependencies as ci + self.assertTrue(hasattr(ci, 'cache_receipt'), 'restored cache validation missing') + with self.pinned_switch() as f: + compiler = self.compiler_fixture(f.build.prefix) + report = dict(key='a' * 64, identity=dict(opam_prefix=str(f.build.prefix), + opam_root=str(f.root / 'opam'), compiler='ocaml-base-compiler.5.5.0', + opam_version=f.build.opam('--version'))) + receipt = ci.cache_receipt(report, f.build.opam) + receipt_path = f.build.prefix / '.journal-ci-cache.json' + receipt_path.write_text(json.dumps(receipt)) + self.assertEqual(ci.validate_restored_cache(report, True)['status'], 'accepted') + self.assertEqual(ci.validate_restored_cache(report, False)['status'], 'rejected') + changed = dict(report, key='b' * 64) + self.assertEqual(ci.validate_restored_cache(changed, True)['status'], 'rejected') + for path in (compiler, f.build.prefix / '.opam-switch/switch-state', + f.build.prefix / '.opam-switch/switch-config', f.root / 'opam/config'): + with self.subTest(path=path): + original = path.read_bytes() + path.write_bytes(original + b'\nchanged\n') + self.assertEqual(ci.validate_restored_cache(report, True)['status'], 'rejected') + path.write_bytes(original) + self.assertEqual(ci.validate_restored_cache(report, True)['status'], 'accepted') + for malformed in ('{}', 'invalid json'): + receipt_path.write_text(malformed) + self.assertEqual(ci.validate_restored_cache(report, True)['status'], 'rejected') + receipt_path.write_text(json.dumps(receipt)) + compiler.unlink() + self.assertEqual(ci.validate_restored_cache(report, True)['status'], 'rejected') + receipt_path.unlink() + self.assertEqual(ci.validate_restored_cache(report, True)['status'], 'rejected') + + def test_ci_rejection_quarantines_only_owned_runner_paths(self): + import prepare_ci_dependencies as ci + self.assertTrue(hasattr(ci, 'quarantine_cache'), 'safe cold fallback missing') + with tempfile.TemporaryDirectory() as directory: + workspace = Path(directory).resolve() / 'workspace' + runner_temp = Path(directory).resolve() / 'runner-temp' + workspace.mkdir() + runner_temp.mkdir() + opam_root = Path(directory).resolve() / 'opam' + prefix = workspace / '_opam' + for path in (opam_root, prefix): + path.mkdir() + (path / 'keep').write_text('owned cached state') + report = dict(identity=dict(opam_prefix=str(prefix), opam_root=str(opam_root))) + env = dict(GITHUB_WORKSPACE=str(workspace), RUNNER_TEMP=str(runner_temp), + RUNNER_ENVIRONMENT='github-hosted', GITHUB_ACTIONS='true') + with patch.dict(os.environ, env), patch('prepare_ci_dependencies.Path.home', return_value=opam_root.parent): + # A foreign root is refused; no caller-provided path is moved. + with self.assertRaises(ValueError): + ci.quarantine_cache(report) + self.assertTrue((prefix / 'keep').exists()) + opam_root.rename(opam_root.parent / '.opam') + report['identity']['opam_root'] = str(opam_root.parent / '.opam') + destination = ci.quarantine_cache(report) + self.assertFalse(prefix.exists()) + self.assertFalse(Path(report['identity']['opam_root']).exists()) + self.assertEqual(len(list(Path(destination).glob('*/keep'))), 2) + self.assertTrue(workspace.exists()) + with self.assertRaises(ValueError): + ci.quarantine_cache(report) + def test_final_verification_rejects_changed_gmp_and_probe_input(self): self.assertTrue(hasattr(self.build.Build, 'verify_inputs'), 'final input verification boundary missing') with self.pinned_switch() as f: diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e14a010..7f0a67b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -13,22 +13,27 @@ jobs: # The production transport explicitly requires Apple DNS-SD. runs-on: macos-latest timeout-minutes: 60 + env: + CI_COMPILER: ocaml-base-compiler.5.5.0 + # This commit binds compiler source checksums, patches, build commands and + # every repository package recipe. Update deliberately with the toolchain. + CI_OPAM_REPOSITORY: git+https://github.com/ocaml/opam-repository.git#7b5e9a2b084fe7ff750c4a9449289065ef9f9216 + CI_OPAM_VERSION: 2.6.1 + HOMEBREW_NO_AUTO_UPDATE: 1 + OPAMJOBS: 3 + OPAMEXTERNALSOLVER: builtin-0install + OPAMPRECISETRACKING: 1 steps: - uses: actions/checkout@v4 - name: Verify Apple app icon packaging run: python3 .agents/skills/journal-build/scripts/test_apple_app_icon.py - - uses: ocaml/setup-ocaml@v3 - with: - ocaml-compiler: "5.5.0" - # This workflow owns installed-switch caching with stricter identity. - cache: false + - name: Prepare native system libraries before cache lookup + run: | + brew install pkg-config gmp sqlite zlib zstd + echo "PKG_CONFIG_PATH=$(brew --prefix gmp)/lib/pkgconfig:$(brew --prefix sqlite)/lib/pkgconfig:$(brew --prefix zlib)/lib/pkgconfig:$(brew --prefix zstd)/lib/pkgconfig" >> "$GITHUB_ENV" - name: Resolve declared dependency sources run: python3 .agents/skills/journal-build/scripts/prepare_ci_dependencies.py - - name: Prepare native system dependencies - run: | - python3 .agents/skills/journal-build/scripts/prepare_ci_dependencies.py --pin - opam install . --deps-only --with-test --locked --ignore-pin-depends --depext-only --yes - - name: Identify installed dependency cache + - name: Identify dependency build inputs before compiler setup id: dependency-identity run: | python3 .agents/skills/journal-build/scripts/prepare_ci_dependencies.py --cache-identity @@ -38,17 +43,34 @@ jobs: report = json.loads(Path('.ci-reports/opam-cache.json').read_text()) with open(os.environ['GITHUB_OUTPUT'], 'a') as output: print('prefix=' + report['identity']['opam_prefix'], file=output) + print('root=' + report['identity']['opam_root'], file=output) print('key=' + report['key'], file=output) PY - - name: Cache installed dependencies + - name: Restore compiler, installed dependencies and opam metadata id: dependency-cache - uses: actions/cache@v4 + uses: actions/cache/restore@v4 + continue-on-error: true with: - path: ${{ steps.dependency-identity.outputs.prefix }} + path: | + ${{ steps.dependency-identity.outputs.prefix }} + ${{ steps.dependency-identity.outputs.root }} # Exact only: compiled switches are not relocatable or cross-platform. - key: journal-opam-v2-${{ runner.os }}-${{ runner.arch }}-${{ steps.dependency-identity.outputs.key }} + key: journal-opam-v3-${{ runner.os }}-${{ runner.arch }}-${{ steps.dependency-identity.outputs.key }} + - name: Validate restored cache or select a clean cold setup + env: + CI_CACHE_HIT: ${{ steps.dependency-cache.outputs.cache-hit }} + run: python3 .agents/skills/journal-build/scripts/prepare_ci_dependencies.py --validate-cache + # Pinned because its provisioning behavior participates in the input key. + - uses: ocaml/setup-ocaml@93303b622b2522e4411e295f9e77411a24912ac7 # v3 + with: + ocaml-compiler: ${{ env.CI_COMPILER }} + opam-repositories: | + default: ${{ env.CI_OPAM_REPOSITORY }} + cache: false - name: Pin resolved checkouts after cache restoration run: python3 .agents/skills/journal-build/scripts/prepare_ci_dependencies.py --pin + - name: Verify native system dependencies on the fresh runner + run: opam install . --deps-only --with-test --locked --ignore-pin-depends --depext-only --yes - name: Install locked dependencies run: opam install . --deps-only --with-test --locked --ignore-pin-depends --yes - name: Record installed package versions @@ -59,6 +81,16 @@ jobs: opam exec -- dune build @all app/native_embed.exe.o - name: Run full regression suite run: opam exec -- dune runtest --force + - name: Verify final cache inputs and seal installed state + run: python3 .agents/skills/journal-build/scripts/prepare_ci_dependencies.py --seal-cache + - name: Save compiler, installed dependencies and opam metadata + if: success() && steps.dependency-cache.outputs.cache-hit != 'true' + uses: actions/cache/save@v4 + with: + path: | + ${{ steps.dependency-identity.outputs.prefix }} + ${{ steps.dependency-identity.outputs.root }} + key: journal-opam-v3-${{ runner.os }}-${{ runner.arch }}-${{ steps.dependency-identity.outputs.key }} - name: Upload resolved dependencies and test logs if: always() uses: actions/upload-artifact@v4