diff --git a/.agents/skills/journal-build/references/cache-and-ci.md b/.agents/skills/journal-build/references/cache-and-ci.md index ca7ebd50..0d1e0223 100644 --- a/.agents/skills/journal-build/references/cache-and-ci.md +++ b/.agents/skills/journal-build/references/cache-and-ci.md @@ -11,13 +11,17 @@ The Apple builder consumes a prepared switch; it neither provisions it nor manag CI dependency caching belongs to `.github/workflows/ci.yml` and `.agents/skills/journal-build/scripts/prepare_ci_dependencies.py`. The workflow first installs the native Homebrew inputs (GMP, SQLite, zlib, zstd and pkg-config), selects their explicit pkg-config directories, and resolves locked Git pins once. `--cache-identity` then records `.ci-reports/opam-cache.json` **before setup creates a compiler or switch**. No installed compiler or opam command is needed for this lookup. -The exact `journal-opam-v3-` key binds root opam/lock bytes and resolved Git SHAs, absolute workspace/switch/root paths, native OS/architecture/build, runner image/version, selected Xcode/clang/macOS SDK, library/header bytes and flags, and compiler environment. A fixed opam-repository commit binds compiler source checksums, patches, options/build commands and package recipes; a pinned setup action and expected opam version bind provisioning behavior. These inputs live in the workflow and must be updated deliberately with toolchain changes. Missing inputs fail closed. The shared Apple builder's actual compiler binary hash remains unchanged for native provenance; generated compiler bytes are recorded in the saved-state receipt instead of used as a pre-restore lookup input. +The `journal-opam-v4----` key separates compatibility from native-provider observations. The family binds exact OCaml package and pinned opam-repository recipes, expected opam version and pinned setup action, lock/manifests and resolved Git SHAs, absolute workspace/switch/root prefixes, native OS major/architecture, selected Xcode/SDK/clang, effective build-tool paths/bytes and compilation flags. A changed effective `cc` still changes the family; an extra empty PATH directory does not. Environment values are hashed in reports. Runner ImageVersion, OS patch/build, full PATH digest and unrelated workflow/helper comment bytes do not determine compatibility. Increment schema_version deliberately when the bootstrap contract changes. `CI_OPAM_VERSION` is an expected version check, not a downloader selector; the pinned setup action controls its release selection, and an unexpected opam version fails sealing. The Apple builder's compiler binary hash and native artifact provenance remain strict and unchanged. -Restore both the actual `$GITHUB_WORKSPACE/_opam` switch and `~/.opam` metadata before setup-ocaml, with its own overlapping caches disabled. Only an exact Actions hit with a matching `.journal-ci-cache.json` receipt is accepted. The receipt checks input identity, compiler binary bytes and critical switch/root metadata without running the restored compiler. A miss, partial restoration, absent receipt or corrupt state is recorded in `.ci-reports/cache-restore.json`; only the disposable GitHub-hosted runner's exact switch/root paths are moved to `RUNNER_TEMP/journal-rejected-cache-*` before clean setup. Local/foreign paths and symlinks are refused. setup-ocaml initializes or reuses this switch and does not compile a second compiler for an accepted restore. +The native suffix records current GMP, SQLite, zlib and zstd versions, pkg-config flags, selected headers and libraries. Request an exact variant first, then allow prefix restoration **only within the same family**. Restore the complete `$GITHUB_WORKSPACE/_opam` and `~/.opam` before setup-ocaml, with setup's overlapping caches disabled. A matching switch receipt and compiler/switch metadata hashes are required before the restored compiler is run. Native and bytecode compiler smoke builds must pass before setup reuses the switch. A missing archive, invalid receipt or unusable compiler selects a clean setup; only exact disposable hosted-runner paths may be quarantined. No compiler/SDK/lock/path fallback is allowed. -On every run, setup re-pins local project packages, the preparer re-pins exact resolved checkouts, and `opam install . --deps-only --with-test --locked --ignore-pin-depends --depext-only --yes` verifies system requirements. The ordinary locked install, `dune build @all app/native_embed.exe.o` and `dune runtest --force` remain unconditional. After success, `--seal-cache` rechecks native inputs and actual compiler configuration/version/architecture, absolute prefix and opam version, then writes `.ci-reports/cache-receipt.json` and the switch receipt. The setup action's expected opam executable PATH prefix is normalized for this final comparison; actual selected toolchain/library inputs and other compiler flags are reobserved. A changed final input is an error and cannot be saved. Save a new cache only after all checks pass. +A different SQLite version or archive hash does not itself reinstall anything. After setup, exact re-pinning and depext checks, `--check-native-cache` examines installed C stub undefined symbols and actual Mach-O install-name providers, asks dyld to resolve required symbols in both current and linked providers, and compiles/runs the existing SQLite binding in native and bytecode modes (WAL, foreign keys, prepared statements and blobs). Zarith also runs big-integer arithmetic through both modes. Compatible bindings remain untouched. Failed binding/link probes select only their installed owner packages (for example sqlite3), and `opam reinstall --locked --ignore-pin-depends --yes ROOTS` computes their reverse dependency closure. Empty roots never invoke reinstall. Unidentifiable owners or failed post-repair probes fail the job. SQLite 3.x is not a blanket compatibility promise: a downgrade missing a required symbol is incompatible. Runtime/link checks cover the installed dependencies and full source regression covers application behavior. -Caches are immutable. If an exact key contains corrupt state, the run records/rejects it and safely cold-builds, but cannot replace that entry. Bump the namespace on an authorized CI change (or delete only that identified cache with explicit authorization), then rerun. Never change another owner's switch, relax ABI identity, rewrite locks, or skip final checks to obtain a hit. GitHub PR merge-ref caches are scoped to that PR: sibling PR misses are normal until a matching default-branch cache exists. Image/SDK/library/repository/lock or resolved moving-ref changes also correctly select a cold cache. +The helper test step clears its ambient `PKG_CONFIG_PATH` because Apple CLI fixtures require explicit tool selection; the CI identity fixture supplies its own native providers, and other workflow steps retain the selected CI providers. The ordinary locked install, helper boundary tests, `dune build @all app/native_embed.exe.o` and full `dune runtest --force` stay unconditional. Sealing reobserves family/native inputs, repeats native probes, and checks actual compiler configuration/version/architecture, absolute prefix and opam version. Changed final inputs cannot be saved. A validated fallback is recorded as `source: compatible`, not an exact Actions hit; a successful different native variant is saved under its new immutable key. Corrupt exact entries cannot be overwritten; bump the namespace for an authorized contract repair or request deletion of only that identified cache. Never clear shared caches or alter another owner's switch/pins. + +`cache-restore.json` and CI logs distinguish no accessible archive, family input mismatch, receipt/hash rejection, compiler probe failure, compatible native changes and selective repair. Each known difference reports its field, cached/current values and decision. Audit-only changes are explicitly ignored. A separate tiny data-only `journal-opam-input-v4-` record may restore across families to explain misses; it never authorizes executable restoration. If no readable prior manifest exists, logs show current inputs and `comparison_available: false`, without guessing a cause. Redact personal home names and hash environment values. GitHub default/base/current-ref visibility and immutable archive versions still apply; sibling PR merge-ref caches are isolated. + +The workflow cancels an older run only when a newer `pull_request.synchronize` head enters the same workflow/PR concurrency group. Main and controlled comparison runs have unique groups. No manual cancellation or repository settings change is needed. ### Measure cold and hot CI runs @@ -39,4 +43,8 @@ gh run download "$COLD_RUN_ID" --repo logseq/logseq_journal \ --name journal-ci-evidence --dir "$EVIDENCE_DIR/hot-artifact" ``` -Save each attempt's `journal-ci-evidence` artifact immediately after it completes (including the cold artifact before rerunning); it contains `dependencies.json`, `opam-cache.json`, `cache-restore.json`, `cache-receipt.json` and `packages.txt`. Select the real attempt numbers if not 1/2. Compare the identity JSON/key, resolved SHAs, image/version, compiler, absolute prefix and installed package lists. A changed identity or absent exact hit is a miss, not a hot sample. Record setup/bootstrap, system-dependency preparation, restore, re-pin, locked-install elapsed time and installed/recompiled package counts, final build/test time, cache save time and total job duration. Count setup/compiler cost as well as dependency-install cost; strict matching may trade hit rate for correctness. Keep full build/tests in both attempts. Report measured seconds and counts; do not infer a saving percentage from the earlier two runs, which both installed 170 packages despite a root-cache hit. +Save each attempt's `journal-ci-evidence` artifact immediately after it completes (including the cold artifact before rerunning); it contains `dependencies.json`, `opam-cache.json`, `cache-restore.json`, `cache-receipt.json` and `packages.txt`. Select the real attempt numbers if not 1/2. Compare the identity JSON/key, resolved SHAs, image/version, compiler, absolute prefix and installed package lists. A changed family or absent restored archive is a cold sample. For a compatible native variant, record the matched/requested keys, `source: compatible`, actual provider/runtime versions, required-symbol results and unchanged compiler/stub hashes; do not label it an exact Actions hit. Record setup/bootstrap, system-dependency preparation, restore, re-pin, locked-install elapsed time and installed/recompiled package counts, final build/test time, cache save time and total job duration. Count setup/compiler cost as well as dependency-install cost; strict matching may trade hit rate for correctness. Keep full build/tests in both attempts. Report measured seconds and counts; do not infer a saving percentage from the earlier two runs, which both installed 170 packages despite a root-cache hit. + +### Controlled native-provider comparison + +When explicitly authorized, a temporary push workflow on the owned branch can call this same workflow twice sequentially with `workflow_call` input `sqlite-profile: '3.50.4'` then `'3.49.2'`. `prepare_ci_sqlite.py` downloads checksum-pinned official SQLite amalgamations and compiles them into the same disposable `RUNNER_TEMP/journal-ci-sqlite` prefix/install name. The second profile also prepends an empty PATH directory. It never installs into Homebrew or a user switch. The binding runtime must report the requested provider version in these fixture runs, so a pkg-config-only change cannot pass as a real experiment. Keep both full builds/tests, compare actual compiler/stub bytes and all opam installation actions, then remove the temporary caller and require green normal PR CI on the final head. The reusable workflow remains the single build/test implementation. Evidence belongs outside Git. diff --git a/.agents/skills/journal-build/scripts/ci_native_compatibility.py b/.agents/skills/journal-build/scripts/ci_native_compatibility.py new file mode 100644 index 00000000..ad738b1d --- /dev/null +++ b/.agents/skills/journal-build/scripts/ci_native_compatibility.py @@ -0,0 +1,180 @@ +"""Read-only compiler/native link probes for restored CI switches.""" +from pathlib import Path +import re +import subprocess +import tempfile + +from journal_build import sha + + +def capture(argv, cwd=None): + result = subprocess.run([str(a) for a in argv], cwd=cwd, text=True, capture_output=True) + if result.returncode: + raise RuntimeError((result.stderr or result.stdout or str(argv[0]))[-6000:]) + return result.stdout.strip() + + +def require_native_symbols(library, symbols): + """Ask dyld to resolve every required symbol, including shared-cache libraries.""" + source = r''' +#include +#include +int main(int argc, char **argv) { + void *handle = dlopen(argv[1], RTLD_NOW | RTLD_LOCAL); + if (!handle) { fprintf(stderr, "%s\n", dlerror()); return 2; } + int missing = 0; + for (int i = 2; i < argc; i++) { + dlerror(); (void)dlsym(handle, argv[i]); + const char *error = dlerror(); + if (error) { fprintf(stderr, "missing required symbol %s: %s\n", argv[i], error); missing = 1; } + } + dlclose(handle); return missing; +} +''' + with tempfile.TemporaryDirectory(prefix='journal-symbols-') as directory: + root = Path(directory) + (root / 'probe.c').write_text(source) + capture(['xcrun', '--sdk', 'macosx', 'clang', root / 'probe.c', '-o', root / 'probe']) + capture([root / 'probe', library, *sorted(set(symbols))]) + + +SQLITE_PROGRAM = r''' +let require expected actual = + if expected <> actual then failwith (Sqlite3.Rc.to_string actual) +let () = + let version = Sqlite3.sqlite_version_info () in + if not (String.starts_with ~prefix:"3." version) then failwith "SQLite version is outside the SQLite 3 API family"; + let file = Filename.temp_file "journal-ci-sqlite" ".db" in + let db = Sqlite3.db_open ~mutex:`FULL file in + let exec sql = require Sqlite3.Rc.OK (Sqlite3.exec db sql) in + exec "PRAGMA foreign_keys=ON"; + let wal = ref false in + require Sqlite3.Rc.OK (Sqlite3.exec db ~cb:(fun row _ -> wal := row.(0) = Some "wal") "PRAGMA journal_mode=WAL"); + if not !wal then failwith "SQLite WAL unavailable"; + exec "CREATE TABLE parent(id INTEGER PRIMARY KEY, payload BLOB NOT NULL)"; + exec "CREATE TABLE child(parent_id INTEGER REFERENCES parent(id))"; + exec "BEGIN IMMEDIATE"; + let insert = Sqlite3.prepare db "INSERT INTO parent VALUES (?, ?)" in + require Sqlite3.Rc.OK (Sqlite3.bind_values insert [Sqlite3.Data.INT 42L; Sqlite3.Data.BLOB "\000fixture"]); + require Sqlite3.Rc.DONE (Sqlite3.step insert); + require Sqlite3.Rc.OK (Sqlite3.finalize insert); + let query = Sqlite3.prepare db "SELECT id, payload FROM parent" in + require Sqlite3.Rc.ROW (Sqlite3.step query); + if Sqlite3.column_int query 0 <> 42 || Sqlite3.column_blob query 1 <> "\000fixture" then failwith "SQLite binding data mismatch"; + require Sqlite3.Rc.OK (Sqlite3.finalize query); + exec "COMMIT"; + require Sqlite3.Rc.CONSTRAINT (Sqlite3.exec db "INSERT INTO child VALUES (99)"); + if not (Sqlite3.db_close db) then failwith "SQLite close failed"; + List.iter (fun name -> if Sys.file_exists name then Sys.remove name) [file; file ^ "-wal"; file ^ "-shm"]; + print_endline version +''' + + +def binding_probe(prefix, package, program): + prefix = Path(prefix) + with tempfile.TemporaryDirectory(prefix='journal-binding-') as directory: + root = Path(directory) + source = root / 'probe.ml' + source.write_text(program) + values = {} + for mode, compiler, archive in [('native', 'ocamlopt.opt', 'cmxa'), ('bytecode', 'ocamlc.opt', 'cma')]: + output = root / ('probe-' + mode) + capture([prefix / 'bin' / compiler, '-I', prefix / 'lib' / package, + package + '.' + archive, source, '-o', output], cwd=root) + argv = [output] if mode == 'native' else [prefix / 'bin/ocamlrun', '-I', prefix / 'lib/stublibs', output] + values[mode + '_version'] = capture(argv, cwd=root) + if mode == 'native': + values['native_linkage'] = capture(['otool', '-L', output]).splitlines()[1:] + return values + + +def probe_sqlite_binding(prefix, expected_version=None): + stubs = Path(prefix) / 'lib/sqlite3/libsqlite3_stubs.a' + before = sha(stubs) + result = binding_probe(prefix, 'sqlite3', SQLITE_PROGRAM) + if result['native_version'] != result['bytecode_version']: + raise RuntimeError('native and bytecode SQLite runtime version differ') + if expected_version is not None and result['native_version'] != expected_version: + raise RuntimeError('SQLite runtime version differs from selected provider: ' + result['native_version'] + ' / ' + expected_version) + if before != sha(stubs): + raise RuntimeError('SQLite runtime probe changed the installed binding') + return dict(status='compatible', stub_sha256=before, **result) + + +def probe_zarith_binding(prefix): + program = ''' +let () = + let x = Z.of_string "123456789012345678901234567890123456789" in + let square = Z.mul x x in + if not (Z.equal (Z.sqrt square) x) || not (Z.equal (Z.divexact square x) x) then failwith "GMP arithmetic mismatch"; + print_endline "GMP arithmetic passed" +''' + return dict(status='compatible', **binding_probe(prefix, 'zarith', program)) + + +def probe_compiler(prefix): + prefix = Path(prefix) + with tempfile.TemporaryDirectory(prefix='journal-compiler-') as directory: + root = Path(directory) + source = root / 'probe.ml' + source.write_text('let () = if Array.length [|21;21|] <> 2 then exit 1; print_int (21+21)\n') + for mode, compiler in [('native', 'ocamlopt.opt'), ('bytecode', 'ocamlc.opt')]: + output = root / ('probe-' + mode) + capture([prefix / 'bin' / compiler, source, '-o', output], cwd=root) + argv = [output] if mode == 'native' else [prefix / 'bin/ocamlrun', output] + if capture(argv, cwd=root) != '42': + raise RuntimeError('restored compiler/runtime smoke result differs') + return dict(status='compatible', compiler_sha256=sha(prefix / 'bin/ocamlopt.opt')) + + +SYMBOL_GROUPS = { + 'sqlite3': re.compile(r'^sqlite3_'), 'gmp': re.compile(r'^__gmp'), + 'libzstd': re.compile(r'^ZSTD_'), + 'zlib': re.compile(r'^(?:zlibVersion|adler32|crc32|compress|uncompress|inflate|deflate|gz)'), +} +LIBRARY_GROUPS = {'sqlite3': 'sqlite3', 'gmp': 'gmp', 'zstd': 'libzstd', 'z': 'zlib'} + + +def native_link_checks(prefix, libraries): + """Check actual installed C stubs, not just current pkg-config declarations.""" + prefix = Path(prefix) + owners = {} + for install in (prefix / '.opam-switch/install').glob('*.install'): + for quoted in re.findall(r'"([^"\n]+)"', install.read_text()): + owners.setdefault(Path(quoted).name, set()).add(install.stem) + candidates = set((prefix / 'lib/stublibs').glob('*.so')) + candidates.update(p for p in (prefix / 'lib').rglob('*.a') if '_stubs' in p.name or p.name == 'libzarith.a') + checks, rebuild = [], set() + for path in sorted(candidates): + undefined = capture(['nm', '-u', path]).splitlines() + groups = {name: sorted({line.strip()[1:] for line in undefined if pattern.match(line.strip()[1:])}) + for name, pattern in SYMBOL_GROUPS.items()} + groups = {name: symbols for name, symbols in groups.items() if symbols} + if not groups: + continue + owner = owners.get(path.name, set()) + if not owner and path.parent.name in {'sqlite3', 'zarith'}: + owner = {path.parent.name} + linked = capture(['otool', '-L', path]).splitlines()[1:] if path.suffix == '.so' else [] + for name, symbols in groups.items(): + selected = [p for p in libraries[name]['files'] if p.endswith('.dylib')] + if not selected: + raise RuntimeError('no dynamic provider for native validation: ' + name) + providers = {min(selected, key=len)} + for line in linked: + library = line.strip().split(' (', 1)[0] + if any(re.fullmatch(r'lib' + re.escape(stem) + r'(?:\.[0-9]+)*\.dylib', Path(library).name) + for stem, group in LIBRARY_GROUPS.items() if group == name): + providers.add(library) + try: + for provider in sorted(providers): + require_native_symbols(provider, symbols) + checks.append(dict(file=str(path.relative_to(prefix)), group=name, symbols=len(symbols), + providers=sorted(providers), status='compatible')) + except RuntimeError as error: + if not owner: + raise RuntimeError('cannot identify native rebuild owner for ' + str(path.relative_to(prefix))) from error + rebuild.update(owner) + checks.append(dict(file=str(path.relative_to(prefix)), group=name, packages=sorted(owner), + status='rebuild', reason=str(error))) + return checks, sorted(rebuild) diff --git a/.agents/skills/journal-build/scripts/prepare_ci_dependencies.py b/.agents/skills/journal-build/scripts/prepare_ci_dependencies.py index 2a9b5e0b..3b597502 100644 --- a/.agents/skills/journal-build/scripts/prepare_ci_dependencies.py +++ b/.agents/skills/journal-build/scripts/prepare_ci_dependencies.py @@ -9,9 +9,11 @@ import re import subprocess import shutil +import shlex import tempfile from journal_build import fingerprint, opam_identity, sdk_identity, sha +from ci_native_compatibility import (native_link_checks, probe_compiler, probe_sqlite_binding, probe_zarith_binding, require_native_symbols) def run(*args): print("+", " ".join(args), flush=True) @@ -27,8 +29,6 @@ def capture(*args): if (platform.system(), platform.machine(), os.environ.get("RUNNER_OS"), os.environ.get("RUNNER_ARCH")) != ("Darwin", "arm64", "macOS", "ARM64"): raise ValueError("CI dependency cache requires a native macOS ARM64 runner") image = {name: os.environ.get(name) for name in ("ImageOS", "ImageVersion")} - if not all(image.values()): - raise ValueError("runner ImageOS and ImageVersion are required for cache identity") compiler = os.environ.get("CI_COMPILER", "") repository = os.environ.get("CI_OPAM_REPOSITORY", "") if not re.fullmatch(r"ocaml-base-compiler\.[0-9]+\.[0-9]+\.[0-9]+", compiler): @@ -48,7 +48,6 @@ def capture(*args): raise ValueError("exact resolved dependency SHAs are required for cache identity") sdk = capture("xcrun", "--sdk", "macosx", "--show-sdk-path") clang = capture("xcrun", "--sdk", "macosx", "--find", "clang") - gmp = Path(capture("pkg-config", "--variable=libdir", "gmp")) / "libgmp.a" system_libraries = {} for name in ("gmp", "sqlite3", "zlib", "libzstd"): directory = Path(capture("pkg-config", "--variable=libdir", name)) @@ -64,32 +63,84 @@ def capture(*args): system_libraries[name] = {"version": capture("pkg-config", "--modversion", name), "flags": capture("pkg-config", "--cflags", "--libs", name), "files": {str(p): sha(p) for p in sorted(inputs)}} - recipe = [root / name for name in (".github/workflows/ci.yml", ".agents/skills/journal-build/scripts/prepare_ci_dependencies.py", ".agents/skills/journal-build/scripts/journal_build.py")] - identity.update(schema_version=3, host=dict(system=platform.system(), architecture=platform.machine(), - release=platform.release(), build=capture("sw_vers", "-buildVersion"), image=image), + # Deliberate schema changes govern the bootstrap contract. Comments and + # reporting-only helper edits do not change an installed compiler's ABI. + setup = re.findall(r"uses: ocaml/setup-ocaml@([0-9a-f]{40})", (root / ".github/workflows/ci.yml").read_text()) + if len(setup) != 1: + raise ValueError("one pinned setup-ocaml action is required") + tools = {} + for name, override in (("cc", "CC"), ("c++", "CXX"), ("make", "MAKE"), ("ar", "AR"), ("ranlib", "RANLIB"), ("ld", "LD")): + argv = shlex.split(os.environ.get(override) or name) + selected = shutil.which(argv[0]) + if not selected: + raise ValueError("missing build tool: " + name) + tools[name] = dict(path=str(Path(selected).resolve()), sha256=sha(selected), arguments=fingerprint(argv[1:])) + flags = ("SDKROOT", "CFLAGS", "CXXFLAGS", "CPPFLAGS", "LDFLAGS", "OCAMLPARAM", + "MACOSX_DEPLOYMENT_TARGET", "OCAMLLIB", "LIBRARY_PATH", "CPATH", "C_INCLUDE_PATH", + "CPLUS_INCLUDE_PATH", "CAML_LD_LIBRARY_PATH", "ARCHFLAGS", "OPAMJOBS", "OPAMEXTERNALSOLVER", + "OPAMPRECISETRACKING", "OPAMCRITERIA", "OCAML_CONFIGURE_OPTIONS") + identity.update(schema_version=4, setup_action=setup[0], + host=dict(system=platform.system(), architecture=platform.machine(), os_major=platform.mac_ver()[0].split('.')[0]), workspace=str(root), opam_root=str(Path(os.environ.get("OPAMROOT", Path.home() / ".opam")).resolve()), - resolved=resolved, - manifests={p.name: sha(p) for p in manifests}, - recipe={str(p.relative_to(root)): sha(p) for p in recipe}, + resolved=resolved, manifests={p.name: sha(p) for p in manifests}, xcode=capture("xcodebuild", "-version"), sdk_path=sdk, sdk=sdk_identity(sdk, "macosx", capture("xcrun", "--sdk", "macosx", "--show-sdk-version")), - clang={"path": clang, "sha256": sha(clang)}, - gmp={"path": str(gmp), "sha256": sha(gmp)}, system_libraries=system_libraries, - environment={name: os.environ.get(name) for name in - ("PATH", "DEVELOPER_DIR", "SDKROOT", "CC", "CXX", "CFLAGS", "CXXFLAGS", "CPPFLAGS", - "LDFLAGS", "PKG_CONFIG_PATH", "PKG_CONFIG_LIBDIR", "OCAMLPARAM", "MACOSX_DEPLOYMENT_TARGET", - "OCAMLLIB", "LIBRARY_PATH", "CPATH", "C_INCLUDE_PATH", "CPLUS_INCLUDE_PATH", - "CAML_LD_LIBRARY_PATH", "ARCHFLAGS", - "OPAMJOBS", "OPAMEXTERNALSOLVER", "OPAMPRECISETRACKING", "OPAMCRITERIA", - "OCAML_CONFIGURE_OPTIONS")}) - if lookup_path is not None: - # setup-ocaml only prepends its opam executable to PATH. All actual - # compiler/SDK/library selections above are freshly observed. - identity["environment"]["PATH"] = lookup_path - return {"key": fingerprint(identity), "identity": identity} - - -def cache_files(report): + clang={"path": clang, "sha256": sha(clang)}, tools=tools, + environment={name: fingerprint(os.environ[name]) if name in os.environ else None for name in flags}) + observed = dict(image=image, release=platform.release(), build=capture("sw_vers", "-buildVersion"), + path_sha256=fingerprint(lookup_path if lookup_path is not None else os.environ.get("PATH")), + system_libraries=system_libraries) + return {"key": fingerprint(identity), "native_key": fingerprint(system_libraries), "identity": identity, "observed": observed} + + +def cache_key(report): + return "journal-opam-v4-macOS-ARM64-" + report["key"] + "-" + report["native_key"] + + +def differences(cached, current, field=""): + if isinstance(cached, dict) and isinstance(current, dict): + return [item for name in sorted(set(cached) | set(current)) + for item in differences(cached.get(name), current.get(name), field + "." + name)] + if cached == current: + return [] + return [dict(field=field.lstrip("."), cached=cached, current=current)] + + +def safe_log(value, field=""): + """Logs contain hashes for environment values and no personal absolute paths.""" + if isinstance(value, dict): + if {"field", "cached", "current"} <= value.keys(): + return dict(field=safe_log(value["field"]), cached=safe_log(value["cached"], value["field"]), current=safe_log(value["current"], value["field"])) + return {safe_log(k): safe_log(v, field + "." + k) for k, v in value.items()} + if isinstance(value, list): + return [safe_log(v, field) for v in value] + if isinstance(value, str): + if ".environment." in field and not re.fullmatch(r"[0-9a-f]{64}", value): + return "sha256:" + fingerprint(value) + return re.sub(r"/Users/[^/\s]+", "/Users/", value) + return value + + +def comparable_record(record): + return (isinstance(record, dict) and isinstance(record.get("identity"), dict) + and isinstance(record.get("observed", {}), dict) + and isinstance(record.get("observed", {}).get("system_libraries", {}), dict)) + + +def cache_diagnostics(current, cached, status, reason): + if not comparable_record(cached): + cached = None + changes = differences(cached.get("identity", {}), current["identity"], "identity") if cached else [] + native = differences(cached.get("observed", {}).get("system_libraries", {}), + current.get("observed", {}).get("system_libraries", {}), "native") if cached else [] + audit = differences({k: v for k, v in cached.get("observed", {}).items() if k != "system_libraries"}, + {k: v for k, v in current.get("observed", {}).items() if k != "system_libraries"}, "audit") if cached else [] + return safe_log(dict(status=status, reason=reason, comparison_available=cached is not None, + comparison_reason="recorded candidate inputs" if cached else "no readable prior manifest; differences unavailable", + current=current, differences=changes, native_changes=native, ignored_audit_changes=audit)) + + +def cache_files(report, allow_missing=False): """Critical restored state checked without executing the cached compiler.""" identity = report["identity"] prefix, opam_root = Path(identity["opam_prefix"]), Path(identity["opam_root"]) @@ -97,7 +148,8 @@ def cache_files(report): prefix / ".opam-switch/switch-state", opam_root / "config"] optional = list((prefix / "bin").glob("ocaml*")) optional += list((prefix / ".opam-switch/packages").glob("ocaml*/opam")) - return {str(p): sha(p) for p in sorted(set(required + [p for p in optional if p.is_file()]))} + return {str(p): sha(p) if p.is_file() or not allow_missing else None + for p in sorted(set(required + [p for p in optional if p.is_file()]))} def cache_receipt(report, opam): @@ -112,16 +164,70 @@ def cache_receipt(report, opam): return dict(inputs=report, compiler=observed, files=cache_files(report)) -def validate_restored_cache(report, exact_hit): +def validate_restored_cache(report, exact_hit, matched_key=""): + cached = None + file_changes = [] try: - if not exact_hit: - raise ValueError("no exact Actions cache hit") + if not exact_hit and not matched_key: + raise ValueError("no accessible compatible switch archive") receipt = json.loads((Path(report["identity"]["opam_prefix"]) / ".journal-ci-cache.json").read_text()) - if receipt["inputs"] != report or receipt["files"] != cache_files(report): - raise ValueError("restored input identity or compiler/switch metadata differs") - return dict(status="accepted", compiler=receipt["compiler"]) + cached = receipt["inputs"] + if not comparable_record(cached): + raise ValueError("invalid restored input manifest structure") + if cached["key"] != report["key"] or cached["identity"] != report["identity"]: + raise ValueError("compatibility input mismatch") + if matched_key and matched_key != cache_key(cached): + raise ValueError("Actions matched key differs from switch receipt") + file_changes = differences(receipt["files"], cache_files(report, allow_missing=True), "files") + if file_changes: + raise ValueError("compiler/switch metadata hash mismatch") + result = cache_diagnostics(report, cached, "accepted", "receipt accepted; native probes still required") + result.update(source="exact" if exact_hit and cached["native_key"] == report["native_key"] else "compatible", compiler=receipt["compiler"]) + return result except (OSError, ValueError, KeyError, TypeError) as error: - return dict(status="rejected", reason=str(error)) + result = cache_diagnostics(report, cached, "rejected", str(error)) + result["differences"].extend(safe_log(file_changes)) + return result + + +def rebuild_native_packages(packages, opam): + installed = set(opam("list", "--installed", "--short", "--color=never").splitlines()) + roots = sorted(set(packages) & installed) + if roots: + # opam computes and rebuilds the real reverse dependency closure. + opam("reinstall", "--locked", "--ignore-pin-depends", "--yes", *roots) + return roots + + +def check_native(report, opam, repair=False): + prefix = Path(report["identity"]["opam_prefix"]) + checks, roots = native_link_checks(prefix, report["observed"]["system_libraries"]) + sqlite = None + if (prefix / "lib/sqlite3/sqlite3.cmxa").exists(): + try: + # Fixture runs must demonstrate the selected different provider was + # actually loaded. Normal runs validate the real install-name provider. + expected = report["observed"]["system_libraries"]["sqlite3"]["version"] if os.environ.get("CI_SQLITE_PROFILE", "brew") != "brew" else None + sqlite = probe_sqlite_binding(prefix, expected) + except RuntimeError as error: + roots = sorted(set(roots) | {"sqlite3"}) + checks.append(dict(group="sqlite3", status="rebuild", reason=str(error))) + zarith = None + if (prefix / "lib/zarith/zarith.cmxa").exists(): + try: + zarith = probe_zarith_binding(prefix) + except RuntimeError as error: + roots = sorted(set(roots) | {"zarith"}) + checks.append(dict(group="gmp", status="rebuild", reason=str(error))) + result = dict(status="compatible" if not roots else "repair-required", checks=checks, sqlite=sqlite, zarith=zarith, rebuild_roots=roots) + print(json.dumps(safe_log(result), indent=2), flush=True) + if roots: + if not repair: + raise RuntimeError("native compatibility failed after repair: " + ", ".join(roots)) + result["reinstalled_roots"] = rebuild_native_packages(roots, opam) + result["after_repair"] = check_native(report, opam, repair=False) + result["status"] = "repaired" + return result def quarantine_cache(report): @@ -146,31 +252,65 @@ def main(): mode = parser.add_mutually_exclusive_group() mode.add_argument("--pin", action="store_true", help="pin previously resolved checkouts") mode.add_argument("--cache-identity", action="store_true", help="record CI build-input identity before compiler setup") - mode.add_argument("--validate-cache", action="store_true", help="validate exact restored state or quarantine for cold setup") + mode.add_argument("--validate-cache", action="store_true", help="validate compatible restored state or quarantine for cold setup") + mode.add_argument("--check-native-cache", action="store_true", help="probe restored bindings; repair only incompatible native packages") mode.add_argument("--seal-cache", action="store_true", help="verify final native inputs and record restored-state receipt") args = parser.parse_args() - if args.validate_cache or args.seal_cache: + if args.validate_cache or args.seal_cache or args.check_native_cache: report = json.loads(Path(".ci-reports/opam-cache.json").read_text()) + def opam(*argv): + command = ["opam", *argv] + print("+", " ".join(command), flush=True) + output = subprocess.check_output(command, text=True).strip() + if argv[0] == "reinstall": + print(output, flush=True) + return output if args.validate_cache: - result = validate_restored_cache(report, os.environ.get("CI_CACHE_HIT") == "true") + result = validate_restored_cache(report, os.environ.get("CI_CACHE_HIT") == "true", os.environ.get("CI_CACHE_MATCHED_KEY", "")) + if not result["comparison_available"]: + try: + candidate = json.loads(Path(".ci-reports/cache-candidate/inputs.json").read_text()) + result = cache_diagnostics(report, candidate, "rejected", result["reason"]) + except (OSError, ValueError, TypeError, KeyError): + pass + if result["status"] == "accepted": + try: + result["compiler_probe"] = probe_compiler(report["identity"]["opam_prefix"]) + except RuntimeError as error: + result.update(status="rejected", reason="restored compiler probe failed: " + str(error)) if result["status"] != "accepted": result["quarantine"] = quarantine_cache(report) - Path(".ci-reports/cache-restore.json").write_text(json.dumps(result, indent=2) + "\n") - print(json.dumps(result, indent=2)) + Path(".ci-reports/cache-restore.json").write_text(json.dumps(safe_log(result), indent=2) + "\n") + print(json.dumps(safe_log(result), indent=2)) + elif args.check_native_cache: + restored = json.loads(Path(".ci-reports/cache-restore.json").read_text()) + if restored["status"] != "accepted": + print('Native reuse check: cold setup; installed-state probes run during sealing') + return + result = check_native(report, opam, repair=True) + Path(".ci-reports/native-compatibility.json").write_text(json.dumps(safe_log(result), indent=2) + "\n") else: - if cache_identity(Path.cwd(), lookup_path=report["identity"]["environment"]["PATH"]) != report: - raise ValueError("native CI inputs changed after lookup; refusing to save switch") - def opam(*argv): - return subprocess.check_output(["opam", *argv], text=True).strip() + final = cache_identity(Path.cwd()) + # PATH prefixes introduced by setup are audit-only; selected tools, + # flags, native providers and all compatibility inputs are reobserved. + if final["key"] != report["key"] or final["native_key"] != report["native_key"]: + print(json.dumps(cache_diagnostics(final, report, "rejected", "inputs changed during build"), indent=2)) + raise ValueError("CI inputs changed after lookup; refusing to save switch") + native = check_native(report, opam) + Path(".ci-reports/native-final.json").write_text(json.dumps(safe_log(native), indent=2) + "\n") receipt = cache_receipt(report, opam) path = Path(report["identity"]["opam_prefix"]) / ".journal-ci-cache.json" path.write_text(json.dumps(receipt, indent=2) + "\n") Path(".ci-reports/cache-receipt.json").write_bytes(path.read_bytes()) + candidate = Path(".ci-reports/cache-candidate/inputs.json") + candidate.parent.mkdir(parents=True, exist_ok=True) + candidate.write_text(json.dumps(report, indent=2) + "\n") return if args.cache_identity: report = Path(".ci-reports/opam-cache.json") - report.write_text(json.dumps(cache_identity(Path.cwd()), indent=2) + "\n") - print(report.read_text(), end="") + inputs = cache_identity(Path.cwd()) + report.write_text(json.dumps(inputs, indent=2) + "\n") + print(json.dumps(safe_log(inputs), indent=2)) return report = Path(".ci-reports/dependencies.json") if args.pin: diff --git a/.agents/skills/journal-build/scripts/prepare_ci_sqlite.py b/.agents/skills/journal-build/scripts/prepare_ci_sqlite.py new file mode 100644 index 00000000..e6668b94 --- /dev/null +++ b/.agents/skills/journal-build/scripts/prepare_ci_sqlite.py @@ -0,0 +1,61 @@ +#!/usr/bin/env python3 +"""Prepare a checksum-pinned SQLite provider for an authorized CI comparison.""" +import hashlib +import os +from pathlib import Path +import subprocess +import tempfile + +SOURCES = { + '3.50.4': ('3500400', 'a3db587a1b92ee5ddac2f66b3edb41b26f9c867275782d46c3a088977d6a5b18'), + '3.49.2': ('3490200', '5c6d8697e8a32a1512a9be5ad2b2e7a891241c334f56f8b0fb4fc6051e1652e8'), +} + + +def main(): + version = os.environ.get('CI_SQLITE_PROFILE', 'brew') + if version == 'brew': + return + if version not in SOURCES: + raise ValueError('unsupported CI SQLite profile: ' + version) + if os.environ.get('GITHUB_ACTIONS') != 'true' or os.environ.get('RUNNER_ENVIRONMENT') != 'github-hosted': + raise ValueError('comparison provider is restricted to the disposable hosted CI runner') + root = Path(os.environ['RUNNER_TEMP']).resolve() + prefix = root / 'journal-ci-sqlite' + if prefix.exists(): + raise ValueError('comparison provider path already exists; refusing to replace it') + source_id, expected = SOURCES[version] + with tempfile.TemporaryDirectory(prefix='journal-sqlite-source-', dir=root) as directory: + directory = Path(directory) + archive = directory / 'source.tar.gz' + subprocess.run(['curl', '--fail', '--location', '--retry', '3', 'https://sqlite.org/2025/sqlite-autoconf-' + source_id + '.tar.gz', '-o', archive], check=True) + if hashlib.sha256(archive.read_bytes()).hexdigest() != expected: + raise ValueError('SQLite source checksum mismatch') + subprocess.run(['tar', '-xf', archive, '-C', directory], check=True) + source = directory / ('sqlite-autoconf-' + source_id) + lib, include = prefix / 'lib', prefix / 'include' + (lib / 'pkgconfig').mkdir(parents=True) + include.mkdir() + flags = ['-O2', '-fPIC', '-DSQLITE_THREADSAFE=1', '-DSQLITE_ENABLE_COLUMN_METADATA', '-DSQLITE_ENABLE_FTS5', '-DSQLITE_ENABLE_RTREE'] + obj = directory / 'sqlite3.o' + subprocess.run(['xcrun', '--sdk', 'macosx', 'clang', *flags, '-c', source / 'sqlite3.c', '-o', obj], check=True) + dylib = lib / 'libsqlite3.0.dylib' + subprocess.run(['xcrun', '--sdk', 'macosx', 'clang', '-dynamiclib', obj, '-o', dylib, '-install_name', dylib, '-compatibility_version', '9.0.0', '-current_version', '3.0.0'], check=True) + (lib / 'libsqlite3.dylib').symlink_to(dylib.name) + subprocess.run(['ar', 'rcs', lib / 'libsqlite3.a', obj], check=True) + for name in ('sqlite3.h', 'sqlite3ext.h'): + (include / name).write_bytes((source / name).read_bytes()) + (lib / 'pkgconfig/sqlite3.pc').write_text(f'prefix={prefix}\nlibdir=${{prefix}}/lib\nincludedir=${{prefix}}/include\nName: SQLite\nDescription: Checksum-pinned CI compatibility provider\nVersion: {version}\nLibs: -L${{libdir}} -lsqlite3\nCflags: -I${{includedir}}\n') + with open(os.environ['GITHUB_ENV'], 'a') as output: + print('PKG_CONFIG_PATH=' + str(lib / 'pkgconfig') + ':' + os.environ.get('PKG_CONFIG_PATH', ''), file=output) + if version == '3.49.2': + # A real but irrelevant PATH change must not alter the toolchain key. + empty = root / 'journal-empty-path' + empty.mkdir() + with open(os.environ['GITHUB_PATH'], 'a') as output: + print(empty, file=output) + print('SQLite comparison provider ready: version=' + version + ', source_sha256=' + expected) + + +if __name__ == '__main__': + main() diff --git a/.agents/skills/journal-build/scripts/test_journal_build.py b/.agents/skills/journal-build/scripts/test_journal_build.py index af1f1957..8fb16470 100644 --- a/.agents/skills/journal-build/scripts/test_journal_build.py +++ b/.agents/skills/journal-build/scripts/test_journal_build.py @@ -256,9 +256,7 @@ def test_ci_identity_is_stable_and_invalidates_required_inputs(self): compiler = f.build.prefix / 'bin/ocamlopt.opt' compiler.unlink() self.assertEqual(baseline, ci.cache_identity(f.journal)) - inputs = [f.lock, f.journal / 'journal.opam', - f.journal / '.ci-reports/dependencies.json', workflow, - f.journal / '.agents/skills/journal-build/scripts/prepare_ci_dependencies.py', f.journal / '.agents/skills/journal-build/scripts/journal_build.py'] + inputs = [f.lock, f.journal / 'journal.opam', f.journal / '.ci-reports/dependencies.json'] for path in inputs: with self.subTest(path=path): data = path.read_bytes() @@ -267,7 +265,45 @@ def test_ci_identity_is_stable_and_invalidates_required_inputs(self): path.write_bytes(changed) self.assertNotEqual(baseline['key'], ci.cache_identity(f.journal)['key']) path.write_bytes(data) - for name in ('ImageVersion', 'CFLAGS', 'DEVELOPER_DIR', 'OPAMROOT', 'CI_OPAM_VERSION', 'CI_COMPILER', 'CI_OPAM_REPOSITORY'): + for path in (workflow, fixture_scripts / 'prepare_ci_dependencies.py', fixture_scripts / 'journal_build.py'): + original = path.read_bytes() + path.write_bytes(original + b'\n# unrelated comment\n') + self.assertEqual(baseline['key'], ci.cache_identity(f.journal)['key']) + path.write_bytes(original) + with patch.dict(os.environ, {'ImageVersion': 'other-image', 'PATH': str(f.root / 'empty') + ':' + env['PATH']}): + moved = ci.cache_identity(f.journal) + self.assertEqual(baseline['key'], moved['key']) + self.assertEqual(baseline['native_key'], moved['native_key']) + self.assertNotEqual(baseline['observed'], moved['observed']) + native = f.root / 'sqlite-provider' + for sub in ('lib/pkgconfig', 'include'): + (native / sub).mkdir(parents=True) + (native / 'lib/libsqlite3.a').write_bytes(b'fixture native archive bytes') + for header in ('sqlite3.h', 'sqlite3ext.h'): + (native / 'include' / header).write_text('fixture header bytes') + pc = native / 'lib/pkgconfig/sqlite3.pc' + pc.write_text('prefix=' + str(native) + '\nlibdir=${prefix}/lib\nincludedir=${prefix}/include\n' + 'Name: sqlite3\nDescription: identity fixture\nVersion: 3.54.0\n' + 'Libs: -L${libdir} -lsqlite3\nCflags: -I${includedir}\n') + with patch.dict(os.environ, {'PKG_CONFIG_PATH': str(pc.parent) + ':' + env['PKG_CONFIG_PATH']}): + first = ci.cache_identity(f.journal) + pc.write_text(pc.read_text().replace('3.54.0', '3.53.4')) + (native / 'lib/libsqlite3.a').write_bytes(b'changed fixture native archive') + second = ci.cache_identity(f.journal) + self.assertEqual(first['key'], second['key']) + self.assertNotEqual(first['native_key'], second['native_key']) + empty = f.root / 'empty' + empty.mkdir() + wrapper = empty / 'cc' + wrapper.write_text('#!/bin/sh\nexec /usr/bin/cc "$@"\n') + wrapper.chmod(0o755) + with patch.dict(os.environ, {'PATH': str(empty) + ':' + env['PATH']}): + self.assertNotEqual(baseline['key'], ci.cache_identity(f.journal)['key']) + original = workflow.read_text() + workflow.write_text(original.replace('93303b622b2522e4411e295f9e77411a24912ac7', 'f' * 40)) + self.assertNotEqual(baseline['key'], ci.cache_identity(f.journal)['key']) + workflow.write_text(original) + for name in ('CFLAGS', 'DEVELOPER_DIR', 'OPAMROOT', 'CI_OPAM_VERSION', 'CI_COMPILER', 'CI_OPAM_REPOSITORY'): with self.subTest(environment=name), patch.dict(os.environ, {name: 'changed'}): if name in ('DEVELOPER_DIR', 'CI_COMPILER', 'CI_OPAM_REPOSITORY', 'CI_OPAM_VERSION'): # An unavailable selected toolchain must fail, not restore an old key. @@ -275,7 +311,7 @@ def test_ci_identity_is_stable_and_invalidates_required_inputs(self): ci.cache_identity(f.journal) else: self.assertNotEqual(baseline['key'], ci.cache_identity(f.journal)['key']) - for name, value in [('ImageVersion', ''), ('ImageOS', ''), ('RUNNER_ARCH', 'X64'), ('RUNNER_OS', 'Linux')]: + for name, value in [('RUNNER_ARCH', 'X64'), ('RUNNER_OS', 'Linux')]: with self.subTest(invalid=name), patch.dict(os.environ, {name: value}): with self.assertRaises(ValueError): ci.cache_identity(f.journal) @@ -292,9 +328,10 @@ def test_ci_identity_is_stable_and_invalidates_required_inputs(self): def test_ci_cache_receipt_accepts_exact_state_and_rejects_mismatches(self): import prepare_ci_dependencies as ci self.assertTrue(hasattr(ci, 'cache_receipt'), 'restored cache validation missing') + self.assertTrue(hasattr(ci, 'cache_key'), 'compatible cache family missing') with self.pinned_switch() as f: compiler = self.compiler_fixture(f.build.prefix) - report = dict(key='a' * 64, identity=dict(opam_prefix=str(f.build.prefix), + report = dict(key='a' * 64, native_key='d' * 64, observed={}, identity=dict(opam_prefix=str(f.build.prefix), opam_root=str(f.root / 'opam'), compiler='ocaml-base-compiler.5.5.0', opam_version=f.build.opam('--version'))) receipt = ci.cache_receipt(report, f.build.opam) @@ -304,20 +341,40 @@ def test_ci_cache_receipt_accepts_exact_state_and_rejects_mismatches(self): self.assertEqual(ci.validate_restored_cache(report, False)['status'], 'rejected') changed = dict(report, key='b' * 64) self.assertEqual(ci.validate_restored_cache(changed, True)['status'], 'rejected') + variant = dict(report, native_key='c' * 64, observed={'system_libraries': {'sqlite3': {'version': '3.49.2'}}}) + receipt['inputs']['native_key'] = 'd' * 64 + receipt['inputs']['observed'] = {'system_libraries': {'sqlite3': {'version': '3.50.4'}}} + receipt_path.write_text(json.dumps(receipt)) + match = ci.cache_key(receipt['inputs']) + restored = ci.validate_restored_cache(variant, False, matched_key=match) + self.assertEqual(restored['status'], 'accepted') + self.assertEqual(restored['source'], 'compatible') + incompatible = dict(variant, key='f' * 64, identity=dict(variant['identity'], compiler='ocaml-base-compiler.5.4.0')) + rejected = ci.validate_restored_cache(incompatible, False, matched_key=match) + self.assertEqual(rejected['status'], 'rejected') + self.assertTrue(any(d['field'].endswith('compiler') for d in rejected['differences'])) + report['observed'] = receipt['inputs']['observed'] for path in (compiler, f.build.prefix / '.opam-switch/switch-state', f.build.prefix / '.opam-switch/switch-config', f.root / 'opam/config'): with self.subTest(path=path): original = path.read_bytes() path.write_bytes(original + b'\nchanged\n') - self.assertEqual(ci.validate_restored_cache(report, True)['status'], 'rejected') + rejected = ci.validate_restored_cache(report, True) + self.assertEqual(rejected['status'], 'rejected') + self.assertTrue(any(d['field'].startswith('files.') and d['cached'] != d['current'] for d in rejected['differences'])) path.write_bytes(original) self.assertEqual(ci.validate_restored_cache(report, True)['status'], 'accepted') for malformed in ('{}', 'invalid json'): receipt_path.write_text(malformed) self.assertEqual(ci.validate_restored_cache(report, True)['status'], 'rejected') + for inputs in ('broken', [], None, dict(report, observed='broken')): + receipt_path.write_text(json.dumps(dict(receipt, inputs=inputs))) + self.assertEqual(ci.validate_restored_cache(report, True)['status'], 'rejected') receipt_path.write_text(json.dumps(receipt)) compiler.unlink() - self.assertEqual(ci.validate_restored_cache(report, True)['status'], 'rejected') + rejected = ci.validate_restored_cache(report, True) + self.assertEqual(rejected['status'], 'rejected') + self.assertTrue(any(d['current'] is None and d['field'].endswith('bin/ocamlopt.opt') for d in rejected['differences'])) receipt_path.unlink() self.assertEqual(ci.validate_restored_cache(report, True)['status'], 'rejected') @@ -352,6 +409,91 @@ def test_ci_rejection_quarantines_only_owned_runner_paths(self): with self.assertRaises(ValueError): ci.quarantine_cache(report) + def test_ci_diagnostics_are_truthful_specific_and_redacted(self): + import prepare_ci_dependencies as ci + self.assertTrue(hasattr(ci, 'cache_diagnostics'), 'specific miss diagnostics missing') + current = dict(key='b' * 64, native_key='c' * 64, + identity=dict(compiler='ocaml-base-compiler.5.5.0', environment={'CFLAGS': 'private-token-value'}, + workspace='/Users/private-person/project'), observed={}) + missing = ci.cache_diagnostics(current, None, 'miss', 'no accessible candidate') + self.assertFalse(missing['comparison_available']) + self.assertEqual(missing['differences'], []) + for invalid in ('broken', [], dict(current, observed='broken')): + self.assertFalse(ci.cache_diagnostics(current, invalid, 'miss', 'invalid prior record')['comparison_available']) + cached = dict(current, key='a' * 64, identity=dict(current['identity'], compiler='ocaml-base-compiler.5.4.0')) + result = ci.cache_diagnostics(current, cached, 'miss', 'compatibility key mismatch') + self.assertTrue(result['comparison_available']) + difference = next(d for d in result['differences'] if d['field'].endswith('compiler')) + self.assertEqual(difference['cached'], 'ocaml-base-compiler.5.4.0') + self.assertEqual(difference['current'], 'ocaml-base-compiler.5.5.0') + rendered = json.dumps(result) + self.assertNotIn('private-token-value', rendered) + self.assertNotIn('/Users/private-person', rendered) + same = ci.cache_diagnostics(current, current, 'miss', 'no archive despite matching input record') + self.assertTrue(same['comparison_available']) + self.assertEqual(same['differences'], []) + self.assertEqual(same['status'], 'miss') + + def test_native_symbol_probe_uses_real_dylib_and_rejects_missing_exports(self): + import prepare_ci_dependencies as ci + self.assertTrue(hasattr(ci, 'require_native_symbols'), 'native symbol validation missing') + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + source, library = root / 'fixture.c', root / 'libfixture.dylib' + source.write_text('int sqlite3_fixture(void) { return 42; }\n') + subprocess.run(['xcrun', '--sdk', 'macosx', 'clang', '-dynamiclib', str(source), '-o', str(library)], + check=True, capture_output=True) + ci.require_native_symbols(str(library), ['sqlite3_fixture']) + with self.assertRaisesRegex(RuntimeError, 'sqlite3_missing'): + ci.require_native_symbols(str(library), ['sqlite3_fixture', 'sqlite3_missing']) + library.unlink() + with self.assertRaises(RuntimeError): + ci.require_native_symbols(str(library), ['sqlite3_fixture']) + + def test_native_reinstall_rebuilds_real_reverse_dependents_only(self): + import prepare_ci_dependencies as ci + self.assertTrue(hasattr(ci, 'rebuild_native_packages'), 'selective native repair missing') + with self.pinned_switch() as f: + metadata = f.checkout / 'fixture.opam' + metadata.write_text(metadata.read_text() + 'build: [["sh" "-c" "echo fixture >> %{prefix}%/fixture-events"]]\n') + subprocess.run(['git', '-C', f.checkout, 'add', '.'], check=True, capture_output=True) + subprocess.run(['git', '-C', f.checkout, '-c', 'user.name=Build test', '-c', 'user.email=build@example.invalid', + 'commit', '-qm', 'fixture build marker'], check=True, capture_output=True) + f.build.opam('pin', 'add', '--no-action', 'fixture.dev', str(f.checkout)) + repo = f.root / 'repository' + for name, dependency in [('dependent', 'depends: ["fixture"]'), ('unrelated', '')]: + path = repo / f'packages/{name}/{name}.1/opam' + path.parent.mkdir(parents=True) + path.write_text('opam-version: "2.0"\nsynopsis: "Offline native repair fixture"\n' + dependency + + '\nbuild: [["sh" "-c" "echo ' + name + ' >> %{prefix}%/fixture-events"]]\n') + f.build.opam('update') + f.build.opam('install', '--assume-depexts', 'dependent', 'unrelated') + events = f.build.prefix / 'fixture-events' + events.write_text('') + repaired = ci.rebuild_native_packages(['fixture', 'not-installed'], f.build.opam) + self.assertEqual(repaired, ['fixture']) + self.assertEqual(events.read_text().splitlines(), ['fixture', 'dependent']) + events.write_text('') + self.assertEqual(ci.rebuild_native_packages(['not-installed'], f.build.opam), []) + self.assertEqual(events.read_text(), '') + + def test_sqlite_runtime_probe_reuses_installed_binding_read_only(self): + import prepare_ci_dependencies as ci + self.assertTrue(hasattr(ci, 'probe_sqlite_binding'), 'SQLite native/bytecode runtime probe missing') + switch = os.environ.get('JOURNAL_TEST_SWITCH') + if not switch: + self.skipTest('set JOURNAL_TEST_SWITCH to an already prepared switch; this test never provisions it') + prefix = Path(subprocess.check_output(['opam', 'var', '--safe', '--switch', switch, 'prefix'], text=True).strip()) + stubs = prefix / 'lib/sqlite3/libsqlite3_stubs.a' + before = self.build.sha(stubs) + result = ci.probe_sqlite_binding(prefix) + self.assertEqual(result['status'], 'compatible') + self.assertTrue(result['native_version'].startswith('3.')) + self.assertEqual(result['native_version'], result['bytecode_version']) + self.assertEqual(before, self.build.sha(stubs)) + with self.assertRaisesRegex(RuntimeError, 'version'): + ci.probe_sqlite_binding(prefix, expected_version='4.0.0') + def test_final_verification_rejects_changed_gmp_and_probe_input(self): self.assertTrue(hasattr(self.build.Build, 'verify_inputs'), 'final input verification boundary missing') with self.pinned_switch() as f: diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 7f0a67bf..77023313 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,10 +1,21 @@ name: Journal CI on: + workflow_call: + inputs: + sqlite-profile: + type: string + default: brew pull_request: push: branches: [main] +# Only a newer head for this PR supersedes its old run. Main and controlled +# comparisons use unique run groups and cannot cancel one another. +concurrency: + group: ${{ github.workflow }}-${{ github.event_name == 'pull_request' && format('pr-{0}', github.event.pull_request.number) || format('{0}-{1}', github.ref, github.run_id) }} + cancel-in-progress: ${{ github.event_name == 'pull_request' && github.event.action == 'synchronize' }} + permissions: contents: read @@ -14,6 +25,7 @@ jobs: runs-on: macos-latest timeout-minutes: 60 env: + CI_SQLITE_PROFILE: ${{ inputs.sqlite-profile || 'brew' }} CI_COMPILER: ocaml-base-compiler.5.5.0 # This commit binds compiler source checksums, patches, build commands and # every repository package recipe. Update deliberately with the toolchain. @@ -31,6 +43,8 @@ jobs: run: | brew install pkg-config gmp sqlite zlib zstd echo "PKG_CONFIG_PATH=$(brew --prefix gmp)/lib/pkgconfig:$(brew --prefix sqlite)/lib/pkgconfig:$(brew --prefix zlib)/lib/pkgconfig:$(brew --prefix zstd)/lib/pkgconfig" >> "$GITHUB_ENV" + - name: Prepare optional controlled SQLite provider + run: python3 .agents/skills/journal-build/scripts/prepare_ci_sqlite.py - name: Resolve declared dependency sources run: python3 .agents/skills/journal-build/scripts/prepare_ci_dependencies.py - name: Identify dependency build inputs before compiler setup @@ -45,7 +59,18 @@ jobs: print('prefix=' + report['identity']['opam_prefix'], file=output) print('root=' + report['identity']['opam_root'], file=output) print('key=' + report['key'], file=output) + print('native-key=' + report['native_key'], file=output) PY + # This small, data-only record explains misses outside the compatible + # switch family. It never authorizes executable state restoration. + - name: Restore prior input record for miss diagnostics + id: input-cache + uses: actions/cache/restore@v4 + continue-on-error: true + with: + path: .ci-reports/cache-candidate + key: journal-opam-input-v4-${{ runner.os }}-${{ runner.arch }}-${{ steps.dependency-identity.outputs.key }}-${{ steps.dependency-identity.outputs.native-key }} + restore-keys: journal-opam-input-v4-${{ runner.os }}-${{ runner.arch }}- - name: Restore compiler, installed dependencies and opam metadata id: dependency-cache uses: actions/cache/restore@v4 @@ -54,11 +79,14 @@ jobs: path: | ${{ steps.dependency-identity.outputs.prefix }} ${{ steps.dependency-identity.outputs.root }} - # Exact only: compiled switches are not relocatable or cross-platform. - key: journal-opam-v3-${{ runner.os }}-${{ runner.arch }}-${{ steps.dependency-identity.outputs.key }} + # Prefix fallback is confined to the same compiler/SDK/lock family. + # Native provider differences require receipt + real link/runtime probes. + key: journal-opam-v4-${{ runner.os }}-${{ runner.arch }}-${{ steps.dependency-identity.outputs.key }}-${{ steps.dependency-identity.outputs.native-key }} + restore-keys: journal-opam-v4-${{ runner.os }}-${{ runner.arch }}-${{ steps.dependency-identity.outputs.key }}- - name: Validate restored cache or select a clean cold setup env: CI_CACHE_HIT: ${{ steps.dependency-cache.outputs.cache-hit }} + CI_CACHE_MATCHED_KEY: ${{ steps.dependency-cache.outputs.cache-matched-key }} run: python3 .agents/skills/journal-build/scripts/prepare_ci_dependencies.py --validate-cache # Pinned because its provisioning behavior participates in the input key. - uses: ocaml/setup-ocaml@93303b622b2522e4411e295f9e77411a24912ac7 # v3 @@ -71,8 +99,18 @@ jobs: run: python3 .agents/skills/journal-build/scripts/prepare_ci_dependencies.py --pin - name: Verify native system dependencies on the fresh runner run: opam install . --deps-only --with-test --locked --ignore-pin-depends --depext-only --yes + - name: Probe restored native bindings and repair incompatible packages only + run: python3 .agents/skills/journal-build/scripts/prepare_ci_dependencies.py --check-native-cache - name: Install locked dependencies run: opam install . --deps-only --with-test --locked --ignore-pin-depends --yes + - name: Verify build and cache helper boundaries + env: + # Apple CLI boundary fixtures deliberately reject ambient pkg-config. + # The CI identity fixture selects its own explicit providers. + PKG_CONFIG_PATH: '' + JOURNAL_TEST_SWITCH: ${{ github.workspace }} + PYTHONDONTWRITEBYTECODE: 1 + run: python3 .agents/skills/journal-build/scripts/test_journal_build.py - name: Record installed package versions if: always() run: opam list --installed --columns=name,installed-version --short --color=never > .ci-reports/packages.txt @@ -90,12 +128,18 @@ jobs: path: | ${{ steps.dependency-identity.outputs.prefix }} ${{ steps.dependency-identity.outputs.root }} - key: journal-opam-v3-${{ runner.os }}-${{ runner.arch }}-${{ steps.dependency-identity.outputs.key }} + key: journal-opam-v4-${{ runner.os }}-${{ runner.arch }}-${{ steps.dependency-identity.outputs.key }}-${{ steps.dependency-identity.outputs.native-key }} + - name: Save successful input record for miss diagnostics + if: success() && steps.input-cache.outputs.cache-hit != 'true' + uses: actions/cache/save@v4 + with: + path: .ci-reports/cache-candidate + key: journal-opam-input-v4-${{ runner.os }}-${{ runner.arch }}-${{ steps.dependency-identity.outputs.key }}-${{ steps.dependency-identity.outputs.native-key }} - name: Upload resolved dependencies and test logs if: always() uses: actions/upload-artifact@v4 with: - name: journal-ci-evidence + name: journal-ci-evidence${{ inputs.sqlite-profile && format('-{0}', inputs.sqlite-profile) || '' }} include-hidden-files: true if-no-files-found: error path: |