Allow a UK local-area release on an accepted model mismatch (hold: needs María's routing call)

changelog.d/699.fixed.md

@@ -0,0 +1,3 @@
+Require a byte-bound input-surface audit when accepting a UK regional model mismatch, refuse dropped columns, and record defaulted inputs and limitations in the bundle, TRACE and execution receipts. Preserve the regional manifest digest and area build model while keeping area receipts uncertified; dropping the regional release removes its exception record.
+
+Refuse regional releases that explicitly name a different model package, and forward empty CLI reasons so certification rejects them. Document that the UK loader silently skips unknown columns.

changelog.d/703.added.md

@@ -0,0 +1 @@
+Support opt-in local data release manifests pinned by the installed bundle TRO, with fail-closed verification and a distinct runtime certification basis for offline Budget-day runners. Reject conflicting TRO hash assertions and require local and fetched manifest artifacts to agree with the installed bundle.

changelog.d/accept-regional-model-mismatch.added.md

@@ -0,0 +1 @@
+`certify-data --accept-regional-model-mismatch REASON` (UK only) keeps a local-area release routed when it was built on another policyengine-uk version, and records the reviewed mismatch as `regional_certification` in the bundle instead of refusing; area receipts are uncertified.

docs/bundles.md

@@ -116,6 +116,68 @@ policyengine bundle manifest 4.19.1
 `install` targeted. Use `--venv` or `--python` only to inspect a different
 target explicitly.
 
+## Local release manifests for Budget-day runners
+
+Set `POLICYENGINE_UK_DATA_MANIFEST_FILE` before starting a generation process to
+read the UK release manifest from that file. The same setting is available as
+`POLICYENGINE_<COUNTRY>_DATA_MANIFEST_FILE` for other bundled countries. When set,
+it always selects the local file, avoiding a Hugging Face request and its timeout.
+An unset setting preserves the existing fetch and certification behavior.
+
+The trust anchor is the `trov:sha256` on
+`composition/1/artifact/data_release_manifest` in the installed country's
+`data/bundle/{country}.trace.tro.jsonld`. The file's **raw bytes** must match that
+pin. A missing pin, unreadable file, altered bytes, or certification mismatch
+stops the process. Every definition of the artifact ID in the TRO must agree on
+one digest. After revision normalization, the manifest's certified/default
+artifact and all artifacts shared with the bundle must agree with the bundle's
+locations and pinned hashes. This artifact check also applies to fetched
+manifests. The model, core, build ID and fingerprint checks still apply;
+selecting a local file cannot extend the bundle's certification.
+
+At G0 each morning, while Hugging Face is available, use the exact installed
+release that the runner will use and download and verify the manifest before
+enabling the setting. Keep the Hugging Face token in `HUGGING_FACE_TOKEN`,
+`HF_TOKEN` or `HUGGINGFACE_HUB_TOKEN`:
+
+```bash
+env -u POLICYENGINE_UK_DATA_MANIFEST_FILE POLICYENGINE_SKIP_COUNTRY_IMPORTS=1 python - <<'PY' &&
+import hashlib
+from pathlib import Path
+
+import requests
+from policyengine.provenance.manifest import (
+    bundled_data_release_manifest_sha256, get_release_manifest,
+    hugging_face_auth_headers, https_release_manifest_uri,
+)
+
+expected = bundled_data_release_manifest_sha256("uk")
+response = requests.get(
+    https_release_manifest_uri(get_release_manifest("uk").data_package),
+    headers=hugging_face_auth_headers(), timeout=30,
+)
+response.raise_for_status()
+actual = hashlib.sha256(response.content).hexdigest()
+if actual != expected:
+    raise RuntimeError(f"Manifest SHA-256: expected {expected}; actual {actual}")
+target = Path("data/uk-release-manifest.json")
+target.parent.mkdir(parents=True, exist_ok=True)
+staged = target.with_suffix(".json.tmp")
+staged.write_bytes(response.content)
+staged.replace(target)
+print(f"Verified {target}: {actual}")
+PY
+export POLICYENGINE_UK_DATA_MANIFEST_FILE="$PWD/data/uk-release-manifest.json"
+```
+
+Start a fresh runner process with that setting and record its receipt at G0.
+Runtime certification and TRACE output use a distinct basis ending in
+`_local_manifest`, such as `exact_build_model_version_local_manifest`; packaged
+execution receipts record it as `runtime.data_certification_basis`. The
+receipt's certificate keeps the bundle's published basis so certificate
+verification remains reproducible. See [execution receipts](receipts.md).
+Refresh and verify the file whenever the installed certified release changes.
+
 ## Bundle-only PRs
 
 Run:

docs/engineering/skills/data-certification.md

@@ -79,6 +79,80 @@ Microcosm national release with one of:
   claim first. Its `frs_2024_25` is not carried, so keep it loadable with a
   `dataset_overlays.uk` entry.
 
+- **An older local-area release, on an accepted mismatch.** When the national
+  cut moves to a new policyengine-uk before a local-area cut on that version
+  exists, a reviewer can accept retaining the older regional route. A reason
+  and a byte-bound input-surface audit are both required. First audit the
+  actual local H5 with the exact target model installed:
+
+  ```bash
+  .venv/bin/python scripts/audit_regional_input_surface.py \
+    --regional-manifest-uri "<the routed local-area release_manifest.json URI>" \
+    --artifact "microcosm_uk_2024_25_dense=/local/dense.h5" \
+    --model-version 2.147.2 \
+    --audited-by "<reviewer>" \
+    --limitation "the dense cut's VAT predates the policyengine-uk VAT factor fix (#2187, 2.135.7); the dashboard holds a VAT limit" \
+    --output regional-input-surface.json
+
+  .venv/bin/python scripts/bundle.py certify-data --country uk --data-producer populace \
+    --manifest-uri "hf://dataset/policyengine/populace-uk-private@<national-cut-tag>/releases/microcosm-uk-2024-25-national/release_manifest.json" \
+    --regional-manifest-uri "<the routed local-area release_manifest.json URI>" \
+    --accept-regional-model-mismatch "<who accepted it and why>" \
+    --regional-input-surface-evidence regional-input-surface.json \
+    --model-version 2.147.2
+  ```
+
+  The comparison is ported from Microcosm #1348's
+  `tools/certify_uk_cut_in_policyengine_py.py`, and strengthened to inspect the
+  persisted entity-table columns, including each period and country-loader
+  aliases. Comparing the old model's input inventory alone cannot prove which
+  columns the file carries. The audit binds the regional release identity,
+  original manifest bytes, actual H5 bytes and exact target model version.
+  Certification refuses missing or mismatched evidence and **any dropped
+  column**. It records every defaulted input and the free-text limitations in
+  the validated `regional_certification`; the bundle hash binds that record.
+
+  The policyengine-uk 2.147.2 dataset loaders silently skip columns absent
+  from the model's variables: both the
+  [generic dataset loader](https://github.com/PolicyEngine/policyengine-uk/blob/fe0b99a36bd309a722b19937345e34290cb6dd73/policyengine_uk/simulation.py#L1113-L1127)
+  and the
+  [UK multi-year loader](https://github.com/PolicyEngine/policyengine-uk/blob/fe0b99a36bd309a722b19937345e34290cb6dd73/policyengine_uk/simulation.py#L1186-L1196)
+  use `continue` without warning. A removed or renamed input can therefore
+  disappear silently at runtime; the audit must catch that column loss before
+  certification.
+
+  The national certificate never widens. Area status remains `uncertified`:
+  the certificate still covers the national dataset only (#685). TRACE and
+  execution receipts expose the regional identity, build model, acceptance,
+  reason, input-surface result and limitations separately from the national
+  certification. The flag remains UK only; an empty reason, a release that
+  already covers the target model, or a missing regional release refuses.
+  The regional core checks still apply. Re-certifying with
+  `--drop-regional-release` removes the regional record, routes and artifact.
+
+  For the Budget route, the measured **model-inventory comparison** from
+  2.132.6 to 2.147.2 has no dropped names and these 16 new pure inputs:
+  `additional_residential_property_purchased`, `defined_benefit_pension_share`,
+  `directly_held_shares`, `early_years_school_start_deferred`,
+  `is_full_year_category_a_director`, `local_bus_uncapped_single_fare`,
+  `non_residential_property_purchased`, `non_residential_property_secured_debt`,
+  `other_residential_property_secured_debt`,
+  `pension_annual_allowance_taxable_lump_sum_death_benefits`,
+  `pension_contributions_via_salary_sacrifice_pre_2015`, `pip_dl_points`,
+  `pip_m_points`, `salary_sacrifice_returned_to_income`,
+  `unit_and_investment_trusts`, and `war_widows_pension_reported`.
+  Both `salary_sacrifice_returned_to_income` and
+  `local_bus_uncapped_single_fare` default in that comparison. This is not an
+  audit of cut `microcosm-uk-2024-25-dense-20261019T000432Z-ee5c2dbe`:
+  its private H5 was unavailable to this implementation job without
+  authentication. Certification must use the complete list measured from the
+  actual cut by the audit above, including any further missing stored inputs.
+  Record the pre-fix VAT limitation shown above for this dense cut.
+
+  After the receipt schema changes, refresh the Budget dashboard's vendored
+  `tests/fixtures/w4/execution_receipt.v1.schema.json` from
+  `src/policyengine/data/schemas/execution_receipt.v1.schema.json`.
+
 Either way, certification checks that the regional release serves the pinned
 model by its own build or compatibility claims (a `--qualified-pair-id` names
 the primary release only), checks its core claims as for the primary release

docs/receipts.md

@@ -47,6 +47,25 @@ receipt_json = receipt.to_canonical_json()
 
 For a country `Microsimulation` built with `managed_microsimulation()`, call `build_execution_receipt_for_microsimulation(sim, years=..., reform=..., baseline=...)`. It reads the dataset that was loaded from `sim.policyengine_bundle`. The file must still hash to the SHA-256 verified when it was loaded, and the receipt must name the dataset, URI, revision and package recorded then; otherwise it is refused, because it would describe bytes that did not run. A simulation loaded from an unmanaged file or URI recorded no load-time hash, so it cannot be receipted. The microsimulation does not record its reform, so pass the reform you built it with.
 
+With `POLICYENGINE_UK_DATA_MANIFEST_FILE` enabled, a packaged-bundle receipt
+rechecks runtime certification using the local file and records its distinct
+`*_local_manifest` basis in `runtime.data_certification_basis`. Local-file
+verification errors stop receipt creation. The certificate's
+`compatibility_basis` stays the published bundle's assertion, which
+`verify_certificate` can reproduce. Like the other runtime context, this source
+annotation is excluded from execution and receipt digests. Caller-supplied
+bundles do not claim this packaged-bundle runtime annotation.
+
+The execution-receipt schema accepts the annotation as an optional string;
+compatibility bases in the certificate and TRACE TRO are also strings. The UK
+Budget dashboard's receipt adapter uses this package's receipt model and
+`verify_certificate`, so it needs no basis allowlist change. When adopting this
+release, refresh the dashboard's vendored
+`tests/fixtures/w4/execution_receipt.v1.schema.json`: its
+`tests/test_released_receipt.py` checks exact equality against the installed
+schema. Older consumers that forbid extra runtime fields need the updated
+schema before reading these receipts.
+
 There are two ways to bind a policyengine-uk `Scenario`:
 - A scenario built from `parameter_changes` is bound by content with `scenario_reform_identity(scenario)`.
 - `Scenario.from_reform(d)` wraps `d` in a closure that cannot be bound by its code, so bind `d` itself with `reform_identity(d, "policyengine-uk.reform_dict/v1")`.

scripts/audit_regional_input_surface.py

@@ -0,0 +1,56 @@
+"""Write a byte-bound audit of regional H5 columns for an installed UK model.
+
+Run in an environment with the exact requested policyengine-uk version and
+PyTables installed. Supply the H5 files from the immutable regional manifest;
+the audit verifies their hashes without installing models or downloading data.
+"""
+
+from __future__ import annotations
+
+import argparse
+import os
+import sys
+from pathlib import Path
+
+REPO_ROOT = Path(__file__).resolve().parents[1]
+# The target model must be auditable before the current bundle certifies it.
+os.environ.setdefault("POLICYENGINE_SKIP_COUNTRY_IMPORTS", "1")
+sys.path.insert(0, str(REPO_ROOT / "src"))
+
+from policyengine.provenance.regional_input_surface import (  # noqa: E402
+    measure_regional_input_surface,
+)
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--regional-manifest-uri", required=True)
+    parser.add_argument(
+        "--artifact", required=True, action="append", metavar="NAME=PATH"
+    )
+    parser.add_argument("--model-version", required=True)
+    parser.add_argument("--audited-by", required=True)
+    parser.add_argument("--limitation", required=True, action="append")
+    parser.add_argument("--output", required=True, type=Path)
+    args = parser.parse_args(argv)
+    paths: dict[str, Path] = {}
+    for entry in args.artifact:
+        name, separator, path = entry.partition("=")
+        if not separator or not name.strip() or not path.strip() or name in paths:
+            parser.error("Each --artifact must be a unique nonblank NAME=PATH.")
+        paths[name] = Path(path)
+    evidence = measure_regional_input_surface(
+        regional_manifest_uri=args.regional_manifest_uri,
+        artifact_paths=paths,
+        model_version=args.model_version,
+        audited_by=args.audited_by,
+        limitations=args.limitation,
+        token=os.environ.get("HUGGING_FACE_TOKEN") or os.environ.get("HF_TOKEN"),
+    )
+    args.output.write_text(evidence.model_dump_json(indent=2) + "\n")
+    print(f"Saved {len(evidence.artifacts)} artifact input audits to {args.output}.")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

scripts/bundle.py

@@ -48,6 +48,14 @@ def _certify_data(args: argparse.Namespace) -> int:
         argv.extend(["--regional-release-kind", args.regional_release_kind])
     if args.drop_regional_release:
         argv.append("--drop-regional-release")
+    if args.accept_regional_model_mismatch is not None:
+        argv.extend(
+            ["--accept-regional-model-mismatch", args.accept_regional_model_mismatch]
+        )
+    if args.regional_input_surface_evidence:
+        argv.extend(
+            ["--regional-input-surface-evidence", args.regional_input_surface_evidence]
+        )
     if args.no_generate:
         argv.append("--no-generate")
     if args.no_changelog:
@@ -226,6 +234,19 @@ def _parser() -> argparse.ArgumentParser:
             "Cannot be combined with --regional-manifest-uri."
         ),
     )
+    certify.add_argument(
+        "--accept-regional-model-mismatch",
+        metavar="REASON",
+        help=(
+            "UK only: route regions to a regional release on another model "
+            "version, recording this reviewed reason; forwarded to "
+            "scripts/certify_data_release.py."
+        ),
+    )
+    certify.add_argument(
+        "--regional-input-surface-evidence",
+        help="Audited regional input-surface JSON; required for an accepted mismatch.",
+    )
     certify.add_argument(
         "--regional-artifact-prefix",
         help="Legacy per-state artifact prefix to import. Defaults to states/.",

scripts/certify_data_release.py

@@ -24,6 +24,7 @@
 from __future__ import annotations
 
 import argparse
+import json
 import os
 import sys
 from pathlib import Path
@@ -115,7 +116,22 @@ def main(argv=None) -> int:
             "qualification of this pair (requires --core-version)."
         ),
     )
+    parser.add_argument(
+        "--accept-regional-model-mismatch",
+        metavar="REASON",
+        help=(
+            "UK only: route regions to a regional release that does not cover "
+            "the pinned model, recording this reviewed reason in the bundle "
+            "instead of refusing. Requires --regional-input-surface-evidence; "
+            "dropped columns refuse. The core must still be covered."
+        ),
+    )
     parser.add_argument("--no-generate", action="store_true")
+    parser.add_argument(
+        "--regional-input-surface-evidence",
+        type=Path,
+        help="Audited JSON evidence bound to the regional bytes and exact target model.",
+    )
     parser.add_argument("--no-changelog", action="store_true")
     parser.add_argument(
         "--skip-artifact-check",
@@ -149,6 +165,12 @@ def main(argv=None) -> int:
         drop_regional_release=args.drop_regional_release,
         core_version=args.core_version,
         qualified_pair_id=args.qualified_pair_id,
+        regional_model_mismatch_reason=args.accept_regional_model_mismatch,
+        regional_input_surface_evidence=(
+            json.loads(args.regional_input_surface_evidence.read_text())
+            if args.regional_input_surface_evidence is not None
+            else None
+        ),
     )
     print(result.summary())