diff --git a/changelog.d/699.fixed.md b/changelog.d/699.fixed.md new file mode 100644 index 00000000..08497a7f --- /dev/null +++ b/changelog.d/699.fixed.md @@ -0,0 +1,3 @@ +Require a byte-bound input-surface audit when accepting a UK regional model mismatch. The audit compares the regional H5's stored columns with the regional release's build model as well as the target model: it refuses columns the build model defines and the target does not, records columns neither model defines as never modelled, and reports as defaulted only target inputs the file does not store that were not build-model inputs. Certification reads both models again rather than trusting the evidence's lists, and adds a limitation naming the defaulted inputs. The bundle and TRACE record the result and limitations and preserve the regional manifest digest and area build model; an area run's execution receipt carries a summary bound to the record's digest and stays uncertified. Dropping the regional release removes its exception record. + +Refuse regional releases that explicitly name a different model package, and forward empty CLI reasons so certification rejects them, as it rejects reasons with no visible character. Document that the UK loader silently skips unknown columns. diff --git a/changelog.d/accept-regional-model-mismatch.added.md b/changelog.d/accept-regional-model-mismatch.added.md new file mode 100644 index 00000000..22626b70 --- /dev/null +++ b/changelog.d/accept-regional-model-mismatch.added.md @@ -0,0 +1 @@ +`certify-data --accept-regional-model-mismatch REASON --regional-input-surface-evidence AUDIT.json` (UK only) keeps a local-area release routed when it was built on another policyengine-uk version, and records the reviewed mismatch as `regional_certification` in the bundle instead of refusing; area receipts are uncertified. `scripts/audit_regional_input_surface.py` writes the audit, reading the release's build model in a throwaway `uv run` environment, and `verify_regional_certification` checks an area receipt's record of the exception. diff --git a/docs/engineering/skills/data-certification.md b/docs/engineering/skills/data-certification.md index 23c5d813..fd54ffc3 100644 --- a/docs/engineering/skills/data-certification.md +++ b/docs/engineering/skills/data-certification.md @@ -79,6 +79,129 @@ Microcosm national release with one of: claim first. Its `frs_2024_25` is not carried, so keep it loadable with a `dataset_overlays.uk` entry. +- **An older local-area release, on an accepted mismatch.** When the national + cut moves to a new policyengine-uk before a local-area cut on that version + exists, a reviewer can accept retaining the older regional route. A reason + and a byte-bound input-surface audit are both required. First audit the + actual local H5 with the exact target model installed. The audit also needs + `uv` and access to PyPI (see below): + + ```bash + .venv/bin/python scripts/audit_regional_input_surface.py \ + --regional-manifest-uri "" \ + --artifact "microcosm_uk_2024_25_dense=/local/dense.h5" \ + --model-version 2.147.2 \ + --audited-by "" \ + --limitation "the dense cut's VAT predates the policyengine-uk VAT factor fix (#2187, 2.135.7); the dashboard holds a VAT limit" \ + --output regional-input-surface.json + + .venv/bin/python scripts/bundle.py certify-data --country uk --data-producer populace \ + --manifest-uri "hf://dataset/policyengine/populace-uk-private@/releases/microcosm-uk-2024-25-national/release_manifest.json" \ + --regional-manifest-uri "" \ + --accept-regional-model-mismatch "" \ + --regional-input-surface-evidence regional-input-surface.json \ + --model-version 2.147.2 + ``` + + The audit compares the entity-table columns the H5 actually stores, in each + period, with two models: the build model and core the regional manifest + records (`build.built_with_model_package` and + `build.built_with_core_package`), and the target. As in Microcosm #1348's + `tools/certify_uk_cut_in_policyengine_py.py`, the build model is the + baseline: + + - **Dropped columns** are stored columns the build model defines and the + target does not. The target's loaders skip them. Certification refuses + **any dropped column**; there is no override. + - **Never-modelled columns** are stored columns neither model defines: + Microcosm's own area codes (among them `constituency_code_oa` and + `la_code_oa`), build and source ids, clone indices and SPI donor fields. + No model ever loaded them, so they are recorded, not refused. + - **Defaulted inputs** are target inputs the file does not store, after the + target's input aliases, that were not inputs of the build model. Area runs + default them where the build model did not. Inputs absent under both + models are not changes. + + The audit reads the build model's variables and inputs in a throwaway + `uv run --no-project --no-config --default-index https://pypi.org/simple + --exclude-newer --with policyengine-uk== --with + policyengine-core==` environment, so it needs `uv` and access to + PyPI. `--exclude-newer` is the manifest's build time, so the build model's + other dependencies resolve as they did then, and every `UV_*` setting + except the cache, Python-install and connection ones is removed. A regional + manifest that does not record both build versions has no baseline, and the + audit refuses it. The evidence binds the regional release identity, + original manifest bytes, actual H5 bytes, both build versions and the exact + target model version, and keeps both models' full variable and input lists. + + Certification does not trust those lists. It reads the target model in + process, so the environment that runs `certify-data` must have the target + policyengine-uk installed, as the audit's does, and reads the build model + again with `uv`; any list or input alias that differs refuses. Only the + stored columns rest on the auditor's word: they cannot be re-read without + the H5, and the evidence binds them to its SHA-256. Certification then + recomputes all three lists and refuses missing or mismatched evidence, + including evidence without `schema_version: 2`, from before the comparison + was relative to the build model. It records the never-modelled columns, + every defaulted input and the free-text limitations in the validated + `regional_certification`, and adds one limitation naming the defaulted + inputs; the bundle hash binds that record. + + The policyengine-uk 2.147.2 dataset loaders silently skip columns absent + from the model's variables: both the + [generic dataset loader](https://github.com/PolicyEngine/policyengine-uk/blob/fe0b99a36bd309a722b19937345e34290cb6dd73/policyengine_uk/simulation.py#L1113-L1127) + and the + [UK multi-year loader](https://github.com/PolicyEngine/policyengine-uk/blob/fe0b99a36bd309a722b19937345e34290cb6dd73/policyengine_uk/simulation.py#L1186-L1196) + use `continue` without warning. A removed or renamed input can therefore + disappear silently at runtime; the audit must catch that column loss before + certification. + + The national certificate never widens. Area status remains `uncertified`: + the certificate still covers the national dataset only (#685). TRACE + carries the whole regional record, evidence included, separately from the + national certification. An area run's execution receipt carries a summary + of it (both model versions, the reason, the three lists and the + limitations) and `record_sha256`, the digest of the whole record; check it + with `verify_regional_certification`. A national receipt carries none: its + `certificate_sha256` already digests the record. The flag remains UK only; + an empty or invisible reason, a release that already covers the target + model, or a missing regional release refuses. + The regional core checks still apply. Re-certifying with + `--drop-regional-release` removes the regional record, routes and artifact. + + For the Budget route, the audit of dense cut + `microcosm-uk-2024-25-dense-20261019T000432Z-ee5c2dbe`, built with + policyengine-uk 2.132.6 and policyengine-core 3.33.3, measured in #699: + + - **83 never-modelled columns**, and **no dropped column** under either + 2.147.2 or 2.151.0. The first version of this audit counted those 83 as + dropped and refused the cut, even against its own build model. + - **16 new defaults under 2.147.2:** + `additional_residential_property_purchased`, + `defined_benefit_pension_share`, `directly_held_shares`, + `early_years_school_start_deferred`, `is_full_year_category_a_director`, + `local_bus_uncapped_single_fare`, `non_residential_property_purchased`, + `non_residential_property_secured_debt`, + `other_residential_property_secured_debt`, + `pension_annual_allowance_taxable_lump_sum_death_benefits`, + `pension_contributions_via_salary_sacrifice_pre_2015`, `pip_dl_points`, + `pip_m_points`, `salary_sacrifice_returned_to_income`, + `unit_and_investment_trusts`, and `war_widows_pension_reported`. + - **37 new defaults under 2.151.0:** those 16 and 21 more. The 21 include + the gambling duty inputs, liquid and solid fuel consumption, and bonus + sacrifice (`pension_contributions_via_bonus_sacrifice`), so area + breakdowns of those measures use defaults on 2.151.0. Certification + names every defaulted input in the limitations. + - Another **123 inputs already defaulted under 2.132.6**. They are absent + under both models, so they are not changes; the first version listed + them too, giving 139 defaults under 2.147.2 and 160 under 2.151.0. + + Record the pre-fix VAT limitation shown above for this dense cut. + + After the receipt schema changes, refresh the Budget dashboard's vendored + `tests/fixtures/w4/execution_receipt.v1.schema.json` from + `src/policyengine/data/schemas/execution_receipt.v1.schema.json`. + Either way, certification checks that the regional release serves the pinned model by its own build or compatibility claims (a `--qualified-pair-id` names the primary release only), checks its core claims as for the primary release diff --git a/docs/receipts.md b/docs/receipts.md index 48686f6e..c39bcded 100644 --- a/docs/receipts.md +++ b/docs/receipts.md @@ -11,6 +11,7 @@ Schema v1 covers UK results and is defined by `policyengine.provenance.receipt.E | `wrapper`, `model`, `core` | The `policyengine`, `policyengine-uk` and `policyengine-core` versions that actually ran, read from the installed distributions rather than from the bundle manifest. `install_origin` records a PEP 610 direct URL as `archive`, `directory`, `editable` or `vcs`; it is absent for an ordinary index install with no `direct_url.json`. `wheel_sha256` appears only when the receipt builder verified it. One way is `wheel_paths=` (evidence `installed_record_verified`): the installed files, and any bytecode cache the interpreter would load instead of a source file, are checked against that wheel's RECORD. The other is the installer's recorded archive hash (`installer_archive_hash`), taken only when the install came from a `.whl` URL; a source archive's hash is never recorded as a wheel hash. | | `data` | The data release and build, and the model version the data was built with (`build_model_version`, distinct from `model.version`). Also the dataset's logical name, URI, revision and the SHA-256 of the bytes that were simulated. The dataset is resolved exactly as `materialize_dataset` resolves it (overlays, per-dataset repository, package and revision, and the release manifest revision when an artifact has none of its own), and the bytes must be the ones the bundle pins for it: a receipt never gives an artifact's name to other bytes. | | `certification_status`, `certificate` | `certified` only when the bundle certifies all of: the installed model and core versions, that dataset name and those bytes, and the running `policyengine` version. The UK certificate names one dataset, the national default, so a run on any other dataset, the local-area file that constituency and local-authority regions load included, gets an uncertified receipt. Each package must also have an absent or `archive` install origin, or a `directory` or `vcs` origin with `installed_record_verified` wheel evidence. A directory or VCS install without that evidence produces an `uncertified` receipt, which carries no certificate. An editable install is never certified, whatever the evidence: its wheel holds a pointer to a source tree (a `.pth` file or an import hook) rather than the source, so checking the installed files against it does not bind the code that ran. `certificate_source` records whether the builder read the certificate from the installed bundle (`packaged`) or from a manifest the caller passed (`caller_supplied`). It is the producer's statement, not evidence: trust a certificate only after `verify_certificate` (below). With the packaged bundle, the composition fingerprint of its TRACE TRO is recorded as well. | +| `regional_certification` | Present only on an area run: a run on the regional release that the bundle routes constituency and local-authority regions to on an accepted model mismatch (`data_releases.uk.regional_certification`). Such a run is uncertified. The member summarises that exception: the model the release was built with and the one area runs use, the reviewer's reason, the dropped, never-modelled and defaulted lists, and the limitations. `record_sha256` binds the whole record, its evidence included; the evidence's model surfaces stay in the bundle and its TRACE TRO. A national receipt carries none, because its `certificate_sha256` already digests the record. Check it with `verify_regional_certification` (below). | | `reform`, `baseline` | Both are required. Each is `current_law` or a digest of the reform bound to its representation (see below). | | `fiscal_year` | What a year label means: year 2027 is the fiscal year from 6 April 2027 to 5 April 2028, shown as `2027-28`. v1 allows only this convention. | | `years` | The reported years, sorted and unique. | @@ -149,6 +150,8 @@ A reader who holds the members' identities can recompute the composition's diges To check a certificate, call `verify_certificate(receipt, bundle)` with the policyengine manifest for `certificate.bundle_version`. It rebuilds the certificate from that manifest's certification record, the same way the builder does. Then it compares every identity member: the record digest, the basis, the certified model, core and dataset digest, the certifier and the bundle version. It also checks that the receipt's whole `data` identity is the certified dataset with the certified bytes. An empty list means the bundle certifies exactly what the receipt claims, and through the receipt's own rules that pins `model.version`, `core.version` and `wrapper.version` too. `certificate_source`, `qualification_receipt_sha256`, `tro_composition_fingerprint` and wheel hashes are not determined by the bundle, so it does not check them. +To check an area receipt's regional exception, call `verify_regional_certification(receipt, bundle)` with the policyengine manifest the run used. It rebuilds the summary from that manifest's `regional_certification` record, the same way the builder does, and compares every member, `record_sha256` included. It also checks that the receipt's `data` is the regional artifact that record covers, with the identity the bundle determines. An empty list means the receipt names exactly the bundle's exception; the area run stays uncertified. + Pass the release artifact that was loaded as `dataset_path`. A derived per-year file such as `enhanced_frs_2024_25_year_2026.h5` has different bytes from the artifact the bundle pins, so a receipt for it is refused. ## Digests @@ -195,7 +198,8 @@ Consumers that validate with the JSON Schema rather than the Python model must a - a `current_law` reform has no digest, while any other reform has both a representation and a digest; - a wheel digest always comes with its evidence; - a certified receipt has no `editable` package, and no `directory` or `vcs` package without `installed_record_verified` wheel evidence; -- a certificate is trusted only after it, and the receipt's `data`, have been checked against the published manifest it names (`verify_certificate`). +- a certificate is trusted only after it, and the receipt's `data`, have been checked against the published manifest it names (`verify_certificate`); +- `regional_certification` appears only on an uncertified area run, and is checked against the manifest the run used (`verify_regional_certification`). The schema itself fixes the package names (`policyengine`, `policyengine-uk`, `policyengine-core`) and the UK fiscal-year convention, which starts on 6 April. diff --git a/scripts/audit_regional_input_surface.py b/scripts/audit_regional_input_surface.py new file mode 100644 index 00000000..1c18cb37 --- /dev/null +++ b/scripts/audit_regional_input_surface.py @@ -0,0 +1,70 @@ +"""Write a byte-bound audit of regional H5 columns for an installed UK model. + +Run in an environment with the exact requested policyengine-uk version and +PyTables installed. Supply the H5 files from the immutable regional manifest; +the audit verifies their hashes and downloads no microdata. The baseline is the +build model and core the regional manifest records: the audit reads their +variables in a throwaway ``uv run --with`` environment, resolved from PyPI as +of the manifest's build time, so it needs ``uv`` and PyPI. Certification reads +both models again. Columns the build model defines and the target does not +refuse; columns neither model defines are recorded as never modelled; and +target inputs the files do not store that were not build inputs are recorded +as defaulted. +""" + +from __future__ import annotations + +import argparse +import os +import sys +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parents[1] +# The target model must be auditable before the current bundle certifies it. +os.environ.setdefault("POLICYENGINE_SKIP_COUNTRY_IMPORTS", "1") +sys.path.insert(0, str(REPO_ROOT / "src")) + +from policyengine.provenance.regional_input_surface import ( # noqa: E402 + measure_regional_input_surface, +) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--regional-manifest-uri", required=True) + parser.add_argument( + "--artifact", required=True, action="append", metavar="NAME=PATH" + ) + parser.add_argument("--model-version", required=True) + parser.add_argument("--audited-by", required=True) + parser.add_argument("--limitation", required=True, action="append") + parser.add_argument("--output", required=True, type=Path) + args = parser.parse_args(argv) + paths: dict[str, Path] = {} + for entry in args.artifact: + name, separator, path = entry.partition("=") + if not separator or not name.strip() or not path.strip() or name in paths: + parser.error("Each --artifact must be a unique nonblank NAME=PATH.") + paths[name] = Path(path) + evidence = measure_regional_input_surface( + regional_manifest_uri=args.regional_manifest_uri, + artifact_paths=paths, + model_version=args.model_version, + audited_by=args.audited_by, + limitations=args.limitation, + token=os.environ.get("HUGGING_FACE_TOKEN") or os.environ.get("HF_TOKEN"), + ) + args.output.write_text(evidence.model_dump_json(indent=2) + "\n") + print( + f"Saved {len(evidence.artifacts)} artifact input audits to {args.output}: " + f"policyengine-uk {evidence.model_version} against build model " + f"{evidence.build_model_version} (core {evidence.build_core_version}); " + f"{len(evidence.dropped_columns)} dropped, " + f"{len(evidence.never_modelled_columns)} never-modelled, " + f"{len(evidence.defaulted_inputs)} defaulted." + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/bundle.py b/scripts/bundle.py index 53b74a26..9dd4040a 100644 --- a/scripts/bundle.py +++ b/scripts/bundle.py @@ -48,6 +48,14 @@ def _certify_data(args: argparse.Namespace) -> int: argv.extend(["--regional-release-kind", args.regional_release_kind]) if args.drop_regional_release: argv.append("--drop-regional-release") + if args.accept_regional_model_mismatch is not None: + argv.extend( + ["--accept-regional-model-mismatch", args.accept_regional_model_mismatch] + ) + if args.regional_input_surface_evidence: + argv.extend( + ["--regional-input-surface-evidence", args.regional_input_surface_evidence] + ) if args.no_generate: argv.append("--no-generate") if args.no_changelog: @@ -226,6 +234,19 @@ def _parser() -> argparse.ArgumentParser: "Cannot be combined with --regional-manifest-uri." ), ) + certify.add_argument( + "--accept-regional-model-mismatch", + metavar="REASON", + help=( + "UK only: route regions to a regional release on another model " + "version, recording this reviewed reason; forwarded to " + "scripts/certify_data_release.py." + ), + ) + certify.add_argument( + "--regional-input-surface-evidence", + help="Audited regional input-surface JSON; required for an accepted mismatch.", + ) certify.add_argument( "--regional-artifact-prefix", help="Legacy per-state artifact prefix to import. Defaults to states/.", diff --git a/scripts/certify_data_release.py b/scripts/certify_data_release.py index f3a720ee..60b0c9b6 100644 --- a/scripts/certify_data_release.py +++ b/scripts/certify_data_release.py @@ -24,6 +24,7 @@ from __future__ import annotations import argparse +import json import os import sys from pathlib import Path @@ -115,7 +116,26 @@ def main(argv=None) -> int: "qualification of this pair (requires --core-version)." ), ) + parser.add_argument( + "--accept-regional-model-mismatch", + metavar="REASON", + help=( + "UK only: route regions to a regional release that does not cover " + "the pinned model, recording this reviewed reason in the bundle " + "instead of refusing. Requires --regional-input-surface-evidence; " + "dropped columns (defined by the regional build model, not by the " + "pinned model) refuse. The core must still be covered." + ), + ) parser.add_argument("--no-generate", action="store_true") + parser.add_argument( + "--regional-input-surface-evidence", + type=Path, + help=( + "Audit JSON from scripts/audit_regional_input_surface.py, bound to " + "the regional bytes, its build model and the exact target model." + ), + ) parser.add_argument("--no-changelog", action="store_true") parser.add_argument( "--skip-artifact-check", @@ -149,6 +169,12 @@ def main(argv=None) -> int: drop_regional_release=args.drop_regional_release, core_version=args.core_version, qualified_pair_id=args.qualified_pair_id, + regional_model_mismatch_reason=args.accept_regional_model_mismatch, + regional_input_surface_evidence=( + json.loads(args.regional_input_surface_evidence.read_text()) + if args.regional_input_surface_evidence is not None + else None + ), ) print(result.summary()) diff --git a/src/policyengine/data/schemas/execution_receipt.v1.schema.json b/src/policyengine/data/schemas/execution_receipt.v1.schema.json index d10d1955..2bacf34b 100644 --- a/src/policyengine/data/schemas/execution_receipt.v1.schema.json +++ b/src/policyengine/data/schemas/execution_receipt.v1.schema.json @@ -457,6 +457,104 @@ "title": "ReformIdentity", "type": "object" }, + "RegionalCertificationIdentity": { + "additionalProperties": false, + "description": "The accepted regional model mismatch an area run used, bound by digest.\n\nIt summarises the bundle's ``regional_certification`` record. The record's\nevidence (both models' variable and input lists and the measured columns)\nstays in the bundle and its TRACE TRO; ``record_sha256`` binds it.", + "properties": { + "accepted_reason": { + "title": "Accepted Reason", + "type": "string" + }, + "area_receipts": { + "const": "uncertified", + "title": "Area Receipts", + "type": "string" + }, + "built_with_model_version": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "description": "The model the regional release was built with.", + "title": "Built With Model Version" + }, + "certified_for_model_version": { + "description": "The model the national certificate covers and area runs use.", + "title": "Certified For Model Version", + "type": "string" + }, + "compatibility_basis": { + "const": "accepted_model_mismatch", + "title": "Compatibility Basis", + "type": "string" + }, + "defaulted_inputs": { + "description": "Inputs area runs default that the regional file does not store and its build model did not take as inputs.", + "items": { + "type": "string" + }, + "title": "Defaulted Inputs", + "type": "array" + }, + "dropped_columns": { + "items": { + "type": "string" + }, + "title": "Dropped Columns", + "type": "array" + }, + "limitations": { + "items": { + "type": "string" + }, + "title": "Limitations", + "type": "array" + }, + "never_modelled_columns": { + "description": "Stored columns neither model defines.", + "items": { + "type": "string" + }, + "title": "Never Modelled Columns", + "type": "array" + }, + "record_sha256": { + "description": "RFC 8785 SHA-256 of the bundle manifest's data_releases..regional_certification record, its evidence included (see regional_certification_identity()).", + "pattern": "^[0-9a-f]{64}$", + "title": "Record Sha256", + "type": "string" + }, + "regional_manifest_sha256": { + "pattern": "^[0-9a-f]{64}$", + "title": "Regional Manifest Sha256", + "type": "string" + }, + "regional_manifest_uri": { + "title": "Regional Manifest Uri", + "type": "string" + } + }, + "required": [ + "compatibility_basis", + "certified_for_model_version", + "accepted_reason", + "area_receipts", + "regional_manifest_uri", + "regional_manifest_sha256", + "defaulted_inputs", + "dropped_columns", + "never_modelled_columns", + "limitations", + "record_sha256" + ], + "title": "RegionalCertificationIdentity", + "type": "object" + }, "RuntimeContext": { "additionalProperties": false, "description": "Where the execution ran. Part of no digest.", @@ -655,6 +753,18 @@ "reform": { "$ref": "#/$defs/ReformIdentity" }, + "regional_certification": { + "anyOf": [ + { + "$ref": "#/$defs/RegionalCertificationIdentity" + }, + { + "type": "null" + } + ], + "default": null, + "description": "Present only on a run of the dataset the bundle routes area regions to on an accepted model mismatch (an area run), which is uncertified: the exception's model versions, reason, audited lists and limitations, and the digest of its full record, evidence included. It does not widen the national certificate." + }, "runtime": { "anyOf": [ { @@ -711,10 +821,11 @@ "x-policyengine-digests": { "canonicalization": "RFC 8785 (JSON Canonicalization Scheme) over the stated value after removing object members whose value is null (array elements are kept). Numbers are IEEE 754 doubles; integers must lie within +/-(2**53 - 1).", "certificate.certificate_sha256": "the bundle manifest's data_releases. record", - "execution_identity_sha256": "{schema_version, country_id, wrapper, model, core, data, certification_status, certificate?} where certificate keeps only bundle_version, country_id, certificate_sha256, compatibility_basis, certified_for_model_version, certified_for_core_version, certified_dataset_sha256 and certified_by", + "execution_identity_sha256": "{schema_version, country_id, wrapper, model, core, data, certification_status, certificate?, regional_certification?} where certificate keeps only bundle_version, country_id, certificate_sha256, compatibility_basis, certified_for_model_version, certified_for_core_version, certified_dataset_sha256 and certified_by", "outputs_sha256": "the canonical output payload the receipt binds", "receipt_sha256": "the whole receipt without runtime and without reform.name and baseline.name", - "reform.reform_sha256": "RFC 8785 over {representation, payload} where payload is the reform exactly as given in that representation, nulls kept (a null in a reform can mean something, unlike an absent receipt field); for policyengine-uk.scenario_composition/v1 the payload is {applied_before_data_load, application: parameter_changes_then_modifiers, members: [{representation, reform_sha256, binding}, ...]} with members in application order; an infinite number in a payload is the object {\"$nonfinite\": \"Infinity\"} or {\"$nonfinite\": \"-Infinity\"}, and a payload may not use the key $nonfinite itself" + "reform.reform_sha256": "RFC 8785 over {representation, payload} where payload is the reform exactly as given in that representation, nulls kept (a null in a reform can mean something, unlike an absent receipt field); for policyengine-uk.scenario_composition/v1 the payload is {applied_before_data_load, application: parameter_changes_then_modifiers, members: [{representation, reform_sha256, binding}, ...]} with members in application order; an infinite number in a payload is the object {\"$nonfinite\": \"Infinity\"} or {\"$nonfinite\": \"-Infinity\"}, and a payload may not use the key $nonfinite itself", + "regional_certification.record_sha256": "the bundle manifest's data_releases..regional_certification record, its evidence included" }, "x-policyengine-rules": [ "years is sorted ascending with no duplicates, and each year is written as an integer (2027, not 2027.0).", @@ -728,6 +839,8 @@ "reform/baseline kind current_law has no representation or reform_sha256 and binding content; kind reform has both.", "wheel_sha256 and wheel_sha256_evidence are present together or not at all.", "certification_status certified requires each of wrapper, model and core to have install_origin absent or archive, or install_origin directory or vcs with wheel_sha256_evidence installed_record_verified; an editable install is never certified.", - "A consumer trusts a certificate only after checking it against the data_releases. record of the published policyengine manifest for certificate.bundle_version (verify_certificate()): certificate_sha256 is that record's digest, every other member of certificate except certificate_source, qualification_receipt_sha256 and tro_composition_fingerprint equals what the record states, and data equals the data identity the record determines. certificate_source is the producer's statement and proves nothing." + "A consumer trusts a certificate only after checking it against the data_releases. record of the published policyengine manifest for certificate.bundle_version (verify_certificate()): certificate_sha256 is that record's digest, every other member of certificate except certificate_source, qualification_receipt_sha256 and tro_composition_fingerprint equals what the record states, and data equals the data identity the record determines. certificate_source is the producer's statement and proves nothing.", + "regional_certification is present only when data is the artifact that the bundle's accepted regional model mismatch (data_releases..regional_certification) covers: an area run, which is uncertified. A certified receipt carries none.", + "A consumer checks regional_certification against the data_releases..regional_certification record of the manifest the run used (verify_regional_certification()): record_sha256 is that record's digest, every other member equals what the record states, and data is the regional artifact the record covers." ] } diff --git a/src/policyengine/provenance/__init__.py b/src/policyengine/provenance/__init__.py index deb2a8bc..b9eea8d3 100644 --- a/src/policyengine/provenance/__init__.py +++ b/src/policyengine/provenance/__init__.py @@ -108,6 +108,9 @@ from .receipt import ( verify_certificate as verify_certificate, ) +from .receipt import ( + verify_regional_certification as verify_regional_certification, +) from .trace import ( build_simulation_trace_tro as build_simulation_trace_tro, ) diff --git a/src/policyengine/provenance/certification.py b/src/policyengine/provenance/certification.py index 381c1e70..5ea7436c 100644 --- a/src/policyengine/provenance/certification.py +++ b/src/policyengine/provenance/certification.py @@ -47,6 +47,8 @@ from typing import Literal, Optional, cast, get_args import requests +from packaging.version import InvalidVersion, Version +from pydantic import ValidationError from policyengine.data.uk_geography_assets import ( CONSTITUENCY_ASSET_SPEC, @@ -57,6 +59,7 @@ HF_REQUEST_TIMEOUT_SECONDS, DataReleaseArtifact, DataReleaseManifest, + RegionalInputSurfaceEvidence, RegionalReleaseKind, _specifier_matches, fetch_pypi_wheel_metadata, @@ -413,6 +416,258 @@ def model_compatibility_basis( return basis, [] if warning is None else [warning] +#: A regional release's model coverage accepted by a reviewer, not by the +#: release's own build or claims (``--accept-regional-model-mismatch``). +REGIONAL_BASIS_ACCEPTED_MISMATCH = "accepted_model_mismatch" +REGIONAL_VAT_LIMITATION = ( + "the dense cut's VAT predates the policyengine-uk VAT factor fix " + "(#2187, 2.135.7); the dashboard holds a VAT limit" +) + + +def regional_defaults_limitation( + model_package: str, model_version: str, built_with: str, defaulted: list[str] +) -> str: + """The limitation certification records for inputs area runs default. + + Names every target input the regional file does not store that was not + an input of its build model, so a reader of the limitations alone knows + which area results rest on defaults. + """ + return ( + f"area runs on {model_package} {model_version} default {len(defaulted)} " + "inputs that the regional file does not store and its build model, " + f"{model_package} {built_with}, did not take as inputs, so area results " + f"for measures that read them use those defaults: {', '.join(defaulted)}" + ) + + +def _regional_build_model_version( + regional_manifest: DataReleaseManifest, model_package: str +) -> Optional[str]: + """Return the recorded regional build version; refuse a different package.""" + built = ( + regional_manifest.build.built_with_model_package + if regional_manifest.build is not None + else None + ) + if built is not None and built.name != model_package: + raise CertificationError( + f"Regional release was built with {built.name}, not {model_package}; " + "regional releases must use the target model package." + ) + return built.version if built is not None else None + + +def accepted_regional_mismatch( + regional_manifest: DataReleaseManifest, + model_package: str, + model_version: str, + reason: str, + *, + regional_manifest_uri: str, + regional_manifest_sha256: str, + evidence: Optional[RegionalInputSurfaceEvidence], +) -> tuple[dict, str]: + """Record a reviewed decision to route regions to a release on another model. + + Area runs on such a release are uncertified twice over: the certificate + covers the national dataset only, and the regional file was built and + calibrated under another model version. The record names both versions + and the reason so the bundle, its TRO and every consumer can say so. + + The evidence must bind the regional cut, its raw manifest bytes, its H5 + bytes, its recorded build model and core, and the exact target model. + Both models' surfaces in it are re-read from the models (the target must + be installed); the stored columns are the auditor's, bound to the H5 + bytes. Certification recomputes the comparison against the build model: + stored columns the build model defines and the target does not are + dropped and always refuse; stored columns neither model defines are + recorded as never modelled; and target inputs the file does not store + that were not build inputs are recorded as defaulted, and named in a + limitation. Returns ``(record, warning)``. + """ + + from policyengine.provenance.regional_input_surface import readable + + reason = reason.strip() + if not readable(reason): + raise CertificationError( + "--accept-regional-model-mismatch needs a reason a reviewer can read." + ) + built_with = _regional_build_model_version(regional_manifest, model_package) + if evidence is None: + raise CertificationError( + "An accepted regional model mismatch requires audited input-surface " + "evidence (--regional-input-surface-evidence)." + ) + try: + audit = RegionalInputSurfaceEvidence.model_validate(evidence) + except ValidationError as exc: + raise CertificationError( + f"Invalid regional input-surface evidence: {exc}" + ) from exc + if ( + audit.regional_manifest_uri != regional_manifest_uri + or audit.regional_manifest_sha256 != regional_manifest_sha256 + or audit.model_package != model_package + or audit.model_version != model_version + ): + raise CertificationError( + "Regional input-surface evidence must bind the exact regional cut, " + "raw manifest bytes and target model package/version." + ) + built_with_core, _ = _core_claims(regional_manifest) + if built_with is None or built_with_core is None: + raise CertificationError( + "The regional release records no build model and core versions, " + "so an accepted mismatch has no baseline to audit against." + ) + if (audit.build_model_version, audit.build_core_version) != ( + built_with, + built_with_core, + ): + raise CertificationError( + "Regional input-surface evidence must be measured against the " + f"regional release's build model, {model_package} {built_with} with " + f"policyengine-core {built_with_core}." + ) + expected_artifacts = { + name: artifact + for name, artifact in regional_manifest.artifacts.items() + if artifact.kind == "microdata" and Path(artifact.path).suffix.lower() == ".h5" + } + if not expected_artifacts or set(audit.artifacts) != set(expected_artifacts): + raise CertificationError( + "Evidence must measure every regional microdata artifact." + ) + for name, artifact in expected_artifacts.items(): + measured = audit.artifacts[name] + if any( + getattr(measured, key) != getattr(artifact, key) + for key in ("repo_id", "path", "revision", "sha256") + ) or ( + measured.repo_type is not None + and measured.repo_type + != ( + artifact.repo_type + or parse_manifest_uri(regional_manifest_uri)["repo_type"] + ) + ): + raise CertificationError( + f"Regional evidence artifact {name!r} does not match its identity/bytes." + ) + if not all(column.strip() for column in measured.columns): + raise CertificationError("Regional evidence has an empty column name.") + if measured.columns_by_period is not None: + if ( + not measured.columns_by_period + or any(not columns for columns in measured.columns_by_period.values()) + or set(measured.columns) + != { + column + for columns in measured.columns_by_period.values() + for column in columns + } + ): + raise CertificationError( + "Regional evidence has inconsistent period columns." + ) + if not set(audit.model_inputs).issubset(audit.model_variables): + raise CertificationError( + "Evidence model inputs must belong to its variable surface." + ) + if not readable(audit.audited_by) or not all( + readable(item) for item in audit.limitations + ): + raise CertificationError( + "Regional evidence needs an auditor and limitations a reader can see." + ) + from policyengine.provenance.regional_input_surface import ( + compare_input_surface, + verify_model_surfaces, + ) + + try: + # The lists are re-read from the models themselves, so an edited + # evidence file cannot hide a drop; only the stored columns rest on + # the auditor's word, bound to each H5's SHA-256. + verify_model_surfaces(audit, regional_manifest) + except ValueError as exc: + raise CertificationError( + "Certifying an accepted regional mismatch re-reads both models' " + f"surfaces: {exc}" + ) from exc + try: + comparison = compare_input_surface( + audit.artifacts, + audit.model_variables, + audit.model_inputs, + audit.input_aliases, + build_model_variables=audit.build_model_variables, + build_model_inputs=audit.build_model_inputs, + ) + except ValueError as exc: + raise CertificationError( + f"Invalid regional input-surface evidence: {exc}" + ) from exc + if ( + audit.dropped_columns != comparison["dropped_columns"] + or audit.never_modelled_columns != comparison["never_modelled_columns"] + or audit.defaulted_inputs != comparison["defaulted_inputs"] + ): + raise CertificationError( + "Regional evidence comparison disagrees with its measured surfaces." + ) + if comparison["dropped_columns"]: + raise CertificationError( + f"Dropped regional input columns are refused ({model_package} " + f"{built_with} defines them and {model_version} does not): " + + ", ".join(comparison["dropped_columns"]) + ) + limitations = list(audit.limitations) + try: + predates_vat_fix = ( + Version(built_with) < Version("2.135.7") <= Version(model_version) + ) + except InvalidVersion as exc: + raise CertificationError( + "Regional model versions must be valid versions." + ) from exc + if predates_vat_fix and REGIONAL_VAT_LIMITATION not in limitations: + limitations.append(REGIONAL_VAT_LIMITATION) + if comparison["defaulted_inputs"]: + defaults = regional_defaults_limitation( + model_package, model_version, built_with, comparison["defaulted_inputs"] + ) + if defaults not in limitations: + limitations.append(defaults) + record = { + "compatibility_basis": REGIONAL_BASIS_ACCEPTED_MISMATCH, + "certified_for_model_version": model_version, + "built_with_model_version": built_with, + "accepted_reason": reason, + "area_receipts": "uncertified", + "regional_manifest_uri": regional_manifest_uri, + "regional_manifest_sha256": regional_manifest_sha256, + "defaulted_inputs": comparison["defaulted_inputs"], + "dropped_columns": comparison["dropped_columns"], + "never_modelled_columns": comparison["never_modelled_columns"], + "limitations": limitations, + "evidence": audit.model_dump(mode="json", exclude_none=True), + } + warning = ( + f"regional release: built with {model_package} {built_with} and routed " + f"under {model_version} on an accepted mismatch ({reason}); area runs " + "are uncertified; audited dropped columns: none; never-modelled " + "columns (stored, defined by neither model): " + f"{len(comparison['never_modelled_columns'])}; inputs defaulted since " + f"the build model: {len(comparison['defaulted_inputs'])}. Limitations: " + + "; ".join(limitations) + ) + return record, warning + + def _check_core_compatibility( manifest: DataReleaseManifest, core_version: str, @@ -1097,6 +1352,8 @@ def build_bundle_data_release_payload( regional_manifest_uri: Optional[str] = None, regional_uri_parts: Optional[dict] = None, regional_release_kind: Optional[RegionalReleaseKind] = None, + regional_certification: Optional[dict] = None, + regional_manifest_sha256: Optional[str] = None, ) -> dict: """Map the country certification payload into bundle ``data_releases``.""" payload = copy.deepcopy(country_payload) @@ -1110,12 +1367,16 @@ def build_bundle_data_release_payload( payload["release_manifest_uri"] = https_manifest_url(uri_parts) if regional_manifest_uri is not None: payload["regional_source_manifest_uri"] = regional_manifest_uri + if regional_manifest_sha256 is not None: + payload["regional_source_manifest_sha256"] = regional_manifest_sha256 if regional_uri_parts is not None: payload["regional_release_manifest_uri"] = https_manifest_url( regional_uri_parts ) if regional_release_kind is not None: payload["regional_release_kind"] = regional_release_kind + if regional_certification is not None: + payload["regional_certification"] = regional_certification payload["default_dataset_uri"] = certified_artifact.get("uri") return payload @@ -1139,6 +1400,8 @@ def certify( core_version: Optional[str] = None, qualified_pair_id: Optional[str] = None, bundle_core_version: Optional[str] = None, + regional_model_mismatch_reason: Optional[str] = None, + regional_input_surface_evidence: Optional[RegionalInputSurfaceEvidence] = None, ) -> tuple[dict, str, list[str]]: raise NotImplementedError @@ -1162,6 +1425,8 @@ def certify( core_version: Optional[str] = None, qualified_pair_id: Optional[str] = None, bundle_core_version: Optional[str] = None, + regional_model_mismatch_reason: Optional[str] = None, + regional_input_surface_evidence: Optional[RegionalInputSurfaceEvidence] = None, ) -> tuple[dict, str, list[str]]: raise CertificationError( "Legacy data-producer certification updates are not implemented in " @@ -1188,20 +1453,36 @@ def certify( core_version: Optional[str] = None, qualified_pair_id: Optional[str] = None, bundle_core_version: Optional[str] = None, + regional_model_mismatch_reason: Optional[str] = None, + regional_input_surface_evidence: Optional[RegionalInputSurfaceEvidence] = None, ) -> tuple[dict, str, list[str]]: manifest, manifest_sha256, uri_parts = fetch_release_manifest( manifest_uri, token=token ) primary_manifest = manifest regional_manifest = None + regional_manifest_sha256 = None regional_uri_parts = None recorded_release_kind = None regional_artifacts: dict[str, DataReleaseArtifact] = {} regional_warnings: list[str] = [] + regional_certification: Optional[dict] = None + mismatch_warning: Optional[str] = None + if regional_model_mismatch_reason is not None and regional_manifest_uri is None: + raise CertificationError( + "--accept-regional-model-mismatch applies to a regional release; " + "pass --regional-manifest-uri." + ) + if regional_model_mismatch_reason is not None and country != "uk": + raise CertificationError( + "--accept-regional-model-mismatch is only supported for the UK, " + "whose area receipts are uncertified." + ) if regional_manifest_uri is not None: - regional_manifest, _, regional_uri_parts = fetch_release_manifest( - regional_manifest_uri, token=token + regional_manifest, regional_manifest_sha256, regional_uri_parts = ( + fetch_release_manifest(regional_manifest_uri, token=token) ) + _regional_build_model_version(regional_manifest, model_package) # The regional release runs on the same model and core, so it must # cover the pinned versions by its own build or claims. A qualified # pair names the primary release only, so it does not extend here. @@ -1209,6 +1490,30 @@ def certify( _, regional_warnings = model_compatibility_basis( regional_manifest, model_package, model_version ) + except CertificationError as exc: + if regional_model_mismatch_reason is None: + raise CertificationError( + f"Regional release {regional_manifest_uri}: {exc}" + ) from exc + regional_certification, mismatch_warning = accepted_regional_mismatch( + regional_manifest, + model_package, + model_version, + regional_model_mismatch_reason, + regional_manifest_uri=regional_manifest_uri, + regional_manifest_sha256=regional_manifest_sha256, + evidence=regional_input_surface_evidence, + ) + regional_warnings = [] + else: + if regional_model_mismatch_reason is not None: + raise CertificationError( + f"Regional release {regional_manifest_uri} serves " + f"{model_package} {model_version} by its own build or " + "claims; drop --accept-regional-model-mismatch so the " + "certificate does not record a mismatch that is not there." + ) + try: if core_version is not None: _check_core_compatibility( regional_manifest, core_version, None, regional_warnings @@ -1220,6 +1525,8 @@ def certify( regional_warnings = [ f"regional release: {warning}" for warning in regional_warnings ] + if mismatch_warning is not None: + regional_warnings.append(mismatch_warning) primary_artifact_names = set(manifest.artifacts) manifest, recorded_release_kind = merge_regional_release_manifest( country, @@ -1313,6 +1620,8 @@ def certify( regional_manifest_uri=regional_manifest_uri, regional_uri_parts=regional_uri_parts, regional_release_kind=recorded_release_kind, + regional_certification=regional_certification, + regional_manifest_sha256=regional_manifest_sha256, ), manifest_sha256, warnings, @@ -1358,6 +1667,8 @@ def certify_data_release( drop_regional_release: bool = False, core_version: Optional[str] = None, qualified_pair_id: Optional[str] = None, + regional_model_mismatch_reason: Optional[str] = None, + regional_input_surface_evidence: Optional[RegionalInputSurfaceEvidence] = None, ) -> CertificationResult: """Certify a data release into the bundle manifest. @@ -1373,8 +1684,24 @@ def certify_data_release( given, by its own build or claims: a qualified pair names the primary release only. While the bundle routes regions, a run without ``regional_manifest_uri`` is refused unless ``drop_regional_release``. + + ``regional_model_mismatch_reason`` (UK only) requires + ``regional_input_surface_evidence`` (``scripts/audit_regional_input_surface.py``) + bound to the regional manifest and artifact bytes, the regional build + model and core, and the exact target model. Dropped columns, which the + build model defines and the target does not, are always refused. Every + never-modelled column, defaulted input and limitation is recorded in + ``regional_certification`` alongside the reason and both model versions. + The core must still be covered; area receipts remain uncertified. """ certified_country = _certified_country(country) + if ( + regional_input_surface_evidence is not None + and regional_model_mismatch_reason is None + ): + raise CertificationError( + "Regional input-surface evidence requires an accepted mismatch reason." + ) if drop_regional_release and regional_manifest_uri is not None: raise CertificationError( "--drop-regional-release removes the bundle's regional routing, so it " @@ -1419,6 +1746,8 @@ def certify_data_release( bundle_core_version=( (bundle.get("packages") or {}).get("policyengine-core") or {} ).get("version"), + regional_model_mismatch_reason=regional_model_mismatch_reason, + regional_input_surface_evidence=regional_input_surface_evidence, ) previous_release = (bundle.get("data_releases") or {}).get(country) or {} bundle.setdefault("data_releases", {})[country] = data_release diff --git a/src/policyengine/provenance/manifest.py b/src/policyengine/provenance/manifest.py index 7729313e..bf4345ee 100644 --- a/src/policyengine/provenance/manifest.py +++ b/src/policyengine/provenance/manifest.py @@ -211,6 +211,81 @@ class CertifiedDataArtifact(BaseModel): RegionalReleaseKind = Literal["local_area", "area_coded_national"] +class RegionalInputArtifactEvidence(BaseModel): + """Measured columns of one byte-verified regional H5 artifact.""" + + model_config = {"extra": "forbid"} + + repo_id: str = Field(min_length=1) + path: str = Field(min_length=1) + revision: str = Field(min_length=1) + sha256: str = Field(pattern=r"^[0-9a-f]{64}$") + repo_type: Optional[Literal["model", "dataset"]] = None + columns: list[str] = Field(min_length=1) + columns_by_period: dict[str, list[str]] = Field(min_length=1) + + +class RegionalInputSurfaceEvidence(BaseModel): + """An operator's audit of actual regional columns against two exact models. + + The baseline is the regional release's build model and core, as its + manifest records them; the target is the model the bundle certifies. Both + surfaces are retained in full so certification recomputes the comparison + rather than trusting a declared zero-loss result. The artifact hashes + identify the bytes inspected by the auditor, not merely a model's possible + inputs. Schema version 2 made the comparison relative to the build model; + version 1 evidence, which counted columns no model defines as dropped, is + refused. + """ + + model_config = {"extra": "forbid"} + + schema_version: Literal[2] + regional_manifest_uri: str = Field(min_length=1) + regional_manifest_sha256: str = Field(pattern=r"^[0-9a-f]{64}$") + model_package: str = Field(min_length=1) + model_version: str = Field(min_length=1) + build_model_version: str = Field(min_length=1) + build_core_version: str = Field(min_length=1) + artifacts: dict[str, RegionalInputArtifactEvidence] = Field(min_length=1) + model_variables: list[str] = Field(min_length=1) + model_inputs: list[str] = Field(min_length=1) + build_model_variables: list[str] = Field(min_length=1) + build_model_inputs: list[str] = Field(min_length=1) + input_aliases: Optional[dict[str, str]] = None + dropped_columns: list[str] + """Stored columns the build model defines and the target model does not.""" + never_modelled_columns: list[str] + """Stored columns neither model defines; no model loads them.""" + defaulted_inputs: list[str] + """Target inputs the file does not store, after the target's input + aliases, that were not inputs of the build model.""" + audited_by: str = Field(min_length=1, pattern=r"\S") + limitations: list[str] = Field(min_length=1) + + +class RegionalCertification(BaseModel): + """A reviewer's acceptance of a regional release on another model version. + + Written only by ``certify-data --accept-regional-model-mismatch``: the + regional release neither was built with nor claims the certified model, + and area runs on it are uncertified. + """ + + compatibility_basis: Literal["accepted_model_mismatch"] + certified_for_model_version: str + built_with_model_version: Optional[str] = None + accepted_reason: str = Field(min_length=1, pattern=r"\S") + area_receipts: Literal["uncertified"] = "uncertified" + regional_manifest_uri: str + regional_manifest_sha256: str = Field(pattern=r"^[0-9a-f]{64}$") + defaulted_inputs: list[str] + dropped_columns: list[str] + never_modelled_columns: list[str] + limitations: list[str] + evidence: RegionalInputSurfaceEvidence + + class CountryReleaseManifest(BaseModel): schema_version: int = 1 bundle_id: Optional[str] = None @@ -227,6 +302,11 @@ class CountryReleaseManifest(BaseModel): ``local_area`` for a non-default local-area release, and ``area_coded_national`` for another release's national dataset that carries area codes (and its area weight matrices).""" + regional_certification: Optional[RegionalCertification] = None + """Present only when the regional release was routed on an accepted + model mismatch rather than its own build or claims.""" + regional_source_manifest_sha256: Optional[str] = None + """SHA-256 of the original regional release manifest bytes.""" certified_data_artifact: Optional[CertifiedDataArtifact] = None certification: Optional[DataCertification] = None source_sha256: Optional[str] = Field(default=None, exclude=True) diff --git a/src/policyengine/provenance/receipt.py b/src/policyengine/provenance/receipt.py index 1c7b43e0..e8192c58 100644 --- a/src/policyengine/provenance/receipt.py +++ b/src/policyengine/provenance/receipt.py @@ -56,6 +56,10 @@ - A certificate is trusted only after :func:`verify_certificate` has checked it, and the receipt's data identity, against the published bundle manifest it names. ``certificate_source`` is the producer's statement, not evidence. +- An area run on a regional release the bundle routes on an accepted model + mismatch is uncertified and carries a summary of that exception with the + digest of its full record; :func:`verify_regional_certification` checks it + against the manifest the run used. Adding an optional field is a compatible v1 change. Removing or redefining a field, or changing what a digest covers, requires schema v2. @@ -133,6 +137,7 @@ drop_null_members, ) from policyengine.provenance.loaded import loaded_versions +from policyengine.provenance.manifest import RegionalCertification from policyengine.utils.hashing import sha256_file __all__ = [ @@ -146,6 +151,7 @@ "PackageIdentity", "ReceiptError", "ReformIdentity", + "RegionalCertificationIdentity", "RuntimeContext", "WrapperIdentity", "build_execution_receipt", @@ -156,10 +162,12 @@ "fiscal_year_label", "installed_package_identity", "reform_identity", + "regional_certification_identity", "scenario_reform_identity", "strict_json_value", "uk_fiscal_year_convention", "verify_certificate", + "verify_regional_certification", ] RECEIPT_SCHEMA_VERSION: Literal[1] = 1 @@ -218,14 +226,17 @@ "wheel_sha256 and wheel_sha256_evidence are present together or not at all.", "certification_status certified requires each of wrapper, model and core to have install_origin absent or archive, or install_origin directory or vcs with wheel_sha256_evidence installed_record_verified; an editable install is never certified.", "A consumer trusts a certificate only after checking it against the data_releases. record of the published policyengine manifest for certificate.bundle_version (verify_certificate()): certificate_sha256 is that record's digest, every other member of certificate except certificate_source, qualification_receipt_sha256 and tro_composition_fingerprint equals what the record states, and data equals the data identity the record determines. certificate_source is the producer's statement and proves nothing.", + "regional_certification is present only when data is the artifact that the bundle's accepted regional model mismatch (data_releases..regional_certification) covers: an area run, which is uncertified. A certified receipt carries none.", + "A consumer checks regional_certification against the data_releases..regional_certification record of the manifest the run used (verify_regional_certification()): record_sha256 is that record's digest, every other member equals what the record states, and data is the regional artifact the record covers.", ) #: How each digest is computed. DIGEST_RULES: dict[str, str] = { "canonicalization": "RFC 8785 (JSON Canonicalization Scheme) over the stated value after removing object members whose value is null (array elements are kept). Numbers are IEEE 754 doubles; integers must lie within +/-(2**53 - 1).", - "execution_identity_sha256": "{schema_version, country_id, wrapper, model, core, data, certification_status, certificate?} where certificate keeps only bundle_version, country_id, certificate_sha256, compatibility_basis, certified_for_model_version, certified_for_core_version, certified_dataset_sha256 and certified_by", + "execution_identity_sha256": "{schema_version, country_id, wrapper, model, core, data, certification_status, certificate?, regional_certification?} where certificate keeps only bundle_version, country_id, certificate_sha256, compatibility_basis, certified_for_model_version, certified_for_core_version, certified_dataset_sha256 and certified_by", "receipt_sha256": "the whole receipt without runtime and without reform.name and baseline.name", "certificate.certificate_sha256": "the bundle manifest's data_releases. record", + "regional_certification.record_sha256": "the bundle manifest's data_releases..regional_certification record, its evidence included", "reform.reform_sha256": 'RFC 8785 over {representation, payload} where payload is the reform exactly as given in that representation, nulls kept (a null in a reform can mean something, unlike an absent receipt field); for policyengine-uk.scenario_composition/v1 the payload is {applied_before_data_load, application: parameter_changes_then_modifiers, members: [{representation, reform_sha256, binding}, ...]} with members in application order; an infinite number in a payload is the object {"$nonfinite": "Infinity"} or {"$nonfinite": "-Infinity"}, and a payload may not use the key $nonfinite itself', "outputs_sha256": "the canonical output payload the receipt binds", } @@ -398,6 +409,46 @@ class CertificateIdentity(_Contract): ) +class RegionalCertificationIdentity(_Contract): + """The accepted regional model mismatch an area run used, bound by digest. + + It summarises the bundle's ``regional_certification`` record. The record's + evidence (both models' variable and input lists and the measured columns) + stays in the bundle and its TRACE TRO; ``record_sha256`` binds it. + """ + + compatibility_basis: Literal["accepted_model_mismatch"] + certified_for_model_version: str = Field( + description="The model the national certificate covers and area runs use." + ) + built_with_model_version: Optional[str] = Field( + default=None, description="The model the regional release was built with." + ) + accepted_reason: str + area_receipts: Literal["uncertified"] + regional_manifest_uri: str + regional_manifest_sha256: str = Field(pattern=SHA256_PATTERN) + defaulted_inputs: list[str] = Field( + description=( + "Inputs area runs default that the regional file does not store " + "and its build model did not take as inputs." + ) + ) + dropped_columns: list[str] + never_modelled_columns: list[str] = Field( + description="Stored columns neither model defines." + ) + limitations: list[str] + record_sha256: str = Field( + pattern=SHA256_PATTERN, + description=( + "RFC 8785 SHA-256 of the bundle manifest's " + "data_releases..regional_certification record, its evidence " + "included (see regional_certification_identity())." + ), + ) + + class ReformIdentity(_Contract): """A reform or baseline, bound by the digest of its canonical form.""" @@ -529,6 +580,16 @@ class ExecutionReceipt(_Contract): data: DataIdentity certification_status: Literal["certified", "uncertified"] certificate: Optional[CertificateIdentity] = None + regional_certification: Optional[RegionalCertificationIdentity] = Field( + default=None, + description=( + "Present only on a run of the dataset the bundle routes area regions " + "to on an accepted model mismatch (an area run), which is " + "uncertified: the exception's model versions, reason, audited lists " + "and limitations, and the digest of its full record, evidence " + "included. It does not widen the national certificate." + ), + ) reform: ReformIdentity baseline: ReformIdentity fiscal_year: FiscalYearConvention @@ -549,6 +610,13 @@ class ExecutionReceipt(_Contract): ) runtime: Optional[RuntimeContext] = None + @model_serializer(mode="wrap") + def _omit_absent_regional_certification(self, handler): + payload = handler(self) + if self.regional_certification is None: + payload.pop("regional_certification", None) + return payload + @field_validator("schema_version", mode="before") @classmethod def _schema_version_is_an_integer(cls, value: Any) -> Any: @@ -576,6 +644,11 @@ def _identity_rules(self) -> ExecutionReceipt: if self.certificate is not None: raise ValueError("An uncertified receipt carries no certificate.") return self + if self.regional_certification is not None: + raise ValueError( + "A certified receipt carries no regional_certification: the " + "accepted regional exception covers uncertified area runs only." + ) certificate = self.certificate if certificate is None: raise ValueError("A certified receipt needs a certificate.") @@ -627,6 +700,10 @@ def execution_identity(self) -> dict[str, Any]: for key in CertificateIdentity.IDENTITY_FIELDS if key in payload } + if self.regional_certification is not None: + identity["regional_certification"] = ( + self.regional_certification.canonical_payload() + ) return identity @property @@ -841,6 +918,24 @@ def certificate_sha256(certification_record: Mapping[str, Any]) -> str: return content_sha256(certification_record) +def regional_certification_identity( + regional_record: Mapping[str, Any], +) -> RegionalCertificationIdentity: + """What an area receipt carries of a bundle's ``regional_certification``. + + Every member of the record except its evidence, and the digest of the + whole record, evidence included. + """ + + summary = RegionalCertification.model_validate(regional_record).model_dump( + mode="json", exclude_none=True + ) + del summary["evidence"] + return RegionalCertificationIdentity( + **summary, record_sha256=content_sha256(regional_record) + ) + + def reform_identity( payload: Optional[Union[Mapping[str, Any], Sequence[Any]]], representation: Optional[ReformRepresentation] = None, @@ -3771,7 +3866,23 @@ def _data_identity( dataset_name: str, dataset_sha256: str, ) -> DataIdentity: - """The data identity of ``dataset_name`` with those bytes, from the bundle. + """The data identity of ``dataset_name`` with those bytes, from the bundle.""" + + identity, _ = _resolved_data( + country_id, record, bundle, dataset_name, dataset_sha256 + ) + return identity + + +def _resolved_data( + country_id: str, + record: Mapping[str, Any], + bundle: Mapping[str, Any], + dataset_name: str, + dataset_sha256: str, +) -> tuple[DataIdentity, bool]: + """The data identity of ``dataset_name`` with those bytes, from the bundle, + and whether they are the artifact the accepted regional exception covers. The dataset is resolved exactly as :func:`policyengine.provenance.dataset_materialization.materialize_dataset` @@ -3811,17 +3922,30 @@ def _data_identity( datasets = record.get("datasets") or {} entry = datasets.get(dataset_name) primary = datasets.get(record.get("default_dataset")) or {} - # A regional release's artifacts are certified beside the primary release - # but are not part of it, so only entries at the primary release's own - # revision take its build and manifest identity. - release = entry is not None and entry.get("revision") in ( - None, - primary.get("revision"), + # A regional release's artifacts keep their own build provenance, even + # when acceptance allows them to run beside another national model. + regional_certification = manifest.regional_certification + regional = regional_certification is not None and any( + artifact.repo_id == spec.repo_id + and artifact.path == spec.path + and artifact.revision == spec.revision + and artifact.sha256 == spec.sha256 + and artifact.repo_type in (None, spec.repo_type) + for artifact in regional_certification.evidence.artifacts.values() + ) + release = ( + not regional + and entry is not None + and entry.get("revision") + in ( + None, + primary.get("revision"), + ) ) data_package = record["data_package"] certified_artifact = record.get("certified_data_artifact") or {} certification = record.get("certification") or {} - return DataIdentity( + identity = DataIdentity( data_package=spec.data_package_name, release_version=data_package["version"] if release else spec.revision, build_id=( @@ -3832,9 +3956,13 @@ def _data_identity( build_fingerprint=certification.get("data_build_fingerprint") if release else None, - build_model_version=certification.get("built_with_model_version") - if release - else None, + build_model_version=( + regional_certification.built_with_model_version + if regional and regional_certification is not None + else certification.get("built_with_model_version") + if release + else None + ), release_manifest_revision=data_package.get("release_manifest_revision") if release else None, @@ -3843,6 +3971,7 @@ def _data_identity( dataset_revision=spec.revision, dataset_sha256=dataset_sha256, ) + return identity, regional def _build( @@ -3892,7 +4021,9 @@ def identity(name: str) -> dict[str, Any]: dataset_name = dataset or record["default_dataset"] actual_sha256 = sha256_file(dataset_path) - data = _data_identity(country_id, record, bundle, dataset_name, actual_sha256) + data, regional = _resolved_data( + country_id, record, bundle, dataset_name, actual_sha256 + ) runtime_basis = None if source == "packaged" and ( @@ -3946,6 +4077,13 @@ def identity(name: str) -> dict[str, Any]: data=data, certification_status=status, certificate=certificate, + # National receipts need no copy: certificate_sha256 digests the + # record that holds the regional exception. + regional_certification=( + regional_certification_identity(record["regional_certification"]) + if regional + else None + ), reform=reform, baseline=baseline, fiscal_year=uk_fiscal_year_convention(), @@ -4040,6 +4178,66 @@ def verify_certificate( return problems +def verify_regional_certification( + receipt: ExecutionReceipt, bundle: Mapping[str, Any] +) -> list[str]: + """Problems with an area receipt's regional exception against a bundle. + + ``bundle`` is the policyengine manifest the run used. The summary is + rebuilt from that manifest's ``data_releases..regional_certification`` + record exactly as :func:`build_execution_receipt` builds it and compared + member by member, ``record_sha256`` included, and the receipt's ``data`` + must be the regional artifact that record covers, with the identity the + bundle determines. An empty list means the receipt names exactly the + bundle's accepted regional exception; the area run stays uncertified. + """ + + claimed = receipt.regional_certification + if claimed is None: + return ["the receipt carries no regional_certification"] + record = (bundle.get("data_releases") or {}).get(receipt.country_id) + regional_record = ( + record.get("regional_certification") if isinstance(record, Mapping) else None + ) + if not isinstance(record, Mapping) or not isinstance(regional_record, Mapping): + return [ + f"the bundle's {receipt.country_id} data release has no " + "regional_certification" + ] + try: + expected = regional_certification_identity(regional_record) + data, regional = _resolved_data( + receipt.country_id, + record, + bundle, + receipt.data.dataset, + receipt.data.dataset_sha256, + ) + except (KeyError, ValueError) as exc: + return [f"the bundle cannot determine regional_certification: {exc}"] + problems = [] + claimed_payload = claimed.canonical_payload() + expected_payload = expected.canonical_payload() + for field in sorted(set(claimed_payload) | set(expected_payload)): + if claimed_payload.get(field) != expected_payload.get(field): + problems.append( + f"regional_certification.{field} differs from the bundle's record" + ) + if not regional: + problems.append( + f"data.dataset {receipt.data.dataset!r} is not the artifact the " + "bundle's regional exception covers" + ) + claimed_data = receipt.data.model_dump() + for field, value in data.model_dump().items(): + if claimed_data[field] != value: + problems.append( + f"data.{field} is {claimed_data[field]!r}, " + f"but the bundle determines {value!r}" + ) + return problems + + def build_execution_receipt_for_microsimulation( microsimulation: Any, *, diff --git a/src/policyengine/provenance/regional_input_surface.py b/src/policyengine/provenance/regional_input_surface.py new file mode 100644 index 00000000..676941cb --- /dev/null +++ b/src/policyengine/provenance/regional_input_surface.py @@ -0,0 +1,664 @@ +"""Audit regional H5 bytes against their build model and the exact UK target. + +A local-area release keeps serving area runs after the national bundle moves +to a newer policyengine-uk. The audit compares the columns actually stored in +every artifact and period with two models: the release's build model, which +its manifest records (``build.built_with_model_package`` and +``build.built_with_core_package``), and the installed target model. As in +Microcosm PR #1348's handoff comparison, the build model is the baseline: + +- **dropped**: stored columns the build model defines and the target model + does not. The target's loaders skip them, so they refuse acceptance. +- **never modelled**: stored columns neither model defines, such as + Microcosm's area codes, build and source ids, clone indices and SPI donor + fields. No model ever loaded them, so they are recorded as information. +- **defaulted**: target inputs a period does not store, after the target's + input aliases, that were not inputs of the build model. Area runs default + them where the build model did not. + +Columns neither model defines, and inputs absent under both, are not changes. +The build model is read in a throwaway ``uv run`` environment, from PyPI as it +was when the release was built, so the audit needs ``uv`` and PyPI. Input +moves are read from the installed target model's constructor, so an alias +cannot make a supplied input look defaulted. Certification reads both models +again (``verify_model_surfaces``); only the stored columns rest on the +auditor's word, bound to each H5's SHA-256. No household values are included +in the resulting audit. + +policyengine-uk 2.147.2 silently skips unknown columns in both dataset loaders; +the audit must detect that loss before certification. See the ``continue`` +branches in ``build_from_dataset`` and ``build_from_multi_year_dataset``: +https://github.com/PolicyEngine/policyengine-uk/blob/fe0b99a36bd309a722b19937345e34290cb6dd73/policyengine_uk/simulation.py#L1113-L1127 +https://github.com/PolicyEngine/policyengine-uk/blob/fe0b99a36bd309a722b19937345e34290cb6dd73/policyengine_uk/simulation.py#L1186-L1196 +""" + +from __future__ import annotations + +import ast +import hashlib +import inspect +import json +import os +import re +import subprocess +import textwrap +from collections.abc import Callable +from datetime import UTC, datetime, timedelta +from importlib.metadata import PackageNotFoundError, version +from pathlib import Path +from typing import Any + +from packaging.version import InvalidVersion, Version + +from policyengine.provenance.manifest import ( + DataReleaseManifest, + RegionalInputArtifactEvidence, + RegionalInputSurfaceEvidence, +) + +UK_ENTITIES = ("person", "benunit", "household") +#: The interpreter of the throwaway environment that reads the build model. +BUILD_MODEL_PYTHON = "3.13" +#: The only index that environment resolves from. +BUILD_MODEL_INDEX = "https://pypi.org/simple" +#: uv settings that environment keeps: where uv caches and how it connects, +#: none of which changes what resolves. Every other ``UV_*`` setting (indexes, +#: constraints, overrides, the caller's project) and ``VIRTUAL_ENV`` are +#: removed. +BUILD_MODEL_KEPT_UV_SETTINGS = frozenset( + { + "UV_CACHE_DIR", + "UV_CONCURRENT_DOWNLOADS", + "UV_CONCURRENT_INSTALLS", + "UV_HTTP_TIMEOUT", + "UV_LINK_MODE", + "UV_NATIVE_TLS", + "UV_NO_PROGRESS", + "UV_PYTHON_CACHE_DIR", + "UV_PYTHON_INSTALL_DIR", + } +) +#: Runs in the build model's environment: the model API ``_model_surface`` +#: uses, and the package versions it actually imported. +BUILD_MODEL_PROBE = """ +import json +from importlib.metadata import version + +import policyengine_uk + +system = policyengine_uk.CountryTaxBenefitSystem() +print(json.dumps({ + "model_version": version("policyengine-uk"), + "core_version": version("policyengine-core"), + "variables": sorted(system.variables), + "inputs": sorted( + name + for name, variable in system.variables.items() + if variable.is_input_variable() + ), +})) +""" + + +class RegionalInputSurfaceError(ValueError): + """A complete, byte-bound regional surface cannot be established.""" + + +def readable(text: str) -> bool: + """Whether ``text`` shows a reader anything. + + ``str.strip`` and a ``\\S`` pattern both accept text made only of + invisible format characters, such as a zero-width space, so this needs + one printable character that is not whitespace. + """ + return any(char.isprintable() and not char.isspace() for char in text) + + +def _names(values: list[str], label: str) -> set[str]: + if not values or any( + not isinstance(value, str) or not value.strip() or value != value.strip() + for value in values + ): + raise RegionalInputSurfaceError(f"{label} must contain nonblank names.") + names = set(values) + if len(names) != len(values): + raise RegionalInputSurfaceError(f"{label} contains duplicate names.") + return names + + +def compare_input_surface( + artifacts: dict[str, RegionalInputArtifactEvidence], + model_variables: list[str], + model_inputs: list[str], + input_aliases: dict[str, str] | None = None, + *, + build_model_variables: list[str], + build_model_inputs: list[str], +) -> dict[str, list[str]]: + """Recompute dropped, never-modelled and defaulted names per period. + + ``dropped_columns`` are stored columns the build model defines and the + target model does not. ``never_modelled_columns`` are stored columns + neither model defines. ``defaulted_inputs`` are target inputs a period + does not supply that were not inputs of the build model. Inputs supplied + by the target's unconditional ``move_values`` calls use the destination + name after the move. Every artifact and period must pass: the union of + supplied columns cannot hide a missing input in another period. + """ + variables = _names(model_variables, "model_variables") + inputs = _names(model_inputs, "model_inputs") + if not inputs <= variables: + raise RegionalInputSurfaceError("model_inputs must be model_variables.") + build_variables = _names(build_model_variables, "build_model_variables") + build_inputs = _names(build_model_inputs, "build_model_inputs") + if not build_inputs <= build_variables: + raise RegionalInputSurfaceError( + "build_model_inputs must be build_model_variables." + ) + aliases = input_aliases or {} + for donor, target in aliases.items(): + if ( + donor not in variables + or target not in inputs + or donor == target + or target in aliases + ): + raise RegionalInputSurfaceError( + f"Unsupported input alias {donor!r} -> {target!r}." + ) + if not artifacts: + raise RegionalInputSurfaceError("No regional artifact columns measured.") + dropped: set[str] = set() + never_modelled: set[str] = set() + defaulted: set[str] = set() + for name, artifact in artifacts.items(): + columns = _names(artifact.columns, f"{name} columns") + if not artifact.columns_by_period: + raise RegionalInputSurfaceError(f"{name} has no measured periods.") + periods = [] + for period, period_columns in artifact.columns_by_period.items(): + if not re.fullmatch(r"[0-9]{4}", period): + raise RegionalInputSurfaceError( + f"{name} has an invalid period {period!r}." + ) + periods.append(_names(period_columns, f"{name}/{period} columns")) + if set().union(*periods) != columns: + raise RegionalInputSurfaceError( + f"{name} columns must equal the union of columns_by_period." + ) + for supplied in periods: + dropped.update((supplied & build_variables) - variables) + never_modelled.update(supplied - build_variables - variables) + effective = set(supplied) + for donor, target in aliases.items(): + if donor in effective: + effective.remove(donor) + effective.add(target) + defaulted.update((inputs - effective) - build_inputs) + return { + "dropped_columns": sorted(dropped), + "never_modelled_columns": sorted(never_modelled), + "defaulted_inputs": sorted(defaulted), + } + + +def _input_aliases(simulation_class: Any) -> dict[str, str]: + """Read unconditional literal moves; refuse unsupported dynamic semantics. + + policyengine-uk has no public alias inventory. Its public Simulation + constructor declares the input moves. Reading that source avoids running + a simulation or encoding one version's aliases as a permanent allowlist. + """ + try: + tree = ast.parse(textwrap.dedent(inspect.getsource(simulation_class.__init__))) + except (OSError, TypeError, SyntaxError) as exc: + raise RegionalInputSurfaceError( + "Cannot inspect the installed UK Simulation input aliases." + ) from exc + constructor = tree.body[0] + if not isinstance(constructor, (ast.FunctionDef, ast.AsyncFunctionDef)): + raise RegionalInputSurfaceError("Cannot locate the UK Simulation constructor.") + unconditional = { + id(statement.value) + for statement in constructor.body + if isinstance(statement, ast.Expr) + } + aliases: dict[str, str] = {} + for node in ast.walk(constructor): + if not ( + isinstance(node, ast.Call) + and isinstance(node.func, ast.Attribute) + and isinstance(node.func.value, ast.Name) + and node.func.value.id == "self" + and node.func.attr == "move_values" + ): + continue + if ( + id(node) not in unconditional + or len(node.args) != 2 + or node.keywords + or any( + not isinstance(argument, ast.Constant) + or not isinstance(argument.value, str) + for argument in node.args + ) + ): + raise RegionalInputSurfaceError( + "UK Simulation has a conditional or dynamic move_values input alias; " + "an explicit audit of its semantics is required." + ) + donor = str(node.args[0].value) # type: ignore[attr-defined] + target = str(node.args[1].value) # type: ignore[attr-defined] + if donor in aliases: + raise RegionalInputSurfaceError(f"Duplicate move_values alias {donor!r}.") + aliases[donor] = target + return aliases + + +def _model_surface(model_version: str) -> tuple[list[str], list[str], dict[str, str]]: + try: + installed = version("policyengine-uk") + except PackageNotFoundError as exc: + raise RegionalInputSurfaceError( + "Install the exact policyengine-uk target first." + ) from exc + if installed != model_version: + raise RegionalInputSurfaceError( + f"Installed policyengine-uk {installed} differs from requested {model_version}." + ) + # Keep country-model loading out of certification's evidence-validation path. + import policyengine_uk + + system = policyengine_uk.CountryTaxBenefitSystem() + variables = sorted(system.variables) + inputs = sorted( + name + for name, variable in system.variables.items() + if variable.is_input_variable() + ) + return variables, inputs, _input_aliases(policyengine_uk.Simulation) + + +def regional_build_versions(manifest: DataReleaseManifest) -> tuple[str, str]: + """Return the regional release's build model and core: the audit baseline. + + A release that records neither, or records other packages, refuses: there + is no model to tell a dropped column from one no model ever defined. + """ + build = manifest.build + model = build.built_with_model_package if build is not None else None + core = build.built_with_core_package if build is not None else None + if model is None or core is None: + raise RegionalInputSurfaceError( + "The regional manifest records no build model and core versions " + "(build.built_with_model_package and build.built_with_core_package), " + "so the audit has no baseline." + ) + if model.name != "policyengine-uk" or core.name != "policyengine-core": + raise RegionalInputSurfaceError( + f"The regional release was built with {model.name} and {core.name}, " + "not policyengine-uk and policyengine-core." + ) + for package in (model, core): + try: + Version(package.version) + except InvalidVersion as exc: + raise RegionalInputSurfaceError( + f"The regional manifest's {package.name} build version " + f"{package.version!r} is not a version." + ) from exc + return model.version, core.version + + +def regional_build_cutoff(manifest: DataReleaseManifest) -> str | None: + """Return when the regional release was built, as a UTC ``--exclude-newer``. + + The build model is read with the packages PyPI had then, so its other + dependencies resolve as they did at the build however they move on. + Rounded up to the second; ``None`` when the manifest records no + ``build.built_at``. + """ + built_at = manifest.build.built_at if manifest.build is not None else None + if built_at is None: + return None + try: + moment = datetime.fromisoformat(built_at) + except ValueError as exc: + raise RegionalInputSurfaceError( + f"The regional manifest's build.built_at {built_at!r} is not a timestamp." + ) from exc + if moment.tzinfo is None: + moment = moment.replace(tzinfo=UTC) + moment = moment.astimezone(UTC) + if moment.microsecond: + moment = moment.replace(microsecond=0) + timedelta(seconds=1) + return moment.strftime("%Y-%m-%dT%H:%M:%SZ") + + +def _build_model_surface( + model_version: str, + core_version: str, + run: Callable[..., Any] | None = None, + exclude_newer: str | None = None, +) -> tuple[list[str], list[str]]: + """Read the build model's variables and inputs in a throwaway environment. + + ``uv run --no-project --no-config`` installs exactly the two pins from + PyPI, with packages uploaded before ``exclude_newer`` when it is given, + and without the caller's uv settings or project. The probe reports the + versions it imported, and any other resolution refuses. + """ + run = run if run is not None else subprocess.run + label = f"policyengine-uk {model_version} with policyengine-core {core_version}" + command = [ + "uv", + "run", + "--no-project", + "--no-config", + "--default-index", + BUILD_MODEL_INDEX, + *(["--exclude-newer", exclude_newer] if exclude_newer else []), + "--python", + BUILD_MODEL_PYTHON, + "--with", + f"policyengine-uk=={model_version}", + "--with", + f"policyengine-core=={core_version}", + "python", + "-I", + "-c", + BUILD_MODEL_PROBE, + ] + env = { + key: value + for key, value in os.environ.items() + if key != "VIRTUAL_ENV" + and (not key.startswith("UV_") or key in BUILD_MODEL_KEPT_UV_SETTINGS) + } + try: + result = run(command, capture_output=True, text=True, check=True, env=env) + except subprocess.CalledProcessError as exc: + detail = " | ".join((exc.stderr or "").strip().splitlines()[-5:]) + raise RegionalInputSurfaceError( + f"Cannot read the build model {label}: {detail or exc}" + ) from exc + except OSError as exc: + raise RegionalInputSurfaceError( + f"Cannot run uv to read the build model {label}." + ) from exc + try: + surface = json.loads((result.stdout or "").strip().splitlines()[-1]) + except (IndexError, ValueError) as exc: + raise RegionalInputSurfaceError( + f"The probe of the build model {label} printed no surface." + ) from exc + if not isinstance(surface, dict): + raise RegionalInputSurfaceError( + f"The probe of the build model {label} printed no surface." + ) + resolved = (surface.get("model_version"), surface.get("core_version")) + if resolved != (model_version, core_version): + raise RegionalInputSurfaceError( + f"The build-model probe resolved policyengine-uk {resolved[0]} with " + f"policyengine-core {resolved[1]}, not {label}." + ) + variables, inputs = surface.get("variables"), surface.get("inputs") + if not isinstance(variables, list) or not isinstance(inputs, list): + raise RegionalInputSurfaceError( + f"The probe of the build model {label} printed no variable lists." + ) + return ( + sorted(_names(variables, "build_model_variables")), + sorted(_names(inputs, "build_model_inputs")), + ) + + +def _surface_difference(claimed: list[str], actual: list[str]) -> str: + extra = sorted(set(claimed) - set(actual)) + missing = sorted(set(actual) - set(claimed)) + shown = [f"+{name}" for name in extra[:5]] + [f"-{name}" for name in missing[:5]] + return ", ".join(shown) or "order or duplicates" + + +def verify_model_surfaces( + evidence: RegionalInputSurfaceEvidence, + regional_manifest: DataReleaseManifest, + *, + run: Callable[..., Any] | None = None, +) -> None: + """Refuse evidence whose model surfaces are not the models' own. + + Certification calls this, so editing the evidence cannot hide a dropped + column. The target model must be installed at ``evidence.model_version``: + its variables, inputs and input aliases are read in process, as the audit + reads them. The build model is read again in a throwaway ``uv run`` + environment, as the audit reads it. The stored columns cannot be re-read + without the H5: the auditor attests them, and the evidence binds them to + each H5's SHA-256. + """ + variables, inputs, aliases = _model_surface(evidence.model_version) + target = f"policyengine-uk {evidence.model_version}" + for field, claimed, actual in ( + ("model_variables", evidence.model_variables, variables), + ("model_inputs", evidence.model_inputs, inputs), + ): + if claimed != actual: + raise RegionalInputSurfaceError( + f"The evidence's {field} are not {target}'s " + f"({_surface_difference(claimed, actual)})." + ) + if (evidence.input_aliases or {}) != aliases: + raise RegionalInputSurfaceError( + f"The evidence's input_aliases are not {target}'s." + ) + build_variables, build_inputs = _build_model_surface( + evidence.build_model_version, + evidence.build_core_version, + run=run, + exclude_newer=regional_build_cutoff(regional_manifest), + ) + build = ( + f"policyengine-uk {evidence.build_model_version} with policyengine-core " + f"{evidence.build_core_version}" + ) + for field, claimed, actual in ( + ("build_model_variables", evidence.build_model_variables, build_variables), + ("build_model_inputs", evidence.build_model_inputs, build_inputs), + ): + if claimed != actual: + raise RegionalInputSurfaceError( + f"The evidence's {field} are not {build}'s " + f"({_surface_difference(claimed, actual)})." + ) + + +def _sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as file: + for chunk in iter(lambda: file.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def _hdf_columns(path: Path) -> dict[str, list[str]]: + import pandas as pd + + try: + with pd.HDFStore(path, mode="r") as store: + keys = set(store.keys()) + flat = {f"/{entity}" for entity in UK_ENTITIES} & keys + nested = { + key + for key in keys + if any(key.startswith(f"/{entity}/") for entity in UK_ENTITIES) + } + if flat and nested: + raise RegionalInputSurfaceError( + "H5 mixes single-year and multi-year entity keys." + ) + entity_keys: dict[str, dict[str, str]] = {} + if flat: + if "/time_period" not in keys: + raise RegionalInputSurfaceError( + "Single-year H5 requires /time_period." + ) + metadata = store.select("/time_period") + values = metadata.to_numpy().ravel().tolist() + if len(values) != 1 or not re.fullmatch(r"[0-9]{4}", str(values[0])): + raise RegionalInputSurfaceError( + "H5 /time_period must name one year." + ) + entity_keys[str(values[0])] = { + entity: f"/{entity}" + for entity in UK_ENTITIES + if f"/{entity}" in flat + } + else: + for key in sorted(nested): + parts = key.strip("/").split("/") + if len(parts) != 2 or not re.fullmatch(r"[0-9]{4}", parts[1]): + raise RegionalInputSurfaceError( + f"Invalid H5 entity period key {key!r}." + ) + entity_keys.setdefault(parts[1], {})[parts[0]] = key + if not entity_keys: + raise RegionalInputSurfaceError("H5 has no UK entity periods.") + columns_by_period = {} + for period, entities in sorted(entity_keys.items()): + missing = set(UK_ENTITIES) - set(entities) + if missing: + raise RegionalInputSurfaceError( + f"H5 period {period} is missing entity tables: {sorted(missing)}." + ) + columns: set[str] = set() + for entity in UK_ENTITIES: + # This obtains the same column metadata for fixed and table + # stores without selecting household or person rows. + frame = store.select(entities[entity], start=0, stop=0) + columns.update( + _names(list(frame.columns), f"{entity}/{period} columns") + ) + columns_by_period[period] = sorted(columns) + return columns_by_period + except RegionalInputSurfaceError: + raise + except (OSError, KeyError, ValueError, ImportError) as exc: + raise RegionalInputSurfaceError( + f"Cannot read H5 entity metadata from {path.name}." + ) from exc + + +def measure_regional_input_surface( + regional_manifest_uri: str, + artifact_paths: dict[str, Path], + model_version: str, + audited_by: str, + limitations: list[str], + token: str | None = None, + *, + run: Callable[..., Any] | None = None, +) -> RegionalInputSurfaceEvidence: + """Audit actual H5 columns against the build model and the exact target. + + The operator supplies local files named by the immutable regional release. + Every microdata H5 must be measured and match its manifest's SHA-256 both + before and after metadata inspection. The installed policyengine-uk must + be ``model_version``. The build model and core named by the manifest are + read in a throwaway ``uv run`` environment through ``run`` + (``subprocess.run`` by default), resolving as of the manifest's + ``build.built_at``. Dropped columns refuse. This reads no unit values for + output. + """ + # Imported lazily: the certification gate reuses compare_input_surface. + from policyengine.provenance.certification import fetch_release_manifest + + if ( + not readable(audited_by) + or not limitations + or not all(readable(item) for item in limitations) + ): + raise RegionalInputSurfaceError( + "An auditor and limitations a reader can see are required." + ) + manifest, manifest_sha256, parts = fetch_release_manifest( + regional_manifest_uri, token=token + ) + build_model_version, build_core_version = regional_build_versions(manifest) + build_cutoff = regional_build_cutoff(manifest) + expected = { + name: artifact + for name, artifact in manifest.artifacts.items() + if artifact.kind == "microdata" and Path(artifact.path).suffix.lower() == ".h5" + } + if not expected or set(artifact_paths) != set(expected): + raise RegionalInputSurfaceError( + "Supplied artifact paths must exactly cover the regional microdata H5 artifacts: " + f"{sorted(expected)}." + ) + variables, inputs, aliases = _model_surface(model_version) + build_variables, build_inputs = _build_model_surface( + build_model_version, build_core_version, run=run, exclude_newer=build_cutoff + ) + artifacts = {} + for name, artifact in sorted(expected.items()): + path = Path(artifact_paths[name]) + if not artifact.sha256: + raise RegionalInputSurfaceError( + f"Artifact {name!r} has no manifest sha256." + ) + before = _sha256_file(path) + if before != artifact.sha256: + raise RegionalInputSurfaceError( + f"Artifact {name!r} bytes differ from its manifest sha256." + ) + periods = _hdf_columns(path) + after = _sha256_file(path) + if after != before: + raise RegionalInputSurfaceError( + f"Artifact {name!r} bytes changed during metadata inspection." + ) + artifacts[name] = RegionalInputArtifactEvidence( + repo_id=artifact.repo_id, + repo_type=artifact.repo_type or parts["repo_type"], + path=artifact.path, + revision=artifact.revision, + sha256=after, + columns=sorted( + {column for columns in periods.values() for column in columns} + ), + columns_by_period=periods, + ) + comparison = compare_input_surface( + artifacts, + variables, + inputs, + aliases, + build_model_variables=build_variables, + build_model_inputs=build_inputs, + ) + if comparison["dropped_columns"]: + raise RegionalInputSurfaceError( + f"Columns policyengine-uk {build_model_version} defines and " + f"{model_version} does not would be dropped, which refuses " + f"acceptance: {comparison['dropped_columns']}." + ) + return RegionalInputSurfaceEvidence( + schema_version=2, + regional_manifest_uri=regional_manifest_uri, + regional_manifest_sha256=manifest_sha256, + model_package="policyengine-uk", + model_version=model_version, + build_model_version=build_model_version, + build_core_version=build_core_version, + artifacts=artifacts, + model_variables=variables, + model_inputs=inputs, + build_model_variables=build_variables, + build_model_inputs=build_inputs, + input_aliases=aliases, + dropped_columns=comparison["dropped_columns"], + never_modelled_columns=comparison["never_modelled_columns"], + defaulted_inputs=comparison["defaulted_inputs"], + audited_by=audited_by, + limitations=limitations, + ) diff --git a/src/policyengine/provenance/trace.py b/src/policyengine/provenance/trace.py index bcffda78..0e79b34e 100644 --- a/src/policyengine/provenance/trace.py +++ b/src/policyengine/provenance/trace.py @@ -556,6 +556,15 @@ def _build_bundle_performance( performance["pe:dataBuildId"] = certification.data_build_id if certification.certified_by is not None: performance["pe:certifiedBy"] = certification.certified_by + if country_manifest.regional_certification is not None: + # An explicit JSON literal preserves evidence member names through + # JSON-LD expansion; they are JSON contract keys, not RDF predicates. + performance["pe:regionalCertification"] = { + "@type": "@json", + "@value": country_manifest.regional_certification.model_dump( + mode="json", exclude_none=True + ), + } performance.update(dict(emission_context or _emission_context())) return performance diff --git a/tests/test_certify_data_release.py b/tests/test_certify_data_release.py index 092420f6..7d2e2c93 100644 --- a/tests/test_certify_data_release.py +++ b/tests/test_certify_data_release.py @@ -1412,6 +1412,254 @@ def test__given_regional_release_without_a_claim__then_refuses(self, tmp_path): regional_release_kind="area_coded_national", ) + def _older_dense(self) -> dict: + dense = _microcosm_uk_dense_payload() + dense["build"]["built_with_model_package"]["version"] = "2.120.0" + dense["build"]["built_with_core_package"] = { + "name": "policyengine-core", + "version": "3.33.3", + } + dense["compatible_model_packages"] = [ + {"name": "policyengine-uk", "specifier": "==2.120.0"} + ] + return dense + + def test__given_older_dense_cut__then_refuses_without_an_accepted_mismatch( + self, tmp_path + ): + with pytest.raises(CertificationError, match="Regional release .*2.126.12"): + _certify_uk( + _tmp_bundle(tmp_path), + [ + _response(_microcosm_uk_national_payload()), + _response(self._older_dense()), + ], + regional_manifest_uri=UK_DENSE_URI, + ) + + def test__given_accepted_mismatch__then_routes_and_records_it( + self, tmp_path, monkeypatch + ): + bundle_path = _tmp_bundle(tmp_path) + reason = "area receipts are uncertified; the dense cut lags one release" + from policyengine.provenance.certification import ( + regional_defaults_limitation, + ) + from tests.test_regional_input_evidence import ( + fake_model_surfaces, + input_evidence, + ) + + fake_model_surfaces(monkeypatch) + evidence = input_evidence(self._older_dense()) + + result = _certify_uk( + bundle_path, + [ + _response(_microcosm_uk_national_payload()), + _response(self._older_dense()), + ], + regional_manifest_uri=UK_DENSE_URI, + regional_model_mismatch_reason=reason, + regional_input_surface_evidence=evidence, + ) + + release = json.loads(bundle_path.read_text())["data_releases"]["uk"] + assert release["regional_release_kind"] == "local_area" + assert release["regional_certification"] == { + "compatibility_basis": "accepted_model_mismatch", + "certified_for_model_version": UK_MODEL_VERSION, + "built_with_model_version": "2.120.0", + "accepted_reason": reason, + "area_receipts": "uncertified", + "regional_manifest_uri": UK_DENSE_URI, + "regional_manifest_sha256": evidence["regional_manifest_sha256"], + "defaulted_inputs": evidence["defaulted_inputs"], + "dropped_columns": [], + "never_modelled_columns": [], + "limitations": [ + *evidence["limitations"], + regional_defaults_limitation( + "policyengine-uk", + UK_MODEL_VERSION, + "2.120.0", + evidence["defaulted_inputs"], + ), + ], + "evidence": evidence, + } + # The national certificate itself is untouched by the regional record. + assert release["certification"]["compatibility_basis"] == ( + "built_with_model_package" + ) + assert _sim_api_routes_both_area_types(release) + manifest = CountryReleaseManifest.model_validate(release) + assert manifest.regional_certification is not None + assert manifest.regional_certification.built_with_model_version == "2.120.0" + mismatch = [w for w in result.warnings if "accepted mismatch" in w] + assert len(mismatch) == 1 and "2.120.0" in mismatch[0] + + def test__given_a_covering_regional_release__then_an_accepted_mismatch_refuses( + self, tmp_path + ): + with pytest.raises(CertificationError, match="drop --accept-regional"): + _certify_uk( + _tmp_bundle(tmp_path), + [ + _response(_microcosm_uk_national_payload()), + _response(_microcosm_uk_dense_payload()), + ], + regional_manifest_uri=UK_DENSE_URI, + regional_model_mismatch_reason="not needed", + ) + + # A zero-width space is not whitespace, so strip() alone kept it. + @pytest.mark.parametrize("reason", ["", " ", "\t\n", "\u200b", "\ufeff\u2060"]) + def test__given_an_empty_reason__then_refuses(self, tmp_path, reason): + with pytest.raises(CertificationError, match="needs a reason"): + _certify_uk( + _tmp_bundle(tmp_path), + [ + _response(_microcosm_uk_national_payload()), + _response(self._older_dense()), + ], + regional_manifest_uri=UK_DENSE_URI, + regional_model_mismatch_reason=reason, + ) + + def test__given_an_accepted_mismatch__then_the_core_must_still_be_covered( + self, tmp_path, monkeypatch + ): + from tests.test_regional_input_evidence import ( + fake_model_surfaces, + input_evidence, + ) + + fake_model_surfaces(monkeypatch) + national = _microcosm_uk_national_payload() + national["compatible_core_packages"] = [ + {"name": "policyengine-core", "specifier": "==3.33.0"} + ] + with pytest.raises( + CertificationError, match="Regional release .*policyengine-core 3.33.0" + ): + _certify_uk( + _tmp_bundle(tmp_path), + [_response(national), _response(self._older_dense())], + regional_manifest_uri=UK_DENSE_URI, + regional_model_mismatch_reason="area receipts are uncertified", + regional_input_surface_evidence=input_evidence(self._older_dense()), + core_version="3.33.0", + ) + + def test__given_an_accepted_mismatch_without_a_regional_release__then_refuses( + self, tmp_path + ): + with pytest.raises(CertificationError, match="pass --regional-manifest-uri"): + _certify_uk( + _tmp_bundle(tmp_path), + [_response(_microcosm_uk_national_payload())], + drop_regional_release=True, + regional_model_mismatch_reason="area receipts are uncertified", + ) + + def test__given_bundle_wrapper_accepted_mismatch__then_forwards_it( + self, monkeypatch + ): + bundle_script = _load_bundle_script(monkeypatch) + captured: dict[str, list[str]] = {} + fake = ModuleType("certify_data_release") + + def fake_main(argv): + captured["argv"] = argv + return 0 + + fake.main = fake_main + monkeypatch.setitem(sys.modules, "certify_data_release", fake) + reason = "area receipts are uncertified" + assert ( + bundle_script.main( + [ + "certify-data", + "--country", + "uk", + "--manifest-uri", + UK_NATIONAL_URI, + "--regional-manifest-uri", + UK_DENSE_URI, + "--accept-regional-model-mismatch", + reason, + "--regional-input-surface-evidence", + "regional-audit.json", + ] + ) + == 0 + ) + argv = captured["argv"] + assert argv[argv.index("--accept-regional-model-mismatch") + 1] == reason + assert ( + argv[argv.index("--regional-input-surface-evidence") + 1] + == "regional-audit.json" + ) + + def test__given_bundle_wrapper_empty_reason__then_certifier_refuses_it( + self, monkeypatch, tmp_path, capsys + ): + bundle_script = _load_bundle_script(monkeypatch) + spec = importlib.util.spec_from_file_location( + "certify_data_release", + Path(__file__).resolve().parents[1] / "scripts" / "certify_data_release.py", + ) + certifier = importlib.util.module_from_spec(spec) + assert spec.loader is not None + spec.loader.exec_module(certifier) + monkeypatch.setitem(sys.modules, "certify_data_release", certifier) + bundle_path = _tmp_bundle(tmp_path) + original_bundle = bundle_path.read_bytes() + captured = {} + + def certify_temporary_bundle(**kwargs): + captured.update(kwargs) + kwargs["bundle_path"] = bundle_path + return certify_data_release(**kwargs) + + monkeypatch.setattr(certifier, "certify_data_release", certify_temporary_bundle) + with ( + patch( + "policyengine.provenance.certification.requests.get", + side_effect=[ + _response(_microcosm_uk_national_payload()), + _response(self._older_dense()), + ], + ), + pytest.raises(SystemExit) as error, + ): + bundle_script.main( + [ + "certify-data", + "--country", + "uk", + "--manifest-uri", + UK_NATIONAL_URI, + "--model-version", + UK_MODEL_VERSION, + "--regional-manifest-uri", + UK_DENSE_URI, + "--accept-regional-model-mismatch", + "", + "--no-generate", + "--no-changelog", + "--skip-artifact-check", + ] + ) + # bundle.py reports the certifier's refusal as a usage error. + assert error.value.code == 2 + assert isinstance(error.value.__context__, CertificationError) + assert "needs a reason" in str(error.value.__context__) + assert capsys.readouterr().out == "" + assert captured["regional_model_mismatch_reason"] == "" + assert bundle_path.read_bytes() == original_bundle + def test__given_pinned_core__then_the_regional_release_must_cover_it( self, tmp_path ): diff --git a/tests/test_execution_receipt.py b/tests/test_execution_receipt.py index 66fe4d22..c290c362 100644 --- a/tests/test_execution_receipt.py +++ b/tests/test_execution_receipt.py @@ -433,6 +433,263 @@ def test_a_regional_release_dataset_keeps_its_own_identity(dataset_file): assert receipt.data.release_manifest_revision is None +def _bundle_with_regional_acceptance(dataset_file): + """A synthetic, audited area exception beside a national certificate.""" + + bundle = _bundle_for(dataset_file.read_bytes()) + record = bundle["data_releases"]["uk"] + artifact = { + "path": "area_file.h5", + "repo_id": record["data_package"]["repo_id"], + "revision": "regional-release-cut", + "sha256": hashlib.sha256(dataset_file.read_bytes()).hexdigest(), + } + record["datasets"]["area_file"] = artifact + manifest_uri = ( + "hf://dataset/policyengine/populace-uk-private@regional-release-cut/" + "releases/dense/release_manifest.json" + ) + defaulted = [ + "local_bus_uncapped_single_fare", + "salary_sacrifice_returned_to_income", + ] + built = ["employment_income", "household_weight"] + stored = ["constituency_code_oa", *built] + limitations = [ + "The dense cut's VAT predates the policyengine-uk VAT factor fix " + "(#2187, 2.135.7); the dashboard holds a VAT limit." + ] + record["regional_certification"] = { + "compatibility_basis": "accepted_model_mismatch", + "certified_for_model_version": INSTALLED_UK, + "built_with_model_version": "2.132.6", + "accepted_reason": "UK hub accepted this measured input surface.", + "area_receipts": "uncertified", + "regional_manifest_uri": manifest_uri, + "regional_manifest_sha256": "b" * 64, + "defaulted_inputs": defaulted, + "dropped_columns": [], + "never_modelled_columns": ["constituency_code_oa"], + "limitations": limitations, + "evidence": { + "schema_version": 2, + "model_package": "policyengine-uk", + "model_version": INSTALLED_UK, + "build_model_version": "2.132.6", + "build_core_version": "3.33.3", + "regional_manifest_uri": manifest_uri, + "regional_manifest_sha256": "b" * 64, + "artifacts": { + "area_file": { + **artifact, + "columns": stored, + "columns_by_period": {"2025": stored}, + } + }, + "model_variables": sorted([*built, *defaulted]), + "model_inputs": sorted([*built, *defaulted]), + "build_model_variables": built, + "build_model_inputs": built, + "dropped_columns": [], + "never_modelled_columns": ["constituency_code_oa"], + "defaulted_inputs": defaulted, + "audited_by": "UK hub review", + "limitations": limitations, + }, + } + return bundle + + +def test_an_area_receipt_carries_a_digest_bound_summary_of_the_exception( + dataset_file, schema_validator +): + from policyengine.provenance.receipt import regional_certification_identity + + bundle = _bundle_with_regional_acceptance(dataset_file) + record = bundle["data_releases"]["uk"]["regional_certification"] + + receipt = _build(dataset_file, bundle, dataset="area_file") + + payload = json.loads(receipt.model_dump_json(exclude_none=True)) + summary = payload["regional_certification"] + assert summary == regional_certification_identity(record).canonical_payload() + assert summary["record_sha256"] == content_sha256(record) + assert summary["never_modelled_columns"] == ["constituency_code_oa"] + assert summary["limitations"] == record["limitations"] + # The evidence's model surfaces stay in the bundle and TRACE. + assert "evidence" not in summary + assert "model_variables" not in json.dumps(payload) + assert receipt.certification_status == "uncertified" + assert receipt.certificate is None + assert receipt.data.build_model_version == "2.132.6" + schema_validator.validate(payload) + restored = ExecutionReceipt.model_validate_json(receipt.model_dump_json()) + assert restored.regional_certification == receipt.regional_certification + assert restored.receipt_sha256 == receipt.receipt_sha256 + + +def test_a_national_receipt_leaves_the_exception_to_its_certificate(dataset_file): + """Review of #699: certificate_sha256 already digests the regional record.""" + bundle = _bundle_with_regional_acceptance(dataset_file) + record = bundle["data_releases"]["uk"] + + receipt = _build(dataset_file, bundle) + + assert receipt.certification_status == "certified" + assert "regional_certification" not in receipt.canonical_payload() + assert receipt.certificate.certificate_sha256 == certificate_sha256(record) + assert verify_certificate(receipt, bundle) == [] + changed = copy.deepcopy(bundle) + changed["data_releases"]["uk"]["regional_certification"]["limitations"].append( + "Additional accepted area limitation." + ) + assert any( + "certificate_sha256" in problem + for problem in verify_certificate(receipt, changed) + ) + + +def test_a_certified_receipt_cannot_carry_the_regional_exception(dataset_file): + bundle = _bundle_with_regional_acceptance(dataset_file) + area = _build(dataset_file, bundle, dataset="area_file") + payload = _build(dataset_file, bundle).model_dump(mode="json") + payload["regional_certification"] = area.regional_certification.model_dump( + mode="json" + ) + with pytest.raises(ValidationError, match="regional_certification"): + ExecutionReceipt.model_validate(payload) + + +@pytest.mark.parametrize("change", ["limitations", "evidence"]) +def test_the_regional_record_is_bound_to_area_receipt_identity(dataset_file, change): + bundle = _bundle_with_regional_acceptance(dataset_file) + receipt = _build(dataset_file, bundle, dataset="area_file") + changed_bundle = copy.deepcopy(bundle) + record = changed_bundle["data_releases"]["uk"]["regional_certification"] + if change == "limitations": + record["limitations"].append("Additional accepted area limitation.") + record["evidence"]["limitations"].append("Additional accepted area limitation.") + else: + # Not in the summary, but in the record its digest covers. + record["evidence"]["audited_by"] = "Another reviewer" + + changed = _build(dataset_file, changed_bundle, dataset="area_file") + + assert changed.execution_identity_sha256 != receipt.execution_identity_sha256 + assert changed.receipt_sha256 != receipt.receipt_sha256 + + +def test_regional_verification_checks_an_area_receipt_against_the_bundle( + dataset_file, +): + from policyengine.provenance.receipt import verify_regional_certification + + bundle = _bundle_with_regional_acceptance(dataset_file) + receipt = _build(dataset_file, bundle, dataset="area_file") + assert verify_regional_certification(receipt, bundle) == [] + + payload = receipt.model_dump(mode="json") + payload["regional_certification"]["accepted_reason"] = "A different acceptance." + claimed = ExecutionReceipt.model_validate(payload) + assert verify_regional_certification(claimed, bundle) == [ + "regional_certification.accepted_reason differs from the bundle's record" + ] + + edited = copy.deepcopy(bundle) + edited["data_releases"]["uk"]["regional_certification"]["evidence"][ + "audited_by" + ] = "Another reviewer" + assert verify_regional_certification(receipt, edited) == [ + "regional_certification.record_sha256 differs from the bundle's record" + ] + assert verify_regional_certification(_build(dataset_file, bundle), bundle) == [ + "the receipt carries no regional_certification" + ] + + +def test_trace_exposes_regional_acceptance_separately_from_national(dataset_file): + from policyengine.provenance.manifest import ( + CountryReleaseManifest, + RegionalCertification, + ) + from policyengine.provenance.trace import build_trace_tro_from_release_bundle + + record = _bundle_with_regional_acceptance(dataset_file)["data_releases"]["uk"] + tro = build_trace_tro_from_release_bundle( + CountryReleaseManifest.model_validate(record), + None, + fetch_pypi=lambda name, version: {}, + ) + performance = tro["@graph"][0]["trov:hasPerformance"] + + assert performance["pe:regionalCertification"]["@type"] == "@json" + assert performance["pe:regionalCertification"][ + "@value" + ] == RegionalCertification.model_validate( + record["regional_certification"] + ).model_dump(mode="json", exclude_none=True) + assert ( + performance["pe:compatibilityBasis"] + == record["certification"]["compatibility_basis"] + ) + assert ( + performance["pe:regionalCertification"]["@value"]["area_receipts"] + == "uncertified" + ) + + +def test_trace_regional_acceptance_survives_jsonld_expansion(dataset_file): + rdflib = pytest.importorskip("rdflib") + from policyengine.provenance.manifest import CountryReleaseManifest + from policyengine.provenance.trace import ( + POLICYENGINE_TRACE_NAMESPACE, + build_trace_tro_from_release_bundle, + ) + + record = _bundle_with_regional_acceptance(dataset_file)["data_releases"]["uk"] + manifest = CountryReleaseManifest.model_validate(record) + tro = build_trace_tro_from_release_bundle( + manifest, + None, + fetch_pypi=lambda name, version: {}, + ) + + graph = rdflib.Graph().parse(data=json.dumps(tro), format="json-ld") + values = list( + graph.objects( + None, rdflib.URIRef(POLICYENGINE_TRACE_NAMESPACE + "regionalCertification") + ) + ) + + assert len(values) == 1 + assert isinstance(values[0], rdflib.Literal) + assert values[0].datatype == rdflib.RDF.JSON + assert json.loads(str(values[0])) == manifest.regional_certification.model_dump( + mode="json", exclude_none=True + ) + + +def test_removing_regional_acceptance_removes_downstream_record(dataset_file): + from policyengine.provenance.manifest import CountryReleaseManifest + from policyengine.provenance.trace import build_trace_tro_from_release_bundle + + bundle = _bundle_with_regional_acceptance(dataset_file) + record = bundle["data_releases"]["uk"] + del record["regional_certification"] + del record["datasets"]["area_file"] + + receipt = _build(dataset_file, bundle) + tro = build_trace_tro_from_release_bundle( + CountryReleaseManifest.model_validate(record), + None, + fetch_pypi=lambda name, version: {}, + ) + + assert "regional_certification" not in receipt.canonical_payload() + assert "pe:regionalCertification" not in tro["@graph"][0]["trov:hasPerformance"] + assert receipt.certification_status == "certified" + + def test_an_omitted_artifact_revision_is_the_release_manifest_revision(dataset_file): """Round-4 review M5: the receipt names what the materializer downloads.""" diff --git a/tests/test_measure_regional_inputs.py b/tests/test_measure_regional_inputs.py new file mode 100644 index 00000000..814c0dab --- /dev/null +++ b/tests/test_measure_regional_inputs.py @@ -0,0 +1,726 @@ +"""Measure the actual dense H5 surface against its build model and the target.""" + +from __future__ import annotations + +import hashlib +import importlib +import importlib.metadata +import importlib.util +import json +import os +import subprocess +import sys +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import MagicMock, patch + +import pandas as pd +import pytest + +from policyengine.provenance.manifest import RegionalInputArtifactEvidence + +MODEL_VERSION = "2.147.2" +BUILD_VERSION = "2.132.6" +BUILD_CORE_VERSION = "3.33.3" +BUILT_AT = "2026-10-19T02:07:51.099627+00:00" +#: BUILT_AT in UTC, rounded up to the second: the probe's --exclude-newer. +BUILD_CUTOFF = "2026-10-19T02:07:52Z" +URI = ( + "hf://dataset/policyengine/regional@dense-cut/releases/dense/release_manifest.json" +) +LIMITATION = "Reviewed dense VAT limitation; dashboard holds a VAT limit" +DEFAULTS = ["local_bus_uncapped_single_fare", "salary_sacrifice_returned_to_income"] +#: An input of both models that the files never store: not a change. +ABSENT_UNDER_BOTH = "legacy_default_input" +#: A build-model variable that the target model no longer defines. +REMOVED = "removed_in_target" +#: A Microcosm column that neither model defines. +AREA_CODE = "constituency_code_oa" +BUILD_INPUTS = [ + "benunit_id", + "employment_income_before_lsr", + "household_weight", + ABSENT_UNDER_BOTH, + REMOVED, +] +BUILD_VARIABLES = sorted([*BUILD_INPUTS, "employment_income"]) +PERSON = ("employment_income",) + + +class FakeSimulation: + def __init__(self): + self.move_values("employment_income", "employment_income_before_lsr") + + def move_values(self, source, target): + pass + + +class FakeVariable: + def __init__(self, is_input): + self.is_input = is_input + + def is_input_variable(self): + return self.is_input + + +class FakeCountryTaxBenefitSystem: + def __init__(self): + inputs = { + "employment_income_before_lsr", + "household_weight", + "benunit_id", + ABSENT_UNDER_BOTH, + *DEFAULTS, + } + self.variables = { + name: FakeVariable(name in inputs) + for name in sorted(inputs | {"employment_income"}) + } + + +class FakeRunner: + """Stands in for ``uv run`` reading the build model, so nothing installs.""" + + def __init__(self, **surface): + self.surface = { + "model_version": BUILD_VERSION, + "core_version": BUILD_CORE_VERSION, + "variables": BUILD_VARIABLES, + "inputs": BUILD_INPUTS, + } | surface + self.calls = [] + + def __call__(self, command, **kwargs): + self.calls.append((command, kwargs)) + stdout = "Installed 40 packages\n" + json.dumps(self.surface) + "\n" + return subprocess.CompletedProcess(command, 0, stdout=stdout, stderr="") + + +def _module(): + return importlib.import_module("policyengine.provenance.regional_input_surface") + + +def _h5(path, *, person=PERSON, periods=None, fmt="fixed", missing=None): + """Write UK entity tables: one year, or each period's person columns.""" + with pd.HDFStore(path, mode="w") as store: + if periods is None: + store.put("time_period", pd.Series([2025])) + periods = {None: person} + for period, person_columns in periods.items(): + for entity, columns in { + "person": person_columns or ("benunit_id",), + "benunit": ("benunit_id",), + "household": ("household_weight",), + }.items(): + if (period, entity) == missing: + continue + key = entity if period is None else f"{entity}/{period}" + frame = pd.DataFrame({column: [1.0] for column in columns}) + store.put(key, frame, format=fmt) + return path + + +def _response(artifact_paths, build=None): + payload = { + "schema_version": 1, + "data_package": {"name": "microcosm-data", "version": "1"}, + "dataset_role": "non_default_local_area", + "is_default": False, + "default_datasets": {}, + "build": { + "built_at": BUILT_AT, + "built_with_model_package": { + "name": "policyengine-uk", + "version": BUILD_VERSION, + }, + "built_with_core_package": { + "name": "policyengine-core", + "version": BUILD_CORE_VERSION, + }, + } + if build is None + else build, + "artifacts": { + name: { + "kind": "microdata", + "path": f"{name}.h5", + "repo_id": "policyengine/regional", + "revision": "dense-cut", + "sha256": hashlib.sha256(path.read_bytes()).hexdigest(), + } + for name, path in artifact_paths.items() + }, + } + response = MagicMock(status_code=200) + response.content = json.dumps(payload, indent=2).encode() + return response + + +@pytest.fixture +def model(monkeypatch): + module = _module() + monkeypatch.setattr(module, "version", lambda name: MODEL_VERSION) + monkeypatch.setitem( + sys.modules, + "policyengine_uk", + SimpleNamespace( + CountryTaxBenefitSystem=FakeCountryTaxBenefitSystem, + Simulation=FakeSimulation, + ), + ) + return module + + +def _measure(module, paths, response=None, run=None, **kwargs): + response = _response(paths) if response is None else response + with patch( + "policyengine.provenance.certification.requests.get", return_value=response + ): + return module.measure_regional_input_surface( + URI, + paths, + MODEL_VERSION, + "Budget operator", + [LIMITATION], + run=FakeRunner() if run is None else run, + **kwargs, + ) + + +def _artifact(columns_by_period): + return RegionalInputArtifactEvidence( + repo_id="r", + path="dense.h5", + revision="cut", + sha256="a" * 64, + columns=sorted({column for c in columns_by_period.values() for column in c}), + columns_by_period=columns_by_period, + ) + + +@pytest.mark.parametrize("fmt", ["fixed", "table"]) +def test_actual_h5_columns_and_aliases_produce_bound_audit(tmp_path, model, fmt): + paths = {"dense": _h5(tmp_path / "dense.h5", fmt=fmt)} + response = _response(paths) + result = _measure(model, paths, response) + assert result.schema_version == 2 + assert result.model_version == MODEL_VERSION + assert result.model_package == "policyengine-uk" + assert result.build_model_version == BUILD_VERSION + assert result.build_core_version == BUILD_CORE_VERSION + assert result.build_model_variables == BUILD_VARIABLES + assert result.build_model_inputs == BUILD_INPUTS + assert result.regional_manifest_uri == URI + assert ( + result.regional_manifest_sha256 == hashlib.sha256(response.content).hexdigest() + ) + artifact = result.artifacts["dense"] + assert artifact.sha256 == hashlib.sha256(paths["dense"].read_bytes()).hexdigest() + assert artifact.repo_type == "dataset" + assert artifact.columns == ["benunit_id", "employment_income", "household_weight"] + assert artifact.columns_by_period == {"2025": artifact.columns} + assert result.input_aliases == {"employment_income": "employment_income_before_lsr"} + assert result.defaulted_inputs == DEFAULTS + assert result.dropped_columns == [] + assert result.never_modelled_columns == [] + assert result.limitations == [LIMITATION] + assert result.audited_by == "Budget operator" + + +def test_a_default_in_one_period_is_not_hidden_by_another_period(tmp_path, model): + paths = { + "dense": _h5( + tmp_path / "dense.h5", + periods={ + "2025": (*PERSON, "salary_sacrifice_returned_to_income"), + "2026": PERSON, + }, + ) + } + result = _measure(model, paths) + assert result.defaulted_inputs == DEFAULTS + periods = result.artifacts["dense"].columns_by_period + assert "salary_sacrifice_returned_to_income" in periods["2025"] + assert "salary_sacrifice_returned_to_income" not in periods["2026"] + + +@pytest.mark.parametrize("entity", ["person", "benunit", "household"]) +def test_each_period_requires_every_entity(tmp_path, model, entity): + paths = { + "dense": _h5( + tmp_path / "dense.h5", + periods={"2025": PERSON, "2026": PERSON}, + missing=("2026", entity), + ) + } + with pytest.raises(ValueError, match="2026|entity"): + _measure(model, paths) + + +def test_single_year_layout_requires_a_time_period(tmp_path, model): + path = _h5(tmp_path / "dense.h5") + with pd.HDFStore(path, mode="a") as store: + store.remove("time_period") + with pytest.raises(ValueError, match="time_period|period"): + _measure(model, {"dense": path}) + + +def test_extra_metadata_is_not_mistaken_for_a_model_column(tmp_path, model): + path = _h5(tmp_path / "dense.h5") + with pd.HDFStore(path, mode="a") as store: + store.put("metadata", pd.Series(["producer build notes"])) + result = _measure(model, {"dense": path}) + assert result.dropped_columns == [] + assert result.never_modelled_columns == [] + + +def test_a_column_neither_model_defines_is_recorded_not_refused(tmp_path, model): + path = _h5(tmp_path / "dense.h5", person=(*PERSON, AREA_CODE)) + result = _measure(model, {"dense": path}) + assert result.never_modelled_columns == [AREA_CODE] + assert result.dropped_columns == [] + assert AREA_CODE in result.artifacts["dense"].columns + + +def test_a_column_the_build_model_defines_and_the_target_does_not_blocks_audit( + tmp_path, model +): + path = _h5(tmp_path / "dense.h5", person=(*PERSON, REMOVED)) + with pytest.raises(ValueError, match=f"dropped.*{REMOVED}"): + _measure(model, {"dense": path}) + + +def test_an_input_absent_under_both_models_is_not_reported(tmp_path, model): + paths = {"dense": _h5(tmp_path / "dense.h5")} + assert ABSENT_UNDER_BOTH not in _measure(model, paths).defaulted_inputs + # Had the build model computed it, area runs would now default it. + computed = FakeRunner( + inputs=[name for name in BUILD_INPUTS if name != ABSENT_UNDER_BOTH] + ) + assert ABSENT_UNDER_BOTH in _measure(model, paths, run=computed).defaulted_inputs + + +def test_build_model_is_read_at_its_manifest_pins_without_the_callers_settings( + tmp_path, model, monkeypatch +): + # Start from no uv settings: CI's setup-uv sets kept ones of its own. + for name in [name for name in os.environ if name.startswith("UV_")]: + monkeypatch.delenv(name) + for name in ( + "UV_FROZEN", + "UV_PROJECT_ENVIRONMENT", + "VIRTUAL_ENV", + "UV_INDEX_URL", + "UV_EXTRA_INDEX_URL", + "UV_CONSTRAINT", + "UV_OVERRIDE", + "UV_CONFIG_FILE", + ): + monkeypatch.setenv(name, "from the caller") + monkeypatch.setenv("UV_CACHE_DIR", "/kept/cache") + monkeypatch.setenv("KEPT_SETTING", "1") + run = FakeRunner() + _measure(model, {"dense": _h5(tmp_path / "dense.h5")}, run=run) + ((command, kwargs),) = run.calls + assert command[:-1] == [ + "uv", + "run", + "--no-project", + "--no-config", + "--default-index", + "https://pypi.org/simple", + "--exclude-newer", + BUILD_CUTOFF, + "--python", + "3.13", + "--with", + f"policyengine-uk=={BUILD_VERSION}", + "--with", + f"policyengine-core=={BUILD_CORE_VERSION}", + "python", + "-I", + "-c", + ] + assert command[-1] == model.BUILD_MODEL_PROBE + assert kwargs["check"] is True + assert "VIRTUAL_ENV" not in kwargs["env"] + assert {name for name in kwargs["env"] if name.startswith("UV_")} == { + "UV_CACHE_DIR" + } + assert kwargs["env"]["KEPT_SETTING"] == "1" + + +@pytest.mark.parametrize( + ("built_at", "cutoff"), + [ + (BUILT_AT, BUILD_CUTOFF), + ("2026-10-19T03:07:51+01:00", "2026-10-19T02:07:51Z"), + ("2026-10-19T02:07:51", "2026-10-19T02:07:51Z"), + (None, None), + ], +) +def test_the_build_model_resolves_as_of_the_release_build_in_utc( + tmp_path, model, built_at, cutoff +): + paths = {"dense": _h5(tmp_path / "dense.h5")} + response = _response(paths) + payload = json.loads(response.content) + payload["build"]["built_at"] = built_at + response.content = json.dumps(payload).encode() + run = FakeRunner() + _measure(model, paths, response, run=run) + ((command, _),) = run.calls + if cutoff is None: + assert "--exclude-newer" not in command + else: + assert command[command.index("--exclude-newer") + 1] == cutoff + + +def test_an_unreadable_build_time_refuses_before_any_model_is_read(tmp_path, model): + paths = {"dense": _h5(tmp_path / "dense.h5")} + response = _response(paths) + payload = json.loads(response.content) + payload["build"]["built_at"] = "the day before the Budget" + response.content = json.dumps(payload).encode() + run = FakeRunner() + with pytest.raises(ValueError, match="built_at"): + _measure(model, paths, response, run=run) + assert run.calls == [] + + +@pytest.mark.parametrize( + "resolved", + [{"model_version": "2.132.7"}, {"core_version": "3.33.4"}], + ids=["model", "core"], +) +def test_a_build_model_resolved_at_other_versions_refuses(tmp_path, model, resolved): + paths = {"dense": _h5(tmp_path / "dense.h5")} + with pytest.raises(ValueError, match="resolved"): + _measure(model, paths, run=FakeRunner(**resolved)) + + +@pytest.mark.parametrize( + "unrecorded", ["built_with_model_package", "built_with_core_package", "build"] +) +def test_an_unrecorded_build_version_leaves_no_baseline_and_refuses( + tmp_path, model, unrecorded +): + paths = {"dense": _h5(tmp_path / "dense.h5")} + response = _response(paths) + payload = json.loads(response.content) + if unrecorded == "build": + del payload["build"] + else: + del payload["build"][unrecorded] + response.content = json.dumps(payload).encode() + run = FakeRunner() + with pytest.raises(ValueError, match="baseline"): + _measure(model, paths, response, run=run) + assert run.calls == [] + + +def test_a_failed_build_model_probe_refuses(tmp_path, model): + paths = {"dense": _h5(tmp_path / "dense.h5")} + + def unresolvable(command, **kwargs): + raise subprocess.CalledProcessError( + 1, command, stderr="No solution found when resolving --with dependencies" + ) + + with pytest.raises(ValueError, match="build model.*No solution"): + _measure(model, paths, run=unresolvable) + + +@pytest.mark.parametrize( + ("audited_by", "limitations"), + [("\u200b", [LIMITATION]), ("Budget operator", ["\u200b"]), (" ", [LIMITATION])], +) +def test_an_invisible_auditor_or_limitation_refuses( + tmp_path, model, audited_by, limitations +): + paths = {"dense": _h5(tmp_path / "dense.h5")} + with patch( + "policyengine.provenance.certification.requests.get", + return_value=_response(paths), + ): + with pytest.raises(ValueError, match="a reader can see"): + model.measure_regional_input_surface( + URI, paths, MODEL_VERSION, audited_by, limitations, run=FakeRunner() + ) + + +def test_installed_model_must_be_the_requested_exact_version( + tmp_path, model, monkeypatch +): + monkeypatch.setattr(model, "version", lambda name: "2.146.4") + paths = {"dense": _h5(tmp_path / "dense.h5")} + with pytest.raises(ValueError, match="2.147.2|2.146.4"): + _measure(model, paths) + + +def test_artifact_bytes_must_match_the_manifest(tmp_path, model): + paths = {"dense": _h5(tmp_path / "dense.h5")} + response = _response(paths) + payload = json.loads(response.content) + payload["artifacts"]["dense"]["sha256"] = "0" * 64 + response.content = json.dumps(payload).encode() + with pytest.raises(ValueError, match="sha256|[Hh]ash|bytes"): + _measure(model, paths, response) + + +def test_file_changing_during_metadata_read_refuses(tmp_path, model, monkeypatch): + paths = {"dense": _h5(tmp_path / "dense.h5")} + response = _response(paths) + original = pd.HDFStore.select + mutated = False + + def select_and_modify(store, *args, **kwargs): + nonlocal mutated + result = original(store, *args, **kwargs) + if not mutated and args[0] != "/time_period": + with paths["dense"].open("ab") as file: + file.write(b"changed while inspecting metadata") + mutated = True + return result + + monkeypatch.setattr(pd.HDFStore, "select", select_and_modify) + with pytest.raises(ValueError, match="chang|sha256|[Hh]ash|bytes"): + _measure(model, paths, response) + + +@pytest.mark.parametrize("names", [[], ["other"], ["dense", "other"]]) +def test_every_routed_h5_requires_exactly_one_supplied_path(tmp_path, model, names): + path = _h5(tmp_path / "dense.h5") + response = _response({"dense": path}) + with pytest.raises(ValueError, match="artifact|[Hh]5"): + _measure(model, {name: path for name in names}, response) + + +def test_comparison_is_relative_to_the_build_model(): + result = _module().compare_input_surface( + {"dense": _artifact({"2025": ["weight", "area_code", "removed"]})}, + ["weight", "new_input", "old_input", "was_formula"], + ["weight", "new_input", "old_input", "was_formula"], + build_model_variables=["weight", "old_input", "was_formula", "removed"], + build_model_inputs=["weight", "old_input", "removed"], + ) + assert result == { + "dropped_columns": ["removed"], + "never_modelled_columns": ["area_code"], + "defaulted_inputs": ["new_input", "was_formula"], + } + + +def test_comparison_uses_each_artifact_and_every_period(): + artifact = _artifact( + {"2025": ["income", "weight", "new_input"], "2026": ["income", "weight"]} + ) + result = _module().compare_input_surface( + {"dense": artifact}, + ["income", "income_reported", "weight", "new_input"], + ["income_reported", "weight", "new_input"], + {"income": "income_reported"}, + build_model_variables=["income", "income_reported", "weight"], + build_model_inputs=["income_reported", "weight"], + ) + assert result == { + "dropped_columns": [], + "never_modelled_columns": [], + "defaulted_inputs": ["new_input"], + } + + +def test_comparison_refuses_a_period_union_that_hides_columns(): + artifact = RegionalInputArtifactEvidence( + repo_id="r", + path="dense.h5", + revision="cut", + sha256="a" * 64, + columns=["weight"], + columns_by_period={"2025": ["weight", "removed"]}, + ) + with pytest.raises(ValueError, match="columns|union|period"): + _module().compare_input_surface( + {"dense": artifact}, + ["weight"], + ["weight"], + build_model_variables=["weight"], + build_model_inputs=["weight"], + ) + + +def test_comparison_refuses_build_inputs_outside_the_build_variables(): + with pytest.raises(ValueError, match="build_model_inputs"): + _module().compare_input_surface( + {"dense": _artifact({"2025": ["weight"]})}, + ["weight"], + ["weight"], + build_model_variables=["weight"], + build_model_inputs=["weight", "unknown"], + ) + + +def test_conditional_alias_cannot_silently_claim_a_supplied_input( + tmp_path, model, monkeypatch +): + class ConditionalSimulation: + def __init__(self): + if self.condition: + self.move_values("employment_income", "employment_income_before_lsr") + + monkeypatch.setattr( + sys.modules["policyengine_uk"], "Simulation", ConditionalSimulation + ) + paths = {"dense": _h5(tmp_path / "dense.h5")} + with pytest.raises(ValueError, match="alias|move_values|conditional"): + _measure(model, paths) + + +def _manifest_with_build_time(): + from policyengine.provenance.manifest import DataReleaseManifest + + return DataReleaseManifest.model_validate( + { + "schema_version": 1, + "data_package": {"name": "microcosm-data", "version": "1"}, + "build": {"built_at": BUILT_AT}, + } + ) + + +def test_certification_rereads_both_models_and_accepts_their_own_surfaces( + tmp_path, model +): + evidence = _measure(model, {"dense": _h5(tmp_path / "dense.h5")}) + run = FakeRunner() + model.verify_model_surfaces(evidence, _manifest_with_build_time(), run=run) + ((command, _),) = run.calls + assert f"policyengine-uk=={BUILD_VERSION}" in command + assert command[command.index("--exclude-newer") + 1] == BUILD_CUTOFF + + +@pytest.mark.parametrize( + ("field", "edit"), + [ + # A stored column the build model defines, claimed as never modelled. + ("build_model_variables", lambda names: [n for n in names if n != REMOVED]), + # The same column claimed as a target variable, so not dropped. + ("model_variables", lambda names: sorted([*names, REMOVED])), + # New defaults claimed as build inputs, so not defaulted. + ("build_model_inputs", lambda names: sorted([*names, *DEFAULTS])), + ("model_inputs", lambda names: names[1:]), + ], +) +def test_an_edited_model_surface_refuses_at_certification(tmp_path, model, field, edit): + evidence = _measure(model, {"dense": _h5(tmp_path / "dense.h5")}) + edited = evidence.model_copy(update={field: edit(getattr(evidence, field))}) + with pytest.raises(ValueError, match=field): + model.verify_model_surfaces( + edited, _manifest_with_build_time(), run=FakeRunner() + ) + + +def test_edited_input_aliases_refuse_at_certification(tmp_path, model): + evidence = _measure(model, {"dense": _h5(tmp_path / "dense.h5")}) + edited = evidence.model_copy(update={"input_aliases": {}}) + with pytest.raises(ValueError, match="input_aliases"): + model.verify_model_surfaces( + edited, _manifest_with_build_time(), run=FakeRunner() + ) + + +def test_certification_needs_the_target_model_installed(tmp_path, model, monkeypatch): + evidence = _measure(model, {"dense": _h5(tmp_path / "dense.h5")}) + monkeypatch.setattr(model, "version", lambda name: BUILD_VERSION) + with pytest.raises(ValueError, match=f"{BUILD_VERSION}.*{MODEL_VERSION}"): + model.verify_model_surfaces( + evidence, _manifest_with_build_time(), run=FakeRunner() + ) + + +def test_the_build_model_probe_reads_the_model_api_the_audit_uses(): + """Run the probe with this interpreter, in place of a uv environment.""" + pytest.importorskip("policyengine_uk") + module = _module() + model_version = importlib.metadata.version("policyengine-uk") + core_version = importlib.metadata.version("policyengine-core") + + def run_here(command, **kwargs): + python = command.index("python") + return subprocess.run([sys.executable, *command[python + 1 :]], **kwargs) + + surface = module._build_model_surface(model_version, core_version, run=run_here) + variables, inputs, _ = module._model_surface(model_version) + assert surface == (variables, inputs) + + +def test_audit_cli_writes_pydantic_evidence_json(tmp_path, model, monkeypatch, capsys): + script_path = ( + Path(__file__).parents[1] / "scripts" / "audit_regional_input_surface.py" + ) + spec = importlib.util.spec_from_file_location( + "audit_regional_input_surface", script_path + ) + script = importlib.util.module_from_spec(spec) + spec.loader.exec_module(script) + path = _h5(tmp_path / "dense.h5", person=(*PERSON, AREA_CODE)) + result = _measure(model, {"dense": path}) + captured = {} + + def measure(**kwargs): + captured.update(kwargs) + return result + + monkeypatch.setattr(script, "measure_regional_input_surface", measure) + output = tmp_path / "evidence.json" + assert ( + script.main( + [ + "--regional-manifest-uri", + URI, + "--artifact", + f"dense={path}", + "--model-version", + MODEL_VERSION, + "--audited-by", + "Budget operator", + "--limitation", + LIMITATION, + "--output", + str(output), + ] + ) + == 0 + ) + assert captured["artifact_paths"] == {"dense": path} + assert captured["limitations"] == [LIMITATION] + assert json.loads(output.read_text()) == result.model_dump(mode="json") + assert "0 dropped, 1 never-modelled, 2 defaulted" in capsys.readouterr().out + + +def test_audit_cli_bootstraps_before_target_model_is_certified(tmp_path): + """The target must be auditable before it matches the current bundle.""" + script = Path(__file__).parents[1] / "scripts" / "audit_regional_input_surface.py" + code = """ +import importlib.util +import os +import runpy +import sys +os.environ.pop('POLICYENGINE_SKIP_COUNTRY_IMPORTS', None) +original = importlib.util.find_spec +def refuse_country_autoload(name, *args, **kwargs): + if name in {'policyengine_uk', 'policyengine_us'}: + raise RuntimeError('audit triggered automatic country import') + return original(name, *args, **kwargs) +importlib.util.find_spec = refuse_country_autoload +sys.argv = [sys.argv[1], '--help'] +runpy.run_path(sys.argv[0], run_name='__main__') +""" + result = subprocess.run( + [sys.executable, "-c", code, str(script)], + capture_output=True, + text=True, + cwd=tmp_path, + ) + assert result.returncode == 0, result.stderr + assert "--model-version" in result.stdout diff --git a/tests/test_regional_input_evidence.py b/tests/test_regional_input_evidence.py new file mode 100644 index 00000000..af8e3bf2 --- /dev/null +++ b/tests/test_regional_input_evidence.py @@ -0,0 +1,614 @@ +"""A regional model exception requires an audit of the exact dense bytes.""" + +import copy +import hashlib +import json + +import pytest + +from policyengine.provenance.certification import CertificationError +from policyengine.provenance.manifest import CountryReleaseManifest +from tests.test_certify_data_release import ( + UK_DENSE_URI, + UK_MODEL_VERSION, + _certify_uk, + _microcosm_uk_dense_payload, + _microcosm_uk_national_payload, + _response, + _tmp_bundle, +) + +REASON = "Budget accepts the audited dense model gap; area runs remain uncertified" +VAT_LIMIT = ( + "the dense cut's VAT predates the policyengine-uk VAT factor fix " + "(#2187, 2.135.7); the dashboard holds a VAT limit" +) +DENSE = "microcosm_uk_2024_25_dense" +BUILD_CORE_VERSION = "3.33.3" +DEFAULTED_INPUTS = [ + "local_bus_uncapped_single_fare", + "salary_sacrifice_returned_to_income", +] +#: What certification adds to the limitations for input_evidence's defaults. +DEFAULTS_LIMIT = ( + f"area runs on policyengine-uk {UK_MODEL_VERSION} default 2 inputs that the " + "regional file does not store and its build model, policyengine-uk 2.120.0, " + "did not take as inputs, so area results for measures that read them use " + "those defaults: local_bus_uncapped_single_fare, " + "salary_sacrifice_returned_to_income" +) + + +def older_dense() -> dict: + dense = _microcosm_uk_dense_payload() + dense["build"]["built_with_model_package"]["version"] = "2.120.0" + dense["build"]["built_with_core_package"] = { + "name": "policyengine-core", + "version": BUILD_CORE_VERSION, + } + dense["compatible_model_packages"] = [ + {"name": "policyengine-uk", "specifier": "==2.120.0"} + ] + return dense + + +def manifest_sha256(dense: dict) -> str: + """The digest of the bytes ``_response`` serves for ``dense``.""" + return hashlib.sha256(json.dumps(dense).encode()).hexdigest() + + +def input_evidence(dense: dict | None = None) -> dict: + """An audit whose build model lacks the two inputs the target adds.""" + dense = older_dense() if dense is None else dense + artifact = dense["artifacts"][DENSE] + build = dense["build"] + return { + "schema_version": 2, + "regional_manifest_uri": UK_DENSE_URI, + "regional_manifest_sha256": manifest_sha256(dense), + "model_package": "policyengine-uk", + "model_version": UK_MODEL_VERSION, + "build_model_version": build["built_with_model_package"]["version"], + "build_core_version": build["built_with_core_package"]["version"], + "artifacts": { + DENSE: { + key: artifact[key] for key in ("repo_id", "path", "revision", "sha256") + } + | { + "columns": ["employment_income"], + "columns_by_period": {"2025": ["employment_income"]}, + } + }, + "model_variables": ["employment_income", *DEFAULTED_INPUTS], + "model_inputs": ["employment_income", *DEFAULTED_INPUTS], + "build_model_variables": ["employment_income"], + "build_model_inputs": ["employment_income"], + "dropped_columns": [], + "never_modelled_columns": [], + "defaulted_inputs": DEFAULTED_INPUTS.copy(), + "audited_by": "Budget regional input-surface review", + "limitations": [VAT_LIMIT], + } + + +def store_column(evidence: dict, column: str) -> None: + measured = evidence["artifacts"][DENSE] + measured["columns"] = sorted([*measured["columns"], column]) + measured["columns_by_period"]["2025"] = measured["columns"] + + +def fake_model_surfaces(monkeypatch) -> dict: + """Stand in for the two models certification re-reads, installing neither. + + The surfaces match ``input_evidence``. Returns them, for a test to change + what a model reports, and the reads made. + """ + from policyengine.provenance import regional_input_surface + + surfaces: dict = { + "target": { + "variables": ["employment_income", *DEFAULTED_INPUTS], + "inputs": ["employment_income", *DEFAULTED_INPUTS], + "aliases": {}, + }, + "build": {"variables": ["employment_income"], "inputs": ["employment_income"]}, + "reads": [], + } + + def target_surface(model_version): + surfaces["reads"].append(("target", model_version)) + target = surfaces["target"] + return sorted(target["variables"]), sorted(target["inputs"]), target["aliases"] + + def build_surface(model_version, core_version, run=None, exclude_newer=None): + surfaces["reads"].append(("build", model_version, core_version, exclude_newer)) + build = surfaces["build"] + return sorted(build["variables"]), sorted(build["inputs"]) + + monkeypatch.setattr(regional_input_surface, "_model_surface", target_surface) + monkeypatch.setattr(regional_input_surface, "_build_model_surface", build_surface) + return surfaces + + +@pytest.fixture(autouse=True) +def model_surfaces(monkeypatch): + return fake_model_surfaces(monkeypatch) + + +def certify_with_evidence(bundle_path, evidence: dict, dense: dict | None = None): + dense = older_dense() if dense is None else dense + return _certify_uk( + bundle_path, + [_response(_microcosm_uk_national_payload()), _response(dense)], + regional_manifest_uri=UK_DENSE_URI, + regional_model_mismatch_reason=REASON, + regional_input_surface_evidence=evidence, + ) + + +def test_accepted_mismatch_without_audited_evidence_refuses(tmp_path): + with pytest.raises(CertificationError, match="[Ee]vidence|[Aa]udit"): + _certify_uk( + _tmp_bundle(tmp_path), + [ + _response(_microcosm_uk_national_payload()), + _response(older_dense()), + ], + regional_manifest_uri=UK_DENSE_URI, + regional_model_mismatch_reason=REASON, + ) + + +@pytest.mark.parametrize("model_package", ["policyengine-us", "another-model"]) +@pytest.mark.parametrize("accept_mismatch", [False, True]) +def test_regional_release_built_with_another_model_package_refuses( + tmp_path, model_package, accept_mismatch +): + dense = older_dense() + dense["build"]["built_with_model_package"]["name"] = model_package + if not accept_mismatch: + dense["compatible_model_packages"] = [ + {"name": "policyengine-uk", "specifier": f"=={UK_MODEL_VERSION}"} + ] + bundle_path = _tmp_bundle(tmp_path) + before = bundle_path.read_bytes() + + with pytest.raises( + CertificationError, + match=f"built with {model_package}, not policyengine-uk", + ): + if accept_mismatch: + certify_with_evidence(bundle_path, input_evidence(dense), dense) + else: + _certify_uk( + bundle_path, + [_response(_microcosm_uk_national_payload()), _response(dense)], + regional_manifest_uri=UK_DENSE_URI, + ) + + assert bundle_path.read_bytes() == before + + +@pytest.mark.parametrize( + ("field", "wrong"), + [ + ("regional_manifest_uri", UK_DENSE_URI.replace("dense", "other-dense")), + ("regional_manifest_sha256", "0" * 64), + ("model_package", "policyengine-us"), + ("model_version", "2.147.1"), + ], +) +def test_evidence_for_another_cut_bytes_or_model_refuses(tmp_path, field, wrong): + evidence = input_evidence() + evidence[field] = wrong + with pytest.raises(CertificationError): + certify_with_evidence(_tmp_bundle(tmp_path), evidence) + + +@pytest.mark.parametrize( + ("field", "wrong"), + [ + ("repo_id", "policyengine/another-dense"), + ("path", "different_dense.h5"), + ("revision", "different-cut"), + ("sha256", "0" * 64), + ], +) +def test_evidence_for_other_artifact_bytes_or_identity_refuses(tmp_path, field, wrong): + evidence = input_evidence() + evidence["artifacts"][DENSE][field] = wrong + with pytest.raises(CertificationError): + certify_with_evidence(_tmp_bundle(tmp_path), evidence) + + +def test_a_dropped_column_refuses_even_with_an_acceptance_reason( + tmp_path, model_surfaces +): + """The build model defines the stored column and the target does not.""" + evidence = input_evidence() + store_column(evidence, "removed_model_variable") + for field, surface in ( + ("build_model_variables", "variables"), + ("build_model_inputs", "inputs"), + ): + evidence[field] = sorted([*evidence[field], "removed_model_variable"]) + model_surfaces["build"][surface].append("removed_model_variable") + evidence["dropped_columns"] = ["removed_model_variable"] + with pytest.raises(CertificationError, match="[Dd]ropped.*removed_model_variable"): + certify_with_evidence(_tmp_bundle(tmp_path), evidence) + + +@pytest.mark.parametrize( + ("claim", "refused_field"), + [ + ("never_modelled", "build_model_variables"), + ("a_target_variable", "model_variables"), + ], +) +def test_an_edited_evidence_file_cannot_hide_a_dropped_column( + tmp_path, model_surfaces, claim, refused_field +): + """The lists in the file are checked against the models, not just each other.""" + model_surfaces["build"]["variables"].append("removed_model_variable") + evidence = input_evidence() + store_column(evidence, "removed_model_variable") + if claim == "never_modelled": + evidence["never_modelled_columns"] = ["removed_model_variable"] + else: + for field in ("build_model_variables", "model_variables"): + evidence[field] = sorted([*evidence[field], "removed_model_variable"]) + bundle_path = _tmp_bundle(tmp_path) + before = bundle_path.read_bytes() + with pytest.raises(CertificationError, match=f"re-reads.*{refused_field}"): + certify_with_evidence(bundle_path, evidence) + assert bundle_path.read_bytes() == before + + +def test_an_edited_evidence_file_cannot_hide_a_default(tmp_path): + evidence = input_evidence() + for field in ("build_model_variables", "build_model_inputs"): + evidence[field] = sorted([*evidence[field], *DEFAULTED_INPUTS]) + evidence["defaulted_inputs"] = [] + with pytest.raises(CertificationError, match="re-reads.*build_model"): + certify_with_evidence(_tmp_bundle(tmp_path), evidence) + + +def test_certification_reads_the_build_model_as_of_the_regional_build( + tmp_path, model_surfaces +): + dense = older_dense() + dense["build"]["built_at"] = "2026-10-19T02:07:51.099627+00:00" + certify_with_evidence(_tmp_bundle(tmp_path), input_evidence(dense), dense) + assert model_surfaces["reads"] == [ + ("target", UK_MODEL_VERSION), + ("build", "2.120.0", BUILD_CORE_VERSION, "2026-10-19T02:07:52Z"), + ] + + +def test_certification_refuses_without_the_target_model_installed( + tmp_path, monkeypatch +): + from policyengine.provenance import regional_input_surface + + def not_installed(model_version): + raise regional_input_surface.RegionalInputSurfaceError( + f"Installed policyengine-uk 2.132.6 differs from requested {model_version}." + ) + + monkeypatch.setattr(regional_input_surface, "_model_surface", not_installed) + with pytest.raises(CertificationError, match="re-reads.*2.132.6 differs"): + certify_with_evidence(_tmp_bundle(tmp_path), input_evidence()) + + +def test_a_column_neither_model_defines_is_recorded_not_refused(tmp_path): + evidence = input_evidence() + store_column(evidence, "constituency_code_oa") + evidence["never_modelled_columns"] = ["constituency_code_oa"] + bundle_path = _tmp_bundle(tmp_path) + + result = certify_with_evidence(bundle_path, evidence) + + release = json.loads(bundle_path.read_text())["data_releases"]["uk"] + record = release["regional_certification"] + assert record["never_modelled_columns"] == ["constituency_code_oa"] + assert record["dropped_columns"] == [] + assert record["evidence"] == evidence + (warning,) = [w for w in result.warnings if "accepted mismatch" in w] + assert "never-modelled columns (stored, defined by neither model): 1" in warning + + +def test_an_input_absent_under_both_models_is_not_reported(tmp_path, model_surfaces): + evidence = input_evidence() + for field in ( + "model_variables", + "model_inputs", + "build_model_variables", + "build_model_inputs", + ): + evidence[field] = sorted([*evidence[field], "legacy_input"]) + for model in ("target", "build"): + for surface in ("variables", "inputs"): + model_surfaces[model][surface].append("legacy_input") + bundle_path = _tmp_bundle(tmp_path) + certify_with_evidence(bundle_path, evidence) + record = json.loads(bundle_path.read_text())["data_releases"]["uk"][ + "regional_certification" + ] + assert record["defaulted_inputs"] == DEFAULTED_INPUTS + + evidence["defaulted_inputs"] = sorted([*DEFAULTED_INPUTS, "legacy_input"]) + with pytest.raises(CertificationError, match="disagrees"): + certify_with_evidence(_tmp_bundle(tmp_path), evidence) + + +@pytest.mark.parametrize( + ("field", "wrong"), + [("build_model_version", "2.120.1"), ("build_core_version", "3.33.4")], +) +def test_evidence_against_another_build_model_refuses(tmp_path, field, wrong): + evidence = input_evidence() + evidence[field] = wrong + with pytest.raises(CertificationError, match="build model"): + certify_with_evidence(_tmp_bundle(tmp_path), evidence) + + +@pytest.mark.parametrize( + "unrecorded", ["built_with_model_package", "built_with_core_package"] +) +def test_a_regional_release_without_build_versions_has_no_baseline( + tmp_path, unrecorded +): + dense = older_dense() + evidence = input_evidence(dense) + del dense["build"][unrecorded] + evidence["regional_manifest_sha256"] = manifest_sha256(dense) + with pytest.raises(CertificationError, match="baseline"): + certify_with_evidence(_tmp_bundle(tmp_path), evidence, dense) + + +@pytest.mark.parametrize("version", [1, None], ids=["version_1", "unversioned"]) +def test_evidence_must_declare_schema_version_2(tmp_path, version): + evidence = input_evidence() + if version is None: + del evidence["schema_version"] + else: + evidence["schema_version"] = version + with pytest.raises(CertificationError, match="schema_version"): + certify_with_evidence(_tmp_bundle(tmp_path), evidence) + + +@pytest.mark.parametrize("field", ["audited_by", "limitations"]) +def test_invisible_auditor_or_limitations_refuse(tmp_path, field): + evidence = input_evidence() + evidence[field] = "\u200b" if field == "audited_by" else ["\u200b"] + with pytest.raises(CertificationError, match="a reader can see"): + certify_with_evidence(_tmp_bundle(tmp_path), evidence) + + +@pytest.mark.parametrize("reason", [" ", "\t\n"]) +def test_a_bundle_record_with_a_blank_reason_does_not_validate(tmp_path, reason): + from pydantic import ValidationError + + bundle_path = _tmp_bundle(tmp_path) + certify_with_evidence(bundle_path, input_evidence()) + release = json.loads(bundle_path.read_text())["data_releases"]["uk"] + release["regional_certification"]["accepted_reason"] = reason + with pytest.raises(ValidationError, match="accepted_reason"): + CountryReleaseManifest.model_validate(release) + + +def test_an_h5_suffix_in_any_case_must_be_measured_as_the_audit_does(): + from policyengine.provenance.certification import accepted_regional_mismatch + from policyengine.provenance.manifest import DataReleaseManifest + + dense = older_dense() + dense["artifacts"][DENSE]["path"] = "microcosm_uk_2024_25_dense.H5" + evidence = input_evidence(dense) + manifest = DataReleaseManifest.model_validate(dense) + + def accept(evidence): + return accepted_regional_mismatch( + manifest, + "policyengine-uk", + UK_MODEL_VERSION, + REASON, + regional_manifest_uri=UK_DENSE_URI, + regional_manifest_sha256=manifest_sha256(dense), + evidence=evidence, + ) + + record, _ = accept(evidence) + assert set(record["evidence"]["artifacts"]) == {DENSE} + evidence["artifacts"] = {"another_file": evidence["artifacts"][DENSE]} + with pytest.raises(CertificationError, match="every regional microdata"): + accept(evidence) + + +@pytest.mark.parametrize( + "field", ["dropped_columns", "never_modelled_columns", "defaulted_inputs"] +) +def test_the_declared_comparison_must_match_the_measured_surfaces(tmp_path, field): + evidence = input_evidence() + evidence[field] = ["invented_variable"] + with pytest.raises(CertificationError): + certify_with_evidence(_tmp_bundle(tmp_path), evidence) + + +def test_a_missing_artifact_measurement_refuses(tmp_path): + evidence = input_evidence() + evidence["artifacts"] = {} + with pytest.raises(CertificationError): + certify_with_evidence(_tmp_bundle(tmp_path), evidence) + + +def test_every_default_and_limitation_survives_validation_and_roundtrip(tmp_path): + bundle_path = _tmp_bundle(tmp_path) + evidence = input_evidence() + certify_with_evidence(bundle_path, evidence) + + release = json.loads(bundle_path.read_text())["data_releases"]["uk"] + manifest = CountryReleaseManifest.model_validate(release) + assert manifest.regional_certification is not None + record = manifest.regional_certification.model_dump(mode="json", exclude_none=True) + assert record["defaulted_inputs"] == DEFAULTED_INPUTS + assert record["dropped_columns"] == [] + assert record["limitations"] == [VAT_LIMIT, DEFAULTS_LIMIT] + assert record["evidence"] == evidence + assert record["area_receipts"] == "uncertified" + assert record["accepted_reason"] == REASON + assert ( + release["regional_source_manifest_sha256"] + == evidence["regional_manifest_sha256"] + ) + rebuilt = CountryReleaseManifest.model_validate_json(manifest.model_dump_json()) + assert rebuilt.regional_certification == manifest.regional_certification + assert release["certification"]["compatibility_basis"] == "built_with_model_package" + + +def test_certification_names_the_defaulted_inputs_in_the_limitations(tmp_path): + """Review of #699: a reader of the limitations alone sees the defaults.""" + evidence = input_evidence() + bundle_path = _tmp_bundle(tmp_path) + result = certify_with_evidence(bundle_path, evidence) + record = json.loads(bundle_path.read_text())["data_releases"]["uk"][ + "regional_certification" + ] + assert record["limitations"] == [VAT_LIMIT, DEFAULTS_LIMIT] + assert record["evidence"]["limitations"] == [VAT_LIMIT] + (warning,) = [w for w in result.warnings if "accepted mismatch" in w] + assert "inputs defaulted since the build model: 2." in warning + assert DEFAULTS_LIMIT in warning + + +def test_the_defaults_limitation_is_not_repeated(tmp_path): + evidence = input_evidence() + evidence["limitations"].append(DEFAULTS_LIMIT) + bundle_path = _tmp_bundle(tmp_path) + certify_with_evidence(bundle_path, evidence) + record = json.loads(bundle_path.read_text())["data_releases"]["uk"][ + "regional_certification" + ] + assert record["limitations"] == [VAT_LIMIT, DEFAULTS_LIMIT] + + +def test_no_defaults_limitation_when_nothing_defaults(tmp_path, model_surfaces): + for surface in ("variables", "inputs"): + model_surfaces["target"][surface] = ["employment_income"] + evidence = input_evidence() + evidence["model_variables"] = evidence["model_inputs"] = ["employment_income"] + evidence["defaulted_inputs"] = [] + bundle_path = _tmp_bundle(tmp_path) + certify_with_evidence(bundle_path, evidence) + record = json.loads(bundle_path.read_text())["data_releases"]["uk"][ + "regional_certification" + ] + assert record["limitations"] == [VAT_LIMIT] + + +def test_evidence_digest_binds_original_manifest_bytes(tmp_path): + dense = older_dense() + response = _response(dense) + response.content = json.dumps(dense, indent=2).encode() + evidence = input_evidence(dense) + with pytest.raises(CertificationError): + _certify_uk( + _tmp_bundle(tmp_path), + [_response(_microcosm_uk_national_payload()), response], + regional_manifest_uri=UK_DENSE_URI, + regional_model_mismatch_reason=REASON, + regional_input_surface_evidence=evidence, + ) + + evidence["regional_manifest_sha256"] = hashlib.sha256(response.content).hexdigest() + bundle_path = _tmp_bundle(tmp_path) + _certify_uk( + bundle_path, + [_response(_microcosm_uk_national_payload()), response], + regional_manifest_uri=UK_DENSE_URI, + regional_model_mismatch_reason=REASON, + regional_input_surface_evidence=evidence, + ) + release = json.loads(bundle_path.read_text())["data_releases"]["uk"] + assert ( + release["regional_source_manifest_sha256"] + == evidence["regional_manifest_sha256"] + ) + + +def test_drop_regional_release_removes_acceptance_and_preserves_national_cert(tmp_path): + bundle_path = _tmp_bundle(tmp_path) + certify_with_evidence(bundle_path, input_evidence()) + before = json.loads(bundle_path.read_text())["data_releases"]["uk"] + _certify_uk( + bundle_path, + [_response(_microcosm_uk_national_payload())], + drop_regional_release=True, + ) + after = json.loads(bundle_path.read_text())["data_releases"]["uk"] + assert after["certification"] == before["certification"] + assert "regional_certification" not in after + assert "regional_source_manifest_sha256" not in after + assert "regional_source_manifest_uri" not in after + assert "regional_release_kind" not in after + assert "microcosm_uk_2024_25_dense" not in after["datasets"] + assert set(after["region_datasets"]) == {"national"} + + +def test_limitations_and_evidence_are_in_hashed_bundle_material(tmp_path): + from policyengine.provenance.canonical import canonical_sha256 + + bundle_path = _tmp_bundle(tmp_path) + certify_with_evidence(bundle_path, input_evidence()) + before = json.loads(bundle_path.read_text()) + after = copy.deepcopy(before) + after["data_releases"]["uk"]["regional_certification"]["limitations"].append( + "An additional reviewed regional limitation" + ) + + def digest(bundle): + manifest = CountryReleaseManifest.model_validate(bundle["data_releases"]["uk"]) + return canonical_sha256(manifest.model_dump(mode="json")) + + assert digest(before) != digest(after) + + +def test_missing_period_measurements_refuse(tmp_path): + evidence = input_evidence() + del evidence["artifacts"][DENSE]["columns_by_period"] + with pytest.raises(CertificationError): + certify_with_evidence(_tmp_bundle(tmp_path), evidence) + + +def test_budget_target_records_vat_limit_and_both_named_defaults(tmp_path, monkeypatch): + from tests import test_certify_data_release as fixtures + + monkeypatch.setattr(fixtures, "UK_MODEL_VERSION", "2.147.2") + dense = older_dense() + dense["build"]["built_with_model_package"]["version"] = "2.132.6" + dense["compatible_model_packages"][0]["specifier"] = "==2.132.6" + audit = input_evidence(dense) + audit["model_version"] = "2.147.2" + audit["limitations"] = ["The dense cut was calibrated under an older model."] + path = _tmp_bundle(tmp_path) + certify_with_evidence(path, audit, dense) + release = json.loads(path.read_text())["data_releases"]["uk"] + record = release["regional_certification"] + assert record["certified_for_model_version"] == "2.147.2" + assert record["built_with_model_version"] == "2.132.6" + assert VAT_LIMIT in record["limitations"] + assert set(DEFAULTED_INPUTS) <= set(record["defaulted_inputs"]) + assert release["certification"]["compatibility_basis"] == "built_with_model_package" + + +def test_regional_acceptance_never_widens_national_coverage(tmp_path): + national = _microcosm_uk_national_payload() + national["build"]["built_with_model_package"]["version"] = "2.120.0" + national["compatible_model_packages"][0]["specifier"] = "==2.120.0" + with pytest.raises(CertificationError, match="new data build|compatibility claim"): + _certify_uk( + _tmp_bundle(tmp_path), + [_response(national), _response(older_dense())], + regional_manifest_uri=UK_DENSE_URI, + regional_model_mismatch_reason=REASON, + regional_input_surface_evidence=input_evidence(), + )