UK: constituency and local-authority runs use enhanced-FRS weight matrices, and every UK run requires la_code_oa; both break on a Microcosm UK default #725

Summary

policyengine.py plans to replace the UK default, the enhanced FRS 1.56.16, with a Microcosm national release. Constituency and local-authority runs would move to a separate Microcosm local-area dataset (PolicyEngine/policyengine.py#553, PolicyEngine/microcosm#1114). Two parts of the executor assume the enhanced FRS and would fail on that default. The second blocks every UK run, not only local ones. Line numbers below are at d1d4834.

1. Constituency and local-authority runs use enhanced-FRS weight matrices

• How the fallback is reached: policyengine.py's default UK registry has no constituency or local-authority regions; build_uk_region_registry() loads them only on request. So _resolve_region (simulation_runtime.py:317) gets None from get_region and falls back to _build_uk_weight_replacement_region (simulation_runtime.py:241).
• What the fallback builds: a WeightReplacementStrategy over parliamentary_constituency_weights.h5 or local_authority_weights.h5. These are looked up through constituencies_2024.csv or local_authorities_2021.csv in the policyengine-uk-data-private GCS bucket. Both runtime paths (simulation_runtime.py:588, stage12_runtime/simulation.py:157) and precompute.py go through it.
• Why it fails on another dataset: the matrices have one row per area and one column per enhanced-FRS household. The certified 1.56.16 copies on Hugging Face are 650 × 52,846 for constituencies and 360 × 52,846 for local authorities. WeightReplacementStrategy.apply raises Weight matrix row length (…) does not match household count (…) on any dataset of a different size. The published Microcosm national file has 63,827 households, so every constituency and local-authority run would fail on it.
• Worth checking: the 1.56.16 copies on Hugging Face hold a single 2025 key, and apply reads f[str(year)]. I could not read the GCS copies the executor downloads, so check that runs for other years find their key.

2. Every UK run requires la_code_oa

• Per-run check: detect_uk_local_authority_metadata (uk_local_authority_metadata.py:280) runs on every UK dataset right after loading (simulation_runtime.py:608, stage12_runtime/simulation.py:183). It raises UK dataset household table contains no la_code_oa column when the column is missing.
• Start-up check: validate_country_worker runs the same check on the installed UK default (_check_uk_local_authority_dataset, stage12_worker_validation.py:80).
• Effect on a Microcosm default: the Microcosm national file carries region and no area codes. With it as the default, every UK run fails at dataset load and UK workers fail validation.
• Geographic reports: they also copy constituency_code_oa and la_code_oa into their outputs (UK_GEOGRAPHIC_DATASET_VARIABLES, stage12_runtime/output_planning.py:26).

What already works for a Microcosm local-area file

• Boundary detection: with the aliases from UK local-area exports: add the constituency_code_oa, la_code_oa and region_code_oa columns that policyengine.py and the simulation API read, and record the area-code frames in the release manifest microcosm#1114, the local file carries la_code_oa on the April 2023 frame. detect_uk_local_authority_boundary_version identifies that frame as LAD23. The packaged coordinates_lad23.csv has all 361 codes, Ards and North Down included.
• Local outputs: build_uk_constituency_impact and build_uk_local_authority_impact already group by constituency_code_oa and la_code_oa, so they work on any file that carries those columns.
• Dataset pins: runtime_dataset_uri keeps an explicit revision for repositories outside the GCS-mirrored policyengine-*-data repos. A Microcosm dataset in populace-uk-private therefore keeps its own pin.

Proposed changes

1. Make the local-authority check conditional.
   • detect_uk_local_authority_metadata returns None when the dataset has no la_code_oa.
   • The worker check accepts a default without area codes.
   • Local-authority outputs and scoped runs fail with a clear message only when they are requested on such a dataset.
   • This is needed before any policyengine.py release with a Microcosm UK default reaches the executor, so it gates the next policyengine[models] pin bump (currently ==6.2.1 in projects/policyengine-simulation-executor/pyproject.toml).
2. Use policyengine.py's constituency and local-authority regions instead of building weight-replacement ones.
   • UK: certify a Microcosm national dataset as the UK default, with separate national and local-area datasets and a corrected local-authority list policyengine.py#553 ships the area lists in the package and points those regions at the local-area dataset, the way Use ACS-local data for US regional simulations policyengine.py#552 does for US states. get_region then returns a row-filter region with its own dataset_path, and the fallback is no longer reached.
   • Until then, keep the fallback only for the enhanced FRS. Compare the resolved dataset's household count with the matrix before running, and stop with a clear message otherwise.
3. Geographic outputs of national reports: run them on the local-area dataset, or skip them, when the national dataset carries no area codes. This follows decision 2 in UK: certify a Microcosm national dataset as the UK default, with separate national and local-area datasets and a corrected local-authority list policyengine.py#553.
4. Loading cost: the size of the local-area file depends on the Microcosm local release. If it is large, the per-area extracts discussed in Load the ACS local-area dataset for US state and district runs #717 for the US local-area file may apply here too.

Related

• UK: certify a Microcosm national dataset as the UK default, with separate national and local-area datasets and a corrected local-authority list policyengine.py#553: certifying a Microcosm national dataset, with separate national and local-area datasets.
• UK local-area exports: add the constituency_code_oa, la_code_oa and region_code_oa columns that policyengine.py and the simulation API read, and record the area-code frames in the release manifest microcosm#1114: _oa aliases on Microcosm local-area exports.
• Use ACS-local data for US regional simulations policyengine.py#552 and Load the ACS local-area dataset for US state and district runs #717: the US counterpart (local-area dataset for state and district runs).
• policyengine-api (v1) uk_constituency_breakdown downloads the same enhanced-FRS constituency matrix.

🤖 Generated with Claude Code