UK: keep constituency and local-authority results available through a switch to Microcosm, then serve them from its local-area dataset #728

Summary

#727 (closing #725) lets the executor accept a UK dataset without area codes, but only when the policyengine.py bundle declares where constituency and local-authority results come from. Both region types must be routed to another dataset through policyengine.py's regional_dataset_defaults. A missing column on its own is never read as "national-only", so a local-area dataset that lost its area codes still fails.

We don't necessarily intend to publish the Microcosm national dataset ahead of its local-area dataset; #727 keeps that option open in case it becomes necessary. Either way, constituency and local-authority results stay available throughout:

• National and local-area datasets ship together (with policyengine.py routing): constituency and local-authority runs, and the breakdowns in national reports, move straight to the Microcosm local-area dataset.
• The national dataset ships first: those runs and breakdowns keep using the enhanced FRS local-area data until the Microcosm local-area dataset is ready. That means the enhanced FRS file with its area codes, plus its constituency and local-authority weight matrices, which serve them today. The UK default does not switch to a dataset without area codes before this routing is in place.

Omitting the breakdowns is not an acceptable interim state, and #727 does not omit them: on a dataset without area codes, a constituency or local-authority breakdown raises a clear error. A routed national default would therefore fail national reports until the work below lands, so the default does not switch before then.

The executor pins policyengine[models] exactly, so this work lands with, or right after, the bump to the policyengine.py release that changes the UK default. That release is tracked in PolicyEngine/policyengine.py#553; the area-code aliases it needs are in PolicyEngine/microcosm#1114. The check-policyengine-updates workflow opens the bump.

Work

1. Interim routing to the enhanced FRS, needed only if the national dataset ships first.
   • The routing declaration: a UK regional_dataset_defaults in policyengine.py that names the enhanced FRS (as a dataset overlay) for both constituency and local_authority. This is what Admit UK datasets without area codes only when local-area runs are routed, and guard the weight-matrix fallback #727's uk_area_regions_routed reads.
   • Constituency and local-authority runs: keep the weight-matrix regions from _build_uk_weight_replacement_region. Have them load the dataset routed for their region type, resolve_region_dataset_path("uk", region_type), instead of the default; in the interim that is the enhanced FRS. _require_uk_weight_matrix_matches_dataset then confirms each matrix fits it. Keeping the matrices keeps today's numbers.
   • Breakdowns in national reports: replace Admit UK datasets without area codes only when local-area runs are routed, and guard the weight-matrix fallback #727's clear error with a run on the routed dataset, and build the constituency and local-authority outputs from that run. Record which dataset each part of the report came from: the national figures from Microcosm, the local breakdowns from the enhanced FRS.
2. Constituency and local-authority runs on the Microcosm local-area dataset. _resolve_region already asks policyengine.py first (get_region). It only builds a weight-matrix region when policyengine.py returns nothing.
   • If policyengine.py lists the regions by default: the release puts constituency and local-authority regions in its default UK registry, with dataset_path pointing at the local-area dataset (decision 3 in UK: certify a Microcosm national dataset as the UK default, with separate national and local-area datasets and a corrected local-authority list policyengine.py#553). The bump alone then moves these runs to row filters on the local file. Remove _build_uk_weight_replacement_region and _require_uk_weight_matrix_matches_dataset once nothing routes to the enhanced FRS, and update their tests.
   • If policyengine.py keeps them opt-in: build the full UK registry here with build_uk_region_registry(include_constituencies=True, include_local_authorities=True).
   • Checks:
     • The local file's revision pin should survive runtime_dataset_uri. It keeps an explicit revision for repositories outside the GCS-mirrored policyengine-*-data repos, which covers populace-uk-private.
     • detect_uk_local_authority_metadata should identify the local file's codes as LAD23.
3. Maps on national reports from the Microcosm local-area dataset. A national run on a file without area codes has nothing to group by. Build these outputs from a second run on the local-area dataset; omitting them is ruled out. Stage 12's geographic aggregate (UK_GEOGRAPHIC_DATASET_VARIABLES) follows the same route.
4. Check the routed dataset at worker start-up.
   • Today: Admit UK datasets without area codes only when local-area runs are routed, and guard the weight-matrix fallback #727 checks a routed dataset when a run loads it, because workers install only the default dataset. Validate the routed dataset's area codes when the worker starts, once workers materialize it.
   • Smoke apps: smoke_app.py and v2_static_runtime_files_smoke.py read the boundary version off the installed default. Point them at the routed dataset when the default carries no area codes.
5. Loading cost. The local-area file's size depends on the Microcosm local release. If it is large, the per-area extracts discussed in Load the ACS local-area dataset for US state and district runs #717 for the US local-area file apply here too.
6. Watch the method change.
   • The risk: a policyengine.py release could add constituency and local-authority regions to the default UK registry while the default dataset is still the enhanced FRS, or before the local-area dataset is wired in. The bump would then switch these runs from the calibrated enhanced-FRS weight matrices to row filters on the national file, and the published numbers would change.
   • Mitigation: hold that bump, or keep the weight-matrix route for that release.

Done when

• A UK default without area codes is never live while constituency and local-authority runs, or the breakdowns in national reports, are unavailable. They come from the enhanced FRS in the interim, or from the Microcosm local-area dataset.
• Every dataset that serves constituency or local-authority results is checked for its area codes when the worker starts and when a run loads it.
• UK constituency and local-authority runs on the Microcosm local-area dataset use policyengine.py's regions, with no weight matrices involved.
• National UK reports carry constituency and local-authority maps from the local-area dataset.

Related

• UK: constituency and local-authority runs use enhanced-FRS weight matrices, and every UK run requires la_code_oa; both break on a Microcosm UK default #725 and Admit UK datasets without area codes only when local-area runs are routed, and guard the weight-matrix fallback #727: accepting UK datasets without area codes only when routed, including the release-sequence discussion on Admit UK datasets without area codes only when local-area runs are routed, and guard the weight-matrix fallback #727.
• Load the ACS local-area dataset for US state and district runs #717: the US local-area dataset.
• UK: certify a Microcosm national dataset as the UK default, with separate national and local-area datasets and a corrected local-authority list policyengine.py#553: certifying a Microcosm national dataset, with separate national and local-area datasets.
• Use ACS-local data for US regional simulations policyengine.py#552: regional dataset defaults.
• UK local-area exports: add the constituency_code_oa, la_code_oa and region_code_oa columns that policyengine.py and the simulation API read, and record the area-code frames in the release manifest microcosm#1114: _oa aliases on Microcosm local-area exports.

🤖 Generated with Claude Code