[juaristi22]

Fixes #880

Part of #728. This makes the executor serve policyengine.py 6.10.0 (PolicyEngine/policyengine.py#685), which makes the Microcosm national dataset the UK default and routes constituency and local-authority regions to the Microcosm local-area dataset. UK-wide and nation-level reports run on the national file alone. Their constituency and local-authority results come from a separate run, requested on demand with breakdown: "local_areas", on the local-area file alone.

Same-Repository Draft PR Check

• This PR is a draft.
• This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Summary

After this PR, on a release built with .py 6.10.0:

• UK-wide and nation-level requests (region absent, uk or country/<nation>) load only microcosm_uk_2024_25 and build one baseline and reform. They compute no constituency or local-authority results: both fields are null, and the job status body omits them. This removes the second baseline and reform run on the local-area file from every such request (about 70-90 s and 12 GiB each), from every UK budget-window year and from the UK release sentinel.
• Local-area breakdowns. A UK-wide or nation-level request with breakdown: "local_areas" loads only microcosm_uk_2024_25_dense and builds one baseline and reform there with the region's filter. Both area levels then come from one disclosure-controlled release (Use disclosure-controlled UK geography outputs for constituency and local authority impacts #860), and the job returns a LocalAreaBreakdownOutput.
• Constituency and local-authority requests are unchanged: they run on the local-area file, as María's commits below set up.
• Older releases. Each deployment rebuilds only the worker app for the current .py version (the deploy_executor job redeploys policyengine-simulation-py<current version> from main); apps for older .py versions are not rebuilt, and they ignore fields they do not know. The gateway therefore sends a breakdown only to a worker whose routing bundle manifest declares the new capability, and refuses it with 400 for any other.
• Which bundles count as routing UK area regions. The manifest's regional_release_kind must be local_area, and both area region types must route to one certified dataset. A bundle of kind area_coded_national, which .py certifies without registering area regions, or one that routes the two types to different datasets, counts as not routed and declares no capability.
• Dataset checks before simulating. A routed local-area dataset must carry all four household columns the area outputs read (constituency_code_oa, la_code_oa, region_code_oa, source_household_id) as it loads, for breakdown, constituency and local-authority runs alike, in the runtime and in the Stage 12 worker.
• Dataset checks before merge. The executor image smoke and the Stage 12 static-files smoke check the same four columns on the downloaded routed file, reading only the household table's column names, so the PR image smoke checks the real file.

Contract

• SimulationRequest.breakdown: Literal["local_areas"] | None = None. Accepted only for UK requests whose region is absent, uk, country/england, country/scotland, country/wales or country/northern_ireland (the nations .py 6.10.0 registers; an executor test checks the list against the installed registry). Refused with 422 for other countries, any other region (constituency, local-authority, bare-name, country/ni or an unknown nation), include_cliffs: true, segmented: true, a region_group or any other value. include_cliffs: false stays accepted, because the Public API always sends it. BudgetWindowBatchRequest does not take the field.
• LocalAreaBreakdownOutput (extra="forbid"): kind: "local_areas", model_version, policyengine_version, dataset, data_version, constituency_impact: list[ConstituencyImpactRecord] and local_authority_impact: list[LocalAuthorityImpactRecord]. Each list takes only its own level's records, and either may be empty.
• JobStatusResponse.result is SingleYearMacroOutput | LocalAreaBreakdownOutput | None. Both models forbid extra fields, so a payload validates as only one of them.
• Routing bundle manifest key local_area_breakdown, version 1, written by the routing publisher only for apps whose bundle routes UK area regions as described above. The gateway refuses breakdown with 400, before spawning, for any route whose manifest lacks it, including routes pinned to an older version and routes from the legacy version dicts. The error names the .py version.
• The entry starts no Stage 12 comparison for a breakdown request (skip reason unsupported_options).

The gateway OpenAPI golden and the generated client are regenerated.

Commits

María's commits (unchanged):

• 5ad72b0f Build routed UK area results from the local-area dataset
• 6aad6520 Check the routed area dataset in the image smokes
• c2728bce Copy no area codes into Stage 12 for a routed UK national report
• 24690a27 Record the routed UK area changes in the changelog

Added on top:

SHA	Commit	Kind
a96c72c2	Merge origin/main into uk-728-area-breakdowns-microcosm	merge, no conflicts
7dad9206	Load the routed UK area pair through main's dataset and span helpers	red-green
acc1d2bf	Use disclosure-controlled UK geography outputs for constituencies and local authorities	cherry-pick of #860's 3c99e59d (Pavel), adapted for this PR
897e4b56	Withhold the inner areas that sum to a withheld UK area	cherry-pick of #860's 8fb0b788 (Pavel)
0e132423	Type the UK area pair counts that the containment helper reads	Pyright fix for the previous commit
73dfd895	chore(deps): update policyengine to 6.10.0	cherry-pick of #883's 2e98847d (github-actions)
0ec2f4d4	Require the disclosure columns on the routed area dataset before simulating	red-green
3f031fd2	Add breakdown=local_areas to the simulation request and its result model	red-green
f14cc7ba	Run a local-area breakdown on the local-area dataset alone	red-green
841fc732	Stop building the area pair in UK-wide and nation runs on routed bundles	red-green
362a904c	Declare the local-area breakdown capability in the routing manifest	red-green
ef6b7eca	Refuse breakdown requests for workers that do not declare serving them	red-green
ad9c35da	Check UK local-area breakdowns and area-free national results in beta	red-green (local helper tests)
3cdc8149	Document UK local-area breakdowns and record them in the changelog	docs
895d20dc	Remove the UK weight-matrix region fallback	red-green

Added after the first review:

SHA	Commit	Kind
17c73b1e	Count UK area regions as routed only to one local-area release dataset	red-green
5ecf8d47	Check every area-output column before a routed area run simulates	red-green
f2b7685e	Accept only the UK nations .py registers in a breakdown request	red-green
1fd5c004	Type each breakdown level's records and the area pair precisely	red-green (contract) and typing
dad34498	Build local-area breakdowns in their own builder	refactor, no behaviour change
f77a8bee	Test the installed bundle's routing and a version-pinned refusal	tests of existing behaviour
2e3c1bf3	Correct the changelog and gateway README for the review's points	docs
ff0e3d97	Name the breakdown builder in the microdata-weights guidance	docs

Added after the second review:

SHA	Commit	Kind
1eab140d	Check every area-output column of the routed file in the image smokes	red-green
64fdfed7	Share the output builders' span and version helpers	refactor, no behaviour change
b34a7052	Type the breakdown builder's country module and drop an unused field	typing, no behaviour change
dbf75b34	Test a Scotland breakdown on the installed bundle and the check fallback	tests of existing behaviour
9503823d	Record the image smokes' column check in the changelog	docs

Correction to 895d20dc's commit message: it says only the gateway and entry are redeployed from main. The deploy_executor job also redeploys the app for the current .py version from main on each deployment. The removal is still safe, because that app is now 6.10.0, which routes UK area regions; apps for older versions are not rebuilt.

The cherry-picks use git cherry-pick -x, so each keeps its author and names its source commit. acc1d2bf's message has an "Adapted for #881" paragraph. Its disclosure release reads the pair the area breakdowns use, #881's builder tests patch compute_uk_geography_disclosure_output, and the planning test expects the four household columns. #860 and #883 are still open; this PR now contains their commits.

Red-green

Each commit marked red-green was written tests first. The new tests were run and failed for the expected reason before the code was written, and the tests and the code were committed together. Each commit message records how its tests failed. For example:

• the request tests failed with extra_forbidden where they expect value_error or literal_error;
• gateway polling of a breakdown result returned 500;
• invalid breakdown requests reached the national path (DID NOT RAISE);
• a routed national run loaded both datasets;
• the routing manifest had no local_area_breakdown (KeyError);
• the gateway returned 200 and spawned the job for workers without the capability;
• after the first review: an area_coded_national bundle or one routing the two area types to different datasets counted as routed (assert True is False) and declared the capability; constituency and local-authority runs on a dense file missing a column built both simulations (DID NOT RAISE, and in Stage 12 "simulated a dataset it cannot use"); country/ni and country/atlantis passed the contract (DID NOT RAISE ValidationError); a local-authority record validated in the constituency list (DID NOT RAISE ValidationError);
• after the second review: both image smokes passed a routed file missing constituency_code_oa, region_code_oa or source_household_id (DID NOT RAISE <class 'ValueError'>), and raised a generic error that does not name the file for a missing la_code_oa.

A few new tests pinned behaviour that already held, and passed before the change; their commit messages say which. Examples are the Stage 12 adapter skipping an unknown field, the non-default-dataset rule for deterministic baseline ids, f77a8bee's two tests (the installed bundle's real routing and revision, and a refusal pinned by UK model version), and dbf75b34's tests (a country/scotland breakdown resolved against the installed bundle, and the other branch of detect_uk_run_dataset_metadata).

A merge and cherry-picks cannot be written test-first. They were checked by running the full suites, as below.

Testing

On the final tree, with Python 3.13 and the pinned ruff 0.15.22:

• Unit tests, each in its own locked environment (uv run --frozen pytest tests/):

  • executor: 1862 passed, 22 skipped;
  • contract: 322 passed;
  • gateway: 268 passed;
  • entry: 668 passed;
  • observability: 35 passed;
  • Stage 12 persistence: 40 passed;
  • fastapi: 14 passed.

• make check passes. It regenerates the client, runs ruff format and check, and runs Pyright in every package. The executor's Pyright errors went from 77 to 68: Use disclosure-controlled UK geography outputs for constituency and local authority impacts #860 removed 6 and the weight-matrix removal 3, and no new error is recorded.

• Other checks:

  • python scripts/run_quality_guards.py passes.
  • python -m scripts.rules render leaves no diff.
  • uv lock --check passes for the executor.

• Generated client: ./scripts/generate-clients.sh, then the policyengine-apis-integ tests that need no services (24 passed). test_ping needs the docker-compose services and was not run.

• On the real 6.10.0 bundle, without loading data, the breakdown path resolves uk, country/wales and country/northern_ireland to microcosm_uk_2024_25_dense with its revision microcosm-uk-2024-25-dense-20261019T000432Z-ee5c2dbe, and refuses constituency/Aldershot. A unit test now checks the routing and the revision against the installed bundle.

• OpenAPI golden regenerated in 3f031fd2 and again in 1fd5c004 (the two area lists now reference their record schemas); docs/generated/rules.md renders with no diff.

• Not run locally:

  • the Docker builds, the Modal image smoke and the local integration job;
  • the beta integration tests: test_calculate_uk_local_area_breakdown is new, and test_calculate_uk_model gains an assertion;
  • an end-to-end breakdown on the private data.

  CI and the beta deploy run them.

Environment variables

None of the changed code reads a new environment variable.

• Routing publisher (update_version_registry.py, changed in 362a904c; its condition tightened in 17c73b1e). The new key comes from uk_area_regions_routed(), which reads regional_release_kind, region_datasets and datasets from the bundle manifest packaged inside the installed policyengine package (policyengine/data/bundle/manifest.json), with no network access and no environment variable.
  • The publisher's existing variables are unchanged. MODAL_TOKEN_ID and MODAL_TOKEN_SECRET are GitHub Actions repository secrets; both names are listed in the repository's secrets. simulation-deploy.yml passes them to simulation-deploy.reusable.yml with secrets: inherit, and the plan, record and publish steps set them in env:.
  • MODAL_ENVIRONMENT, APP_NAME, POLICYENGINE_VERSION, POLICYENGINE_US_VERSION, POLICYENGINE_UK_VERSION, FORCE_LATEST and RELEASE_ENVIRONMENT come from workflow inputs and job outputs.
  • GITHUB_WORKFLOW_REF, GITHUB_ACTIONS, GITHUB_REPOSITORY, GITHUB_RUN_ID and GITHUB_RUN_ATTEMPT are set by GitHub.
• Executor. A breakdown run downloads the private local-area file through the existing run_simulation Modal function. That function already attaches the pe-uk-private-hf-read-token Modal secret, with required key HUGGING_FACE_TOKEN, and gcp-credentials. Constituency and local-authority runs on 6.10.0 already use the same download. I confirmed the attachment in code only; I did not check that the Modal secret exists in Modal.
• Gateway and entry read nothing new.

Code-owner review

These commits change files under rules/, which need code-owner approval (sim.process.rule-machinery-code-owned):

• rules/baselines/pyright-executor.txt and its count in rules/registry.toml: 77 to 71 in acc1d2bf, then 71 to 68 in 895d20dc. Both only remove entries.
• rules/baselines/sim-microdata-weight-stripping.txt and its count: 1 to 0, from Use disclosure-controlled UK geography outputs for constituency and local authority impacts #860's commit.

The commits added after the two reviews change nothing under rules/; the executor's Pyright errors stay at 68.

Rollout

1. Merge and deploy this PR. That creates policyengine-simulation-py6-10-0 with the local_area_breakdown capability, and moves latest to 6.10.0. The Public API still pins 6.7.5, so its users see no change. Unpinned clients get national UK results without area data.
2. The app-v2 change can merge on its own. Outputs from 6.7.5 carry area data, so its Compute buttons do not appear yet.
3. The Public API change sends breakdown and moves to 6.10.0. It needs this PR deployed first.

Risks and follow-ups

• The 6.10.0 executor is frozen once main's lock moves past it, so later fixes to the breakdown code reach only newer versions.
• An entry rollback to an older Cloud Run revision would return 422 for breakdown.
• The local-area file (about 720 MB) is downloaded on every cold container. Measure this in beta, and consider baking it into the image.
• Disclosure control may withhold many areas on the local-area file, or publish nothing for a level. The cost of its 200 bootstrap replicates on that file has not been measured. The beta breakdown test fails if either level is empty.
• country/ni, the Public API's spelling, is refused with 422; .py names it country/northern_ireland. The Public API change maps it before sending.
• data_version of a breakdown falls back to the release's data version if the manifest records no revision for the local-area dataset, as the macro output does. 6.10.0 records one.
• data_version on single-area UK runs still names the national release (_selected_dataset_revision is US-only).
• No Stage 12 local-area report exists, so the shadow comparison does not cover breakdown numbers.
• The executor's synchronous FastAPI surface (compat_models.SimulationOptions, used by docker-compose) does not take breakdown.

🤖 Generated with Claude Code

[PavelMakarchuk]

Thanks María and Anthony, this looks good to me. I approve.

I merged main (with #904) into it locally, and checked:

• It merges cleanly. .py 6.10.0 still gives relative_change as a percentage, so #904's /100 stays right. Its test runs on 6.10.0 and passes.
• All tests pass on the merged tree: executor 1,867, contract 322, gateway 269, entry 668. Ruff 0.15.22 and pyright pass.
• The 6.10.0 registry has 650 constituencies (18 in NI), 361 local authorities and country/northern_ireland, which matches api#4024.
• The breakdown run loads the dense file, so it cannot share a baseline ID or a cache entry with a national run. api#4024 also gives breakdown its own options hash.

One question before the UK switch: a region=uk breakdown and a region=country/* breakdown each make their own withholding decisions over the same households. An area that one run withholds could appear in the other, and a user can request both. Is that acceptable under our disclosure policy? If not, a nation request could run through the UK-wide release and then filter. This doesn't need to block the merge.

A small note: breakdown baselines always get a random ID, because the dense file is not the default dataset. So each "Compute" click recomputes the current-law baseline. That is fine for now.

For the record: 6.10.0 pins policyengine-uk 2.132.6, which doesn't have the UC Act uplift yet (policyengine-uk#2242, from 2.135.0). That needs a later .py bump.

Merge order note: #904 is now merged and deployed to prod, so the remaining order is api#4026, then this PR.

[anth-volk]

Thanks, Pavel. On your question about withholding decisions: a region=uk breakdown and a region=country/* breakdown each make their own decisions over the same households. We'll deal with that in #907, which covers deciding a disclosure policy for subnational results (including the option of running every breakdown UK-wide and filtering it to the nation) and measuring how far breakdowns can be split by nation and English region. As you said, it doesn't block this PR.