Skip to content

Filter UK country simulations by stored regions - #565

Merged
anth-volk merged 2 commits into
mainfrom
fix/uk-country-region-filter
Oct 9, 2026
Merged

anth-volk merged 2 commits into
mainfrom
fix/uk-country-region-filter

Conversation

@anth-volk

@anth-volk anth-volk commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #564

UK country simulations fail before calculation when they filter raw household inputs on country: the dataset stores region, and the UK model derives country later. This caused the Scotland population test failure in API deployment run 37838389100.

Scotland, Wales, and Northern Ireland must be selected through the stored household region column. The country/ prefix in the public selector does not name a dataset column.

Public selector Required input filter
country/scotland region == "SCOTLAND"
country/wales region == "WALES"
country/northern_ireland region == "NORTHERN_IRELAND"
country/england region is one of the nine English ITL1 values

England selects NORTH_EAST, NORTH_WEST, YORKSHIRE, EAST_MIDLANDS, WEST_MIDLANDS, EAST_OF_ENGLAND, LONDON, SOUTH_EAST, and SOUTH_WEST. Unknown or missing regions are excluded, rather than classified as England.

Extend RowFilterStrategy to accept multiple allowed values while preserving existing scalar filters and cache keys. All four countries retain RowFilterStrategy, which the simulation worker can combine through RegionGroupStrategy. Preserve household membership, related people/benefit units, and weights; handle text and byte region encodings without requiring a country column.

AI-facing guidance in AGENTS.md and docs/engineering/skills/repository-guidance.md explicitly requires these mappings, explains the distinction between public selectors and input columns, and requires regression tests with no country input column. docs/regions.md gives users the same mapping. Do not replace the fix with an extra dataset column or a preliminary model calculation.

Validation: the original code change passed all 158 affected regional/filtering tests. After adding the explicit guidance, 85 focused UK registry, scoping, and entity utility tests passed; repository formatting, Ruff lint, and whitespace checks passed. Existing targeted mypy checks report two pre-existing no-any-return errors in unchanged weight-replacement code, reproduced at the base revision.

Earlier filtering checks also passed against an older cached UK dataset for all four countries, preserving membership and weights. That file differs from the certified artifact; fetching the certified artifact failed local authentication. The exact certified artifact and deployed Scotland population simulation remain unvalidated with this change.

Expected release: 6.2.3, based on current main/PyPI version 6.2.2 and this patch-level changelog fragment. The release must actually contain this PR; if another change consumes that version first, update the dependent PRs to the actual fixed version.

Release sequence: temporarily remove the failing API test in PolicyEngine/policyengine-api#3875 (issue PolicyEngine/policyengine-api#3874); merge and publish this fix; upgrade and deploy simulation workers in PolicyEngine/policyengine-sim-api#733 (issue PolicyEngine/policyengine-sim-api#732); then upgrade API v1 and restore the exact Scotland test in PolicyEngine/policyengine-api#3877 (issue PolicyEngine/policyengine-api#3876). The downstream PRs intentionally require the unpublished version and must fail dependency resolution until publication. Regenerate their authentic lockfiles after publication and qualify the deployed worker before merging the API restoration.

@anth-volk anth-volk reopened this Oct 9, 2026
@anth-volk
anth-volk marked this pull request as ready for review October 9, 2026 00:23
@anth-volk
anth-volk merged commit 3a5ea9a into main Oct 9, 2026
26 checks passed
@anth-volk
anth-volk deleted the fix/uk-country-region-filter branch October 9, 2026 00:23
MaxGhenis added a commit that referenced this pull request Oct 9, 2026
Since #565 the nation regions filter the stored household region column,
and policyengine.py lists which regions make up each nation. policyengine-uk
keeps its own mapping in the formula that derives country from region.
These tests take the expected nations from that formula, for every value of
policyengine-uk's Region enum, so the two copies cannot drift apart
unnoticed. A Hypothesis property checks region-only data with several
people per household, separately ordered tables and str or bytes regions,
and each nation runs end to end through the UK model.

Fixes #569

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Filter UK countries using stored household regions

1 participant