Repository navigation
Filter UK country simulations by stored regions - #565
Merged
Merged
Conversation
This was referenced Oct 8, 2026
Draft
anth-volk
marked this pull request as ready for review
October 9, 2026 00:23
MaxGhenis
added a commit
that referenced
this pull request
Oct 9, 2026
Since #565 the nation regions filter the stored household region column, and policyengine.py lists which regions make up each nation. policyengine-uk keeps its own mapping in the formula that derives country from region. These tests take the expected nations from that formula, for every value of policyengine-uk's Region enum, so the two copies cannot drift apart unnoticed. A Hypothesis property checks region-only data with several people per household, separately ordered tables and str or bytes regions, and each nation runs end to end through the UK model. Fixes #569 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #564
UK country simulations fail before calculation when they filter raw household inputs on
country: the dataset storesregion, and the UK model derivescountrylater. This caused the Scotland population test failure in API deployment run 37838389100.Scotland, Wales, and Northern Ireland must be selected through the stored household
regioncolumn. Thecountry/prefix in the public selector does not name a dataset column.country/scotlandregion == "SCOTLAND"country/walesregion == "WALES"country/northern_irelandregion == "NORTHERN_IRELAND"country/englandregionis one of the nine English ITL1 valuesEngland selects
NORTH_EAST,NORTH_WEST,YORKSHIRE,EAST_MIDLANDS,WEST_MIDLANDS,EAST_OF_ENGLAND,LONDON,SOUTH_EAST, andSOUTH_WEST. Unknown or missing regions are excluded, rather than classified as England.Extend
RowFilterStrategyto accept multiple allowed values while preserving existing scalar filters and cache keys. All four countries retainRowFilterStrategy, which the simulation worker can combine throughRegionGroupStrategy. Preserve household membership, related people/benefit units, and weights; handle text and byte region encodings without requiring acountrycolumn.AI-facing guidance in
AGENTS.mdanddocs/engineering/skills/repository-guidance.mdexplicitly requires these mappings, explains the distinction between public selectors and input columns, and requires regression tests with nocountryinput column.docs/regions.mdgives users the same mapping. Do not replace the fix with an extra dataset column or a preliminary model calculation.Validation: the original code change passed all 158 affected regional/filtering tests. After adding the explicit guidance, 85 focused UK registry, scoping, and entity utility tests passed; repository formatting, Ruff lint, and whitespace checks passed. Existing targeted mypy checks report two pre-existing
no-any-returnerrors in unchanged weight-replacement code, reproduced at the base revision.Earlier filtering checks also passed against an older cached UK dataset for all four countries, preserving membership and weights. That file differs from the certified artifact; fetching the certified artifact failed local authentication. The exact certified artifact and deployed Scotland population simulation remain unvalidated with this change.
Expected release: 6.2.3, based on current main/PyPI version 6.2.2 and this patch-level changelog fragment. The release must actually contain this PR; if another change consumes that version first, update the dependent PRs to the actual fixed version.
Release sequence: temporarily remove the failing API test in PolicyEngine/policyengine-api#3875 (issue PolicyEngine/policyengine-api#3874); merge and publish this fix; upgrade and deploy simulation workers in PolicyEngine/policyengine-sim-api#733 (issue PolicyEngine/policyengine-sim-api#732); then upgrade API v1 and restore the exact Scotland test in PolicyEngine/policyengine-api#3877 (issue PolicyEngine/policyengine-api#3876). The downstream PRs intentionally require the unpublished version and must fail dependency resolution until publication. Regenerate their authentic lockfiles after publication and qualify the deployed worker before merging the API restoration.