Skip to content

UK nation regions fail because the household data has no country column #558

Description

@MaxGhenis

What happens

Every UK nation region fails on the certified UK data. country/england, country/scotland, country/wales and country/northern_ireland scope with RowFilterStrategy(variable_name="country", ...) (src/policyengine/countries/uk/regions.py). That filter reads a country column from the input household table. The certified year file does not store one: policyengine-uk derives country from region in a formula.

Reproduced on 2026-10-08 with main at 9663cf2 (policyengine-uk 2.102.3). pe.uk.ensure_datasets(years=[2026]) wrote enhanced_frs_2024_25_year_2026.h5, with 52,846 households. The household table has region, region_code_oa, la_code_oa and constituency_code_oa, but no country. Both of these raise:

region = uk_region_registry.get("country/scotland")  # same for country/england
Simulation(dataset=year_2026, tax_benefit_model_version=pe.uk.uk_latest,
           scoping_strategy=region.scoping_strategy).run()
ValueError: Variable 'country' not found in household data. Available columns: ['household_id', 'household_weight', 'region', ...]

#553 (section 3) reports the same failure on the Microcosm national file.

Fix

Before scoping, the model version checks the household variables the strategy filters on. If one is missing from the input household table, policyengine-uk calculates it on the unscoped dataset. The derived column is used for the filter only and dropped from the scoped input. policyengine.py then keeps no copy of the region-to-nation mapping: the nations are whatever policyengine-uk's country formula says. A strategy now declares the variables it filters on.

On the 2026 year file, computing country through policyengine-uk takes about 2.7s on the full dataset. It splits the 52,846 households 37,961 England / 5,405 Scotland / 3,821 Wales / 5,659 Northern Ireland, with each region mapped to exactly one nation.

Activity

  1. MaxGhenis commented on Oct 9, 2026

    @MaxGhenis
    ContributorAuthor

    Fixed on main by #565 (released in 6.2.3), from #564: the nation regions now filter the stored region column.

    I checked it on the certified enhanced_frs_2024_25_year_2026.h5 with main at 07bae75 and policyengine-uk 2.102.3. Each nation filter keeps exactly the households policyengine-uk's own country formula assigns to that nation:

    region households weighted households
    country/england 37,961 26,354,698.47
    country/scotland 5,405 2,729,247.77
    country/wales 3,821 1,590,754.53
    country/northern_ireland 5,659 783,847.56

    The four sets are disjoint and cover all 52,846 households (31,458,548.32 weighted). Simulation(..., scoping_strategy=country/scotland).run() returns the same 5,405 households and leaves the caller's tables unchanged.

    #568 no longer changes scoping. It now adds the tests from #569, which pin #565's region lists to policyengine-uk's country formula. Its earlier approach calculated country through the model before filtering, which the repository guidance added in #565 rules out. Separately, scoped UK runs still carry UK-wide business rates and region-relative ranks: #567.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions