Anchor UK year-file runs on the survey year so the State Pension follows the triple lock

[MaxGhenis]

Fixes #556

Summary

A UK population run through policyengine.py did not reproduce policyengine_uk.Microsimulation on the same certified dataset and year. The State Pension followed CPI instead of the triple lock, and everything downstream moved with it. Under the current release's policyengine-uk (2.102.3), State Pension rate reforms didn't reach it at all. With this PR, a run of a UK year file matches the direct policyengine-uk run on every output, record by record, with or without a reform.

Mechanism (from the code; issue #556 has the per-record evidence)

• create_datasets cut each year file from the projection policyengine-uk makes of the certified file (sim.dataset[year]).
• run() passed that frame back as UKSingleYearDataset(fiscal_year=year).
• policyengine-uk's three State Pension formulas take the data year as min(simulation.dataset.years). They split state_pension_reported at the data year against that year's legislated rates, then scale to the period's rates.
• A direct run's data year is the survey year (2024). In the wrapper, the projected year became the data year, and the projection uprates state_pension_reported by CPI. So for every recipient, the wrapper's State Pension was the reported amount uprated by CPI rather than by the triple lock.

Fix

• PolicyEngineUKDataset gains data_year and data_year_data.
• create_datasets records the source's first year and stores its tables in each projected year file (data_year, data_year_person, … HDF5 keys).
• run() builds the policyengine-uk input a direct run builds. It projects the data year's tables forward with policyengine-uk's own extend_single_year_dataset, and puts the year file's tables (scoped, if a region is set) at the simulated year. Records are matched by entity ID, so region scoping carries over to the data year; records missing from the data year raise a clear error. The tables are copied before handing them over. policyengine-uk encodes enum columns in place on a multi-year dataset's tables, which this path builds; it copies a single-year dataset when projecting it.
• policyengine.py's person_weight / benunit_weight columns are dropped from the data year's tables before projecting, because a direct run calculates them. The projected years are passed in order.
• ensure_datasets regenerates year files that lack the data-year record. load_datasets refuses them, and now loads each file once. The record is written last, so it marks a complete file, and an unreadable file raises instead of reading as unrecorded. A dataset built in memory without data_year still runs as observed data for its own year, which is how policyengine-uk treats a single-year dataset.
• Saved UK simulation outputs record the data year their run anchored on. Simulation.load() refuses UK outputs saved without one, as it refuses US outputs without their marker, and Simulation.ensure() runs them again.
• docs/microsim.md documents the year-file contract and the scoping caveat (Scoped UK runs give the region UK-wide business rates and region-relative ranks #567).

Evidence (real runs, 2026)

Record-level check, through the public API: ensure_datasets → Simulation.run() vs policyengine_uk.Microsimulation on the same materialized file, over 69 output columns (the default outputs plus the State Pension components, Housing Benefit, CTR and the gov_* totals):

	main + this PR (uk 2.102.3, data 1.56.16)	#555 + this PR (uk 2.123.4, data 1.58.0)
columns compared	69	69
records differing	0	0
State Pension, £bn (wrapper / direct)	133.69 / 133.69	128.31 / 128.31

Before the fix, the same comparison (a prototype of this construction, against the old one) found 26 of 68 columns differing. They included in_poverty_* for 313–372 households and the income deciles. A Scotland-scoped run's State Pension components, income tax and Pension Credit matched the direct run's for the records kept with the fix, and differed without it.

With a reform: new State Pension £260/wk and basic £200/wk from 2026, against 2026 law of £241.30 / £184.90. The policyengine.py runs use Simulation(policy=...). The direct runs apply the same compiled policy with policyengine.py's own modifier (reform_impact.py). Changes are against each column's own baseline, in £bn:

Reform effect	6.2.x before	6.2.x fixed	direct, uk 2.102.3	#555 before	#555 fixed	direct, uk 2.123.4
State Pension	+0.000	+10.679	+10.679	+9.939	+10.250	+10.250
Income tax	0.000	+1.959	+1.959	+1.734	+1.846	+1.846
Pension Credit	0.000	−1.065	−1.065	−0.905	−0.886	−0.886
Household net income	0.000	+7.309	+7.309	+7.176	+7.390	+7.390

The reformed totals with this PR equal the direct run's to three decimals: State Pension £144.369bn on 6.2.x and £138.564bn on #555.

How each column was run:

• 6.2.x before: policyengine 6.2.1 from PyPI, on its own year files.
• Certify policyengine-uk-data 1.58.0 with policyengine-uk 2.123.4 and core 3.32.21 #555 before: the Certify policyengine-uk-data 1.58.0 with policyengine-uk 2.123.4 and core 3.32.21 #555 worktree at cdad33a, which is the branch merged with main (6.2.2), without this PR, on its own year files. Its baseline (pepy_branch.json) was run on the branch before the main merge, which reports 6.2.1. The merge only brought US changes (Use ACS-local data for US regional simulations #552).
• Fixed: main or Certify policyengine-uk-data 1.58.0 with policyengine-uk 2.123.4 and core 3.32.21 #555 with this PR's commits cherry-picked. The changed Python sources are identical to this branch's; Certify policyengine-uk-data 1.58.0 with policyengine-uk 2.123.4 and core 3.32.21 #555's bundle manifest and TRACE files differ, as they belong to Certify policyengine-uk-data 1.58.0 with policyengine-uk 2.123.4 and core 3.32.21 #555.
• Direct: policyengine-uk on the same materialized file, with the same compiled policy applied by policyengine.py's modifier.

Impact on published UK figures

Every UK population result computed through policyengine.py's Simulation on ensure_datasets year files carries this error, verified on policyengine 6.2.1 and the #555 branch. Both columns below come from running policyengine.py's own API (pepy_poverty.py: ensure_datasets, Simulation, calculate_uk_poverty_rates / _by_age, Aggregate) before and after the fix. In both cases the fixed result equals the direct policyengine-uk run on every aggregate and on overall poverty, to three decimals.

2026-27	6.2.x before	6.2.x fixed	#555 before	#555 fixed
State Pension, £bn	127.50	133.69	124.42	128.31
Pension Credit, £bn	7.51	6.99	7.42	7.05
Income tax, £bn	312.35	313.45	319.89	320.55
Housing Benefit, £bn	13.75	13.71	9.18	9.14
Household net income, £bn	1,760.2	1,764.7	1,708.7	1,711.5
Universal Credit, Child Benefit	unchanged		unchanged
Absolute BHC poverty, everyone	14.75%	14.41%	15.16%	15.02%
Absolute AHC poverty, everyone	19.00%	18.71%	20.32%	20.25%
Relative BHC poverty, everyone	18.75%	19.01%	16.61%	16.56%
Relative AHC poverty, everyone	23.06%	23.27%	21.10%	20.96%
Absolute BHC, 65 and over (senior)	15.18%	13.46%	15.67%	14.95%
Absolute AHC, 65 and over	13.41%	11.91%	13.73%	13.36%
Relative BHC, 65 and over	21.46%	21.17%	17.51%	16.79%
Absolute BHC, children	16.32%	16.31%	19.97%	19.97%

• Bump policyengine-uk to 2.102.3 for the FYE 2025 absolute poverty line #536 (merged) published absolute poverty of 14.7% BHC and 19.0% AHC for everyone on 6.2.x. The direct run, and this fix, give 14.4% and 18.7%. Its child figures (16.3% / 23.7%) are unchanged at one decimal.
• Certify policyengine-uk-data 1.58.0 with policyengine-uk 2.123.4 and core 3.32.21 #555 (open) publishes the "before" columns in its UK tables. After this fix the branch gives State Pension £128.3bn (not £124.4bn), Pension Credit £7.1bn, income tax £320.5bn, household net income £1,711bn, and pensioner absolute BHC poverty of 15.0% (not 15.7%). Its direct-Microsimulation "model versus data" table is unaffected.
• Reform results. Under policyengine 6.2.x (uk 2.102.3), every State Pension rate reform scored zero through policyengine.py. The data year was the simulated year, so each recipient's pension was their CPI-uprated reported amount whatever the rates were; a rate change only moved money between the flat-rate and additional components. Under uk 2.123.4, which reads the data year's rates from the unreformed baseline (policyengine-uk#2163, released in 2.123.1), reforms did take effect but were understated: +£9.9bn against +£10.25bn for the reform above. Any triple-lock or State Pension rate analysis run through policyengine.py should be run again on a release with this fix.
• On the 1.56.16 data under uk 2.102.3, part of the State Pension gap comes from a policyengine-uk double count that 2.123.4 fixed (policyengine-uk#1921). It inflated the direct run's additional State Pension. The wrapper reproduces whatever policyengine-uk computes, so with this PR it now includes that double count too, as policyengine-uk 2.102.3 does.

Invariants

• Differential. For any source dataset and any year it projects to, every output of a policyengine.py run of the year file equals the direct policyengine-uk run of the source, record by record. This holds with or without a reform applied the same way to both, and for a year file loaded back from disk. Under row filtering (RowFilterStrategy, the tested strategy) it holds, for the records kept, for person and benefit-unit variables that don't depend on dataset-wide normalisation. policyengine-uk calculates income deciles, the relative-poverty median and shareholding over every household in the simulation (read in its source). A row-filtered run therefore gets region-relative values for those, including household flags mapped to people. WeightReplacementStrategy changes them through the weights. That is pre-existing issue Scoped UK runs give the region UK-wide business rates and region-relative ranks #567, which this PR doesn't change. Tested by example for 2024 / 2025 / 2026 / 2028, by a Hypothesis property over random pensioners (age, sex, reported amount) and years 2025–2030, with a reform, and through load_datasets. Checked on the real certified data for 2026 under both pe-uk versions (tables above).
• Round trip. Saving and loading a year file preserves data_year and the data year's tables.
• No aliasing. A run, scoped or not, leaves the caller's dataset tables unchanged.
• Validation. data_year after year, a projected dataset without its data-year tables, and data-year tables without data_year are all refused. Outputs record a data year without tables.
• Saved outputs. A saved UK output records its data year. An output saved without one is refused by load() and re-run by ensure().

Tests: tests/test_uk_year_file_data_year.py, 20 tests on a seven-person 2024 dataset written in policyengine-uk's own file format, real policyengine-uk, no licensed data, about 50 s. All 20 also pass under policyengine-uk 2.123.4 (#555's pin).

• Mutation check. With run() reverted to the old single-year construction, 7 of the first 15 fail: every differential test for a projected year, the scoped run, the Hypothesis property, regeneration, and record matching. The 8 that still pass don't depend on the anchoring: 2024 is its own data year, plus storage, validation and aliasing. The reform test and the loaded-from-disk test also fail under the mutant.
• Full suite: on f384ec2, 1214 passed, 9 skipped, 2 failed; origin/main gives 1201 passed and 9 skipped in the same environment. Both failures were in tests/test_dataset_runtime.py, which mocks policyengine-uk and calls create_datasets(years=[]). 875924c skips the data-year read when no years are requested, and those tests then pass. On 1629ee7b, 237 targeted tests pass: every test file that saves, loads or ensures simulations, plus the UK dataset, region, scoping, run-record and release-manifest tests.

Compatibility

• Year-file format. Projected year files gain the data year's tables, which doubles them: Enhanced FRS 2026 goes from 113 MB to 226 MB.
• Old year files. ensure_datasets rewrites them once. load_datasets refuses them with a message to regenerate. Direct PolicyEngineUKDataset(filepath=...) opens are not checked, matching the US precedent for legacy-input records.
• Old saved outputs. UK outputs saved by earlier releases are refused by Simulation.load() and re-run by Simulation.ensure().
• Disk. Every projected year file repeats the data year's tables; the default five years add about 565 MB. Storing them once per data folder would be a possible follow-up.
• policyengine-uk API. The fix uses policyengine_uk.data.economic_assumptions.extend_single_year_dataset, which has the same signature in 2.102.3 and 2.123.4.
• Not changed. managed_microsimulation, household calculations, and the US path.

Related

• Test the UK nation filters against policyengine-uk's country formula #568 (UK nation scoping) adds a function with the same name and signature, _policyengine_uk_input(dataset). Whichever PR lands second keeps this body. Test the UK nation filters against policyengine-uk's country formula #568 also calls it on the unscoped dataset to derive filter variables; with this body that dataset includes the data-year tables, which is correct, just costlier.
• Scoped UK runs give the region UK-wide business rates and region-relative ranks #567. Row-filtered UK runs normalise dataset-wide outputs over the region.

Independent review

Opus (subfleet run --task review --tier standard) requested changes at 9e16ad72, with no blockers. It confirmed the mechanism and the construction by reading both policyengine-uk versions, and confirmed every number but one. All ten findings are addressed in 1629ee7b:

1. The Certify policyengine-uk-data 1.58.0 with policyengine-uk 2.123.4 and core 3.32.21 #555 pensioner figure is 15.0%, not 14.9%.
2. The scoping claim is narrowed, and the caveat is in the docs.
3. The reform behaviour change is documented, measured and tested.
4. A year file loaded from disk is compared against the direct run.
5. Weight columns are dropped from the projected data-year tables.
6. Projected years are sorted.
7. The data-year key is written last.
8. Unreadable files raise.
9. Each year file is loaded once.
10. The changelog is updated, and saved-output reuse is now refused rather than left as a follow-up.

Round 3 at efaeb13a approved the nit fixes. The branch was then brought up to date with main (#565, #559) at 0110ff97, with no conflicts; its diff against main is still these six files.

Round 2 at 1629ee7b approved: every finding was fixed in code, with no regressions. Its nits are addressed in efaeb13a: changelog wording, citing the policyengine-uk fix PR (#2163, 2.123.1), how each reform column was run, docs on weight replacement, and a clear error when an output dataset is simulated. Logging expected re-runs below warning level touches the US path too, so it is split out.

Follow-ups (split out)

• The UK registry's country/* regions filter on a country column that the certified year file doesn't store, so they raise. Found while testing scoping.
• Year files are named by dataset stem only, so ensure_datasets reuses year files cut from an earlier data release or model version (UK and US).

axiom: n/a: wrapper data-loading fix; no policy rule changes

🤖 Generated with Claude Code

[MaxGhenis]

Merged at head 0110ff97 with a merge commit (e080ce00).

Gates

• CI: gh pr checks exited 0 on the latest run at 0110ff97: all 13 checks, including the full test matrix on Python 3.11–3.14.
• Merge state: MERGEABLE, not a draft, no change request on GitHub.
• Base: main was still 07bae750, the commit merged into the branch.

Independent review: Opus via subfleet run --task review --tier standard.

• Round 1, at 9e16ad72: requested changes, with ten findings and no blockers.
• Round 2, at 1629ee7b: approved. All ten were fixed in code, with six nits.
• Round 3, at efaeb13a: approved the nit fixes.

0110ff97 is efaeb13a plus a conflict-free merge of main (#565, #559). Its diff against main is the same six files.

Why this merges without Max's sign-off: it is a correctness fix that moves published aggregates (2026 State Pension £127.5bn → £133.7bn on 6.2.x). It makes the wrapper match policyengine-uk on the same data, which Max's standing rule covers.

Follow-ups split out

• Year-file reuse across data releases.
• Quieter logging of expected re-runs.
• The UK country-region filter, since fixed on main by Filter UK country simulations by stored regions #565.