Repository navigation
Map FRS household codes from explicit data-dictionary tables - #636
Open
vahid-ahmadi wants to merge 1 commit into
Open
vahid-ahmadi wants to merge 1 commit into
vahid-ahmadi wants to merge 1 commit into
Conversation
Map CTBAND through an explicit table of the UKDS data dictionary codes.
Code 10 ("Information not found" from 2024-25, "Household not valued
separately" before), blanks and missing-value codes all take band D, the
policyengine-uk default; a blank band used to become band A. A CTBAND,
GVTREGNO, PTENTYP2 or TYPEACC code outside the dictionary now fails the
build naming the FRS variable and code.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Calibration fit against the base build: pass
A drop of more than 2.0% fails. Notes
National targets in a worse error band on the head (1)
Flags are for review: builds of the same code move 2 to 12 national targets to a worse band (#598). Bands are of |estimate / target - 1|; no target's estimate or error is published. Income distribution and single ages against the base build: passHBAI FYE 2025 (DWP), period 2024: equivalised income in £ a week, rates and Gini in per cent. A value outside HBAI's 95% interval is marked (outside).
FlagsNone. These measures report and flag but do not fail yet: their same-code spread is unmeasured (#598). Aggregate impact of the data change: passBase dataset Checks: 0 broken by this change, 4 failing on both sides, 0 fixed. Moves flagged for review: 0 (thresholds in Notes
Moves flagged for reviewNone. Receipts by tax head
Spending by benefit and service
Caseloads
Government balance, each component counted once
Model totals
Components the model counts twice
Household income totals
Population
Poverty rates
People in poverty
Inequality
Mean household net income in each side's own deciles
Standard reform costings (change in gov_balance)
Standard reforms: change in each reform's target
Household net income by decileThe datasets hold different households, so households are not matched; compare the mean income in each side's own deciles. Checks failing on both base and head (4)
Counted-once checkCounted-once check not run: not requested. Provenance
Base dataset: Head dataset: Receipts by tax head: head levels
Spending by benefit and service: head levels
Caseloads: head levels
Government balance, each component counted once: head levels
Model totals: head levels
Components the model counts twice: head levels
Household income totals: head levels
Population: head levels
Poverty rates: head levels
People in poverty: head levels
Inequality: head levels
Mean household net income in each side's own deciles: head levels
Standard reform costings (change in gov_balance): head levels
Standard reforms: change in each reform's target: head levels
|
vahid-ahmadi
marked this pull request as ready for review
October 23, 2026 22:03
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An external replication using raw FRS 2024-25 hit council tax band code 10 and other codes it could not place. This PR makes the build read the FRS household codes from explicit data-dictionary tables and fail clearly on a code it does not know.
main, code 10 already became D but a blank band became A (thecategoricaldefault), so the same "no band" case got two different bands. Northern Ireland households, which leave CTBAND blank, now get D, as Microcosm already gives them.create_frswith an error naming the FRS variable and the code (never a count). Onmain, an unknown CTBAND code silently became D, and an unknown region, tenure or accommodation code became a missing value that policyengine-uk later rejects ("contains NaN values" or "Invalid value(s) ['nan']") without naming the survey variable.Evidence
UKDS FRS variable listings (household table), downloaded from the public UKDS documentation pages:
The 2024-25 "NEW FOR 2024-25" sheet says CTBAND is "Based on VOA / SAA data for CT band and not respondents' answers". GVTREGNO (1, 2, 4 to 13), PTENTYP2 (1 to 6) and TYPEACC (1 to 7) have the same codes in all three releases, so the new errors cannot fire on any of them; the production build would already fail on such codes when policyengine-uk loads the dataset.
Open question
Band D for a household with no band is the existing default, kept here. An alternative is to impute the band (for example from the reported bill and region), which would matter if the 2024-25 "Information not found" share is material. I have no access to the microdata to measure that share.
Tests
tests/test_frs_categorical_codes.py: every CTBAND code 1 to 10 against the dictionary, blanks and missing-value codes, codes outside the dictionary, a property test that every output is a validCouncilTaxBand, and the generic mapper's error.tests/test_legacy_benefit_proxies.py: throughcreate_frs, CTBAND 1, 8, 9, 10 and blank, and an unknown code for each of the four variables. Againstmain'sfrs.py, 5 of these 9 fail (blank becomes A; the four unknown codes pass silently). The fixture's PTENTYP2 and TYPEACC were 0, which neither dictionary has; they are now 3 and 1.ruff checkandruff format --checkclean on the changed files.Docs for building from a UKDS download are in a separate PR: #635.
🤖 Generated with Claude Code