Repository navigation
Extend LA council tax to all eight bands (MHCLG Table 9, 2026-27) - #482
Open
vahid-ahmadi wants to merge 3 commits into
Open
vahid-ahmadi wants to merge 3 commits into
vahid-ahmadi wants to merge 3 commits into
Conversation
Adds band_A_amount..band_H_amount to storage/la_council_tax.csv for the
296 English billing authorities, taken from MHCLG "Council tax levels
set by local authorities in England 2026-27", Tables_1-9_2026-27.ods,
sheet Table_9 ("Area council tax for a dwelling occupied by two adults
by band"). The columns are England only: Table 9 does not cover Wales
or Scotland, so those rows stay blank, and NI has no council tax.
Adds storage/build_la_council_tax.py, which downloads the workbook,
parses the ODS with zipfile plus ElementTree (no odfpy dependency),
re-maps the post-2023 South Yorkshire ONS codes to the pre-2023 codes
used by local_authorities_2021.csv, and rewrites only the new columns.
Every pre-existing cell is round-tripped as its literal text, so the
build is idempotent.
band_d_amount already held the 2026-27 Table 9 Band D figure for all
294 directly-matching English LAs, and _YEAR_BAND_D_ENGLAND / the
England reference URL already point at 2026-27, so no year attribution
changed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Sep 18, 2026
Council tax integration: Nuffield Workstream 2 deliverable tracker
PolicyEngine/policyengine-uk#1854
Open
juaristi22
approved these changes
Sep 18, 2026
juaristi22
left a comment
juaristi22
left a comment
Collaborator
There was a problem hiding this comment.
Approving as a data-only change. The useful result is the ratio check: all 296 authorities are exact ninths of Band D to within half a penny, so the eight columns carry nothing beyond Band D and the statutory ratios. That settles how Microcosm treats them: no port and no Chronicle ask, since our route is the published Band D fact times the ratio parameter, and, as #484 says, none of these amounts may ever become an additive target.
Three nits, none blocking:
band_D_amountduplicatesband_d_amountfor England by construction. The test guards the pair, but one column is simpler and cannot desync on a refresh.- The builder rewrites
band_d_amountfor England from Table 9 whenever it differs by more than half a penny. That is a no-op today, but on a future refresh it silently moves that column's provenance from Table 10 to Table 9 without a log line. Print the count of rewritten Band D cells. - The hard-coded asset URL is fine for now; scraping the release page the way
voa_council_tax.pydoes can wait for the 2027-28 refresh.
Once this and #484 both land, we re-pin the Microcosm inventory sha256s for the loader and the test file.
🤖 Generated with Claude Code
Drops the duplicate band_D_amount column (nothing in uk-data reads it; English values were identical to band_d_amount), so the pair cannot desync on a refresh. The builder now prints how many band_d_amount cells it rewrote from Table 9, since each rewrite moves that cell's source from Table 10. Addresses review nits on #482. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Review of #482 — Extend LA council tax to all eight bands (MHCLG Table 9, 2026-27)Base repository: PolicyEngine/policyengine-uk-data Source DocumentsCriticalNone. Should AddressNone. SuggestionsNone. Evidence GapsNo material gaps reported. Notes
Validation Summary40 passed, 14 skipped, focused snapshot module tests; official workbook builder reproduced committed CSV twice byte-identically; original 16 cells per row preserved; 3231 emitted targets exact equality to merge-base; pr-482-focused-pytest.log/pr-482-builder-diagnostic.log/pr-482-cell-preservation-diagnostic.json 2368 of 2368 official Table9 values matched exactly; all 296 code mappings checked; year £ units precepts country coverage and notes p/ac established; source/derivative manifest integrity passed; 2 originals 1 extract no PDF renders. CI at 4fc43cc: Check changelog fragment=SUCCESS, Lint=SUCCESS, Validate bundle release manifest contract=SUCCESS, Test=IN_PROGRESS. Detailed code, policy, source and reproduction artifacts were retained locally. Timingelapsed seconds: 1115.90s; setup seconds: 21.75s Review SeverityAPPROVE. Open findings: 0 critical, 0 should address, 0 suggestions. |
juaristi22
added a commit
to PolicyEngine/microcosm
that referenced
this pull request
Oct 15, 2026
_local_contract_covers still keyed the local council-tax rows on the voa.council_tax_stock.by_area. prefix that #929 retired, so the 25 mhclg, welshgov and scotgov by-area rows covered only the LA loss module. They now cover uk-data's la_council_tax source as well, ahead of the re-pin that follows PolicyEngine/policyengine-uk-data#482 and PolicyEngine/policyengine-uk-data#484. Three stale texts are corrected while the register regenerates: the English region concern names the taxbase rows and the nation grain, and the two private-rent concerns describe the calendar-2025 PIPR series in the pinned feed, not a June 2026 snapshot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22
added a commit
to PolicyEngine/microcosm
that referenced
this pull request
Oct 15, 2026
_local_contract_covers still keyed the local council-tax rows on the voa.council_tax_stock.by_area. prefix that #929 retired, so the 25 mhclg, welshgov and scotgov by-area rows covered only the LA loss module. They now cover uk-data's la_council_tax source as well, ahead of the re-pin that follows PolicyEngine/policyengine-uk-data#482 and PolicyEngine/policyengine-uk-data#484. Three stale texts are corrected while the register regenerates: the English region concern names the taxbase rows and the nation grain, and the two private-rent concerns describe the calendar-2025 PIPR series in the pinned feed, not a June 2026 snapshot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Calibration fit against the base build: pass
A drop of more than 2.0% fails. Notes
National targets in a worse error band on the head (9)
Flags are for review: builds of the same code move 2 to 12 national targets to a worse band (#598). Bands are of |estimate / target - 1|; no target's estimate or error is published. Income distribution and single ages against the base build: passHBAI FYE 2025 (DWP), period 2024: equivalised income in £ a week, rates and Gini in per cent. A value outside HBAI's 95% interval is marked (outside).
FlagsNone. These measures report and flag but do not fail yet: their same-code spread is unmeasured (#598). Aggregate impact of the data change: passBase dataset Checks: 0 broken by this change, 4 failing on both sides, 0 fixed. Moves flagged for review: 148 (thresholds in Notes
Moves flagged for review
108 more flags are in the report artifact. Receipts by tax head
Spending by benefit and service
Caseloads
Government balance, each component counted once
Model totals
Components the model counts twice
Household income totals
Population
Poverty rates
People in poverty
Inequality
Mean household net income in each side's own deciles
Standard reform costings (change in gov_balance)
Standard reforms: change in each reform's target
Household net income by decileThe datasets hold different households, so households are not matched; compare the mean income in each side's own deciles. Checks failing on both base and head (4)
Counted-once checkCounted-once check not run: not requested. Provenance
Base dataset: Head dataset: Receipts by tax head: head levels
Spending by benefit and service: head levels
Caseloads: head levels
Government balance, each component counted once: head levels
Model totals: head levels
Components the model counts twice: head levels
Household income totals: head levels
Population: head levels
Poverty rates: head levels
People in poverty: head levels
Inequality: head levels
Mean household net income in each side's own deciles: head levels
Standard reform costings (change in gov_balance): head levels
Standard reforms: change in each reform's target: head levels
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
storage/la_council_tax.csv: adds eight columns,band_A_amount…band_H_amount, populated for all 296 English billing authorities. All pre-existing columns and all 360 rows are untouched — verified cell-by-cell againstHEAD(0 rows of drift in the original 16 columns).storage/build_la_council_tax.py(new): reproducible builder. Downloads the MHCLG workbook, parsesTable_9from the ODS withzipfile+xml.etree(honouringtable:number-columns-repeated; noodfpy/pandas ODS engine needed), re-maps the post-2023 South Yorkshire ONS codes, and rewrites only the new columns. Every other cell is round-tripped as its literal text, so a re-run is a no-op (md5-identical on the second run).targets/sources/la_council_tax.py: module docstring only — documents the new columns, their England-only provenance, and the rebuild script. No behaviour change;get_targets()output is unchanged.tests/test_la_council_tax_targets.py: seven new tests (see below).changelog.d/la-council-tax-all-bands.added.md.There is no existing builder directory in this repo —
la_council_tax.csvwas hand-built in #374 with no committed script. I put the builder inpolicyengine_uk_data/storage/, alongsidedownload_private_prerequisites.py/upload_completed_datasets.py, as the closest existing pattern. Happy to move it if you'd rather it lived elsewhere.Data source
MHCLG, Council tax levels set by local authorities in England: 2026 to 2027
https://www.gov.uk/government/statistics/council-tax-levels-set-by-local-authorities-in-england-2026-to-2027
Workbook:
https://assets.publishing.service.gov.uk/media/69de1fa63e81003ae0422508/Tables_1-9_2026-27.ods, sheetTable_9— "Area council tax for a dwelling occupied by two adults by band, 2026 to 2027". 296 English authority rows, bands A–H, inclusive of all precepts.England only. Table 9 has no Welsh or Scottish coverage, so the 22 Welsh, 32 Scottish and 10 NI rows leave all eight new columns blank rather than inheriting an English figure. Wales additionally has a Band I and Scotland has used non-statutory band ratios since the 2017 reform, so neither could be back-filled from Band D anyway. A test enforces the blanks.
Band-ratio verification
Checked every one of the 296 authorities against the statutory ratios (A=6/9, B=7/9, C=8/9, D=1, E=11/9, F=13/9, G=15/9, H=18/9 of Band D). All 296 obey the ratios. Worst absolute deviation by band:
That is pure independent-to-the-penny rounding (at most half a penny per band). No authority deviates and nothing in the source was altered. The test tolerance is £0.01, comfortably above the observed £0.0044 worst case.
Year attribution (step 3) — no change was needed
I checked before touching anything:
band_d_amountfor England already holds the 2026-27 Table 9 Band D figure, exactly, for all 294 English LAs whose ONS code matches directly (max difference £0.00); the remaining two are the South Yorkshire code re-map below, which also match._YEAR_BAND_D_ENGLANDis already2026and_ENGLAND_REFalready points at the 2026-27 release.So
_year_for_band_d,_ref_for_band_dand the reference URL are unchanged, and no target's year attribution moved. The builder still writesband_d_amountfrom Table 9 so the file stays reproducible, but on today's data that write is a no-op. The newband_D_amountcolumn duplicatesband_d_amountfor England by construction; a test asserts they agree exactly.South Yorkshire code re-map
MHCLG uses the post-2023 codes
E08000038(Barnsley) andE08000039(Sheffield);local_authorities_2021.csvusesE08000016/E08000019. The builder re-maps them, confirmed by authority name. Their Band D values also matched the existing CSV exactly.Tests
Added to
test_la_council_tax_targets.py:band_*_amountcolumns exist;band_D_amount == band_d_amountfor England.uv run ruff format .reformatted my two files (now clean).uv run ruff check .reports 61 errors — the identical count with my changes stashed, i.e. entirely pre-existing onorigin/main(verified by stashing and re-running, not assumed). None are in code this PR adds.What I could NOT verify / gaps
get_targets()is unchanged, so the band-amount columns are currently data-only, for downstream band-level work. Wiring them into the loss matrix would ripple intotest_la_loss_council_tax.pyand felt out of scope here — flagging as follow-up rather than doing it silently.assets.publishing.service.gov.ukmedia path rather than scraped from the release page (asvoa_council_tax.pydoes). It will need updating for 2027-28. Scraping would be more robust; say the word and I'll switch it.🤖 Generated with Claude Code