[juaristi22]

This PR carries the facts chronicle#395 asks for, for the rows of PolicyEngine/microcosm#1095's next policyengine-uk bump. Closes #395. Its two asks for later editions, the next Childcare Offer for Wales report and the 2027 CPI weights, are #398. One ask has no published source (Wales's bus fare receipts), and the PR says so below rather than deriving anything.

What it adds

Thirteen new packages (1,075 facts) and one line in the March 2026 EFO receipts package (1 fact). scripts/build_uk_395_packages.py writes the new packages from the pinned bytes; --check compares instead of writing.

• Item 1, Northern Ireland domestic rates. No published table gives the amount levied on domestic property. This item carries the two that give domestic receipts, and the two domestic reliefs that have statistics of their own.

  • lps-domestic-rates-paid-2026 (144): LPS's domestic rates paid against the year's accounts by 31 March, and the domestic occupancies that made a payment, for the 11 councils and Northern Ireland, rating years 2020-21 to 2025-26. 2024-25: GBP 889,543,160, the figure Give both OBR placements of Northern Ireland domestic rates in the fiscal partition policyengine-uk#2663 cites.
  • ons-crpsf-ni-domestic-rates-fye2025 (52): ONS country and regional public sector finances, Northern Ireland's regional and district domestic rates, 1999-00 to 2024-25 (2024-25: GBP 498m and GBP 441m). Each is its own concept; no total is derived.
    • The workbook doesn't state whether these rows are accrued or cash, or gross or net of reliefs and rebates, so they are a different measure from LPS's cash paid.
    • LPS's dataset doesn't say whether credits such as rate rebate count as payments. The docs say both.
  • lps-lone-pensioner-allowance-ye-2024 (420):
    • owner occupiers with a current award at 31 March, and the net amount payable to them over each financial year, by council and for Northern Ireland;
    • Northern Ireland's net amounts for private renters, NIHE and housing association tenants and all tenures (GBP 9,241,619 in 2023-24);
    • years 2009-10 to 2023-24, with every amount under one concept and its tenure as lpa_tenure.
  • lps-disabled-persons-allowance-ye-2024 (360): properties with a current award and the net amount awarded over each financial year, by council and for Northern Ireland, 2009-10 to 2023-24 (GBP 4,687,743 in 2023-24).
  • The relief statistics stamp counts by the calendar year of the 31 March they are current on, and amounts by the financial year's opening year. Tables that restate ported rows (LPA 1.3 and 1.4, DPA 1.3) are asserted equal and not ported again.
  • Rate rebate has no published series by year, and LPS's Trust Statement doesn't split its rate revenue by property type, so the sources don't give the whole bill net of reliefs.

• Item 2, Wales.

  • dft-bus01-wales-journeys-2025 (57): BUS01a passenger journeys and BUS01c concessionary journeys for Wales, read from the workbook dft-bus01-passenger-journeys-2025 already pins (that package leaves the devolved columns out of its scope). Year ending March 2025: 71.7m journeys, 32.3m of them concessionary.
  • The same journeys sit in welshgov-bus-statistics-2024-25, stamped by opening year. This package's fiscal_year 2025 is that package's fiscal_year 2024, and the header and docs say to join on the coverage dates.
  • Passenger fare receipts are not published for Wales. DfT splits revenue by type for England only (BUS05ai), BUS04ai's Wales operating revenue includes BSOG, concessionary reimbursement and support (its note 1), and the Welsh Government release doesn't present the receipts its survey collects.

• Item 3, welshgov-childcare-offer-report-2023-24 (3). The report's Table 4 for September 2023 to August 2024:

  • 21,283 children in Offer-funded childcare (administrative);
  • an estimated eligible population of 36,600 (model output: "Welsh Government analysis of the ONS Annual Population Survey 2023");
  • an estimated 58% take-up.

  No later report is published yet.

• VED (comment of 29 October).

  • The OBR's 'Vehicle excise duties' line (sheet 3.8, cash basis) joins the March package as a March line, obr.vehicle_excise_duties, for its 2024-25 outturn only (GBP 8.205bn).
  • Of two vintages' numbers for one line and year Chronicle keeps the later, so October carries 2025-26 on and the two vintages share no VED key. The October package carried VED as a table-scoped row (obr.cash_receipts.vehicle_excise_duties); its build now carries it under the March line id and concept, with the same cells, values and assertions.
  • The other March lines' overlap with October predates the rule and is chronicle#397.
  • The split by payer is published and ported: ONS CDDZ (vehicle duty paid by households, 'Other taxes') and EKED (by businesses, 'Taxes on production'), calendar years 2018 to 2025, in ons-pusf-vehicle-duty-{households,businesses}-october-2026 (16). The series state no accounting basis, so use them for shares, not as levels against the OBR's cash line.

• Bottled gas (comment of 29 October).

  • ONS MM23 CPI weights for 04.5.2 (CJXB), 04.5.2.1 (L89X) and 04.5.2.2 (L89Y), weight years 2024 to 2026 (9).
  • The two classes' indices L7E9 and L7EA: annual averages 2023 to 2025 and January values 2023 to 2026 (14).
  • L89X's and L89Y's metadata leave the unit blank; they sum to CJXB's parts per thousand in each year, which the tests assert.

Licences

• All the sources are OGL v3.0.
• OpenDataNI's record for the LPS rates-paid dataset says "License not specified". nidirect's Open data page states that all data on the OpenDataNI portal falls under the OGL. The manifest quotes both, and the CSV is committed.
• The two allowance workbooks state the OGL on their cover sheets.

A parser change

LPS prints amounts as text such as '£51,877,954'. _delimited_scalar (in chronicle/sources/rows.py and cells.py) already strips '$', and now strips '£' too. No artifact that reaches that function held a '£' followed by a digit before. DWP's Stat-Xplore band labels such as '£200' are parsed by their own parser and are unchanged.

Pins

These are measured from partial bundles of the new packages, both OBR receipts packages, and the packages that share their dimensions (the DfT, Scottish and LPS packages). Those bundles are valid, with no warning beyond the bundle's standing duplicate-key one.

• fact_count/row_count 1,661,755 → 1,662,831 (+1,076).
• By source: dft +57, lps +924, obr +1, ons +91, welshgov +3.
• Packages 428 → 441, and source tables 370 → 383.
• country:K02000001 40,878 → 40,918.
• Entities: households +227, people +481, dwellings +360, firms +8.
• Period counts: existing keys move, and none is new.
• No new geography key, no new warning, and semantic duplicate keys unchanged at 2,687.
• The full merged-bundle test needs the R2-only artifacts, which aren't cached locally, so CI runs it.

Role and paths

ledger-source-ingestor. Paths outside the role's list, as #391, #392 and #394 declared:

• db/data/**;
• chronicle/source_package.py and chronicle/bundle.py (registration);
• docs/** and changelog.d/**;
• scripts/build_uk_395_packages.py (new), and VED in scripts/build_uk_excise_packages.py and scripts/build_obr_efo_october_2026.py.

The parser change sits in chronicle/sources/**, which is inside the role.

Checks run

• validate-package, the consumer contract, build-suite and agent acceptance on all thirteen new packages and the two changed OBR packages: valid, with no warnings, no issues and no duplicate key within any package.
• publish-raw uploaded the twelve new artifacts, and verify-raw confirms each object's sha256.
• tests/test_source_uk_395_facts.py: 35 passed. It covers:
  • registration, the generator rebuilding each YAML byte for byte, the manifests and the '£' parse;
  • values re-read from each file with csv, openpyxl and pandas;
  • LPS's councils against their total, LPA's tenures against their total, and ONS's NI sheet against the UK sheet;
  • L89X + L89Y = CJXB, the annual indices as monthly averages, and VED's years split across the two vintages.
• tests/test_issue_381_uk_excise_facts.py and tests/test_source_obr_efo_october_2026.py pass, including the October generator's byte-for-byte check and March-line parity.
• The parser, registry, OBR, bus, pension and source-package tests, and tests/test_chronicle_bundle.py without the merged build: 464 passed.
• ruff check on every edited Python file.

Review

Vahid's first round (comment of 30 October) is addressed: the relief statistics (1), the LPS and ONS bases (2), the Wales stamps (3), APS 2023 (4), the PUSF basis (5) and --check (6).

🤖 Generated with Claude Code

[vahid-ahmadi]

Review of f343b52 (draft, so a comment rather than a formal review). First round; no earlier comments or threads.

The numbers are right. Every fact I re-read from the pinned bytes matches, and every pinned file I could fetch matches the publisher's current file. No musts. Three shoulds and three nits, mostly about what the docs tell Microcosm.

Findings

1. should: published relief figures exist for two domestic reliefs.
The PR body says only the Trust Statement lists reliefs, and docs/data-sources/uk-facts-395.md:74 gives that as the reason none is ported. LPS also publishes Lone Pensioner Allowance and Disabled Person's Allowance statistics (OGL, 29 May 2024): publication page, files lps-lpa-ye-2024.XLSX and lps-dpa-ye-2024.XLSX. They give the net amount awarded per financial year, by council (owner occupiers) and for NI, for years ending March 2010 to March 2024. That is the latest edition I found; no -ye-2025 file exists at the same path.

• LPA total, Table 1.5: £9,241,619 in 2023-24;
• DPA, Table 1.3: £4,687,743 in 2023-24.

Both reliefs are domestic only, so the "revenue isn't split by property type" reason doesn't apply to them. They are small and end at 2023-24, and rate rebate is still missing, so they don't give the whole bill net of reliefs. But #395 asks for reliefs "as separate facts where the source gives them". Please port them, or name them in the docs and on #395 with the reason for leaving them out.

2. should: say how the LPS and ONS figures relate.
For 2024-25, ONS gives £498m regional plus £441m district, £939m in all. LPS gives £889.5m paid. The gap is £49.5m (5.6%). The docs (uk-facts-395.md:43, :64-72) present both as "domestic receipts" but don't explain the gap. Neither package states the ONS basis: national accounts accruals or cash, and gross or net of reliefs and rebates. Microcosm has to choose one of these to bind, so one line on each basis would help, even if it only says "not stated by the publisher".

A data point for #1095: #395 names two OBR lines, £494m (expenditure 4.1, obr.domestic_rates) and £485.3m (receipts 3.2). Both are near ONS's regional rate (£498m), and neither is near the district rate (£441m). The bases may differ, so this doesn't settle which OBR line is which.

3. should: Wales's bus journeys are now in Chronicle twice, one year apart.
welshgov-bus-statistics-2024-25 already carries 71.7m for 2024-25 as welshgov.bus.passenger_journeys, fiscal_year 2024 (opening year). The new package carries the same 71.7m as dft.local_bus_passenger_journeys, fiscal_year 2025, because DfT stamps by closing year (bus01_wales_journeys_2025/source_package.yaml:1434). Both are allowed by chronicle/core.py, and both have correct coverage dates. But a consumer that joins on the period value gets 71.4m from one package and 71.7m from the other for "2024". The concessionary journeys exist only in the DfT package, so Microcosm will use that stamp. Please add a sentence near uk-facts-395.md:91 and in the package header: DfT's fy2025 is Welsh Government's fy2024, and the series is the same.

4. nit: record that the eligible population comes from APS 2023.
Table 4's source note says "Welsh Government analysis of the ONS Annual Population Survey 2023". The header (childcare_offer_report_2023_24/source_package.yaml:8) says only "modelled from the ONS Annual Population Survey". The denominator rests on calendar 2023 data while the fact is stamped academic_year 2023. Please add the survey year to the eligible-population notes.

5. nit: give the PUSF split's basis.
The OBR VED facts carry accounting_basis: cash. CDDZ and EKED carry no basis and are calendar-year national accounts figures. The header (pusf_vehicle_duty_households_october_2026/source_package.yaml:7) says "national accounts' split" but not accrued against cash. For the record: households paid 61% in 2025 (5,535 of 9,036), against OBR's £8.205bn cash for 2024-25. One line saying the split is for shares and not levels against the OBR line would stop a basis mix.

6. nit: scripts/build_uk_395_packages.py has no --check.
main() (:1350) ignores its arguments, so --help rewrites the eleven packages. My run left the tree clean, and the test compares the YAML byte for byte, so nothing is wrong. A --check flag like the October script's would match the other generators.

Verdict

Sound and carefully sourced. The two "no published source" statements are honest for Wales fares, and nothing is derived. For NI reliefs, item 1 qualifies that statement. Items 1 to 3 are worth doing before this leaves draft, and the bundle job needs to go green.

What I ran

Fresh clone at f343b52, fresh venv, deleted afterwards.

Hashes. Fetched from the publishers today, all byte-identical to the pins:

• the LPS CSV (OpenDataNI);
• the CRPSF workbook;
• the Childcare Offer PDF;
• the seven ONS series CSVs;
• BUS01 (51b5aa5e…).

Facts re-read independently. I read the source files myself (csv, openpyxl, pandas, pypdf) and compared them with build_facts:

• LPS: all 144 match. Councils sum to NI within £2, counts exactly. The LGD 2014 codes are correct and match lps-housing-stock-lgd-2026.
• CRPSF: all 52 match rows 51 and 52, which equal the UK sheet's in every year.
• BUS01 Wales: all 57 match columns R and P, ×10^6. YE March 2025: 71.71m journeys, 32.29m concessionary.
• Childcare Offer: Table 4 gives 21,283, 36,600 and 58%.
• CDDZ and EKED: all 16 match.
• CJXB, L89X and L89Y: all 9 match, and the classes sum to CJXB in each year.
• L7E9 and L7EA: all 14 match. The January values are month periods, so they don't share keys with the annual averages.

OBR VED.

• March sheet 3.8 row 52 and October 4.8 row 52 match the 14 facts.
• A base-against-head fact diff of both receipts packages: +7 March and 7 October facts re-keyed (the same values, assertions and basis), nothing else changed.
• The 5 new shared keys are 2026-27 to 2030-31, as the body says.

Pins. From the built facts: households +227 (144 + 52 + 8 + 9 + 14), people +67, firms +8, country:K02000001 +46, and by source dft +57, lps +144, obr +7, ons +91, welshgov +3. All match the body.

Labels and parser.

• Group and measure labels in the 11 new packages: no stray whitespace or tabs.
• The '£' change: every other text artifact under db/data with '£' followed by a digit is read by html_tables_and_text or the Stat-Xplore parser. Neither calls _delimited_scalar, so no other package changes.

Overlaps. No existing package carries CPI weights, the two class indices, CRPSF, PUSF vehicle duty or LPS rates paid. The only overlap is item 3.

Commands.

• build_uk_395_packages.py and build_uk_excise_packages.py re-run: tree clean.
• build_obr_efo_october_2026.py --check: clean.
• Pytest:
  • test_source_uk_395_facts.py, test_issue_381_uk_excise_facts.py, test_source_obr_efo_october_2026.py, alias drift, and source rows and cells: 134 passed.
  • test_chronicle_bundle.py without the merged build: 26 passed.
  • test_chronicle_source_package.py -k "lps or ons or dft or welshgov or obr": 16 passed.
• validate-package on the 13 new or changed packages: valid, no warnings.
• ruff 0.14.10 (locked), over CI's lint paths: passed.

Not verified

• The merged-bundle test, its absolute pins and "no new period key". CRPSF reaches back to 1999-00, so the period key check is CI's. CI's Chronicle bundle and Chronicle surface jobs were both pending on f343b52 when I posted.
• The licence point, beyond reading both statements. On your question, I think nidirect's portal-wide OGL statement is enough to commit the CSV.
• Welsh Government and StatsWales having no Welsh fare receipts. I checked only that BUS05i has no Wales columns and that BUS04i's Wales revenue is total operating revenue.
• The Trust Statement editions after 2022-23.

🤖 Generated with Claude Code

[juaristi22]

Thanks, Vahid. All six are addressed in 2f4c387 and ec9a9e9, and the body is updated.

1. Relief statistics. Both are ported:
   • lps-lone-pensioner-allowance-ye-2024 (420 facts): owner occupiers with a current award and the net amount payable to them, by council and for NI (Tables 1.1 and 1.2), and Table 1.5's NI amounts for the other tenures and in total. Every amount is one concept with its tenure as lpa_tenure.
   • lps-disabled-persons-allowance-ye-2024 (360): properties with a current award and net amounts, by council and for NI (Tables 1.1 and 1.2).
   • Both run 2009-10 to 2023-24. Counts are stamped by the calendar year of the 31 March they are current on, and amounts by financial year.
   • The tables that restate ported rows (LPA 1.3 and 1.4, DPA 1.3) are asserted equal and not ported again. DPA's 2017-18 amount differs between Tables 1.2 and 1.3 only in float noise.
   • The docs now say rate rebate has no published series, so the sources still don't give the whole bill net of reliefs.
2. LPS against ONS.
   • The docs and the CRPSF header now say that the workbook states no basis: neither accrued nor cash, nor gross or net of reliefs and rebates. So the ONS rows and LPS's cash paid are different measures.
   • They also say that LPS's dataset doesn't say whether credits such as rate rebate count as payments.
   • I've left the OBR-line reading to #1095.
3. Wales stamps. The BUS01 Wales header and the docs now say that DfT's fiscal_year 2025 is the Welsh Government package's fiscal_year 2024 (and likewise a year back), the same series, so join on the coverage dates.
4. APS 2023. The eligible-population notes quote Table 4's source line ("Welsh Government analysis of the ONS Annual Population Survey 2023"). The header says it is calendar-2023 data behind a fact for the 2023-24 Offer year.
5. PUSF basis. The headers and docs now say that the series state no accounting basis, so use the split for shares, not as levels against the OBR's cash line.
6. --check. scripts/build_uk_395_packages.py --check compares instead of writing, as the October script does.

One more change. María's rule is that of two vintages' numbers for one line and year Chronicle keeps the later. So the March package now carries only VED's 2024-25 outturn, October carries 2025-26 on, and the two share no VED key: semantic duplicates are back at 2,687. The older March lines' overlap is #397. The pins moved to match: +1,076 facts against main.

🤖 Generated with Claude Code

[vahid-ahmadi]

Review of ec9a9e9, round 2

Thanks, María. All six round-1 items are done, and the 780 new relief facts match LPS's files exactly. One small nit below, no musts or shoulds.

Round 1

1. Relief statistics: done. lps-lone-pensioner-allowance-ye-2024 (420) and lps-disabled-persons-allowance-ye-2024 (360), built at scripts/build_uk_395_packages.py:462-901, documented at docs/data-sources/uk-facts-395.md:86-112. Checks below.
2. LPS against ONS: done. uk-facts-395.md:76-84 and the CRPSF header (crpsf_ni_domestic_rates_fye2025/source_package.yaml:12) say the workbook states no basis, and that the two are different measures.
3. Wales stamps: done. bus01_wales_journeys_2025/source_package.yaml:14-18 and uk-facts-395.md:130-134. I checked the pairing: DfT 2025, 2024 and 2023 have the same coverage dates as Welsh Government 2024, 2023 and 2022 (71.7m, 71.4m, 61.2m rounded).
4. APS 2023: done. childcare_offer_report_2023_24/source_package.yaml:8 and the measure notes (:93, :159).
5. PUSF basis: done. Both PUSF headers (:9-10) and uk-facts-395.md:197-198.
6. --check: done. build_uk_395_packages.py:1829-1852. It exits 0 on the clean tree. When I appended a line to the LPA YAML, it printed stale: for that file, exited 1 and wrote nothing.

The relief packages

• Hashes. I fetched lps-lpa-ye-2024.XLSX and lps-dpa-ye-2024.XLSX from finance-ni.gov.uk today. Both are byte-identical to the pins (f2ea22b1…, 41b5aaa7…). Both cover sheets carry the OGL statement and "Published: 29 May 2024", as the manifests quote.
• Values. I read all five LPA tables and three DPA tables with openpyxl, separately from Chronicle's parser. All 780 built facts match on concept, tenure, geography, period and value: none missing, none extra, none different. The 780 semantic keys are distinct.
• 2023-24. LPA all tenures £9,241,619.46. DPA £4,687,742.79.
• Restated tables. The asserts at :743-752 (LPA 1.3 against 1.1 and 1.2, 1.4 against 1.5, 1.5's column B against 1.2) and :863-864 (DPA 1.3) are real. They run in CI, because test_source_uk_395_facts.py:94 runs every builder.
• DPA 2017-18. Table 1.2 has 3374738.5700000003 and Table 1.3 has 3374738.57, so the difference is only float noise. The 1e-4 tolerance at :685 is fine.
• LPA tenures. The four tenure columns sum to Table 1.5's total to within 2e-9 in every year.
• Stamps. Counts are calendar years dated 31 March, and amounts are fiscal years stamped by their opening year. This matches lps-housing-stock-lgd-2026 (calendar year dated 1 April) and lps-domestic-rates-paid-2026 (fiscal year, opening year). Using dwelling for DPA and person for LPA follows what each table counts.

The VED vintage change

• Consumers. Nothing reads March's dropped VED years.
  • March's VED line is new in this PR, so main never had those years.
  • Outside the two receipts packages, vehicle_excise_duties appears only in the two test files, the build script and the docs.
  • build_obr_efo_october_2026.py --check is clean, so the October build still finds the line through March's 2024-25 record set.
  • Alias drift passes.
• Pins. They reconcile against main (53815b2): 1,661,755 → 1,662,831 is +1,076 = 295 (the eleven UK facts for the next microcosm policyengine-uk bump: Northern Ireland's whole domestic rates bill, Wales's local bus fare receipts and concessionary journeys, and the Childcare Offer for Wales take-up #395 packages) + 780 (the relief packages) + 1 (March's VED outturn). Against round 1's +302: +780 for the relief packages, and −6 for March's dropped VED years. By source: lps 72 → 996 (+144 +780) and obr +1. Semantic duplicates are back at main's 2,687, and the period key set is unchanged.
• Against Add SPI 2023-24 Table 3.10 and the OBR October 2026 EFO vintage (PolicyEngine/microcosm#1418) #394. Add SPI 2023-24 Table 3.10 and the OBR October 2026 EFO vintage (PolicyEngine/microcosm#1418) #394 kept both vintages side by side (tests/test_chronicle_bundle.py:525-528: "kept as published beside the March package"). So "keep the later" is a new rule, and this PR is the first place it is applied. OBR EFO March 2026: drop the years the October 2026 vintage also publishes (PolicyEngine/microcosm#1095) #397 is open and states the rule. It sizes the rest (426 March facts to drop, 74 to keep) and gives the timing constraint against the Microcosm re-pin. It covers the remaining lines.
• Microcosm timing. No Microcosm VED row reads March's later years today (#1095 has VED waiting on the NTS-vehicle work). A VED row that binds the calendar-2025 window will need October for 2025-26 from the start, as uk-facts-395.md:173-177 says.

New findings

1. nit: DPA's published councils don't sum to Northern Ireland's count in one year. In Table 1.1, the eleven councils sum to 10,675 at 31 March 2010, but the NI row prints 10,674. Every other DPA and LPA year sums exactly, and the facts carry what the sheet prints. The rates-paid section already flags its own £2 gap "as published" (uk-facts-395.md:63-65). A matching line under DPA (:98-101) or in its header would save someone chasing it.

Verdict

Ready from my side once CI's bundle job is green. The nit is optional.

What I ran

Fresh clone at ec9a9e9 and a fresh venv, both deleted afterwards.

• build_uk_395_packages.py --check, build_obr_efo_october_2026.py --check: clean. build_uk_excise_packages.py (no --check) re-run: tree clean.
• Pytest:
  • test_source_uk_395_facts.py, test_issue_381_uk_excise_facts.py and test_source_obr_efo_october_2026.py: 75 passed.
  • Alias drift and source rows: 50 passed.
  • test_chronicle_source_package.py -k "lps or obr or dft or ons or welshgov": 16 passed.
  • test_chronicle_bundle.py: 26 passed, 1 skipped (the merged build).
• validate-package on the two new and eight changed packages: valid, no errors or warnings.
• ruff 0.14.10 (locked) over CI's lint paths: passed.

Not verified

• The merged-bundle test and its absolute pins. I reconciled the pin deltas by hand from the diff. CI's Chronicle bundle and Chronicle surface jobs were still pending on ec9a9e9 when I posted.
• OBR EFO March 2026: drop the years the October 2026 vintage also publishes (PolicyEngine/microcosm#1095) #397's counts (426 and 74). I read the issue but didn't recount them.
• Whether LPS has published anything after the 2024 edition beyond the publication page, or any rate rebate series.

🤖 Generated with Claude Code