Skip to content

Budget 2026 slower data fixes, PR C of #1418: electricity at the ONS level, pension income by area - #1420

Open
juaristi22 wants to merge 46 commits into
mainfrom
uk-budget-2026-data-fixes-c-1418
Open

juaristi22 wants to merge 46 commits into
mainfrom
uk-budget-2026-data-fixes-c-1418

Conversation

@juaristi22

@juaristi22 juaristi22 commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

PR C of microcosm#1418 covers the Budget 2026 data fixes that weren't quick ones. It is stacked on PR A (#1419), whose branch is merged in. #1418 stays open.

Ready for review. Merging waits on CI at the head and on María. The feed is re-pinned onto PolicyEngine/chronicle#394's merge (53815b2), which carries SPI Table 3.10 and the October 2026 EFO. Anthony's two rounds, Vahid's review and the independent review are answered (commits 36 and 37), and so are the six re-reviews since (commits 38, 39, 41, 42, 43 and 44).

A national built from commit 40 (staging only) stopped at certification on two £1m-and-over SPI rows. Whether to hold the new £1m-and-over other-income row out, as the ten other £1m-and-over cells are held, is María's call; the national is rebuilt after it.

A deadline: from 2026-11-27, test_no_committed_exclusion_is_about_to_lapse (commit 42) fails on every PR until orr.rail_government_support, which expires on 2026-12-11, is renewed or bound.

Not in this PR:

Commits, in order

  1. The energy stage's level becomes ONS household spending (UK: data fixes for the Budget 2026 household measures #1418 item 12; the commit itself is titled for electricity, and gas follows in commit 6).

    • Before: the lcfs_consumption stage raked kWh to DESNZ Energy Trends' calendar-2024 volume. Priced at QEP, that sits about 5% above ONS Consumer Trends for both fuels. The 30 September 2026 release revised 2024 spending up, electricity from £26.10bn to £28.13bn and gas from £14.64bn to £18.30bn.
    • The new declaration: a fuel declared in spend_level_fuels takes the kWh ONS household spending buys: its class (04.5.1, 04.5.2) for 2024, less QEP's UK average fixed cost on every UK domestic meter DESNZ counts (Great Britain's sub-national count plus Northern Ireland's row, now vendored), divided by QEP's UK average unit cost. Electricity: 84.9 TWh, against Energy Trends' 91.7 TWh.
    • Expected frame spend: on the last national's design weights, each fuel's frame spend sits about 1% below ONS, where Energy Trends put it about 5% above. The 1% is the standing charges of meters that aren't frame households' main homes. The next build measures it.
    • One function, three places: energy_pricing.published_energy_level carries the level to the stage's rake, to the uk_stage_lcfs_consumption_energy_rake gate's recomputation, and to the factor that levels the dense build's local DESNZ kWh rows. So the 980 local electricity rows move with it (Great Britain's cells now sum to 82.7 TWh, not 89.2), and no £/kWh bridge is needed.
    • Unchanged: the H5's energy_price_basis stays calendar 2024.
    • Text updates: the texts that described the level as Energy Trends now say what it is.
  2. The Gambling Commission's machine yield by category is vendored (item 5, row 56).

    • OOTLAR 1.50 moves machines with a maximum charge above £1, or a maximum prize above £100, to 25% from 1 April 2027. That means Category B.
    • excise_facts.json gains the Commission's FY2024-25 machine GGY by category for Great Britain, all venues: Category B £2.29bn of £2.59bn.
    • The split itself is held for the engine input.
  3. Re-pin of the gate digests and the coverage manifest.

  4. SPI pension income by area (item 14).

    • hmrc.pension_income.amount: each area's taxpayers' private and State Pension income together, SPI 2023-24 Tables 3.15 and 3.14, count times mean.
    • It grows at ITL Table 2.5's all-ranges total income growth (×1.1340). The national State Pension and private pension bands grew ×1.1348 together, a 0.1% gap, before UK pensions after #1084: open rulings, residual gaps and the engine pieces #1069's State Pension index: ×1.151 since, a 1.5% gap (commit 36).
    • A fan-out sum bridge reconciles the cells per grain to those two families summed. This is the first bridge to sum several fan-outs, and it refuses a control that is missing one of them.
    • At source the bands sit 3.0% above the constituency cells and 3.2% above the authority cells. The area tables leave out the taxpayers HMRC places in no area, who are more often pensioners: ITL Tables 2.1 and 2.2 put 1.8% of taxpayers of State Pension age in no region, against 0.8% of all taxpayers. The bridge therefore allowed 4% at source and 0.6% in the index gap. Commits 27 and 36 revise both: 2.5% to 3.7% at source, and 2% in the index gap after UK pensions after #1084: open rulings, residual gaps and the engine pieces #1069's State Pension index.
    • The frame reads total_pension_income_annual.
  5. Merge of PR A, then a test-stub fix and the H2 fixture.

  6. Gas follows electricity (María's ruling; commit 30 takes gas back to Energy Trends): ONS 04.5.2 gives 236.5 TWh against 252.3 TWh. Great Britain's local gas cells now sum to 234.0 TWh, not 249.7.

  7. The ONS electricity and gas rows are bound again (commit 30 holds the gas row out again). Decision D1's two measure exclusions are deleted (María's ruling: PR C carries the electricity deletion PR A had planned).

  8. The 29 register entries tracking UK local surface, consumption and high-value homes: remaining items (consolidation) #1278 now track UK: every open data item: uk-data parity, engine locks, the local surface, consumption and high-value homes #1095, which UK local surface, consumption and high-value homes: remaining items (consolidation) #1278 closed into. Text only.

  9. The taxpayers with pension income by area (María's ruling, option c).

    • hmrc.pension_income.count binds at HMRC's own area level, grown by ITL Table 2.5's all-ranges taxpayer growth.
    • It is a local family with no higher control, because no national row counts the union of State Pension and private pension recipients. It is declared a subset of ITL's taxpayers.
  10. Re-pin of the gate digests, the coverage manifest and the H2 fixture.

  11. The cars fuel duty hold quotes the 30 September ONS edition (María's ruling: refresh text that quotes the old release). Text only: ONS 07.2.2 for 2025 converts to about £13.1bn of duty, 9% below the OBR cars line, where the June edition gave £13.5bn and 7%.

  12. Rural Fuel Duty Relief on the dense build (item 6, row 3).

    • Before: policyengine-uk relieves petrol and diesel duty for a household with in_rural_fuel_duty_relief_area, which no build wrote.
    • The areas: HMRC Notice 2001 opens the relief in the Inner and Outer Hebrides, the Northern Isles, the Islands of the Clyde, the Isles of Scilly, the post town of Hawes and sixteen mainland postcode districts. It names the island groups without bounding them, so tools/build_uk_rfdr_area_layer.py declares Microcosm's translation into postcodes, each part with its sources. Islands that share a sector with the mainland (Raasay, Seil, Easdale, Luing, Kerrera, Lismore) and Hawes are listed by unit postcode, and mainland units inside island districts are taken out. The points the sources leave open are recorded in the tool; none changes an area's flag.
    • The layer: from the ONS postcode lookup (August 2026), uk/rfdr_area_layer.json lists the output areas more than half of whose postcodes are in the scheme: 41 in England and 1,053 in Scotland.
    • The node: uk.full.geography.rfdr sets the flag on each placed household from its assigned output area, after the local-authority resolve, on the BRMA node's pattern. --rfdr-localisation area is the default on the atomic assignment; none is for measurement builds, and a release candidate refuses it. Northern Ireland has no area in the scheme.
    • The national: it has no areas, so it leaves the engine's default of false (María's ruling).
    • The column joins the export surface's reviewed extras. The next dense build measures the flag.
  13. Re-pin of the gate digests for the RFDR export column.

  14. The six texts that still said UK local surface, consumption and high-value homes: remaining items (consolidation) #1278 tracks an item now name UK: every open data item: uk-data parity, engine locks, the local surface, consumption and high-value homes #1095, finishing commit 8. Text only.

  15. Merge of PR A's newer commits (the UC take-up gate at 0.06 of DWP's caseload, and the care-hours test). The gate digests are recomputed on the merged gates.json.

  16. OBR's betting and gaming receipts are bound (María's ruling).

    • The old reasoning: the exclusion held the row out as an identity. The stage levels each gambling base to HMRC's FY2024-25 receipts, or Machine Games Duty's liabilities, over the engine's rates, so the engine charges the receipts back by construction.
    • Why it no longer holds: that is true only at prior weights. Calibration moves the weights of the households carrying the bases, and nothing holds their level.
    • Before, on PR A's national (run uk-national-1418-7757add7a, H5 a912a7044836), at final weights:
      • the twelve bases shared out by betting and gaming stakes each sit 12.3% above HMRC's level;
      • the National Lottery base, on its own stakes, sits 2.4% above;
      • Machine Games Duty comes to £0.688bn against £0.612bn of liabilities;
      • at the engine's 2024 rates the bases carry £3.972bn of duty against OBR's £3.616bn (+9.9%). At HMRC's levels it would be £3.619bn.
    • The change: the exclusion is deleted, so the national binds the row as it binds the ONS energy rows. The tests and notes that called the row held out now say it is bound.
    • Undone by commit 40, pending María: the national solve can't measure the row's 2024 duty, so the exclusion is restored.
  17. Merge of PR A's review round (the UC take-up gate at 0.055, and the Carer's Allowance wording), with the gate and spine-part digests and the coverage manifest recomputed.

  18. The H2 fixture, regenerated on the merged tree.

  19. The local references, membership and compile-parity receipt are regenerated on the 272c987 pin. It ran from the feed's local cut, whose source hash matches the pin.

    • Pension income by area: 2,022 cells, 2,016 of them compiled. The two authorities without SPI local measures and the City of London's support-floor exclusion sign off the other six, as for the other SPI local families.
    • Energy: the DESNZ electricity and gas rows move by exactly the new level factor over the old, ×0.9258 on all 980 electricity rows and ×0.9371 on all 977 gas rows. No row's status changes.
    • Nothing else moves. The target-surface gate passes, and its count pins follow.
  20. Merge of main after PR A merged (two changelog lines).

  21. The Carer's Allowance take-up draws are balanced by weight within strata, for the next pair build (option B of the pool investigation).

    • Before: the newly entitled carers' identity-keyed uniforms were compared with 0.6 one by one, so the claiming share reached 0.6 only in expectation. On the few source households over the limit, one draw put the 2031 taper's cost well below its expectation.
    • The change: the uniforms are now placed by weight within strata, as policyengine-uk's stratified_uniform does (an engine test holds the two identical). The strata are:
      • the route (in-household or not);
      • whether the benefit unit reports a means-tested benefit at the survey year;
      • the survey-year weekly earnings band, one weekly rate wide above the limit and open above three.
    • Result: each stratum claims 0.6 of its eligible weight to within one record, and the outside pool's draw is balanced the same way at its measured share. The draws stay keyed on identity.
    • Unchanged: the rate, the pool and the caseload.
    • Receipt: it lists each stratum's realised share against its target, support-banded.
  22. Re-pin of the coverage manifest and the H2 fixture for the stage.

  23. Anthony's review: the scorer test's pin. test_uk_local_candidate_scorer.py pins the 27,208 active local references that commit 19 left (the 2,016 pension-income cells added).

  24. Anthony's review: the Rural Fuel Duty Relief checks.

    • In area mode the node refuses a Great Britain output-area code that isn't well-formed for its system, and a frame with Great Britain households none of which is flagged. Commit 31 makes the second check per support system, and commit 36 applies it at full scale only.
    • The layer tool's column check tests what it means.
    • Tests cover the release refusal of --rfdr-localisation none and the committed-layer requirement.
  25. Anthony's review: the energy level's bound and Northern Ireland's meters.

    • The energy rake gate holds each ONS-spend level to between 0.85 and 1.0 of the Energy Trends volume (0.926 and 0.937 now), so a wrong input row the recomputation shares fails it.
    • Northern Ireland's electricity meters stay in the standing-charge subtraction, and the level block says why: its tariffs carry the charge in the unit rate. The level doesn't move.
    • The gate battery digests are recomputed.
  26. Anthony's review: the Carer's Allowance strata are enforced. The stage refuses a take-up or pool stratum whose drawn weight misses its share by more than one record. The receipt records the check, and the H2 fixture carries the stage's new notes.

  27. Anthony's review: the pension bridge's tolerance. spi_pension_amount_bands_vs_area allows the larger measured gap plus half a point, 3.7%, not 4%. The coverage manifest is regenerated for commits 23 to 27.

  28. Anthony's re-review: the council tax pin. The support-floor membership has the City of London's two SPI pension income cells (51 rows, 95 area ids). This was the only line both engine-free jobs failed on.

  29. SPI Table 3.7's other income by total-income band is bound (María's ruling of 29 October). No row held that column, so the Income Tax on savings rows were met mostly through it.

  30. Gas back on Energy Trends; the QEP payment method checked directly (María's ruling of 29 October, Anthony's re-review 2 and 3).

    • ONS 04.5.2 counts bottled gas (COICOP 04.5.2.2), which the frame holds in liquid_fuel_consumption. Bottled gas is 2.2% to 3.2% of the class on ONS's CPI weights.
    • So gas and the local DESNZ gas rows return to the Energy Trends level, and ons.household_gas_expenditure returns to the measure exclusions, tracked on UK: every open data item: uk-data parity, engine locks, the local surface, consumption and high-value homes #1095. Electricity stays on ONS 04.5.1.
    • Every QEP payment method's electricity level falls inside the ratio bound, so the gate now checks the stage's declared payment_method against qep_payment_method.
    • The gate battery digests are recomputed.
  31. Anthony's re-review: count pins, RFDR per system, the Carer's Allowance tolerance, and 7b.

    • The second council tax assertion and the subset relations (22) are re-pinned.
    • The RFDR node refuses each support system with households none of which is flagged.
    • The Carer's Allowance stage refuses a stratum off its share by more than half its heaviest record, the most the construction allows. A stage test covers the pool branch. Commit 36 rotates each stratum by a start, which makes the bound one record.
    • The RFDR layer is rebuilt from the registered NSPL zip (downloaded from R2, sha256 checked). It flags the same output areas, and a test holds its provenance to the zip.
  32. SPI savings rows on the OBR's savings income; SPI Table 3.10 bound (María's rulings of 29 October).

    • The Table 3.7 interest and other-income rows grow by the October 2026 EFO's Table 4.5 savings income growth, vendored as obr_savings_income_growth_facts.json, in place of ×1.192.
    • Table 3.10's individuals with self-employment income, and that income, bind in the eight bands from £1 to £10,000. The others are signed out row by row.
  33. Re-pin onto Add SPI 2023-24 Table 3.10 and the OBR October 2026 EFO vintage (PolicyEngine/microcosm#1418) chronicle#394's merge (53815b2).

    • On the national surface only the intended values move: the 13 interest rows up 13.4%, 13 other-income rows and 16 Table 3.10 rows.
    • obr.winter_fuel_allowance is pinned to March's expenditure table, since October's FY2025-26 outturn would match it twice.
    • On the local surface only the 977 DESNZ gas cells move, back to the Energy Trends level after commit 30 (×1.067); the statuses and the references are unchanged.
  34. The national calibration test's pin moves to 1,403 active references. It had stopped engine-free CI on commit 33; the 94 test files that read the regenerated surfaces pass locally.

  35. Each UK active reference count is pinned once in the tests (María's request).

    • National: test_uk_target_references.py's ACTIVE_REFERENCE_COUNT, which now carries the count's history.
    • Local: the battery bindings test's candidate_targets.
    • The other copies check against the committed membership, or drop an incidental count. A change to the target set still fails one test, and no longer one per copy, each found in its own CI round.
  36. The independent review and Vahid's review (María's rulings of 29 October where the points were hers).

    • Pension income by area.
      • The bridge's index bound is 0.02: the 1.5% gap after UK pensions after #1084: open rulings, residual gaps and the engine pieces #1069's State Pension index, plus half a point. A test recomputes every fan-out bridge's gap from the committed registers.
      • The source factor is bounded on both sides, at 1.025 to 1.037, because the gap runs one way.
      • The rounded-thousand floor defers the pension income counts HMRC publishes at 5,000 or fewer, and their amounts with them (17 constituency and 3 local-authority cells of each). The bridge still counts them at source.
      • A deferred cell joins the reconciliation grown by its bound cells' own index, as an off-surface cell does. The dense target surface compiles end to end, and the pension cells take ×1.045 at both grains.
      • The SPI-frame fence and both adjudication records name the pension metrics.
    • Table 3.10's amounts grow by the count index times mixed income per head.
    • Electricity is measured at 2024, as road fuel is, with a signed matched-period hold.
    • The zero-support floor covers Table 3.7's other income and Table 3.10's bands.
    • RFDR: sampled rungs check the codes only, and the tool pins the NSPL registration and gains --check.
    • Carer's Allowance: each stratum's placement is rotated by a start drawn per stratum. So the stage refuses past one record, not half.
    • Cache keys and timing: the stages key their caches on the modules they read, and the energy level is computed once per process.
    • Texts: the stale texts are corrected, and the savings path's notes record its scope against ONS's household interest.
  37. Regenerated surfaces: the national and local references, the census, both parity receipts, the coverage manifest, the H2 fixture and the gate digests.

    • On the national surface only the intended values move: the eight Table 3.10 amount rows (×1.0763, the count index), and electricity onto ONS's calendar-2024 figure (×0.975).
    • On the local surface only the 40 pension income cells move, from bound to deferred, leaving 27,168 active.
  38. The re-review of commits 36 and 37.

    • tools/diagnose_uk_consumption_taxes.py reads each row's measurement period from the contract.
    • Stage-level tests pin the Carer's Allowance placement and the RFDR strictness switch through the graph. A test checks each fan-out bridge's source half on the committed registers, and another the surface's deferral wiring.
    • The Carer's Allowance start is keyed on each stratum's members.
    • Only the transferred spine gate keys on the energy level.
    • The small-cell floor reads published values.
    • The widened floor and fence records say they are pending María's confirmation.
    • The texts are fixed.
    • The local references regenerate unchanged, and the national compile-parity receipt changes only in the income-anchor rows' rationale.
  39. The round-3 re-review of commit 38.

    • The fan-out bridges' gaps are checked by running the runtime surface on the committed registers, at both grains.
    • A Carer's Allowance stratum's start is keyed on its lowest-draw member, so a membership change re-draws a stratum only when that member changes.
    • The certifier compares support-floor drops on the floor's signature fields only.
    • The diagnostic tool weights every row at the calibration year and reads periods from the run's registry.
    • A test pins the take-up strata's route.
    • The RFDR layer is rebuilt with its corrected description (same areas).
    • The texts are fixed.
  40. The betting and gaming row is held out again, pending María's decision.

    • The first national built from this PR (at 10f434d) refused at the national problem. The row declares the frame's period, 2024, but its value is the engine's gambling_duty, not a stored column.
    • The solve measures a declared period only on a stored column or a CGT dated measure, so it can't read the engine's 2024 duty. No PR C national had been built since commit 16, and the tests don't run the national problem's measure routes.
    • María's signed exclusion of 23 October is restored as it was. Its adjudication now notes that her 28 October bind ruling can't take effect until the solve can measure the duty, either through a dated duty measure or through bases bound at the frame's period.
    • Main holds the other excise rows out under María's 23 October ruling, since the build sets their level; they would hit the same refusal if bound. The row is still measured on every evaluation.
  41. The round-4 re-review of commits 39 and 40.

    • A test runs the solve's own period check over every bound row of the packaged contract, so a binding the solve can't measure now fails in CI.
    • The bridge mutation's scale is derived from the baseline, and the committed deferrals are pinned.
    • The diagnostic tool reads the solved weights as the certifier does.
    • The texts are fixed.
  42. The round-5 re-review of commit 41.

    • A test fails 14 days before any committed measure exclusion lapses, so a lapse shows in CI and not at a national's target step.
    • The diagnostic tool's own weight path is tested on synthetic release and solved H5s. Its id check refuses a household on either side alone. With --registry, its default year is the run's calibration period, and its texts say it needs a national H5.
    • The period test reads the release's calibration year and also runs the solve's fact-period check.
  43. The round-6 re-review of commit 42.

    • The exclusion lapse clock also reads the register's support floor, and passes on an empty register.
    • The period test runs the solve's fact-period check on every binding kind, so the CGT provider rows are covered, and a required row missing from the membership fails.
    • The diagnostic tool weights its composition with the solved weights and refuses a registry with mixed periods when no --year is given.
  44. The round-7 re-review of commit 43.

    • The period test reads the rows the national compiles: national-line targets, less rows signed out of a fan-out and rows held on their own, except held rows in the support floor's scope, which are compiled as probes.
    • A deferred required row is flagged like a missing one. The national compiles only the active references, so this test is the only guard there.

Checks run locally

All on the merged tree, with policyengine-uk 2.162.14:

  • the touched engine-free tests: energy pricing, local energy targets, stage health, local census, cross-grain declarations, fan-out bridges, local targets, population targets, measure simulation, data-target parity, gate-battery pins and contract;
  • for RFDR: the layer and node tests, country spec, Orrery contract and publication, full-build preparation, CLI and calibration graph, national graph, rowwise national role, terminal gates and dense sign-off draft;
  • the annual-income engine test;
  • the coverage manifest, local census and parity-register checks.

Also run since: the RFDR engine admission test (test_uk_full_graph_admission.py), which passes, and the local surface tests after commit 19 (battery bindings, local targets, target references, local census, local release preflight, fan-out bridges, cross-grain declarations), which pass.

For commits 23 to 27: the RFDR layer and node tests, the atomic, BRMA, rent, property and home-value graph tests, the full target and population graphs, the CLI, Carer's Allowance, stage health, energy pricing, fan-out bridges, cross-grain declarations, national reconciliation, country spec, scorer, battery bindings, gate-battery pins and contract tests, and the coverage manifest check.

For commits 28 to 33: 31 touched engine-free test files (target references and membership, local targets, battery bindings, local census, local release preflight, fan-out bridges, cross-grain declarations, the full target graph, energy pricing, domestic energy targets, the constituency energy resource, high-value homes, stage health, measure simulation, Carer's Allowance, the RFDR layer and graph, subset relations, council tax, HMRC uprating, population targets, the feed pin, full targets, the scorer, national reconciliation, bus and CGT targets, LCFS consumption, country spec, gate-battery pins and contract), the HMRC uprating and RFDR admission engine tests, the coverage manifest check, and the feed-gated high-value home stock check in the re-pin. CI runs the rest.

For commits 36 and 37: the 107 engine-free test files that read the regenerated surfaces or that this round touches; the engine tests for HMRC uprating, Carer's Allowance and graph admission; the coverage manifest check; the H2 fixture's readers; tools/build_uk_rfdr_area_layer.py --check against the registered NSPL zip, which reproduces the committed layer byte for byte; and tools/uk_target_surface_receipt.py, which compiles the dense target surface end to end on the pinned feed. All pass. The vendoring register's check that each consumer names its file failed once and is fixed in commit 36. The Orrery reconstruction test needs a Node package this checkout doesn't have; CI runs it.

For commit 38: the 31 engine-free test files this round touches or that read what it changed; the Carer's Allowance and HMRC uprating engine tests; the coverage manifest check; the round-2 mutations, each of which now fails a test; and tools/build_uk_rfdr_area_layer.py --check against the registered NSPL zip, which still reproduces the committed layer byte for byte. All pass.

For commit 39: the 34 engine-free test files this round touches or that read what it changed; the Carer's Allowance and weight-year engine tests; the coverage manifest check; the re-review's mutations (local-authority pension cells x1.005, and the take-up strata without the route), each of which now fails a test; and tools/build_uk_rfdr_area_layer.py --check against the registered NSPL zip, which reproduces the rebuilt layer byte for byte. All pass.

For commits 40 to 44: the touched engine-free test files (measure simulation, measurement period, consumption-tax diagnostics, excise quantity inputs, uprating holds, support floor, fan-out bridges and off-surface cells). All pass.

CI runs the suites (this PR targets main).

🤖 Generated with Claude Code

juaristi22 and others added 10 commits October 28, 2026 20:28
…g buys

The lcfs_consumption stage raked electricity kWh to DESNZ Energy Trends'
calendar-2024 volume, which priced at QEP sits about 5% above ONS
Consumer Trends 04.5.1 on the 30 September 2026 release. A fuel the
stage now declares in spend_level_fuels takes its level from ONS
household spending instead: the class for the level year, less QEP's
United Kingdom average fixed cost on every UK domestic meter DESNZ
counts (Great Britain's sub-national count with Northern Ireland's row,
now vendored), over QEP's United Kingdom average unit cost. Electricity
is declared, giving 84.9 TWh against Energy Trends' 91.7 TWh; gas keeps
Energy Trends.

published_energy_level carries the new level everywhere it was read:
the stage's rake, the energy_rake gate's recomputation, and the factor
that puts the dense build's local DESNZ electricity rows on the stage's
level, so those rows move with it. The texts that described the level
as Energy Trends follow (gate notes, census fence, cross-grain route,
holds, adjudication, contract notes), and the ONS resource's purpose now
names the release its rows come from.

microcosm#1418, item 12 of the Budget data fixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OOTLAR 1.50 moves machines with a maximum charge to play above GBP 1,
or a maximum prize above GBP 100, to Machine Games Duty's higher rate
from 1 April 2027: Category B. The Machine Games Duty standard-rate base
the stage levels has no category split, so the Gambling Commission's
FY2024-25 gaming machine gross gambling yield by machine category
(Industry Statistics sheet 4, all venues, Great Britain) is vendored
beside it, a sixth excise_facts.json selection. The split column that
reads it waits for the engine input (policyengine-uk#2719).

microcosm#1418, item 5 of the Budget data fixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The energy_rake gate's notes moved the gates manifest digest, its
fingerprint and the spine part's manifest digest (the policy digest is
unchanged), and the stage's declaration and vendored resources moved the
release input coverage manifest's source digest. Regenerated and
checked; the local references, the local compile parity receipt and the
H2 fixture follow in the pin pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ands

The dense surface had no pension income below the national SPI bands.
hmrc.pension_income.amount binds each constituency's and local
authority's taxpayers' private and State Pension income together from
SPI 2023-24 Tables 3.15 and 3.14 (count times mean). It is grown to
calendar 2025 by ITL Table 2.5's all-ranges total income growth, which
the national State Pension and private pension bands match together to
within 0.1%. The frame reads total_pension_income_annual, the engine's
sum of the two pensions on the annual simulation.

A fan-out sum bridge reconciles the cells per grain to the national
bands, and may now sum several fan-outs: the State Pension and private
pension bands together are the area cells' quantity. A control missing
one of its fan-outs is refused. At source the bands sit 3.0% above the
constituency cells and 3.2% above the local-authority cells, since the
area tables leave out the taxpayers HMRC places in no area, who are
more often pensioners; the bridge refuses past 4% at source or 0.6% in
the index gap. The two authorities without SPI measures are deferred as
for employment, and the metric is appended last at both grains.

Counts are not bound here: the area tables count taxpayers with any
pension, a union no national row binds, and the bridge for them is a
ruling still owed (microcosm#1418, item 14 of the Budget data fixes).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR C stacks on PR A. The two conflicts were the contract's target counts
(PR A's three ITL Table 2.6 savings rows and PR C's pension income by
area: 452 targets, 391 national, 61 local) and the release input
coverage manifest, regenerated on the merged tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The engine-free annual measure test checks every annual measure is one
the resolver knows; total_pension_income_annual reads the engine's
total_pension_income, which the stub now sums from its two pensions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The lcfs_consumption stage's spend_level declaration is mirrored into
the fixture's stage parameters and the graph's contract hashes move with
it and with the re-vendored resources; nothing else moves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
María's ruling of 2026-10-28 on PR C: gas follows electricity. The 30
September 2026 Consumer Trends release puts 2024 gas spending at
GBP 18.30bn, so the stage's Energy Trends level priced at QEP sat about
5% above ONS for both fuels. The stage now declares gas in
spend_level_fuels: ONS 04.5.2 less QEP's United Kingdom average gas
standing charge on every UK domestic gas meter, over the unit cost,
236.5 TWh against Energy Trends' 252.3 TWh. Great Britain's local gas
cells follow at 234.0 TWh. The texts that named Energy Trends for gas
now name the shared level.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Decision D1 held ons.household_electricity_expenditure and
ons.household_gas_expenditure out of the objective as published-source
residuals against the stage's DESNZ-volume level. With the stage's level
now the kWh ONS household spending buys, the frame starts about 1% below
each row at design weights, so both exclusions are deleted and the
national binds ONS 04.5.1 and 04.5.2 again (María's rulings of
2026-10-28 on microcosm#1418: PR C carries the electricity deletion PR A
planned, and gas follows).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
microcosm#1278 closed into microcosm#1095 on 25 October, which now
tracks every open UK data item. The 29 register entries whose tracking
still named #1278 (five measure exclusions, two small-cell rules and 22
uprating holds) name #1095 (María's ruling of 2026-10-28). Text only:
no entry's dates, reason or scope changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 and others added 7 commits October 28, 2026 21:16
María's ruling of 2026-10-28 on PR C (option c): SPI Tables 3.14 and
3.15 count each area's taxpayers with any pension income, State or
private, and no national row counts that union, since the national SPI
bands count State Pension and private pension recipients apart and most
taxpayers of State Pension age are both. hmrc.pension_income.count binds
the cells at HMRC's own area level, grown by ITL Table 2.5's all-ranges
taxpayer growth, as a local family with no higher control, declared a
subset of ITL's taxpayers. The metric follows the amount, appended last
at both grains; the same two authorities are deferred.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The energy_rake gate's notes moved the gates manifest digest, its
fingerprint and the spine part's manifest digest again (the policy digest
is unchanged); the stage's gas declaration moved the coverage manifest's
source digest and the fixture's stage parameters and contract hashes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The obr.fuel_duties_cars reason converted ONS 07.2.2 for calendar 2025
to duty on the June Consumer Trends edition; the feed carries the 30
September 2026 edition, whose figure is 3.2% lower, so the conversion is
about GBP 13.1bn, 9 percent below the OBR cars line. Text only (María's
ruling of 2026-10-28: refresh reason text that quotes the old release).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
policyengine-uk relieves petrol and diesel duty for a household with
in_rural_fuel_duty_relief_area, which no Microcosm build wrote. HMRC
Notice 2001 (SI 2011/2935 Schedule 1) opens the relief in the Inner and
Outer Hebrides, the Northern Isles, the Islands of the Clyde, the Isles
of Scilly, the post town of Hawes and sixteen mainland postcode
districts, and names the island groups without bounding them.

tools/build_uk_rfdr_area_layer.py declares Microcosm's translation into
postcode districts, with unit postcodes for the islands that share a
sector with the mainland (Raasay, Seil, Easdale, Luing, Kerrera,
Lismore) and for Hawes, and mainland units inside island districts
excluded, each part with its sources. From the ONS postcode lookup it
writes uk/rfdr_area_layer.json: the output areas more than half of
whose postcodes are in the scheme, 41 in England and 1,053 in Scotland.

The post-geography node uk.full.geography.rfdr sets the flag on each
placed household from its assigned output area, after the local
authority resolve, on the BRMA node's pattern: --rfdr-localisation area
is the default on the atomic assignment, none is for measurement
builds and a release candidate refuses it, and the layer is pinned like
the BRMA layer. Northern Ireland has no area in the scheme, and the
national line leaves the engine's default of false (María's ruling of
2026-10-28). The column joins the export surface's reviewed extras.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The uk_export_surface gate's allowed_extra_columns gains the RFDR flag,
moving the policy, gates manifest and fingerprint digests and the
release-cut part's digests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tracking fields moved to #1095 in c4b08a5, but six texts still
said an open item is tracked on #1278: the obr.fuel_duties_cars
reason's business-use rebind, two uprating-hold reasons, and the
high-value homes coverage decision (#1240 fix 6) in the local census
description, the scope constant's comment and a test comment. Text
only, completing María's ruling of 2026-10-28; texts citing #1278 as
where something was decided or changed are history and stay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o PR C

PR A allows the UC take-up stage gate 0.06 of DWP's caseload and names
the in-household care hours carrier among the release-dropped columns.
The gate battery's policy, manifest and fingerprint digests and the
spine part's manifest digest are recomputed on the merged gates.json,
which carries both PR A's tolerance and PR C's notes and RFDR export
column; the coverage manifest and the H2 fixture don't move.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 and others added 3 commits October 28, 2026 22:57
The obr.betting_and_gaming_duties exclusion held the row out as an
identity: the lcfs_consumption stage levels each gambling base to
HMRC's FY2024-25 receipts, or Machine Games Duty's liabilities, over
the engine's rates (microcosm#1359), so the engine charged the receipts
back by construction. That holds only at prior weights. Calibration
moves the weights of the households that carry the bases, and nothing
held their level: on the 3c0d11b6 national the bases sat about 12%
above HMRC's receipts at final weights (Machine Games Duty £0.684bn
against £0.609bn). The level is not held at final weights, so the
exclusion is deleted and the national binds the row, as it binds the
ONS electricity and gas rows (María's ruling of 2026-10-28).

The notes and tests that described the row as held out now say it is
bound, and the consumption-tax tool's docstring names the bound rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR A narrows the UC take-up stage gate to 0.055 and rewords the Carer's
Allowance pool and the unmapped account codes. The gate battery's
policy, manifest and fingerprint digests and both spine-part digests
are recomputed on the merged gates.json, and the coverage manifest is
regenerated; the H2 fixture follows in its own commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR A's Carer's Allowance wording and PR C's stages both reach the spine
graph; tools/graph_uk_spine_fixture.py regenerates it on the merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
juaristi22 and others added 2 commits October 29, 2026 04:24
The local surface regenerates on the 272c987 pin for PR C's two local
changes, from the feed's local cut (constituency and local-authority
rows and the statistical_scope rows but the small areas, as the
272c987 re-pin cut it; its source hash matches the pin):

- SPI pension income and the taxpayers with it by area add 2,022 cells,
  2,016 of them compiled. The two authorities without SPI local
  measures and the City of London's support-floor exclusion sign off
  the other six, as for the other SPI local families.
- The DESNZ electricity and gas rows move with the energy level, by
  exactly the new factor over the old: x0.9258 on all 980 electricity
  rows and x0.9371 on all 977 gas rows, with no status changing.

Nothing else moves: the references only gain the pension cells, and the
receipt replaces the energy rows and adds the pension rows. The
target-surface gate passes; its count pins follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22
juaristi22 changed the base branch from uk-budget-2026-static-fixes-1418 to main October 29, 2026 10:04
juaristi22 and others added 2 commits October 29, 2026 11:22
The stage compared each newly entitled carer's identity-keyed uniform
with the take-up rate on its own, so the claiming share reached 0.6
only in expectation. On the few source households over the limit, one
draw had put the 2031 taper's cost well below its expectation (the
1418-ca-pool investigation, option B).

weight_balanced_uniforms places a stratum's uniforms by weight: each
record takes the midpoint of its slice of the stratum's cumulative
weight, in the order of its draw, as policyengine-uk's
stratified_uniform does (an engine test holds the two identical). The
take-up strata are the route (in-household or not), whether the
benefit unit reports a means-tested benefit at the survey year, and the
survey-year weekly earnings band, one weekly rate wide above the limit
and open above three, read from the engine's carers_allowance_earnings
as its own test reads them. Each stratum then claims 0.6 of its
eligible weight to within one record. The outside pool's draw is
balanced the same way over the other carers who report no allowance, by
receipt and band, at its measured share. The rate, the pool and the
caseload are unchanged, and the draws stay keyed on identity. The
receipt lists each stratum's realised share against its target,
support-banded and guarded as its other shares are.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Carer's Allowance take-up stage's reviewed payload gains the
carers_allowance_earnings predictor and the balanced-draw wording, so
the source manifest's hash and the spine graph move.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22
juaristi22 marked this pull request as ready for review October 29, 2026 20:48
juaristi22 and others added 2 commits October 29, 2026 21:02
Commit 33's regenerated national membership holds 1,403 active
references (SPI Table 3.7's other income and Table 3.10's bands added),
and test_uk_target_references.py already pins it. The national
calibration test still asserted 1,374, which stopped engine-free CI on
6353e65. The 94 test files that read the regenerated surfaces pass
locally, so CI's --maxfail=1 should hide no further pin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The national count was asserted in four places across three files, and
the local count in two, so every legitimate change to the target set had
to move each copy, and with CI's --maxfail=1 each missed copy cost its own
CI round (twice on 29 Oct). One pin per surface stays:

- national: ACTIVE_REFERENCE_COUNT in test_uk_target_references.py, now
  with the count's history the country spec test carried;
- local: the candidate_targets pin in test_uk_battery_bindings.py.

The country spec test checks the packaged spec against the committed
membership's active_reference_count, the national calibration test drops
its incidental count (it exists for the UC and OBR references), and the
scorer test checks the scorer reads the committed membership. A change to
the target set still fails one test, so it is reviewed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vahid-ahmadi

Copy link
Copy Markdown
Contributor

Review of 6919f44

I read the body, all 35 commit messages and the thread first. Anthony's two rounds cover the energy level, RFDR checks, the Carer's Allowance strata, the pension bridge and the count pins, so I don't go over those again. The points below are new. Paths are relative to packages/microcosm-build/src/microcosm/build/ unless they start with tools/.

Findings

1. should: the OBR savings path runs well ahead of ONS's household interest.

  • uk_runtime/hmrc_uprating.py:250 grows the 13 Table 3.7 interest rows and the new other-income rows by EFO October 2026 Table 4.5's savings income line. The vendored rates match the pinned workbook exactly: +34.96% in 2024-25, then +0.65%. Chained over the calendar 2025 window that gives about ×1.356, against ×1.192 before.
  • Table 4.5 is headed "Self-employed income streams". It is the savings income of people who file self assessment, the line the OBR uses for SA receipts. The SPI Table 3.7 rows cover all taxpayers, most of whom don't file. A 35% rise in 2024-25 could partly be more savers entering SA as interest crossed the Personal Savings Allowance, which is a change in who files, not interest growth.
  • ONS households' interest received (D.41, HAXV, release of 30 September 2026) gives FY2023-24 £91.1bn, FY2024-25 £97.5bn (+7.1%) and calendar 2025 £95.5bn. That is ×1.048 from FY2023-24 to calendar 2025, against ×1.356 here. HAXV is after the FISIM adjustment and includes ISA interest and non-taxpayers, so it isn't like for like, but the gap is about 29 points.
  • The engine still projects interest at GDP per head (pending policyengine-uk#2638), so the solve can only reach the new level by moving weight toward savers.
  • Ask: record the scope difference and the HAXV comparison in the resource notes, and add the records that carry interest to the targeted weight check Anthony asked for in his finding 9. Whether to keep the ruling is María's call. I'm not asking for a hold.

2. should: Table 3.10's two lowest bands only count people who file.

  • Note 1a to Table 3.10 counts an individual "if there is evidence of business activity in the tax return". The £1 to £500 and £500 to £1,000 bands (111k and 110k in 2023-24) are therefore filers only. People whose trading income stays under the £1,000 trading allowance need not file at all, and the FRS frame may hold them.
  • Binding those bands (uk/uk_population_targets.json:3789, :3830) could move weight away from the small-profit group that the trading allowance measures affect.
  • Ask: were the eight bands measured on PR A's national before binding, and what is the gap at design weights in the two lowest? If the gap is small, this is a nit. If it is large, binding from £1,000 is an option.

3. should (question): RFDR zero-flag refusal on sampled dense builds.

  • uk_runtime/rfdr_area_layer.py:279 refuses any system with households but none flagged. England and Wales have only 41 flagged output areas. Area mode is the default.
  • The private-rent node already relaxes its checks on sampled rungs (strict=full_scale, uk_runtime/graph_population.py:1324). The RFDR node gets no such flag.
  • I couldn't work out from the code how many England and Wales clones a 0.01 rung places, so I haven't shown that it refuses. Has a 0.01 dense build been run in area mode? If not, pass full_scale through, or document that sampled rungs use none.

4. nit: the ratio bound would reject the previous ONS vintage.

  • On the June release (£26.10bn for 2024), the electricity level is (26.10 − 6.19) / 0.25827 = 77.1 TWh, a ratio of 0.840. That is outside [0.85, 1.0] (uk/gates.json:335).
  • So a revision as large as September's moves the ratio across the band in either direction. That's fine as a trigger to re-adjudicate, but the gate notes should say so, so the next re-pin isn't read as a data error. This builds on Anthony's finding 5 and his new finding 3.

5. nit: text that commit 30 left stale.

  • uk_runtime/stage_health.py:315-317: "held to the bound ONS 04.5.1 and 04.5.2 spend rows". 04.5.2 is held out again.
  • uk/uk_population_targets.json:17541: the gas row "bound on the frame's gas spend column".
  • uk/uk_population_targets.json:710: electricity "a measured diagnostic since microcosm#1113". It is bound again.
  • uk/uk_population_targets.json:17465 and uk/target_reference_signed_exclusions.json:55 say no per-fuel ONS energy row is fitted.

6. nit: the cars fuel duty hold updates the ONS figure but keeps the engine reads.

  • uk/calibration_measure_exclusions.json (obr.fuel_duties_cars) now quotes £13.1bn and 9% on the September edition, but keeps "-9.9 percent ... (-8.1 percent on DESNZ's)".
  • If those were measured with the stage levelled to the June 07.2.2, say so, or re-measure.

7. nit: PR state. The body says "Draft until CI passes on commit 35 and María marks it ready", but the PR isn't a draft. One of the two should change.

Checked and found true

  • The electricity level reproduces from the vendored rows: (£28.129bn − 30,192,447 meters × £205.11) / £0.25827 = 84.93 TWh, a ratio of 0.926 to Energy Trends' 91.74 TWh.
  • The RFDR mainland districts in tools/build_uk_rfdr_area_layer.py match HMRC Notice 2001 on GOV.UK exactly: EX35, LA17 and NE48, plus 13 Scottish districts.
  • 53815b2 is chronicle#394's merge (merged 16:13 today), and its tree is identical to 2ba21f3's, the head of my round 2 review there.
  • No file this PR touches has changed on main since the merge base. CI's run at this head was on the merge with main after Pin the fact decoder in both UK target kernel hashes #1424.

Verdict

No must-fix findings. Fine to merge once engine-free (3.13) is green and the Draft wording is settled. Findings 1 and 2 are about what the next national should measure, not reasons to hold this PR. Finding 3 is worth answering first if sampled dense builds are in regular use.

What I ran

  • A fresh clone at 6919f44 with a fresh uv sync --all-packages --locked. Both are deleted afterwards.
  • 21 engine-free test files, all passed: energy pricing, stage health, the RFDR layer and graph, fan-out bridges, HMRC uprating, Carer's Allowance, stochastic assignment, target references, council tax, subset relations, the local scorer, battery bindings, local targets, measure simulation, population targets, full targets, the Chronicle feed, cross-grain declarations, country spec and the data contract.
  • Read OBR EFO October 2026 Table 4.5 from the workbook pinned in chronicle#394, ONS HAXV (UKEA), SPI 2023-24 Table 3.10 and its notes, and HMRC Notice 2001.
  • CI at this head: engine-free (3.14), engine-uk, engine-us, integration-uk, lint, wheels and select-countries pass. engine-free (3.13) was still running when I posted.

Not verified: any build, engine tests, Python 3.13 locally, the pension gaps (3.0% and 3.2%), the 41 and 1,053 output areas against NSPL, and the Gambling Commission rows.

🤖 Generated with Claude Code

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Independent review of #1420 at b3f4ebf70

This review ran separately from the sessions that wrote the PR. Twelve reviewers went through the diff:

  • three read it line by line, one per area;
  • nine others took one angle each: removed behaviour, callers in other files, Python pitfalls, wrappers, reuse, simplification, efficiency, altitude and conventions.

Verifiers then checked every candidate against the code, and a last pass looked for gaps. Finding 1 was reproduced with the repo's own check. 6919f44 changes only test pins, so nothing below moves.

Anthony's two rounds and Vahid's review aren't repeated. Where a finding overlaps Vahid's, it says so and adds what is new. Paths are relative to packages/microcosm-build/src/microcosm/build/ unless they start with packages/, tools/ or changelog.d/.

Verdict: finding 1 should be fixed before merge. From this head, every dense build fails at target compilation. National builds are unaffected.

Must fix

  1. The pension bridge refuses the committed registers, so every dense build stops at target compilation.
    • uk/cross_grain_declarations.json:546 declares max_index_shift 0.006 for spi_pension_amount_bands_vs_area. The real index gap is 1.5%:
    • _assert_uk_fanout_sum_agreement (uk_runtime/ledger_targets.py:3446) raises at both grains. At the constituency grain: "the national bands' uprating index and its constituency cells' index differ by 1.0150532826197245, beyond the declared 0.006, so the cells would take 1.0450232420644112". The local-authority grain gives the same index and 1.0479.
    • The check runs unconditionally in uk.full.target_compilation, through uk_local_target_surface, so every dense build hits it. So does tools/uk_target_surface_receipt.py.
    • Reproduced engine-free with the runtime's own appliers and assertion on the committed membership values.
    • The 0.1% gap in the bridge's reason, the selector note (uk/uk_population_targets.json:34445) and the changelog was measured before UK pensions after #1084: open rulings, residual gaps and the engine pieces #1069. On the old State Pension index the two families grew ×1.1348 together.
    • Why CI can't see it:
      • test_uk_fanout_sum_bridges.py uses synthetic registers;
      • the local reference regeneration and the compile-parity tool never call uk_local_target_surface.
    • Options:
      • put both sides on one index;
      • or re-declare about 0.021, the gap plus half a point, which is the bridge's own rule. That passes, but it scales every area pension cell by ×1.045 to ×1.048, not the ~×1.03 the PR describes.
    • Two more problems on the same declaration:
      • the changelog still says 4%, though commit 27 set 0.037;
      • the source check abs(source - 1.0) > 0.037 is centred on 1.0, but the gap is one-sided (+3.0% and +3.2%). A factor anywhere down to 0.963 passes.

Should fix: data and calibration

  1. Table 3.10's counts and amounts grow on indices that pull apart.

    • In each band from £1 to £10,000:
      • counts grow ×1.0763, ITL all-band taxpayer growth (uk/uk_population_targets.json:3807);
      • amounts grow ×1.0384, OBR per-capita mixed income (:3848).
    • So each band's implied mean profit falls 3.5%. The engine meanwhile raises every self-employment income by ×1.0384 against fixed nominal band edges, which moves people up and out of these bands.
    • To meet both rows, calibration has to add about 7.6% more people to each band and tilt each band toward its bottom edge.
    • No other self-employment pair does this:
      • Table 3.6 grows counts by per-band taxpayer growth and amounts by per-band total-income growth;
      • the area pair is ×1.0763 for counts and ×1.1340 for amounts.
    • This adds to Vahid's finding 2. Both move weight within the small-profit group that the trading allowance measures read.
    • Suggestion: one pairing for each band, for example amounts grown by the count index times per-capita mixed income. Each band's mean then moves as the engine moves incomes.
  2. The RFDR zero-flag refusal does fire on sampled dense rungs (Vahid's finding 3).

    • Only 41 of the 188,880 England and Wales output areas are flagged, about 2 in 10,000 households.
    • Estimated refusal odds per seed, from the supports' household counts:
      • f001: about 27% at K=15 (the default), 71% at K=4 and 94% at K=1;
      • f010: about 16% at K=2;
      • full scale: about 0.
    • Documented runs it would stop: experiments/762's R2 (f001, K=4) and experiments/901's dense smoke (f010, K=2).
    • Reproduced with the committed layer on a 400-row England and Wales pool.
    • The assignment is identity-keyed, so a failing seed fails every time. The message ("the layer's output areas and the assignment's cannot be the same codes") misreads a small-sample miss.
    • Commit 31's per-system check made it worse. Commit 24's check over Great Britain would have refused about 31% at f001 with K=1, and about 1% with K=4.
    • Release builds are unaffected, because they refuse sampling.
    • Suggestion, as Vahid's: pass full_scale through and refuse a zero-flag system only at full scale. Keep the malformed-code refusal unconditional.
  3. The balanced Carer's Allowance draw is deterministic in one- and two-record strata.

    • weight_balanced_uniforms (stochastic_assignment.py:81) places each record at the midpoint of its weight slice. A one-record stratum therefore sits at 0.5 whatever its draw:
      • in take-up (uk_runtime/carers_allowance_take_up.py:636, positions < rate), it claims in every build at rates 0.6 and 0.75, and never at 0.45;
      • in the pool (:323, draws < share), it is never drawn while the measured share is 0.5 or less.
    • Two equal-weight records claim exactly half at a 0.6 rate. Three or more records land within half a point of the rate (simulated).
    • Independent draws were right in expectation. These are fixed rounding errors that don't average out over seeds, and the half-record refusal lets them through.
    • Thin strata are plausible here: 16 take-up and 10 pool strata, over what the module docstring calls "the few source households over the limit", the taper's population. Counting records per stratum on a real stage input would settle it; those counts stay private.
    • policyengine-uk's stratified_uniform is identical. But the engine uses it only to place birthdays, never against a rate, so the engine test's docstring "as the engine's own take-up draws do" doesn't hold.
    • Suggestion: an identity-keyed random start in each stratum (systematic sampling). It keeps the half-record balance and restores the rate in expectation.
  4. The cache keys don't cover the modules this PR moves behaviour into.

    • source_hash hashes only the modules a kernel lists:
      • uk_runtime/graph_kernels.py:166 binds lcfs_consumption to (uk_engine_adapter,);
      • the spine gate's code hash (uk_runtime/graph_evidence.py:371) covers gate_battery and battery_bindings.
    • Neither covers energy_pricing.py, which now computes the ONS-spend level, or stage_health.py, which holds the energy gate. The target-compile kernels do hash energy_pricing.
    • A probe confirms it: editing energy_pricing.py re-keys target compilation but not uk.stage.lcfs_consumption or the spine gates.
    • Example: a code-only change to household_spend_level, such as Anthony's Northern Ireland meters alternative, resumed on the same --graph-store:
      • the 980 local electricity cells are re-levelled;
      • the frame keeps the old kWh, and the cached energy-rake pass is reused;
      • the frame and the local targets end up on different levels, with no error.
    • graph_kernels.py:207 has the same gap for the Carer's Allowance stage and stochastic_assignment. It matters under the two-build flow that reuses build 1's store.
    • The gap is older than this PR, which widens it.
    • Suggestion:
      • add energy_pricing and stochastic_assignment to those stages' helper modules;
      • add energy_pricing and stage_health to the spine gate's code hash.
  5. The energy level re-reads the QEP file 52 times per call, which adds about 20 minutes to every local compile.

    • household_spend_level (uk_runtime/energy_pricing.py:1037) calls qep_prices(parameters) to read two UK averages. That rebuilds the whole table: 52 vendored_rows calls, each re-parsing the 2.77 MB qep_energy_prices.json.
    • Measured at the merge base and at the head: published_energy_level goes from 4 ms to 627 ms.
    • align_energy_to_stage_level (uk_runtime/domestic_energy_targets.py:369) recomputes it for each reference, and 1,957 local references carry the energy index. So one local compile goes from about 0.6 to about 21 minutes.
    • Who pays:
      • the dense build's target compilation (the first build after merge recompiles);
      • certify_uk_release_cut.py;
      • the compile-parity tool;
      • uk_target_surface_receipt.py;
      • the local reference generator.
    • Suggestion, any one of:
      • read only the two UK-average rows;
      • pass in the EnergyPrices the stage already holds;
      • compute the factors once per compile.
  6. The new SPI families sit outside the zero-support floor.

    • The floor's hmrc_spi_income_band_counts and hmrc_spi_income_band_amounts scopes (uk/calibration_measure_exclusions.json:36, :49) list the seven older families by exact id.
    • Table 3.7's other income by band (up to £1m+) and Table 3.10's bands aren't in them. Yet the floor's reason covers "the HMRC SPI income bands by total income".
    • If one of the 29 new rows has no carrier on a spine, the dense size search refuses it by name and the national carries a permanent miss. A sibling row would instead be signed out and receipted. The most likely rows are other income at £500k to £1m and £1m+.
    • test_uk_support_floor.py pins the seven families, so CI won't notice. A support count on the next spine settles it.
  7. The rounded-thousand floor misses the pension income counts.

    • hmrc_rounded_thousand_floor (uk/local_small_cell_rule.json:23, floor 5,500) matches only hmrc.higher_rate_taxpayers.count.
    • The PR added pension income to the generator's count × mean set and its coverage-gap tuple, but not here.
    • 17 constituency and 3 local-authority hmrc.pension_income.count cells published at 5,000 or less bind. The smallest is 2,000, so the rounding error is up to 25%.
    • These counts have no higher control, so the floor is their only guard.
    • Self-employment counts have the same gap from before this PR (405 cells). Extending the rule needs María's ruling, but nothing records the omission as deliberate.
  8. The pension metrics bind under adjudications that don't name them.

    • They join hmrc_income_by_area through the hmrc/ prefix (uk_runtime/local_target_census.py:159).
    • require_adjudicated_uk_local_binding checks adjudications per family (uk_runtime/local_rowwise.py:823-826). So the pension metrics pass under records that name only the employment, self-employment and higher-rate references (uk/local_binding_adjudications.json:20).
    • The hmrc_spi_frame_model_proxy fence authority (local_target_census.py:924-930) lists five declarations and no pension income, though both new bindings use the filter it covers.
    • When the higher-rate counts were added, bdb9648 and 5a0c3f6 extended these records. The 28 October ruling (option c) covers how the count binds, not the fences.
  9. The re-bound ONS electricity row is measured through the engine's CPI, not on 2024.

    • ons.household_electricity_expenditure (uk/uk_population_targets.json:17490) has no measurement_period. It is therefore measured at 2025 through the engine, which grows electricity by CPI (+3.4% in 2.162.14), while ONS 04.5.1 grew 2.5% from 2024.
    • That puts about 0.85% of electricity spending's projection error into the weights. Meanwhile the stage level and the local DESNZ rows sit on 2024.
    • The road fuel row (:17580) is measured at 2024 for exactly this reason (UK national calibration reads stored input columns at the frame period: 76 £ rows compare 2024 values with 2025 targets #1324). No written rule covers electricity, so this is María's call.
  10. Nothing ties the ONS gas exclusion to spend_level_fuels.

    • The re-added exclusion's reason (uk/calibration_measure_exclusions.json:398) says netting bottled gas out "rebinds this row". It doesn't say that gas must also leave Energy Trends.
    • test_uk_energy_pricing.py pins spend_level_fuels and test_uk_measure_simulation.py pins the exclusion, separately. The gate doesn't read the exclusions.
    • Following the reason once UK: every open data item: uk-data parity, engine locks, the local surface, consumption and high-value homes #1095 lands would bind ONS 04.5.2 against a frame levelled about 6.7% higher (252.3 against 236.5 TWh). That is D1's original pull, and every check would pass.
    • A test that each bound ONS energy row's fuel is in spend_level_fuels would close the gap.
  11. One OBR gambling total can't hold the two gambling carriers.

    • obr.betting_and_gaming_duties (uk/uk_population_targets.json:17759) binds one duty-weighted sum of two separately weighted carriers:
      • National Lottery stakes;
      • betting and gaming stakes, of which every other base, MGD included, is a fixed share (uk_runtime/lcfs_consumption.py:478-484).
    • Commit 16 gives drift in each carrier as its cause. But one row can be met by shrinking both. That can leave Lottery below HMRC's level while betting and gaming, which carries the MGD base, stays above it.
    • excise_facts.json already vendors HMRC receipts by duty. So a Lottery row and a row for the remaining duties could bind instead, with the OBR total kept as a diagnostic. The next national can measure the split.

Should fix: tests and texts

  1. The shared helper's test asserts one record; the stage enforces half.

    • Commit 31 tightened the Carer's Allowance refusal to half the heaviest record. But packages/microcosm-build/tests/engine_free/shared/test_stochastic_assignment.py:203 and the CA tests still assert within one whole record.
    • A regressed helper that places records at the end of their slice passes that test, yet misses the half-record bound in 3 of 18 stratum × rate cells. Only the CA stage's runtime refusal would catch it, so other users of the helper are unguarded.
    • These texts still say "within one record":
      • CARERS_ALLOWANCE_RATE_DERIVATION (uk_runtime/carers_allowance_take_up.py:212), copied verbatim into sources.yaml, source_stages.json and the H2 fixture;
      • the pool docstring;
      • the helper's docstring.
    • Each receipt prints that rule next to a constant strata_within_half_a_record: True.
  2. More texts that commits 16, 30 and 32 made false, adding to Vahid's finding 5:

    • uk/uk_population_targets.json:17772: the betting and gaming row's notes still say it "reads the HMRC receipts back by construction … not the weights. Held out of the objective as a measured diagnostic". Commit 16 deleted that exclusion. Its message says the notes were updated, but only line 885 was.
    • tools/diagnose_uk_consumption_taxes.py:7-15, which is also its --help, says gas takes the ONS-spend level and is bound.
    • uk/spec/sources.yaml:4314 says the stage scales both fuels to Energy Trends. So do source_stages.json, the H2 fixture, uk_runtime/diagnostics.py:655 and uk_runtime/lcfs_consumption.py:390.
    • uk/gates.json:1095 says "no bound row … holds calibrated domestic energy".
    • uk_runtime/local_target_census.py:271, also in the committed census, says the domestic energy cells sit "at the lcfs stage's Energy Trends level".
    • The regenerated compile-parity receipt still says the 13 Table 3.7 interest amount rows grow by GDP per head (tools/build_uk_ledger_compile_parity_signed_differences.py:83-91).
    • uk_runtime/spi_income.py:203-205 says the donors and the band targets share one index, held by an engine lockstep test. That test's branch (packages/microcosm-build/tests/engine/uk/test_uk_hmrc_uprating.py:78) can no longer run, because commit 32 removed the assertion that showed it ran.
  3. The RFDR layer isn't tied to the registered NSPL, and rerunning the tool changes its bytes.

    • In tools/build_uk_rfdr_area_layer.py:449, _registered_sha256 accepts any manifest with one file. It checks no package id or pinned digest, unlike the BRMA tool.
    • The zip member is picked by glob, and the provenance hard-codes the August 2026 manifest path, whatever --nspl-manifest says.
    • The 7b test (test_uk_rfdr_area_layer.py:242-243) compares two fields the tool writes in the same run, so it passes for any zip. uk/high_value_home_stock.provenance.json holds an independent copy of the registered digest to check against.
    • --built-on defaults to today (:485), and there is no --check. The 7b rebuild changed the layer's sha256 only through built_on, which moves the run-time pin, the checkpoint identity and the node's source digest.

Smaller items

  • --rfdr-localisation defaults to area, so a legacy-assignment build through the CLI fails unless it passes none. docs/uk-full-build-graph.md doesn't mention RFDR.
  • The new ratio bounds are read with a bare float() (uk_runtime/stage_health.py:436), so infinite bounds pass. The rest of the gate uses _finite_number.
  • Copies of existing code:
    • weight_balanced_uniforms repeats birth_month.stratified_positions, which is already on main. The two already differ on NaN weights.
    • _prepare_rfdr_localisation repeats the BRMA one.
    • energy_pricing.py:135 and :146-151 repeat lcfs_consumption's ONS resource and row constants, which docs/shared-constants.md asks to define once.
  • The Gambling Commission machine rows are vendored with no reader. Their purpose still names the split that left the PR.

Checked and found true

  • The electricity level reproduces from the vendored rows, and so do the other QEP payment methods' ratios.
  • The 977 local gas cells are exactly back at their merge-base values. The 980 electricity cells move ×0.925811.
  • The constituency energy facts are unchanged apart from order and feed identity.
  • The OBR savings factor reproduces: ×1.356187 against ×1.19613 before, +13.4%.
  • Only the intended national references move: 29 added, 13 re-indexed and the winter fuel pin.
  • Table 3.10's band edges and its signed-out complement are right, and the count pins match the registers.
  • The gate digest mirror tests pass.
  • weight_balanced_uniforms matches policyengine-uk's stratified_uniform.
  • RFDR postcode matching uses exact outward codes, so KW1 and KW10 don't collide.
  • The Carer's Allowance receipt's per-stratum shares are support-banded.

Not covered

  • Any build or engine test. Everything above ran engine-free on Python 3.13.
  • Spine support for the 29 new SPI rows.
  • Carer's Allowance record counts per stratum.

One disclosure-control item went to the author separately.

🤖 Generated with Claude Code

juaristi22 and others added 2 commits October 30, 2026 01:18
The independent review (at b3f4ebf) and Vahid's (at 6919f44), with
María's rulings of 29 October on the points that were hers.

Pension income by area (independent 1, 8, 9):
- The pension bridge's index bound is 0.02, the measured 1.5% gap
  (State Pension bands x1.224 on #1069's index, private x1.103, together
  x1.151 against the cells' x1.134) plus half a point. A test recomputes
  every fan-out bridge's index gap from the committed registers.
- A bridge may declare min_factor_shift; the pension bridge's source
  factor must sit between 1.025 and 1.037, as the gap runs one way.
- The rounded-thousand floor reaches the pension income counts and,
  through a new floor_reads, the amounts, which are the count times the
  mean (María: counts and amounts). The bridge counts the small-cell
  deferrals at source, grown by the bound cells' own index, and a
  deferral joins the reconciliation grown the same way, as an
  off-surface cell does (the pension cells are the first deferred ones
  an index moves).
- The SPI-frame fence's authority, the census family and both
  adjudication records name the pension metrics.

Data and calibration:
- Table 3.10's amounts grow by the count index times OBR mixed income
  per head (independent 2).
- The RFDR node refuses a system with nothing flagged only at full
  scale; sampled rungs check the codes (independent 3, Vahid 3).
- The Carer's Allowance placement is rotated by a start drawn per
  stratum, and the stage refuses a stratum off by more than one record
  (independent 4 and 13).
- The LCFS stage, the Carer's Allowance stage and the spine gate key
  their caches on energy_pricing, stochastic_assignment and stage_health
  (independent 5); the GB energy level factors are computed once per
  process (independent 6).
- The zero-support floor's HMRC scopes take Table 3.7's other income and
  Table 3.10's bands (independent 7).
- ONS electricity is measured at 2024, as road fuel is, with a signed
  matched-period hold (independent 10; María: at 2024).
- A test holds each bound ONS energy row to a fuel levelled to ONS
  spending (independent 11).

Texts (independent 14, 15 and smaller items; Vahid 1, 4, 5, 6):
- The stale energy, betting and gaming, census, parity, SPI and gate
  texts are corrected; the savings path's resource notes record its
  scope and the HAXV comparison; the gate notes record the June
  vintage's 0.840; the cars fuel duty figures say which edition.
- The RFDR tool pins the NSPL registration and gains --check; its 7b
  test reads the high value home stock's record of the zip.
- --rfdr-localisation follows the geography assignment when unset; the
  ratio bounds are read as finite numbers; energy_pricing defines the
  ONS row variant once; the build graph doc describes the RFDR node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the pinned feed (53815b2):
- National references and membership: 1,403 active, as before. Only
  the intended values move: the eight Table 3.10 amount rows (x1.0763,
  the count index now applied) and ONS electricity onto its calendar-2024
  figure (x0.975), which the membership now lists as a matched-period
  hold.
- Local references and membership: 27,168 active. The rounded-thousand
  floor defers 40 pension income cells (17 constituency and 3
  local-authority counts, and their amounts); nothing else moves.
- The local census, both compile-parity receipts, the release input
  coverage manifest and the H2 spine fixture.
- The gate battery's manifest digests and the certification parts' move
  with the gate notes; the policy digests are unchanged.
- The battery bindings test's local pins: 27,168 candidate targets and
  4,179 reviewed exclusions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Thanks, Vahid. All seven are addressed at ead8943d6: commit 36 makes the changes and commit 37 regenerates the surfaces from them.

  1. The OBR savings path: notes added. The purpose of obr_savings_income_growth_facts.json, and of the selection it is vendored from, now records three things:

    • the scope: Table 4.5's row is the savings income of self assessment filers, while Table 3.7 counts every taxpayer, so part of 2024-25's +35.0% may be savers entering self assessment;
    • your HAXV comparison: ×1.048 from FY2023-24 to calendar 2025, against ×1.356 on this path, and why the two aren't like for like;
    • the engine still projects interest at GDP per head (Project savings interest by interest received per person policyengine-uk#2638), so the solve reaches the level only by moving weight toward savers.

    The next national, built once the reviews close, adds the records that carry interest to Anthony's targeted check (the maximum-to-median weight ratio and the effective sample size), beside the betting carriers. The ruling stays María's.

  2. Table 3.10's two lowest bands: measured, and the gap is small.

    • PR A's national (4374e0bf) binds no Table 3.10 row, so I measured on its final weights rather than design weights. The two lowest bands' count ratios to the targets are 1.01 (£1 to £500) and 0.94 (£500 to £1,000).
    • The largest gaps are higher up: 0.83 at £3,000 to £4,000 and 0.77 at £7,500 to £10,000.
    • Every band rests on 10 or more source households.
    • So all eight bands stay bound, and María can still choose to bind from £1,000. The amounts now grow on the count index times mixed income per head (the independent review's finding 2).
  3. RFDR on sampled rungs: fixed.

    • The node takes strict_flags from the build's full_scale, as the private-rent node does, and only a full-scale build refuses a system with nothing flagged.
    • The malformed-code refusal stays unconditional.
    • The independent review estimated the odds of a refusal on a sampled rung (about 27% per seed at f001 with K=15), so no area-mode run at 0.01 was needed to show it.
    • The CLI's --rfdr-localisation now follows the geography assignment when unset.
  4. The revision note: added. The gate's notes and the stage-health docstring now say the June vintage gave 0.840. A re-pin that crosses the bounds asks for the level to be re-adjudicated, and isn't by itself a data error.

  5. The stale texts: fixed.

    • stage_health.py says only the 04.5.1 row holds the calibrated frame.
    • The gas row's notes say it is measured and held out while it counts bottled gas.
    • Electricity's incumbent note says it is bound again.
    • The 04.5 total's notes and its signed exclusion say electricity is bound and gas is held out.
  6. The fuel duty figures: dated. The −9.9% and −8.1% were measured with the stage levelled to the June 07.2.2, and the reason now says so. It adds that the September edition's 2024 figure is 4.2% lower. The next national re-measures them with tools/diagnose_uk_consumption_taxes.py.

  7. The Draft wording: fixed in the body. The PR is ready for review, and merging waits on CI and María.

🤖 Generated with Claude Code

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Reply to the independent review. Every finding is addressed at ead8943d6 (commit 36 makes the changes, commit 37 regenerates the surfaces), except 12: María ruled to measure that one first.

Must fix

  1. Fixed.
    • max_index_shift is now 0.02, the 1.5% gap plus half a point.
    • The bridge's reason, the selector note and the changelog give the indices after UK pensions after #1084: open rulings, residual gaps and the engine pieces #1069: ×1.224 for the State Pension bands and ×1.103 for the private ones, together ×1.151 against the cells' ×1.134. The changelog now says 3.7% where it said 4%.
    • It also gives the factors the compiled surface applies, ×1.045 at both grains. The local-authority grain implies its two authorities without SPI measures from the constituencies, so it reads ×1.045 rather than the ×1.048 the cells alone give.
    • A new test, test_each_committed_bridge_holds_its_declared_index_gap, recomputes every bridge's index gap from the committed national membership with the runtime's own ITL readers. A stale declaration now fails in CI.
    • The source check is one-sided where the gap is. A bridge may declare min_factor_shift, and the pension bridge holds its source factor between 1.025 and 1.037.

Should fix: data and calibration

  1. Fixed as suggested. Table 3.10's amounts grow by the count index times OBR mixed income per head, so each band's mean moves as the engine moves incomes. The new applier is hmrc.itl_2026.taxpayer_count_growth_all_bands_times_obr_per_capita_mixed_income. A unit test pins the factor, and the engine lockstep test checks the mixed-income half against the engine's projection.
  2. Fixed as suggested. strict_flags follows full_scale, and the malformed-code refusal stays unconditional. Tests cover both.
  3. Fixed, with one correction.
    • Each stratum's placement is rotated by a start drawn per stratum from the identity seed (systematic sampling), so each record claims with probability equal to the rate.
    • Rotation doesn't keep the half-record balance, though. The claimed records form an arc, and the arc can end mid-record at both ends, so the bound is one record: half at each end.
    • The stage now refuses a stratum off its share by more than its heaviest record. The receipt key is strata_within_one_record, and the texts say one record.
    • Tests: a one-record stratum claims at the rate over many seeds; the helper without starts still holds half a record, and with starts one.
  4. Fixed.
    • lcfs_consumption hashes energy_pricing.
    • The Carer's Allowance stage hashes stochastic_assignment.
    • The spine gate hashes energy_pricing and stage_health.
    • Tests pin each.
  5. Fixed. The Great Britain level factors are computed once per process from the declared inputs: 642 ms on the first call, then 0.03 ms.
  6. Fixed.
    • The HMRC count scope adds Table 3.10's counts.
    • The amount scope adds Table 3.7's other income and Table 3.10's amounts, within the existing ceilings.
    • The floor's reason names them, and test_uk_support_floor.py pins them.
    • The support count on a spine comes with the next national, built once the reviews close.
  7. Fixed (María's ruling: pension counts and amounts).
    • hmrc_rounded_thousand_floor now reaches hmrc.pension_income.count.
    • It also reaches hmrc.pension_income.amount through a new floor_reads: each amount is the count times the mean, so it defers where its count does. That defers 17 constituency and 3 local-authority cells of each.
    • The bridge now counts small-cell deferrals at source, grown by the bound cells' own index. Without that, the 17 deferred constituency amounts (0.7% of the total) would have moved the source factor from 1.030 to 1.037, onto the bound.
    • A deferral keeps its published 2023-24 value. The surface now grows it by its target's bound cells' own index before it joins the reconciliation, as an off-surface cell already was: ×1.076 for the counts and ×1.134 for the amounts. Before this, it took a share 7% and 12% too small. No other deferred family carries an index, so their rows are unchanged.
    • The dense target surface now compiles end to end (tools/uk_target_surface_receipt.py on the pinned feed). The pension bridge reads source factors of 1.0295 and 1.0296, with the deferred cells counted, and applies ×1.0450 and ×1.0451.
    • The rule's reason records that the ruling reaches pension income only. The 405 self-employment count cells below the floor stay bound, as a known rounding error.
  8. Fixed. The fence's authority and the census family name the pension declarations. Both adjudication records note the extension and say the basis is unchanged.
  9. Fixed (María's ruling: at 2024).
    • The electricity row declares measurement_period 2024 and require_matching_fact_period. It reads the frame's 2024 column against ONS's 2024 figure, as road fuel does.
    • That makes the row a period hold, which the full target compile refuses unless declared. So uk/uprating_holds.json gains a signed matched_period entry beside road fuel's. Without it, every build would have stopped at target compilation.
  10. Fixed.
    • test_each_bound_ons_energy_row_is_a_fuel_levelled_to_ons_spending holds every bound per-fuel ONS row to a fuel in spend_level_fuels.
    • The gas exclusion's reason says the row rebinds only with gas in that list.
  11. Measure first (María's ruling). The OBR total stays bound. The next national measures each carrier against HMRC's receipts by duty, and María decides on the split from that.

Should fix: tests and texts

  1. Fixed, with 4. The stage, its receipt and every text say one record, which is what the rotated helper guarantees. The helper's own tests assert half a record without starts and one record with them.
  2. Fixed, at each place you listed.
    • The betting and gaming notes say the row is bound since UK: data fixes for the Budget 2026 household measures #1418, why, and what one row can't hold.
    • The consumption-tax tool's docstring is updated.
    • sources.yaml, source_stages.json, the regenerated fixture, diagnostics.py and lcfs_consumption.py give each fuel's own level.
    • gates.json's note for the NHS budget gate is updated.
    • The census's domestic energy text is updated, and the census is regenerated.
    • The parity tool's rationale now covers other income too, and the receipt is regenerated.
    • spi_income.py's comment is updated. The engine lockstep test now asserts both savings indices unconditionally and checks the new Table 3.10 amount index.
  3. Fixed.
    • The tool pins the NSPL registration's package id, file, sha256 and zip member, and refuses any other manifest.
    • The provenance names the manifest the tool read.
    • --check rebuilds the layer in memory and compares it with the committed files, writing nothing. Run against the registered zip, it reproduces the committed layer and provenance byte for byte.
    • The 7b test holds the provenance to high_value_home_stock.provenance.json's registration and to the tool's pin.

Smaller items

  • --rfdr-localisation now follows the geography assignment when unset, as --private-rent-localisation follows the BRMA mode. docs/uk-full-build-graph.md describes the node.

  • The ratio bounds are read with _finite_number.

  • energy_pricing defines the ONS resource and row variant once, and lcfs_consumption imports them. A test holds the two views together.

  • The machine rows' purpose now says they have no reader until the Machine Games Duty split's own PR.

  • Two are not changed:

    • weight_balanced_uniforms now rotates by a start per stratum, which stratified_positions doesn't, so merging them would move birth months.
    • _prepare_rfdr_localisation has the BRMA one's shape but pins a different resource.

    Either can be merged in a follow-up.

Checks

Run locally for commits 36 and 37: the 107 engine-free test files that read the regenerated surfaces or that this round touches; the engine tests for HMRC uprating, Carer's Allowance and graph admission; the coverage manifest check; the H2 fixture's readers; tools/build_uk_rfdr_area_layer.py --check against the registered NSPL zip, which reproduces the committed layer byte for byte; and tools/uk_target_surface_receipt.py, which compiles the dense target surface end to end on the pinned feed. All pass. The vendoring register's check that each consumer names its file failed once and is fixed in commit 36. The Orrery reconstruction test needs a Node package this checkout doesn't have; CI runs it.

🤖 Generated with Claude Code

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Re-review of #1420 at ead8943d6

This checks the independent review's 15 findings (issuecomment-6510008298) against commits 36 and 37 and the reply, then reviews the new code. Five reviewers re-checked the findings and scanned the two commits, and a verifier checked each new candidate. Everything ran engine-free; where an item rests on a probe or a mutation run, it says so. Paths are relative to packages/microcosm-build/src/microcosm/build/ unless they start with packages/, tools/, docs/ or changelog.d/.

Verdict: finding 1 is resolved, and nothing in this round blocks merge.

  • The pension bridge, re-run on the runtime surface at this head with the deferrals counted, passes at both grains: source 1.0295 and 1.0296, index 1.0151, applied ×1.0450 and ×1.0451.
  • Two groups are worth doing before the next national:
    • A: the tool the PR names for re-measuring now misreads the electricity row;
    • B to D: test gaps that would let round-1 problems come back with CI green.

Round-1 findings

# Status Evidence
1 Resolved Passes at both grains as above. test_each_committed_bridge_holds_its_declared_index_gap fails on the round-1 registers, so it isn't vacuous. The source half is still checked only in a dense build (D).
2 Resolved Table 3.10 amounts grow ×1.1176 (×1.0763 × ×1.0384) on exactly the 8 bound rows, through the one applier table.
3 Resolved strict_flags follows full_scale end to end, every default is strict, and it enters the node key. Untested (C).
4 Partly The per-stratum rotation gives each record the rate over starts and holds one record. But the start is keyed on the stratum code alone (G), and no stage-level test checks that the stage applies it (B).
5 Resolved Probes re-key the LCFS stage, the Carer's Allowance stage and the spine gates when those modules change. The spine gate addition is broader than needed and has no test pin (E).
6 Resolved lru_cache on the zero-argument default path: 0.63 s on the first call, then 0.03 ms. It returns copies, and explicit parameters stay uncached.
7 Resolved The scopes are extended. The 2-row cap signed for seven amount families now covers nine (H).
8 Resolved Exactly 40 references are deferred (17 constituency and 3 local-authority cells, count and amount), so active local references go from 27,208 to 27,168. Deferrals grow by their bound cells' index; nothing else moved. The floor reads the uprated value (F).
9 Resolved in text The fence authority, the census family and both adjudications name the pension metrics. The signing is open and the filter name in the new text is wrong (H).
10 Resolved The row is measured at 2024 under a signed matched-period hold (María's ruling), and the compile reads the 2024 fact. The diagnostic tool didn't follow (A).
11 Resolved The new test fails if the gas exclusion is deleted, or if electricity leaves spend_level_fuels.
12 Declined María's ruling: measure each carrier on the next national first. The notes now say what one row can't hold.
13 Resolved "One record" everywhere. The helper tests assert half a record without starts and one with them.
14 Resolved Every listed text is fixed. Two residuals: the receipt's new rationale also lands on the 13 count rows (I), and uk_runtime/lcfs_consumption.py:3614-3615 still says "the NEED shape at the DESNZ level".
15 Partly The registration pins, the manifest label, --check and the anchored test are in. Two gaps remain: --built-on still defaults to today when writing, and --check without --out raises FileNotFoundError instead of reporting a difference.

New findings

A. The consumption-tax diagnostic compares the engine's 2025 electricity with the 2024 target.

  • tools/diagnose_uk_consumption_taxes.py:55-61 (ROW_YEARS) lists only the five excise rows.
  • So the electricity row is now measurement_period 2024, against ONS's 2024 £28.129bn, but the tool still measures it through the engine at --year (default 2025).
  • On the next national, a frame that fits the bound 2024 row exactly will report about +3.4%. That is the CPI projection the matched-period hold was added to keep out, and the PR's texts name this tool for the re-measurement.
  • test_excise_rows_are_measured_at_their_own_period checks only that ROW_YEARS is a subset of the declared periods. Road fuel has had the same gap since UK national calibration reads stored input columns at the frame period: 76 £ rows compare 2024 values with 2025 targets #1324.
  • Suggestion: derive ROW_YEARS from each row's measurement_period in the contract.

B. The Carer's Allowance stage's placement isn't tested at stage level.

  • Commit 36 removed the assertions that tied the stage's flags to the balanced uniforms (assert_array_equal(flag[eligible], positions < 0.6) and the pool's equivalent). The replacements check only share bounds.
  • Mutation runs: with the stage's starts= dropped, or its identity-keyed draws replaced by row-position draws, every test in test_uk_carers_allowance_take_up.py still passes, though 99 and 152 of the fixture's 640 flags change.
  • So round-1 populace-data: registry-driven loader for published populations #4 could return with CI green.
  • Suggestion:
    • assert that the flags equal weight_balanced_uniforms(..., starts=stratum_starts(...)) < rate, for take-up and for the pool;
    • add a row-permutation check.

C. The RFDR strictness switch isn't exercised by any test.

  • Neither of these lines is reached:
    • strict=full_scale at uk_runtime/graph_population.py:1334-1336;
    • the kernel's require_flagged=strict (:738).
  • The new tests call _rfdr_area_nodes(strict=False) and area_flags(require_flagged=False) directly.
  • Mutation run: forcing strict everywhere passes every RFDR graph and layer test, which would bring round-1 populace-calibrate: representation operator (targets -> calibrated weights) #3 back on sampled rungs.
  • The private-rent node's tests cover all three levels; the same pattern would work here.

D. The bridge's source half is still checked only inside a dense build.

  • test_each_committed_bridge_holds_its_declared_index_gap asserts only the index half (test_uk_fanout_sum_bridges.py:466).
  • The source half, including the new one-sided floor, runs only in uk_local_target_surface: dense compilation, or tools/uk_target_surface_receipt.py with the ledger facts. No test passes committed controls through it, so the deferred_cells wiring (uk_runtime/ledger_targets.py:3274) is never exercised.
  • The constituency source factor (1.0295) clears the 1.025 floor by 0.45 points. A feed re-pin, or a change in which cells are deferred, could leave CI green and stop every dense build: round 1's failure, on the other half of the factor.
  • Dropping the deferral wiring gives 1.0366, still inside the bound, so even the runtime wouldn't notice the applied factor rising by about 0.7%.
  • A check of both halves on the committed registers runs engine-free in about 20 s.

E. The spine gate's hash now re-keys 14 spine stages that never read energy.

  • One UKSpineGateKernel hash serves both gate phases (uk_runtime/graph_evidence.py:374-380).
  • The assembled phase evaluates no energy gate, yet its key now includes energy_pricing, and that key feeds frs_hmrc_spine_leaves and everything after it.
  • Probe on the production spine graph:
    • an edit to energy_pricing.py moves 36 of 49 stage keys;
    • an edit to lcfs_consumption.py moves 22;
    • the extra 14 run from frs_hmrc_spine_leaves to nts_bus_travel, including the SPI support channel and the income spine.
  • stage_health belongs in both phases, but energy_pricing only in the transferred one. It could go in as a node param on spine.gates.transferred, or through a kernel per phase.
  • Round 1 suggested adding both modules, so this is partly the review's own suggestion. No test pins the gate's modules.

F. The rounded-thousand floor reads the uprated count, not the published one.

  • tools/generate_uk_local_target_references.py:220-241 compares each count's 2025-indexed resolved_value with 5,500.
  • The rule's reason, the deferral rationale ("published value … below 5500") and the changelog all describe the published count. Pension counts are the first indexed family under this rule.
  • Today it defers the right 20 cells: published counts are multiples of 1,000, and 5,000 × 1.0763 is under 5,500.
  • At a 2026 period (×1.1049 on the vendored ITL rows), 9 of the 17 constituency cells published at 5,000 would bind again, with their amounts. CI's pin would fail, but it would read as data movement.
  • Suggestion: compare the published value (value_before_uprating).

G. The Carer's Allowance start is keyed on the stratum code alone.

  • stratum_starts (uk_runtime/carers_allowance_take_up.py:342-354) hashes only the code, under seed 0. So each stratum's start is the same in every build and every FRS vintage.
  • A one-record stratum's outcome is then fixed per code. Computed from the public constants alone: at 0.6, the one-record strata of 9 of the 16 take-up codes always claim and 7 never do; at 0.45, 5 always claim.
  • Within a build the expected claimed weight is still right. But this claim holds only across codes: "every carer claims with probability equal to the rate, however few share the stratum". It appears in the module docstring, rate_derivation, the pool docstring and the changelog.
  • Suggestion, either:
    • key the start on the code plus the stratum's sorted member ids. That stays reproducible and order-free, and a singleton's outcome follows its own identity;
    • or reword the texts to "across strata".

H. Three signed records were widened without a recorded approval.

  • Commit 36 extended these records, citing "the independent review's finding 7/9", with approved_on left at 22 and 14 October:
    • the zero-support floor's scopes (uk/calibration_measure_exclusions.json): its 2-row cap, signed for seven amount families, now covers nine;
    • both pension-related fence adjudications (uk/local_binding_adjudications.json:15-23).
  • The cars fuel duty reason (:392, approved 8 October) was also edited, and its adjudication doesn't record it.
  • Earlier changes to scope re-dated approved_on and cited María's ruling: b176dff, and the small-cell rule in this same commit. Earlier re-wordings recorded who made them (5a0c3f6).
  • The floor's reason and adjudication are part of the signature its readers compare (uk_runtime/support_floor.py), so this edit amounts to a re-signature.
  • The new adjudication text says the pension references "stand on the same income_tax_annual > 0 filter", but the declarations use income_tax_before_winter_fuel_payment_charge_annual > 0.
  • These need María's ruling recorded, and the 2-row cap re-sized or confirmed for nine families. No more code is needed.

I. Texts.

  • uk/calibration_measure_exclusions.json:392: the inserted clause makes the cars fuel duty reason read "…so the next national re-measures it (…), so binding it would pull weight toward fuel buyers…".
  • uk_runtime/energy_pricing.py:147-149: the comment says the ONS row variant serves the road-fuel level. In fact the road-fuel level and the heating recomposition (lcfs_consumption.py:2131, :3040) filter on their own literals, and the new test checks only the import.
  • The regenerated parity receipt gives the 13 Table 3.7 interest count rows the OBR savings rationale, but they grow by taxpayer counts. The prefix at tools/build_uk_ledger_compile_parity_signed_differences.py:221-225 matches both measures.
  • docs/uk-full-build-graph.md:33 says "postcode districts and sectors". The definition has no sectors; it works through districts, unit postcodes and excluded units.
  • docs/uk-local-cross-grain-reconciliation-802.md:96-106 and :120-121 still describe a symmetric 2% source check, and count a deferral at its published value.
  • The PR body's items 4, 24 and 31 still give the 0.1% gap and 4%, the Great Britain-wide zero-flag refusal, and half a record, with no note of the later commit.
  • tools/build_uk_rfdr_area_layer.py:66 pins the NSPL member as a new literal beside NSPL_DATA_MEMBER in the high-value home stock tool. The 7b test cross-checks the package, file and digest, but not the member.

Checked and found true

  • Nationally only these move:
    • the 8 Table 3.10 amount rows (×1.0384 → ×1.1176);
    • the electricity row (2025 → 2024);
    • the 04.5 rationale text.
  • Locally only the 40 pension cells move, plus the re-dated rationale on the 192 higher-rate deferrals.
  • The H2 fixture changes only the two stage contract digests that source_stages.json changed. The coverage manifest changes only its source digest, and the gate digests match their pins.
  • Every new default is safe, and every production caller passes it: strict_flags, require_flagged, min_factor_shift, starts, deferred_cells and floor_reads.
  • An unset --rfdr-localisation on the atomic assignment resolves to area and pins the committed layer, before the checkpoint identity is computed.
  • The level-factor cache returns the uncached result and returns copies.
  • The new uprating hold's approver and date match María's ruling of 29 October.
  • CI passes on every job at ead8943d6.

Not covered

  • Any build or engine test. Everything ran engine-free on Python 3.13.
  • Spine support for the 29 new SPI rows and the floor's 2-row cap.
  • Carer's Allowance record counts per stratum.

Two disclosure-control items went to the author separately.

🤖 Generated with Claude Code

The re-review (issuecomment-6513556513) found nothing blocking; these
are its groups A to I and round 1's residuals 14 and 15.

- A: tools/diagnose_uk_consumption_taxes.py derives each row's period
  from the contract's measurement_period, so electricity and road fuel
  are measured at 2024 as the national binds them.
- B: stage-level tests pin the Carer's Allowance placement (identity
  draws, member-keyed starts, flags against the pool share and the
  rate) and its independence from row order.
- C: the RFDR strictness switch is tested through the graph builder and
  the kernel: a full-scale run refuses an unflagged system, a sampled
  rung runs.
- D: each fan-out bridge's source half is checked on the committed
  registers, and the surface's deferral wiring is tested.
- E: only the transferred spine gate keys on energy_pricing, as a node
  param; the kernel hash keeps the gate battery, its bindings and
  stage_health.
- F: the small-cell floor reads each cell before its index.
- G: a Carer's Allowance stratum's start is keyed on its code and its
  members' sorted identities.
- H: the widened floor and fence records say they are pending María's
  confirmation (no approved_on moves); the pension adjudication names
  the right filter; the cars fuel duty record notes its re-wording.
- I and residuals: the texts the re-review listed, the RFDR tool's
  build date and --check on a missing file, and the NSPL member
  cross-check.

The national compile-parity receipt is regenerated for the interest
count rows' rationale; the local references regenerate unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Reply to the re-review. Everything in it is addressed at 216f22a5f (commit 38), except H's approvals, which are María's to record.

Worth doing before the next national

  • A. Fixed. tools/diagnose_uk_consumption_taxes.py now derives ROW_YEARS from each row's measurement_period in the contract. Electricity and road fuel are measured at 2024 on the frame's own columns, and the excise rows at their FY2024-25 period. The test now requires ROW_YEARS to equal the contract's declared periods for every measured row, so the road fuel gap from UK national calibration reads stored input columns at the frame period: 76 £ rows compare 2024 values with 2025 targets #1324 is closed too.

  • B. Fixed.

    • test_the_stage_places_each_carer_by_their_identity_and_their_strata spies on both balanced placements. It checks that each draw is the carer's own identity-keyed uniform, that each start is the member-keyed one, and that the flags are those positions against the pool share and then the rate.
    • test_the_stage_draws_the_same_carers_whatever_the_row_order permutes the person rows and compares the flags by person.
    • Both of your mutations now fail. Dropping the take-up starts= fails the starts check. Row-position draws fail the identity check.
  • C. Fixed. Two tests go through the real graph builder:

    • strict_flags is the build's full_scale;
    • with a layer whose England and Wales areas no household is placed in, a full-scale run refuses and a sampled run flags none and goes on.

    Forcing strictness on, in the kernel or in the caller, now fails both.

  • D. Fixed.

    • test_each_committed_bridge_holds_its_declared_source_gap recomputes the source half of every fan-out bridge at each grain from the committed registers, the deferred cells included, and holds it to the bridge's bounds. The one-sided pension floor is among them.
      • Constituency readings match the compiled surface (pension 1.0295, employment 1.0046).
      • Local-authority readings are slightly higher (pension 1.0321), because the surface also fills in the two authorities without SPI measures.
    • test_the_surface_hands_its_placed_deferrals_to_the_fanout_bridges covers the deferred_cells wiring in uk_local_target_surface.

Smaller items

  • E. Fixed.
    • The spine gate kernel's hash covers gate_battery, battery_bindings and stage_health.
    • Only spine.gates.transferred, which evaluates the energy rake, carries energy_pricing's hash, as a node param. An energy edit no longer re-keys the assembled gate or the spine stages after it.
    • Tests pin both: the kernel's modules, and the param on the transferred node only.
  • F. Fixed.
    • The rule reads each cell at its value before the index (value_before_uprating from the authoring pass), the published count its floor is set on.
    • A new test grows a count published at 5,000 past 5,500 and still defers it.
    • The regenerated local references are unchanged, as expected: published counts are multiples of 1,000.
  • G. Fixed by keying. stratum_starts keys each stratum's start on its code plus a digest of its members' sorted identities. A rebuild of the same people rotates each stratum the same way, whatever the row order. A carer alone in a stratum now claims with probability equal to the rate across identities, so the texts' claim holds per carer and per build. The one-record test now samples identities under fixed codes.
  • H. Texts fixed; the approvals are María's.
    • The three widened records now say plainly that they were extended in this PR, in answer to findings 7 and 9, and are pending María's confirmation. The floor's record adds that its 2-row amounts cap, signed for seven families, now covers nine, for her to confirm or re-size. No approved_on moves, since that would record a signature she hasn't given.
    • The pension adjudication now names income_tax_before_winter_fuel_payment_charge_annual > 0.
    • The cars fuel duty record notes the re-wording; its ruling and figures are unchanged.
  • I. Fixed.
    • The cars fuel duty sentence now reads cleanly.
    • energy_pricing's comment says what reads the ONS row variant: the energy level and the division diagnostic. The road-fuel level and the heating recomposition read one row per class on its code and frequency.
    • The parity receipt's rationale gives the interest counts their own growth: HMRC's projected taxpayers in the same Table 2.5 band.
    • The build graph doc and the RFDR changelog say "districts, unit postcodes and excluded units".
    • docs/uk-local-cross-grain-reconciliation-802.md describes the declared, possibly one-sided source bounds and how deferrals are counted and grown.
    • The 7b test checks that the two tools read the same NSPL member.
    • The PR body's items 4, 24 and 31 now point to the commits that revised them.
  • Round 1's 14. lcfs_consumption's rake docstring gives each fuel's own level.
  • Round 1's 15.
    • Without --built-on, the layer keeps its committed date unless something else in it moves.
    • --check against a missing file reports a difference (exit 1).
    • Run against the registered zip, it still reproduces the committed layer and provenance byte for byte.

Checks

Run locally for commit 38: the 31 engine-free test files this round touches or that read what it changed; the Carer's Allowance and HMRC uprating engine tests; the coverage manifest check; the mutation runs above, each of which now fails a test; and tools/build_uk_rfdr_area_layer.py --check against the registered NSPL zip, which still reproduces the committed layer byte for byte. All pass. The local references regenerate unchanged, and the national compile-parity receipt changes only in the income-anchor rows' rationale.

🤖 Generated with Claude Code

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Re-review of #1420 at 216f22a5f (round 3)

This checks round 2's items (issuecomment-6513556513) against commit 38 and the reply, then reviews commit 38. Two reviewers re-checked the items and re-ran round 2's mutations, and a verifier checked each new candidate. Everything ran engine-free. Paths are relative to packages/microcosm-build/src/microcosm/build/ unless they start with packages/, tools/ or docs/.

Verdict: nothing blocks merge.

Round-2 items

Item Status Evidence
A Resolved ROW_YEARS comes from each row's measurement_period in the contract, and the test requires the two to match. Two residuals remain (5).
B Resolved The stage tests spy on both placements and permute the rows. Dropping either starts= (take-up or pool) now fails, and so do row-position draws.
C Resolved Both tests go through the graph builder. Forcing strictness in the caller or in the kernel fails a test.
D Partly The source half is tested on the committed registers, and the deferral wiring has a test. But the local-authority reading isn't the runtime's (1).
E Resolved An energy_pricing.py edit now moves 22 stage keys, from lcfs_consumption on, and leaves the assembled gate alone. The new param enters the node key, and tests pin both.
F Resolved The floor reads value_before_uprating, and reverting that fails the new test. No reference moved.
G Resolved Simulated with the real helpers: a one-record stratum claims at the rate over identities (0.59 to 0.60 at a rate of 0.6). The start is order-free and the one-record bound holds. One cost remains (2).
H María's The three records say they're pending her confirmation. The filter is named correctly, and no approved_on moved. Two text gaps remain (4).
I Resolved Every listed text is fixed; small residuals are in 6.
#4, #14, #15 Resolved —

New findings

1. The local-authority source check reads a different quantity from the runtime.

  • test_each_committed_bridge_holds_its_declared_source_gap (test_uk_fanout_sum_bridges.py:469-510) reads the pension bridge's local-authority source factor as 1.0321.
  • The runtime surface reads 1.0296, because it also counts the two authorities without SPI measures. The test's docstring says it leaves them out.
  • Mutation run: with the local-authority cells ×1.005, the runtime reads 1.0245 and refuses, below the 1.025 floor. The test reads about 1.027 and passes.
  • So a rise of about 0.45% to 0.7% confined to the local-authority cells keeps CI green and stops every dense build.
  • The wiring test (test_uk_off_surface_cells.py:737) replaces the check with a spy and passes no fan-out controls, so it covers only the hand-off.
  • Suggestion: compute the factor through uk_local_target_surface on the committed pension cells, with the real controls. The review's script does this in about 20 s. Both grains would then read the runtime's quantity.

2. Keying the start on every member re-draws about half a stratum when one member changes.

  • stratum_starts (uk_runtime/carers_allowance_take_up.py:352-382) digests the sorted person ids of each stratum's placed set. Any change to that set re-randomises the start.
  • When one member leaves, about 2r(1−r) of the other members' flags flip, whatever the stratum's size: 0.48 at a rate of 0.6. Keyed on the code alone (round 2's scheme), it was 0.11 at 5 members and under 0.01 at 80.
  • In a stage-like set of 320 eligible carers in 20 strata, one carer changing stratum flips on average 27.5 of the other 319 flags (1.3 with code keying). The weight-based pool share adds to this: a prior-weight change that moves one carer across the pool line re-draws whole take-up strata.
  • So a policyengine-uk re-pin or an SPI refit that moves one carer's band or earnings test re-draws most of that stratum. Household-level comparisons between builds churn, while the aggregates stay within one record.
  • Nothing documents this. The test comment at test_uk_carers_allowance_take_up.py:433-434 ("the stream does not depend on which other people the frame carries") still holds for the uniforms, but no longer for the flags.
  • One alternative keeps G's per-carer property: key the start on the stratum's lowest-draw member.
    • A carer alone claims at about 0.594 at a rate of 0.6.
    • Churn is roughly code keying's plus 0.48/n: 0.20, 0.06, 0.011 and 0.006 at 5, 20, 80 and 200 members.
    • The cost: the full ~48% re-draw comes back when the lowest-draw member leaves, or a lower one arrives, which is about 1 change in n.
  • Otherwise, document the cost.

3. Re-certifying an earlier national will refuse if its floor dropped a row.

  • tools/certify_uk_release_cut.py:306-313 loads the floor from the certifier's own package. uk_runtime/support_floor.py:571-576 then requires each recorded drop's reason, tracking, approved_by, adjudication, approved_on and expires_on to equal that floor's.
  • Commits 36 and 38 changed the reason and the adjudication. A probe confirms that a drop recorded under the floor at fddbc7073, edad8aee9 or ead8943d6 is refused at this head, with "not under the declared signature".
  • Runs with no drops are unaffected. So are the in-graph read-backs, which compare against the run's own record. The runbook certifies at the build's own head, so the standard chain is fine.
  • The next trigger will be María's confirmation edit of the floor's adjudication. After it, a national built at this head that dropped a row couldn't be re-certified at the later head without a rebuild.
  • Per-row exclusions are compared by name only (tools/certify_uk_release_cut.py:298-302), so the floor is the odd one out.
  • Suggestion: compare only the signature fields (tracking, approved_by, approved_on, expires_on), or accept the run's own declaration digest.

4. H: two text gaps.

  • The floor's note asks María to confirm or re-size the amounts scope's 2-row cap (seven families, now nine). But the counts scope widened too, from seven families to eight (Table 3.10's counts), under the same 2-row cap.
  • The cars fuel duty adjudication's new note says "the ruling and its figures are unchanged". Yet this PR restated the reason's figures on the September edition (a58bc1a) and added the June comparison. The note also doesn't say who re-worded it.
  • Older, and not from this PR's edits: the fence's signed reason, and its 23 October re-wording, still name income_tax_annual > 0.

5. The diagnostic tool has two residuals.

  • Weights. Rows measured at 2024 are weighted by the engine's weights at 2024.
    • The national H5 stores the solved weights divided by the engine's 2024→2025 factor (1.0030), and tests/engine/uk/test_uk_weight_year.py pins that the engine returns the solved weights at 2025 and solved ÷ factor at 2024.
    • So on a release H5, every 2024-period row (electricity, road fuel and the five excise rows) reads the calibration's figure ÷ 1.003. An exact fit therefore shows −0.30%, though the tool's comment says it "measures what the national binds".
    • The excise rows have read this way since UK: impute alcohol, tobacco and gambling consumption so policyengine-uk's excise duties stop scoring zero #1359.
    • Suggestion: weight the 2024 values with the calibration-year weights, as the certifier (uk_solve_year_frame) and the scorer do.
  • Periods. The tool takes each row's period from the package's contract, but its value from the run's --registry, whose specs carry their own measurement_period.
    • On a national built before this PR, electricity's registry value is the 2025 fact, which the tool now reads off the 2024 frame: about −2.5 points, spurious.
    • Suggestion: read the period from each registry spec, and fall back to the contract only without --registry.
  • Aside: the docstring at uk_runtime/weight_year.py:11 still says "0.72% from 2024 to 2025". The pinned growth is 0.3%.

6. Smaller items.

  • No test pins the take-up strata's route dimension (this predates commit 38).
    • With take_up_strata = pool_strata, 61 of 800 flags change on _outside_pool_frame, and all 37 tests in the file pass.
    • The new stage test reads its strata from the stage's own call.
  • "Postcode districts and sectors" survives in two places:
    • tools/build_uk_rfdr_area_layer.py:8-9 and :589, which writes it into the committed layer's description, so fixing it moves the layer's digest;
    • uk_runtime/rfdr_area_layer.py:9-10.
  • uk_runtime/graph_evidence.py:342: the spine gate kernel's "Node parameters" list doesn't name energy_pricing_code.
  • The PR body's commit-38 checks line cites "the mutation runs above", which only the reply describes.

Checked and found true

  • Every round-2 mutation, re-run against this head, now fails a test.
  • energy_pricing_code is computed from the module's bytes and enters only the transferred gate's key. Nothing pinned needed regenerating: the H2 fixture has no gate nodes, and the Carer's Allowance stage constants are unchanged.
  • Each start digests exactly the set its placement balances over, from unique person ids: eligible carers for take-up, drawable carers for the pool.
  • The diagnostic's ROW_YEARS follows the calibration's own period rule, and --help runs without the engine.
  • Commit 38 moves no reference or membership. The national compile-parity receipt changes only in rationale text, which now gives the interest counts their own growth.
  • CI at 216f22a5f: every finished job passes. engine-free (3.13) and engine-us (3.13) were still running when this was posted.

Not covered

  • Any build or engine test: everything ran engine-free on Python 3.13.
  • The RFDR --check wasn't re-run against the registered zip.

🤖 Generated with Claude Code

The re-review (issuecomment-6514923017) found nothing blocking; these
are its six points.

1. The fan-out bridges' gaps are checked by running the runtime surface
   itself on the committed registers (cells, controls, small-cell
   deferrals and off-surface cells), so each grain reads what a build
   reads; local-authority pension cells x1.005 now refuse in a test.
2. A Carer's Allowance stratum's start is keyed on its code and its
   lowest-draw member, so a membership change re-draws a stratum only
   when that member changes; a carer alone still claims at the rate.
3. The certifier compares a run's support-floor drops with the floor's
   signature fields only (tracking, approver, approval and expiry), so a
   re-worded floor can still certify a run built under it.
4. The floor's note names both widened caps; the cars fuel duty note
   says what this PR changed and that it awaits María.
5. tools/diagnose_uk_consumption_taxes.py weights every row at --year
   and takes each row's period from the run's registry when given one;
   weight_year's docstring says 0.3%.
6. A test pins the take-up strata's route; the RFDR wording reads
   "districts, unit postcodes and excluded units" in the tool, the
   runtime docstring and the rebuilt layer (same areas; description and
   built_on only); the spine gate docstring names its new param.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Reply to the round-3 re-review. All six points are addressed at 10f434dfd (commit 39), except H's approvals, which stay María's.

  1. Fixed. The source-gap test now runs the runtime surface itself on the committed registers, for every fan-out bridge.
    • It calls uk_local_target_surface with the committed cells at their resolved and published values, each bridge's national control, and the committed small-cell deferrals and off-surface cells.
    • Both grains now read what a build reads. For the pension bridge, the source factors are 1.0295 and 1.0296, and the applied factors ×1.0450 and ×1.0451.
    • test_the_runtime_surface_refuses_local_authority_cells_moved_past_the_floor is your mutation: with the local-authority pension cells ×1.005, the surface now refuses.
    • It runs in under a second.
  2. Fixed with your alternative. Each stratum's start is keyed on its code and its lowest-draw member's identity.
    • A carer alone in a stratum still claims at the rate over identities.
    • A change in membership re-draws the stratum only when that member leaves or a lower one joins.
    • A test checks both: removing another member keeps the start, and removing the lowest-draw member moves it.
  3. Fixed. A recorded drop must now share the floor's signature fields: tracking, approved_by, approved_on and expires_on. The reason and adjudication texts no longer enter the comparison.
    • A test accepts a drop recorded under older texts and refuses each moved signature field.
    • So María's confirmation edit won't stop a national built at this head from being re-certified later.
  4. H's texts fixed.
    • The floor's note now names both caps: the amounts scope's (seven families, now nine) and the counts scope's (seven, now eight).
    • The cars fuel duty note now says what this PR changed. a58bc1a restated the figures on the September edition beside the June ones, and commit 36 added which edition the engine reads were measured on. The note says the change is pending María's confirmation and the ruling itself is unchanged.
    • The fence's signed reason and its 23 October re-wording still say income_tax_annual > 0. That is María's signed text from before this PR, so it is left for her.
  5. Fixed.
    • Weights: every row is weighted at --year, the calibration year, with its values at its own period, so a 2024 row reads on the solved weights. A test with a stand-in engine pins this.
    • Periods: with --registry, each row's period comes from the run's own registry specs; without it, from the contract. A test pins this too.
    • weight_year.py's docstring now says 0.3%, from the vendored pins.
  6. Fixed.
    • The stage test now checks the take-up strata's route: no stratum mixes carers inside and outside the household. Setting take_up_strata = pool_strata fails it.
    • "Districts and sectors" now reads "districts, unit postcodes and excluded units" in the tool, the layer description and the runtime docstring. I rebuilt the layer from the registered zip. It flags the same 41 and 1,053 areas; only its description and built_on changed, and --check reproduces it.
    • The spine gate kernel's docstring names energy_pricing_code.
    • The body's commit-38 checks line no longer cites mutation runs it doesn't describe.

Checks

Run locally for commit 39: the 34 engine-free test files this round touches or that read what it changed; the Carer's Allowance and weight-year engine tests; the coverage manifest check; the re-review's mutations (local-authority pension cells x1.005, and the take-up strata without the route), each of which now fails a test; and tools/build_uk_rfdr_area_layer.py --check against the registered NSPL zip, which reproduces the rebuilt layer byte for byte. All pass.

🤖 Generated with Claude Code

The first PR C national (built from 10f434d) refused at the national
problem: obr.betting_and_gaming_duties declares the frame's period
2024, but its value, the engine's gambling_duty, is not a stored
column. The national solve measures a declared period only on a stored
column or a CGT dated measure, so it cannot read the engine's 2024
duty. Commit 16 had deleted the row's exclusion under María's ruling
of 2026-10-28 to bind it; no PR C national had been built since, and
the tests do not run the national problem's measure routes.

María's signed exclusion of 2026-10-23 is restored as it was, with a
note in its adjudication that the bind ruling cannot take effect until
the solve can measure the duty (a dated duty measure, or bases bound at
the frame's period), pending her decision. The row stays measured on
every evaluation, beside the other excise rows, which main holds for
the same reason. The row's notes, the consumption-tax tool's docstring
and the tests' census follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Commit 40 (5aba83cd2) puts the betting and gaming row back in the held-out set until the solve can measure it. This undoes commit 16's binding, pending María's decision, and it corrects my reply to the independent review's finding 12, which said the OBR total stays bound.

  • What failed. The first national built from this PR (at 10f434dfd, staging local only) refused at uk.full.national_problem. obr.betting_and_gaming_duties declares the frame's period, 2024, but its value is the engine's gambling_duty, which isn't a stored column. The solve measures a declared period only on a stored column or a CGT dated measure, so it can't read the engine's 2024 duty.
  • Why CI didn't catch it. No national had been built from PR C since commit 16 bound the row, and the tests don't run the national problem's measure routes.
  • Other rows. I checked every other bound row that declares a measurement period. All read stored columns or CGT's dated measures. Main holds the other excise rows out for the same reason as this one.
  • The change.
    • María's signed exclusion of 23 October is restored as it was.
    • Its adjudication notes that her 28 October ruling to bind the row can't take effect until the solve can measure the duty, either through a dated duty measure or through bases bound at the frame's period.
    • The row is still measured on every evaluation.
    • The row's notes, the consumption-tax tool's docstring and the tests' exclusion census are updated to match.
  • Next. The national is rebuilding from 5aba83cd2. On it I'll measure each gambling carrier against HMRC's receipts by duty (finding 12).

🤖 Generated with Claude Code

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Re-review of #1420 at 5aba83cd2 (round 4)

This checks round 3's six items (issuecomment-6514923017) against commit 39 (10f434dfd), then reviews commits 39 and 40. A reviewer re-checked the items and re-ran the mutations. A verifier checked commit 40 and every bound row that declares a measurement period. Everything ran engine-free. Paths are relative to packages/microcosm-build/src/microcosm/build/ unless they start with packages/ or tools/.

Verdict: nothing blocks merge.

  • All six round-3 items are fixed. H's approvals stay with María.
  • Commit 40 is right:
    • the restored exclusion matches María's signed 23 October entry field for field, apart from the appended adjudication note;
    • every other bound row that declares a measurement period reads a stored column or a CGT dated measure. That is 24 targets (113 references), re-checked independently with the production predicate.
  • The refusal that commit 40 fixes was a miss in the independent review. Round 1 treated binding the betting and gaming row as a design question (finding 12), and didn't check whether the solve could measure it. A cheap engine-free guard would have caught it, and would catch the next one (1).

Round-3 items

Item Status Evidence
1 Resolved The test runs uk_local_target_surface with each bridge's real control, the committed deferrals and the off-surface cells. It reads 1.0295 and 1.0296, applies ×1.0450 and ×1.0451, and refuses under the ×1.005 mutation. Two residuals (3).
2 Resolved The start is keyed on the lowest-draw member, by the placement's own uniform, with ties broken by person id. A carer alone in a stratum claims at 0.59 at a rate of 0.6. Moving one carer between strata now flips 1.7 of the other 319 flags on average, against 1.3 with code keying; the full re-draw happens only when the lowest member changes. Four start mutants fail the new test.
3 Resolved Only the signature fields are compared. A re-worded record certifies, and each moved signature field refuses. Its consequence is finding 2.
4 Resolved Both caps are named, and the cars note says what changed. The fence text stays María's.
5 Resolved Weights are read at --year, values at each row's own period summed to households, and periods come from the run's registry. One residual (4).
6 Resolved The route test fails the mutant, and the wording is fixed. The rebuilt layer flags the same areas, and its provenance digest matches.

New findings

1. No engine-free test checks that a bound row with a declared period can be measured.

  • The solve refuses a bound plain-column row whose declared period (2024) isn't held on the frame. _refuse_ignored_measurement_period (packages/microcosm-build/src/microcosm/build/target_materialization.py:991-1032) raises a MeasurementPeriodError, which stops the whole run.
  • "Held" means one of the frame's own columns (UKFrameTargetAdapter.value_period, uk_runtime/ledger_targets.py:1806-1818) or a CGT dated measure.
  • Nothing runs this rule on the packaged contract:
    • the existing tests use synthetic bindings;
    • the exclusion census simply moved with commit 16;
    • tools/check_uk_measurement_periods.py runs only after a build.
  • That is why commit 16 passed CI and the national refused at uk.full.national_problem, after the spine and engine stages had run.
  • The same refusal will come back if any excise row is bound again, or if one of the five excise exclusions is left to lapse when they expire on 2027-01-08.
  • A guard is cheap. It would run the production predicate over the packaged contract, membership and exclusions, taking the stored columns from uk_spine_graph() / uk_spine_endpoint(...).inputs and the frame period from load_uk_frs_release().
    • A prototype flags exactly obr.betting_and_gaming_duties on commit 16's register, with the build's message, and nothing at this head.
    • It takes milliseconds after the graph import (about 20 s, shared with the graph tests).
    • Its natural home is tests/engine_free/uk/test_uk_measurement_period.py, next to :536.

2. The signature-only floor check no longer ties a drop to the scope it was made under.

  • After commit 39, a recorded drop certifies if its tracking, approved_by, approved_on and expires_on match the floor (uk_runtime/support_floor.py:98, :577-582). Scope membership is re-checked only against the certifier's own floor.
  • Commit 36 widened the floor to Table 3.7's other income and Table 3.10 without moving approved_on. So a national built at this head that drops one of those rows certifies under the 22 October signature, which never covered them. A probe confirms this for drops recorded at edad8aee9 through this head.
  • The reply's point (that María's confirmation edit won't block re-certification) holds only if she confirms by editing the text and leaves approved_on at 22 October. If she moves approved_on, every earlier run with drops refuses again, which is round 3's finding 3 back.
  • Options:
    • record a digest of the scope (families and caps) in each drop, and compare it;
    • or move approved_on with any widening, and let the reader accept named superseded signatures.
  • The share ceiling isn't re-checked at certification either, but that predates this PR.

3. Two test residuals on the bridge.

  • The ×1.005 test is brittle. tests/engine_free/uk/test_uk_fanout_sum_bridges.py:582-595 hard-codes the scale. At ×1.005 the local-authority factor lands only 0.052% below the 1.025 floor. So if the factor legitimately rises past 1.0301, still well inside the bounds, the test fails with "DID NOT RAISE". Deriving the scale from the baseline (baseline ÷ floor × 1.001) keeps it a mutation test.
  • Lost deferrals in the data aren't caught. If a regenerated membership drops the pension deferrals, nothing notices: the factors read 1.0366 and about 1.030, both inside [1.025, 1.037]. Code-level losses are caught elsewhere. A one-line pin on the committed small_cell_deferrals counts (17 and 3) would cover the data.

4. The diagnostic tool assumes the weights were divided for release.

  • It always projects the stored weights to --year (tools/diagnose_uk_consumption_taxes.py:174-213). That equals the solved weights only on H5s divided for release, which started with 0f0bb86 on 14 October.
  • On H5s written as solved, every row now reads +0.3%. These are nationals released before 14 October, and the engine scratch file. The registry-period path is aimed at exactly those older nationals.
  • main()'s --registry wiring and the map_to argument are untested, and --year is hard-coded at 2025 rather than read from the release's calibration year.
  • Suggestion: load the weights the way the certifier does, with load_uk_national_frame and then uk_solve_year_frame.

5. Smaller.

  • The restored betting exclusion still gives María's 23 October reason ("Binding it would move weight onto the households that gamble to hold a level the build already sets"). Her 28 October ruling rejected that argument, and only the appended note says so. Re-signing, or choosing one of the two routes the note names, is her call.
  • Commit 40's message, PR body item 40, the reply and the tool's docstring say main holds the other excise rows "for the same reason". Their registered reason is María's 23 October ruling: the level is set by the build. They would hit the same refusal if bound, but that is a different statement.
  • The cars fuel duty note doesn't say who re-worded it.

Checked and found true

  • Commit 40's entry: its name, reason, tracking, approver and dates are byte-identical to the entry commit 16 deleted, and to main's. Only the adjudication gains a paragraph. The entry isn't expired; the census test, the tool and the row's notes match it; and PR body item 16 says "Undone by commit 40".
  • Period-declared rows: 30 national targets declare 2024. 24 of them are bound:
    • 8 read stored household columns (bus support and fares, electricity, road fuel);
    • 1 reads a CGT dated measure;
    • 15 are CGT threshold rows that carry the year in the measure name.
      The 5 excise rows and the ORR rail row are held. Local bindings can't declare a period (they're refused at load), and the DESNZ kWh rows are resolver measures read at the frame's period.
  • Commit 39: the start keying, the signature fields, the weighting path and the rebuilt RFDR layer are as in the table, and nothing pins the old layer digest.
  • CI at 5aba83cd2: every finished job passes (engine-uk 3.14, integration-uk, lint, select-countries, wheels). The engine-free, engine-uk 3.13 and engine-us jobs were still running when this was posted.

Not covered

  • Any build or engine test: everything ran engine-free on Python 3.13.
  • The national now rebuilding from 5aba83cd2.

🤖 Generated with Claude Code

The re-review (issuecomment-6516126464) found nothing blocking.

1. test_every_bound_row_with_a_declared_period_can_be_measured_there
   runs the solve's own period check over the packaged contract, with
   the spine's stored columns and the release's frame period: it passes
   at this head and flags exactly the betting and gaming row without
   its exclusion, the refusal the first PR C national hit.
3. The bridge mutation's scale is derived from the committed baseline,
   and the pension bridge's committed small-cell deferrals (17 and 3)
   are pinned.
4. tools/diagnose_uk_consumption_taxes.py reads the solved weights as
   the certifier does (load_uk_national_frame, uk_solve_year_frame) by
   household id, and --year defaults to the FRS release's calibration
   year; tests cover the weighting, the households' alignment and
   main()'s --registry wiring.
5. The tool's docstring gives the excise rows' registered reason (María's
   ruling of 2026-10-23), and the cars fuel duty note says who re-worded
   it.

Finding 2 (the floor's drops and its widened scope) is María's, with H.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Reply to the round-4 re-review. Points 1, 3, 4 and 5 are addressed at 125d6e73b (commit 41). Point 2 belongs with H and is María's.

  1. Fixed. test_every_bound_row_with_a_declared_period_can_be_measured_there (in test_uk_measurement_period.py) runs the solve's own _refuse_ignored_measurement_period over every bound plain-column row of the packaged contract that declares a period. The spine's stored columns (uk_spine_endpoint(uk_spine_graph()).inputs) and the release's frame period stand in for the frame, through UKFrameTargetAdapter.value_period. The test does two things:

    • it passes at this head;
    • with the betting and gaming exclusion removed, it flags exactly that row, which is the refusal the first PR C national hit.

    So the next binding of an excise row, or a lapsed excise exclusion, fails in CI and not at the national problem.

  2. María's, with H. Your reading is right: a drop of a newly covered row would certify under the 22 October signature, which never covered it. The fix depends on how María confirms the widening:

    • if she re-signs by moving approved_on, earlier runs with drops need an accepted list of superseded signatures;
    • if she confirms by text, a scope digest in each drop is the stronger check.

    I've left the choice with the widening itself. For the national now building, the floor's receipt will show whether it dropped any row; if it dropped none, nothing rides on this yet.

  3. Fixed. The mutation's scale is now derived from the committed baseline: the local-authority source factor over the floor, ×1.001. The pension bridge's committed small-cell deferrals are pinned at 17 and 3.

  4. Fixed.

    • The tool reads the solved weights as the certifier does (load_uk_national_frame, then uk_solve_year_frame) and aligns them to the engine's households by id. A mismatch refuses.
    • --year defaults to the FRS release's calibration year.
    • Tests cover the weighting, the alignment and main()'s --registry wiring.
  5. Fixed where the text is ours.

    • The tool's docstring now gives the excise rows' registered reason, María's 23 October ruling. "For the same reason" in commit 40's message, the PR body and my commit-40 note was wrong. The body is corrected, and this reply corrects the note.
    • The cars fuel duty note says it was re-worded by the PR C session, committing in María's name, pending her confirmation.
    • The restored betting reason is María's signed text. Re-signing it, or choosing a route to binding, is hers.

Checks

Run locally for commit 41: the touched engine-free files (measurement period, fan-out bridges, consumption-tax diagnostics, measure simulation, support floor and off-surface cells). All pass.

The national is building from 5aba83cd2. Commit 41 changes tests, the tool and texts only, so the build is unaffected.

🤖 Generated with Claude Code

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Re-review of #1420 at 125d6e73b (round 5)

This checks round 4's items (issuecomment-6516126464) against commit 41. One reviewer re-ran the new tests and mutated them, and everything ran engine-free. Paths are relative to packages/microcosm-build/src/microcosm/build/ unless they start with packages/ or tools/.

Verdict: nothing blocks merge, and the four fixes hold. What's left is low severity.

Item Status Evidence
1 Resolved test_every_bound_row_with_a_declared_period_can_be_measured_there runs the solve's own _refuse_ignored_measurement_period, through UKFrameTargetAdapter.value_period, against exactly the spine's carried columns and the release's frame period. It passes engine-free, even with policyengine-uk blocked. With the betting exclusion removed, it flags exactly obr.betting_and_gaming_duties. It isn't vacuous: with no stored columns, it flags the eight stored-column rows.
2 María's The reply's description is right. More precisely, what matters is a drop outside the 22 October scope (Table 3.7's other income or Table 3.10), not any drop at all.
3 Resolved The mutation's scale is now derived from the baseline, so it crosses the floor for any baseline in [1.025, 1.037]. The deferral pin (17 and 3) fails if either grain loses its deferrals.
4 Resolved The solved weights come from load_uk_national_frame and then uk_solve_year_frame, aligned to the engine's households by household_id. Checked on synthetic H5s: a release-divided file gives back the solved weights exactly, a solved file gives them back as stored, and a reversed engine order still aligns.
5 Resolved The tool's docstring and PR body item 40 give the excise rows' registered reason. The cars note names who re-worded it. Commit 40's message keeps the old wording, but that's history.

Remaining, all low severity

1. A lapsed exclusion still passes CI.

  • The exclusion loader (uk_runtime/measure_simulation.py:1770-1845) neither drops nor refuses an expired entry. The window is enforced only when the national targets are applied.
  • So suppose the excise exclusions are still in the register when they expire on 2027-01-08. CI stays green, and every national then refuses at uk.full.national_targets, after the spine, with the window error.
  • The new test catches a deleted exclusion, but not an expired one. The earliest committed exclusion expires on 2026-12-11.
  • Suggestion: a check on the committed register, like the uprating holds' test_no_committed_hold_is_about_to_lapse, would show each lapse in CI ahead of time.

2. The tool's real weight path has no test.

  • The weighting test passes weights=, and the main() test replaces engine_estimates. So a regression anywhere in load, then solve year, then alignment by id would pass CI.
  • Running the default path on a small synthetic H5 inside the test would close this.

3. The tool's scope and robustness.

  • The docstring (:20) and the --dataset help (:244) still say "any H5 the engine loads". The tool now needs a file that load_uk_national_frame accepts.
  • The id check refuses engine households that are missing from the frame, but silently drops frame households the engine lacks. On the default path both sides read one file, so this only matters for callers that pass weights=.
  • With --registry and no --year, the default year comes from the package's release, not the run's.
  • The tool reads the whole H5 a second time, all three tables with Frame validation, after the engine has already loaded it. The household id and weight would be enough, and that matters on the dense file.

4. Test details.

  • The measurability test uses a literal 2025 as the calibration period (test_uk_measurement_period.py:630), rather than load_uk_frs_release().calibration_year.
  • It doesn't run the solve's fact-period check for require_matching_fact_period rows (packages/microcosm-build/src/microcosm/build/target_materialization.py:748-754). No row is affected today.
  • Its second assertion pins today's held betting row, so it will change when María picks a route.

CI and coverage

  • CI at 125d6e73b: lint, select-countries, wheels, integration-uk and engine-uk (3.13 and 3.14) pass. engine-free and engine-us were still running when this was posted.
  • Not covered: any build or engine test.
  • The national building from 5aba83cd2: commit 41 changes only tests, the tool and one register text, so it doesn't affect that build.

🤖 Generated with Claude Code

The re-review (issuecomment-6516738969) found nothing blocking and only
low-severity items.

1. test_no_committed_exclusion_is_about_to_lapse fails 14 days before
   the earliest exclusion expires, as the uprating holds' clock does, so
   a lapse shows in CI and not at a national's target step.
2. A test runs the tool's real weight path on synthetic H5s, one
   divided for release and one written as solved, through
   load_uk_national_frame, uk_solve_year_frame and the alignment by id.
3. The tool says it needs a national H5 load_uk_national_frame accepts,
   refuses a household on either side alone, and with --registry takes
   the run's own calibration period when --year is not given.
4. The measurability test reads the release's calibration year and also
   runs the solve's fact-period check for rows that require their fact
   at the declared period.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Reply to the round-5 re-review. The low-severity items are addressed at 162690525 (commit 42).

  1. Fixed. test_no_committed_exclusion_is_about_to_lapse fails 14 days (UK_HOLD_EXPIRY_NOTICE_DAYS) before the earliest exclusion expires, as the uprating holds' clock does. A lapse now shows in CI, not at a national's target step.
  2. Fixed. test_the_tool_reads_the_solved_weights_of_a_release_and_a_solved_file runs the default weight path on two synthetic H5s, one divided for release and one written as solved. The path goes through load_uk_national_frame and uk_solve_year_frame. The test checks that both read back the solved weights, and that the engine path aligns them to a reordered engine by id.
  3. Fixed, except one item.
    • The docstring and --dataset help now say the tool needs a national H5 that load_uk_national_frame accepts.
    • The id check now refuses a household on either side alone.
    • With --registry and no --year, the tool takes the run's own calibration period from the registry. A test covers this.
    • Not changed: the second read of the H5. The solve-year logic needs the frame's weight record, and the tool is a development diagnostic, so I kept the one code path the certifier uses.
  4. Fixed, except one item.
    • The measurability test reads the release's calibration year.
    • It now also runs the solve's fact-period check: every bound row that requires its fact at its declared period resolves to that period in the committed membership.
    • Not changed: its betting assertion. It will change with María's route.

National build: blocked at certification

The national built from 5aba83cd2 got through the solve, but the calibration-seam fit gate blocked the artifact on two rows in the £1m-and-over band:

  • the savings interest amount;
  • the other-income amount, which this PR binds.

Every lower band of both families fits. I've asked María whether to hold the new £1m-and-over other-income row as a measured distortion, as the ten other £1m-and-over cells are held. That needs her signature, so the build waits for her. The figures stay out of this thread because the band rests on the SPI tape.

Checks

Run locally for commit 42: the measurement-period, consumption-tax diagnostics and measure-simulation test files. All pass.

🤖 Generated with Claude Code

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Re-review of #1420 at 162690525 (round 6)

This checks round 5's items (issuecomment-6516738969) against commit 42. One reviewer ran the new tests under fake clocks and against in-memory mutants. Everything ran engine-free. Paths are relative to packages/microcosm-build/src/microcosm/build/ unless they start with packages/ or tools/.

Verdict: nothing in the code blocks merge. The fixes hold, with two small coverage gaps. The open item is the certification block, and that is María's call.

Item Status Evidence
1 Partly test_no_committed_exclusion_is_about_to_lapse uses the holds' clock: 14 days ahead of date.today(). Under fake clocks it passes until 2026-11-26 and fails from 2026-11-27, naming 2026-12-11. It doesn't cover the support_floor block in the same register (finding 1).
2 Resolved The new test runs the real loader and the real solve-year step on two synthetic H5s. Skipping the solve-year step, aligning by position, or reverting the id check each makes it fail.
3 Resolved The id check refuses a household that appears on only one side. The registry's single compiled period sets the default year, with a fallback to the release's year. The second read of the H5 stays, by choice.
4 Partly The test reads the release's calibration year and runs the fact-period check, which catches a mutated plain-column row. It skips the 15 bound CGT rows on the provider route (finding 2).

Findings (all low)

1. The lapse clock skips the support floor.

  • The floor block (uk/calibration_measure_exclusions.json:3-82, in force until 2027-01-08) is enforced at the same targets step. uk_runtime/full_targets.py:122-139 calls the floor's assert_in_force.
  • But the new test reads only the per-row list, and test_uk_support_floor.py pins its own date.
  • Today the floor is covered only by coincidence: seven per-row entries also expire on 2027-01-08. If those are renewed and the floor isn't, CI stays green and every national refuses from 2027-01-09 with "UK support floor is outside its reviewed window".
  • Adding the floor's expires_on to the clock closes it.
  • Also, min() there has no default. If the last per-row exclusion is ever bound, the test errors instead of passing. The sibling clocks share this pattern.

2. The fact-period check misses the CGT provider rows.

  • The test skips any binding with a kind, but the solve checks the fact period before it branches on kind.
  • 18 bound rows require their fact period. The test checks 3 of them (CGT liability, electricity and road fuel), and not the 15 parameter_gated_threshold CGT rows.
  • So a re-pin that resolves one of those rows to a 2025 fact passes CI, and the national stops after the spine. A bound row missing from the membership also passes.
  • The reply's "every bound row" overstates the coverage.

3. Smaller.

  • The registry-year test hard-codes 2026, so it goes vacuous once the release's calibration year is 2026. Using calibration_year + 1 would hold.
  • A registry with mixed periods falls back to the release's year silently.
  • Older than this commit: composition() reads the H5's stored (dataset-year) weights, while the estimates now use the solved weights. Its totals therefore sit about 0.3% off; the shares are unaffected.

The certification block

  • The national built from 5aba83cd2 stopped at the calibration-seam fit gate on two £1m-and-over amount rows: savings interest, and the other income this PR binds. As in the reply, no figures for that band here.
  • Round 1's removed-behaviour pass raised the other-income row: that £1m-and-over cell is bound without the distortion measurement that held ten sibling cells out. The verifier set it aside under the 21 October rule, which holds a row out only on a measured distortion. The build has now measured it.
  • The savings-interest £1m-and-over row was already bound on PR A's certified national. What this PR changed for it is commit 32's uprating: the 13 interest amounts now grow a further ×1.134 on the OBR savings path, which Vahid's finding 1 questioned.
  • So before deciding, it may be worth checking whether holding only the new other-income row clears the gate, or whether the interest row's new level is part of the miss. That is María's call.

CI and coverage

  • CI at 162690525: lint, select-countries, wheels, integration-uk and engine-uk 3.13 pass. The other jobs were still running when this was posted.
  • Not covered: any build or engine test.

🤖 Generated with Claude Code

- The measure-exclusion lapse clock also reads the register's support
  floor, which the national targets refuse at the same step, and passes
  on an empty register instead of erroring.
- The measurability test runs the solve's fact-period check on every
  binding kind, before it branches on kind, as the solve does, so the 15
  CGT provider rows are covered. A required row missing from the committed
  membership also fails. A new test mutates one CGT row to a later fact
  and drops another from the membership; both fail.
- diagnose_uk_consumption_taxes.py weights its composition with the
  solved weights by household id, as the estimates are, reading them once
  in main(). With --registry and no --year, a registry whose specs span
  several periods refuses rather than falling back. The registry-year test
  uses the release's calibration year plus one, not a literal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Reply to the round-6 re-review. All three findings are addressed at 059225c89 (commit 43).

  1. Fixed.
    • test_no_committed_exclusion_is_about_to_lapse also reads the register's support floor (load_uk_support_floor_declaration), which the national targets refuse at the same step. The failure message names every entry that lapses on the earliest date, the floor included.
    • On an empty register the test passes instead of erroring.
    • The three sibling clocks (uprating holds, small-cell rule, local adjudications) predate this PR and are left as they are.
  2. Fixed. You're right that the round-5 reply's "every bound row" overstated the coverage.
    • The helper now runs the fact-period check on every binding kind, before it branches on kind, as target_materialization does. So the 15 parameter_gated_threshold CGT rows are covered.
    • A required row with no active candidate in the committed membership also fails.
    • test_the_fact_period_check_covers_every_binding_kind is the mutation: with one CGT provider row resolved to a 2025 fact, it flags exactly that row; with another row dropped from the membership, it flags exactly that one.
  3. Fixed.
    • The registry-year test uses the release's calibration year plus one.
    • With --registry and no --year, a registry whose specs span several periods now refuses ("pass --year"). A test covers it.
    • composition() is weighted with the solved weights by household id, as the estimates are. main() reads them once and passes them to both.

The certification block

I checked your question against the build's own fit report: is commit 32's interest uprating part of the £1m-and-over miss, or is it only the new other-income row? The answer, and the band's ratios, went to María privately with the decision. The band's figures, and the direction of either miss, stay out of this thread. Before she signs, I've also offered her a local, unsigned re-solve of this national problem with the other-income row held, to confirm the gate clears.

Checks

Run locally for commit 43: the consumption-tax diagnostics, measurement-period and measure-simulation test files, and ruff on the touched files. All pass.

🤖 Generated with Claude Code

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Re-review of #1420 at 059225c89 (round 7)

This checks round 6's findings (issuecomment-6517309229) against commit 43. One reviewer re-ran round 6's mutations and ran the new clocks under fake dates. Everything ran engine-free. Paths are relative to packages/microcosm-build/src/microcosm/build/ unless they start with packages/ or tools/.

Verdict: all three are fixed, and nothing in the code blocks merge. Two low items remain in the measurability helper. The certification block stays with María.

Finding Status Evidence
1 Resolved Under fake clocks, renewing all 39 per-row entries but not the floor now fails from 2026-12-25, naming support_floor. An empty per-row list no longer errors.
2 Resolved The fact-period check now runs before the kind branch. Moving any of the 18 bound required rows to 2025 flags exactly that row, and so does dropping one from the membership; that includes the 15 CGT provider rows. The plain-column part still flags betting when its exclusion is removed.
3 Resolved composition() uses the same refusal and id alignment as the estimates. main() reads the solved weights once and never weights by the stored column. A registry with mixed periods refuses.

Low items

1. The new clock's first deadline. It is by design, but worth noting now: from 2026-11-27, test_no_committed_exclusion_is_about_to_lapse fails on every PR until orr.rail_government_support (expires 2026-12-11) is renewed or bound.

2. The measurability helper's account of holds differs from the solve's (packages/microcosm-build/tests/engine_free/uk/test_uk_measurement_period.py:613-645).

  • Possible false positives. Some of the 15 CGT provider targets fan out into one row per region or band, which the check reaches only since commit 43. Two cases misfire:
    • The helper still checks a fan-out row that is held on its own by a per-row exclusion, although the national drops that spec before materialising it. Example: a re-pin moves one CGT region's fact, and the reviewer holds just that row. The national runs, but this test fails.
    • It also flags a fan-out target whose rows are all signed out one by one.
  • Wider loop. It loops over all 456 contract targets, against the 394-target national membership. So a local-only target that later declares require_matching_fact_period would be flagged too.
  • Latent miss. Held rows inside the floor's scope are compiled as probes and checked by the solve, but the helper skips every held target. No such row declares a period today.
  • Suggestion: skip a row whose own name is held, unless its target is in the floor's scope, and restrict the loop to national targets.

3. The docstrings overstate the "no active candidate" rule (:572-576, :678-681).

  • They present it as the solve's own refusal ("stops the national after the spine"). In fact the national never reads uk/target_reference_membership.json: it compiles uk/target_references.json, which holds active rows only.
  • So a re-pin that defers a required row drops it from the national silently.
  • The test does catch that case, which makes it the only guard. The docstrings should say so rather than describe a build refusal.

CI and coverage

  • CI at 059225c89: lint, select-countries, wheels, integration-uk and engine-uk (3.13 and 3.14) pass. engine-free and engine-us were still running when this was posted.
  • Note on engine-free: that job was cancelled unfinished on the previous three heads by later pushes, so the PR's engine-free result is still to come.
  • Not covered: any build or engine test.

🤖 Generated with Claude Code

The measurability helper now reads the rows the national compiles, as
graph_national does:
- the targets on the national line only (national geography and the
  national calibration line), not every contract target;
- less the rows signed out of a fan-out;
- less the rows a per-row exclusion holds, except held rows in the
  support floor's scope, which the national compiles as probes and so
  checks.

A required row that is deferred (not active) is now flagged, like a
row missing from the membership. The national compiles only the active
references, so either one leaves the national without a refusal, and
this test is the only guard. The docstrings now say so, instead of
describing a build refusal.

A new test covers the three hold cases: a row held on its own with a
moved fact passes; the same row in the floor's scope is flagged; a
fan-out signed out row by row passes. The deferral case is added to the
fact-period test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Reply to the round-7 re-review. Items 2 and 3 are addressed at 54d9d5b6d (commit 44). Item 1 is a deadline for María.

  1. Noted, by design. From 2026-11-27, test_no_committed_exclusion_is_about_to_lapse fails on every PR until orr.rail_government_support (expires 2026-12-11) is renewed or bound. Both are María's signature. I've added the date to the PR body, under what merging waits on, so it doesn't surprise anyone.
  2. Fixed, as you suggested. The helper now reads the rows the national compiles (graph_national):
    • Targets: only those on the national line (national geography and the national calibration line). That is 391 targets, not the contract's 456, so a local-only target can't trip it.
    • Signed out: rows signed out of a fan-out are skipped. A fan-out signed out row by row compiles nothing and isn't flagged.
    • Held: a row held by its own per-row exclusion is skipped, unless its target is in the support floor's scope. Those held rows are compiled as probes, so they're checked, which closes the latent miss.
    • Test: test_the_period_checks_read_the_rows_the_national_compiles covers each case. A CGT region row with a moved fact passes when held on its own, is flagged when its target is in the floor's scope, and passes when the fan-out is signed out row by row.
  3. Fixed.
    • The docstrings now say the national compiles uk/target_references.json, the active references only. So a required row that is deferred or missing from the membership leaves the national without a refusal, and this test is the only guard.
    • I also widened the guard to the deferred case you describe. Before, only a required target with no active row at all failed. Now any required row that is not active fails, and the fact-period test checks it with a deferred CGT age-band row.
    • No required row is deferred in the committed membership.

Checks

Run locally for commit 44: test_uk_measurement_period.py (20 tests) and ruff. All pass.

🤖 Generated with Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants