Repository navigation
Set UK road-fuel and energy levels in the LCFS stage on a calendar-2024 basis (#1113) - #1167
Draft
juaristi22 wants to merge 14 commits into
Draft
juaristi22 wants to merge 14 commits into
juaristi22 wants to merge 14 commits into
Conversation
…ith a diagnose tool (#1113) The national solve was closing three level gaps by moving weights. obr.vat reads +24.7 percent at design weights and +24.1 percent calibrated on #1121's head calibration (policyengine-uk 2.122.2), against the 25 percent bound: the engine divides household VAT by microdata_vat_coverage (0.38, a constant since January 2023), and the frame's raw household VAT is about 46 percent of the OBR receipts (policyengine-uk#1996). The ONS 04.5.1 and 04.5.2 rows sit below the stage's DESNZ volume at QEP prices on every aligned basis (gas below unit cost alone), and fitting them cut electricity 8.6 percent, gas 18.9 percent and the gas-connected share from 84.5 to 78.9 percent. A re-solve of the 2026-09-30 problem with the rows masked changed no other row's fit (98.5 against 98.4 percent within 10 percent). The three rows join the measure-exclusion register on María's 2026-10-08 ruling (VAT until the engine factor is split; the energy rows as a published-source residual, with DESNZ volume at QEP prices kept as the level), windowed to 2027-01-08. tools/diagnose_uk_consumption_taxes.py keeps them measured on any H5 the engine loads, against the values the run's contract registry declares, beside the fuel and energy composition the rows pulled. The twelve COICOP division columns become a named view beside the LCFS rename map so the tool imports them rather than copying them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The LCFS diary records about a quarter less road fuel than ONS household spending, and binding the OBR cars receipts closed that gap by moving weight onto fuel-buying households. A declared level_road_fuel step after the fuel flag now scales petrol and diesel by one factor, so their prior-weighted total equals ONS Consumer Trends COICOP 07.2.2 for calendar 2024 (the year of the DESNZ pump prices the donor uprating uses) less the donor's own c72213 other-fuels share. The drawn mix and each household's relative spend are kept. - lcfs_consumption reads ons_household_expenditure_facts.json, now its declared consumer (re-vendored; values unchanged). - The uk_stage_lcfs_consumption_road_fuel_level stage-health gate recomputes the level from the vendored row and checks that the factor reproduces it; the licensed donor's other-fuels share is bounded at 5%. - Petrol and diesel join the levelled columns, so they carry no committed support bound (regenerated from the pinned LCFS tabs). - The existing litres audit runs after the step, so its frame litres at DESNZ pump prices describe the levelled frame. - Mirrors: operation schema and kind, stage_health parameter vocabulary, spine gate scope, gate-battery digests and entry lists, coverage manifest. The graph parity fixture is regenerated once at the end of the series. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The LCFS diary covers two weeks, so about a third of car-owning donors record no fuel purchase, and the chain reproduced that zero share among the households the fuel flag marks as buying petrol or diesel (34.3% at prior weights on #1121's head). Over a year such a car is refuelled; binding the OBR cars row had moved weight off those households to compensate. A declared redraw_zero_road_fuel step, between the fuel flag's zeroing and the level step, redraws a flagged household whose chain draw is zero from the donor's distribution among households with positive road fuel, on the chain's predictors. A weighted regime-gated QRF draws the total, positive by construction, then the petrol share of it, which keeps diesel-only and mixed households. The quantiles are keyed on the household id (predict_from_uniforms), so a household's draw doesn't depend on row order. Positive chain draws are kept, and the level step then rescales the total. - The receipt records the flagged zero share before and after, the mean among positive households before, the redrawn mean and mix, and the model's regimes. Its two fits add fit-weight records. - The uk_stage_lcfs_consumption_road_fuel_incidence gate checks the declared rule, seed, salt and trees; that no flagged household is left at zero and no unflagged one carries fuel; and the total's positive-only regime. It fails if the chain's zero share passes one half, which would point at the chain rather than the diary. - The end-to-end stage test's synthetic donor now has one-car diaries with no purchase, so the redraw runs there, and the stage's receipts pass both road-fuel gates. - Mirrors: operation schema and kind, stage_health parameter vocabulary, spine gate scope, gate-battery digests and entry lists, coverage manifest, seed pin. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#1113) The contract declares ons.household_road_fuel_expenditure: ONS Consumer Trends COICOP 07.2.2 (fuels and lubricants for personal transport), calendar year, household scope, with value_expression "petrol_spending + diesel_spending". It resolves to GBP 35.099bn for 2025. Its notes cover the other motor fuels and lubricants the class carries and the measure doesn't (0.4% on the LCFS diary), and why its design-weight gap is the engine's 2024-to-2025 uprating, since the stage levels the columns to the 2024 row. obr.fuel_duties_cars moves to the measure-exclusion register, still measured on every evaluation. Its gap breakdown: - The OBR cars line includes business-paid car fuel. ONS 07.2.2 at DESNZ pump prices implies about GBP 13.5bn of duty, 7% under it. - With the level step in place, the engine's fuel_duty reads -9.9% against it (-8.1% on DESNZ divisors; policyengine-uk#2169 A). Binding it would move weight onto fuel buyers to stand in for business use. This commit is the only one that changes if the OBR cars row is kept bound instead. - National references regenerated: 1,232 active, with 7 no_fact_at_or_before_period and 16 signed excluded unchanged. The local census is unchanged. - Both compile-parity receipts gain the row as ledger_only. - tools/diagnose_uk_consumption_taxes.py measures the new row beside the held-out ones. The register census and the contract and reference counts are re-pinned. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The chain draws COICOP 04 (housing, water and fuel) and 07 (transport) as whole diary totals, while the stage re-levels the energy and road fuel inside them. Electricity and gas move to DESNZ volume at QEP prices, and petrol and diesel to ONS 07.2.2, so the totals no longer contained their own parts: 2.7% of households had energy above their COICOP 04 total. PE-UK's explicit energy VAT base (policyengine-uk#2166) can only take energy out of the generic base if the totals hold it. A declared recompose_from_remainder step runs after the levelling steps. Each total keeps its own chain draw, less the chain's draws of the parts the stage re-levels, and adds the levelled parts back: - housing_water_and_electricity_consumption: its draw, less the drawn electricity and gas, plus the levelled electricity and gas; - transport_consumption: its draw, less the drawn petrol and diesel, plus the levelled petrol and diesel. The drawn parts are kept right after the support clip, before any step touches them. A remainder floors at zero where the parts outgrow the total: 2,056 and 1,240 of 63,956 households on the head-r3 spine, about GBP 1.0bn and 0.3bn. The chain draws each part after, and conditional on, its total, so the remainder is the chain's own split. Drawing the remainders as chain targets of their own (the plan's first version) over-drew the vehicle-purchase tail of transport: 10 to 13% above the donor in-sample, and GBP 27bn on the spine. The chain and its targets are unchanged, and output columns are unchanged. On the head-r3 spine, consumption across the twelve divisions moves -0.7% (GBP 979.0bn to 972.3bn): - housing -3.4%, as DESNZ x QEP energy (GBP 47.9bn) replaces the drawn electricity and gas (GBP 54.7bn); - transport +0.2%, as the levelled fuel (GBP 36.9bn) replaces the chain's fuel draws (GBP 28.7bn), including those zeroed for households without a fuel car. - The chain's domestic-energy target is the diary's electricity plus gas, so the liquid and solid fuels of COICOP 04.5, which have no column of their own, stay inside the housing remainder at the diary's level. The receipt records their published ONS 04.5.3 and 04.5.4 spend beside it (CY2024, vendored rows): GBP 1.55bn. - The uk_stage_lcfs_consumption_recomposed_totals gate checks each total against its declaration. It requires no negative remainder and no household below its components. The remainder must equal the draw less the drawn parts plus the floored mass, and the total must equal the remainder plus the parts. It also recomputes the uncarried ONS spend from the vendored rows. - The two totals leave the committed support bounds (regenerated); the gate holds them instead. - Mirrors: operation schema and kind, spine gate scope, gate-battery digests and entry lists, coverage manifest, source-stage pins. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The audit scales each fuel's all-road-user HMRC litres by the all-fuel OBR cars share (58.3%), giving petrol x1.30 and diesel x0.37. Diesel is mostly vans and lorries, so those per-fuel ratios mislead; only the total ratio compares like with like. The receipt now says so in per_fuel_ratio_basis until a per-fuel cars benchmark exists (DESNZ sub-national road fuel by vehicle and fuel, requested in PolicyEngine/chronicle#322). The plan's quarterly 04.5.x and 07.2.2 vendoring is dropped: no code would read those rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The lcfs_consumption stage now sets the levels of electricity, gas, petrol and diesel, the gas connection and the fuel incidence, so calibration should barely move them. Before this series, the bound rows moved them a lot: gas fell 19% through the weights and the gas-connected share 84.5% to 78.9%. uk_diagnostics gains a report-only consumption_drift block, computed from the calibration's initial and final weights on the frame's own columns: - totals: electricity, gas, domestic energy, petrol, diesel, and consumption across the twelve COICOP divisions; - shares: gas-connected households, fuel-buying households, and flagged fuel-car households without fuel. Each value carries design, final, change and relative change. The block is omitted for a frame without the consumption columns. The typed schema (UKConsumptionDrift, UKDriftValue) is optional and forbids undeclared fields, as the rest of the UK extension does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PE-UK projects each household spend column from the H5's year, 2024, with
calendar-year growth, and Microcosm's calibration uprates the same values to
the calendar-2025 window, so both read every column as calendar-2024 prices.
Two of the stage's levels were on FY2024-25 instead:
- energy: DESNZ volume and QEP prices for FY2024-25;
- bus fares: the yield per fare-paying boarding from FY2024-25 receipts and
journeys.
Each column would then have needed its own basis to be projected correctly.
Instead every column now sits on calendar 2024, so none does:
- Energy is priced at QEP's calendar-2024 averages paid and levelled to the
DESNZ Energy Trends volume over the four quarters of 2024, both vendored
beside the fiscal rows. The level now takes a calendar or fiscal year.
- Each area's FY2024-25 bus yield is re-priced to calendar 2024 by its fares
index:
- London and England outside London: DfT's quarterly local bus fares
index, calendar-year mean over fiscal-year mean;
- Scotland: its calendar-year index, taking the fiscal year as three
quarters of 2024 and one of 2025;
- Northern Ireland: England's index.
The factors are 1.000 for London (fares held flat), 0.965 for England
outside London (the fare cap rose in January 2025), 0.988 for Scotland and
0.982 for Northern Ireland. The bus pricing gate checks them.
- The diary columns (CPI-uprated to 2024) and road fuel (ONS 07.2.2 for 2024)
were already on calendar 2024.
The uk_stage_lcfs_consumption_basis gate holds every level-setting step to
calendar 2024: the donor uprating's target year, the energy prices' and the
DESNZ level's periods, the bus prices' year and the road-fuel level's period.
A step whose receipt is absent fails.
On the head-r3 spine all seven lcfs stage-health gates pass. Energy moves
+1.7% (GBP 47.9bn to 48.8bn) and bus fares -2.1%; fuel and the diary columns
don't move. PE-UK can then project every column from the H5's year with no
basis of its own. It still has to project energy: an energy price index from
2024 in place of the uk-data assumption that bills are stored at FY2026-27
prices.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
experiments/1113-consumption-tax-levels-receipts.md is the receipt the register entries cite. It holds: - the commit list; - microcosm#1121's head calibration; - the measurements that put the levels in the stage (energy volumes and sources, road fuel, the VAT factor); - the masked re-solve of the 2026-09-30 problem; - the stage-only development run on the head-r3 spine, where all seven lcfs stage-health gates pass, with the in-sample check that moved the recomposed totals off chain-drawn remainders. One changelog fragment per change. The charter-H2 graph parity fixture and the national remeasure follow Group B (PolicyEngine/chronicle#322), whose feed re-pin moves the fixture again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#323) (#1113) The artifact is chronicle main at the #323 merge: 352,549 rows (825406f's 344,402 plus 8,147), consumer_fact.v4 with an unchanged schema. It brings ONS Consumer Trends by COICOP division and class, with the national and domestic totals, and DESNZ road transport fuel consumption by vehicle type and fuel. - No existing value moved. Every 825406f row is byte-identical except the seven Consumer Trends series already in the feed. #323 restates their 217 rows on ONS's domestic concept (territory role; price-basis, seasonal-adjustment, unit, sheet and concept dimensions), under new keys with the same values. - The national and local reference files are byte-identical. The electricity, gas and road-fuel targets resolve to the same values through the new keys, which is all the national membership records beside the feed label. - The three compile-parity receipts are unchanged. - The vendored resources carry the same rows under the new feed identity. The Consumer Trends resource names the domestic concept, and the road-fuel target's note says why no national bridge applies: ONS splits the tourism adjustment by no class. - The charter-H2 fixture moves at the branch head, after the stage changes that follow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The stage receipt gains a report-only division_capture block. For each of the twelve COICOP divisions it sets the prior-weighted frame total against ONS Consumer Trends for the uprating's target year (calendar 2024), less the classes a household diary does not record: 02.3 narcotics, 04.2 owner-occupiers' imputed rent and 12.6.1 FISIM. - The divisions are on ONS's domestic concept. Consumer Trends publishes the tourism adjustment for the total only, so only the twelve together are also compared on the national concept (NAT0). International students' tuition has no published split either, and stays in the domestic gap of division 10. - The Consumer Trends resource vendors the 0CN totals and divisions and the three out-of-survey classes for 2023 to 2025 (42 to 93 rows). - Nothing is gated on it. The engine-free stage test has no uprating step, so it checks the block is absent there; a unit test checks the arithmetic against the vendored rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The litres audit's per-fuel benchmark is now HMRC's clearances of that fuel times the cars share of its UK road use, from DESNZ sub-national road transport fuel consumption by vehicle type and fuel (PolicyEngine/chronicle#322), for the uprating's target year. Within one fuel the ktoe cancel, so no conversion factor enters. - For 2024 cars burn 97.8% of road petrol and 39.2% of road diesel; the OBR cars share of fuel duty receipts applied one share to both, which A8 labelled. The OBR-based total stays on the receipt beside the new one. - desnz_road_fuel_by_vehicle.json vendors the UK rows for 2022 to 2024 (30 rows), and the lcfs_consumption stage declares it as an artifact. - Report-only, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixture moves once, at the branch head, for every stage change above. - The bundle gains the lcfs_consumption steps: redraw_zero_road_fuel, level_road_fuel and recompose_from_remainder. It also gains the calendar-2024 energy and bus-fare parameters, the ONS and DESNZ artifact roles, and the updated source and notes text. - The synthetic LCFS donor gains c72213, the other motor fuels whose donor share level_road_fuel nets from ONS 07.2.2. It is added after the positional columns, so no other synthetic value moves. - In the graph JSON only stage_contract_sha256 moves: for lcfs_consumption and for the ten stages that declare a vendored resource the 1ee7dfe re-pin restamped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The receipts gain B1 to B3 and the fixture commit, plus the stage-only run at B3: all seven lcfs gates pass, and the Group A figures hold. - Division capture: the frame's twelve divisions are 72.8% of ONS's survey scope for 2024, against 58.5% for the donor's own diary. Housing reads 1.059, GBP 8.0bn of its excess being D1's energy level. Furnishings read 1.251: the frame draws 66% more per household than the donor. - Litres audit: petrol reads 1.011 of its cars benchmark and diesel 0.709. The total is 0.894, against 0.951 on the OBR cars share. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Automated review pass (Claude Code, high effort) — round 1 at
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Part of #1113 and of #1123 (item 3). The UK LCFS stage now sets household road fuel and domestic energy at their published levels, at calendar-2024 prices, instead of leaving the calibration weights to reach them. The rows whose gap is not a weight question become measured diagnostics. This follows María's rulings of 2026-10-08 (D1–D3), plus two of 2026-10-09: the ONS road-fuel row is bound, and every LCFS spend column sits on one calendar-2024 basis.
obr.vat,ons.household_electricity_expenditure,ons.household_gas_expenditureandobr.fuel_duties_carsmove to the measure-exclusion register. They are still measured on every evaluation, and the entries expire 2027-01-08. The contract binds a new household-scope row in their place:ons.household_road_fuel_expenditure(ONS Consumer Trends 07.2.2, calendar 2025, measured as petrol plus diesel spend).redraw_zero_road_fuelgives every flagged fuel-car household a positive fuel spend;level_road_fuelscales petrol and diesel to ONS 07.2.2 for 2024, less the donor's other-fuels share;recompose_from_remainderwrites housing and transport around their levelled parts.road_fuel_incidence,road_fuel_level,recomposed_totalsandconsumption_basis.uk_diagnostics.consumption_drift: the levelled totals and shares at design and final weights;division_capturein the stage receipt: each COICOP division against ONS Consumer Trends;1ee7dfe(UK consumption-tax facts: Consumer Trends for every COICOP division, and road fuel by vehicle type and fuel (PolicyEngine/microcosm#1113, PolicyEngine/microcosm#1123) chronicle#322 via Expand UK consumption-tax expenditure and road fuel facts chronicle#323, 352,549 rows), which brings Consumer Trends by division and DESNZ road fuel by vehicle type and fuel. No compiled target value moves on either surface, and the three compile-parity receipts are unchanged.Receipts, findings and rulings are in
experiments/1113-consumption-tax-levels-receipts.md.Why the levels belong to the stage
On #1121's head calibration (policyengine-uk 2.122.2, 1,167 rows):
obr.vatreads +24.7% at design weights, against a 25% bound. Its level is the engine'smicrodata_vat_coveragefactor (0.38, with no recorded derivation; policyengine-uk#1996), not a weight question.obr.fuel_duties_carsreads −34.5% at design weights. It is a fiscal-year duty total that includes business use of cars. The LCFS diary records about a quarter less road fuel than ONS, and a third of fuel-car households show no fuel purchase in the two-week diary.Stage-only measurements
These re-run
lcfs_consumptionon #1121's head-r3 spine (63,956 households), before the rebase. All seven lcfs stage-health gates pass.Not in this PR
obr.vat. That waits until policyengine-uk splits its VAT factor (policyengine-uk#1996).Testing
7f235941c, a check ran after every commit. Each commit loads the UK spec, has a current release-input coverage manifest, and has current gate digests mirrored inmicrocosm-data.lcfs_consumption's, and those of the ten stages declaring a vendored resource the re-pin restamped. The commits before that one failtest_h2_uk_spine_parityon their own.🤖 Generated with Claude Code