[MaxGhenis]

WIP. Basis: cos d1817 (Max, 21 Oct 2026): run historical UK Budgets on the newest Microcosm population as policy experiments to catch model bugs.

These are policy experiments on the modern population, not replays, and not PolicyEngine's costing of any past event. The certified replay path is untouched: compute_uk_event.refuse_unexecutable still refuses every historical registry.

Design note

Full design: docs/uk_replay/POLICY_EXPERIMENTS.md. Running findings: docs/uk_replay/EXPERIMENTS.md. Evidence for the claims below: docs/uk_replay/evidence/policy_experiments/.

• Registry flag. None is added; the historical registries are not edited. An experiment spec data/uk/experiments/<event>.json (kind uk_modern_population_policy_experiments) binds its registry by SHA-256 and reads its source rows for the OBR side. The execution: requires_event_population block keeps refusing replays.

• What an experiment is. One registry measure restated as a parameter change on modern law, as either reversal_on_modern_law (modern law contains the measure, so restore the pre-measure policy and keep later changes) or forward_on_modern_law (modern law lacks it, so apply it). The replay's orientation rule applies: positive means a gain to the Exchequer.

• Output location. results/uk/experiments/<population>/<event>/: per experiment-year artifacts, MODERN_LAW_<year>.json, EXPERIMENT_MANIFEST.json, FINDINGS.json, FINDINGS.md. Nothing is written under results/uk/events, results/uk/replays or the variant index, and a property test checks that for any output path.

• Populations are not variants. data/uk/experiments/populations.json (status experiment_population, ids exp-…) has the variant's identity shape, so the replay's environment checks apply. It is never added to replay_variants.json and never sealed. A test checks that no replay reader (the DB build, the namespace scan, the results-PR builder, workflows) names the experiment namespace.

• Labels. Every experiment-year artifact, EXPERIMENT_MANIFEST.json, FINDINGS.json and SUMMARY.json carries label: modern_population_sensitivity (PLAN.md's name for this question), is_replay: false and "Policy experiment on the modern population: not a replay or a costing of ." Specs carry the label and the sentence; MODERN_LAW files carry the label and is_replay; RUN_LOG files are timings and carry none. All of them live only under the experiment namespaces.

• Runs. Each run reuses compute_uk_event.run_sim: the policyengine.py managed microsimulation, the dataset SHA-256 before and after, the dataset year set, and every touched parameter read back from the simulation's processed tree. Runs execute on Modal (pipeline/modal_uk_experiment.py) with network blocked, holding one of the replay's two shared simulation slots.

• Which constructions count as anomalous (classify, a pure function):

  Code	Severity	Condition
  crash	investigate	alternate world raised
  reform_not_applied	investigate	processed parameter ≠ construction intent
  construction_noop	note	modern law already equals the pre-measure world (superseded, or missing from this engine's law)
  inert	investigate (OBR ≥ £20m) / check	a parameter changed but the effect is under £1m
  sign_mismatch	investigate	OBR ≥ £20m and PE the opposite sign, ≥ £5m and ≥ 10%
  magnitude_outside_10x / _3x	investigate / check	scaled ratio outside [1/10, 10] / [1/3, 3]
  obr_zero_pe_moves	note	OBR < £1m, PE ≥ £100m
  mirror_mismatch	investigate	mirrored experiments (same reform, opposite constructions) are not exact negatives
  modern_law_differs	investigate	a touched parameter's processed current-law value differs from the one the construction assumed (assumes_modern)

  Magnitude flags apply only when magnitude_check is scaled, the expected value is at least £100m, and the mapped heads hold at least half the OBR's costing.

• OBR scaling. For each head group, scale = PE modern baseline / OBR event-vintage forecast of that head, where PE's variables cover the whole head (income_tax with Income tax, the four NICs classes with NICs, capital_gains_tax with CGT). Otherwise scale = OBR newest forecast / OBR event-vintage forecast. The forecasts are rows of the OBR Historical official forecasts database (Spring 2026), extracted with cell references into data/uk/experiments/obr_head_forecasts.json and cross-checked value for value against the March 2025 edition that the vintage locks name. The scaling is a bug screen, not a vintage comparison, which is why the bands are wide and every flag needs a diagnosis.

Status (23 Oct, 14:00Z)

Bank. 43 experiments across 19 events, from Budget 2010 to Autumn Statement 2022, built by pipeline/uk_experiment_specs.py with every counterfactual's arithmetic and source.

• Events with a registry (AB2021, SS2022, AS2022) bind it. Older events bind the inventory (Scope the vintage-faithful replay of 32 OBR-costed fiscal events #158).
• Events whose costing window ends before 2024 compare with their last costing year, scaled to the run year (carried_from_fy).
• Uprating counterfactuals use scale: each named parameter's own processed modern value times a factor, so every later uprating of the engine it runs on is kept.
• Every value a reform sets states the current law it is written against, for every year the experiment can run (assumes_modern), or is set outright with a reason (set_outright); validate_spec refuses anything else. An engine whose processed value differs gets modern_law_differs, on any population.

Populations.

• certified_pair: policyengine.py 6.10.0 with policyengine-uk 2.132.6 on the interim Microcosm cut, probed on Modal (data years 2024–2032).
• model_swap: pe-uk 2.143.0 on the same cut. Against the certified pair on the same data, it separates model changes from data. It is valid only where the data carries the newer model's inputs: three variables 2.132.6 computed are dataset inputs by 2.143.0, and the interim cut has none of them (model_swap_input_gaps).

Findings (full log in docs/uk_replay/EXPERIMENTS.md):

• Bug filed: PolicyEngine/policyengine-uk#2551. Setting the statutory Marriage Allowance rate switches on the CPS expanded-MA reform, which keeps 10% and extends it to higher-rate taxpayers. Removing the allowance therefore lowered income tax by £0.23bn. Reproduced on synthetic couples on 2.143.0 (script and output in the evidence folder). Fixed upstream by PolicyEngine/policyengine-uk#2553, first released in 2.148.2. On 2.148.2 the same couples behave as the statute says.
• pe-uk 2.132.6 currency gaps, all fixed upstream and confirmed by the swap or the new detector:
  • dividend bands at £37,500 and £150,000 until April 2026 (fixed in 2.139.4): modern_law_differs on the Budget 2018 and Spring Budget 2021 freezes and the additional-rate threshold;
  • no Growth Plan SDLT relief, and no Autumn Budget 2024 +2pp on additional dwellings (both fixed in 2.141.8, 4044d6ccf): the surcharge reversal removes 1pp, not 3pp, on 2.132.6;
  • additional-dwelling purchases overstated (#2238, fixed in 2.134.1); from then the purchase is a dataset input that the interim cut lacks, so the surcharge is inert on the swap until the pre-Budget cut (microcosm#1308).
• Diagnosed, not bugs: Budget 2017's Class 4 rise (the OBR's costing is net of incorporation); Budget 2016's trading allowance (neither data line carries gross receipts); Budget 2012's 45p (a behavioural costing); construction_noop notes on measures modern law supersedes.
• Clean otherwise: every other experiment returns the OBR's sign, and every magnitude-checked ratio is inside the 3x band. That includes the HSCL and its reversal (exact negatives across containers), the 20% and 19% basic-rate pair, the UC taper, the double lock, and the two-child limit.
• The population. The income tax base is 0.94–0.98 of the OBR's March 2026 forecast, and NICs 1.01–1.05.

Answers to the first review (Opus 5.5, at bb445279; ~/reviews/uk-policy-experiments/review-out-opus.md): the four construction findings are corrected and rerun, modern_law_differs added, findings refuses a run on another spec, adopt checks before copying, committed findings and summaries regenerate byte for byte in a test, containment refuses any checkout's replay roots, and the PR's claims have evidence files. The SS2022 primary-threshold path has more than one defensible reading; the choice is logged and both experiments are sign only.

Answers to the second review (Opus 5.5, at 767a695e; ~/reviews/uk-policy-experiments/review-out-opus-r2.md):

• B1, the swap's model changes. The seven undiagnosed swap rows are diagnosed: VAT grossing (pe-uk#1996, 2.135.7: 0.38 → 0.67 × 0.7, so VAT scales by 0.810 and the rate rise's ratio moves 1.31 → 1.06); the savings starting-rate taper (35f992c6a, #1799, 2.139.2: ratio 0.43 → 0.73); relieved pension contributions in adjusted net income (1dbc88172, 2.134.8); and the Growth Plan first-time-buyer limits (4044d6ccf). The two income tax ones have synthetic-household evidence. The log now lists every swap change with its cause.
• B2, undeclared current law. Every reform value now states its current law: filled from one table that both engines match (at that round, 204 declarations over 144 path-years and 50 parameters; now 207 over 147 and 51), plus values set outright. validate_spec enforces it, so the detector covers the pre-Budget pair too.
• Smaller fixes: containment is case-insensitive (N1); the threshold hold is a stated convention (N2); the lower profits limit is 52 × the weekly threshold (N3); claim wording and prose-only claims (N4, N5); lint (N6); a comment's arithmetic (N7).

Answers to the third review (GPT-6.1 Sol, at 575f4cc8; ~/reviews/uk-policy-experiments/review-out-r3.md):

• Blocking: the Growth Plan SDLT relief has three legs. Both constructions set only the main scale. They now also set the additional-dwellings first band to £250,000 (Temporary Relief Act 2023 s. 1(3)) and the lease-rent nil-rate band to £250,000 (s. 1(4)). Synthetic checks are in the evidence folder: a £200,000 additional dwelling moves from £6,000 to £7,500 when the relief is removed in 2024-25, and from £11,500 to £10,000 when it is kept in 2025-26. Neither engine has the lease leg itself, which is now recorded on policyengine-uk#2332.
• Found while fixing it: brackets that share a threshold add their rates in policyengine. So the first attempt's zero-width band taxed additional dwellings above £250,000 at 13%. validate_spec now refuses shared thresholds among the brackets a reform sets; a collision with a bracket the reform leaves alone needs the engine's schedule and shows in the effect.
• Smaller fixes: the citations, case-insensitive repository containment, a crash keeping modern_law_differs, lint, the narrowed label claim, and the declaration count (now 207 declarations over 147 distinct path-years and 51 parameters).

Reruns. All 17 events whose specs changed in round 2 were rerun on both populations and adopted. Autumn Statement 2022 was rerun again for the SDLT correction and adopted. Every flag on either population and every model-swap change is diagnosed, and the experiment tests pass (130). Every committed run is on the current spec, every flag on either population and every model-swap change is diagnosed, and the experiment tests pass (127).

Results pipeline.

• PE_UK_PRIVATE_HF_READ_TOKEN reads populace-uk-private: a scratch workflow's HEAD requests got 302 on the Microcosm file and 200 on its release manifest, against 401 without a token (run link in the evidence folder).
• 6.10.0's qualification lost its runner at the same AB2025 step in both attempts (47fe63f4, dfc9d655), and 6.10.1's first attempt (201e753c) also ended run_artifact_failed, though its run job left no log to say where. Run the UK replay's year workers that fit the runner's memory, and log memory around each event #218 measures, on Modal, 10.2 GiB for one worker over the whole of AB2025 and 18.5 GiB for two on Microcosm, against 9.3 GiB for two on the enhanced FRS, so two Microcosm workers exceed the 16 GB hosted runner. It carries the fix and the retries.

Invariants (property-tested, tests/test_uk_policy_experiment.py)

1. Orientation: reversal and forward give exact negatives, and spending negates tax.
2. A total is the exact (math.fsum) sum of its head groups.
3. The mapped and unmapped OBR values sum to the registry's FY rows, in Decimal.
4. classify is pure and invariant under joint negation and joint positive scaling above the floors. A ratio further from 1 than a flagged one is also flagged. crash, inert and construction_noop exclude the sign and magnitude codes.
5. No output path resolves into a replay root of this checkout or any other.
6. Mirrors return exact negatives (checked on real runs).
7. Every committed FINDINGS and SUMMARY file regenerates byte for byte from the committed artifacts, and findings refuses a run on another spec.
8. Where a model swap and its certified pair differ on the current law of a parameter a construction sets, the construction declares the law it assumes or sets the parameter outright.

Plan

• Harness, specs, scaling bases, tests
• Modal probe → population records (certified pair and model swap)
• AS2022, SS2022, AB2021 and 16 older events on the interim pair; model swap in progress
• Diagnose anomalies; pe-uk#2551 filed and handed to the UK hub
• Opus review of bb445279; its findings answered
• Reruns of the six changed events on both populations
• Round 2 (Opus 5.5 at 767a695e); its findings answered
• Reruns of the 17 changed events on both populations
• Round 3 (GPT-6.1 Sol at 575f4cc8); its findings answered
• Autumn Statement 2022 rerun on both populations
• Rounds 4 and 5 (GPT-6.1 Sol at 9b15c476 and 378b8496); their findings answered
• Pre-Budget pair when published

axiom: n/a: replay and experiment tooling; no policy encoding.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
policyengine-scorecard	Ready	Preview	Oct 23, 2026 1:14pm UTC

[MaxGhenis]

Merged by squash at head 378b8496 (merge commit 9d64bf9a).

Reviews. Six rounds, each answered before the next:

1. Opus 5.5 at bb445279;
2. Opus 5.5 at 767a695e;
3. GPT-6.1 Sol at 575f4cc8, which found the blocking Growth Plan SDLT legs;
4. Sol at 9b15c476;
5. Sol at 378b8496;
6. Sol confirmation of the PR body at 378b8496: MERGE-SAFE.

The reports are in ~/reviews/uk-policy-experiments/.

Gates. gh pr checks exits 0 on the latest run (2,455-test suite, DB drift and replay namespace checks). The PR is MERGEABLE (CLEAN), ready, with no changes requested. Separation from replays was rechecked in every round: additions only, with no registry, variant index, DB or workflow change.

Upstream from this work:

• Setting the statutory Marriage Allowance rate switches on the CPS expanded Marriage Allowance reform policyengine-uk#2551, the Marriage Allowance trigger, fixed by #2553 in 2.148.2;
• policyengine-uk#2556, wealth_tax taxing gross wealth (the scorecard registry marks the measure not expressible in Mark the reported wealth tax not expressible: the engine taxes gross wealth #221);
• the missing 2022-25 lease-rent band, recorded on policyengine-uk#2332.

Next. The pre-Budget Microcosm cut runs as a new population when it is published. It will carry the fixed engine, so it gives the national Marriage Allowance rerun and the additional-dwellings legs that the interim cut's input gap hides.

🤖 Generated with Claude Code