[juaristi22]

Follows Vahid's review of #1381 (#1359).

Microcosm's regime-gated QRF decides whether each recipient row is zero or positive with a boosted classifier, the sign gate. The gate is calibrated only on the donors it was fitted to, so on every other row it draws rare outcomes too rarely.

This PR adds an opt-in that calibrates each gate out of fold, and turns it on for the LCFS consumption chain only. Every other UK stage and every US fit keep today's gate, and their draws do not move.

Stacked on #1381, which is stacked on #1383. This PR gets no CI until both merge. Merge them without deleting their branches, or this PR closes with them. The tests below ran locally on the stack.

The problem

Addressing Vahid's review of #1381 showed that the excise shares the LCFS chain draws lose incidence on the frame. Tobacco sat on 8.2% of frame households against 10.3% of donors, and betting and gaming on 2.0% against 3.6%.

Held-out tests trace it to the gate, not to the frame.

• A row is drawn positive when its keyed uniform falls under the gate's probability, so the share of positive draws is the gate's mean probability.

• The gate is a default HistGradientBoostingClassifier, fitted and calibrated on the same donors: 100 rounds, 31 leaves, 20 rows a leaf, and no early stopping below 10,000 rows. It memorises a rare class's donors, so on rows it was not fitted to, its mean probability of that class falls short of the class's rate.

• Out of fold on the 4,995 LCFS 2024-25 donors, the current gates' mean probability over the true rate is:

  • education 0.42, solid fuels 0.11 and liquid fuels 0.81;
  • tobacco 0.73, gambling stakes 0.82 and betting and gaming 0.88.

  For education, solid fuels and gambling stakes, the held-out log loss is worse than a constant at the class rate.

On the frame, rare chain targets lose incidence against the donor. In a stage-only run on #1333's test spine, at prior weights:

• solid fuels: 0.33% of households, against 2.47%;
• liquid fuels: 3.09%, against 4.54%;
• education: 4.31%, against 5.83%;
• tobacco: 8.16%, against 10.34%;
• betting and gaming: 1.96%, against 3.58%.

How the QRF model changes

An opt-in on RegimeGatedQRF. RegimeGatedQRF(gate_calibration=...) takes a name from the new GateCalibration constants.

• in_sample, the default, is the gate as before, byte for byte.
• cross_fitted works in four steps:
  1. It cuts the training rows into five folds, stratified by class.
  2. For each fold, it fits a gate the same way, with the same classifier and weights, on the other four folds, and that gate scores the fold it left out.
  3. It fits one weighted calibration on the pooled out-of-fold scores: a slope and a per-class offset on the gates' log-probabilities. For two classes this is Platt scaling; for three-sign gates, a shared temperature with class offsets. At the fit, the calibrated out-of-fold probabilities' weighted mean equals each class's weighted rate.
  4. At draw time, each row's probability is the mean of the five mapped fold gates.

What does not change.

• The fold assignment and the fold seeds derive from the one seed the in-sample gate takes, so the fit's random stream advances by the same draw. Every magnitude forest, every bootstrap and every keyed uniform is identical; only which side of zero a row falls on moves.
• Every draw path takes the cross-fitted gate unchanged: predict, predict_from_uniforms, keyed fit_draw_next and conditional_uniforms. predict_positive_from_uniforms bypasses the gate, as before.
• A class with fewer rows than folds cannot appear in every fold, so its target keeps the in-sample gate.
• QRFChainState records the calibration, but writes it into a checkpoint's model_config only when it is cross-fitted. So in-sample checkpoints are byte-identical, the US banks and PUF chain included. Resuming under a different calibration is refused.

Why this form. I compared the candidates out of fold on the LCFS donors, over one five-fold pass. The figure is the mean over the seven rare targets of |log(mean probability / rate)|:

• current gate: 0.578;
• regularisation alone (early stopping, 25 rounds, or small trees): 0.09 to 0.12;
• a logit shift that matches the out-of-fold mean to the rate: 0.117, or 0.028 on an early-stopped gate;
• one calibration map applied to a gate fitted on every row: 0.066. It overshoots the rarest targets.
• the cross-fitted fold ensemble: 0.005, with the lowest log loss on every target, common ones included.

It is written in qrf.py, not through sklearn's CalibratedClassifierCV. sklearn's stratified splits, and HistGradientBoostingClassifier's own early stopping, fail on a class with too few rows.

Where it runs: only the LCFS chain

• lcfs_consumption's fit_weighted_qrf_chain declares gate_calibration: cross_fitted, which the schema allows as an enum on that operation. impute_lcfs_consumption passes it to the model.
• Nothing else opts in. The other UK stages, the LCFS road-fuel redraw and prepayment flag, and every US fit keep the in-sample gate.
• A control run of the LCFS stage, with the new qrf.py and no declaration, reproduces Write policyengine-uk's alcohol, tobacco and gambling duty inputs from the LCFS stage #1381's stage run byte for byte.

Results on the frame

This is a stage-only run of lcfs_consumption on #1333's test spine (78,440 households, policyengine-uk 2.151.0), the control against this head. All eight LCFS stage-health gates pass in both.

Rare targets. Share of households with the spending, at prior weights, control → head, with LCFS donors in brackets:

• solid fuels: 0.33% → 2.25% (2.47%);
• liquid fuels: 3.09% → 3.88% (4.54%);
• education: 4.31% → 6.65% (5.83%);
• tobacco: 8.16% → 11.16% (10.34%);
• gambling stakes: 14.63% → 16.70% (17.03%);
• National Lottery: 10.68% → 10.85% (11.42%);
• betting and gaming: 1.96% → 2.63% (3.58%).

Common targets. The in-sample gates also under-drew the zeros of the near-universal divisions. With those zeros drawn at their rate:

• Incidence moves towards the donor's:
  • food: 99.67% → 99.05%, against 99.15%;
  • recreation: 98.98% → 97.10%, against 97.27%;
  • miscellaneous: 98.43% → 97.30%, against 97.31%.
• The twelve divisions' prior-weighted total falls 1.4%. The seven divisions held at the shared frame-to-donor ratio fall 1.42% with it, and restaurants and hotels falls 4.2%.
• On-trade alcohol: 30.37% → 29.25% (31.01%).

Excise inputs. They keep their national levels; who carries them changes:

• Tobacco duty is about 1.8 times tobacco spending, from 2.2, on 11.2% of households.
• The other gambling duties are 5.5 times betting and gaming stakes, from 6.4, on 2.6%.
• Lottery Duty is 0.61 times National Lottery stakes, from 0.56.
• Alcohol duty is 0.45 times alcohol spending, from 0.41.

The stage notes carry the new figures.

Columns and cost.

• 42 of the stage's 46 written columns move: all twenty chain targets and 22 of the 24 excise inputs.
• The stage takes 264 s against 110 s, since each gate is now five.

Read before reviewing

• Re-pins. Any byte edit of qrf.py moves these, so they are re-pinned:

  • the fit.qrf graph-parity pins (its draws are positive-only and do not move);
  • the spec-loader golden vector;
  • the US seed protocol and seed map digests, with the regenerated F0 coverage report. The second commit does this and cites 1734b9e;
  • the US bundle hash in the multispine pool tool test.

  Only CI's engine-us and engine-workflow tiers check the US ones.

• What remains is the chain's own. The chain conditions each target on its own earlier draws, not on a donor's spending. Out of fold, with the chain's earlier draws as features:

  • liquid fuels are drawn at 0.75 of their rate;
  • the National Lottery at 0.88;
  • betting and gaming at 0.91.

  On the frame, tobacco now sits above the donor. No gate calibration reaches that.

• Write policyengine-uk's alcohol, tobacco and gambling duty inputs from the LCFS stage #1381's receipts and changelog record the in-sample figures as of Write policyengine-uk's alcohol, tobacco and gambling duty inputs from the LCFS stage #1381; this PR's changelog records the change.

Should the other gated fits move too?

This PR turns the calibration on only where it was measured. Every other gated fit keeps the in-sample gate, and is likely to under-draw its rare classes on recipients in the same way:

• UK:
  • etb_services, etb_vat and nts_bus_travel;
  • spi_housing_shell;
  • spi_income, stages 1 and 2, and through them spi_rank_transport;
  • uc_reporter_redraw, was_wealth and salary_sacrifice;
  • the LCFS road-fuel redraw's petrol share and the prepayment flag.
• US:
  • puf_qrf_chain and acs_transfer;
  • about twenty predict() fits, among them child support, childcare, disability benefits, retirement contributions and distributions, SCF wealth and auto loans, and the SIPP fits.

Do we want to measure each of these with the same held-out harness and opt in the ones that need it, stage by stage? A global change of the default would do more than move every gated draw:

• it would move the US exact-draw pin in test_spec_engine_seeds.py;
• it would let resumed UK graph builds reuse stage outputs from the old gate, because UK stage hashes do not include qrf.py.

Opting in per stage avoids both, because a stage's declaration changes its hash.

Tests

Run locally on the stack, targeted to the touched surfaces, with JUnit counts and no failures or skips:

• microcosm-fit: 149, 10 of them new;
• graph parity pins, H parity and the executor: 105;
• the spec loader, 10; inventory coverage and seeds, 24; and spec_engine_coverage.py --check;
• engine-free UK and spec, 531:
  • the LCFS stage, 133, with one new test;
  • stage health and source stages;
  • coverage and the UK graph;
  • re-emission, country spec and bundles;
• the engine tier, 64: H2 UK spine parity on the regenerated fixture, the engine excise inputs and engine coverage;
• the coverage manifest's --check and ci_test_plan verify.

Not run here, because they need the US engine: engine_contract/us/test_spec_engine_coverage_tool.py and engine_workflow/us/test_us_multispine_pool_tool.py. The wheels job did not run either.

🤖 Generated with Claude Code

[MaxGhenis]

Thanks María. This is a clear diagnosis, and calibrating the gate out of fold is the right fix. I'd land it after the Budget rather than in Monday's cut, since it moves 22 of the 24 excise inputs and the COICOP totals that the VAT coverage is being derived on.

Two changes before merge, and one wording fix:

1. Fall back when a fold has too little weight. _can_cross_fit counts rows, so a target with zero-weight rows can pass it and still leave a training fold with no weight. sklearn then raises ZeroDivisionError: Weights sum to zero. A reproducer: ten donors, five per class, eight of them at weight zero. The in-sample gate fits; the cross-fitted one crashes. Please check each fold's training weight per class, and keep the in-sample gate when it falls short. The LCFS weights should all be positive, but this matters before other stages opt in.
2. Check convergence before returning the calibration. When the out-of-fold scores are saturated, the Newton step is huge and the line search hits its 1e-10 floor. The loop then returns the identity map with no warning, so the rate balance fails silently. A reproducer: ten rows, five per class, every logit 29. That gives a positive-class mean of about 1 against a rate of 0.5. Please check the weighted residuals at exit, and fall back to an offsets-only fit or the class rates, or raise. Real gates probably never get there, and your LCFS figures show the balance held, but it should fail loudly.
3. Wording. The balance is exact on the pooled out-of-fold donor scores, not on recipients. Also, in the chain, a sign that changes moves later targets' amounts through their predictors, even though the forests and uniforms are the same. The changelogs and the module docstring should say both.

Please also add Hypothesis tests for the invariants:

• the weighted rate balance, including saturated and constant scores;
• the sparse-weight fallback;
• base-versus-head byte identity on the default path.

The two reviews (one by Claude, one by GPT-6.1 Sol) checked these, all on synthetic data:

• The default path is byte-identical to the base, and so are 36 checkpoint byte hashes across all seven regimes.
• Resuming under a different calibration is refused.
• LCFS is the only opt-in.
• The 22 UK manifest re-pins and the US seed-protocol and seed-map pins recompute.
• On held-out data the cross-fitted gate lands at about 1.0 of the true rate with 100 or more positives; the in-sample gate lands at 0.05 to 0.5.

I can send you the reproducers and the proposed property tests.