Repository navigation
Sign the draught shares' degenerate exclusions and build on policyengine-uk #2695 - #1411
Conversation
The excise step (microcosm#1359) writes HMRC's published draught-relief share of beer and of cider clearances to every household, so household.beer_draught_share and household.cider_draught_share are constant by construction, and uk_degenerate_release_surface refuses them on both the local and the national line. María approved both entries on 2026-10-27 (adjudication microcosm#1359, six-month window, every line), like the Tax-Free Childcare routed share's. The register is solve-scoped (solve_scope.py keeps it in the solve fingerprint), so the release contract's degenerate evidence digest moves with it, and the battery tests' exclusion clock moves to the newest receipt's approval date. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PolicyEngine/policyengine-uk#2695 keeps every property allowance election and stops working out every benefit for the whole population on each of its comparison branches, the cost behind Microcosm UK microsimulations running several times slower since 2.157.0 (policyengine-uk#2694). Until a release carries it, the workspace installs policyengine-uk from that pull request's head commit (ead3444d, the 2.162.10 tag plus its three commits), which the engine freeze accepts as policyengine-uk 2.162.10; the lock moves to the release once it is out. The uprating and bonus-sacrifice pins are re-taken on the new engine with every value unchanged, concept coverage finds no input added or removed, the coverage manifest and the H2 spine fixture are regenerated, the version mentions move, the US worker identity pins the new lock, and the engine pin test reads the lock's direct reference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PolicyEngine/policyengine-uk#2695's head now merges the 2.162.12 release (b71ce793), so the workspace installs that commit, which the engine freeze accepts as policyengine-uk 2.162.12. 2.162.11 imputes non-workers' wages once per participation step with unchanged outputs, and 2.162.12 changes only CI. The uprating and bonus-sacrifice pins keep every value, concept coverage finds no input added or removed, and the coverage manifest, the H2 spine fixture, the version mentions and the US worker identity's lock digest move with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drafted by tools/draft_uk_dense_signoff.py from the first dense attempt of this pair, uk-local-candidate-f100-s42-20261027T022836Z-d5f72865 at 90ff43f, with its degenerate-surface failure set aside: that failure named only the two draught-share columns, which the previous commits' register excludes. Approved on 2026-10-27 by juaristi22 before the rebuild, so that it can pass without a reuse build: three national target-fit rows and twenty local target-fit cells added, the entries still earned kept, and one national and six local entries the attempt no longer earns retired. Every entry is scoped to the local line and suppresses only its release fence; the rows stay bound in the solve. The rebuild at this head solves anew on policyengine-uk 2.162.12 plus #2695. A row that moves there is re-drafted from the rebuild for a reuse build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2.162.14 is the first release on PyPI that carries PolicyEngine/policyengine-uk#2695: 2.162.13's release run stopped at its YAML test job's 30-minute limit, so it never published, and PolicyEngine/policyengine-uk#2697 raised the limit. Its package code outside the tests is the head commit the previous pin installed (b71ce793), so the interim git source goes and the lock takes the registry release. This pair's local-area cut stays the build at db50ae8, which records policyengine-uk 2.162.12; its national is built at this commit on 2.162.14. The uprating and bonus-sacrifice pins keep every value, concept coverage finds no input added or removed, and the coverage manifest, the H2 spine fixture, the version mentions and the US worker identity's lock digest move with the lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merge, not rebase: the local-area and national releases record this branch's db50ae8 and daaf543 as their git commits, so those stay as they are. The only conflict was the H2 spine fixture. #1414 moved the spi_income_band_donors stage's hash through hmrc_itl_taxpayer_counts.json, and this branch moved the engine. tools/graph_uk_spine_fixture.py regenerated it on the merged tree, and --check passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@juaristi22 review of #1411 at Earlier findings. There are no earlier comments or reviews on this PR. The finding it answers is my #1405 verification run (issuecomment-6443298627, summary issuecomment-6443428518): What I checked on the draught-share entries
Engine
Findings
Verdict: ready. Nothing blocks merging. Please merge with a merge commit, as the body asks: squashing would drop db50ae8 and daaf543, which the two releases cite. Point 1 can be a follow-up issue. What I ran: a shallow clone at Not checked:
🤖 Generated with Claude Code |
A git install of an engine pull request carries the release's version string. So the spine's rules_engine provenance, and the engine identity the builds hash from it, could not tell b71ce793 from policyengine-uk 2.162.12 (review of #1411, point 1). The provenance now adds what PEP 610's direct_url.json records: a git install's commit, an archive's hash, or a directory, and never a URL. A release from an index writes no direct_url.json, so its provenance and identity are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The comment above CLOCK still named the 2026-09-23 approval and a 2026-10-15 expiry. The latest approval is now 2026-10-27, and the earliest expiry is 2026-11-16 (review of #1411, point 3). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@vahid-ahmadi thanks. All three are answered at
Locally, these passed:
CI is running on the new head. It still merges with a merge commit. |
PavelMakarchuk
left a comment
There was a problem hiding this comment.
Review of #1411 at 40a1de543. I checked the head after María's answers (8a3bec1, 40a1de5) to Vahid's review at 9a6bcce8f, then the parts his review left out: the 23 new target-fit entries, the engine provenance code and its tests, and the tooling checks.
Vahid's three items at head
- 1 (provenance): fixed for new builds.
_rules_engine_provenance()now adds PEP 610'sdirect_url.jsonrecord. The local-area release6528c865was built before the fix, so its receipt still records 2.162.12 with no commit (see Nice to have 2). - 2 (pairing): fixed. The body's "Before the next cut" says it.
- 3 (clock comment): fixed.
test_uk_battery_bindings.py:58-64now names 2026-10-27 and 2026-11-16, and no register at head has an approval later than 2026-10-27.
Must fix
None.
Should fix
- The size of the 23 new target-fit misses is not on record anywhere a reviewer can read.
- Each new entry says only that the row "breached the target-fit release bound (max_abs_relative_error 0.25)" and cites the private gate report
a859f46b…. The register test refuses a%in a reason, and the body gives counts only. - Some of these rows feed Budget measures. The three national rows are SPI savings interest by income band (£15k, £20k and £30k bands). #1418 item 11 already finds savings interest at about half of the base HMRC costed (row 62). The local cells include UC households with three or more children (E14001098), and council tax bands in Westminster, Kensington and Chelsea, Tower Hamlets and Na h-Eileanan Siar.
- All entries are scoped to the local line. So the national release and national dashboard figures are not affected. A dashboard view that reads the local-area release is.
- Fix: post the drafter's table for the 23 added rows (target, estimate, relative error) in the body or on #1301. Add the 22 kept rows' error on the rebuild
6528c865. Then a reviewer can see whether any row is far past 25%.
- Each new entry says only that the row "breached the target-fit release bound (max_abs_relative_error 0.25)" and cites the private gate report
Nice to have
_direct_installdrops the hash of an archive without sha256 (spine_build.py:765-771). PEP 610 allows any hash algorithm inarchive_info.hashes. With only, say,sha512, the record is{"kind": "archive", "sha256": null}, so two different archives share one engine identity. Fix: record the first algorithm present, for example{"kind": "archive", "hash": "sha512=…"}, or refuse. Also add tests for"unrecognised"and an editabledir_info. Neither branch has a test now.- The local-area release keeps a version-only receipt. The body says both releases "keep theirs". For
6528c865this means it still records policyengine-uk 2.162.12, with the same engine identity as a PyPI 2.162.12 install. The fix only covers builds from 8a3bec1 on. Say so in "Which commit builds what", so a reader does not think the fix covers this release. The commit is traceable only throughgit_commitdb50ae8 and its lock (and policyengine.py#725's limitation). - A second engine identity stays version-only.
country_adapter.py:160-167(build_uk_country_graph) builds its default identity frommetadata.versionalone, not from_rules_engine_provenance(). Today onlytests/engine/uk/test_uk_country_adapter.pycalls it. Call_rules_engine_provenance()there too, so the two definitions do not drift. - Merge-order note. Open PRs that touch the same pins will need them re-taken after this merges: #1410 (
uv.lock, soAPPROVED_UV_LOCK_SHA256, andspine_build.py), and #1405, #1391 and #1342 (contract.py, next to the degenerate evidence digest).
Verified as correct
- Draught-share entries. Writing the published share to each household is right. policyengine-uk 2.162.14's
beer_draught_shareandcider_draught_sharedefault to 0, so if the column is not written, all beer and cider is charged at the non-draught rate. Excluding the constant column from the degenerate gate is therefore the correct fix. The fields, the six-month window and the absence ofrelease_linesmatch the Tax-Free Childcare routed-share entry. - Register counts match the diff. National: 3 added, 5 kept, 1 retired (8 entries). Local: 20 added, 17 kept, 6 retired (37 entries). The kept entries are byte-identical to
main. Every entry hasapproved_by,adjudication,approved_on,expires_onandrelease_lines == ["local"]. 18 entries expire on 2026-11-16, 4 on 2026-11-23 and 23 on 2026-11-24. The test sets intest_uk_terminal_gates.pymatch the files. - Engine pin. All three
pyproject.tomlextras and the threerequires-distlines inuv.lockread>=2.162.14. The lock takes 2.162.14 from PyPI. No[tool.uv.sources]entry for policyengine-uk is left in anypyproject.toml. The installed 2.162.14 wheel writes nodirect_url.json, so the head's provenance is{package, version}as before. - Provenance code. No URL, path or credential reaches the record: only
vcs,commit_id, a sha256,editable, or a kind. A missing package and a missing file both return the old record. - Small code changes.
frs_take_up.py,hmrc_uprating.py,uc_capital_coherence.pyandpolicyengine_uk_concepts.pychange only the version in comments, a docstring or the concept mapping'sengine_version.worker_identity.pytakes the new lock's sha256. The new_resolvedbranch intest_engine_core_pins.pymakes a future direct reference fail with its URL, which is a clear message. - Battery clock. The latest approval in every UK register is 2026-10-27. The earliest expiry is 2026-11-16.
CLOCK = 2026-10-27is inside the window. - Head versus the built pair. The body records it clearly (the "Which commit builds what" and "Before the next cut" sections). The difference is #1414's pensioner taxpayer targets. It is not a problem for the merge, because the body says which commit built each release. A re-run of the dashboard on this pair gets the pair, not the head.
Tests run locally
At 40a1de543, in a fresh worktree, with uv sync --all-packages --locked --extra uk (policyengine-uk 2.162.14 from PyPI, Python 3.14):
pytestontest_uk_rules_engine_provenance.py,test_uk_terminal_gates.py,test_uk_battery_bindings.py,test_engine_core_pins.py,microcosm-data/.../test_contract.pyandtest_uk_frs_stochastic.py: 468 passed, 0 failed, 0 skipped.tools/build_uk_release_input_coverage_manifest.py --check: passed (144 required, 1 reviewed exclusion).tools/graph_uk_spine_fixture.py --check: passed, the fixture regenerates byte-identically (about 6 minutes).ruff checkandruff format --checkonspine_build.pyand the two touched test files: clean.- Not run: any build, certify or test that needs the private FRS data or credentials.
🤖 Generated with Claude Code
PEP 610 allows any algorithm in archive_info.hashes. With only sha512,
the record was {"kind": "archive", "sha256": null}, so two archives shared
one engine identity (Pavel's review of #1411, nice to have 1).
The record now keeps every hash the archive lists, plus the legacy hash
field. An archive that records no hash is refused. Tests cover the sha512
case, two archives taking two identities, the refusal, an editable
directory and the unrecognised record.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
build_uk_country_graph hashed its default engine identity from the version alone, so it would drift from the spine and full builds, which now hash _rules_engine_provenance() (Pavel's review of #1411, nice to have 3). It now hashes the same provenance. A release install's identity is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@PavelMakarchuk thanks. At
Locally, these passed:
CI is running on the new head. |
|
@PavelMakarchuk on Should fix 1. The per-row targets, estimates and errors stay in the private gate reports, since the estimates come from the licensed data. Reviewers on the licensed lane can read them there. Here they are as counts by error band. They are the same on the first attempt (
Every row is in the same band on both builds. |
PavelMakarchuk
left a comment
There was a problem hiding this comment.
Follow-up review of #1411 at 55d5c9fd7 (new since 40a1de543: f6fdc53, 55d5c9f, the body edits and the answer on target fit).
Must fix
None.
Should fix
- Target-fit misses: mostly answered. The banded counts are the same on
d5f72865and6528c865. All 8 national rows, including the three SPI savings interest bands, are between 0.25 and 0.5. One question is still open: the local line has 1 added row above 1 and 5 rows (1 added, 4 kept) between 0.5 and 1. The row names are public, so please name these 6 rows. Then a reader can see whether a Budget-relevant cell, such as UC households with three or more children, is among them. This can go in the body; the values can stay private.
Nice to have
None new.
Verified as correct
- Nice to have 1 (archive hash): fixed in f6fdc53.
_direct_installkeeps every hash inarchive_info.hashes, adds the legacyhashfield withsetdefault, sorts the keys, and raises a clearValueErrorwhen no hash is recorded. Two archives with different hashes now get different identities. No URL reaches the record. - Nice to have 2 (old receipt): fixed in the body. "Which commit builds what" now says that the local-area release was built before 8a3bec1 and keeps a version-only identity.
- Nice to have 3 (second identity): fixed in 55d5c9f.
build_uk_country_graphnow hashes_rules_engine_provenance(). The import inside the function avoids a cycle. A release install's identity is unchanged. - Nice to have 4 (merge order): fixed in the body. "Before the next cut" lists #1410, #1405, #1391 and #1342.
Tests run locally
At 55d5c9fd7, uv sync --all-packages --locked --extra uk:
pytestontest_uk_rules_engine_provenance.pyandtest_uk_country_adapter.py: 12 passed.ruff checkonspine_build.py,country_adapter.pyand the provenance test: clean.
🤖 Generated with Claude Code
PavelMakarchuk
left a comment
There was a problem hiding this comment.
Follow-up review of #1411 at 55d5c9fd7 (no new commits; the body now has "Local rows past 0.5").
Must fix
None.
Should fix
None. Should fix 1 is answered. The body names the six local rows past 0.5. Five are council tax band or age-structure cells, and one council tax cell in S12000013 (Na h-Eileanan Siar) is above 1. The UC three-or-more-children cell and the three SPI savings interest bands are between 0.25 and 0.5 on both builds.
Nice to have
None open.
Verified as correct
- The six rows named in the body are all in the local register with
release_lineslocal only, so the national release is not affected.
I have no open findings on this PR. Note that 18 of the target-fit sign-offs expire on 16 November, as the body says.
🤖 Generated with Claude Code
PavelMakarchuk
left a comment
There was a problem hiding this comment.
Approving at 55d5c9fd7. All findings from my reviews and Vahid's are closed. Note that 18 target-fit sign-offs expire on 16 November, so a cut certified after that date needs them renewed.
🤖 Generated with Claude Code
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The next UK national + local (dense) pair was built from this branch. Five commits made it buildable and faster, a merge of
mainbrings it up to date, and four more answer the reviews.What changes
1. d444eb5 Sign the draught shares' degenerate-surface exclusions
The excise step (#1359) writes HMRC's published draught-relief share of beer and of cider clearances to every household. So
household.beer_draught_shareandhousehold.cider_draught_shareare constant by construction, anduk_degenerate_release_surfacerefuses them on both lines: it runs in the dense build's calibrated battery and in the national release-cut scope. No release built since the step landed could pass it.solve_scope.pykeeps it in the solve fingerprint), so the release contract's degenerate evidence digest moves with it.2. 5bd7273 Build on policyengine-uk 2.162.10 plus #2695 from its head commit, and 3. 469a44c Move the engine to policyengine-uk 2.162.12 plus #2695
PolicyEngine/policyengine-uk#2695 keeps every property allowance election, but stops each comparison branch from working out every benefit for the whole population. That cost is why Microcosm UK microsimulations have run several times slower since 2.157.0 (PolicyEngine/policyengine-uk#2694). On 2.162.8 this branch's spine took about three times as long as the 2.151.0 pair's, and the dense build's pre-solve stages about twice as long.
Until a release carries the change, the workspace installs policyengine-uk from the pull request's head commit,
b71ce793. That commit merges the 2.162.12 release into the pull request, and is set through[tool.uv.sources]in the rootpyproject.toml; the second commit pinned its earlier head,ead3444don 2.162.10. The engine freeze accepts it as policyengine-uk 2.162.12: an exact version, a non-editable install and an intact RECORD.The move from 2.162.8 also brings 2.162.9 to 2.162.12: the 2010 to 2012 minimum wage history, the ESA permitted-work limit set from the national minimum wage, participation responses that impute non-workers' wages once per step (outputs unchanged), and a CI-only release.
4. db50ae8 Sign the dense build's gate-only failures and retire the stale entries
The first dense attempt (at 90ff43f, on policyengine-uk 2.162.8) failed only the degenerate surface, which the first commit fixes, and the two gate-only target-fit gates.
tools/draft_uk_dense_signoff.pydrafted its sign-off with the degenerate failure set aside, after checking that the failure named only the two draught-share columns the register now excludes. María approved it on 2026-10-27, before the rebuild, so the rebuild can pass without a reuse build.The rebuild solves anew on the new engine. A row that moves there is re-drafted from the rebuild for a reuse build.
Local rows past 0.5. These are the error bands on both builds (
d5f72865and6528c865); the values stay in the private gate reports.council_tax/S12000013/council_tax/band_f(added).council_tax/W06000018/council_tax/band_h;age_structure/E14001131/age/20_30,age_structure/E14001319/age/20_30,council_tax/E09000020/council_tax/band_bandcouncil_tax/S12000013/council_tax/band_a.Every other signed-off row, national and local, is between 0.25 and 0.5. That includes the UC households with three or more children cell (
uc_households/E14001098/uc_hh_3plus_children) and the three SPI savings interest bands.5. daaf543 Pin policyengine-uk 2.162.14 from PyPI
2.162.14 is the first release on PyPI that carries #2695. 2.162.13's release run stopped at its YAML test job's 30-minute limit, so it never published; PolicyEngine/policyengine-uk#2697 raised the limit. Outside the tests, 2.162.14's package code is
b71ce793, so the[tool.uv.sources]entry goes and the lock takes the registry release. The pins, coverage manifest and fixture move as in the third commit, with every value unchanged.Merge of
mainThe branch was brought up to date with
mainby a merge, not a rebase. So db50ae8 and daaf543, the commits the two releases record, stay as they are.spi_income_band_donorsstage's hash, throughhmrc_itl_taxpayer_counts.json;graph_uk_spine_fixture.pyregenerated the fixture on the merged tree, and--checkpasses.hmrc_uprating.pymerged cleanly: this branch changed a docstring, Bind HMRC's pensioner taxpayers by area and re-index the SPI State Pension bands (#1069) #1414 the State Pension rows.Review answers: 8a3bec1, 40a1de5, f6fdc53 and 55d5c9f
_rules_engine_provenance(), and the engine identity the builds hash from it, couldn't tellb71ce793from policyengine-uk 2.162.12.direct_url.jsonrecords: a git install's commit, an archive's hash, or a directory, and never a URL.direct_url.json, so its provenance and identity are unchanged.archive_info, so an archive hashed only with sha512 no longer records a null.build_uk_country_graphhashes_rules_engine_provenance()as the spine and full builds do.Which commit builds what
b71ce793, recording policyengine-uk 2.162.12.git_commit, db50ae8, whose lock namesb71ce793, and through policyengine.py#725's recorded limitation.Before the next cut
main; the pair doesn't.uv.lock, soAPPROVED_UV_LOCK_SHA256, andspine_build.py;contract.py, beside the degenerate evidence digest.Builds
Both releases are built and published to the inspect lane. Neither is promoted.
populace-uk-private.microcosm-uk-2024-25-dense-20261027T154955Z-6528c865.uk_high_value_homes_support, which is diagnostic.microcosm-uk-2024-25-national-20261028T061416Z-3c0d11b6.Testing
Run locally:
build_uk_release_input_coverage_manifest.py --checkandgraph_uk_spine_fixture.py --check;test_uk_rules_engine_provenance.py), with the spine, assembler, full-gates and full-build tests that read the provenance;main, the same tests again, plus the tests Bind HMRC's pensioner taxpayers by area and re-index the SPI State Pension bands (#1069) #1414 touched (uprating, ITL age targets, measurement period, national calibration, population targets, subset relations, target references), Cite US and UK spec sources by symbol or at a commit, never by a drifting line #1407's spec-source citation guard, and the tests that read the sign-off registers.The full suite runs in CI.
Merge with a merge commit, not squash or rebase: the releases built from this branch record its commits.
🤖 Generated with Claude Code