Skip to content

Add a trading allowance claim status, so allowance reforms reach claimants without turnover data - #2765

Merged
juaristi22 merged 13 commits into
mainfrom
claude/trading-allowance-claimant-2764-1294f6
Nov 3, 2026
Merged

juaristi22 merged 13 commits into
mainfrom
claude/trading-allowance-claimant-2764-1294f6

Conversation

@juaristi22

@juaristi22 juaristi22 commented Oct 31, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #2764

Survey data hold profit, not turnover, so a raise in the trading allowance missed its main base: traders who already claim £1,000 against larger receipts. The Budget 2026 raise to £3,000 (in current law since #2719) is costed on that base; HMRC's policy paper puts it at about 400,000 taxpayers with a lower bill, by about £380 on average. This PR lets a dataset say who claims, so the engine reaches that base at any allowance level.

What changes

  • Input trading_allowance_claim_status (person, enum, default UNKNOWN): whether the person claimed the allowance in the year their input profit was measured. It describes a positive profit, so a person with no profit is read as unknown, whichever status they have.
    • CLAIMANT: the profit is receipts less the allowance, as under partial relief (ITTOIA 2005 s. 783AI).
    • NON_CLAIMANT: the profit is receipts less actual expenses (capital allowances come off separately), and those deductions exceed the allowance.
  • Input allowance: the allowance legislated for the year the data were measured.
    • For microdata that is the dataset's first year, as the State Pension variables read it. For other data without years, such as a policyengine-core Dataset, it is the earliest year the profit was supplied for.
    • For anything else, such as a situation, it is the year simulated, as the State Pension variables read a situation. Nothing is inferred from which years a profit was written for, or when, so writes by a modifier, a derivative, a branch or set_input can't move it.
    • It is read from parameters.baseline, which reforms leave alone.
    • Data measured before April 2027 keep £1,000 when projected past the raise to £3,000; data measured from 2027-28 get £3,000.
  • Parameter gov.simulation.trading_allowance_claimant_expense_share (default 1, bounded to [0, 1]): claimants' expenses and capital allowances as a share of the input allowance. It sets claimants' ordinary treatment, so it matters under a cut below the input allowance and, for a claimant with a supplied current loss, at any allowance.
  • Formula in utils/trading_allowance.py, which trading_allowance_deduction, miscellaneous_income_trading_allowance_deduction and uses_trading_allowance share:
    • Claimant. Receipts are profit + input allowance, whatever receipts are supplied.
      • With no current loss, the deduction at allowance A is min(profit, max(A, expenses) − input), less their capital allowances, where their expenses are the share of the input allowance, or their capital allowances if larger, never above the input allowance. Partial relief replaces capital allowances, so the deduction gives them back, and it can be negative for a claimant.
      • So a raise relieves up to A − input (£2,000 at £3,000 on 2025 data), with full relief when profit + input is within A.
      • With the default share, a cut leaves their profit as reported.
      • Their options are the allowance and ordinary treatment, which deducts their expenses: the share of the input allowance, or their capital allowances if larger, never above the input allowance.
      • They make the same election as everyone else: the allowance (uses_trading_allowance, giving up a current loss) where it covers their expenses, unless a supplied current loss is worth more than its saving over them; ties take the allowance. The election weighs the loss against the saving in taxable trade profit and casual income, so Class 4, and income tax where a relief reads trade profit, can step where it switches, under the convention every status shares.
      • Where a deduction is an input (supplied, or set by the model) or neutralised, it applies at its own value whichever treatment they elect. So their election weighs it as the allowance's, and the other deduction follows that election: a pin to the value the formula gives changes nothing, and a positive pin means the allowance applies. Non-claimants and unknown status keep main's election.
      • Their receipts take up the allowance before income not otherwise charged.
    • Non-claimant. No election for partial relief, so ordinary treatment, except full relief, which needs no election (s. 783AE). It applies where supplied receipts plus income not otherwise charged are within the allowance. A larger current loss can still be claimed instead (s. 783AL).
    • Unknown. Unchanged: the receipts branch if self_employment_gross_receipts is supplied, otherwise the no-receipts rule. Existing datasets and situations give the same results.
    • Class 4 (ni_class_4_profits_before_losses) and relevant UK earnings read the same deduction, and their formulas are unchanged.
  • CI: .github/workflows/test-suite.yaml raises the Python shards' time limit from 75 to 90 minutes. Shards took up to 75 minutes on slower runners and were cancelled with no test failing (runs 41303444320 and 41326415023). The YAML job stays at 75 minutes; its comment now says Release each YAML case's simulation and its file's reformed systems #2742 released the cases and the job takes about 9 minutes. Editing this file also makes each push run versioning.yaml's Release test suite, roughly six more runner-hours, which isn't a required check.

Design choices for review

An enum, not a bool with a known flag. One column holds three exclusive states. With a bool and a known flag, the two columns must agree: a dataset that sets the flag but not its partner is silently ignored, and one that marks everyone known turns every unflagged trader into a non-claimant. The default UNKNOWN is what leaves existing data alone. Datasets store the enum name (CLAIMANT), which Simulation encodes like other enum columns, and the value carries forward from the year supplied.

The input allowance is a fact about the data, not a simulation-year parameter. Reading it for the data year lets one model version serve data from before and after the April 2027 raise, with no separate copy of the allowance's history.

A dataset's year is a fact about the whole file: its survey year, which the engine projects forward. A situation states no such year, so it uses the year simulated, the model's existing rule for situations (State Pension). Earlier rounds inferred a situation's year from the history of profit writes instead; that needed state, and each fix to it opened new edge cases, so this PR no longer does it.

  • A situation's profit for 2027 is read as already net of £3,000, and one for 2026 as net of £1,000.
  • Costing is unchanged by this choice: microsimulation reads a dataset, whose first year applies to every record.

Two limits remain, documented in _data_year:

  • data can't declare the year their incomes were measured, so a file saved from a later projected year is read as measured in its first year;
  • a situation's profit given for an earlier year and carried forward is read as net of the later year's allowance. A claimant's 2025-26 profit simulated in 2027-28 gains nothing from the raise, where the same record in a dataset measured in 2025 gains £2,000. Give a situation's profit for the year simulated, or use a dataset. A simulation that takes a data-built simulation's populations is read the same way.

A cut below the input allowance is set by an explicit share. Claimants' expenses are below the input allowance but unknown. Under a cut, a claimant with no current loss keeps claiming while the new allowance covers their expenses, and otherwise deducts those (ordinary treatment). So their taxable profit rises by the input allowance less the larger of the two.

A claimant with a supplied current loss claims it where it is worth more than the allowance's saving over their expenses; ties take the allowance. At the input allowance and the default share that saving is nil, so they claim it there: their claim and the loss contradict each other, and the election settles it as it does for everyone. So the share matters without a cut too: in 2026, a claimant with £30,000 of pay, a £2,500 profit and a £1,500 loss pays £3,686 at the default share and £3,886 at a share of 0.

  • The default share of 1 takes the lower bound: a cut adds nothing.
  • A share of 0 gives the upper bound for claimants with no current loss: they all keep claiming.
  • A share outside [0, 1] is taken as the nearer bound, since expenses above the allowance would contradict the claim.

Non-claimants keep full relief, but not partial relief. Partial relief needs an election (s. 783AM), and electing after a raise is behaviour, as HMRC's costings book it. Full relief needs none.

Without supplied receipts, a non-claimant's receipts are unknown but above their profit plus the input allowance. So the model shows full relief only from supplied receipts. Elsewhere, under a raise it gives none, which is a lower bound. A non-claimant with no profit is read as unknown, as a claimant is, so their casual income gets the allowance as anyone's does.

Chapter 8 miscellaneous income (#2725, #2726) is handled, not left as is. A claimant's receipts take up the allowance first (trades first, s. 783AK), and income not otherwise charged gets the rest.

  • At the input allowance that rest is nothing.
  • At £3,000 on 2026 data, a claimant with £500 of profit and £900 of casual income has both relieved in full.

A non-claimant gets it only through full relief.

For the data side (PolicyEngine/microcosm#1439)

  • Flag only profits net of the allowance. A profit at or below £1,000 flagged CLAIMANT is read as receipts above £1,000 and taxed at baseline, where UNKNOWN relieves it in full. Traders whose receipts were within £1,000 belong at UNKNOWN.
  • The stored receipts can go. With the flag, receipts of profit + £1,000 or + £3,000 are no longer needed: a claimant's status overrides self_employment_gross_receipts, and the input allowance is nominal, so the gap no longer grows with mixed-income uprating.
  • Data year. The input allowance comes from the dataset's first year, so a 2024-25 or 2025-26 base reads £1,000. A file saved from a later projected year would be read as measured then, so save releases from their base year.
  • Calibration. HMRC's figure of about 400,000 gainers may include expense claimers who switch to the allowance, which the costing counts as behaviour. If it does, calibrating the static claimant share to it would overstate claimants; the costing note doesn't say.
  • Don't pin the deduction. A dataset that pins trading_allowance_deduction to zero, as policyengine-uk-data does on SPI records, bypasses the formula and blocks every allowance reform. A flagged claimant's election weighs a pinned deduction at its value, so their treatment stays consistent, but the raise can't reach them. Flagging claimants and non-claimants, without the pin, avoids that.

Tests

policyengine_uk/tests/policy/baseline/gov/hmrc/income_tax/allowances/trading_allowance_claim_status.yaml (57 cases):

  • Claimants:
    • at the input allowance, in 2026 and 2027;
    • under raises to £3,000 and £5,000 (partial and full relief), and £3,000 to £5,000 in 2027;
    • under a cut to £500 and abolition, each beside an unknown-status person so they can't pass without the reform;
    • under cuts at expense shares of 0 and 0.5, and with a share of 1.5 taken as 1;
    • with capital allowances, at £1,000 and under a raise;
    • a claimant's status overriding supplied receipts;
    • claimants with no profit;
    • capital allowances above the share's expenses, which count as expenses;
    • an allowance equal to their expenses to the penny, or to their capital allowances, which keeps the allowance;
    • losses brought forward that absorb both options: they don't keep a claimant on an allowance below their expenses, and they leave a covered claimant on the allowance.
  • Non-claimants:
    • with no receipts, under a raise, and with low-expense receipts above the allowance;
    • with receipts within it, alone and with casual income;
    • with relevant income exactly at it, and above it;
    • with no profit, read as unknown: casual income within, exactly at and above the allowance.
  • Unknown status, with and without receipts.
  • Class 4 and income tax.
  • The current-loss election:
    • a claimant at £1,000 (who claims the loss), under a raise that beats a smaller loss and one that doesn't, under a cut below their expenses and under abolition;
    • at a share of 0.5: allowances of £499 and £500, and the switch at £2,000, with no step;
    • at a share of 0 and today's allowance, where the loss is claimed on more profit;
    • a non-claimant, including electing out of full relief.
  • Chapter 8 income for claimants, including a claimant who gives up a loss for the allowance's relief on casual income.
  • Pinned deductions, which the claimant's election weighs at their values:
    • on trades, under a raise to £3,000, where the loss is still claimed;
    • on casual income, where the claimant takes ordinary treatment throughout (£3,626);
    • on trades with casual income under £3,000, where the loss is still claimed and the raise changes nothing;
    • pinned to the value the formula gives, which changes nothing;
    • pinned positive, which means the allowance applies.

policyengine_uk/tests/test_trading_allowance_claim_status_properties.py (one hypothesis test, 30 people per draw, 15 scenarios of allowance and expense share, one at a share of 1.5, with every invariant checked on one run of each scenario):

  • Every status matches a reference written from the Act's options (full, partial, ordinary), with every election, a claimant's included, weighing a supplied current loss against the allowance's saving over the person's own ordinary treatment. It covers taxable profit, Class 4 profits, the Chapter 8 deduction and uses_trading_allowance, with Chapter 8 and miscellaneous income drawn separately.
  • A claimant's deductions never exceed the allowance's excess over £1,000.
  • Neither taxable profit nor Class 4 profits ever rise with the allowance.
  • A smaller expense share never lowers the taxable profit of a claimant with no supplied current loss. With one, a smaller share can move them from the loss to the allowance, which lowers that profit; the YAML cases check their income tax.

policyengine_uk/tests/test_trading_allowance_claim_status_data_year.py (18 tests on small datasets and situations):

  • data measured in 2025 get the 2027 raise, data measured in 2027 gain nothing from it, and a reform of the data year leaves the input allowance;
  • a raise never raises a claimant's tax when they have a loss (dataset), and a carried loss doesn't enter the election (situation);
  • a policyengine-core simulation over data reads the profit's first year, and a household subset of data reads the dataset's first year;
  • a situation reads its profit as measured in the year simulated, including a carried profit (the documented limit), and a situation that takes a dataset's populations reads the year simulated (the other documented limit);
  • a situation's data year doesn't depend on how its profit was given: in the dictionary, by set_input in any order, or for two people in turn;
  • derivatives (this package's and policyengine-core's, before or after a first calculation), a branch given the profit and a modifier that writes it back leave the data year;
  • the status carries forward;
  • a neutralised deduction, and a deduction the model sets (set_model_value), each keep a raise from costing a claimant their loss;
  • pinning either deduction, or both, to the values their formulas give, by set_input or set_model_value, changes no status's election, deductions, Class 4 or income tax, over a grid of 216 people at three allowances and shares.

Commands run in the locked environment (uv run --frozen --extra dev):

  • policyengine-core test on the claim-status YAML at edda47ea2: 57 passed. On all of policyengine_uk/tests/policy at 870b07f3b: 5,466 passed. Since then only the claim-status file and the non-claimant gate changed, and no other YAML file sets the status.
  • pytest at edda47ea2 on the data-year and property files, the parameter and variable metadata tests, the tests that scan the package (test_household_subset.py's review tests, test_warning_filters.py) and all of policyengine_uk/tests/code_health: 3,176 passed.
  • ruff format and ruff check: clean. .github/check_changelog_fragments.py: OK.

Not run locally: the full suite, left to CI, and the microsimulation tests, which need the dataset token.

axiom: n/a: adds an input that records how the data's profits were measured and a gov.simulation convention parameter; the Part 6A rules (ss. 783AE, 783AF, 783AI) are unchanged

🤖 Generated with Claude Code

Add trading_allowance_claim_status, a person enum (UNKNOWN, CLAIMANT,
NON_CLAIMANT; default UNKNOWN), and
gov.simulation.trading_allowance_in_input_profits, the allowance a
claimant's input profit already deducts: £1,000 from 2017-18, not moved
by reforms to the trading allowance and not uprated.

A claimant's receipts are their profit plus that amount, so the deduction
at allowance A is min(profit, A - £1,000) less capital allowances, floored
at nil, and a cut below £1,000 leaves their profit as reported. A
non-claimant gets no allowance at any level, on trade receipts or on
income not otherwise charged. Unknown keeps the current rules, so
existing datasets are unchanged. Class 4 and the current-loss election
read the same deduction.

Issue #2764.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Aggregate impact on certified data: pass

Base 2da212abcf against head e1bed2b8ee, both on enhanced_frs_2024_25.h5 (policyengine-uk-data 1.58.1, sha256 056e80fd8e91), fiscal years 2025-26 to 2031-32. Each year is a full microsimulation run on each side.

Checks: 0 broken by this change, 1 failing on both sides, 0 fixed. Moves flagged for review: 0 (thresholds in report_spec.yaml; a flag does not fail the job).

Workflow run and full report

Notes

  • In the years a figure from fewer than 10 records is suppressed (spending.disabled_students_allowance), every total containing it, and the decile incomes, are rebuilt from their other components, so no published total includes it. Those totals are marked †. A change is withheld where the two sides leave out different figures and can't be lined up.

Moves flagged for review

None.

Spending by benefit and service

Measure 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Disabled Students' Allowance suppressed suppressed suppressed suppressed suppressed suppressed suppressed

Caseloads

Measure 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Disabled Students' Allowance suppressed suppressed suppressed suppressed suppressed suppressed suppressed

Standard reform costings (change in gov_balance)

Measure 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Raise Class 1 main employee NICs rate to 10% 0 0 0 0 0 suppressed suppressed
Raise additional rate by 3pp 0 0 0 0 0 suppressed suppressed

Standard reforms: change in each reform's target

Measure 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Raise Class 1 main employee NICs rate to 10%: change in national_insurance 0 0 0 0 0 suppressed suppressed
Raise additional rate by 3pp: change in income_tax 0 0 0 0 0 suppressed suppressed

Household net income by decile

No household's net income changes in any year.

No change in any year: receipts by tax head; government balance, each component counted once; model totals; household income totals; population; poverty rates; people in poverty; inequality; mean household net income in each side's own deciles.

Checks failing on both base and head (1)
  • VAT excluding private school fees is never negative (sign:vat): 2025-26: 76 units below zero, a weighted £-53,888,684 in total; 2026-27: 76 units below zero, a weighted £-55,777,260 in total; 2027-28: 76 units below zero, a weighted £-57,727,917 in total
Counted-once check

Each of 92 components was raised by £1 per unit in 2026-27, and every head of its totals was read household by household: its own head must rise by exactly £1, no other tax head may move, no other benefit may rise, and the model's own total must move by what its heads moved. A head that breaks a rule is measured again twice: with every other head held at its baseline value, and with every other head held at its value after the first £1 while the component rises again. It fails where either still shows the response. 0 failed; 0 were skipped.

  • widowed_parents_allowance: pension_credit responds only through other heads in fewer than 10 households (other heads that moved there: carers_allowance; gov_spending; not a failure)
  • widowed_parents_allowance: pension_credit responds only through other heads in fewer than 10 households (other heads that moved there: carers_allowance; household_benefits; not a failure)
Provenance
Side Commit Runtime Environment hash Partition hash Time
base main 2da212abcf uk 2.170.0 (files e654eb5f6f55), core 3.33.12, py 3.13.18 aarch64 77aac721ac5e 7cb61324a46b 2037.05s
head #2765 merged into main e1bed2b8ee uk 2.170.0 (files 627f8abbe5b8), core 3.33.12, py 3.13.18 aarch64 77aac721ac5e 7cb61324a46b 2056.84s

Dataset: hf://policyengine/policyengine-uk-data-private/enhanced_frs_2024_25.h5@2526edbbb687a3b54dbb11272e5389e4b62d2ac2, sha256 056e80fd8e9151d78f432a25cde863340baee81304c6216a37fb575ce8563784 (verified on both sides), policyengine-uk-data 1.58.1.

Receipts by tax head: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Final charge on cash interest in non-cash ISAs £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Lifetime ISA withdrawal charge £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Income tax £294.46bn £315.23bn £335.83bn £351.28bn £366.96bn £385.36bn £400.00bn
National Insurance (employee and self-employed) £51.70bn £54.14bn £56.13bn £57.73bn £59.76bn £61.64bn £63.51bn
National Insurance (employer) £148.71bn £155.37bn £160.88bn £165.09bn £172.06bn £177.36bn £182.57bn
Employer NI change borne by capital £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Employer NI change borne by consumers £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Capital gains tax £12.60bn £13.15bn £13.60bn £14.11bn £14.67bn £15.24bn £15.80bn
Student loan repayments £7.82bn £8.13bn £8.39bn £8.58bn £8.79bn £9.04bn £9.29bn
VAT excluding private school fees £249.37bn £256.87bn £266.33bn £272.70bn £278.97bn £285.24bn £291.59bn
Change in VAT from its baseline £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
VAT on private school fees £1.95bn £2.01bn £2.09bn £2.14bn £2.20bn £2.26bn £2.31bn
Change in private school VAT from its baseline £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Fuel duties £24.71bn £24.13bn £26.00bn £26.83bn £26.51bn £25.78bn £24.63bn
Alcohol duties £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Tobacco duties £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Vehicle excise duties £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Betting and gaming duties £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
TV licence fee £4.26bn £4.40bn £4.41bn £4.42bn £4.43bn £4.44bn £4.45bn
Council tax £49.21bn £52.16bn £54.89bn £57.95bn £61.16bn £64.47bn £67.91bn
High value council tax surcharge £0.00bn £0.00bn £0.00bn £0.48bn £0.49bn £0.50bn £0.51bn
Northern Ireland domestic rates £0.55bn £0.58bn £0.60bn £0.63bn £0.66bn £0.69bn £0.72bn
Business rates £34.68bn £36.20bn £38.00bn £39.46bn £42.27bn £43.25bn £44.29bn
Stamp duty land tax on the household's own purchases £6.88bn £7.32bn £7.66bn £8.06bn £8.47bn £8.91bn £9.35bn
Stamp duty land tax paid by companies £4.72bn £4.57bn £5.29bn £6.03bn £6.37bn £6.92bn £7.45bn
Land transaction tax (Wales) £0.07bn £0.08bn £0.08bn £0.09bn £0.10bn £0.10bn £0.11bn
Land and buildings transaction tax (Scotland) £0.31bn £0.32bn £0.33bn £0.35bn £0.36bn £0.38bn £0.40bn
Unspecified tax change borne by capital £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Unspecified tax change borne by consumers £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Wealth tax £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Wealth tax excluding the main residence £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Land value tax £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Carbon tax £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Spending by benefit and service: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Lifetime ISA bonus £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
State Pension £121.68bn £125.92bn £126.73bn £129.50bn £132.99bn £136.66bn £140.43bn
Pension Credit £6.66bn £7.03bn £7.29bn £7.48bn £7.60bn £7.84bn £8.03bn
Winter Fuel Payment £1.86bn £1.85bn £1.79bn £1.77bn £1.78bn £1.78bn £1.78bn
Pension Age Winter Heating Payment (Scotland) £0.19bn £0.20bn £0.20bn £0.21bn £0.21bn £0.22bn £0.22bn
Universal Credit £75.48bn £80.52bn £82.95bn £84.96bn £86.11bn £87.61bn £88.21bn
Housing Benefit £8.56bn £8.89bn £9.23bn £9.46bn £9.69bn £9.99bn £10.27bn
Council Tax Reduction £2.07bn £2.20bn £2.31bn £2.45bn £2.58bn £2.72bn £2.86bn
Employment and Support Allowance (income-related) £0.79bn £0.82bn £0.84bn £0.86bn £0.88bn £0.90bn £0.92bn
Employment and Support Allowance (contribution-based) £5.20bn £5.37bn £5.56bn £5.69bn £5.82bn £5.95bn £6.09bn
Jobseeker's Allowance (income-based) £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Jobseeker's Allowance (contribution-based) £0.29bn £0.30bn £0.31bn £0.31bn £0.32bn £0.33bn £0.34bn
Income Support £0.11bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Working Tax Credit £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Child Tax Credit £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Incapacity Benefit £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Personal Independence Payment £26.95bn £28.03bn £29.05bn £30.09bn £30.75bn £31.51bn £32.21bn
Disability Living Allowance £7.49bn £7.79bn £8.07bn £8.36bn £8.55bn £8.76bn £8.95bn
Attendance Allowance £8.68bn £9.03bn £9.36bn £9.69bn £9.90bn £10.15bn £10.38bn
Carer's Allowance £4.48bn £4.66bn £4.83bn £5.00bn £5.11bn £5.27bn £5.42bn
Carer Support Payment (Scotland) £0.31bn £0.32bn £0.33bn £0.34bn £0.35bn £0.36bn £0.37bn
Carer's Allowance Supplement (Scotland) £0.00bn £0.04bn £0.04bn £0.05bn £0.05bn £0.05bn £0.05bn
Severe Disablement Allowance £0.03bn £0.03bn £0.03bn £0.03bn £0.03bn £0.03bn £0.03bn
Industrial Injuries Disablement Benefit £0.50bn £0.52bn £0.54bn £0.55bn £0.56bn £0.58bn £0.59bn
Armed Forces Compensation Scheme £0.61bn £0.63bn £0.65bn £0.66bn £0.68bn £0.69bn £0.71bn
War Widow's or Widower's Pension £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Child Benefit £16.19bn £16.82bn £17.39bn £18.00bn £18.39bn £18.84bn £19.25bn
Scottish Child Payment £0.49bn £0.52bn £0.53bn £0.55bn £0.56bn £0.57bn £0.56bn
Two Child Limit Payment (Scotland, counterfactual) £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Healthy Start vouchers £0.05bn £0.05bn £0.05bn £0.05bn £0.05bn £0.05bn £0.05bn
Maternity Allowance £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Statutory Maternity Pay £3.30bn £3.41bn £3.54bn £3.62bn £3.70bn £3.78bn £3.87bn
Statutory Paternity Pay £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Statutory Sick Pay £1.12bn £1.16bn £1.20bn £1.23bn £1.26bn £1.29bn £1.32bn
Sure Start Maternity Grant £0.01bn £0.01bn £0.01bn £0.01bn £0.01bn £0.01bn £0.01bn
Bereavement Support Payment £0.67bn £0.69bn £0.71bn £0.73bn £0.75bn £0.76bn £0.78bn
Widowed Parent's Allowance £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Tax-Free Childcare £0.56bn £0.57bn £0.57bn £0.58bn £0.60bn £0.61bn £0.59bn
Extended childcare entitlement £2.99bn £3.50bn £3.59bn £3.71bn £3.78bn £3.86bn £3.93bn
Universal childcare entitlement £2.06bn £2.22bn £2.29bn £2.37bn £2.43bn £2.48bn £2.53bn
Targeted childcare entitlement £0.50bn £0.51bn £0.52bn £0.54bn £0.55bn £0.57bn £0.56bn
Childcare Offer for Wales £0.26bn £0.28bn £0.28bn £0.29bn £0.30bn £0.31bn £0.32bn
Care to Learn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Childcare Grant £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Parents' Learning Allowance £0.20bn £0.20bn £0.20bn £0.20bn £0.20bn £0.20bn £0.20bn
Adult Dependants' Grant £0.06bn £0.06bn £0.04bn £0.04bn £0.04bn £0.04bn £0.04bn
Travel Grant £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Disabled Students' Allowance suppressed suppressed suppressed suppressed suppressed suppressed suppressed
16 to 19 Bursary Fund £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
NHS spending £203.89bn £204.30bn £204.56bn £204.91bn £205.34bn £205.85bn £206.35bn
State education spending £0.02bn £0.02bn £0.02bn £0.02bn £0.02bn £0.02bn £0.02bn
Transport subsidy spending £25.05bn £25.10bn £25.14bn £25.18bn £25.23bn £25.30bn £25.36bn
Cost-of-living support payment £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Energy Bills Rebate £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Household Energy Payment £0.00bn £2.11bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Energy Price Guarantee subsidy £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Basic income £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Disability basic income £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Other public spending budget change £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Caseloads: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Lifetime ISA bonus 0k 0k 0k 0k 0k 0k 0k
State Pension 11,264k 11,145k 10,819k 10,700k 10,723k 10,750k 10,776k
Pension Credit 1,486k 1,480k 1,472k 1,452k 1,450k 1,455k 1,464k
Winter Fuel Payment 7,844k 7,783k 7,473k 7,375k 7,391k 7,409k 7,427k
Pension Age Winter Heating Payment (Scotland) 803k 802k 800k 797k 799k 801k 802k
Universal Credit 6,342k 6,347k 6,336k 6,297k 6,292k 6,271k 6,223k
Housing Benefit 1,249k 1,245k 1,244k 1,227k 1,227k 1,229k 1,232k
Council Tax Reduction 3,176k 3,195k 3,200k 3,214k 3,224k 3,235k 3,242k
Employment and Support Allowance (income-related) 71k 71k 71k 71k 71k 71k 71k
Employment and Support Allowance (contribution-based) 719k 720k 721k 722k 724k 726k 728k
Jobseeker's Allowance (income-based) 0k 0k 0k 0k 0k 0k 0k
Jobseeker's Allowance (contribution-based) 67k 67k 67k 67k 67k 68k 68k
Income Support 25k 0k 0k 0k 0k 0k 0k
Working Tax Credit 0k 0k 0k 0k 0k 0k 0k
Child Tax Credit 0k 0k 0k 0k 0k 0k 0k
Incapacity Benefit 0k 0k 0k 0k 0k 0k 0k
Personal Independence Payment 3,567k 3,575k 3,579k 3,585k 3,593k 3,602k 3,611k
Disability Living Allowance 1,291k 1,294k 1,296k 1,298k 1,300k 1,304k 1,307k
Attendance Allowance 1,695k 1,698k 1,700k 1,703k 1,707k 1,711k 1,715k
Carer's Allowance 1,034k 1,037k 1,038k 1,040k 1,042k 1,060k 1,062k
Carer Support Payment (Scotland) 71k 71k 71k 72k 72k 72k 72k
Carer's Allowance Supplement (Scotland) 0k 71k 71k 72k 72k 72k 72k
Severe Disablement Allowance 4k 4k 4k 4k 4k 4k 4k
Industrial Injuries Disablement Benefit 137k 137k 137k 138k 138k 138k 139k
Armed Forces Compensation Scheme 93k 93k 94k 94k 94k 94k 94k
War Widow's or Widower's Pension 0k 0k 0k 0k 0k 0k 0k
Child Benefit 8,123k 8,120k 8,120k 8,130k 8,145k 8,161k 8,178k
Scottish Child Payment 351k 352k 352k 353k 350k 348k 337k
Two Child Limit Payment (Scotland, counterfactual) 0k 0k 0k 0k 0k 0k 0k
Healthy Start vouchers 200k 200k 200k 201k 201k 201k 202k
Maternity Allowance 0k 0k 0k 0k 0k 0k 0k
Statutory Maternity Pay 333k 333k 334k 334k 335k 336k 337k
Statutory Paternity Pay 0k 0k 0k 0k 0k 0k 0k
Statutory Sick Pay 181k 182k 182k 182k 182k 183k 183k
Sure Start Maternity Grant 26k 26k 26k 26k 26k 26k 26k
Bereavement Support Payment 98k 98k 98k 98k 98k 99k 99k
Widowed Parent's Allowance 0k 0k 0k 0k 0k 0k 0k
Tax-Free Childcare 1,043k 1,027k 1,014k 1,009k 1,011k 1,009k 993k
Extended childcare entitlement 795k 800k 796k 797k 798k 798k 792k
Universal childcare entitlement 1,071k 1,072k 1,072k 1,073k 1,076k 1,079k 1,074k
Targeted childcare entitlement 165k 156k 154k 154k 155k 155k 150k
Childcare Offer for Wales 39k 39k 39k 39k 39k 39k 39k
Care to Learn 0k 0k 0k 0k 0k 0k 0k
Childcare Grant 0k 0k 0k 0k 0k 0k 0k
Parents' Learning Allowance 98k 98k 98k 99k 99k 99k 99k
Adult Dependants' Grant 18k 18k 12k 12k 12k 12k 12k
Travel Grant 0k 0k 0k 0k 0k 0k 0k
Disabled Students' Allowance suppressed suppressed suppressed suppressed suppressed suppressed suppressed
16 to 19 Bursary Fund 0k 0k 0k 0k 0k 0k 0k
Cost-of-living support payment 0k 0k 0k 0k 0k 0k 0k
Energy Bills Rebate 0k 0k 0k 0k 0k 0k 0k
Household Energy Payment 0k 21,065k 0k 0k 0k 0k 0k
Energy Price Guarantee subsidy 0k 0k 0k 0k 0k 0k 0k
Basic income 0k 0k 0k 0k 0k 0k 0k
Disability basic income 0k 0k 0k 0k 0k 0k 0k
Government balance, each component counted once: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Government receipts, each component once £892.00bn £934.66bn £980.50bn £1,015.93bn £1,054.24bn £1,091.56bn £1,124.90bn
Government spending £529.38bn † £545.66bn † £550.77bn † £559.52bn † £567.20bn † £575.96bn † £583.61bn †
Government balance, each receipt once £362.62bn † £389.00bn † £429.73bn † £456.41bn † £487.04bn † £515.61bn † £541.29bn †
Benefits in household_benefits but not in gov_spending £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn £0.00bn
Model totals: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
gov_tax £892.00bn £934.66bn £980.50bn £1,015.93bn £1,054.24bn £1,091.56bn £1,124.90bn
household_tax £491.97bn £520.41bn £551.21bn £576.01bn £601.00bn £626.70bn £648.42bn
pre_budget_change_household_tax £484.15bn £512.28bn £542.82bn £566.95bn £591.72bn £617.16bn £638.62bn
gov_spending £529.38bn † £545.66bn † £550.77bn † £559.52bn † £567.20bn † £575.96bn † £583.61bn †
household_benefits £529.38bn † £545.66bn † £550.77bn † £559.52bn † £567.20bn † £575.96bn † £583.61bn †
gov_balance £362.62bn † £389.00bn † £429.73bn † £456.41bn † £487.04bn † £515.61bn † £541.29bn †
Household income totals: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Household market income £1,637.96bn £1,702.77bn £1,757.35bn £1,806.41bn £1,855.97bn £1,912.43bn £1,970.44bn
Household net income £1,636.69bn † £1,687.82bn † £1,715.46bn † £1,747.50bn † £1,778.70bn † £1,817.02bn † £1,859.69bn †
Household net income (HBAI definition) £1,449.57bn £1,499.83bn £1,530.87bn £1,565.18bn £1,597.89bn £1,635.60bn £1,676.62bn
Population: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
People 69,245k 69,383k 69,473k 69,592k 69,738k 69,912k 70,080k
Benefit units 35,110k 35,181k 35,226k 35,286k 35,360k 35,449k 35,534k
Households 31,046k 31,108k 31,149k 31,201k 31,267k 31,345k 31,420k
Poverty rates: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Absolute poverty, before housing costs: everyone 16.3% 15.5% 16.4% 16.3% 16.5% 16.5% 16.6%
Absolute poverty, before housing costs: children 24.7% 22.1% 23.3% 22.9% 23.1% 23.2% 23.5%
Absolute poverty, before housing costs: working-age adults 14.1% 13.8% 14.7% 14.7% 14.8% 14.8% 15.0%
Absolute poverty, before housing costs: pensioners 13.7% 13.4% 14.0% 13.8% 13.9% 13.9% 13.9%
Absolute poverty, after housing costs: everyone 21.6% 21.0% 22.0% 22.6% 22.9% 23.1% 23.4%
Absolute poverty, after housing costs: children 33.3% 31.3% 32.3% 32.5% 32.9% 33.4% 34.2%
Absolute poverty, after housing costs: working-age adults 19.9% 19.8% 20.8% 21.6% 21.9% 22.0% 22.3%
Absolute poverty, after housing costs: pensioners 12.7% 12.4% 13.2% 13.2% 13.3% 13.4% 13.6%
Relative poverty, before housing costs: everyone 17.0% 16.0% 16.4% 16.3% 16.4% 16.3% 16.6%
Relative poverty, before housing costs: children 25.9% 22.9% 23.3% 22.9% 23.0% 22.9% 23.5%
Relative poverty, before housing costs: working-age adults 14.5% 14.1% 14.6% 14.7% 14.7% 14.7% 15.0%
Relative poverty, before housing costs: pensioners 14.5% 14.0% 13.9% 13.8% 13.8% 13.9% 13.9%
Relative poverty, after housing costs: everyone 21.4% 21.0% 21.3% 21.3% 21.3% 21.5% 21.7%
Relative poverty, after housing costs: children 33.1% 31.1% 31.2% 31.0% 31.1% 31.2% 31.8%
Relative poverty, after housing costs: working-age adults 19.9% 19.8% 20.2% 20.4% 20.5% 20.6% 20.8%
Relative poverty, after housing costs: pensioners 12.5% 12.4% 12.2% 11.7% 11.7% 11.8% 12.0%
People in poverty: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Absolute poverty, before housing costs: everyone 11,278k 10,779k 11,402k 11,365k 11,475k 11,513k 11,651k
Absolute poverty, before housing costs: children 3,688k 3,314k 3,496k 3,445k 3,480k 3,502k 3,549k
Absolute poverty, before housing costs: working-age adults 5,965k 5,893k 6,308k 6,362k 6,420k 6,430k 6,521k
Absolute poverty, before housing costs: pensioners 1,624k 1,572k 1,598k 1,559k 1,575k 1,580k 1,581k
Absolute poverty, after housing costs: everyone 14,944k 14,600k 15,310k 15,721k 15,936k 16,141k 16,426k
Absolute poverty, after housing costs: children 4,975k 4,682k 4,837k 4,888k 4,945k 5,040k 5,179k
Absolute poverty, after housing costs: working-age adults 8,463k 8,461k 8,971k 9,344k 9,491k 9,584k 9,706k
Absolute poverty, after housing costs: pensioners 1,506k 1,457k 1,502k 1,490k 1,500k 1,517k 1,542k
Relative poverty, before housing costs: everyone 11,750k 11,084k 11,388k 11,364k 11,416k 11,415k 11,651k
Relative poverty, before housing costs: children 3,872k 3,423k 3,496k 3,445k 3,466k 3,454k 3,549k
Relative poverty, before housing costs: working-age adults 6,159k 6,013k 6,301k 6,362k 6,389k 6,389k 6,521k
Relative poverty, before housing costs: pensioners 1,720k 1,648k 1,590k 1,557k 1,562k 1,572k 1,581k
Relative poverty, after housing costs: everyone 14,849k 14,552k 14,775k 14,809k 14,874k 15,014k 15,234k
Relative poverty, after housing costs: children 4,942k 4,660k 4,673k 4,663k 4,679k 4,707k 4,806k
Relative poverty, after housing costs: working-age adults 8,426k 8,440k 8,711k 8,823k 8,873k 8,967k 9,062k
Relative poverty, after housing costs: pensioners 1,482k 1,452k 1,391k 1,323k 1,322k 1,339k 1,365k
Inequality: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Gini coefficient, equivalised HBAI income before housing costs 0.3080 0.3052 0.3058 0.3058 0.3063 0.3069 0.3085
Mean household net income in each side's own deciles: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Decile 1: mean household_net_income per household £17,681 † £17,997 † £17,915 † £18,152 † £18,307 † £18,531 † £18,717 †
Decile 2: mean household_net_income per household £27,939 † £29,023 † £28,711 † £29,013 † £29,276 † £29,691 † £30,151 †
Decile 3: mean household_net_income per household £34,320 † £35,485 † £36,073 † £36,611 † £37,197 † £37,737 † £38,187 †
Decile 4: mean household_net_income per household £38,933 † £40,687 † £41,724 † £42,727 † £43,334 † £44,132 † £44,434 †
Decile 5: mean household_net_income per household £43,239 † £45,567 † £46,012 † £46,960 † £47,714 † £48,584 † £50,279 †
Decile 6: mean household_net_income per household £47,927 † £49,210 † £50,893 † £51,939 † £52,942 † £53,942 † £54,680 †
Decile 7: mean household_net_income per household £55,662 † £56,571 † £57,442 † £57,670 † £58,730 † £59,651 † £61,082 †
Decile 8: mean household_net_income per household £68,789 † £70,803 † £71,858 † £73,466 † £73,880 † £75,317 † £77,178 †
Decile 9: mean household_net_income per household £78,890 † £81,013 † £80,861 † £82,646 † £85,464 † £86,946 † £89,509 †
Decile 10: mean household_net_income per household £117,005 † £120,339 † £124,742 † £126,632 † £125,971 † £128,600 † £130,938 †
Standard reform costings (change in gov_balance): head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Raise basic rate by 1pp £6.41bn £6.66bn £6.63bn £6.85bn £7.03bn £7.25bn £7.49bn
Raise higher rate by 1pp £3.61bn £3.87bn £4.00bn £4.22bn £4.44bn £4.71bn £4.83bn
Raise personal allowance by ~800GBP/year £-3.80bn £-3.92bn £-3.98bn £-4.04bn £-4.10bn £-4.23bn £-1.95bn
Raise child benefit by 25GBP/week per additional child £-1.99bn £-1.75bn £-1.55bn £-1.38bn £-1.30bn £-1.18bn £-1.07bn
Reduce Universal Credit taper rate to 20% £-20.77bn £-21.32bn £-21.75bn £-21.99bn £-22.34bn £-22.75bn £-22.94bn
Raise Class 1 main employee NICs rate to 10% £10.89bn £11.30bn £11.71bn £12.01bn £12.31bn suppressed suppressed
Raise VAT standard rate by 2pp £24.89bn £25.71bn £26.63bn £27.24bn £27.87bn £28.50bn £29.14bn
Raise additional rate by 3pp £2.90bn £3.15bn £3.22bn £3.41bn £3.61bn suppressed suppressed
Raise the Universal Credit standard allowance by 10% £-3.06bn £-3.18bn £-3.26bn £-3.35bn £-3.44bn £-3.49bn £-3.50bn
Raise the State Pension by 10% £-7.78bn £-8.01bn £-8.24bn £-8.47bn £-8.82bn £-8.55bn £-8.89bn
Standard reforms: change in each reform's target: head levels
Measure (head) 2025-26 2026-27 2027-28 2028-29 2029-30 2030-31 2031-32
Raise basic rate by 1pp: change in income_tax £6.55bn £6.86bn £6.79bn £7.00bn £7.20bn £7.44bn £7.66bn
Raise higher rate by 1pp: change in income_tax £3.61bn £3.92bn £4.01bn £4.23bn £4.45bn £4.72bn £4.84bn
Raise personal allowance by ~800GBP/year: change in income_tax £-3.93bn £-4.07bn £-4.13bn £-4.18bn £-4.22bn £-4.40bn £-1.75bn
Raise child benefit by 25GBP/week per additional child: change in child_benefit £2.33bn £2.14bn £1.94bn £1.76bn £1.65bn £1.52bn £1.40bn
Reduce Universal Credit taper rate to 20%: change in universal_credit £20.74bn £21.31bn £21.78bn £22.04bn £22.37bn £22.78bn £22.99bn
Raise Class 1 main employee NICs rate to 10%: change in national_insurance £11.15bn £11.63bn £12.00bn £12.29bn £12.61bn suppressed suppressed
Raise VAT standard rate by 2pp: change in vat £24.69bn £25.51bn £26.42bn £27.03bn £27.65bn £28.28bn £28.91bn
Raise additional rate by 3pp: change in income_tax £2.90bn £3.15bn £3.22bn £3.42bn £3.62bn suppressed suppressed
Raise the Universal Credit standard allowance by 10%: change in universal_credit £3.06bn £3.19bn £3.28bn £3.37bn £3.43bn £3.49bn £3.51bn
Raise the State Pension by 10%: change in state_pension £10.68bn £11.17bn £11.28bn £11.60bn £12.04bn £12.47bn £12.97bn

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Independent review of #2765 at 5520b59b5

This is an independent adversarial review. Three reviewers read the PR from different angles:

  • statute and a line-by-line pass;
  • callers and Python/numpy pitfalls;
  • cleanup and conventions.

A verifier then reproduced each finding with synthetic Simulation runs and re-read ITTOIA 2005 Part 6A, and a last pass looked for gaps. No dataset sets the claim status yet, so nothing below moves existing results.

Verdict: fix 1 before the data side adopts the flag. The property test can't catch it (3). The rest are edge cases, one parameter-design point, and texts.

Statute

1. A NON_CLAIMANT status blocks full relief, which needs no election.

  • utils/trading_allowance.py:117 and :131-135 zero both reliefs for every non-claimant. But full relief is automatic when relevant income is within the allowance (ss. 783AE(1), 783AF, 783AG). Only partial relief is elective (ss. 783AH(c), 783AM).
  • 2027 (allowance £3,000), with employment of £30,000:
    • A non-claimant with profit £600 and supplied receipts of £2,800 is taxed on the £600 (income tax £3,606). UNKNOWN with the same inputs gives nil (£3,486), as the Act does.
    • With receipts of £2,000 plus £900 of income not otherwise charged, both amounts are taxed instead of nil.
  • 2026, at today's £1,000: a non-claimant with no trade and £800 of income not otherwise charged is taxed on the £800 (£3,646 against £3,486), although £800 is within £1,000.
  • The "receipts unknown" reasoning doesn't apply when receipts are supplied. The YAML precedence case (line 192) uses receipts of £4,000, which is above the allowance.
  • Suggestion: let supplied receipts within the allowance, and a no-trade person's casual income, reach full relief whatever the status. Or say in the input's doc that NON_CLAIMANT is only for traders whose receipts exceed the allowance.

2. A claimant's capital allowances are still deducted (:113).

  • Partial relief replaces both expenses and capital allowances (s. 783AI; HMRC BIM86050).
  • A 2026 claimant with profit £2,500 and capital allowances of £500 is taxed on £2,000, where the Act gives £2,500. Under the raise the gain is then £1,500 rather than £2,000.
  • policyengine-uk-data sets capital_allowances from SPI CAPALL, so an enhanced FRS that adopts the flag would hit this. At least, the input's doc should tell datasets to zero capital allowances for claimants.

Tests

3. The property test's "reference written from the Act" copies the implementation (tests/test_trading_allowance_claim_status_properties.py:125-161, :212, :229-235).

  • It restates the supplied-receipts rule, profit + £1,000, the two non-claimant zeros and the claimant's capital allowances. So it asserts the results in 1 and 2: each kind of row turns up in 4-10% of drawn people. It would fail a statutory fix.
  • It always sets miscellaneous_income equal to income_not_otherwise_charged, so the min() cap is never exercised.
  • Nothing tests the status carrying forward from the data year.

Design

4. The input allowance is a fact about the data, read as a simulation-year parameter.

  • parameters/gov/simulation/trading_allowance_in_input_profits.yaml is read at utils/trading_allowance.py:90-92 and holds £1,000 for every year.
  • Take data measured under the £3,000 allowance (2027-28 onwards), flagged CLAIMANT with profit £2,500, run in 2028 at defaults. It gets a £2,000 deduction and is taxed on £500; with the parameter at £3,000 it would be taxed on £2,500. That is spurious relief in the baseline.
  • One model version can't serve datasets of both vintages without a reform per dataset.
  • policyengine-uk already has two ways to let a dataset declare its basis:
    • it reads dataset facts from the H5: energy_price_basis, savings_income_projection and isa_income's contract, read in simulation.py;
    • basic_state_pension reads the legislated value at min(simulation.dataset.years).

5. A cut's lower bound isn't visible where users reform the allowance.

  • Taking nil under a cut assumes each claimant's expenses equal £1,000, the extreme of the plausible range. So abolition yields £0 static from every flagged claimant.
  • That is stated in the PR body, in the YAML and in one clause of the deduction's doc, but not on gov.hmrc.income_tax.allowances.trading_allowance.
  • The repo's pattern for such assumptions is an explicit gov.simulation parameter, for example pensions/prior_year_input_ratio.yaml.

Edge cases

6. No-trade claimants get £1,000 of invented trade receipts (:101).

  • A claimant with no trade or zero profit is read as having exactly £1,000 of receipts. Those take the allowance ahead of Chapter 8 income:
    • in 2026, a no-trade claimant with £2,000 of casual income is taxed on £2,000 (UNKNOWN: £1,000);
    • in 2027, one with £2,800 is taxed on £800 (the Act: nil).
  • income_not_otherwise_charged.py:20-22 still says a trade with no profit has no receipts.
  • The data guidance excludes these people, so the input's doc should say so.

7. A float32 round trip flips uses_trading_allowance (:99-107).

  • Receipts are float32 profit + 1,000, and receipts − profit isn't exactly 1,000. A 2026 claimant with profit £1,800.40 gets a deduction of £0.0001 and uses_trading_allowance True. That happens for 14% of penny-valued profits under £4,500.
  • On datasets the money effect is nil, but counts of allowance users at £1,000 are inflated. In situation runs with a carried loss, loss relief can drop to 0.
  • Fix: expenses = where(claimant, input_allowance, ...).

8. uses_trading_allowance is False for every claimant at the input allowance (uses_trading_allowance.py:37).

  • The deduction now measures only relief beyond the input profit, but the variable's doc says it marks people who take the allowance, and the loss-relief rules gate on it.
  • A claimant with profit £2,500 and a supplied £1,500 loss gets the loss relieved in 2026, and the relief is withdrawn under the raise. The measured gain is £100 instead of £400.
  • The result matches UNKNOWN, so it isn't a regression. But anyone counting claimants with this variable gets zero.

9. A raise can withdraw a carried-forward loss and raise tax (:152-157).

  • Under any raise every claimant gets a positive deduction. That makes uses_trading_allowance True and withdraws a loss the engine carried forward. The current-loss election only weighs supplied losses.
  • A situation with 2025 inputs (employment £30,000, profit £5,000, loss £5,000, CLAIMANT) run in 2027 gives income tax of £4,558.85 under current law, against £3,895.89 if the allowance stayed at £1,000.
  • Datasets are unaffected, since projected losses count as supplied.

Texts and cleanup

10. miscellaneous_income_trading_allowance_deduction.py:27-30, the docstring this PR edits, still tells datasets to pin the deduction to zero, as policyengine-uk-data does on SPI records. The PR body says pinning blocks every allowance reform.

11. The NON_CLAIMANT definition (trading_allowance_claim_status.py:27-28) says the profit is net of capital allowances. But self_employment_income is profit before them, and they are deducted separately. A dataset that follows the doc would deduct them twice.

12. The non-claimant rule is applied by two where(non_claimant, 0, ...) masks plus a redundant claimant | term in receipts_known (:106, :116-117, :130-135, :157-161).

  • One ordinary_treatment = non_claimant | (current_loss > allowance_saving) at the return gives identical outputs: the maximum difference was 0.0 over 18,000 synthetic people.
  • It also matches the statute: not electing under s. 783AM is ordinary treatment.

Checked and found true

  • UNKNOWN is unchanged. Outputs and dtypes are identical to the merge base for 400 random people at five allowance levels, including supplied receipts, capital allowances, losses and Chapter 8 income.
  • Consistent claimants. The formula matches the Act at every allowance from £1,000 up, including the Chapter 8 trades-first split. The YAML cases at lines 36, 50, 61, 73, 304, 319 and 334 recompute by hand.
  • The status input. It encodes from enum names and carries forward from the data year, on both the situation and the dataset paths. An H5 without the column reads as UNKNOWN.
  • Tests and lint. The new YAML (24 cases) and the property tests pass, and so do ruff and the parameter and variable metadata tests.

Not covered

  • The full suite, and the microsimulation tests, which need the dataset token.

🤖 Generated with Claude Code

Non-claimants keep full relief, which needs no election (ITTOIA 2005
s. 783AE), where supplied receipts, or casual income alone with no
profit, are within the allowance; partial relief stays elective. One
ordinary-treatment mask replaces the two zeroing masks.

A claimant's input allowance is now the allowance legislated for the
year the data were measured (a dataset's first year, or the year
simulated), read from the unreformed baseline parameters, so data from
either side of the April 2027 raise read right. This replaces
gov.simulation.trading_allowance_in_input_profits.

Claimants: capital allowances are not deducted (partial relief replaces
them), the status applies only to a positive profit, the deduction is
computed from the allowances directly (no float32 round trip), and the
claim is their election, so they take the allowance and give up current
losses at any allowance above nil. A cut's effect is set by the new
gov.simulation.trading_allowance_claimant_expense_share (default 1, the
lower bound), which the trading allowance's description points to.

Tests: YAML cases for each review point, a property test whose reference
works from the Act's options, and dataset tests for the data year and
the status carrying forward.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Response to the review at 5520b59b5

All twelve points are addressed in 4d5b6c1. UNKNOWN is still unchanged, and every existing HMRC YAML test passes, as do the trading allowance, loss, Class 4 and income concept property tests.

Statute

1. Non-claimants keep full relief. Fixed. Partial relief stays elective (s. 783AM). A non-claimant now takes ordinary treatment only where full relief can't be shown; it needs no election (s. 783AE). It can be shown in two ways:

  • supplied receipts plus income not otherwise charged are within the allowance;
  • with no profit, that income alone is within it.

Your three cases now give income tax of £3,486, nil on both amounts, and £3,486. YAML cases cover each of them, plus the s. 783AL election out of full relief to claim a larger current loss.

2. A claimant's capital allowances are no longer deducted. Fixed in the engine, not only in the docs. Partial relief replaces them (s. 783AI), so for a claimant the deduction gives back any capital_allowances deducted separately. That means it can now be negative, which the deduction's documentation says. Your 2026 case is taxed on £2,500, and the raise gains the full £2,000. Class 4 and relevant UK earnings read the same sum, so they follow.

Tests

3. The property test. Rewritten.

  • The reference now works through the Act's options for each person instead of restating the formula: full relief without an election, partial relief only with one (never for a non-claimant), and ordinary treatment. A claimant's ordinary treatment deducts the expense share in 5.
  • It draws miscellaneous_income and income_not_otherwise_charged separately, so the min() cap is exercised.
  • It runs 14 scenarios: allowances from £0 to £10,000, and cuts at expense shares of 1, 0.5 and 0.
  • I re-applied four bugs from this review as mutations, and each fails both the property test and the YAML: the non-claimant zeros (1), the claimant's capital allowances (2), invented receipts for a claimant with no profit (6), and claimants not counted in uses_trading_allowance (8).
  • A new test_trading_allowance_claim_status_data_year.py covers the status carrying forward from the year supplied, and the data-year reading in 4. Reading the input allowance at the simulation year fails it.

Design

4. The input allowance is now read for the data year. Done the basic_state_pension way. gov.simulation.trading_allowance_in_input_profits is gone.

  • The engine reads the legislated allowance for the data year: the dataset's first year, or without a dataset the year simulated.
  • It reads it from parameters.baseline, which reforms leave alone (YAML inline ones included), sampled on 30 April as the fiscal-year conversion does. So there's no second copy of the allowance's history to keep in line.
  • On small datasets:
    • data measured in 2025 get the Budget raise in 2027 (£2,000);
    • data measured in 2027 get nothing from it in 2027 or 2028;
    • a reform that also changes the data year's allowance doesn't move the input allowance.

One consequence for situations: a claimant's 2027 profit is read as already net of £3,000. So the YAML raise cases now run in 2026 with reforms to £3,000 and £5,000, and a 2027 case shows £5,000 giving £2,000 over £3,000.

5. The cut assumption is now an explicit parameter. It is gov.simulation.trading_allowance_claimant_expense_share, a convention parameter like prior_year_input_ratio: claimants' expenses and capital allowances as a share of the input allowance.

  • Under a cut, a claimant keeps claiming while the new allowance covers those expenses, and otherwise deducts them. So a cut adds the input allowance less the larger of the two.
  • The default of 1 keeps the lower bound, so results don't change. 0 gives the upper bound.
  • The description of gov.hmrc.income_tax.allowances.trading_allowance now points to it.

Edge cases

6. Claimants with no profit. Fixed. The status describes a positive profit, and a claimant with none is read as unknown, so no receipts are invented. Your cases give £1,000 relieved (as UNKNOWN) and nil.

7. float32. Fixed. The claimant deduction is computed straight from the allowances, as min(profit, max(A, share × input) − input) − capital allowances. So it is exactly 0 at the input allowance (your £1,800.40 case included), and uses_trading_allowance no longer depends on its sign for claimants.

8. uses_trading_allowance for claimants. Fixed. Their claim was the election, so a claimant takes the allowance at any level above nil: they count, and a supplied current loss stays given up. Your example now gains £400. Under abolition there is nothing to take, so a supplied loss can be claimed.

9. Carried-forward losses. This follows from 8. A claimant gives up current losses at the input allowance and under a raise alike, so a raise can't withdraw one. It is tested on the dataset path (the raise lowers tax by exactly £400) and on the situation path (no change).

What remains predates this PR and also affects unknown-status traders: the current-loss election weighs only supplied losses, while Income Tax relieves inputs carried from earlier years. I've left that as it is.

Texts and cleanup

10. The Chapter 8 deduction's doc now says to flag claimants and non-claimants rather than pin trading_allowance_deduction, as policyengine-uk-data does on SPI records.

11. The NON_CLAIMANT definition now says the profit is net of actual expenses, with capital_allowances deducted from it separately.

12. Done. One ordinary mask, with the full-relief exception, replaces the two zeroing masks, and the redundant claimant | term is gone. Claimants skip the comparison.

Commands

Run in the locked environment (uv run --frozen --extra dev):

  • policyengine-core test on the new YAML (37 cases) and on policy/baseline/gov/hmrc: 1,294 passed.
  • pytest on the new property and data-year files and the existing related tests: 258 passed and 1 skipped (test_build_metadata.py's bundle-contract test, which needs policyengine_bundles and is on CI's allowlist).
  • ruff format and ruff check: clean.

Not run: the full suite, and the microsimulation tests, which need the dataset token.

🤖 Generated with Claude Code

Without a dataset the year simulated stands in for the year the inputs
were measured, even for inputs supplied for an earlier year and carried
forward, so a claimant given 2025-26 inputs and simulated in 2027-28 is
read as net of £3,000 and gets nothing from the raise. Keep that
convention, shared with the State Pension variables, and say so in
_data_year, the claim status input and the deduction's documentation,
with the workaround (build a dataset for the earlier year). A test pins
the behaviour so that a change of convention has to update the docs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Re-review of #2765 at 6b9a638d7

This re-reviews 4d5b6c1bd and 6b9a638d7 against my review at 5520b59b5. One reviewer re-ran each earlier case and re-applied each fixed bug as a mutation. A second read the new code for new defects, and I re-ran the main new case myself. Everything used synthetic situations and datasets.

Verdict: all twelve round-1 points are fixed. The property test and the YAML now each fail all four re-applied bugs.

  • Fix before the data side adopts the flag: a new defect in the cut logic (1), which the rewritten property test locks in (2).
  • Cheap fix available: the situation data year (3).
  • Small: 4 and 5.

New

1. Under a cut below a claimant's expenses, the claimant still counts as taking the allowance (utils/trading_allowance.py:212).

  • When the new allowance is below claimant_expenses (share × input allowance, :131), the claimant deducts their expenses instead. That is ordinary treatment, as the comment at :160 says.
  • But the third return value stays claimant & (allowance > 0). So uses_trading_allowance is True, and a supplied current loss stays given up. Only abolition releases it.
  • Example: 2026, employment of £30,000, profit of £2,500, a supplied trading_loss of £1,500, CLAIMANT, the default share of 1:
    • an allowance of £0 relieves the loss, for income tax of £3,686;
    • an allowance of £1, £500 or £999 doesn't, for £3,986;
    • the taxable trade profit is £2,500 in every case.
  • So moving the allowance from £0 to £1 raises this person's tax by £300, though not electing under s. 783AM would leave the loss claimable. A costing of a small cut books that £300 as yield, and abolition looks cheaper than a cut to £500.
  • Fix: claimant & (allowance > 0) & (allowance >= claimant_expenses). All 37 YAML cases still pass with it.

2. The property test copies the claimant rule, so it would fail that fix (tests/test_trading_allowance_claim_status_properties.py:189-196).

  • The reference's claims_loss = ~claimant & … and takes_allowance = where(claimant, allowance > 0, …) restate the code. With the fix above, it fails at an allowance of £300 with a share of 1.
  • The full-relief boundary is untested. s. 783AE gives full relief when relevant income "does not exceed" the allowance, and the code compares with <= (:150, :158, :203). But no YAML case sits exactly at the allowance, so nothing in the YAML pins <= against <.
  • One test can't fail. test_a_raise_leaves_a_situation_claimants_carried_loss (tests/test_trading_allowance_claim_status_data_year.py:138) runs on the situation path, where the raise gives nothing anyway (3). The dataset test at :84 is the real check.

3. A situation's data year can be read from its inputs.

  • 6b9a638d7 documents the limitation and says a situation "does not record when its inputs were measured". It does: after a 2027 calculation, the simulation's _user_input_keys holds only 2025 for both trading_allowance_claim_status and self_employment_income.
  • Suggested fix:
    • utils/supplied_inputs.py:133-168 (supplied_input_periods) already reads these keys. With the status added to SUPPLIED_INPUT_VARIABLES, _data_year could take the first year the status was supplied for.
    • Inputs carried from 2025 would then read as datasets do (£2,000 from the raise in 2027, not 0), and same-year inputs would stay as they are. Datasets could keep min(dataset.years).
  • Docs wording: a situation does have a dataset. build_from_situation stores a UKSingleYearDataset at default_input_period (simulation.py:1088-1090). _data_year falls back only because that class has no years, so the docs' "without a dataset" is not quite the mechanism.

4. The expense share has no upper bound (parameters/gov/simulation/trading_allowance_claimant_expense_share.yaml).

  • Its description says an allowance at or above the input allowance is unaffected. Above a share of 1 that is false: at 1.5, a claimant gets a £500 deduction under today's law, so the baseline moves.
  • A share above 1 also contradicts the claim itself, since a claimant whose expenses exceeded the allowance would have deducted them instead.
  • Suggestion: cap claimant_expenses at the input allowance, or bound the share to [0, 1].

5. A dataset can't declare the year it was measured (low). With the parameter gone, the input allowance follows min(dataset.years) alone.

  • A file whose first year is 2027 or later but whose incomes were measured under £1,000 would be read as net of £3,000. An example is a dataset projected forward and saved from a later year.
  • A dataset attribute, read the way energy_price_basis is, would let such a file say so.

Round-1 items

# Status
1. Non-claimants and full relief Fixed. The three cases give £3,486, nil on both amounts, and £3,486. Casual income above the allowance still gets nothing, as partial relief needs the election. The s. 783AL election out works in both directions.
2. Claimant capital allowances Fixed. Taxable profit is £2,500, and the raise gains the full £2,000. Class 4 and relevant UK earnings follow.
3. Property test Fixed. It works through the Act's options and exercises the min() cap. It fails each of the four re-applied bugs, as does the YAML. See 2 above for what it still copies.
4. Input allowance Fixed for datasets. Data measured in 2025 get £2,000 in 2027 and 2028, and data measured in 2027 get nothing. Five forms of reform all leave the input allowance alone. Situations: see 3 above.
5. Cut assumption Fixed. The new share parameter defaults to 1, and the allowance's description points to it. See 4 above.
6. No-profit claimants Fixed. They are read as unknown.
7. float32 Fixed. £1,800.40 gives exactly 0, and a sweep of 9,460 penny-valued profits found no non-zero deduction.
8. uses_trading_allowance Fixed at the input allowance and under a raise. Under a cut, see 1 above.
9. Carried-forward losses Fixed on datasets: the raise lowers tax by exactly £400, with the loss given up both ways.
10-12. Texts and the single mask Fixed.

Checked

  • The new YAML (37 cases), the data-year file (6 tests) and the property file pass, and ruff is clean.
  • On CI at 6b9a638d7, the YAML, lint and smoke-import jobs pass. The Python test shards were still running.
  • Not covered: the full suite, and the microsimulation tests, which need the dataset token.

🤖 Generated with Claude Code

A claimant takes the allowance only while it is above nil and at least
their expenses (the expense share of the input allowance); below them
they deduct those expenses, which is ordinary treatment, so a supplied
current loss is claimable under any cut, not only under abolition.

A situation's data year is now the latest year, up to the one simulated,
for which trading_allowance_claim_status was supplied (read through
supplied_input_periods; the status joins SUPPLIED_INPUT_VARIABLES), so
inputs carried from an earlier year keep that year's allowance. Microdata
keep their first year, chosen through built_from_data. This replaces the
limitation recorded in 6b9a638.

The expense share is clamped to [0, 1]. A dataset still cannot declare
the year its incomes were measured; that is now documented.

Tests: the property test's reference derives a claimant's choice by
comparing the allowance with deducting their expenses, and gains a
scenario at a share of 1.5; YAML cases cover a cut with a current loss,
full relief exactly at the allowance and a share above 1; the situation
data-year tests now check the supplied year.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Response to the re-review at 6b9a638d7

All five new points are addressed in 8944500. Unknown-status results are still identical to main, and every earlier case and mutation check still holds.

1. A cut below a claimant's expenses is ordinary treatment

Fixed as you suggested. The rule is now claimant & (allowance > 0) & (allowance >= claimant_expenses). Below their expenses, a claimant deducts those expenses (s. 783AM: no election), so a supplied current loss is claimable.

Your example now gives:

  • £3,686 at an allowance of £0, £1, £500 and £999, so there is no jump between £0 and £1;
  • £3,986 at £1,000, where their claim gives the loss up;
  • £3,586 at £3,000.

With the default share, the one jump left sits at the input allowance itself. A claimant with a supplied current loss gives it up there and claims it under any cut. Consistent data have no such person, since the claim makes the loss nil. A new YAML case covers the cut.

2. The property test

  • The claimant rule. The reference now compares the claimant's two options: their trade figure under the allowance, and under deducting the expense share. A claimant keeps the allowance while it does at least as well; a current loss doesn't enter. It passes with the fix and fails the old rule as a mutation. The near-boundary exclusions no longer apply to claimants, whose choice compares the allowances exactly, so the tie at the input allowance is checked rather than skipped.
  • The full-relief boundary. Two YAML cases now sit exactly at the allowance: a non-claimant with relevant income of 2,100 + 900 = 3,000, and one with £3,000 of casual income and no trade. Switching <= to < fails both.
  • The test that couldn't fail. The situation loss test can fail now: after 3, it expects the raise to lower tax by exactly £400.

3. A situation's data year comes from its inputs

Done as you suggested, with one refinement. _data_year now works like this:

  • Microdata: min(dataset.years), chosen through built_from_data.
  • A situation: the latest year, up to the one simulated, for which trading_allowance_claim_status was supplied. It reads that through supplied_input_periods, and the status is now in SUPPLIED_INPUT_VARIABLES.

The refinement: I take the latest supplied year rather than the first, so a situation with explicit inputs for 2025 and 2027 reads 2027's as measured in 2027. That case has a test.

Carried 2025 inputs now get £2,000 from the raise in 2027, while same-year inputs are unchanged, and so is every YAML case. The limitation text from 6b9a638d7 is gone, and its pinning test now checks the new rule.

The wording no longer says "without a dataset": the docs distinguish microdata from situations. One simplification is documented: one year serves a whole situation, so people whose status was supplied for different years all take the latest.

4. The expense share is bounded

The formula clamps it to [0, 1], and its description says so. A share of 1.5 now leaves today's law unchanged. A YAML case and a property-test scenario at 1.5 cover it, and the property test's bound on a claimant's deductions fails without the clamp.

5. A dataset can't declare its measurement year

Documented, not built. _data_year and the input's documentation now say a file saved from a later projected year is read as measured in its first year. A declared attribute like energy_price_basis would touch dataset_schema, H5 I/O and Simulation, so I've left it for a separate change.

Aggregate impact

The bot's FAIL comment is the run for 4d5b6c1bd. It was cancelled when 6b9a638d7 was pushed a minute later, so the report never ran; nothing crashed. The run for this commit will replace it.

Checks

Run in the locked environment (uv run --frozen --extra dev):

  • YAML: policyengine-core test on policy/baseline/gov/hmrc: 1,298 passed. That includes the 41 claim-status cases.
  • Python: pytest on the claim-status property and data-year files and the related trading allowance, Chapter 8, trading loss, Class 4, income concept, parameter and metadata tests: 260 passed and 1 skipped (test_build_metadata.py's bundle-contract test, which needs policyengine_bundles and is on CI's allowlist).
  • Mutations: Each fix in this round, re-applied as the old rule, fails the tests meant for it:
  • keeping the allowance for a claimant below their expenses fails the YAML and the property test;
  • a strict < for full relief fails the two exact-boundary YAML cases;
  • an unclamped share fails the YAML and the property test;
  • reading a situation's data year as the year simulated fails two data-year tests.

Round 1's four mutations still fail both files. Unknown status is still identical to main, checked over 400 random people at five allowances, dtypes included.

  • Lint: ruff format and ruff check are clean, and the changelog fragment check passes.

Not run: the full suite, and the microsimulation tests, which need the dataset token.

🤖 Generated with Claude Code

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Third review of #2765 at 894450054

This re-reviews 894450054 against my second review at 6b9a638d7. As before:

  • one reviewer re-ran each earlier case and re-applied each fix's old rule as a mutation;
  • a second read the commit for new defects;
  • I reproduced the main new one myself.

Everything used synthetic situations and datasets.

Verdict: the five round-2 points are fixed as the reply describes. But the data-year change breaks an existing test (1), so that needs fixing before merge. Items 2 and 3 are the remaining design points, on situations and on claimants with losses. The rest are small.

Fix before merge

1. _data_year crashes a policyengine-core simulation built over UK data, so an existing test fails (utils/trading_allowance.py:41).

  • Why it crashes: getattr(getattr(simulation, "dataset", None), "years", None) swallows only AttributeError. A core Dataset answers unknown attributes through __getattr__, which calls self.load(name). That raises TypeError for Dataset.from_dataframe data and an h5py KeyError for files.
  • Who it hits: built_from_data is True for these simulations (is_over_dataset), and _data_year runs for everyone. So income tax fails whether or not anyone has a status.
  • The failing test: tests/test_data_built_imputations.py::test_core_simulation_over_data_gets_imputations (:291) fails at this head, and I reproduced it. Round 2's version caught TypeError, and with it swapped back in, the test passes. CI's Python jobs were still running when I checked.
  • Why catching alone isn't enough: such data would then fall into the situation rule. With the status stored for every year, "the latest supplied year up to the one simulated" is the year simulated, which zeroes the raise.
  • Fix: catch the loader's errors, or test for UKMultiYearDataset. For data without years, take the earliest year the status was supplied for.

Remaining design points

2. A situation's data year follows the year the status was supplied, not the profit (:44-51).

  • The input's definition (variables/input/trading_allowance_claim_status.py:17-18) ties the claim to "the tax year their input self-employment profit was measured". But the code reads the year the status was supplied for.
  • Status with no year key: it is stored for 2025. With self_employment_income of £5,000 given for 2027, a 2027 run deducts £2,000, reading a 2027 profit as net of £1,000. Round 2's head gave 0. Axes over that 2027 profit give £2,000 at every point.
  • The reverse: a status for 2027 with a profit for 2025 gives 0, reading the 2025 profit as net of £3,000.
  • Round trip: a situation exported with to_input_dataframe() and reloaded counts as microdata, so it gives a different 2027 answer from the situation itself (£2,000 against 0).
  • Suggestion: anchor on the latest year the profit was supplied for, up to the year simulated, or say in the input's docs that the status must carry the profit's year.

3. A claimant with a supplied current loss still jumps where the allowance crosses their expenses (:230).

  • The reply notes this jump at the input allowance, at the default share. But it moves with the share:
    • at 0.5 it sits at £499 to £500 (income tax £3,786 to £4,086);
    • at 0, the round-2 example comes back exactly (£3,886 at £0, £4,185.80 at £1).
  • It grows with the loss and the rate: +£2,400 at the higher rate (pay £60,000, profit £5,000, loss £6,000).
  • At the default share, every cut lowers such a claimant's tax by the loss's relief, so a costing books a cut as a cost for them.
  • The reply calls this inconsistent data, since a claim leaves the trade with no loss. If so, the PR's own rule for a claimant with no profit fits: read a claimant with a supplied current loss as unknown. Alternatively, compare the loss with the allowance's saving for claimants too, as for everyone else.

Tests and small items

4. The "up to the year simulated" filter is untested (:49).

  • Removing it passes all three touched test files.
  • The 2026 assertion (tests/test_trading_allowance_claim_status_data_year.py:149) can't fail: at £1,000, a default-share claimant gets 0 whichever year is read.
  • The filter matters once there's a loss. Take a status for 2025 and 2027, a 2026 loss of £1,500 and £30,000 of pay: 2026 income tax is £4,515.70 with the filter and £4,215.70 without.
  • Suggestion: assert uses_trading_allowance, or income tax with a loss, in 2026.

5. Small items:

  • Float tie: with a £3,000 input allowance and a share of 0.55, the expenses compute to 1,650.0000000000002. So a claimant at an allowance of £1,650 loses the allowance the docs say they keep: with a £1,500 loss, income tax is £3,956 instead of £4,256. Nine of 202 combinations of share and input allowance do this, all at £3,000. Rounding the expenses to pennies would fix it.
  • Capital allowances in the choice: below a share of 1, supplied capital allowances above the share's expenses don't count. With a share of 0.5, capital allowances of £800, an allowance of £600 and profit of £2,500, the claimant keeps the allowance (taxable £2,900), though ordinary treatment gives at most £2,700. Using the larger of share × input allowance and the capital allowances, capped at the input allowance, would cover it.
  • Texts:
    • _data_year says people whose status was supplied for different years "all take the latest". That can't happen as written: in any year one person's status is given, the builder gives everyone else the default, so the other claim reads UNKNOWN.
    • The changelog, trading_allowance_deduction.py:24-25 and the YAML header leave out "up to the year simulated".

Round-2 items

# Status
1. Cut below expenses Fixed at the default share. The round-2 case gives £3,686 at £0, £1, £500 and £999, £3,986 at £1,000, and £3,586 at £3,000. Over 175 allowance levels and 21 profiles, the only wrong-way steps are the loss jump in 3 above. Dataset and situation paths agree on all 2,856 values compared.
2. Property test Fixed. The reference compares the claimant's options rather than restating the rule, and it fails the old rule. Two YAML cases now sit exactly at the allowance, and < fails them. The situation loss test can now fail. Limit: it still shares the model's conventions (receipts as profit plus the input allowance, the clamp, a claimant's loss left out), so it can't test those.
3. Situation data year Fixed for situations whose status carries the profit's year. Every UK dataset path reads the dataset's first year: single- and multi-year datasets, h5 files, data frames, hf:// and gs:// cache hits, Microsimulation, and clones, branches, reforms and baselines. But see 1, 2 and 4 above.
4. Share bounds Fixed: 1.5 matches 1, and −0.5 matches 0. The lower clamp makes no difference for any allowance of £0 or more, which is harmless.
5. Dataset measurement year Documented, as the reply says.

The round-1 fixes all still hold. UNKNOWN status still gives the merge base's results, dtypes included: 400 random people × 7 scenarios × 10 variables.

Checked

  • Tests: the touched test files pass: the YAML (41 cases), the data-year file (8) and the property file (4).
  • Mutations: each mutation the reply lists fails its tests.
  • CI on 894450054: the YAML, lint, docs, smoke-import and aggregate impact jobs pass. The Python jobs were still running.
  • Not covered: the full suite, and the microsimulation tests, which need the dataset token.

🤖 Generated with Claude Code

_data_year checks the dataset's type, so a policyengine-core Dataset,
which loads any attribute it is asked for, no longer crashes; data
without years take the earliest year the profit was supplied for. In a
situation the data year is now the latest year, up to the one simulated,
for which self_employment_income was supplied, which replaces the status
in SUPPLIED_INPUT_VARIABLES.

Claimants make the same election as everyone else, against their own
options: the allowance unless its saving over deducting their expenses
is negative or smaller than a supplied current loss, so income tax no
longer jumps as the allowance or the loss moves. Their expenses are the
share of the input allowance in pennies, or their capital allowances if
larger, capped at the input allowance, and their figures are computed in
float64 so the two options tie exactly where the allowance equals those
expenses.

Tests: YAML cases for claimants with losses, the switch, capital
allowances in the choice and the penny tie; data-year tests for the
profit's year and the year filter; the property reference weighs a
claimant's loss; a deterministic sweep of ties.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Response to the third review at 894450054

All five points are addressed in 2435155. Unknown-status results still match the merge base, and earlier rounds' cases and mutations still hold, except where 3 changes a claimant-with-a-loss expectation on purpose.

1. Core datasets no longer crash

_data_year now checks the dataset's type:

  • a UKMultiYearDataset gives min(dataset.years), as before;
  • any other data, such as a policyengine-core Dataset, which loads any attribute it's asked for, gives the earliest year the profit was supplied for.

Data stored for every year therefore keep their first year rather than the year simulated. test_core_simulation_over_data_gets_imputations passes again. It was the only failure in CI's full run at 894450054: 11,383 passed, 1 failed, with the dataset present.

2. The data year follows the profit

In a situation, the data year is now the latest year, up to the one simulated, for which self_employment_income was supplied, whatever year the status was given for. self_employment_income joins SUPPLIED_INPUT_VARIABLES and the status leaves it. Nothing in the package writes self_employment_income as an input, so branches such as the marginal-rate ones, which perturb employment_income, don't move the year.

Your two cases, both tested:

  • status for 2025 with a profit for 2027 gives 0 in 2027;
  • status for 2027 with a profit for 2025 gives £2,000.

The round trip still differs: a situation saved over several years and reloaded is read as measured in its first year. _data_year documents that as part of the declared-measurement-year gap, which is proposed separately.

3. Claimants with a current loss: no jumps

I took your second option: a claimant makes the same election as everyone else, against their own options.

  • Their saving is the allowance's over deducting their expenses, and it is negative where the expenses do better.
  • They take ordinary treatment where a supplied current loss is worth more than that saving.
  • At the switch both treatments give the same income, so income tax is continuous in the allowance and in the loss.

On your example (pay £30,000, profit £2,500, loss £1,500), income tax is flat and then falls as the allowance rises from £0 to £3,000, with no step:

  • at share 1: £3,686 up to £2,500, then £3,586 at £3,000;
  • at share 0.5: £3,786 up to £2,000, then down to £3,586;
  • at share 0: £3,886 at £0 and £1, falling from £1,500.

YAML cases at a share of 0.5 pin £499, £500 and the switch at £2,000.

I preferred this to reading such claimants as unknown. That would discard their receipts, add a step at the first penny of loss, and leave them under the unknown rule's own step where a profit crosses the allowance.

One consequence, on purpose: at the input allowance and the default share, the allowance saves nothing over a claimant's expenses, so a claimant with a supplied current loss claims it. Round 1's example now gains £100 from the raise, not £400. A claimant with no loss still takes the allowance there, so uses_trading_allowance still counts them.

4. The filter is tested

With profits supplied for 2025 and 2027, test_a_situation_reads_each_year_supplied_as_measured_then now asserts two things in 2026: the claimant takes the allowance, and a 2026 raise to £3,000 gives £2,000. Without the filter, 2026 reads 2027's profit and its £3,000, and both assertions fail.

5. Small items

  • Float tie. The share's expenses are now rounded to pennies, and a YAML case at a £3,000 input allowance, a share of 0.55 and an allowance of £1,650 keeps the allowance.
    • The case uses a £600 profit. With £2,500, the float error happened to round away inside the saving sum, so the case couldn't fail.
    • The property test then found a deeper version. At a tie, the two options' figures were computed in float32 and float64, so the saving came out as ±1e-5 instead of 0 and could flip the choice.
    • At the default share that could hit a claimant with capital allowances at today's allowance.
    • The claimant's figures are now all in float64. In a sweep of 2,000 random claimants at today's allowance, all take it, and their taxable profit is the reported profit.
    • A new deterministic test runs that sweep at three exact ties: £1,000 at a share of 1, £500 at 0.5, and £1,650 at 0.55 of £3,000. Each uses 2,000 claimants with no loss, capital allowances below their expenses and losses brought forward, and all must take the allowance.
  • Capital allowances in the choice. A claimant's expenses are the larger of the share's and their supplied capital allowances, capped at the input allowance. Your example is now taxed on £2,700 under ordinary treatment, with a YAML case.
  • Texts. The "all take the latest" sentence is gone: _data_year now says the builder stores a year's inputs for everyone together. The changelog, the deduction's documentation and the YAML header now say "up to the one simulated", and name the profit as the anchor.

Checks

Run in the locked environment (uv run --frozen --extra dev):

  • YAML: policyengine-core test on policy/baseline/gov/hmrc: 1,303 passed. That includes 46 claim-status cases.
  • Python: pytest on the claim-status property and data-year files, test_data_built_imputations.py, the aggregate-impact tests and the related trading allowance, Chapter 8, trading loss, Class 4, income concept, parameter and metadata tests: 572 passed and 10 skipped. Nine skips are the certified-dataset aggregate-impact tests, which need the dataset token, and the tenth is the bundle-contract test on CI's allowlist. After the last test changes I re-ran the property and data-year files and test_data_built_imputations.py: 25 passed, including the new tie sweep.
  • Mutations: Each fix in this round, re-applied as the old rule, fails tests meant for it:
  • the old dataset lookup fails test_core_simulation_over_data_gets_imputations;
  • leaving a claimant's loss out of their election fails three YAML cases, the dataset loss test and the property test;
  • dropping the year filter fails the data-year filter test;
  • dropping the penny rounding fails the YAML tie case and the 2027 tie sweep;
  • dropping the capital-allowance floor fails its YAML case and the property test;
  • mixing float32 and float64 again fails all three tie sweeps.
  • Lint: ruff format and ruff check are clean, and the changelog fragment check passes.

Not run locally: the full suite, which CI runs.

🤖 Generated with Claude Code

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Fourth review of #2765 at 243515502

This re-reviews 243515502 against my third review at 894450054. As before:

  • one reviewer re-ran each earlier case and re-applied each fix's old rule as a mutation;
  • a second read the commit for new defects;
  • I reproduced the main new case myself.

Everything used synthetic situations and data.

Verdict: the five round-3 points are fixed, and none of the fixes regresses what it set out to do.

  • The core-data crash is gone.
  • The data year follows the profit.
  • A claimant's choice is now continuous in the allowance and the loss.

UNKNOWN status still gives the merge base's results, with labour supply responses on too.

The changes bring two new issues, both in situations or unusual data:

  • brought-forward losses can tie a claimant's choice, which moves Class 4 (1);
  • anything that writes the profit after the build moves a situation's data year (2).

The rest are test gaps and texts. Separately, two of CI's Python shards now hit the job time limit (see Checked).

New

1. Brought-forward losses can tie a claimant's choice, so they keep the allowance below their expenses (utils/trading_allowance.py:239-246, :264-266).

  • Why: claimant_saving compares the two options after loss_relief, each floored at zero. When the losses brought forward cover both options, both come to nil, and the tie keeps the allowance, even below the claimant's expenses.
  • Effect: income tax doesn't change, but Class 4 does, since it has its own losses brought forward and doesn't see these. Its profits rise by the expenses less the allowance.
  • Examples, all in 2026 with no pay and profit of £20,000 (I re-ran the first two):
    • With loss_relief of £25,000 and a cut to £1 at the default share, the deduction is −£999, uses_trading_allowance is True and Class 4 is £505.74. At round 3 these were 0, False and £445.80.
    • Steps come back in Class 4. With loss_relief of £20,500, raising the allowance from £499 to £500 adds £30, and 2p more profit takes it away again.
    • At £3,000, with loss_relief of £21,000, a £1 current loss adds £119.94.
  • Reach: income tax is nil in each case. Situations reach the cut; data whose loss_relief is the amount actually used can't. The raise case also fits capped data.
  • Fix: keep allowance >= claimant_expenses in the claimant's condition (with it, every test still passes), or compare the options before loss_relief.
  • Not caught today: the property test's reference uses the same floored saving, and its near-tie mask skips exactly these people. The docstring's "a cut leaves their profit as reported" and "never jumps" don't hold here.

2. In a situation, anything that writes the profit moves the data year (:46, :54-59; utils/supplied_inputs.py:58).

  • Why: the anchor is the latest year the profit was supplied for. Any later set_input of the profit for the simulated year counts, even one that writes back the same values.
  • Writers in the package:
    • dynamics/progression.py:67 and :87, where calculate_derivative steps the profit and restores it;
    • utils/dependencies.py:73 and :83-84;
    • any scenario's simulation_modifier.
  • Example: pay £30,000, with profit £5,000 and CLAIMANT given for 2025, run in 2027, gives a deduction of £2,000 and income tax of £4,558.85. After progression.calculate_derivative(sim, input_variable="self_employment_income", year=2027, count_adults=1) (retention 0.40, against 0.80 at round 3), the same simulation gives 0 and £4,958.85.
  • No-op modifier: a modifier that re-sets the 2027 profit to its own value gives the reform £4,958.85 against the baseline's £4,558.85. That is a £400 impact from no change.
  • So "nothing in the package writes self_employment_income as an input" doesn't hold. Round 3's anchor on the status was immune to this.
  • Suggestion: fix the data year when the situation is built, from the inputs as supplied, rather than reading the input history at calculation time.

3. Test gaps:

  • The core-data rule (:53) is untested. Taking the latest profit year instead of the earliest passes every test. Yet a policyengine-core simulation over data with profits for 2025-28 would then give 0 in 2027 instead of £2,000. No test has a claimant in a core simulation.
  • The situation loss test no longer tests losses (tests/test_trading_allowance_claim_status_data_year.py:183-198).
    • Its comment says the claim gives the loss up. Under the new choice, a claimant with a supplied current loss claims it.
    • It passes only because the 2025 loss isn't a supplied 2027 loss. With the loss also supplied for 2027, both runs claim it (£3,958.85).
    • Leaving the loss out of the claimant's choice still passes it.
  • Ties on capital allowances flip only the flag. At share 0, with the allowance equal to capital allowances of £726.21 (stored as 726.210022 in float32), uses_trading_allowance reads False, with the same tax. This is minor.

4. Stale texts:

  • Property test: tests/test_trading_allowance_claim_status_properties.py:5-7 still takes the data year from the status.
  • Share parameter: its comment (trading_allowance_claimant_expense_share.yaml:15-21) says that at 0 "every claimant keeps claiming and a cut adds its full amount". The capital-allowance floor and the loss choice both break that.
  • Changelog: changelog.d/2764.added.md:1 doesn't mention capping the capital-allowance floor at the input allowance.

Round-3 items

# Status
1. Core datasets Fixed. Every UK data path, and core simulations over data, read 2025 and give the 2027 claimant £2,000. The old lookup fails test_core_simulation_over_data_gets_imputations. The new rule is untested (3).
2. Profit year Fixed. A status for 2025 with a profit for 2027 gives 0; a status for 2027 with a profit for 2025 gives £2,000; a status with no year key and a 2027 profit gives 0. Side effects: registering the profit changes nothing for UNKNOWN people: 261 arrays against the merge base show no differences and identical dtypes, with labour supply responses on as well. But see 2.
3. Claimants with a loss Fixed, except where brought-forward losses cover both options (1). Over 5,355 allowance steps and 339,612 loss steps, neither income tax nor Class 4 ever moves the wrong way, and the reply's figures match. Round 1's example gains £100. The only cliffs left are the unknown and non-claimant full-relief cliffs, which the Act has.
4. Filter Fixed. Removing it fails the test at :150.
5. Small items Fixed. Penny rounding gives no flips over 121,200 claimants at exact ties. The float64 figures stay internal, so outputs are float32 or bool as before. The capital-allowance floor gives £2,700. The texts are updated, apart from those in 4.

Checked

  • Tests: the touched test files pass: the YAML (46 cases), the property file (7), the data-year file (9) and test_data_built_imputations.py (9).
  • Mutations: each fix's old rule fails its tests.
  • Performance: _data_year costs under 0.03 ms a call on 50,000 people.
  • CI on 243515502:
    • The YAML, lint, docs, smoke-import, changelog and aggregate-impact jobs pass, and so do Python shards 1, 4, 5 and 6.
    • Shards 2 and 3 hit the 1h15m job limit and were cancelled, so "Test / Result" fails.
    • At 894450054 those two shards took 73.0 and 74.1 minutes, and every shard is 1 to 5 minutes slower here. So the added property and tie-sweep tests have likely tipped them over.
    • A rerun may pass, but the margin is gone. Smaller sweeps, or another shard, would restore it.

🤖 Generated with Claude Code

MaxGhenis added a commit that referenced this pull request Nov 1, 2026
The registration sat where #2765 adds self_employment_income, so the two
conflicted. The set's order means nothing; this place merges cleanly
with #1969, #2661, #2765 and #2776.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A claimant keeps the allowance only where it covers their expenses, so
losses brought forward that absorb both options no longer tie them onto
an allowance below those expenses, which charged Class 4 more.

Simulation.build_from_situation records the years the situation gave a
self-employment profit for, and _data_year reads that record, so a
later write of the profit (a modifier, a derivative) no longer moves a
situation's data year. Clones and branches carry it. Capital allowances
are rounded to pennies before they set a claimant's expenses.

Tests: the property test's four hypothesis tests are now one, which runs
each reformed simulation once per draw; each worker rebuilt the 15
simulations for every test before, which took Python shards 2 and 3 past
the job limit. The tie sweeps use 500 claimants. New tests cover the
core-data rule, a situation claimant's supplied loss, a derivative and a
no-op modifier, losses brought forward under a cut and capital-allowance
ties.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Response to the fourth review at 243515502

All four points, and the CI timeout, are addressed in 7b25254. Unknown-status results still match the merge base, and the earlier rounds' cases and mutations still hold.

CI

Shards 2 and 3 timed out because this PR's property tests were expensive on CI. In run 41303444320's shard logs, each of the four hypothesis tests took 139 to 148 s. Most of that was building the module fixture's 15 reformed simulations, which each xdist worker rebuilt for every one of those tests it ran.

  • The four tests are now one. It runs every simulation once per draw and checks all four invariants on that run. Locally, the fixture takes 63 s and the test about 4 s.
  • The tie sweeps use 500 claimants, not 2,000. They took 7 to 11 s each on CI.

This PR's tests now cost CI less than at 894450054, whose shards all passed. Those shards were already at 73–74 minutes, though, so the suite as a whole has little margin; a seventh shard would be a separate change.

1. Brought-forward losses no longer keep a claimant on the allowance

I took your first fix: a claimant again keeps the allowance only where it covers their expenses. Your examples:

  • a cut to £1 with £25,000 of losses brought forward gives a deduction of 0, uses_trading_allowance False and Class 4 of £445.80;
  • with £20,500 brought forward, allowances of £499 and £500 give the same Class 4.

There is a YAML case for this. The property test also changed:

  • it draws £25,000 of losses brought forward;
  • its reference applies the same condition;
  • its near-tie mask no longer skips claimants whose allowance is below their expenses.

So dropping the condition fails both the YAML and the property test.

Your third example is different: an allowance of £3,000, £21,000 brought forward, and a £1 current loss adding £119.94 of Class 4. That is the election's own convention, not this tie. The election compares a supplied current loss with the income tax saving after losses brought forward. Here those losses reduce the saving to nil, so any loss wins. An unknown-status trader with supplied receipts does the same, and the deduction's documentation describes the comparison. I've left it, since changing it would move results for everyone.

2. A situation's data year is fixed when it's built

I took your suggestion. Simulation.build_from_situation records the years the situation gave a profit for (_situation_profit_years), and _data_year reads that record rather than the input history. Clones, branches and the baseline carry it, so a later write of the profit no longer moves the year:

  • your example gives £2,000 and £4,558.85 both before and after calculate_derivative;
  • a modifier that writes the profit back leaves the reform at the baseline's £4,558.85.

Both are tested, and reading the live history instead fails both tests. A simulation built some other way, such as a core simulation over data, still reads the history as it stands.

3. Test gaps

  • The core-data rule. test_a_core_simulation_over_data_reads_the_first_year_of_the_profit builds a policyengine-core simulation over a frame holding a claimant's profits for 2025–28, and expects £2,000 in 2027. Taking the latest year instead fails it.
  • The situation loss test. The carried-loss test's comment now says what it checks: a 2025 loss carried into 2027 isn't a supplied 2027 loss, so it doesn't enter the election. A new test supplies the loss for 2027: at £1,000 the claimant claims it, and at £3,000 the allowance wins. Leaving the loss out of the claimant's choice fails it.
  • Capital-allowance ties. Capital allowances are rounded to pennies before they set the expenses. So £726.21, stored as 726.210022, ties with an allowance of £726.21 and keeps it, with a YAML case.

4. Texts

  • The property test's docstring now takes the data year from the profit.
  • The share parameter's comment says the upper bound holds for a claimant with no capital allowances and no current loss.
  • The changelog says the capital-allowance floor is capped at the input allowance, and that the allowance must cover the expenses.

Checks

Run in the locked environment (uv run --frozen --extra dev):

  • YAML: policyengine-core test on all of policyengine_uk/tests/policy: 5,458 passed.
  • Python: pytest on the claim-status files, test_data_built_imputations.py, the aggregate-impact tests, the scenario, modifier, input-record, branch and dataset-source tests, and the related trading allowance, Chapter 8, trading loss, Class 4, income concept, parameter and metadata tests: 625 passed and 11 skipped. Ten skips need the dataset token and run in CI: nine certified-dataset aggregate-impact tests and one scenario-baseline test. The eleventh is test_build_metadata.py's bundle-contract test, on CI's allowlist.
  • Mutations: each of this round's fixes, re-applied as the old rule, fails tests meant for it:
    • dropping the expenses condition fails the YAML case and the property test;
    • reading the live input history fails the derivative and modifier tests;
    • the latest core-data year fails the core test;
    • unrounded capital allowances fail the tie case;
    • leaving a claimant's loss out of their choice fails three YAML cases, two data-year tests and the property test.
  • Lint: ruff format and ruff check are clean, and the changelog fragment check passes.

Not run locally: the full Python suite, which CI runs.

🤖 Generated with Claude Code

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Fifth review of #2765 at 7b25254f4

This re-reviews 7b25254f4 against my fourth review at 243515502. As before:

  • one reviewer re-ran each earlier case and re-applied each fix's old rule as a mutation;
  • a second read the commit for new defects.

Everything used synthetic situations.

Verdict: the four round-4 points and the CI timeout are addressed, and CI is green.

  • Claimants now keep the allowance only where it covers their expenses.
  • The data year is fixed when the situation is built.
  • The test gaps are closed.

What's left:

  1. the build-time record misses two paths;
  2. some texts still describe the old rule;
  3. the loss fix is pinned on one side only.

New

1. The build-time record misses two paths (simulation.py:1094-1099, utils/trading_allowance.py:58-67).

  • Profits set after the build don't count. A situation built without a profit records an empty set. _data_year falls back to the live history only when the record is missing, not when it is empty, so it reads the year simulated.
    • A bare situation given a 2025 profit and CLAIMANT by set_input gets 0 in 2027, instead of £2,000.
    • A 2027 profit set after building a 2025-profit situation is read as measured in 2025, giving £2,000 instead of 0.
    • The property tests build this way, so they take their data year from the fallback, not from the profit as their docstring says.
    • No caller outside the tests builds this way today: policyengine.py and the API put every input in the situation.
  • build_from_populations doesn't refresh the record (simulation.py:620-633). It is public, and test_housing_benefit_accommodation_dates.py uses it. Build from a 2025-profit situation, then take a 2027-profit donor's populations: the 2027 profit gets £2,000, where a fresh build gives 0.
  • Suggestion:
    • fall back to the live history when the record is empty;
    • copy or recompute the record in build_from_populations;
    • say in the docs that a profit given after the build doesn't move the year.

2. Texts that still describe the old rule:

  • utils/trading_allowance.py:143-144 says "The tax they pay never jumps as the allowance or the loss moves". That isn't so. Where losses brought forward absorb the saving, a £1 current loss still adds £119.94 of Class 4 at £3,000, and up to £239.94 at £4,000 at a share of 0. That's the convention the reply keeps, and unknown-status traders with receipts have it on the merge base too; only the sentence is wrong.
  • trading_allowance_deduction.py:37-43 says ties take the allowance, "claimants included". But the commit's own YAML case (an allowance of £1, with £25,000 brought forward) is a tie that takes ordinary treatment.
  • trading_allowance_claim_status.py:24-27 and the changelog still say the year is the latest one "for which the profit was supplied". That no longer covers profits given after the build (1).

3. The loss fix is pinned on one side only.

  • The gap: only the allowance-below-expenses side is tested. Take covered claimants whose brought-forward losses absorb both options, and let them take ordinary treatment instead: all three test files still pass. Yet Class 4 then rises from £325.80 to £445.80 (profit £20,000, with £21,000 or £25,000 brought forward, at £3,000).
  • Why nothing catches it: the property test's near-tie mask drops every covered claimant with a nil saving, and the only loss_relief YAML case uses a £1 allowance.
  • Fix: add a YAML case at £3,000 with £21,000 brought forward, expecting Class 4 of £325.80 and uses_trading_allowance true. A property check that Class 4 profits never rise with the allowance would also pin it.

Round-4 items

# Status
1. Brought-forward losses Fixed. The cut to £1 with £25,000 brought forward gives a deduction of 0, False and Class 4 of £445.80, and £499 and £500 now give the same Class 4. Over 10,710 allowance steps (42 profiles, three shares), nothing moves the wrong way. The third example is the merge base's own convention: an unknown-status trader with receipts gets the same +£119.94. But see 3.
2. Data year at build Fixed for derivatives and modifiers: £2,000 and £4,558.85 both before and after. The record survives clones, branches, the reform baseline, deep copies, Microsimulation and axes. But see 1.
3. Test gaps Fixed. The core-data test, the loss tests and the capital-allowance tie case each fail under their mutation.
4. Texts Partly (2).
CI Green. The merged property test still checks each invariant on 25 examples. Margin still thin: the shards ran 73.3, 72.4, 63.2, 69.0, 64.6 and 74.8 minutes. The savings landed on shards 1, 3 and 5, and runner variance is bigger than the cut. Correction to the reply: at 894450054 shard 4 failed on the core-data crash. All the shards finished inside the limit there, which is probably what was meant.

Checked

  • UNKNOWN status still matches the merge base: 501 arrays × 2 comparisons, with identical dtypes.
  • The earlier rounds' fixes still hold.
  • The touched files pass: the YAML (48 cases), the data-year file (13 tests), the property file (4) and test_data_built_imputations.py (9).
  • Not covered: the full suite (CI is green) and the microsimulation tests, which need the dataset token.

🤖 Generated with Claude Code

A situation that gave no profit at its build now reads the years a
profit is set for, and build_from_populations recomputes the recorded
profit years from the populations it takes. A profit set after a build
that gave one still leaves the recorded year, which the docs now say.

Texts: the reliefs docstring no longer claims the tax never jumps (Class
4 can, where losses brought forward absorb the saving, as for every
status); the deduction's documentation says claimants keep the allowance
only where it covers their expenses, ties included; the input's
documentation and the changelog describe the recorded year.

Tests: a YAML case pins covered claimants whose losses brought forward
absorb both options to the allowance; the property test checks Class 4
profits never rise with the allowance and compares those claimants; the
data-year file covers the three record paths. The property fixture and
the tie sweeps change a built simulation's parameters rather than pass a
reform, which skips a baseline build they never read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Response to the fifth review at 7b25254f4

All three points are addressed in b2bbf88, and your CI correction is taken (below). Unknown-status results still match the merge base, and the earlier rounds' cases and mutations still hold.

1. The two paths the build-time record missed

  • Profits set after the build. An empty record now falls back to the live history, so a situation built without a profit reads the years one is set for. Your bare situation, given a 2025 profit and CLAIMANT by set_input, gets £2,000 in 2027. The property tests build that way, so they now read the profit's year through this fallback, as their docstring says.
  • A situation that did give a profit. A profit set afterwards doesn't move the year. _data_year, the input's documentation and the changelog now say so. That keeps your second case (a 2027 profit set after a 2025-profit build) at £2,000 on purpose: it is the same write a modifier or a derivative makes.
  • build_from_populations. It now recomputes the record from the new populations' supplied profits, so your donor case gives 0, as a fresh build does.

Each path has a test: a bare situation given a profit later, a profit set after a 2025 build, and populations taken from a 2027-profit donor.

2. Texts

  • utils/trading_allowance.py no longer says the tax never jumps. It now says income tax is the same either way at the switch. Class 4 can still jump with the loss where losses brought forward absorb the income tax saving, which is the convention every status shares.
  • trading_allowance_deduction.py now says a claimant keeps the allowance only where it covers their expenses, ties included. The loss comparison applies to everyone else, and to claimants where the allowance covers their expenses.
  • The input's documentation and the changelog describe the recorded year: the latest year, up to the one simulated, that the situation gave the profit for when it was built.

3. The covered side of the loss fix is pinned

  • A YAML case at £3,000 with £21,000 brought forward expects Class 4 of £325.80 and uses_trading_allowance true.
  • The property test now also checks that Class 4 profits never rise with the allowance.
  • Its near-tie mask no longer drops covered claimants whose options the losses brought forward clearly absorb. Both savings are exactly nil there, so they are compared.

Sending ties to ordinary treatment for covered claimants now fails six YAML cases, two data-year tests and the property test.

CI

  • Correction. At 894450054, shard 4 failed on the core-data crash. What I meant is that every shard finished inside the limit there.
  • The thin margin is suite-wide. At 7b25254f4, this PR's tests were a small share of the tight shards:
    • about 16 s of shard 6's 74.8 minutes;
    • the merged property test (133 s) plus about 67 s of data-year tests in shard 2's 72.4.
  • This PR's share is smaller again. The property fixture and the tie sweeps now change the parameters of a built simulation instead of passing a reform, which built a baseline they never read. Locally the fixture fell from 63 s to 43 s, and the input allowance is still read from the unreformed tree.

A seventh shard would restore the margin for every pull request; I've left it for a separate change.

Checks

Run in the locked environment (uv run --frozen --extra dev):

  • YAML: policyengine-core test on all of policyengine_uk/tests/policy: 5,459 passed.
  • Python: pytest on the claim-status files, test_data_built_imputations.py, the aggregate-impact tests, the scenario, modifier, input-record, branch, dataset-source and housing benefit accommodation-date tests (which use build_from_populations), and the related trading allowance, Chapter 8, trading loss, Class 4, income concept, parameter and metadata tests: 666 passed and 11 skipped. Ten skips need the dataset token and run in CI: nine certified-dataset aggregate-impact tests and one scenario-baseline test. The eleventh is test_build_metadata.py's bundle-contract test, on CI's allowlist.
  • Mutations: each of this round's fixes, re-applied as the old rule, fails tests meant for it:
    • ties sending covered claimants to ordinary treatment fail six YAML cases (the new one among them), two data-year tests and the property test;
    • an empty record that doesn't fall back fails the bare-situation test;
    • build_from_populations without the refresh fails the donor test.
  • Lint: ruff format and ruff check are clean, and the changelog fragment check passes.

Not run locally: the full Python suite, which CI runs.

🤖 Generated with Claude Code

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Sixth review of #2765 at b2bbf8822

This re-reviews b2bbf8822 against my fifth review at 7b25254f4. As before:

  • one reviewer re-ran each earlier case and re-applied each fix's old rule as a mutation;
  • a second read the commit for new defects.

Everything used synthetic situations and datasets.

Verdict: the round-5 points are fixed, and unknown-status results still match the merge base:

  • an empty record falls back to the input history;
  • build_from_populations refreshes the record;
  • the texts are corrected;
  • the covered side of the loss fix is pinned.

What's left:

  • CI is red. Python shard 3 hit the 75-minute limit at 99%, with no failures. That's runner variance, not this commit: its tests take about 4 s there. But the PR can't merge without a rerun, and the seventh shard is now what blocks it.
  • The two new paths reopen the round-4 problem in narrow cases (1, 2).
  • Texts (3).

New

1. On the fallback path, a derivative or a later write moves the data year again (utils/trading_allowance.py:59-66).

  • Mechanism: a situation built without a profit falls back to the live history, so any later write of the profit for the simulated year re-anchors it. That is the round-4 problem, reopened for this path.
  • Example: a bare situation (pay £30,000 for 2025), then set_input a 2025 profit of £5,000 and CLAIMANT.
    • 2027 gives £2,000 and income tax of £4,558.85, as a dict build does.
    • But calculate_derivative(..., year=2027) returns 0.40, where the dict build gives 0.80. Its reset leaves 2027 recorded as supplied, so afterwards the deduction is 0 and income tax £4,958.85, for good.
    • A branch that sets a 2027 profit gets 0, where the dict build's branch keeps £2,000.
  • Docs and tests: the docstring (:33-41) says a derivative or modifier doesn't move the year, but its last sentence concedes the opposite for this path. The derivative test builds from a dict only.
  • The fallback's condition: it checks for an empty record, not for whether a usable year was supplied.
    • A dict profit of {2025: 0} records 2025, so a later 2027 profit reads 2025 (£2,000); with no key at all, it reads 2027 (0).
    • A dict profit for 2028 only, with a 2025 profit set later, reads 2027 (0). The same inputs, all set by set_input, read 2025.
  • Fix:
    • freeze the fallback's years at first read, or record profits set while the record is empty;
    • fall back when no recorded year is at or before the year simulated.

2. build_from_populations recomputes the record from the donor's live input history (simulation.py:633-635).

  • Post-build writes travel. Take a donor built from a 2025 dict, with a 2027 profit set later. The donor itself reads 2025 (£2,000), but a simulation taking its populations records (2025, 2027) and gets 0. A self-rebuild after such a write moves the record without clearing caches.
  • Data-built donors record every projected year. So a situation taking a 2025 dataset's populations reads 2027 and gets 0, where the donor gets £2,000; round 5 gave £2,000. The reverse case, a data-built simulation taking situation populations, still reads its own dataset's first year, but that predates this commit.
  • Fix: copy the donor simulation's record when it has one, and use the first supplied year for a data-built donor. No package caller passes such populations, so this is low priority.

3. Texts:

  • utils/trading_allowance.py:144-145: "where income tax is the same either way" isn't always so.
    • With personal pension contributions limited by relevant UK earnings, the switch changes earnings, and so the relief. Take claimant profit of £10,000, property income of £40,000, contributions of £12,000 and an allowance of £3,000: a loss 1p larger lowers income tax by £400.
    • The same happens where the s. 24A cap binds, which needs an allowance above about £51,000.
    • The behaviour is the shared convention; only the sentence is new.
  • Omitted rules: the changelog and trading_allowance_deduction.py:24-26 leave out the empty-record fallback and the populations refresh.
  • Old wording: the data-year test docstring (:4-7) still says "for which the profit was supplied".
  • Fixture saving: the property test says the change "halves the fixture's cost"; it is about a third.

CI

  • Shard times: 66.8, 74.1, 75.4, 69.2, 68.4 and 68.3 minutes. Shard 3 was cancelled at 99%, with 1,949 of 1,961 tests passed and no failures.
  • Cause: shard 3's shared tests took 19% longer than at 7b25254f4, with the same tests. This PR's tests take about 4 s there, and other PRs' runs today also time out.
  • Rerun needed: a seventh shard is now what blocks this PR from merging.
  • Correction to the reply's figures: at 7b25254f4 this PR took about 4 s of shard 6, not 16 s, and about 29 s of data-year tests on shard 2, not 67 s. Either way, the share is small.

Round-5 items

# Status
1. Build-time record Partly. The empty-record fallback gives a bare situation its profit's year (£2,000). A profit set after a 2025 build stays at 2025, as documented. build_from_populations refreshes for situation donors. The residuals are in 1 and 2.
2. Texts Fixed. "Never jumps" is gone, ties are described correctly, and the recorded year is explained. Nits are in 3.
3. Covered side of the loss fix Fixed. The new YAML case and the near-tie mask change pin it: sending covered ties to ordinary treatment fails six YAML cases, two data-year tests and the property test. Correction to my round-5 suggestion: the new Class 4 monotonicity check doesn't catch that mutation, since Class 4 profits are flat across allowances there. It pins the below-expenses side instead.
4. Fixture Fixed. Changing a built simulation's parameters leaves parameters.baseline at law and leaks nothing between tests. The outputs match the reform route on every draw.

Checked

  • Grids: the allowance and loss grids are unchanged from round 5, cell by cell (108,360 cells, and over five million).
  • UNKNOWN status: still identical to the merge base (501 arrays × 2, with identical dtypes).
  • Earlier rounds: the round-1 to round-4 fixes hold.
  • Tests: the touched files pass: the YAML (49 cases), the data-year file (16 tests), the property file (4) and test_data_built_imputations.py (9). The tests that rebuild populations pass too.

🤖 Generated with Claude Code

CI: the Python shards' time limit goes from 75 to 90 minutes, and the
YAML job's, which its comment ties to theirs. On 1 November the shards
took 63 to 75 minutes on runners of varying speed, and the limit
cancelled shards with no test failing (runs 41303444320, 41313872613 and
41326415023).

A situation that gave no profit at its build records the years of the
first profit set afterwards, so a later write (a derivative's, a
branch's) no longer moves its data year; _data_year falls back to the
input history only where no recorded year is at or before the one
simulated. build_from_populations takes the donor simulation's record,
or a data-built donor's first year, instead of the donor's live input
history.

Texts: the reliefs docstring says which figures can step where the
election switches; the changelog, the deduction's and the input's
documentation and the data-year tests' docstring describe the recorded
year; the fixture's saving reads about a third.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Response to the sixth review at b2bbf8822

All three points are addressed in cbb8547, and the CI timeout is handled by raising the time limit. Unknown-status results still match the merge base, and the earlier rounds' cases and mutations still hold.

CI

As you found, shard 3 ran out of time at 99% with no failures: runner speed varies by more than this PR's share. Rather than wait for a seventh shard, this PR raises the limits:

  • the Python shards go from 75 to 90 minutes, with a dated comment citing the three runs;
  • the YAML job goes to 90 as well, since its comment ties its limit to the shards' (Result waits for them anyway).

A changed fragment records this, and the code-health tests that read the workflow pass.

Correction taken on my figures. They came from .github/test_durations.py, which times the gap between a worker's consecutive results, set-up included. Either way the share is small.

1. The fallback path no longer moves the data year

  • First profit after a bare build. A situation that gave no profit at its build now records the years of the first profit set afterwards (Simulation.set_input). Later writes leave that record, as they leave a built one's.
    • Your bare situation keeps £2,000 and £4,558.85 after calculate_derivative, which now returns 0.80, as the dictionary build does.
    • A branch that sets a 2027 profit keeps £2,000.
  • The fallback's condition. _data_year now falls back to the input history when no recorded year is at or before the one simulated, not only when the record is empty. A dictionary profit for 2028 alone, with a 2025 profit set later, now reads 2025 (£2,000), as an all-set_input build does.
  • A zero profit still counts. A dictionary profit of {2025: 0} records 2025, so a 2027 profit set later reads 2025: the situation gave a profit for 2025, and a profit set after the record doesn't move the year. The docs say so.

2. build_from_populations brings the donor's record

Populations taken from another simulation now bring its record. For a data-built donor they bring its dataset's first year, or the earliest supplied profit year for data without years. Fresh populations still record what they supply, and a simulation rebuilding from its own populations keeps its record.

Both of your cases are tested:

  • a donor built from a 2025 dictionary, with a 2027 profit set later, now gives the taker £2,000, as it gives itself;
  • a situation taking a 2025 dataset's populations reads 2025 (£2,000).

3. Texts

  • The reliefs docstring no longer says income tax is the same either way at the switch. The election weighs the loss against the saving in taxable trade profit, so other figures can step where it switches, under the convention every status shares:
    • Class 4, where losses brought forward absorb that saving;
    • income tax itself, where a relief reads trade profit (pension relief limited by relevant UK earnings, or the s. 24A cap).
  • The recorded year is now described in the changelog, the deduction's and the input's documentation, and the data-year tests' docstring: when the situation was built, first given a profit, or took another simulation's populations, plus the fallback.
  • The fixture saving now reads "about a third".

Checks

Run in the locked environment (uv run --frozen --extra dev):

  • YAML: policyengine-core test on all of policyengine_uk/tests/policy: 5,459 passed.
  • Python: pytest on the claim-status files, test_data_built_imputations.py, the CI code-health tests, the aggregate-impact tests, the scenario, modifier, input-record, branch, dataset-source and housing benefit accommodation-date tests, and the related trading allowance, Chapter 8, trading loss, Class 4, income concept, parameter and metadata tests: 797 passed and 11 skipped, among them the 126 CI code-health tests that read the workflow. Ten skips need the dataset token and run in CI: nine certified-dataset aggregate-impact tests and one scenario-baseline test. The eleventh is test_build_metadata.py's bundle-contract test, on CI's allowlist.
  • Mutations: each of this round's fixes, re-applied as the old rule, fails tests meant for it:
    • without recording the first profit set after a bare build, the derivative and branch tests fail;
    • without the fallback when no recorded year is early enough, the 2028-only test fails;
    • recomputing a donor's populations from the input history fails both donor tests.
  • Lint: ruff format and ruff check are clean, and the changelog fragment check passes.

Not run locally: the full Python suite, which CI runs.

🤖 Generated with Claude Code

@juaristi22
juaristi22 marked this pull request as ready for review November 1, 2026 18:42
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Seventh review of #2765 at cbb8547e2

This re-reviews cbb8547e2 against my sixth review at b2bbf8822. Since the PR is now ready for review, it adds a fresh pass over the whole PR, from the merge base, looking only for things no earlier round raised. Three reviewers:

  • one re-ran each earlier case and re-applied each fix's old rule as a mutation;
  • one read the commit for new defects;
  • one did the whole-PR pass.

I reproduced the two main new cases myself. Everything used synthetic situations and datasets.

Verdict:

  • Fixed: the round-6 cases. The derivative and branch cases on a bare situation, the 2028-only fallback, both build_from_populations cases and the texts are all fixed. CI is green, and unknown-status results still match the merge base.
  • Before merge, two things:
    • recording the first profit written after a bare build, the option from my round-6 list that was taken, makes the data year depend on write order (1). The other option avoids it.
    • a pinned or neutralised deduction still marks a claimant as taking the allowance, so the raise can increase their tax (2).
  • Also: the expense share matters outside cuts (3), the CI time limits are a repo-wide change (4), and smaller points (5).

New

1. Recording the first profit write makes the data year depend on write order (simulation.py:680-689; utils/trading_allowance.py:64-74).

The hook records only the years in the first write of self_employment_income after a bare build. That has three effects, none of them tested:

  • Several years written one call at a time.
    • Pay £30,000 and CLAIMANT for 2025, with a £5,000 profit for each year from 2025 to 2030. A dict build records all six years: in 2027 it gives no deduction, with income tax of £4,895.89.
    • The same profits written by set_input, one year at a time, record only 2025. In 2027 they give a £2,000 deduction, with income tax of £4,495.89: £400 a year of relief the identical inputs shouldn't get. I re-ran this one.
    • Writing 2027 first, or writing two people's profits in turn, gives different answers again. b2bbf8822 matched the dict build.
  • A modifier's or a derivative's write can be the first one.
    • On a bare situation, a no-op modifier that writes back a 2025 profit records 2025 for the reform but not for the baseline.
    • After both are given a 2027 profit and CLAIMANT, the reform reads 2025 (£2,000, £4,495.89) and the baseline 2027 (0, £4,895.89). That's a −£400 "impact" from a modifier that changes nothing: round 4's problem, back for bare situations.
    • A calculate_derivative run before any profit does the same. No package modifier writes the profit, so only user modifiers and derivative helpers reach this.
  • Writes still move the year on the new fallback path.
    • Take a dict profit for 2028 only, then a 2025 profit set later. 2027 reads 2025 (£2,000).
    • But calculate_derivative(..., year=2027) returns 0.40, where the all-set_input build gives 0.80. Afterwards 2027 reads 0 (£4,958.85) for good.
    • A branch that sets a 2027 profit reads 2027, while its parent reads 2025.
  • Fix: freeze the supplied years at the first read of _data_year, rather than at the first write. Multi-year writes then don't depend on their order, a construction-time modifier's year sits with the user's, and later writes stop moving the fallback. Only a derivative run before any profit is given would remain, and the docs could say so.
  • Docs: utils/trading_allowance.py:38-39 and the hook's comment (simulation.py:685-688) say a later write never moves the year, which isn't so until this is fixed.

2. A pinned or neutralised deduction still marks a claimant as taking the allowance (variables/gov/hmrc/income_tax/allowances/uses_trading_allowance.py:41-49).

  • Mechanism: at the merge base the flag came only from the two deduction variables, so it followed a pinned or replaced deduction. The new claimant term recomputes the election from the formula. So when a dataset pins trading_allowance_deduction, or a structural reform neutralises it, a flagged claimant is marked as taking the allowance and gives up a supplied current loss, while receiving no relief. policyengine-uk-data pins it on SPI records today, though the docs now say to flag rather than pin.
  • Example: 2026, pay £30,000, profit £2,500, a £1,500 loss, CLAIMANT, deduction pinned to 0. Income tax is £3,686 at £1,000 and £3,986 at £3,000, so the raise costs them £300. I re-ran this.
    • An unknown-status trader with the same pin gets £3,686 both ways.
    • Unpinned, the claimant gains £100.
    • With 2025 data run in 2027, the raise adds about £319.
    • Pinning the Chapter 8 deduction to zero, as its docstring once told datasets to, does the same through the casual-income part of the saving.
  • Fix: gate the claimant term on the deduction not being an input, or say in the input's docs that a flagged claimant must not have a supplied deduction.

3. The expense share matters outside cuts too.

  • Mechanism: for a claimant with a supplied current loss, the share sets their ordinary treatment at and above the input allowance. So a reform of the share alone, say to 0 for an "upper bound" run, moves income tax at current law: £3,686 to £3,886 for profit £2,500 with a £1,500 loss in 2026.
  • The PR body is wrong here: it says the share "only matters under a cut below the input allowance". The parameter's own description is right: "for a claimant with no current loss".
  • Invariant 4 is too broad: the property test's invariant 4 ("a smaller expense share never lowers a claimant's taxable trade profit") is false off its grid. At a £500 allowance with a £300 loss, share 0.3 gives taxable profit £3,200 and share 0 gives £3,000. It passes only because the grid is {0, 0.5, 1} × {0, 300, 500}. Income tax is the quantity that is monotone there.

4. The CI time limits are a repo-wide change (.github/workflows/test-suite.yaml:91-93, :188-194).

  • Python shards: the dated comment's facts hold. The cited runs were cancelled with no failing test, and nothing else depends on 75 minutes. But other PRs' shards were cancelled at 75 minutes on 1 November too, and this PR's shards are no slower than nine other runs. At this head the slowest shard took 71.4 minutes, which would have passed under the old limit.
  • YAML job: its limit was raised on a stale rationale. The comment still says every case's simulation is kept to the end (The YAML test job likely runs short of memory and hits its 45-minute limit #2741), but Release each YAML case's simulation and its file's reformed systems #2742, already in the merge base, released them. The job ran in 8.7 minutes here, so 90 has no basis.
  • Extra cost: editing test-suite.yaml also matches versioning.yaml's pull-request paths. So every push to this PR now runs a second full suite, "Release test", which adds roughly six runner-hours per push. It isn't a required check.
  • Suggestion: a change that affects every PR is easier for maintainers to weigh in its own PR.

5. Smaller:

  • set_model_value: it stores through Simulation.set_input before marking the value as the model's, so the hook records a value that supplied_input_periods excludes. A model-set 2027 profit then fixes the year. No package path writes this variable that way today.
  • Data-built donors in a chain: they pass on their dataset's first year.
  • Duplicated rule: the data-year rule for data-built simulations is written twice, in _data_year and in _profit_years_of (simulation.py:1109-1117).
  • Repeated work: uses_trading_allowance runs trading_allowance_reliefs three times per person-year.
  • Mutants that survive:
    • build_from_populations recomputing on a self-rebuild, so the reply's "keeps its record" is untested;
    • a data donor without years bringing its latest year (:1112-1117).
  • Texts:
    • the PR body still says income tax doesn't jump as the allowance or the loss moves, and "until the saving exceeds" the loss, where a tie takes the allowance;
    • utils/trading_allowance.py:1-2 says the module serves two variables, but now serves three;
    • utils/trading_allowance.py:149-150 contradicts :141-142 on ties;
    • ni_class_4_trading_loss.py:58-61 says a positive deduction means the allowance was chosen, but a claimant now chooses it with a deduction of 0 or less;
    • the _profit_years_of docstring (simulation.py:1103-1104) says a self-rebuild returns None, but the code returns the record;
    • the election's "saving in taxable trade profit" (:151) leaves out the relief on income not otherwise charged, which the compared saving includes.

Round-6 items

# Status
1. Fallback path Partly. The cases I reported are fixed. A bare situation given a 2025 profit keeps £2,000 after a derivative (0.80), branches keep £2,000, and the 2028-only case reads 2025. The new rule's own problems are in 1.
2. build_from_populations Fixed. Both of my cases now give £2,000. Hook, branch, core-built and detached-population donors behave consistently, and a self-rebuild keeps its record without a stale cache.
3. Texts Fixed. The reliefs docstring's pension-relief and s. 24A cases check out, and so do the changelog, the deduction's and input's documentation, and the test docstrings. The residuals are in 1 and 5.
4. CI Fixed, and green. The six shards ran 65.8 to 71.4 minutes. The scope points are in 4.

Checked

  • Unknown status: results are identical to the merge base and to round 6 (1,002 comparisons, with identical dtypes).
  • Grids: the allowance and loss grids are unchanged, cell by cell.
  • Earlier rounds: the round-1 to round-5 cases still hold.
  • Performance: the hook doesn't run for data builds; a 50,000-person dataset builds in the same time.
  • Signature: it matches policyengine-core's set_input.
  • Whole-PR checks:
    • the supplied-input registry and the aggregate-impact guard;
    • the branches that write the profit;
    • the uprating of the new input;
    • the new parameter's metadata;
    • the towncrier config;
    • the absence of any downstream caller of trading_allowance_reliefs.
  • Tests: the touched files pass: the data-year file (21 tests), the property file (4), the YAML (49 cases) and the CI code-health tests.

🤖 Generated with Claude Code

- A situation's data year falls back to the profit years supplied when
  the allowance is first calculated after construction, not at the first
  write, so multi-year writes don't depend on their order and later
  writes don't move it. Branches read their root simulation's record.
- uses_trading_allowance adds a claimant's election only where both
  deductions come from their formulas, not inputs or neutralised, and
  skips the reliefs where no one is a claimant with a profit.
- Share the data-built rule (data_profit_year), narrow the property
  test's share invariant to claimants with no current loss, and add
  tests for write order, modifiers, the fallback, set_model_value,
  self-rebuilds, data donors and pinned or neutralised deductions.
- Put the YAML job's time limit back at 75 minutes with a corrected
  comment; the Python shards stay at 90.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Response to the seventh review at cbb8547e2

Both points before merge are addressed in 6f05924, along with the expense share, the CI scope and the smaller points. Unknown-status results still match the merge base, and the earlier rounds' test cases still pass.

1. The data year no longer depends on write order

I took your fix. Simulation.set_input no longer records anything. Where the build's record has no year at or before the one simulated, _data_year reads the years a profit had been supplied for when the allowance was first calculated after the build, kept from then on. (The build's record holds the years a dictionary gave the profit for, or those that taken populations bring.) Your cases:

  • Several years one call at a time. £5,000 for each year from 2025 to 2030, set in ascending, descending or 2027-first order, gives no deduction and £4,895.89 of income tax in 2027, as the dictionary build does. Two people's profits set in turn match the dictionary in either order.
  • A modifier's write on a bare situation. With your no-op modifier, the reform and the baseline both read the 2027 profit as measured in 2027: no deduction, and the same income tax.
  • The 2028-only path. calculate_derivative(..., year=2027) returns 0.80, as the all-set_input build does, and 2027 still reads 2025 (£2,000) afterwards. A branch that sets a 2027 profit reads 2025, as its parent does.

Two details the fix needed:

  • Construction. Simulation.__init__ calculates the allowance itself, in the set-up of the July 2025 Universal Credit reform, before a user can set a profit. So calculations during construction read the years as they stand and don't freeze them. Without that, a dictionary giving only a 2028 profit froze 2028 at construction, and a 2025 profit set afterwards no longer counted.
  • Branches read the record of the simulation they were made from. So a branch made before the first calculation can't freeze its own writes.

What remains is documented in _data_year and the PR body: a write before that first calculation counts. Examples are a derivative's write on a situation not yet given a profit, or a write to a copy (clone, as policyengine-core's derivative makes one) before the original was calculated.

The docs now describe the new rule: _data_year, the input's and the deduction's documentation, the data-year tests' docstring, the changelog and the PR body. The hook and its comment are gone.

2. A pinned or neutralised deduction no longer marks a claimant

uses_trading_allowance now adds a claimant's election only where both deductions come from their formulas. Where either is an input (supplied, or set by the model) or neutralised, the flag follows the deductions applied, as at the merge base. Both deductions are now in SUPPLIED_INPUT_VARIABLES, which stored_as_input requires, as the housing benefit accommodation disregard does. Tests:

  • Your example. In 2026, a claimant with pay of £30,000, a profit of £2,500, a £1,500 loss and the deduction pinned to 0 now pays £3,686 at £3,000, as at £1,000, and as you found for an unknown-status trader with the same pin. The £3,000 case is in the YAML.
  • The Chapter 8 pin. In 2026 under £3,000, take a claimant with a profit of £500, capital allowances of £600, casual income of £1,000 and an £800 loss. Unpinned, they give up the loss for the casual-income relief, though their trade deduction is −£100. With that deduction pinned to 0, they keep the loss. Both are YAML cases.
  • A neutralised trading_allowance_deduction. It keeps the £1,500 loss under a raise to £3,000. YAML can't neutralise, so this is a Python test.

The input's documentation says this. A reform that replaces either deduction's formula isn't detected, and the flag's documentation says to replace the flag's formula too.

3. The expense share

  • The PR body was wrong. It no longer says the share only matters under a cut. It sets claimants' ordinary treatment, so it matters at any allowance for a claimant with a supplied current loss. A YAML case pins your example: £3,886 at a share of 0, against £3,686 at the default.
  • Invariant 4 now covers claimants with no supplied current loss. The docstring says why a loss breaks it, and the YAML cases check income tax there.

4. CI

  • The Python shards: I'm keeping them at 90 minutes in this PR.
  • The YAML job is back at 75 minutes, with its comment corrected: Release each YAML case's simulation and its file's reformed systems #2742 released the cases, and the job has since taken 8.7 to 9.3 minutes in six pull request runs on 1 November.
  • The Release test cost is now in the PR body: editing test-suite.yaml makes each push also run versioning.yaml's Release test suite.
  • The changed fragment now covers the shards only.

5. Smaller points

  • set_model_value. The hook is gone. The freeze reads supplied_input_periods, which excludes the model's values, so a 2027 profit the model sets first leaves 2025 as the year. That has a test.
  • Data-built donors in a chain pass on their dataset's first year. A test now pins that.
  • Duplicated rule. data_profit_year holds the rule for data-built simulations, and _data_year and _profit_years_of both use it. supplied_profit_years replaces Simulation._supplied_profit_years.
  • Repeated work. uses_trading_allowance now works out the reliefs only where someone is a claimant with a profit. So unflagged data and situations don't run them a third time.
  • Surviving mutants. A self-rebuild that recomputes its record now fails a self-rebuild test. A data donor without years that brings its latest year fails a donor test, and the core-data test too, since the rule is now shared.
  • Texts:
    • the PR body no longer says income tax doesn't jump, and it says ties take the allowance;
    • the module docstring names all three variables;
    • the reliefs docstring says a claimant with a loss claims it only where it's worth more than the allowance's saving, with ties going to the allowance. It also says the saving includes the relief on income not otherwise charged;
    • ni_class_4_trading_loss.py says a claimant can choose the allowance with a deduction of nil or less;
    • _profit_years_of's docstring says a self-rebuild keeps its record.

Checks

Run in the locked environment (uv run --frozen --extra dev):

  • YAML: policyengine-core test on all of policyengine_uk/tests/policy: 5,463 passed.
  • Python:
    • pytest on the claim-status files, test_data_built_imputations.py, the CI code-health tests and round 6's related tests: 965 passed and 11 skipped. This round adds the benefit cap, childcare, education, 2022-23 director National Insurance and Universal Credit rebalancing tests, for the registry and construction. The skips are the same as before: ten need the dataset token, and one is the bundle-contract test on CI's allowlist.
    • All of policyengine_uk/tests/code_health: 3,089 passed.
  • Unknown status: the results match the merge base's formula exactly over 400 people at five allowances, with the same dtypes.
  • Mutations: each of this round's fixes, re-applied as the old rule, fails tests meant for it:
    • recording the first profit written after a bare build fails the any-order, two-person, bare-modifier and model-set tests;
    • a fallback that reads the live input history fails the bare-build derivative test and the 2028-only derivative-and-branch test;
    • a branch that reads its own record fails both branch tests;
    • freezing the years during construction fails both 2028-only tests;
    • dropping the deduction gate fails both pinned YAML cases and the neutralised test;
    • gating the trade deduction alone fails the casual-income pin;
    • ignoring neutralisation fails the neutralised test;
    • a self-rebuild that recomputes its record fails the self-rebuild test;
    • a data donor without years that brings its latest year fails the donor and core-data tests.
  • Lint: ruff format and ruff check are clean, and the changelog fragment check passes.

Not run locally: the full Python suite, which CI runs.

🤖 Generated with Claude Code

test_household_subset requires every walk up the parent_branch chain to
be reviewed. branch_root reads the years a profit was supplied for and a
dataset's first year, which are the same for every household, so a
household subset reads the whole population's data year and no formula
needs listing in WHOLE_POPULATION_VARIABLES.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Eighth review of #2765 at c38995151

This re-reviews 6f05924a3 and c38995151 against my seventh review at cbb8547e2. As before:

  • one reviewer re-ran each earlier case and re-applied each fix's old rule as a mutation;
  • a second read the commit for new defects.

Both found the main new defect (2) independently, and I re-ran it. Everything used synthetic situations and datasets.

Verdict:

  • Fixed:
    • the write order, the round-7 pinned-deduction cases, the expense share, the YAML job's limit and the smaller points;
    • unknown-status results still match the merge base exactly;
    • c38995151 registers branch_root, which clears the one CI failure at 6f05924a3 (1). CI is running on it.
  • Before merge: the pin gate mixes the two treatments (2). A new YAML case expects a result neither treatment gives, and pinning a deduction to the value its own formula gives changes income tax.
  • Also: branches no longer read their own profit (3), policyengine-core's derivative (4), pins that still cost a loss (5), untested parts of the new rule (6) and texts (7).

CI

1. branch_root is now registered.

  • The failure at 6f05924a3: test_household_subset.py::test_every_walk_up_to_an_ancestor_simulation_is_reviewed failed in shard 4/6 of both Test and Release test, the only failure of 11,469 tests. The new branch_root walks parent_branch but wasn't in REVIEWED_ANCESTOR_WALKS.
  • c38995151 adds it with its reason, and that's the right fix:
    • Its callers, _data_year and Simulation._profit_years_of, read only the root's record of years, never a household's values. Reading the root is what keeps a subset's year equal to the whole population's.
    • With property-allowance subsets actually made, a claimant measured in 2025 and simulated in 2027 gets £2,000 and £4,558.85, both alone and beside two electing landlords.
    • The one side effect is a write: a branch's calculation can freeze the root's years (3).
  • Status: the test passes locally at c38995151, and CI is running there.

Before merge

2. The pin gate mixes the two treatments (variables/gov/hmrc/income_tax/allowances/uses_trading_allowance.py:66-74; utils/trading_allowance.py:350-372).

  • Mechanism:

    • Where one deduction is an input or neutralised, the gate drops the claimant's election from the flag only.
    • The other deduction still comes from its formula, which routes the claimant by that same election.
    • So the flag reads ordinary treatment and the loss is claimed, while that deduction stays on the allowance route. Full relief makes a relevant trade's current loss nil (s. 783AF), so no claim gives both.
  • The new YAML case expects it (trading_allowance_claim_status.yaml:835-856).

    • Inputs: 2026, a raise to £3,000, pay £30,000, profit £500, capital allowances £600, casual income £1,000, an £800 loss, CLAIMANT, and the casual-income deduction pinned to 0.
    • At the head: the trade deduction stays at −£100, which nils the trade profit, and the loss is claimed: £3,526, as the case expects.
    • Either treatment throughout: ordinary treatment gives £3,626 (a trade deduction of −£600), and the allowance gives £3,686.
    • The comment: "rather than given up for nothing" misses the £500 of allowance relief the trade deduction still gives.
  • Pinning to the formula's own value changes tax. 2026 at current law, an expense share of 0.5, pay £30,000, profit £5,000, a £400 loss, CLAIMANT:

    • unpinned, the loss is given up, and income tax is £4,486;
    • with trading_allowance_deduction pinned to 0, the value its formula gives, the flag turns false and the loss is claimed: £4,406;
    • ordinary treatment would give £4,506.

    A set_model_value pin, or a casual-income deduction pinned to its own nil, does the same. I re-ran both cases.

  • Reach: it takes a pinned or neutralised deduction, a claimant whose supplied current loss doesn't exceed the allowance's saving, and a share below 1, capital allowances or casual income.

  • The docs invite this pin: the casual-income deduction's docs still say "Datasets that hold this income already net of the allowance should set this deduction to zero" (miscellaneous_income_trading_allowance_deduction.py:29-31). Take a synthetic dataset holding that column at 0 for a claimant with no casual income, so the pin changes no value. With 2025 data run in 2027, the raise lowers their income tax by £400, not the £293.70 it gives unpinned.

  • Fix:

    • Let the election weigh the deductions actually applied: where a deduction is an input or neutralised, use its applied value in claimant_saving and allowance_saving, and route the other deduction by that election. Keep the claimant term in the flag.
    • A pin to a formula's own value then changes nothing, and the YAML case gives £3,626.
    • Failing that, don't ship the £3,526 expectation, and reconcile the casual-income deduction's advice with the input's.

New, not blocking

3. Branches no longer read their own profit (utils/trading_allowance.py:118). _data_year reads the root's record and first-read years for every branch.

  • Profit given only on a branch:
    • Take a bare root with pay £30,000, and give a branch a 2025 profit of £5,000 and CLAIMANT.
    • In 2027 the branch gets no deduction and pays £4,958.85, where a simulation or a dict build gives £2,000 and £4,558.85.
    • cbb8547e2 gave £2,000.
  • A branch taking other populations: build_from_populations writes a record the branch never reads. With a donor whose profit was given for 2027, the branch gives £2,000 and £4,495.89, where the donor gives £0 and £4,895.89.
  • A branch's calculation freezes the root:
    • Take a dict profit for 2028 only. A branch that calculates 2027 first freezes the root at 2028.
    • A 2025 profit set on the root afterwards then reads as measured in 2027: £0 and £4,958.85, against £2,000 and £4,558.85 without the branch's calculation.
  • Reach: library users only. The package's only branch write of the profit is Pension Credit's, which zeroes it. policyengine.py, the API and the dashboard don't make branches.
  • Fix: where the root has no year at or before the one simulated, fall back to the branch's own record and first read. Document the third case. Neither _data_year's "Not handled" list nor the PR body names these.

4. policyengine-core's derivative hits the documented residual at full size.

  • Mechanism: core's Simulation.derivative clones, writes the perturbed input on the clone, and only then calculates the original (core simulations/simulation.py:3029-3035). So on a situation given its profit by set_input, the clone counts its own write.
  • Effect:
    • With a 2025 profit of £5,000, pay £30,000 and CLAIMANT, derivative("income_tax", "self_employment_income", 2027) as the first calculation returns 400.20.
    • The dict build, or the same call after one prior calculation, returns 0.20. I re-ran this.
    • This package's calculate_derivative calculates first, so it gives 0.20.
  • Reach: nothing calls .derivative( in this package, policyengine.py, the API or the dashboard.
  • Fix: a cheap one is for Simulation.clone to freeze the original's years first, unless construction is still under way.
  • Correction to my round 7: I said only a derivative run before any profit is given would remain. Core's order also makes a derivative run before the first calculation count its own write. The reply's description of the residual is the right one.

5. A pin can still cost a claimant their loss.

  • Mechanism: with the trade deduction pinned to 0, as policyengine-uk-data does on SPI records, the casual-income deduction's election still counts the trade relief the pin removed. A positive casual-income deduction then sets the flag.
  • Example: a claimant with profit £500, casual income £600 and an £800 loss pays £3,546 at £1,000 and £3,586 at £3,000.
  • Not new: unknown status does the same, by the merge base's design, so this predates the PR.
  • Texts: the reply's "a pinned or neutralised deduction no longer marks a claimant" and the PR body's "each leaving the claimant their loss" go further than the code.
  • The fix in 2 covers this.

6. Untested parts of the new rule. These mutants survive:

  • freezing with no profit supplied (the years and check at utils/trading_allowance.py:76);
  • freezing only where the build's record fails (the comment at :122-123);
  • build_from_populations dropping the donor's first-read years;
  • branch_root stopping one level up, or _profit_years_of skipping it;
  • dropping the baseline's reset at simulation.py:481, after which a baseline without a scenario never freezes;
  • the gate reading supplied values only, though the docs say model-set values count.

Also:

  • simulation.py:712-713 do nothing, since core's clone copies every attribute.
  • The donor-chain test covers data → situation → situation, not round 7's situation → data → situation.

7. Texts:

  • Modifiers: a scenario modifier runs inside __init__ (simulation.py:488-489), before the flag clears at :502. So on the fallback path its write counts, and the new modifier test asserts that its year joins. But three texts say a modifier's write doesn't move the year: _data_year (utils/trading_allowance.py:96-100), the input's docs (trading_allowance_claim_status.py:29-33) and the PR body. That holds on the build-record path only.
  • "The deductions applied are not its": this is false for the deduction still computed by its formula (2). It appears in the input's docs (:65-68), uses_trading_allowance.py:29-35, the changelog ("as for everyone") and the PR body.
  • The PR body's sixth-round mutation line ("without recording the first profit set after a bare build…") describes code that no longer exists.
  • ni_class_4_trading_loss.py:58-63: it says a claimant counts when they choose the allowance with a deduction of nil or less, which no longer holds where a deduction is pinned.

Round-7 items

# Status
1. Write order Fixed, apart from the documented residual. Profits set one year at a time, in any order, give no deduction and £4,895.89 in 2027, as the dict build does. So do two people set in turn. The no-op modifier's impact is 0.00. The 2028-only derivative returns 0.80, and branches read 2025. Each old rule, re-applied, fails the tests the reply names.
2. Pinned deduction Partly. The round-7 cases are fixed. Income tax is £3,686 at both £1,000 and £3,000. 2025 data run in 2027 now moves by £0.00 under the raise, where round 7 measured about +£319. The neutralised deduction keeps the loss. But see 2 and 5.
3. Expense share Fixed. A share of 0 gives £3,886, against £3,686, and the YAML pins it. Invariant 4 now holds off its grid for claimants with no current loss, and its docstring says why a loss breaks it.
4. CI The YAML job is back at 75 minutes with a corrected comment, and the PR body states the Release test's cost. The shards stay at 90; that's the maintainers' call.
5. Smaller Fixed. That covers set_model_value, the shared data-year rule, the repeated work (the reliefs now run only for claimants with a profit), both surviving mutants and the listed texts.

Checked

  • Unknown status: identical to the merge base on 5,760 people, six allowances and eight variables: 368,640 cells, with no difference in value or dtype.
  • Construction: __init__ can't return with the flag still set, and nothing calculated during construction survives it. A bare build given its inputs afterwards matches the dict build for income tax, Class 4 and Universal Credit in every year from 2025 to 2030, in eight cases.
  • The registry: adding the two deductions changes nothing else. Its only other readers are Simulation.calculate's record purge and the aggregate-impact guard, which reads partition components only.
  • The short-circuit returns a bool array in mixed, claimant-free and subset populations.
  • Tests: the touched files pass: the data-year file (30 tests), the property file (4), the YAML (53 cases, through policyengine-core test) and the code-health files (280). At 6f05924a3, CI's other 11,468 tests gave 11,405 passed, 61 skipped and 2 xfailed.

🤖 Generated with Claude Code

juaristi22 and others added 2 commits November 2, 2026 11:13
- A situation's data year is the year simulated, as the State Pension
  variables read it; datasets keep their first year. This removes the
  record, the first-read freeze, the construction flag and the branch
  walk, so a situation's data year no longer depends on how, when or
  where its profit was written. The limit is documented: a profit given
  for an earlier year and carried forward is read as net of the later
  year's allowance.
- A claimant's election weighs a pinned or neutralised deduction at the
  value applied, and the other deduction follows that election, so a pin
  to the formula's own value changes nothing and a positive pin means
  the allowance applies. uses_trading_allowance keeps the claimant term.
- Tests: the data-year file is rewritten for the new rule, with a grid
  test that pinning either deduction to its own value changes nothing
  for every status; YAML cases for round 8's pins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vahid-ahmadi

Copy link
Copy Markdown
Collaborator

Review of #2765 at 8d3a4d4: I checked the rules against ITTOIA 2005 Part 6A Chapter 1, the formula's edge cases and the Class 2/4 knock-ons, and ran the new YAML file (56 passed), the two new Python test files (22 passed), scratch situations and an UNKNOWN-status differential against the merge base.

Must

None found.

Should

  1. A NON_CLAIMANT with no profit loses partial relief on casual income; a CLAIMANT with no profit does not. policyengine_uk/utils/trading_allowance.py:216-217 gates claimant on profit > 0, but not non_claimant. The docs say the status "describes a positive profit", so both should fall back to UNKNOWN there.

    • Case: 2026, £30,000 pay, self_employment_income 0 (or -500), casual income £2,500 (miscellaneous_income = income_not_otherwise_charged).
      • UNKNOWN or CLAIMANT: casual-income deduction £1,000, income tax £3,786.
      • NON_CLAIMANT: deduction £0, income tax £3,986.
    • This changes baseline results as soon as a dataset flags loss-making traders. microcosm#1439 plans to mark "other traders as NON_CLAIMANT".
    • Fix: non_claimant = trading & (status == status.possible_values.NON_CLAIMANT). Add a YAML case with no profit and casual income above the allowance. The current no-trade NON_CLAIMANT cases (lines 382-405) only cover casual income at or below the allowance. The property test's reference (test_trading_allowance_claim_status_properties.py:178) has the same ungated non_claimant, so it needs the same change. If the asymmetry is meant, say so in the variable docs and pin it with that case.
  2. The PR body describes the situation data-year rule that 870b07f3 replaced. The code and docs now read a situation's profit as measured in the year simulated (_data_year, trading_allowance.py:69-71). The body still says a situation uses the latest year its profit was supplied for, and that "a 2025-26 profit simulated in 2027-28 deducted £1,000 and gains £2,000". At the head it gains nothing:

    • Case: £30,000 pay, self_employment_income: {2025: 5000}, CLAIMANT, simulated in 2027: deduction £0.
    • The test counts are also stale (53 YAML cases, now 56; 30 data-year tests, now 18), and several mutation rounds refer to recording code that no longer exists.
    • Fix: rewrite the "Input allowance" and "Design choices" sections and the test list to match the head. A reviewer reading the body alone would expect the opposite result for a carried profit.
  3. The CI time limit change belongs in its own PR. .github/workflows/test-suite.yaml:190-196 raises every Python shard's limit from 75 to 90 minutes. That is a repo-wide change with its own changelog fragment (2764.changed.md). It also triggers the Release test suite on each push (about six more runner-hours, by the body's own estimate). Locally the two new Python files take about 2 minutes together, so they are not obviously what pushes shards past 75. Splitting it out lets it be judged on its own runs, and keeps this PR's diff to the trading allowance.

Nit

  1. The same rules are written out in about eight places. The module docstring (trading_allowance.py:81-186), trading_allowance_claim_status.py:16-68, the docs of trading_allowance_deduction, uses_trading_allowance, the casual-income deduction, income_not_otherwise_charged and self_employment_gross_receipts, and both parameter descriptions. Any later change to the rule has to be made in every one. Suggest keeping the full rule in the module docstring and a one-line pointer elsewhere.

  2. The statutory parameter's description now explains a modelling convention. parameters/gov/hmrc/income_tax/allowances/trading_allowance.yaml:1 adds a sentence about claimants and the expense share. That text shows wherever the parameter is described. Suggest keeping it to the statutory amount and leaving the pointer in gov.simulation.trading_allowance_claimant_expense_share.

  3. changelog.d/2764.added.md is a single 255-word line. One or two sentences would read better in the release notes, with the detail left in the docs.

What I checked and found fine

  • Partial relief. A claimant's receipts are profit plus the input allowance, so a raise relieves up to the new allowance less the input allowance, capped at profit. Full relief applies where profit plus input is within the new allowance. Examples: £5,000 at £3,000 on 2026 law → deduction £2,000; £1,500 → £1,500 (nil profit). Capital allowances are given back, as s. 783AI requires.
  • Unknown status is exactly neutral. On 300 random people per run (profits around the boundaries, supplied receipts, capital allowances, casual income, losses brought forward, supplied current losses), with half set explicitly to UNKNOWN, I ran 2026, 2027 and 2029 at the default allowance and at £0, £500, £3,000 and £5,000. The trade deduction, casual-income deduction and uses_trading_allowance were identical to the merge base's formula in all 15 runs.
  • Non-claimants get full relief only, never partial relief after a raise. This matches how HMRC split the measure. The AB26 policy costings (p. 23) cost existing maximum claimants statically, and count expense claimers who switch as a behavioural response. So the engine's static figure compares with the static part, not with the full published line.
  • Property allowance. Untouched: it is Part 6A Chapter 2's separate allowance, and the PR does not read or change it.
  • NICs follow. Class 4 profits subtract trading_allowance_deduction, Class 2 reads Class 4 profits, and a claimant's negative deduction under a cut raises both consistently.
  • The parameter. £1,000 from 2017-04-06 and £3,000 from 2027-04-06, with Budget 2026 references. The input allowance is read from parameters.baseline on 30 April of the data year, so reforms do not move it. A dataset's first year (min(dataset.years), as the State Pension variables do) gives £1,000 for any base before 2027-28.
  • The expense share. It is clamped to [0, 1], and its default of 1 makes a cut add nothing for claimants: a stated lower bound, not an estimate. The cut and abolition cases, and the ties at equal expenses, are covered in the YAML.
  • Code. Fully vectorised, with no hard-coded policy values. A changelog fragment is present. The YAML arithmetic I checked by hand (Class 4 and income tax cases) is correct. Of the +2,217 lines, about 1,770 are tests and about 235 are the utility module, roughly half of that comments and docstring; the rest is variable docs, parameters and CI.
  • Data side. I found nothing in PolicyEngine/microcosm that sets trading_allowance_claim_status yet. microcosm#1439 (open) tracks storing it once this is released.
    • Its interim rule calibrates claimant rates to HMRC's 400,000 gainers at about £380. If that count includes switchers, the static claimant share would be overstated. The costing note does not say whether it does.

CI at 8d3a4d4: 21 checks pass (lint, changelog, YAML tests, release build and installs, docs). The 12 Python shards (Test and Release test) and Aggregate impact were still pending when I posted.

Verdict: the engine logic is sound and UNKNOWN is neutral. Item 1 needs a fix or an explicit decision before the data side starts flagging traders, and the PR body should be brought up to date with the head.

🤖 Generated with Claude Code

- A non-claimant with no profit is read as unknown, as a claimant is:
  the status describes a positive profit. So their casual income above
  the allowance gets partial relief again; a YAML case pins it, and the
  property test's reference has the same gate. The non-claimant's full
  relief now comes only from supplied receipts.
- Keep the full rules in utils/trading_allowance.py. The deduction,
  casual-income deduction and uses_trading_allowance docs go back to
  main's text with a one-line pointer; income_not_otherwise_charged's
  docs and the statutory parameter's description go back to main's; the
  input's docs and the expense share's description are shorter; the
  changelog fragment is two sentences.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator Author

Response to the eighth review at c38995151

870b07f3b and 8d3a4d435 fix the pin gate (2) and change how a situation's data year works, which removes items 3, 4 and 6 at the source. CI was green at c38995151. Vahid's review then led to edda47ea2: a non-claimant with no profit is now read as unknown, as a claimant is, and the docs now give the rules in one place (his reply is separate).

A situation's data year is now the year simulated

Rounds 4 to 8 kept finding new cases in one mechanism: working out the year a situation's profit was measured from the history of profit writes. Each fix closed the reported cases and opened new ones, because writes by a user, a modifier, a derivative, a branch or a copy all look the same to the engine. So I've stopped inferring it.

  • The rule. Data-built simulations keep the dataset's first year, or for data without years the earliest year the profit was supplied for. Anything else, such as a situation, reads the year simulated, as the State Pension variables read a situation.
  • The machinery is gone. That covers the build record, the first-read freeze, the construction flag, branch_root and the population records. simulation.py and test_household_subset.py are back to main.
  • Costing is unchanged. Microsimulation reads a dataset, whose first year applies to every record.
  • Two limits, documented in _data_year and the PR body:
    • a situation's profit given for an earlier year and carried forward is read as net of the later year's allowance;
    • a simulation that takes a data-built simulation's populations is read the same way.
      Tests pin both limits, so a later change to them has to be deliberate.

Effect on this review's items:

  • 3 (branches): a branch reads the year simulated, as any situation does. Given a profit only on the branch, it gets what a simulation given it gets (tested). Nothing is frozen any more, so a branch's calculation can't change the root's later results.
  • 4 (core's derivative): it gives the same result as the first calculation and after one (tested).
  • 6 (mutants): the surviving mutants were all in the removed code.
  • 7 (modifier texts): nothing is inferred from writes any more, so the texts now say only what holds.

2. The election weighs a pinned deduction at its value

I took your fix, for claimants.

  • The rule. Where a deduction is an input (supplied, or set by the model) or neutralised, it applies at its own value whichever treatment the claimant elects. So trading_allowance_reliefs weighs it as the allowance route's value in the claimant's saving. The other deduction follows that election, and the claimant term stays in the flag.
  • A positive applied deduction means the allowance applies. Without that, a large loss could send the other deduction to ordinary treatment while the positive pin set the flag.
  • Your cases:
    • the YAML case now gives £3,626, ordinary treatment throughout, with a trade deduction of −£600;
    • pinning to the formula's own value changes nothing: £4,486 with trading_allowance_deduction pinned, with the casual-income deduction pinned, and with a set_model_value pin;
    • your dataset holding the casual-income deduction at 0: the raise lowers the claimant's tax by £293.70, as unpinned. It matches unpinned at losses of £0, £500, £1,500 and £2,500 too.
  • A grid test checks the property over 216 people of every status at three allowances and shares. Pinning either deduction, or both, by set_input or set_model_value, to the values their formulas give leaves the election, both deductions, taxable profit, Class 4 and income tax unchanged.
  • Unknown and non-claimant status keep the election from the formulas' values, as on main, pins included. Your item 5 notes that their pin interaction predates the PR, and changing it would move results on pinned data such as the SPI records. Applying the same rule to them is left for a separate change.
  • Texts: the reliefs' docstring holds the rule, and the input's and the flag's docs point to it. The casual-income deduction's advice is main's again: with the claimant's election weighing a pin at its value, it no longer conflicts with the input's.

5. Pins that cost a loss

For claimants, 2 covers this. Your example (profit £500, casual income £600, an £800 loss, the trade deduction pinned to 0) pays £3,546 at £1,000 and at £3,000, with the loss claimed. The £3,000 case is in the YAML. Unknown status is as on main, and the texts no longer claim more than that.

6 and 7: the rest

  • simulation.py:712-713 is gone, with the rest of the record.
  • The donor-chain test is gone: populations carry no record now.
  • The PR body's mutation section is gone: it listed earlier rounds' mutations of code that no longer exists.
  • ni_class_4_trading_loss.py:58-63 holds again: the claimant term is always in the flag.

Checks

Run in the locked environment (uv run --frozen --extra dev):

  • YAML: all of policyengine_uk/tests/policy at 870b07f3b: 5,466 passed. The claim-status file at edda47ea2: 57 passed.
  • Python: at edda47ea2, 3,176 passed. That covers the data-year and property files, the parameter and variable metadata tests, the package-scanning tests (including test_household_subset.py's review of ancestor walks) and all of tests/code_health.
  • Your cases: re-run by script and in the tests above.
  • Unknown status: identical to the merge base's formula at edda47ea2 (400 people at five allowances, the same dtypes).
  • Not run this round: mutation checks, and the full Python suite locally. CI is running the suite on edda47ea2.

🤖 Generated with Claude Code

@juaristi22

Copy link
Copy Markdown
Collaborator Author

Response to @vahid-ahmadi's review at 8d3a4d4

Thanks for the review. edda47ea2 addresses items 1, 2, 4, 5 and 6. Item 3 stays in this PR, for the reason below.

1. A non-claimant with no profit. Fixed as you suggested.

  • The gate. non_claimant now needs a positive profit, as claimant does, so either status with no profit is read as UNKNOWN.
  • Your case (2026, £30,000 pay, casual income £2,500) now gives a casual-income deduction of £1,000 and income tax of £3,786 for all three statuses, at a profit of 0 and of −500.
  • Tests. A YAML case pins it, and the property test's reference has the same gate. The two existing no-profit non-claimant cases keep their values, now as unknown.
  • One simplification. With the gate, the no-profit branch of the non-claimant's full-relief test could no longer apply, so I removed it. A non-claimant's full relief now comes only from supplied receipts.
  • For microcosm#1439, traders flagged NON_CLAIMANT with no profit now behave as UNKNOWN.

2. The PR body now matches the head:

  • the input allowance and design sections describe the year-simulated rule for situations and its two documented limits;
  • the test lists and counts are current (57 YAML cases, 18 data-year tests);
  • the mutation history, which described code that no longer exists, is gone.

I also added your calibration point to the data-side section: if HMRC's 400,000 includes switchers, calibrating static claimants to it would overstate them.

3. The CI time limit. I'd like to keep it here.

  • Why: on 1 November this PR's shards were cancelled at 75 minutes with no test failing (runs 41303444320, 41313872613 and 41326415023). At cbb8547e2, after a 90-minute limit was set, they ran 66 to 71 minutes. Without the change, this PR could be cancelled again on a slow runner until a separate PR merged.
  • What it touches: one hunk in test-suite.yaml, with its own changed fragment, so it can be reverted on its own.
  • You're right on two points: the slowness is suite-wide rather than these tests, and each push to this PR also runs the Release test suite. If you'd still rather judge it separately before merging, I'll split it out.

4. The rules written out in many places. The full rules now live only in utils/trading_allowance.py.

  • Back to main's text plus a one-line pointer: the docs of trading_allowance_deduction, the casual-income deduction and uses_trading_allowance.
  • Back to main's text exactly: income_not_otherwise_charged.
  • One sentence: self_employment_gross_receipts keeps one, that a claimant's status overrides supplied receipts.
  • Shorter: the input's own documentation is cut to what a data producer needs.

5. The statutory parameter's description is back to main's. The convention is described only in gov.simulation.trading_allowance_claimant_expense_share, now shorter.

6. The changelog fragment is two sentences.

Checks at edda47ea2, in the locked environment:

  • the claim-status YAML: 57 passed;
  • pytest: 3,176 passed, covering the data-year and property files, the parameter and variable metadata tests, the package-scanning tests and all of tests/code_health;
  • ruff and the changelog check: clean.
    CI is running.

🤖 Generated with Claude Code

@vahid-ahmadi

Copy link
Copy Markdown
Collaborator

Round 2 at edda47ea2: I re-ran the round-1 cases, the claim-status YAML (57 passed) and the two new Python files (22 passed), a mutation of the new gate, and the UNKNOWN-status differential against the merge base. I also read the 8d3a4d4..edda47ea2 diff.

Round-1 items

  1. Fixed. utils/trading_allowance.py:217 now gates non_claimant on a positive profit. In 2026, with £30,000 pay, casual income £2,500 and a profit of 0 or −£500, UNKNOWN, CLAIMANT and NON_CLAIMANT all give a casual-income deduction of £1,000 and income tax of £3,786. The new YAML case (trading_allowance_claim_status.yaml:407) and the property test's reference (test_trading_allowance_claim_status_properties.py:179) pin it: reverting the gate in the engine fails 3 YAML cases and the property test. Dropping the no-profit branch of the non-claimant's full-relief test (trading_allowance.py:309) changes nothing, since that branch can no longer be reached.
  2. Fixed. The body now describes the year-simulated rule for situations and its two limits, and its counts match the head (57 YAML cases, 18 data-year tests). Its carried-profit example also matches: a 2025 claimant profit of £5,000 simulated in 2027 gets a deduction of £0.
  3. Accepted. The reason holds. At this head, Release test Python 1/6 took 77 minutes, so a 75-minute limit would have cancelled it. The other shards took 63 to 73 minutes. The change is one hunk (.github/workflows/test-suite.yaml:196) with its own fragment, so it can be reverted on its own. I'm dropping this item.
  4. Fixed. The full rules are now only in utils/trading_allowance.py. The variable docs point to it, and income_not_otherwise_charged is back to main's text.
  5. Fixed. parameters/gov/hmrc/income_tax/allowances/trading_allowance.yaml is back to main's text. The convention is described only in the expense-share parameter.
  6. Fixed. changelog.d/2764.added.md is now two sentences (77 words).
  • The 400,000 gainers question is now in the body's data-side section, as a caveat for microcosm#1439.

New findings

None. The diff only adds the gate, removes the branch the gate makes unreachable, and shortens the docs.

Unknown status: still identical to the merge base. This used the same randomised setup as round 1: 300 people per run, with half set explicitly to UNKNOWN, in 2026, 2027 and 2029, at the default allowance and at £0, £500, £3,000 and £5,000. In all 15 runs the trade deduction, the casual-income deduction, uses_trading_allowance, income tax and Class 4 matched exactly.

CI at edda47ea2: every check passes, including all 12 Python shards (Test and Release test) and Aggregate impact, which reports 0 checks broken and no moves flagged against the merge base.

Verdict: all round-1 items are fixed or accepted, and I have nothing further on this PR.

🤖 Generated with Claude Code

@juaristi22
juaristi22 added this pull request to the merge queue Nov 2, 2026
Merged via the queue into main with commit be73a74 Nov 3, 2026
38 checks passed
@juaristi22
juaristi22 deleted the claude/trading-allowance-claimant-2764-1294f6 branch November 3, 2026 01:20
MaxGhenis added a commit that referenced this pull request Nov 3, 2026
Keeps both names where #2765 and this PR each added one to SUPPLIED_INPUT_VARIABLES in policyengine_uk/utils/supplied_inputs.py: lifetime_isa_countable_capital (this PR) and miscellaneous_income_trading_allowance_deduction (#2765).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Let datasets mark trading allowance claimants, so allowance reforms reach their base without turnover data

2 participants