[MaxGhenis]

The failure

policyengine_us/tests/core/test_mt_state_agi_invariants.py::test_drawn_batches failed in Rest Python shard 3 of run 40505543884. That run tested #10402 merged onto main cbd6fb4a9e (12:39Z on 10/27), just after #10392 merged at 12:34Z. Property 3 ("Pooling") expected state AGI 12,000 and got 7,600. The shrunk draw is ten 2021 Montana single filers aged 25, nine with no income and one whose only income is a 12,000 pension (@reproduce_failure('6.168.3', b'AXicY3RkQIWMg04AygIiAAsPFJ0=')).

While this PR was in review, #10496 (11a9dee7da) fixed main by excluding those units: property 3's mask, and the seeded grid's joint_path guard, now also require mt_pension_exemption_indiv and mt_pension_exemption_joint to be zero. So on main, a unit that claims the exemption is no longer checked by property 3. This PR replaces that exclusion with a check of the exemption's effect: those units stay in property 3, and their Montana AGI must equal max(federal AGI − mt_pension_exemption_joint, 0). It also adds the shrunk draw as an @example, a guard that the new term is never vacuous, and a pension-only filer in the seeded grid. The branch is merged with main (merge commits 56ec6ef3e0 and 7a91e5efd0), so the diff against main is this change alone.

The model is right; the test was wrong

#10392 subtracts Montana's partial pension, annuity, and IRA income exemption outside mt_subtractions. mt_agi_joint adds mt_pension_exemption_joint (the spouses' combined amount, reduced once by the joint federal AGI) to the pooled subtractions. mt_agi_indiv adds each spouse's mt_pension_exemption_indiv (reduced by that spouse's own federal AGI). So mt_subtractions stays 0 for a pension-only filer, and property 3's unadjusted filter as it stood before #10496 (mt_additions == 0 & mt_subtractions == 0 & social_security == 0) treated that filer as unadjusted and expected federal AGI.

I checked the model against #10392's cited sources:

• The exemption reduces Montana AGI on Form 2 for 2021–2023. MCA 15-30-2110(2) (2021): "adjusted gross income does not include the following, which are exempt from taxation"; (2)(c) is the pension and annuity exemption. On the 2021, 2022 and 2023 Form 2, the exemption schedule's line 7 goes to the Subtractions Schedule (line 35 in 2021, line 34 in 2022–2023). That schedule's total goes to page 1 line 13, and line 14 (Montana AGI) is line 11 plus line 12 minus line 13.
• The 2021 cap and phase-out. The 2021 Form 2 schedule's line 1 is 4,400 and line 5 is 36,700. Line 6 is twice the federal AGI above line 5. The 2021 instructions (line 35) say the same and give 38,900 as the AGI where the exemption reaches zero. The parameters match: cap.yaml has 4,400, phase_out/threshold.yaml 36,700 and phase_out/rate.yaml 2.
• There is no age condition. Neither 15-30-2110(2)(c) nor the schedule has one; contrast (2)(b), the interest exemption for taxpayers aged 65 or older. The definition in MCA 15-30-2101(25)(a) covers "systematic payments of a definitely determinable amount from a qualified pension plan ... paid to the recipient or recipient's beneficiary upon the cessation of employment", with no age. The premature-distribution exclusion in (25)(d) applies to IRA and deferred-compensation distributions, and sources.yaml already notes it is not separately modeled.

For the drawn filer that gives min(4,400, 12,000) = 4,400, no reduction (federal AGI 12,000 is below 36,700), and Montana AGI 12,000 − 4,400 = 7,600. That is what PE computes.

The change

Only policyengine_us/tests/core/test_mt_state_agi_invariants.py changes, plus this PR's changelog fragment. Relative to main:

• Property 3 drops Check Montana's pension exemption in the Montana AGI pooling test #10496's two zero-exemption conditions from its mask and nets the exemption in its expected value instead. Through 2023, a unit on the joint path with non-negative federal AGI and no additions, subtractions or Social Security must report max(federal AGI − mt_pension_exemption_joint, 0); for a unit with no exemption that is federal AGI, as before. The docstring states this, describes the exemption and its source, and keeps Check Montana's pension exemption in the Montana AGI pooling test #10496's explanation, credited, that the exemption is outside mt_subtractions.
• Run outputs. Unchanged: Check Montana's pension exemption in the Montana AGI pooling test #10496 already reads mt_pension_exemption_joint (tax unit) and mt_pension_exemption_indiv (person, summed to the tax unit).
• Pooling guard. Both PRs require no pension exemption on either path for the joint_path witness; one copy of that condition is kept, on top of property 3's mask (run["pooled"]), which selects the same units as main's spelled-out mask. The exemption is reduced by each spouse's own AGI on the separate path and by the joint AGI on the joint path, so the witness gap must come from pooling alone. The existing witnesses (age-40 couples with a self-employment loss) have no pension, so the guard still holds.
• New guard. Through 2023, some pooled unit must have a nonzero pension exemption, so the new term is never vacuous. The seeded grid had no pooled pension unit: its pensioners are the age-70 units, which also have Social Security. It now includes a single filer aged 40 whose only income is a 12,000 pension.
• Explicit example. The shrunk CI draw is an @example on test_drawn_batches (year 2021). A derandomize=True test is seeded from function_digest(test), which hashes the test's source with decorators and comments removed (hypothesis/internal/reflection.py, _clean_source), so the @example adds a case without changing the random draws. The superset check below confirms it: this PR's five random draws equal main's.

max_examples and the other settings are unchanged.

Why #10392's own CI passed

This section separates what was shown from what is inferred.

Shown from the two CI runs (#10392's, and #10402's run 40505543884). Both used CPython 3.14.8, Hypothesis 6.168.3 and the ci profile (database=None, derandomize=True). The test file was identical, and so was the core-1 process's list of 116 test files. The MT variables and parameters were also identical between the two merge trees (git merge-tree of c81ca258 onto 0e7ced1747, and of 0e66a950 onto cbd6fb4a9e). With derandomize=True, Hypothesis 6.168 seeds the test's Random from function_digest(test) (get_random_for_wrapped_test in hypothesis/core.py), so both runs started from the same seed.

Shown from Hypothesis's code (6.168.5, read for this PR). Each integer draw first has a 5% chance (_maybe_draw_constant, p=0.05) of returning a value from a pool of constants instead. Part of that pool comes from local modules in sys.modules: _get_local_constants adds the constants of every newly loaded module whose file is local (is_local_module_file skips the standard library, site-packages and test files) whenever sys.modules grows, and never drops any. So a test's pool also includes modules that earlier tests in the same process loaded.

Shown by runtime probes (local CPython 3.13.9 and Hypothesis 6.168.5, not CI's environment), at this head fc8bdad577 unless marked:

• After importing the package and the test module, the runtime pool held 67 integers in [0, 150000] (the wage range). The test file does not count as local. At the previous merge head 56ec6ef3e0 (the same test code; its package lacks only main's two newest commits), the pool also held 67 integers, the same set as a recomputation from the loaded modules with Hypothesis's own functions. PE variable modules were in sys.modules under hashed names (for example …_mt_pension_exemption_joint) and counted as local.
• With the local pool frozen empty, test_drawn_batches drew different batches: the five random draws were (year, units) (2021, 10), (2021, 21), (2023, 14), (2022, 19), (2027, 30), against (2021, 10), (2021, 21), (2024, 30), (2025, 30), (2021, 13) with the live pool. Rerunning with the live pool reproduced the live draws. So the derandomized draws depend on that pool, not just on the seed.
• Importing the package loads all nine policyengine_us.tools.social_security modules, and they count as local. At 56ec6ef3e0, in sys.modules order, policyengine_us.tools modules added 21 of the 67 integers.

Withdrawn. An earlier version of this section counted 623 and 639 such constants in the two merge trees. Those counts came from a static scan of every .py file in the package plus the 116 test files. Hypothesis reads only loaded modules and skips test files, so they were not the pools the CI processes built, and I've dropped them.

Inferred, not shown. The #10402 merge tree contained #10401's tools/social_security/*, which the #10392 tree did not, and variables/gov/ssa/social_security/calculator/results.py imports them. If those modules were loaded before test_drawn_batches ran, the #10402 process's pool differed from #10392's, and with it the draw. That would explain why #10392's draw contained no pooled pension-only unit and #10402's did. The probes don't reproduce either CI process's pool, which also depends on the modules its earlier tests loaded, and nothing here shows #10392's actual draw.

Tests

All local runs below are at this head (fc8bdad577), one process each, with thread caps of 4 and OMP_WAIT_POLICY=PASSIVE, on CPython 3.13.9 and Hypothesis 6.168.5. The host's filesystem was heavily contended during them: importing the package took 12 to 17 minutes, against under a minute earlier the same day.

• The file, under a 4 GB / 1,800 s cap: 8 passed (the seeded grid for 2021–2027 and test_drawn_batches) in 752.68 s of pytest time, 1,517 s of wall time including startup and shutdown, with a peak RSS of 3.2 GB.
• Superset check. For each batch that test_drawn_batches draws (the @example and the five derandomized draws, which are the same under main's file and this PR's) and for the seeded grid for 2021–2023, I ran one simulation, then main's _check and this PR's _check on that same run. Both pass every case. Main's property 3 asserts 49 units and this PR's asserts 54. Main's units are a subset of this PR's, with the same expected values (they have no exemption). The five extra units are pension filers: the @example's (federal AGI 12,000, exemption 4,400, Montana AGI 7,600), a drawn 2021 unit (12,291, 4,400, 7,891), and the grid's pension-only filer for 2021, 2022 and 2023 (12,000 less 4,400, 4,640 and 5,060).
• The @example alone (test_drawn_batches with Hypothesis's explicit phase only): 1 passed.
• Mutant (property 3's expected value without - mt_pension_exemption_joint), under a 2 GB / 1,800 s cap: 2 failed, both at property 3's assert_allclose (line 231). test_drawn_batches failed on the explicit example: its pension filer reports 7,600 against the mutant's expected 12,000. test_seeded_grid[2021] failed on the grid's pension-only filer: 7,600 against 12,000.
• CI at this head: all 33 checks passed (run 40664835691). Main's shard plan now puts this file in Rest Python shard 5 (group core-2), not shard 3. That job passed all 8 of the file's tests in 21.6 s of call time, under CPython 3.14.9, Hypothesis 6.168.3 and the ci profile.

impact: none (one test file changes; no variable, parameter or reform file changes)
axiom: n/a (test only)

🤖 Generated with Claude Code

[MaxGhenis]

Merge audit (US + core hub), #10498 at fc8bdad5778c605661d41efc5cc0c55b9deacf94, squash.

• Independent reviews:
  • r1, GPT-6.1 Sol, at 94fb45c700: APPROVE, one P3. It confirmed the model is right for the shrunk draw: the 2021 exemption is capped at $4,400, so a $12,000 pension gives Montana AGI $7,600, with no age condition. It confirmed the invariant, the @example and the grid filer. Its P3 asked the body to separate evidence from inference in "Why Add Montana's 2021–2023 partial pension, annuity, and IRA income exemption #10392's own CI passed".
  • r2, GPT-6.1 Sol, at this head: APPROVE, no findings.
    • It diffed both main merges against both parents. The PR's diff against main is only the test file and this PR's changelog fragment, and Check Montana's pension exemption in the Montana AGI pooling test #10496's released fragment is untouched.
    • Property 3 now includes pension-exemption units and asserts max(federal AGI − mt_pension_exemption_joint, 0). The joint_path guard matches main's.
    • It checked the superset probe: main's 49 asserted units are a subset of the head's 54, with equal expected values. The 5 extra units are pension filers.
    • It found r1's P3 resolved.
    • It reran the file: 8 passed.
• Mutant (drop the exemption term from property 3's expectation):
  • The build's lane run failed both selected tests at line 231, 7,600 against 12,000. It ran under a 1,800 s cap because 900 s attempts died during import.
  • r2's run under the 900 s cap timed out during import inside its lane.
  • The hub reran it at default QoS under the 2 GB / 900 s cap: 2 failed at line 231, 7,600 against 12,000, in 14.4 s of pytest time.
• Gates at merge:
  • gh pr checks exits 0, with a successful Pull request run at this head (33 checks). The file passed 8/8 in Rest Python shard 5.
  • MERGEABLE, not a draft, and no review requests changes.
  • It merges cleanly with current main.
• Impact: none. Only one test file and a changelog fragment change; no variable, parameter or reform file changes.
• Axiom: n/a (test only).