Bind state × AGI-band SOI targets and re-pin the US feed for TY2023 state bands (#940)

[MaxGhenis]

Summary

Binds IRS SOI Historic Table 2 state × AGI-band targets, return counts and AGI amounts, for every state's bands from $100k up, with $500k–$1M and $1M+ split. It re-pins the US Chronicle feed to add the TY2023 state file that publishes those split classes.

Fixes #940.

Why

A state's total return count and AGI calibrate tightly, while its income above $1M, which progressive state rate schedules tax, is unconstrained.

• State top-tail income mass is uncalibrated: CO M+ AGI at 24% of SOI in buildp-acs-local (fails progressive-bracket scoring) #940 measured Build P's ACS-local release within 0.12% of Colorado's SOI returns and AGI. That release held 24% of Colorado's SOI AGI above $1M ($7.8B against $31.7B).
• After Stop the US PUF imputation from predicting income from itself (#982) #1033 (US PUF imputation predicts each income item from the recipient's own survey value, so the PUF clone half reproduces ASEC income #982) the PUF clones reach the national tail. One pre-calibration Colorado record ($125.4M proxy AGI, weight 867 in the ×2 view) carries 84% of Colorado income above $1M.
• National Table 1.1 targets (Bind the SOI Table 1.1 size-of-AGI distribution as calibration targets #960) do not say which state that income lands in.

Before this PR, the state return-count and AGI bands already in the feed (TY2022, 9 bands with a summed 500k_plus) compiled to nothing: _is_untransformed_cross_period_agi_slice refuses every bounded AGI slice from another tax year, and only the EITC and taxable-interest families were rescued.

What changes

The rule (us_runtime/fiscal_targets.py)

• _soi_state_agi_band_status and _is_soi_state_agi_band_fact flag a fact only when all of these hold:
  • its measure is return_count or adjusted_gross_income;
  • it comes from a per-state HT2 AGI record set (irs_soi.<ty>.historic_table_2.state_agi.<st>) at state geography;
  • it is for all filing statuses;
  • its band's lower edge is at or above US_SOI_STATE_AGI_BAND_MINIMUM_LOWER_BOUND ($100,000). The national size-of-AGI floor uses the same edge for the same reason: the SOI slice materializer counts every tax unit in a band, filer or not.
• Every other state HT2 band of those measures is refused at every period, so a same-year vintage can never bind raw sub-floor levels. HT2 us rows never qualify; the national shape belongs to Table 1.1.
• _rebase_soi_state_agi_bands runs after the capital-gains rebase. Each band's value is control × band / partition:
  • partition sums the state's published bands of that vintage (sub-floor bands and negative AGI under $1 included).
  • control is the state's HT2 state_broad all-returns total as the registry binds it: the row latest-vintage selection kept and the exclusions let through, read from the compiled registry, not the facts. A state whose total is excluded binds no bands of that measure; two bound totals for one state and measure raise. Congressional-district <st>_total rows never anchor a state.
• The value lands at the control's period (uprating_to_period). Target aging then ages AGI on the CBO AGI series with the state total; counts never age. Each rebased row records state_agi_band_share, uprating_factor and the control's record id.
• Vintage handling:
  • State bands are the one family selected by vintage rather than per row. Every vintage of a binding band stays a candidate through latest-vintage selection; _with_one_state_agi_band_vintage then keeps, per state and measure, the newest vintage whose published partition tiles the AGI line.
  • The choice is made inside selection, so the exclusion vintage guard and the exclusion receipt see only the band ids that can bind.
  • A gapped newest vintage falls back to the last complete one.
  • In an examined vintage, overlapping bands, a NaN bound or value, or one record id with two different rows raise. An identical re-emission of a sub-floor band counts once. A binding band's own fact is a target fact, so, as for every family, the compile refuses two facts for one record id.

Other code

• target_aging.find_period_contract_violations reads uprating_to_period for rebased rows, the period _age_spec ages them from.
• New release requirements, one per state and measure (irs_state_agi_top_tail_<st>_<measure>, 102 in all): each matches that state's [$1M, inf) row by state_fips and both AGI bounds, so a missing state fails by name and another state's row cannot stand in. The release builder already enforces US_FISCAL_TARGET_COVERAGE_REQUIREMENTS; the scoring tools only record the gate result.
• The alias soi-historic-table-2-state-agi-2023 is registered in source_coverage.py and DIRECT_ACTIVE_ALIASES.

Feed re-pin

• The pin moves from c5e5bf8 to 51aa40f: Chronicle c5e5bf8 plus only the TY2023 state AGI package from Add IRS SOI Historic Table 2 TY2023 state AGI-band facts chronicle#291. That commit lives on branch feed/us-c5e5bf8-state-agi-2023 with tag microcosm-us-feed-940-v1.
• Why not Chronicle main. At main, the artifact-year restamp guard (chronicle#292) makes ten scoped pairs produce no row: the CD file, W-2 items, IRA tables and state_2022 US rows no longer restamp to ty2023. chronicle#304 also renames the HT2 and CD capital-gains concepts. Adopting both is Read restamped Chronicle facts at their data year before aging #1030's re-pin and needs its own decisions. I ran the main-based build; it failed on exactly those pairs.
• What moved (experiments/940-state-agi-bands/feed-repin/compare.json):
  • All 39,158 c5e5bf8 cells are present with equal values and no changed field.
  • 1,020 cells were added, all irs_soi.ty2023.historic_table_2.state_agi.
  • Compiled at 2024 as the release does (CD crosswalk, aging, Medicaid substitution), no target other than the state bands is added, removed, or changed in value or metadata. The state bands go from 306 (TY2022, what the rule binds on the old feed) to 408 (TY2023).
• Pins
  • Scope: 586 → 637 pairs, 62 → 63 runs. Feed: 40,178 rows, 965a29ac….
  • Parity regenerated: still 32 compiled / 52 reviewed; irs_soi.historic_table_2 5,654 → 6,674.
  • Feed-gated pins, re-measured:
    • registry: 32,842 → 33,250 targets;
    • national_state: 5,694 → 6,102 at f96be9f41781;
    • ACS-local state: irs_soi 3,819 → 4,227;
    • ACS-local state_cd: SOI 25,864 → 26,272.
  • Every change is exactly +408.
  • tools/spec_engine_coverage.py --check is unchanged (42,184/42,184).
  • Re-checked after merging main on 2026-10-10: same counts and hashes.

Invariants (property-tested in tests/engine_free/us/test_us_soi_state_agi_bands.py)

1. Share identity: each bound band = control × band / partition (Hypothesis over generated partitions, controls and up to three states).

2. Control homogeneity: scaling one state's total scales exactly that state's bands.

3. Feed-order invariance: shuffled facts compile to the same bands.

4. Partition after aging: each aged AGI band divided by the aged state total equals its published share; counts are unaged (synthetic and feed-gated, for all 408 rows).

5. One vintage per state and measure: the bound bands tile; a collapsed 500k_plus never binds beside split rows.

6. Means inside edges: every state band's aged AGI per return lies inside the band (feed-gated, all 204 AGI rows).

7. Differential against the national owner: summed over the 51 states, the aged $500k–$1M and $1M+ rows are 97–100% of Table 1.1's TY2023 classes aged the same way. They measure 98–99%; Table 1.1 also counts returns filed from other areas and Puerto Rico.

8. Bands are shares of the bound total: with two same-key state totals of different values, in either feed order every band's share of the total the registry binds equals the share it records.

There are also example tests for: the floor at every period, a gap falling back to an older complete vintage, CD totals never anchoring a state, an excluded state total anchoring nothing, the exclusion guard seeing only the chosen vintage, overlap, NaN and duplicate handling, the period contract, and the per-state coverage requirements (a missing Colorado row plus an Alabama stand-in fails exactly Colorado's requirement). The Colorado fixture is the published TY2023 cells, with a test that they sum to the file's stub 0.

Review

• Round 1 (adversarial workflow, math-and-selection lens) found five issues, all fixed in 8042842 and 7a90473:

  • the Colorado fixture's stubs 1–7 were wrong;
  • the floor held only across periods;
  • the period contract read the wrong period;
  • count rows could drop with no gate;
  • malformed vintages that could never bind crashed the compile.

  Fixing the vintage fallback exposed a sixth issue: per-band latest selection shadowed older vintages. It is fixed by the selection-key change.

• Round 2 (independent GPT-6.1 Sol review of ea8940de2): REQUEST_CHANGES with two majors and three minors. It also ran six source mutations (all caught by the tests), verified the feed byte for byte (39,158 old rows unchanged, 1,020 added, each equal to its IRS cell) and compiled the pipeline end to end (33,250 targets, 408 bands, all materialize, coverage passes). Fixed in cdd45625e and 39f37229b:

  • major, control: the control came from raw facts and could differ from the total the registry binds (recorded share 0.1505 against an actual 0.3010 with two same-key totals), and an excluded total still anchored bands. Controls now come from the registry.
  • major, duplicates: an identical duplicate of a binding band's fact is refused by the compile's one-fact-per-record-id rule before the pass. That rule holds for every family and the feed builder never emits such duplicates, so it stays; the docs and test, which claimed otherwise, are corrected.
  • minor: the exclusion guard audited discarded vintages (vintage choice moved into selection); NaN bounds slipped through tiling (now raise); the coverage gate counted names, not states (now one requirement per state and measure).

• On the pinned feed the round-2 changes move nothing: the national_state surface is still f96be9f41781, and the feed comparison is identical to the committed receipt.

• Round 3 (independent GPT-6.1 Sol re-review of 39f37229b): APPROVE. All five round-2 findings resolved or accepted as argued; it re-ran its counterexamples and 17 mutations, and confirmed the pinned-feed registry is byte-identical to before (33,250 at fc776a2b6eb7; national_state 6,102 at f96be9f41781). Its two minors and two nits are fixed in e2c34ac75:

  • partition values must parse and be finite (an infinite sub-floor value had made every share 0.0; an unparseable one counted as 0.0);
  • tests for the guards no test exercised (two bound totals raise, each control filter, a closed upper edge failing its state's requirement); the three mutations that had survived are now caught;
  • "every eligible vintage" wording; Hypothesis too_slow health check suppressed.

• A confirmation review of 39f37229b..e2c34ac75 is linked in a comment.

Launch notes (not done here)

• No build, calibration or release was run, and no numbers are published (decision d228 is about ballot revenue scores).
• Support is thin in places. experiments/940-state-agi-bands/README.md measures pre-calibration support per state and band on the US PUF imputation predicts each income item from the recipient's own survey value, so the PUF clone half reproduces ASEC income #982 outputs, all proxy AGI:
  • no state × band cell is empty, and the count cap never binds;
  • 9 states (DE, IA, ID, ME, MI, MT, RI, WI, WY) have a $1M+ SOI mean above every pre-calibration record, so their $1M+ count and amount cannot both be reached by reweighting alone;
  • 4 states have one record above half their $1M+ amount.
• These rows enlarge the target surface. The next exact-k launch must re-freeze --frozen-target-surface-sha256 and re-score the incumbent.
• Bind the SOI Table 1.1 size-of-AGI distribution as calibration targets #960 (national Table 1.1 classes, d416) is compatible: the differential above measures the agreement. Whichever PR merges second resolves the adjacent edits in fiscal_targets.py and docs/us-fact-to-target.md.

axiom: n/a: calibration-target compilation and data feed (no tax-benefit rule changes)

🤖 Generated with Claude Code

[MaxGhenis]

Merged at e2c34ac75 on Max's ruling (decision d926: merge when green, all four bands, both measures).

Gates at the merged head: all 10 CI checks pass; mergeable CLEAN; independent reviews by GPT-6.1 Sol: round 2 REQUEST_CHANGES on ea8940de2 (fixed), round 3 APPROVE on 39f37229b (minors fixed), round 4 APPROVE on e2c34ac75 with no findings (every surviving mutation now caught). Reports: ~/reviews/microcosm-940/review-out-b.md, review-out-r3.md, review-out-r4.md.

Chronicle side: PolicyEngine/chronicle#291 merged (29b8f84). The feed pin stays on tag microcosm-us-feed-940-v1 (51aa40f) until the next re-pin to Chronicle main (#1030), which carries the 51 TY2023 state-AGI pairs over.

Next launch on this surface: re-freeze the exact-k target surface; expect soft misses in the 9 thin $1M+ states (experiments/940-state-agi-bands/). Nothing was built, calibrated or published here; d228 (ballot revenue scores) stays open.