Rebase stale Historic Table 2 rows onto their concept-matched national controls

[MaxGhenis]

Fixes #1243. Stacked on #1036 (base us-cg-returns-control-concept); this PR's two commits sit directly on #1036's head 4e86ec358.

Summary

On the pinned feed (51aa40f), the route-A national_state surface binds two different national all-AGI targets for 14 SOI quantities. One is the stale Historic Table 2 (HT2) US row (TY2022); the other is a newer Pub 1304 Table 1.x row (TY2023). They are 0.3% to 12.2% apart, and two state families already sum past the smaller one (ira_distributions sum 1.019x, taxable_pension_income count 1.009x).

This PR applies to all 14 the doctrine the taxable-interest (#489/#518) and capital-gains (#1035/#1036) families already use:

• HT2 state rows enter as shares of the latest concept-matched national control;
• the stale national rows retire.

After it, every compared quantity has one national value and no state family exceeds it. The PR moves published target values, so it waits on Max's ruling (decision to be queued after review) and on #1036 (d468).

Per-family decision

All 14 columns rebase. Each one passed both checks for TY2020-TY2023:

• Same line. The HT2 documentation guides (20incmdocguide.doc to 23incmdocguide.doc) define the HT2 column at the same Form 1040 line as the Table 1.4 column.
• Same total. The HT2 US row matches Table 1.4's same-year "All returns, total" within 2.7%, usually a little low. The cause of that small difference is not established: Pub 1304's sample and HT2's population cover the same processing year (Pub 1304 "Domain of Study"; HT2 guide endnote 1).

By contrast, the capital-gains columns differ by 2.36x because they count different populations. Both rows of each pair below already bind the same model variable and measure mode, and a new test enforces this.

HT2 column	Line	Control	HT2 / Table 1.4, TY20-23	51 states before -> after	Move	States / smaller national, before
A02500 taxable SS	6b	Table 1.4	0.993-1.002	$509.58B -> $571.71B	+12.19%	0.998
N02500	6b	Table 1.4	0.991-0.996	24,397,150 -> 25,634,869	+5.07%	0.997
A04470 itemized	12	Table 1.2 (= Table 1.4 col 136)	0.974-0.998	$726.41B -> $748.63B	+3.06%	0.997
A01400 taxable IRA	4b	Table 1.4	0.992-1.001	$485.38B -> $474.78B	-2.18%	1.019
N01400	4b	Table 1.4	0.999-1.009	16,385,750 -> 16,649,213	+1.61%	0.997
N01700 taxable pensions	5b	Table 1.4	0.996-1.005	29,818,930 -> 29,410,814	-1.37%	1.009
N02300 unemployment	Sch 1, 7	Table 1.4	0.983-1.002	4,637,960 -> 4,687,496	+1.07%	0.998
A04800 taxable income	15	Table 1.2 (= Table 1.4 col 142)	0.997-0.999	$12,978.42B -> $12,880.42B	-0.76%	0.999
N00200 wages	1z	Table 1.4	0.995-0.999	127,029,870 -> 127,910,705	+0.69%	0.995
A01700 taxable pensions	5b	Table 1.4	0.986-0.996	$1,001.75B -> $1,008.37B	+0.66%	0.995
N1 returns	-	Table 1.1	0.990-0.998	158,710,000 -> 159,655,171	+0.60%	0.994
A00100 AGI	11	Table 1.1	0.997-0.999	$16,441.78B -> $16,499.02B	+0.35%	0.993
A00200 wages	1z	Table 1.4	0.997-0.999	$10,654.77B -> $10,687.80B	+0.31%	0.992
A02300 unemployment	Sch 1, 7	Table 1.4	0.982-0.994	$33.43B -> $33.52B	+0.28%	0.997

Where the gaps came from. Each gap is a vintage factor times that small same-year difference; neither is a concept difference.

• The vintage factor arises because HT2 amounts age TY2022 to TY2023 by realized AGI growth (x1.0305; wages by realized wage growth), and counts do not age. Each column's own change differed. Taxable Social Security grew 15.0% in Table 1.4; taxable IRA distributions grew 0.1%.
• For the AGI and wage amounts the gap is the same-year difference alone (0.3%).
• docs/us-soi-national-controls.md has the full decomposition table.

Fourth national row. The return count had another national row: HT2's own US workbook (22in54us.xlsx, "Historic Table 2 state data, United States total"), packaged as irs_soi.ty2023.state_2022.us. It holds TY2022 data stamped ty2023 and is retired with the HT2 US row.

What changes (fiscal_targets.py)

• _SOI_HT2_NATIONAL_CONTROLS: a reviewed register mapping each of the 14 HT2 measure ids to a control table and column, plus the Form 1040 line both report.
  • Table 1.1 supplies the return count and AGI.
  • The feed carries the Table 1.4 itemized-deduction and taxable-income totals only in Table 1.2, which prints the same TY2023 cells.
• _rebase_stale_soi_ht2_distributions runs after the capital-gains rebase and before the State top-tail income mass is uncalibrated: CO M+ AGI at 24% of SOI in buildp-acs-local (fails progressive-bracket scoring) #940 bands.
  • Each all-AGI HT2 state row becomes value x control / HT2 US, landing at the control's period, with uprating_* receipts and soi_source_concept / soi_control_concept.
  • HT2-lineage national all-AGI rows (HT2 US, and state_<year>.us) of each rebased model quantity retire, each only if the control is at or after its own data year. The workbook's data year comes from state_<year>, not its stamp. Matching is by quantity, so the workbook's EITC row, which shares measure id return_count, stays.
  • Without an eligible control (none, older than the row, after the build period) or without an HT2 US total, rows keep their levels. Eligibility is per row, so with mixed HT2 vintages a newer HT2 US row keeps its level.
  • Two different HT2 US records for one key raise AmbiguousSoiNationalControlError rather than letting feed order pick the denominator.
• _latest_unambiguous_soi_controls: Choose the capital-gains rebase control by concept and period, not feed order #1036's tie-refusing chooser, extracted and now shared by both passes, with unchanged behavior. A tie at the winning period raises AmbiguousSoiNationalControlError.
• _soi_state_agi_band_controls (State top-tail income mass is uncalibrated: CO M+ AGI at 24% of SOI in buildp-acs-local (fails progressive-bracket scoring) #940) reads a rebased total's uprating_to_period, the period _age_spec ages it from. Without this, AGI bands would age from TY2022 while their total ages from TY2023 and overshoot by 3.05%; mutation M2 below confirms it.

Effect on the pinned feed

Release compile: target_period=2024, age_targets=True, packaged CD crosswalk, Medicaid substitution. Before = main + #1036.

	before	after
compiled targets	33,250	33,235
national_state targets	6,102	6,087
national_state registry	882f4764a248	0e899cd7f893
full registry	f09571684885	0bbbad9b22d1
quantities with two national values (of 63 compared)	14	0
state families past the smaller national target	2	0

• Retired: 15 rows, the 14 HT2 US rows and the workbook return count. Added: none.
• 1,221 values move, each family by one factor:
  • 714 HT2 state rows on national_state;
  • 99 HT2 rows standing in for at-large districts on full;
  • the 408 State top-tail income mass is uncalibrated: CO M+ AGI at 24% of SOI in buildp-acs-local (fails progressive-bracket scoring) #940 AGI bands, which follow their state totals (AGI x1.00348, returns x1.00596).
• No metadata-only changes outside those rows.
• Shares hold exactly. After aging, each family's 51 states equal the control times their share of the HT2 US row in 22in55cmcsv.csv, within 4e-16, and each band stays its recorded share within 6e-16.
• ACS-local release.
  • Its state / totals / full SOI counts are unchanged (4,227 / 607 / 31,321), but the same state rows and bands carry the new values.
  • state_cd district blocks re-reconcile to the rebased HT2 parents; its feed-gated suite passes.

Invariants

• Concept-matched. A control always comes from the table and column registered for the HT2 column. HT2, its US workbook, the district file, Table 4.3 and Table 2.1 never qualify. (Hypothesis property, 300 examples, 7 families.)
• Order-free and deterministic. The chosen controls and the compiled registry are identical under any permutation of the facts, and a tie at the winning period raises. (Hypothesis, both levels.)
• Latest eligible. The control is the latest registered fact not after the build period. With none, or one older than the HT2 row, nothing changes. (Hypothesis; examples.)
• Shares conserved. Rebased states equal HT2 value x control / HT2 US, and so sum to the control times the states' HT2 share; when the HT2 states are at most the HT2 US total, they are at most the control. (Hypothesis over 40 compiles; pinned for all 14 families after aging.)
• One national level. For every rebased quantity, the control's row is the only national all-AGI row. (Hypothesis; pinned.)
• Per-row retirement. A lineage national row retires iff its quantity was rebased and the control is at or after its own data year; a newer HT2 row never retires beside an older control. (Hypothesis over 40 compiles with mixed vintages and a workbook at any data year.)
• One denominator. Conflicting HT2 US totals for one key refuse the compile in either order. (Example tests.)
• States never exceed the nation. On national_state, each of the 63 compared SOI families sums to at most the smallest national target of its quantity, and the national targets of each quantity agree on one value. test_pinned_feed_no_soi_state_family_sums_past_its_national_target is tightened from max to min and adds the one-value check. Table 1.2's copies of Table 1.1's count and AGI are equal and stay.
• Bands stay shares. Each State top-tail income mass is uncalibrated: CO M+ AGI at 24% of SOI in buildp-acs-local (fails progressive-bracket scoring) #940 band is its recorded share of the bound state total, after aging. (Synthetic end-to-end test with a rebased total; pinned for all 408.)

Tests

• New test_us_soi_ht2_national_controls.py (40 tests):
  • register vs measure maps;
  • lineage record sets;
  • all 14 columns rebasing as shares;
  • the workbook retiring only with its own quantity;
  • the US row retiring without state rows;
  • keep-level cases (no, older and future control; missing HT2 US);
  • a same-period control;
  • decoy families never anchoring;
  • tie refusal;
  • mixed HT2 vintages, the workbook's data year and conflicting denominators (from the review);
  • three Hypothesis properties.
• Band tests: a new end-to-end test of bands under a rebased total; the pinned Colorado top-tail check now reads the rebased total.
• Pinned feed: the tightened invariant; a new test pinning each family's control, factor, landing period, equal aging factor and post-aging share; the surface pin moves to 33,235 / 6,087 / 0e899cd7f893.
• Mutation check: twelve source mutations each fail at least one synthetic test:
  1. no workbook retirement;
  2. bands reading the source period;
  3. any family as control;
  4. older control rebasing;
  5. wrong factor;
  6. feed-order tie-break;
  7. HT2 US retired without a control;
  8. HT2 US kept;
  9. workbook retired regardless of quantity;
  10. retirement ignoring the row's year;
  11. denominators overwriting each other;
  12. the workbook read at its stamp.
• Local runs on the pinned feed, 4 workers:
  • on this head (490b67181): fiscal targets, bands, this file, release parity, ACS-local release tool, ACS-local sparse CD, Chronicle feed and source coverage, 424 passed;
  • on main + Choose the capital-gains rebase control by concept and period, not feed order #1036 + the first revision (6367e5cff): the whole engine-free tree, 13,148 passed, 72 skipped, 1 failed. The failure is test_telemetry_emitter.py::test_subprocess_exchanges_ambient_token_and_delivers_events, whose local service did not become ready on a saturated host; it touches no code here;
  • on that revision too, ACS-local CD properties and target-loss weights: 88 passed.
    The review fixes compile the pinned feed to byte-identical specs (all 33,235). The 24 main commits Choose the capital-gains rebase control by concept and period, not feed order #1036 lacks touch no file on the US target-compile path (UK, NZ, source-manifest, data-contract and lockfile changes).
• CI runs only for PRs into main; it will run when Choose the capital-gains rebase control by concept and period, not feed order #1036 merges and this PR retargets.
• ruff check and ruff format --check are clean on the touched files.

Evidence: ~/PolicyEngine/_reviews/soi-national-controls-20261012/. It holds identities.py / identities.json, the doc-guide text, before/ and after/ compiles, diff.json, scripts/ and mutations.log. The IRS workbooks for Table 1.4 TY2020-TY2023 and HT2 TY2022/TY2023 are sha256-identical to the feed's sources.

Interactions

• Choose the capital-gains rebase control by concept and period, not feed order #1036. This PR needs it (the family helper, the shared chooser, the pinned invariant test). It retargets to main when Choose the capital-gains rebase control by concept and period, not feed order #1036 merges.
• State top-tail income mass is uncalibrated: CO M+ AGI at 24% of SOI in buildp-acs-local (fails progressive-bracket scoring) #940 bands. They follow the rebased state totals (above).
• Read restamped Chronicle facts at their data year before aging #1030 (restamp at data year). It changes the stamp on state_2022.us. Retirement matches by quantity, not stamp, so the row retires either way. Whichever lands second re-pins the surface digest.
• Bind Historic Table 2 and district-file business, partnership and rental 'net income (less loss)' rows to signed model concepts #1230 (HT2/CD net income less loss) touches other measures. If it merges first, it may conflict textually in fiscal_targets.py.
• Match the SOI congressional-district file by family, at any vintage #1244 adds the same _soi_record_set_family helper to main. Whichever of it and Choose the capital-gains rebase control by concept and period, not feed order #1036 merges second keeps one copy; this PR only calls the helper.
• chronicle#295 / Ingest ASEC 2024 and 2025 into the US support pool (spine is two vintages behind) #296 package TY2023 HT2 state totals and the TY2023 US workbook. On that re-pin, the same rows rebase by the same-year difference alone (tested), and state_2023.us matches the lineage rule.

Not changed here

• Congressional-district rows of these columns on full. The district file's US row is still a second national level for 11 of the quantities (Table 1.x / CD US = 1.001 to 1.169). A follow-up task is filed, in line with Rebase congressional-district capital gains onto Table 1.4 and drop mislabeled CD SALT and PTC columns on the full surface #1040, Rebase congressional-district taxable interest onto Table 4.3 as shares of the district file's US row #1229 and Bind congressional-district interest paid as mortgage interest shares #1231.
• HT2 N04470 / N04800 counts. Their only national row is Table 4.3, which excludes dependents and so is not an all-returns control.
• The share assumption. TY2022 state shares stand in for TY2023 until chronicle#295 lands.
• An HT2 vintage newer than its control keeps HT2 levels beside the older control. The pinned invariant would fail on that re-pin. IRS currently publishes Table 1.4 first.
• Wages concept (SOI and CBO wage targets bind gross employment_income; the engine taxes irs_employment_income #1191). Both rows bind gross employment_income, unchanged.

Review

Round 1 (independent GPT-6.1 Sol, review-1247.md): REQUEST CHANGES, two majors and one minor, all fixed in 490b67181 (see the commit message). Round 2 (delta re-review, Subfleet job 20261012-155332-pr1247-review-r2): pending.

axiom: n/a: calibration-target compilation (no tax-benefit rule changes)

🤖 Generated with Claude Code

[MaxGhenis]

Stacking note: #1250 is stacked on this branch at 490b67181. It takes the congressional-district file's state and district rows of 11 of these columns as shares of the same Table 1.x controls on full, and retires the district US rows. It reuses _SOI_HT2_NATIONAL_CONTROLS, adding a source_column field per entry, plus the chooser and AmbiguousSoiNationalControlError. national_state is byte-identical. If this branch gets new commits, #1250 rebases onto them.

🤖 Generated with Claude Code

[MaxGhenis]

Queued Max's ruling for this PR as d1449; it closes when this PR merges or closes. This PR's session had stopped after round 2 (GPT-6.1 Sol APPROVE at 490b67181) without queueing it, and #1250 is stacked here. If this PR's session resumes and queues its own, supersede one with cos supersede-decision.

🤖 Generated with Claude Code

[MaxGhenis]

A note for this PR's HT2 → Table 1.x control choice, from #1261 (#1258).

#1261 gives the PUF donor separate partnership_income and s_corp_income leaves. It maps Chronicle's Table 1.4 entity legs (chronicle#180: partnership_net_{income,loss}_{returns,amount} and s_corporation_net_{income,loss}_{returns,amount}) to those leaves, as positive part and negative-part magnitude at return grain. The pinned feed carries none of these ids yet, so nothing changes today: the compiled full and national_state registries are byte-identical.

Once chronicle#180 lands, there is an exact constraint conflict at national level, not a near-duplicate:

• HT2 A26270 is net income less loss for line 32.
• At one vintage, that equals partnership NI − NL + S corp NI − NL.

For TY2022, Table 1.4 gives $487.1B − $175.6B + $812.7B − $93.0B = $1,031.2B. HT2 gives $1,033.3B.

If the national A26270 control and the four entity legs both bind, the solve is asked for two values of the same sum. Whichever family this PR's control selection treats as the national line-32 control should exclude the other at national level, or rebase one onto the other, as this PR does for the state rows.

🤖 Generated with Claude Code