Rebase congressional-district rows of the Table 1.x-controlled SOI columns as national shares

[MaxGhenis]

Follow-up to #1243 / #1247. Stacked on #1247 (base us-soi-national-control-concepts, head 490b67181), which is stacked on #1036. This PR's commits sit directly on #1247's head.

It moves published full-surface target values, so it waits on Max's ruling (d1450) and on #1247 and #1036.

Summary

#1247 made each Table 1.x control the only national level of 14 SOI quantities on national_state. On the default full surface, the congressional-district file's US row was still a second national all-AGI target beside the control for 11 of them. That row is irs_soi.ty2023.congressional_district_2022.all_returns.us.<measure>: TY2022 data from 22incd.csv, stamped ty2023, for a processing-window subset of filers. On the pinned feed (51aa40f), control / district US row ran from 1.001 (taxable pension returns) to 1.169 (taxable Social Security amount).

This PR applies the district doctrine of #1229 (taxable interest) and #1040 (capital gains) to those 11 columns, using #1247's register:

• every district-file state and district row enters as its share of the file's own US row, times the control;
• the district US row retires.

After it, every quantity a control of the register measures has one national value on full, and the district file's states and districts each sum to the control.

The columns

22incddocguide.docx (TY2022) defines each column at the same Form 1040 line, under the same IRS name, as the HT2 guides #1247 checked. The district US row is a narrower count of the same concept. The file covers only returns processed in calendar 2023. It drops returns with a missing or mismatched ZIP code, and APO/FPO, foreign and territory addresses (guide sections C-D). It also suppresses small and single-return-dominated cells (section E).

Column	Line	District US / Table 1.4, TY2022	District US / HT2 US, TY2022	Control / district US, pinned feed	District 51 states, before -> after	Move
A02500 taxable SS	6b	0.9831	0.9888	1.1692	$490.09B -> $573.04B	+16.92%
A04470 itemized	12	0.9404	0.9657	1.0997	$682.97B -> $751.10B	+9.97%
N02500 taxable SS	6b	0.9797	0.9874	1.0641	24,167,050 -> 25,716,763	+6.41%
A00100 AGI	11	0.9724	0.9758	1.0597	$15,682.85B -> $16,619.16B	+5.97%
A02300 unemployment	Sch 1, 7	0.9699	0.9798	1.0546	$31.89B -> $33.64B	+5.46%
A01700 taxable pensions	5b	0.9702	0.9843	1.0538	$961.67B -> $1,013.42B	+5.38%
N01400 taxable IRA	4b	0.9966	0.9877	1.0288	16,227,430 -> 16,694,154	+2.88%
N1 returns	-	0.9690	0.9793	1.0272	156,342,760 -> 160,602,107	+2.72%
N02300 unemployment	Sch 1, 7	0.9671	0.9839	1.0272	4,573,040 -> 4,697,502	+2.72%
A01400 taxable IRA	4b	0.9767	0.9837	1.0247	$464.87B -> $476.36B	+2.47%
N01700 taxable pensions	5b	0.9827	0.9850	1.0014	29,501,160 -> 29,541,284	+0.14%

• Same-year identity. The first two ratio columns compare the same tax year (TY2022) on the sha256-matched 22incd.csv, 22in14ar.xls and 22in55cmcsv.csv, with Table 1.4 columns located by header text (scripts/cd_identities.py).
• The gap. The third ratio column is that coverage gap times each column's TY2022 -> TY2023 change; the district rows age only 2023 -> 2024 because of the stamp.
• Partition. In the file, the 51 states sum exactly to the US row and each state's districts sum exactly to the state, for all 11 columns.
• No other columns. The district package carries no wage or taxable-income rows (N00200/A00200, A04800).

This is the #1229 case (same column, stale vintage and coverage), not the #1040/#1231 case (a proxy for a different concept).

What changes

fiscal_targets.py:

• _SoiHt2NationalControlRule.source_column. Rebase stale Historic Table 2 rows onto their concept-matched national controls #1247's register now records each column's IRS name, which the HT2 guides and 22incddocguide.docx both use (N1, A00100, ..., A04800). A test pins the 14 names against the guide.
• _rebase_stale_soi_district_distributions runs after Rebase stale Historic Table 2 rows onto their concept-matched national controls #1247's HT2 pass, with the same register, the same control chooser and AmbiguousSoiNationalControlError.
  • Each all-AGI district-file row of a registered measure becomes value x control / district US row. The US row is the same record set's national row of the same measure, filing status, universe and period.
  • The value lands at the control's period, with the usual uprating_* receipts, soi_source_concept / soi_control_concept, and soi_share_total_source_record_id naming the denominator.
  • One factor per measure scales states and districts alike, so congressional_district_children_scale_to_state_parent keeps factor 1.
  • The district US row retires.
  • No eligible control (none, after the build period, or older than the row's period): the rows keep their levels and the US row stays.
  • No denominator. With an eligible control but no nonzero district US row, the row is dropped. That is the Rebase congressional-district taxable interest onto Table 4.3 as shares of the district file's US row #1229 / Rebase congressional-district capital gains onto Table 1.4 and drop mislabeled CD SALT and PTC columns on the full surface #1040 district rule; Rebase stale Historic Table 2 rows onto their concept-matched national controls #1247 keeps HT2 rows as levels in the same case. Neither case occurs on the pinned feed.
  • Each package divides by its own US row: the denominator key carries the record set, so another district file at the same stamp never supplies it. Two different US values for one key raise; equal twins name the smaller record id.
  • An AGI slice of the file (the current package has none) takes the measure's one factor, as Rebase congressional-district taxable interest onto Table 4.3 as shares of the district file's US row #1229 and Rebase congressional-district capital gains onto Table 1.4 and drop mislabeled CD SALT and PTC columns on the full surface #1040 rebase district slices.
• The column guard runs inside the pass, on the rows it rebases:
  • Every source fact of a rebased row, and its US row, must read the register's source_column; otherwise UnreviewedSoiNationalControlColumnError refuses the compile. A missing column refuses too.
  • The crosswalk merges source districts into one fact that keeps the first contributor's layout, so translate_congressional_district_facts_to_current_vintage now records each contributor's source_column_id in the lineage. _soi_district_source_columns reads every source of a district fact from there; any other fact is its own source, and facts sharing a record id all count.
  • Rows that keep their levels, and facts that compile to no target (a stale AGI slice, a superseded vintage), are not checked.
  • Reason: the same package already maps its SALT and PTC measure ids to the wrong columns (Congressional-district capital-gains, SALT and PTC targets on the full surface cross IRS concepts #1038, Drop congressional-district SALT and PTC facts read from the wrong IRS column #1224).

tools/us_acs_local_cd_surface.py:

• The problem. The state_cd level bridges lift the district-only concepts (charitable, interest paid, QBI) onto the HT2 basis by a sibling's HT2 / district-file ratio. Their siblings (itemized_deductions_amount, return_count, adjusted_gross_income) are three of the 11 columns. Read as compiled, a rebased sibling cancels the gap the bridge exists to carry. On the pinned feed, the bridged charitable and interest-paid amounts would have fallen 9.07%, QBI amounts 5.63% and bridged counts 2.65%.
• cd_file_level reads a national-share district row at the file's own level: it divides out uprating_factor and, for a bridge, re-ages like the bridged row. That re-aging stands in for the sibling's own aging only when both age on one series, as every registered bridge does, so a pair on different series refuses. The series comes from target aging's own rule (_not_ageable_reason, _cbo_income_source_for_spec), not the receipt, which is a no-op for a share landed at the build year.
• The bridge factor, the sibling's band ratio (its median re-aged by the same constant), and every district block's shares, records (state_cd_cd_file_value, state_cd_cd_file_state_sum, state_cd_rebase_factor, which target_roles.json publishes) and band factor use it.

Docs:

• docs/us-soi-national-controls.md has a new section, "Congressional-district rows (Rebase congressional-district rows of the Table 1.x-controlled SOI columns as national shares #1250)".
• docs/us-acs-local-soi-target-surface.md: rebase-factor and bridge notes; the bridge figures are refreshed to this stack's values.
• docs/us-fact-to-target.md, docs/us-chronicle-feed-repin.md, and a changelog fragment.

Effect on the pinned feed

Release compile: target_period=2024, age_targets=True, packaged CD crosswalk, Medicaid substitution. Before = #1247's head 490b67181.

	before	after
compiled targets	33,235	33,224
full registry	0bbbad9b22d1	ce77f86efb24
national_state	6,087 at 0e899cd7f893	unchanged
controlled quantities with two national values on full	11	0
all quantities with two national values on full	38	27

• Retired: 11 rows, the district US rows. Added: none.
• 5,357 values move, 11 x (51 + 436), each family by its one factor (the table). No other spec changes, in value or metadata.
• Shares hold exactly. After aging, the district file's 51 states, and separately its 436 districts, sum to the control within 1e-12. Every district reconciles to its state with factor 1, and the shares age with the control's own link.
• At-large districts are bound twice, as before. Each has the district file's row and HT2's state row standing in for it (AK, DE, DC, ND, SD, VT, WY and Montana's two; 99 rows).
  • Both are now shares of the same control. HT2 / district file is 0.966-1.072 (median 0.988) after, against 0.986-1.17 (median 1.037) before.
  • The exceptions are Wyoming and South Dakota itemized deductions (1.64 and 1.35, from 1.81 and 1.49), where the two files disagree about the state.
  • Counting both, a measure's 445 district rows sum to 1.015-1.022 x the control (before: 1.016-1.026 x the district US row). Each family alone sums to at most the control.
• ACS-local state_cd. All 26,425 values match Rebase stale Historic Table 2 rows onto their concept-matched national controls #1247's head within 4.2e-15. Its counts, drops, band verdicts and receipt (rebase and bridge factors, recorded ratios) match within 5e-15, and so does each district row's record of the file value. state and totals have no district-file rows of these columns.

Invariants

• Shares conserve the control. Every district-file row of a rebased measure equals raw x control / its file's US row and names that row. The states sum to control x (states / US), so never past the control. Each state's districts sum to the state with factor 1. (Hypothesis property: any carried column, either stamp, any control vintage or none, 1-3 states, an other-area residual, HT2 rows alongside, an equal twin US row sorting before or after, any feed order.)
• Own denominator. District rows divide by their own package's US row and HT2 rows by the HT2 US row; a second district file's US row is never borrowed. (Property; examples.)
• One national level. With an eligible control, the control's row is the only national all-AGI row of the quantity. (Property; pinned on full for all 14 controlled quantities, with every state and district family at most the control.)
• No eligible control, no change. (Property; examples for none, older and future controls.)
• Order-free and deterministic. Shuffling the facts compiles the same registry; denominator conflicts refuse in either order. (Property; examples.)
• Stamp-invariant. The truthful 2022 stamp gives the same values and landing period. (Example.)
• Reviewed columns only. An unreviewed or missing column behind a rebased row refuses: on a state row, on the US denominator, on an AGI slice, or on one contributor hidden behind a crosswalk merge. A row that keeps its level, or a fact that compiles to no target, never refuses. (Examples.)
• Differential: one share rule. Fed the same state values and US total, the HT2 pass (Rebase stale Historic Table 2 rows onto their concept-matched national controls #1247) and the district pass produce identical values and factors. (Hypothesis.)
• state_cd is invariant to national shares. Binding district rows as shares, with any binary-exact factor and aging, moves no state_cd value, drop, verdict, relative gap, bridge factor or recorded sibling ratio. The bridged row and its sibling share one aging series, as every registered bridge does; a pair on different series refuses. Where the share ages like the file (the pinned feed), the block rebase factors and every district row's recorded file value are unchanged too. (Hypothesis, with and without the band; examples for the recorded ratio and the series refusal.)

Tests

• New test_us_soi_district_national_controls.py (36 tests):

  • the register's IRS names;
  • shares and declarations for all 11 columns;
  • own denominators beside HT2, and never another district file's US row;
  • keep-level cases (no, older and future control);
  • the drop cases (no US row, zero US row);
  • stamp invariance;
  • the column guard:
    • bad or missing columns;
    • a contributor hidden behind a crosswalk merge;
    • a duplicate record id;
    • the US denominator;
    • no refusal beside an ineligible control;
    • no refusal for a fact that compiles to no target (a stale slice, a superseded vintage);
  • translated facts compiling to the crosswalk compile's registry;
  • AGI slices (state and US) taking the measure's factor;
  • denominator conflict and twins;
  • the property and the differential.

• test_us_acs_local_cd_properties.py:

  • the state_cd invariance property, which expects the series refusal exactly where a bridge pairs two series;
  • cd_file_level;
  • the recorded-ratio example at two agings;
  • the series-from-the-rule example;
  • Sol's build-year control end to end through state_cd.

• Pinned feed (test_us_fiscal_targets.py):

  • test_pinned_feed_district_rows_take_their_national_control_level: 51 + 436 rows per column, factor, denominator, landing period, aging, sums, reconciliation;
  • test_pinned_feed_full_surface_has_one_national_value_per_controlled_quantity;
  • the fences count re-pins to 33,224.

• Mutation check: 30 source mutations at 5768ffd49 (mutation/mutations.log, earlier rounds in mutations-r*.log). 27 fail at least one synthetic test:

  • pass and denominators:
    • district US row kept;
    • older control rebases;
    • no denominator keeps the level;
    • denominator not named;
    • conflicts not refused;
    • feed order names a twin;
    • wrong landing period;
    • wrong factor;
    • pass not wired;
    • US row retired without a control;
    • HT2 rows taken by the district pass;
    • AGI slices skipped;
    • AGI slices join the denominators;
    • denominator key drops the record set;
  • column guard:
    • guard off;
    • missing column accepted;
    • crosswalk drops the contributor's column;
    • guard reads the merged layout;
    • guard skips the denominator;
    • guard checks rows kept as levels;
  • state_cd:
    • bridge sibling read as compiled;
    • sibling not re-aged;
    • sibling median not re-aged;
    • blocks read compiled shares;
    • aging-series guard off;
    • series read from the receipt;
    • block band on compiled values.

  Three are equivalent mutants, so no test can see them:

  • HT2 US rows joining the district inventories: they are keyed by record set, so no district row can reach them;
  • duplicate ids collapsing to one column: the compiler refuses a duplicate id before the pass, and the crosswalk merges copies into one fact;
  • the vintage ratio read on compiled values: the sibling's band compares the bridge factor with the median times factor / vintage ratio, so the ratio's basis cancels.

• Local runs (4 workers, engine-free venv, MICROCOSM_US_CHRONICLE_FACTS set to the pinned feed):

  • Whole microcosm-build engine-free tree at c13a31b67 (this head): 13,086 passed, 69 skipped, 1 failed (engine-free-all-r3.log). The failure is test_telemetry_emitter.py::test_subprocess_exchanges_ambient_token_and_delivers_events: its local telemetry service was unavailable (LocalTelemetryEmitter.available was False), the same failure Rebase stale Historic Table 2 rows onto their concept-matched national controls #1247's broad run reported. It fails the same way at every head; a solo rerun hung past 10 minutes and was stopped. The diff touches no telemetry code.
  • Neighbouring suites at 5768ffd49: fiscal targets, HT2 and district controls, AGI bands, ACS-local sparse CD and properties, ACS-local release tool, CD vintage, crosswalk and geography, release parity, Chronicle feed, release-gate preflight, fiscal refresh builder. Result: 926 passed, 7 skipped (6 need the US engine extra, 1 a built CD plan registry). It includes the feed-gated test_pinned_feed_state_cd_surface_matches_its_contract.
  • Pinned compile at every head since ab6171978: byte-identical to after/. state_cd values and receipt are identical across the review heads (statecd/head_r2-r4).

• CI runs only for PRs into main; it runs when Choose the capital-gains rebase control by concept and period, not feed order #1036 and Rebase stale Historic Table 2 rows onto their concept-matched national controls #1247 merge and this PR retargets.

• ruff check and ruff format --check are clean on the touched files.

Evidence: ~/PolicyEngine/_reviews/cd-soi-national-controls-20261012/. It holds cd_identities.json, the guide text, the after/ compile (before is #1247's after-r2/), diff.json, the state_cd receipts and values (statecd/), the #1030 overlay, scripts/ and mutation/.

Interactions

• Rebase stale Historic Table 2 rows onto their concept-matched national controls #1247 / Choose the capital-gains rebase control by concept and period, not feed order #1036. Needs both (register, chooser, family helper). Retargets to main after they merge.
• Read restamped Chronicle facts at their data year before aging #1030 (restamp at data year). Overlay: Read restamped Chronicle facts at their data year before aging #1030's head 03ceca6ad merged into this PR. Source files merge clean; the test file conflicts.
  • No refusal, no value change. Read restamped Chronicle facts at their data year before aging #1030's share narrowing admits all 5,357 district shares. uprating_from_period moves 2023 -> 2022 and uprating_to_period stays 2023, so aging is unchanged and no value moves.
  • state_cd holds. All 26,425 values, band rows (with their recorded ratios) and bridge factors are identical with the district pass on or off (1e-14). That is the case cd_file_level's re-aging exists for: unrebased district rows age from 2022 under Read restamped Chronicle facts at their data year before aging #1030, shares from 2023. A block of a rebased amount column records its file values aged from 2023, one link from an unrebased row, so its recorded rebase factor moves (itemized deductions, median 1.062 -> 1.095); its shares and bound values do not.
  • Owed by whichever lands second (pinned tests skip in CI):
    1. ..._restores_the_fences re-pins to national_state 603ebb05a967 on the overlay.
    2. test_pinned_feed_corrects_the_translated_congressional_district_specs becomes {9,156 / 14,824 / 1,092 / 1,705 district, soi-state-2022 1, W-2 1}.
    3. In that test, the count of 60 rebased onto a restamped control becomes 0 (from Choose the capital-gains rebase control by concept and period, not feed order #1036), and "no restamped share" must allow exactly the 5,357 district shares, plus Rebase congressional-district taxable interest onto Table 4.3 as shares of the district file's US row #1229's, Rebase congressional-district capital gains onto Table 1.4 and drop mislabeled CD SALT and PTC columns on the full surface #1040's and Bind congressional-district interest paid as mortgage interest shares #1231's if they land.
• Rebase congressional-district taxable interest onto Table 4.3 as shares of the district file's US row #1229, Rebase congressional-district capital gains onto Table 1.4 and drop mislabeled CD SALT and PTC columns on the full surface #1040, Bind congressional-district interest paid as mortgage interest shares #1231 (other district families). Same doctrine and declarations, distinct helper names (_SoiDistrictShareTotal, UnreviewedSoiNationalControlColumnError), so no symbol collides. Text overlap:
  • neighbouring lines in the compile sequence and __all__;
  • Bind congressional-district interest paid as mortgage interest shares #1231 edits the same STATE_CD_LEVEL_BRIDGES comment block;
  • Rebase congressional-district taxable interest onto Table 4.3 as shares of the district file's US row #1229 edits the state_cd factor-band comment.
  • cd_file_level also reads Rebase congressional-district taxable interest onto Table 4.3 as shares of the district file's US row #1229/Rebase congressional-district capital gains onto Table 1.4 and drop mislabeled CD SALT and PTC columns on the full surface #1040 shares at file level in the band's ratios; their verdicts are unchanged (the relative gap is scale-free). Bind congressional-district interest paid as mortgage interest shares #1231's national-share rows bind as compiled through its own branch, which runs before any sibling lookup.
• Drop congressional-district SALT and PTC facts read from the wrong IRS column #1224 drops the mislabeled district SALT/PTC columns; no overlap beyond the fences count.
• chronicle#345 relabels district columns. It does not touch these 11; if a re-pin changes their columns, the guard refuses.

Not changed here

• The other district families. The district US row is still a second national value on full for 22 quantities whose other national row is the TY2022 HT2 US row: dividends, Schedule C, partnership, rental and tax-exempt income, EITC, ACTC, income tax, medical deductions. Rebase congressional-district capital gains onto Table 1.4 and drop mislabeled CD SALT and PTC columns on the full surface #1040 (capital gains) and Drop congressional-district SALT and PTC facts read from the wrong IRS column #1224 (SALT, PTC returns) handle five more. A follow-up chip is filed.
• At-large double binding on full (above). state_cd already drops HT2's at-large copies.
• The share assumption. A district's share of the ZIP-matched, processing-window file stands in for its share of the full population, as for Rebase congressional-district taxable interest onto Table 4.3 as shares of the district file's US row #1229's interest rows.
• The keep-versus-drop difference between Rebase stale Historic Table 2 rows onto their concept-matched national controls #1247's HT2 pass and the district passes when a family has no US row.

Review

Round 1, in-session Opus 5.5 (review-opus.md, with executed checks: a fresh compile byte-identical to after/, 63 targeted and pinned tests, four extra mutations, two counterexample scripts). Verdict APPROVE WITH NITS, no P1. Responses, in dddcfddd4 and 0072c2159:

• F3 (P2): state_cd per-row records held share values. Fixed.
  • District blocks read their rows through cd_file_level. state_cd_cd_file_value, state_cd_cd_file_state_sum and state_cd_rebase_factor (published in target_roles.json) record the file's own values again, and the receipt's rebase factors match Rebase stale Historic Table 2 rows onto their concept-matched national controls #1247's base.
  • Bound values were never affected.
• F1 (P2): the quoted state_cd rebase-factor range left out itemized deductions. Moot after F3: the receipt is unchanged, and the docs say so.
• F2 (P3): re-aging assumed one aging series. Fixed.
  • cd_file_level refuses a pair on different series (test added).
  • The sibling's band median is re-aged with the factor, so recorded ratios match under Read restamped Chronicle facts at their data year before aging #1030 too. Overlay: Wyoming 1.7527 with the pass on and off.
  • The property's docstring now says what it draws.
• F4 (P3): two extra mutants survived, and the guide test is a change detector.
  • New tests pin the guard's eligibility rule (an unreviewed column beside an older or future control keeps its levels) and AGI slices.
  • AGI slices now take the measure's factor, as in Rebase congressional-district taxable interest onto Table 4.3 as shares of the district file's US row #1229 and Rebase congressional-district capital gains onto Table 1.4 and drop mislabeled CD SALT and PTC columns on the full surface #1040, and the guard checks them.
  • The doc now says the test pins the register to the guide's names.
• F5 (P3): rows without a denominator are dropped silently. Kept, as Rebase congressional-district taxable interest onto Table 4.3 as shares of the district file's US row #1229 and Rebase congressional-district capital gains onto Table 1.4 and drop mislabeled CD SALT and PTC columns on the full surface #1040 do. The pinned test fails if any of the 11 families loses a row (51 + 436 each). A refusal or receipt shared by all three district passes is proposed on Taxable-interest rebase: per-row control choice and per-pair bridge retirement can disagree #1236.
• F6 (P3): eligibility compares the stamp. As documented, and shared with Rebase congressional-district taxable interest onto Table 4.3 as shares of the district file's US row #1229 (Taxable-interest rebase: per-row control choice and per-pair bridge retirement can disagree #1236 corner 2). Taxable-interest rebase: per-row control choice and per-pair bridge retirement can disagree #1236 now notes that Read restamped Chronicle facts at their data year before aging #1030's correction runs after the rebase passes, so it forces a review of a new restamp but does not change the rebase's period test.

Round 2, independent GPT-6.1 Sol (review-sol.md, at 0072c2159): REQUEST CHANGES, three P2s, each with an executed counterexample. Fixed in 01aec387a and abe8df5da:

• A share whose control sits at the build year broke state_cd bridges. Its receipt is the no-op source_equals_build. The aging series now comes from target aging's own rule; Sol's case is an end-to-end test.
• A row could divide by another district file's US row at the same stamp. Denominators are keyed by record set.
• The column guard refused facts that never bind (a stale AGI slice, a superseded vintage). The guard moved into the pass and checks only rebased rows and their denominator.

Round 3, Sol delta (review-sol-r2.md, at abe8df5da): the three earlier findings RESOLVED, two new P2s. Fixed in 5768ffd49:

• A duplicate record id could hide an unreviewed contributor.
• Compiling the translator's own output refused.

The crosswalk now records each contributor's column in the merged fact's lineage, and the guard reads every contributor from there.

Round 4, Sol delta (review-sol-r3.md, at 5768ffd49): APPROVE, both findings RESOLVED, no new defect.

• Probes: older lineage without the column refuses; empty contributor ids cannot hide a column; at-large proxies keep their parent's column; a translated feed compiled again keeps its registry version.
• Its own replay: full registry ce77f86efb24, national_state 0e899cd7f893, state_cd 82f42fbf2ec8, all unchanged.

c13a31b67 after it is docs only.

axiom: n/a: calibration-target compilation (no tax-benefit rule changes)

🤖 Generated with Claude Code