Report each QRF output's largest single record in the US tail-concentration evidence

[MaxGhenis]

Follow-up to #1259 / #1251 / #1239. Reporting only: no threshold, no verdict changes.

The US release's QRF tail gate fails a sparse column whose top 100 records carry more than 75% of its weighted absolute mass. That test cannot see one record carrying a fifth of a column. In the #1251 rebuild, one tax unit of weight 644.7 drew a $218.9M Schedule E line-32 value and carried 21.7% of line 32's weighted absolute mass at initial weights, while the top 100 tax units carried 44.9%. At the certified Route A final weights the two shares were 4.4% and 39.9%. Both pass.

This PR adds the missing number to the release's tail evidence. For each QRF output it reports the largest single record's share of weighted absolute mass (the top-1 share) and that record's weight, beside the top-100 share.

What it adds

• microcosm.build.gates.single_record_concentration(values, weights, top_k=100) returns carriers, top_1_share, top_1_weight and top_k_share for one column.
  • It uses the gate's own population: finite value and weight, positive |value| × weight.
  • No column is skipped as thin.
  • A column with no mass, or a total past the float64 range, returns None for all three measurements. The carriers count is still reported.
• qrf_tail_concentration.json gains surface.single_record_concentration: {top_k, columns: {column: measurements}, unmeasured: {column: reason}}.
  • It covers every numeric QRF output, dense and thin ones included. Absent and boolean outputs are not measured.
  • The gate's details are unchanged, so the UK and US contract readers see the same keys.
• The release dry run (--dry-run-gates-report) adds top_1_share_at_base_weights and top_1_weight_at_base_weights to each tail row. No class or verdict reads them.
• The post-export probe compares the block with a reference release that recorded it. A release built before this PR, such as Route A, compares none.
• Docs. The README (release gate evidence) and changelog.d/us-qrf-tail-single-record-share.added.md describe the block.

The gate path is unchanged

• For every sparse output, _qrf_tail_concentration_gate runs the same statements as on main. It resolves weights only where main did.
• Dense outputs are measured with weights resolved separately. If an entity's weights do not resolve, the output is listed under unmeasured with the reason, and nothing raises. On main, the gate never resolved weights for a dense output, so the diagnostic cannot fail a run the gate would pass.
• A sparse output with unresolvable weights still raises, as it did on main. A test pins this.

Invariants

Each is a Hypothesis property (300 examples for the function, 40 to 60 through the release tool and the dry run), plus example tests.

1. Share chain. A measured column has 0 < top_1_share ≤ top_k_share ≤ 1.
   • With top_k = 1 the two shares are equal.
   • With at most top_k carriers, top_k_share is exactly 1.
2. Pigeonhole floors. top_1_share ≥ 1/carriers and top_k_share ≥ min(top_k, carriers)/carriers, to one rounding step.
3. Scale-free in the weights. Scaling every weight by c > 0 leaves both shares and the carrier count unchanged and scales top_1_weight by c.
   • This is exact for powers of two and holds to 1e-9 relative otherwise.
   • Scaling or negating every value changes nothing.
4. Zero mass is reported, not divided by. Any float64 input gives a JSON-safe report: no NaN, no infinity, no warning.
   • Inputs include NaN, ±inf, zero or negative weights, and products or sums past the float64 range.
   • Either all three measurements are present or all three are None. A column with no carriers is always None.
5. Order-free. Permuting records leaves carriers and top_1_weight unchanged, and the shares unchanged to rounding.
   • Among records tied for the largest mass, top_1_weight is the largest weight.
6. Monotone. Raising the largest record's value never lowers top_1_share. Raising top_k never lowers top_k_share.
7. Differential, against the gate. On every column tail_concentration_gate checks, top_k_share equals the gate's top_share bit for bit, and carriers equals its carrier count. On a thin column, carriers equals the gate's thin count.
8. Differential, against exact arithmetic. Both shares match ratios of the same float masses summed as Python Fractions.
9. Reporting only. For random geometries (0 to 1,200 carriers in each of three columns, random weights, signs and spreads), the release tool's gate has the same verdict, failures and details as the shared gate run on the sparse columns alone.
   • The dry run's status, failure lines, at-risk lines, summary and details are unchanged by any values in the block, or by its absence.
   • Rows differ only in the two new fields.

Mutation check. I ran ten mutants of the function: tie handling, the ≤ 1 cap, the ≤ top_k shortcut, a tail one record short, the finite-total check, zero-mass carriers, the weight finiteness filter, signed mass, median for max, and dividing by the tail. All ten are killed (out/mutants.log). Two survived the first round:

• The cap. The gate sums its tail in partition order and its total in array order, so its own top_share can exceed 1 by one rounding step. Masses [2, 3, 2, 1, 1e16, 7, 1e15, 1, 0.5] at top_k = 8 give 1.0000000000000002. The measurement caps at 1.
• The ≤ top_k shortcut. It returns exactly 1. Summed in partition order, [2, 1e15, 4.5e15, 1e16, 1, 1e15, 4.5e15] gives 0.9999999999999998.

Each now has an example test.

Not changed here. The gate itself can report top_share above 1. The UK data contract requires top_share in [0, 1]. Reaching it takes a column whose top top_k records hold all but a sub-rounding sliver of its mass, so this PR records it in a test rather than changing the gate.

Certified Route A release: which columns exceed 5% and 10%

The release is populace-us-2024-0fb05b6-4b57d15a287c-20260930T150401Z.

• Inputs. The published H5 (sha256 417d23ae…, as the release manifest certifies) and its calibration sidecar. Both are public in the ungated policyengine/populace-us repository; no restricted data was read.
• Method. scripts/route_a_top1.py in the evidence folder runs this branch's _qrf_tail_concentration_gate on the H5's QRF columns, using the post-export probe's loader. It runs once at the release's calibrated weights, then at the sidecar's initial_household_weight.
• Fidelity. At calibrated weights, the branch reproduces the release's recorded qrf_tail_concentration.json (sha256 83119434…) exactly:
  • the same verdict and surface buckets;
  • the gate's details after a JSON round trip, including all 33 top-100 shares and carrier counts;
  • no difference from the probe's compare_qrf_tail;
  • and the sidecar's final weights equal the H5's.

At the release's final weights, 17 of the 68 measured QRF outputs have a top-1 share above 5%, and 7 have one above 10% (the first seven rows below):

Column	Gate class	Carriers	Top-1	Its weight	Largest record (value × weight)	Top-100	Top-1 at initial weights
long_term_capital_gains_on_collectibles	checked, waived	1,138	48.1%	1,757.8	$6,761,367 × 1,757.8 = $11.89B	93.3%	13.9%
investment_income_elected_form_4952	thin	394	44.1%	1,989.2	$6,715,619 × 1,989.2 = $13.36B	98.2%	23.7%
farm_income	thin	445	43.5%	4,673.2	$3,804,933 × 4,673.2 = $17.78B	96.8%	22.1%
casualty_loss	thin	138	18.7%	2,418.8	$618,387 × 2,418.8 = $1.50B	99.8%	7.5%
sstb_w2_wages_from_qualified_business	thin	480	18.4%	3,675.7	$9,294,781 × 3,675.7 = $34.16B	94.7%	20.4%
investment_interest_expense	dense	64,778	17.5%	2,120.1	$3,681,182 × 2,120.1 = $7.80B	65.0%	7.6%
bond_assets	checked, waived	8,887	14.9%	5,381.4	$20,590,000 × 5,381.4 = $110.80B	77.3%	4.4%
estate_income	checked, waived	2,567	8.5%	176.1	$27,594,578 × 176.1 = $4.86B	79.2%	21.6%
w2_wages_from_qualified_business	checked, waived	1,563	7.8%	3,675.7	$9,294,781 × 3,675.7 = $34.16B	77.3%	10.4%
sstb_self_employment_income_before_lsr	checked	11,696	6.6%	2,353.0	$10,162,242 × 2,353.0 = $23.91B	52.4%	2.9%
unadjusted_basis_qualified_property	checked	24,729	6.6%	1,701.6	$119,257,711 × 1,701.6 = $202.93B	47.2%	3.6%
qualified_bdc_income	checked	5,276	6.6%	2,509.2	$2,747 × 2,509.2 = $0.01B	69.0%	4.2%
charitable_non_cash_donations	checked	45,215	6.6%	973.7	$7,753,602 × 973.7 = $7.55B	65.3%	7.4%
unrecaptured_section_1250_gain	checked	6,310	6.3%	2,353.0	$1,349,338 × 2,353.0 = $3.18B	58.9%	16.7%
sstb_unadjusted_basis_qualified_property	checked	5,378	6.1%	2,353.0	$23,399,794 × 2,353.0 = $55.06B	72.3%	4.2%
tax_exempt_interest_income	checked	16,664	5.5%	1,115.9	$3,078,364 × 1,115.9 = $3.44B	51.5%	2.6%
farm_rent_income	checked	2,063	5.1%	3,896.9	$-288,692 × 3,896.9 = $1.12B	74.8%	1.7%

What the table shows:

• Above 10%. Four of the seven are thin and one is dense, so the tail gate never grades any of these five:

  • investment_income_elected_form_4952 and farm_income, each above 43%;
  • casualty_loss;
  • sstb_w2_wages_from_qualified_business;
  • and the dense investment_interest_expense, at 17.5%.

  The other two, collectibles gains and bond_assets, are checked but waived in the run's d490 register.

• Above 5% and passing the gate. Eight of the 12 checked columns above 5% pass the gate unwaived. Their top-100 shares run from 47% to 75%.

• Calibration moves these shares. At the same household total (138.1M), calibration lowers the median household weight from 378 to 118 and raises the 99th percentile from 1,021 to 3,731. The largest per-household final/initial ratio here is 5.0.

  • Seven of the 17 columns were under 5% at initial weights. For example, collectibles gains rose from 13.9% to 48.1% and bond_assets from 4.4% to 14.9%.
  • Others fell. estate_income dropped from 21.6% to 8.5%, unrecaptured_section_1250_gain from 16.7% to 6.3%, and domestic_production_ald from 21.2% to 3.4%.

• At initial weights, 13 columns exceed 5% and 8 exceed 10%. The three above 5% at initial weights only are domestic_production_ald, stock_assets (5.3%) and partnership_income (5.3%).

• Shared largest records. The same person holds the largest record of w2_wages_from_qualified_business and of sstb_w2_wages_from_qualified_business. Another person holds the largest record of both SSTB self-employment and SSTB UBIA. No other column above 5% shares its largest record.

• Non-amount outputs. Four numeric QRF outputs are codes or rates (cps_race, detailed_occupation_recode, hourly_wage, first_home_mortgage_origination_year). They are measured like the rest, but their shares carry no dollar meaning; all are below 0.3%.

• Zero mass. Four outputs have no mass in this release (s_corp_income and the three second-home mortgage columns). They are reported with no share.

• What this release predates. Route A predates Bound each PUF QRF draw by the population slice its recipient stands for #1259's draw bound and Carry Schedule E line 32 and its passive subset on the US PUF donor #1251's line-32 split. The bound acts on raw draws at initial weights; these shares are measured on the finished release frame.

All 68 rows are in out/route_a_top1.md and out/route_a_top1.json in the evidence folder.

Decision

Whether this becomes a failing gate is a methodology choice, so it is queued for Max as d1511 rather than decided here. The choice covers:

• per-column thresholds or one threshold with reviewed exceptions;
• the weights it grades at, final or initial;
• and whether thin and dense columns join it.

Tests

• The new and touched test files pass locally on Python 3.14.7:

  • test_single_record_concentration.py: 28 passed;
  • test_gates.py with that file: 216 passed;
  • the builder's QRF tail tests: 20 passed;
  • test_us_release_gate_dry_run.py: 56 passed;
  • test_us_post_export_probe.py: 23 passed.
  • The whole of test_us_fiscal_refresh_builder.py, test_us_puf_capital_gains_tail.py and the data contract's test_contract.py: 649 passed, 8 skipped.

  The full suites run in CI.

• ruff check and ruff format --check are clean on every changed file.

• CI: pending on the head commit (the repository's runners were stalled for hours earlier today).

Review

Independent review (GPT-6.1 Sol, subfleet run --task review --tier hard) is pending at this head.

Evidence (outside the repository): ~/PolicyEngine/_reviews/us-qrf-tail-top1-share/. It holds scripts/route_a_top1.py, scripts/mutants.py, out/route_a_top1.{md,json} and out/mutants.log.

axiom: n/a: release diagnostics; no policy rule changes.

🤖 Generated with Claude Code