Count households once: sum the weight column unweighted by PavelMakarchuk · Pull Request #767 · PolicyEngine/policyengine-sim-api

PavelMakarchuk · 2026-10-12T20:57:17Z

Fixes #766

Summary

budget.households was the sum of squared household weights.

build_budgetary_impact summed household_weight through _sum_output_variable, which calls data[variable].sum(). Here data is the household MicroDataFrame, which is weighted by household_weight, so .sum() on the weight column is Σ w × w. On the sparse US data that is about 10⁴ times the household count, on every run. For example, state/ny 2026 reports 135,597,233,840 households.

The fix:

_count_output_entities(simulation, entity, weight) counts entities with a plain sum of the weight column.
build_budgetary_impact uses it for households.
There is no aggregate fallback, because a weighted frame always holds its weight column, and the aggregate path would weight it again.

The other budget fields are not changed. They are meant to be weighted sums (Σ w × value).

Tests

New tests/test_simulation_output_budget.py builds the budget from real MicroDataFrames, as policyengine does. The existing builder test uses plain pd.DataFrames, which is why it never caught this.
The new test fails on main with assert 5.0 == 3.0, and it passes with the fix.
The same test checks that baseline_net_income, tax_revenue_impact and state_tax_revenue_impact stay weighted sums.
uv run pytest tests/ --ignore=tests/integration: 733 passed, 23 skipped.
ruff format --check and ruff check are clean on the changed files.
I did not run Pyright or the integration tests.

The test file is separate from test_simulation_output_builder.py so that it does not conflict with #765, which reorders that file's imports.

Downstream

policyengine-app-v2 shows the average change per household as -budget.budgetary_impact / budget.households, so that headline is about 10⁴ times too small today. Cached results keep the wrong count until they are recomputed.

🤖 Generated with Claude Code

budget.households summed household_weight through the household MicroDataFrame, whose columns are already weighted by household_weight, so it returned the sum of squared weights: about 10^4 times the count on every run (state/ny 135.6bn). Count entities with a plain sum of the weight column, and test the budget on a real MicroDataFrame. Fixes #766 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

anth-volk · 2026-10-13T13:11:02Z

Note: this will fail deployment due to issues around SPM handling in .py, but merging anyway, those are under repair

#767 landed on main after this stack branched, and its new test imports are unsorted under the I rule this PR enables. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

#767 landed on main after this stack branched. D401, which this PR enables, wants each docstring's summary line in the imperative mood. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The output-builder tests built household frames as plain pd.DataFrames, which carry no weights, so they did not weight the way production's MicroDataFrames do. That is how #767 escaped. fixtures/weighted_frames.py now builds every builder test's entity frames as MicroDataFrames, and rejects a weight column that is the same on every row. The golden macro result keeps its budget: its households now weigh 0.5 and 1.5, with values chosen so that only a correctly weighted build reproduces it. The materialized-columns budget test now expects weighted totals, derived in the test from an explicit weighted formula; it expected unweighted sums before, which production never returned. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The executor's entity frames are MicroDataFrames weighted by <entity>_weight, so any column's sum is already weighted. #767 fixed a weight column aggregated through one, which returned the sum of squared weights. Four rules now cover that class of mistake: - sim.microdata.weights-not-values (doc-only): weights, values, labels and counts each allow different operations. - sim.microdata.weights-not-aggregated-as-values: a weight is never the variable of a weighted aggregator or frame[<weight>] followed by a weighted reduction. Enforced, with no findings. - sim.microdata.weight-stripping-confined: raw values are read from a weighted frame only in simulation_output_common.py. Ratcheted: the UK geography builders' plain DataFrame handoff (_output_household) is baselined for a decision. - sim.microdata.builder-tests-use-weighted-frames: the output-builder tests build entity frames only through the shared weighted factory. Enforced, with no findings. The AST guards live in the executor's tests, so the executor leg of the Test job runs them. Each rule's proof uses the real pre-#767 budget line or the mistake it mirrors. The rationale is in docs/engineering/skills/microdata-weights.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

budget.households is a plain sum of the household weights. #767's bug published the sum of squared weights instead, about 10^4 times too high (135.6 billion for New York). build_budgetary_impact now raises ImplausibleEntityCountError for a count that is negative, not finite, or above its country's bound: 200 million US households (about 131 million, US Census Bureau Current Population Survey 2023) or 60 million UK households (about 28 million, ONS Families and households 2023). Other countries only need a finite, non-negative count. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The output-builder tests built household frames as plain pd.DataFrames, which carry no weights, so they did not weight the way production's MicroDataFrames do. That is how #767 escaped. fixtures/weighted_frames.py now builds every builder test's entity frames as MicroDataFrames, and rejects a weight column that is the same on every row. The golden macro result keeps its budget: its households now weigh 0.5 and 1.5, with values chosen so that only a correctly weighted build reproduces it. The materialized-columns budget test now expects weighted totals, derived in the test from an explicit weighted formula; it expected unweighted sums before, which production never returned. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The executor's entity frames are MicroDataFrames weighted by <entity>_weight, so any column's sum is already weighted. #767 fixed a weight column aggregated through one, which returned the sum of squared weights. Four rules now cover that class of mistake: - sim.microdata.weights-not-values (doc-only): weights, values, labels and counts each allow different operations. - sim.microdata.weights-not-aggregated-as-values: a weight is never the variable of a weighted aggregator or frame[<weight>] followed by a weighted reduction. Enforced, with no findings. - sim.microdata.weight-stripping-confined: raw values are read from a weighted frame only in simulation_output_common.py. Ratcheted: the UK geography builders' plain DataFrame handoff (_output_household) is baselined for a decision. - sim.microdata.builder-tests-use-weighted-frames: the output-builder tests build entity frames only through the shared weighted factory. Enforced, with no findings. The AST guards live in the executor's tests, so the executor leg of the Test job runs them. Each rule's proof uses the real pre-#767 budget line or the mistake it mirrors. The rationale is in docs/engineering/skills/microdata-weights.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

budget.households is a plain sum of the household weights. #767's bug published the sum of squared weights instead, about 10^4 times too high (135.6 billion for New York). build_budgetary_impact now raises ImplausibleEntityCountError for a count that is negative, not finite, or above its country's bound: 200 million US households (about 131 million, US Census Bureau Current Population Survey 2023) or 60 million UK households (about 28 million, ONS Families and households 2023). Other countries only need a finite, non-negative count. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

#767 landed on main after this stack branched, and its new test imports are unsorted under the I rule this PR enables. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The output-builder tests built household frames as plain pd.DataFrames, which carry no weights, so they did not weight the way production's MicroDataFrames do. That is how #767 escaped. fixtures/weighted_frames.py now builds every builder test's entity frames as MicroDataFrames, and rejects a weight column that is the same on every row. The golden macro result keeps its budget: its households now weigh 0.5 and 1.5, with values chosen so that only a correctly weighted build reproduces it. The materialized-columns budget test now expects weighted totals, derived in the test from an explicit weighted formula; it expected unweighted sums before, which production never returned. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The executor's entity frames are MicroDataFrames weighted by <entity>_weight, so any column's sum is already weighted. #767 fixed a weight column aggregated through one, which returned the sum of squared weights. Four rules now cover that class of mistake: - sim.microdata.weights-not-values (doc-only): weights, values, labels and counts each allow different operations. - sim.microdata.weights-not-aggregated-as-values: a weight is never the variable of a weighted aggregator or frame[<weight>] followed by a weighted reduction. Enforced, with no findings. - sim.microdata.weight-stripping-confined: raw values are read from a weighted frame only in simulation_output_common.py. Ratcheted: the UK geography builders' plain DataFrame handoff (_output_household) is baselined for a decision. - sim.microdata.builder-tests-use-weighted-frames: the output-builder tests build entity frames only through the shared weighted factory. Enforced, with no findings. The AST guards live in the executor's tests, so the executor leg of the Test job runs them. Each rule's proof uses the real pre-#767 budget line or the mistake it mirrors. The rationale is in docs/engineering/skills/microdata-weights.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

budget.households is a plain sum of the household weights. #767's bug published the sum of squared weights instead, about 10^4 times too high (135.6 billion for New York). build_budgetary_impact now raises ImplausibleEntityCountError for a count that is negative, not finite, or above its country's bound: 200 million US households (about 131 million, US Census Bureau Current Population Survey 2023) or 60 million UK households (about 28 million, ONS Families and households 2023). Other countries only need a finite, non-negative count. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Enable ruff group 1 (safe auto-fixes) and fix everything it finds Select FLY, FURB, I, PIE, PYI, RET, RSE, UP and W, and fix every finding: ruff's safe fixes (import sorting, pyupgrade), then by hand: - policyengine-fastapi re-exports JWTDecoder through __all__ and drops two unused imports (the 9 F401s noted in Phase 0). - Two tests build their JSON receipts with json.dumps, not % formatting. - smoke_app uses min() rather than sorted()[0] (ruff 0.16 FURB192). scripts/** is linted as Python 3.11, the shared rules tool's minimum, so import sorting matches policyengine-api byte for byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Run ruff check in CI and enforce the lint rule The lint job now runs ruff check . on the whole repository, and sim.quality.ruff-lint moves from advisory to enforced with the codes selected in ruff.toml. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Add a changelog entry for ruff group 1 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Sort the imports in the budget output test from #767 #767 landed on main after this stack branched, and its new test imports are unsorted under the I rule this PR enables. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Sort the imports #743 and #744 added #743 and #744 landed on main after this stack branched. A dataset-digest test from #743 and the tomllib import #744 added to the deployment assets test are unsorted under the I rule this PR enables. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Sort the imports #787 added #787 landed on main after this stack branched. It renamed the macro result fixture module to fixtures.macro_output and added fixtures.fake_object_store, and those imports in the output builder, Stage 12 artifact and Stage 12 runtime tests are unsorted under the I rule this PR enables. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Sort the imports #791 added #791 landed on main after this stack branched. I, which this PR enables, flags the import block of its new Stage 12 failure-cause tests; ruff's safe fix sorts it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Anthony Volk <14987227+anth-volk@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>