Register weighted-microdata rules, weight the builder tests, and check change weights and household counts

AGENTS.md

@@ -18,6 +18,10 @@ Before changing correlation identifiers, telemetry transport, spans, stage
 names, logging, or observability failure handling, read
 `docs/engineering/skills/observability.md`.
 
+When changing code that aggregates simulation microdata (the output builders,
+the national reduce, Stage 12 aggregation, or anything that reads
+`output_dataset.data`), read `docs/engineering/skills/microdata-weights.md`.
+
 When adding or changing a GitHub Actions workflow step or a script under
 `.github/scripts/`, read `docs/engineering/skills/ci-workflows.md`.
 
@@ -138,6 +142,13 @@ To add or change a rule, read `docs/engineering/skills/rules-registry.md`.
 - `sim.observability.id-is-diagnostic-only`: observability_id has diagnostic meaning only: never use it for idempotency, authorization, database identity, routing or business logic. (doc-only, not checked: follow it yourself)
 - `sim.observability.registered-stages-only`: Span names come from the stage plans in policyengine_simulation_observability.stages; runtime code never adds span-name string literals. (doc-only, not checked: follow it yourself)
 
+### Weighted microdata
+
+- `sim.microdata.builder-tests-use-weighted-frames`: Output-builder tests build entity frames only through fixtures/weighted_frames.weighted_entity_frame, as MicroDataFrames with non-uniform weights, never as plain pd.DataFrames. (blocking: pytest `projects/policyengine-simulation-executor/tests/test_microdata_weight_boundaries.py`)
+- `sim.microdata.weight-stripping-confined`: Raw values are read from a weighted frame or column (np.asarray, np.array, .values, .to_numpy(), .tolist(), pd.DataFrame or pd.Series) only in simulation_output_common.py; there, counting entities is always allowed and every other read is a recorded exception whose list may only shrink. (blocking: pytest `projects/policyengine-simulation-executor/tests/test_microdata_weight_boundaries.py`)
+- `sim.microdata.weights-not-aggregated-as-values`: A weight column is never aggregated as a value: never the variable of a weighted aggregator (_sum_output_variable, _change_output_variable, Aggregate, ChangeAggregate) or frame[<weight>] followed by a weighted reduction; count entities with _count_output_entities. (blocking: pytest `projects/policyengine-simulation-executor/tests/test_microdata_weight_boundaries.py`)
+- `sim.microdata.weights-not-values`: Microdata columns are weights, values, labels or counts: weights are only summed plainly to count entities or passed as weights=, values are aggregated with their own entity's weights, labels only group or filter, and counts derive from weights. (doc-only, not checked: follow it yourself)
+
 ### Code quality and tests
 
 - `sim.quality.ci-cd-scripts-ruff-exemptions`: CI/CD scripts under .github may print their output (T201); every other ruff rule applies to them. (blocking: ruff per-file ignore of T201 for `.github/**`)

changelog_entry.yaml

@@ -1,8 +1,11 @@
 - bump: patch
   changes:
     added:
+      - Made the budget's reform-minus-baseline totals log a warning when the baseline and reform household frames carry different weights, since the change is then weighted by the reform's weights alone
+      - Added weighted-microdata rules to the registry, with proof from the #767 household count; the executor's code never aggregates a weight column as a value, reads raw values from weighted frames only in simulation_output_common.py (the UK local-authority breakdown's plain household frame is its one recorded exception there, in a list that may only shrink), and its output-builder tests build entity frames through one shared factory with non-uniform weights, while a doc-only rule records that weights, values, labels and counts each allow different operations
       - Made CLAUDE.md and the Copilot instructions thin pointers to AGENTS.md, required workflow logic longer than one line to live in tested and shellchecked scripts, and registered the tracking-issue rule
       - Added a rules registry (rules/registry.toml) with a standard-library tool that validates it, checks every rule is enforced where it claims, and renders docs/generated/rules.md and the Rules block in AGENTS.md
+      - Made the budget refuse a household count outside its country's plausible range (above 200 million in the US or 60 million in the UK, negative or not finite), raising ImplausibleEntityCountError instead of publishing a count like #767's
     changed:
       - Enabled every ruff rule (select ALL), ignoring only COM812 and ISC001, which conflict with ruff format, and the D203 and D213 halves of the mutually exclusive docstring pairs; complex functions, long parameter lists and magic numbers were split into smaller helpers and grouped parameters, each behind new characterization tests, across the contract and persistence validation, entry settings, app factory and Stage 12 adapter, gateway route resolution, Stage 12 coordination, workers, parity and qualification, the precompute and national reduce, the routing publisher and its command line, the release sentinel, the package update helper and the shared rules tool; Modal function, FastAPI route, Pydantic validator and command-line signatures and every OpenAPI document are unchanged, and three library APIs that other projects import keep a reasoned PLR0913 suppression
       - Enabled ruff's argument, idiom and pytest-style rules (ARG, ASYNC, B, C4, FBT, PD, PERF, PGH, PLW, PT, PTH, RUF, SIM); unused arguments are removed, underscored or explained, FastAPI auth dependencies use Annotated, re-raised errors keep their cause, and boolean options of in-repo helpers are keyword-only; OpenAPI documents are unchanged, and the shared rules tool's unittest tests are exempt from pytest-style rules; tests take the unused-argument rules (ARG001, ARG002, ARG005) as file-level exceptions, since their fakes and stubs mirror real signatures; best-effort telemetry and cleanup calls that swallow any error use contextlib.suppress, with the reason as a comment

docs/engineering/skills/README.md

@@ -15,6 +15,8 @@ Current skills:
   verification, and title conventions.
 - `testing.md`: test placement, dependency boundaries, and expected validation
   commands.
+- `microdata-weights.md`: weights, values, labels and counts in simulation
+  microdata, and the guards that keep weights from being aggregated as values.
 - `observability.md`: identifier ownership, HTTP and Modal transport,
   persistence, registered runtime stages, trace boundaries, and failure
   isolation.

docs/engineering/skills/microdata-weights.md

@@ -0,0 +1,88 @@
+# Microdata Weights Skill
+
+Read this before changing code that aggregates simulation microdata: the
+`simulation_output_*.py` builders, `segmented_national_reduce` and
+`stage12_runtime/aggregation`, or anything else that reads
+`output_dataset.data` or builds a `MicroDataFrame`.
+
+## Weights aren't values
+
+A microdata column is one of four kinds, and each allows different operations
+(rule `sim.microdata.weights-not-values`):
+
+| Kind | Examples | Allowed | Never |
+| --- | --- | --- | --- |
+| Weight | `household_weight`, `person_weight` | A plain (unweighted) sum, to count entities; use as `weights=` | Aggregation as a value through a weighted frame or aggregator |
+| Value | `household_net_income`, `household_tax` | Weighted aggregation, with its own entity's weights | Aggregation after the weights are stripped |
+| Label | `household_income_decile`, `age`, `is_male`, `race` | Grouping and filtering | Summing or averaging |
+| Count | `households`, people per decile | Derived from weights | |
+
+The executor's entity frames are microdf `MicroDataFrame`s weighted by
+`<entity>_weight`: policyengine builds them that way, and so do
+`segmented_national_reduce._us_dataset_from_frames` and
+`stage12_runtime/aggregation._dataset_from_frames` when they rebuild a dataset.
+So `.sum()`, `.mean()` and `.count()` on any column are already weighted.
+
+In #766 (fixed by #767), `build_budgetary_impact` passed `"household_weight"`
+to `_sum_output_variable`, the helper it uses for ordinary values. That
+weighted the weight column by itself, giving the sum of squared weights, and
+`budget.households` came out about 10^4 times too high on every run. It
+escaped because the builder test used plain `pd.DataFrame`s, which carry no
+weights. Count entities with `_count_output_entities` instead.
+
+## Rules
+
+- `sim.microdata.weights-not-aggregated-as-values`: a weight is never the
+  variable of a weighted aggregator (`_sum_output_variable`,
+  `_change_output_variable`, policyengine's `Aggregate` and `ChangeAggregate`)
+  or `<frame>[<weight>]` followed by a weighted reduction (`sum`, `mean`,
+  `median`, `quantile`, `std`, `count`, `groupby`, `cumsum`).
+- `sim.microdata.weight-stripping-confined`: raw values are read from a
+  weighted frame or column (`np.asarray`, `np.array`, `.values`, `.to_numpy()`,
+  `.tolist()`, `pd.DataFrame(...)`, `pd.Series(...)`) only in
+  `simulation_output_common.py`, which holds `_count_output_entities`. Summing
+  stripped values gives an unweighted sample total, the mirror image of #767.
+  Within that module, counting entities is the one permanent read; every other
+  read is a recorded exception, and the list of exceptions may only shrink.
+- `sim.microdata.builder-tests-use-weighted-frames`: the output-builder tests
+  (`tests/test_simulation_output_*.py`) build entity frames only through
+  `fixtures/weighted_frames.weighted_entity_frame`, which returns a
+  `MicroDataFrame` with non-uniform weights, never as a plain `pd.DataFrame`.
+
+The guards live in
+`projects/policyengine-simulation-executor/tests/test_microdata_weight_boundaries.py`
+and run in the executor's `Test` job.
+
+## How the guard recognises a weight
+
+The guards read the source, so they recognise names, not types:
+
+- a weight is a string or f-string ending in `_weight` or `_weights` (including
+  `f"{entity}_weight"`), a name assigned from one, or a `.weights` attribute;
+- a weighted frame is `<simulation>.output_dataset.data.<entity>` (or
+  `getattr(<simulation>.output_dataset.data, ...)`), `_entity_data(...)`, a
+  `MicroDataFrame` or `MicroSeries` built with `weights=`, a name assigned from
+  one, and anything indexed, filtered or computed from those.
+
+They don't follow data through function calls or containers. A weight passed
+through a helper escapes them, and so do whole-frame readers that iterate
+`output_dataset.data.entity_data` (`simulation_microdata`, the Stage 12
+simulation worker and the precompute determinism check).
+
+## Handling a finding
+
+Each finding needs a decision, recorded in the PR:
+
+- **Fix it.** Count entities with `_count_output_entities`, and aggregate values
+  through the weighted frame or the aggregators.
+- **Move it, as a recorded exception.** A strip that can't be fixed yet moves
+  into `simulation_output_common.py` as a named helper, with a docstring saying
+  why and what would remove it.
+
+Recorded exceptions are listed in
+`rules/baselines/sim-microdata-weight-stripping.txt`. That list may only
+shrink, so adding one is a deliberate change to the rule's baseline. Today it
+holds one: `_unweighted_household_frame`, the plain household frame the UK
+local-authority breakdown hands policyengine's longwise geography impact. It
+goes once that breakdown hands policyengine the simulations, as the
+constituency and US congressional district breakdowns already do.

docs/generated/rules.md

@@ -8,11 +8,11 @@ Every rule that governs `policyengine-sim-api` is listed here with how it is enf
 
 | Tier | enforced | ratcheted | advisory | doc-only | Total |
 | --- | ---: | ---: | ---: | ---: | ---: |
-| mechanical | 40 | 1 | 0 | 14 | 55 |
+| mechanical | 42 | 2 | 0 | 15 | 59 |
 | metric | 0 | 0 | 0 | 0 | 0 |
 | judgment | 0 | 0 | 0 | 0 | 0 |
 
-**Blocking:** 41 of 55 rules.
+**Blocking:** 44 of 59 rules.
 
 ## Gaps
 
@@ -22,6 +22,7 @@ Every rule that governs `policyengine-sim-api` is listed here with how it is enf
 - `sim.boundaries.unit-tests-need-no-services`: Unit tests never require real network credentials, Modal, Hugging Face, GCP or deployed services; those seams are mocked.
 - `sim.contracts.regenerate-clients`: When public API schemas change, regenerate the clients and run the generated-client integration tests.
 - `sim.deploy.image-smoke-covers-every-image-lib`: The image smoke path filter includes every library mounted into a Modal image, so a change to any of them runs the smoke.
+- `sim.microdata.weights-not-values`: Microdata columns are weights, values, labels or counts: weights are only summed plainly to count entities or passed as weights=, values are aggregated with their own entity's weights, labels only group or filter, and counts derive from weights.
 - `sim.observability.failures-never-break-requests`: Invalid observability configuration fails at build or deploy time, and telemetry failures at runtime never fail a request or computation.
 - `sim.observability.header-transport-only`: observability_id travels only in the X-PolicyEngine-Observability-Id header, never in a simulation JSON body, and polling never invents one.
 - `sim.observability.id-is-diagnostic-only`: observability_id has diagnostic meaning only: never use it for idempotency, authorization, database identity, routing or business logic.
@@ -65,6 +66,7 @@ Every rule that governs `policyengine-sim-api` is listed here with how it is enf
 ### Ratcheted (existing violations still tolerated)
 
 - `sim.ci.multiline-run-is-script`: 13 remaining in `rules/baselines/ci-multiline-run.txt`
+- `sim.microdata.weight-stripping-confined`: 1 remaining in `rules/baselines/sim-microdata-weight-stripping.txt`
 
 ### Exceptions and review dates
 
@@ -242,6 +244,31 @@ flowchart LR
   r_sim_observability_registered_stages_only -.-> gap_unenforced
 ```
 
+## Weighted microdata
+
+| Rule | Statement | Tier | Status | Blocking | Enforced by | CI | Proof |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+| `sim.microdata.builder-tests-use-weighted-frames` | Output-builder tests build entity frames only through fixtures/weighted_frames.weighted_entity_frame, as MicroDataFrames with non-uniform weights, never as plain pd.DataFrames. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-executor/tests/test_microdata_weight_boundaries.py` | `pr.yml:test` | yes |
+| `sim.microdata.weight-stripping-confined` | Raw values are read from a weighted frame or column (np.asarray, np.array, .values, .to_numpy(), .tolist(), pd.DataFrame or pd.Series) only in simulation_output_common.py; there, counting entities is always allowed and every other read is a recorded exception whose list may only shrink. | mechanical | ratcheted | yes | pytest `projects/policyengine-simulation-executor/tests/test_microdata_weight_boundaries.py` | `pr.yml:test` | yes |
+| `sim.microdata.weights-not-aggregated-as-values` | A weight column is never aggregated as a value: never the variable of a weighted aggregator (_sum_output_variable, _change_output_variable, Aggregate, ChangeAggregate) or frame[<weight>] followed by a weighted reduction; count entities with _count_output_entities. | mechanical | enforced | yes | pytest `projects/policyengine-simulation-executor/tests/test_microdata_weight_boundaries.py` | `pr.yml:test` | yes |
+| `sim.microdata.weights-not-values` | Microdata columns are weights, values, labels or counts: weights are only summed plainly to count entities or passed as weights=, values are aggregated with their own entity's weights, labels only group or filter, and counts derive from weights. | mechanical | doc-only | no | nothing | — | — |
+
+```mermaid
+flowchart LR
+  r_sim_microdata_builder_tests_use_weighted_frames("sim.microdata.builder-tests-use-weighted-frames (enforced)")
+  e_pytest_projects_policyengine_simulation_executor_tests_test_microdata_weight_boundaries_py["pytest projects/policyengine-simulation-executor/tests/test_microdata_weight_boundaries.py"]
+  j_CI_pr_yml_test["CI pr.yml:test"]
+  r_sim_microdata_weight_stripping_confined("sim.microdata.weight-stripping-confined (ratcheted)")
+  r_sim_microdata_weights_not_aggregated_as_values("sim.microdata.weights-not-aggregated-as-values (enforced)")
+  r_sim_microdata_weights_not_values("sim.microdata.weights-not-values (doc-only)")
+  gap_unenforced>"not enforced"]
+  r_sim_microdata_builder_tests_use_weighted_frames --> e_pytest_projects_policyengine_simulation_executor_tests_test_microdata_weight_boundaries_py
+  e_pytest_projects_policyengine_simulation_executor_tests_test_microdata_weight_boundaries_py --> j_CI_pr_yml_test
+  r_sim_microdata_weight_stripping_confined --> e_pytest_projects_policyengine_simulation_executor_tests_test_microdata_weight_boundaries_py
+  r_sim_microdata_weights_not_aggregated_as_values --> e_pytest_projects_policyengine_simulation_executor_tests_test_microdata_weight_boundaries_py
+  r_sim_microdata_weights_not_values -.-> gap_unenforced
+```
+
 ## Code quality and tests
 
 | Rule | Statement | Tier | Status | Blocking | Enforced by | CI | Proof |

projects/policyengine-simulation-executor/fixtures/weighted_frames.py

@@ -0,0 +1,41 @@
+"""The one shared factory for entity frames in output-builder tests.
+
+policyengine holds each entity's output as a microdf ``MicroDataFrame`` weighted
+by its ``<entity>_weight`` column, so ``.sum()`` on any column is already a
+weighted sum. Builder tests build their entity frames here, so they weight
+exactly as production does: #767 escaped because a builder test used plain
+``pd.DataFrame``s, which carry no weights.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+import pandas as pd
+from microdf import MicroDataFrame
+
+if TYPE_CHECKING:
+    from collections.abc import Mapping, Sequence
+
+
+def weighted_entity_frame(
+    columns: Mapping[str, Sequence[object]], *, entity: str = "household"
+) -> MicroDataFrame:
+    """Return ``columns`` as a MicroDataFrame weighted by ``<entity>_weight``.
+
+    Rows without a weight column get the weights 1, 2, 3 and so on. A given
+    weight column must differ between rows, so that a total which ignores the
+    weights can't match a weighted expectation by accident.
+    """
+    weight_column = f"{entity}_weight"
+    frame = pd.DataFrame(dict(columns))
+    if weight_column not in frame.columns:
+        frame[weight_column] = [float(row + 1) for row in range(len(frame))]
+    weights = frame[weight_column]
+    if len(frame) > 1 and (weights == weights.iloc[0]).all():
+        msg = (
+            f"{weight_column} is the same on every row; give the rows different "
+            "weights so that unweighted totals can't pass for weighted ones"
+        )
+        raise ValueError(msg)
+    return MicroDataFrame(frame, weights=weight_column)