Use disclosure-controlled UK geography outputs for constituency and local authority impacts

changelog_entry.yaml

@@ -1,6 +1,7 @@
 - bump: patch
   changes:
     added:
+      - Added optional disclosure fields to UK constituency and local-authority records (disclosure_tier, reported or approximate, and the 90% interval bounds of the average and relative income change); records stored before this change still validate
       - Added the rule sim.boundaries.sqlalchemy-and-alembic-only, with proof from the psycopg adapter of 08ebe1ca; code connects to databases only through SQLAlchemy, never by importing a database driver, and runtime code never changes the schema, which changes only through Alembic migrations; a guard parses every file under projects/*/src, libs/*/src and .github/scripts and rejects driver imports, alembic imports outside a migrations directory, create_all and drop_all, create and drop on tables, SQLAlchemy DDL statements and DDL text in string literals, which extends the Stage 12 DDL check's pattern to the gateway and every library
       - Made the budget's reform-minus-baseline totals log a warning when the baseline and reform household frames carry different weights, since the change is then weighted by the reform's weights alone
       - Added weighted-microdata rules to the registry, with proof from the #767 household count; the executor's code never aggregates a weight column as a value, reads raw values from weighted frames only in simulation_output_common.py (the UK local-authority breakdown's plain household frame is its one recorded exception there, in a list that may only shrink), and its output-builder tests build entity frames through one shared factory with non-uniform weights, while a doc-only rule records that weights, values, labels and counts each allow different operations
@@ -25,6 +26,7 @@
       - Ran the Modal image smoke for Stage 12 persistence changes and added policyengine-fastapi to the PR test matrix
       - Ran the full PR workflow and the Modal image smoke only on the lowest unmerged pull request in a stack, behind a new CI complete gate job that fails on the others
     fixed:
+      - Built UK constituency and local-authority results with policyengine's disclosure-controlled compute_uk_geography_disclosure instead of the deprecated, unprotected functions, in one release nested in regions and the nation; an area that rests on fewer than 10 survey households, is dominated by one, would be recovered by subtraction, or has a non-positive baseline income is left out of the list, so small areas no longer show exploding relative changes such as +1372% for a £192 loss
       - Preserved bounded reform stop dates in UK and US simulations while retaining current-law baseline artifact reuse
       - Resolved PolicyEngine-managed regional datasets using bundled data versions instead of release-manifest commit hashes when loading from GCS
       - Added a Modal app-release bundle backfill for missing data artifact revisions

docs/engineering/skills/microdata-weights.md

@@ -82,7 +82,6 @@ Each finding needs a decision, recorded in the PR:
 Recorded exceptions are listed in
 `rules/baselines/sim-microdata-weight-stripping.txt`. That list may only
 shrink, so adding one is a deliberate change to the rule's baseline. Today it
-holds one: `_unweighted_household_frame`, the plain household frame the UK
-local-authority breakdown hands policyengine's longwise geography impact. It
-goes once that breakdown hands policyengine the simulations, as the
-constituency and US congressional district breakdowns already do.
+is empty: the last one, `_unweighted_household_frame`, went when the UK
+constituency and local-authority breakdowns moved to policyengine's
+disclosure-controlled geography output, which takes the simulations.

docs/generated/rules.md

@@ -66,7 +66,7 @@ Every rule that governs `policyengine-sim-api` is listed here with how it is enf
 ### Ratcheted (existing violations still tolerated)
 
 - `sim.ci.multiline-run-is-script`: 13 remaining in `rules/baselines/ci-multiline-run.txt`
-- `sim.microdata.weight-stripping-confined`: 1 remaining in `rules/baselines/sim-microdata-weight-stripping.txt`
+- `sim.microdata.weight-stripping-confined`: 0 remaining in `rules/baselines/sim-microdata-weight-stripping.txt`
 
 ### Exceptions and review dates
 

libs/policyengine-simulation-contract/src/policyengine_simulation_contract/macro_output.py

@@ -2,7 +2,7 @@
 
 from __future__ import annotations
 
-from typing import Generic, TypeVar
+from typing import Generic, Literal, TypeVar
 
 from pydantic import BaseModel, ConfigDict, RootModel
 
@@ -234,12 +234,38 @@ class CliffImpactOutput(MacroOutputModel):
     reform: CliffImpactInSimulation
 
 
-class ConstituencyImpactRecord(MacroOutputModel):
+UKGeographyDisclosureTier = Literal["reported", "approximate"]
+
+
+class UKAreaDisclosureFields(MacroOutputModel):
+    """Disclosure tier and 90% intervals of a published UK area's income changes.
+
+    UK area results come from licensed survey microdata and pass policyengine's
+    disclosure control. An area whose income change is withheld is left out of
+    the list, so every record here has values. ``disclosure_tier`` is
+    ``reported``, or ``approximate`` when the area's effective sample is thin
+    and the interval understates the uncertainty. The ``_lower`` and ``_upper``
+    fields bound each change's 90% bootstrap interval, rounded outward.
+
+    All of these fields are optional, so results stored before disclosure
+    control still validate; there they are null.
+    """
+
+    disclosure_tier: UKGeographyDisclosureTier | None = None
+    average_household_income_change_lower: float | None = None
+    average_household_income_change_upper: float | None = None
+    relative_household_income_change_lower: float | None = None
+    relative_household_income_change_upper: float | None = None
+
+
+class ConstituencyImpactRecord(UKAreaDisclosureFields):
     """Reform impact on household net income in one UK parliamentary constituency.
 
     ``average_household_income_change`` is the average change per household and
     ``relative_household_income_change`` the relative change in the area's total
-    household net income. ``x`` and ``y`` are integer map-rendering coordinates.
+    household net income, both rounded to the order of their standard error.
+    ``population`` is the weighted number of households. ``x`` and ``y`` are
+    integer map-rendering coordinates.
     """
 
     constituency_code: str
@@ -251,7 +277,7 @@ class ConstituencyImpactRecord(MacroOutputModel):
     population: float
 
 
-class LocalAuthorityImpactRecord(MacroOutputModel):
+class LocalAuthorityImpactRecord(UKAreaDisclosureFields):
     """Reform impact on household net income in one UK local authority.
 
     Fields mean the same as in ``ConstituencyImpactRecord``.
@@ -269,7 +295,10 @@ class LocalAuthorityImpactRecord(MacroOutputModel):
 class GeographicImpactOutput(
     MacroRootModel[list[ConstituencyImpactRecord | LocalAuthorityImpactRecord]]
 ):
-    """UK constituency or local-authority impact records."""
+    """UK constituency or local-authority impact records.
+
+    Areas whose income change disclosure control withholds are left out.
+    """
 
 
 class CongressionalDistrictImpactRecord(MacroOutputModel):

libs/policyengine-simulation-contract/tests/test_macro_output_uk_geography.py

@@ -0,0 +1,74 @@
+"""UK area records stay readable before and after disclosure control."""
+
+from __future__ import annotations
+
+import pytest
+from policyengine_simulation_contract.macro_output import (
+    ConstituencyImpactRecord,
+    GeographicImpactOutput,
+    LocalAuthorityImpactRecord,
+)
+from pydantic import ValidationError
+
+STORED_BEFORE_DISCLOSURE = {
+    "constituency_code": "E14001063",
+    "constituency_name": "Aldershot",
+    "x": 56,
+    "y": -40,
+    "average_household_income_change": -150.95,
+    "relative_household_income_change": -0.0026,
+    "population": 40_530.65,
+}
+DISCLOSURE_FIELDS = {
+    "disclosure_tier": "approximate",
+    "average_household_income_change_lower": -170.0,
+    "average_household_income_change_upper": -130.0,
+    "relative_household_income_change_lower": -0.003,
+    "relative_household_income_change_upper": -0.0022,
+}
+
+
+def test_records_stored_before_disclosure_control_still_validate() -> None:
+    record = ConstituencyImpactRecord.model_validate(STORED_BEFORE_DISCLOSURE)
+
+    assert record.disclosure_tier is None
+    assert record.average_household_income_change_lower is None
+    assert record.relative_household_income_change_upper is None
+
+
+def test_records_carry_the_disclosure_tier_and_intervals() -> None:
+    record = LocalAuthorityImpactRecord.model_validate(
+        {
+            "local_authority_code": "E06000063",
+            "local_authority_name": "Cumberland",
+            "x": 4,
+            "y": 18,
+            "average_household_income_change": -150.0,
+            "relative_household_income_change": -0.0026,
+            "population": 120_000.0,
+            **DISCLOSURE_FIELDS,
+        }
+    )
+
+    assert record.model_dump(mode="json")["disclosure_tier"] == "approximate"
+    assert record.average_household_income_change_lower == -170.0
+
+
+def test_a_withheld_tier_is_not_a_record() -> None:
+    # Withheld areas are left out of the list, so no record holds that tier.
+    with pytest.raises(ValidationError):
+        ConstituencyImpactRecord.model_validate(
+            {**STORED_BEFORE_DISCLOSURE, "disclosure_tier": "withheld"}
+        )
+
+
+def test_published_values_stay_required() -> None:
+    record = {**STORED_BEFORE_DISCLOSURE, **DISCLOSURE_FIELDS}
+    record["average_household_income_change"] = None
+
+    with pytest.raises(ValidationError):
+        ConstituencyImpactRecord.model_validate(record)
+
+
+def test_an_empty_area_list_is_valid() -> None:
+    assert GeographicImpactOutput.model_validate([]).root == []

projects/policyengine-simulation-executor/src/policyengine_simulation_executor/simulation_output_builder.py

@@ -68,6 +68,7 @@ class SimulationOutputBuilder:
     uk_local_authority_metadata: UKLocalAuthorityMetadata | None = None
     progress: JobProgressReporter = NULL_JOB_PROGRESS
     _analysis: Any = field(default=None, init=False)
+    _uk_geography_disclosure: Any = field(default=None, init=False)
 
     def __post_init__(self) -> None:
         """Normalize ``country`` to lower case."""
@@ -257,18 +258,53 @@ def _build_congressional_district_impact(
                 region_code=self.resolved_region_code,
             )
 
+    def _uk_geography_disclosure_output(
+        self,
+    ) -> simulation_output_geographic.UKGeographyDisclosureOutput | None:
+        """Return UK area changes under disclosure control, computed once.
+
+        Constituencies and local authorities go through one release, so the
+        disclosure check sees both. None for other countries.
+        """
+        if self.country != "uk":
+            return None
+        if self._uk_geography_disclosure is None:
+            self._uk_geography_disclosure = (
+                simulation_output_geographic.compute_uk_geography_disclosure_output(
+                    self.country,
+                    self.baseline,
+                    self.reform,
+                    year=self._simulation_year(),
+                )
+            )
+        return self._uk_geography_disclosure
+
+    def _simulation_year(self) -> int:
+        """Return the simulated year, which labels the disclosure release."""
+        for value in (
+            self.simulation_params.get("time_period"),
+            self.simulation_params.get("year"),
+            getattr(self.dataset, "year", None),
+        ):
+            if value is None:
+                continue
+            try:
+                return int(value)
+            except (TypeError, ValueError):
+                continue
+        return 0
+
     def _build_uk_constituency_impact(self) -> GeographicImpactOutput | None:
         with self._span(ANNUAL_IMPACT_STAGES.name(Stage.OUTPUT_UK_CONSTITUENCY)):
             return simulation_output_geographic.build_uk_constituency_impact(
-                self.country, self.baseline, self.reform
+                self.country, self._uk_geography_disclosure_output()
             )
 
     def _build_uk_local_authority_impact(self) -> GeographicImpactOutput | None:
         with self._span(ANNUAL_IMPACT_STAGES.name(Stage.OUTPUT_UK_LOCAL_AUTHORITY)):
             return simulation_output_geographic.build_uk_local_authority_impact(
                 self.country,
-                self.baseline,
-                self.reform,
+                self._uk_geography_disclosure_output(),
                 uk_local_authority_metadata=self.uk_local_authority_metadata,
             )
 

projects/policyengine-simulation-executor/src/policyengine_simulation_executor/simulation_output_common.py

@@ -8,10 +8,10 @@
 from importlib import import_module
 from typing import TYPE_CHECKING, Any
 
-import pandas as pd
 from policyengine_simulation_contract.spm import spm_error_detail
 
 if TYPE_CHECKING:
+    import pandas as pd
     from _typeshed import ConvertibleToFloat
     from policyengine.core import Simulation
 
@@ -77,23 +77,6 @@ def _count_output_entities(simulation: Simulation, entity: str, weight: str) ->
     return float(np.asarray(data[weight], dtype=float).sum())
 
 
-def _unweighted_household_frame(simulation: Simulation) -> pd.DataFrame:
-    """Return the household output as a plain DataFrame, without its weights.
-
-    policyengine's ``compute_longwise_uk_geography_impacts`` takes plain frames
-    and weights each area by the baseline's ``household_weight`` column itself.
-    The UK local-authority breakdown calls it directly, so that it can apply
-    this service's boundary metadata. Handing it the weighted frame would weight
-    twice if it ever reduced a column directly.
-
-    This is a recorded exception to sim.microdata.weight-stripping-confined,
-    listed in rules/baselines/sim-microdata-weight-stripping.txt, which may only
-    shrink. It goes once the breakdown hands policyengine the simulations, as
-    the constituency and US congressional district breakdowns already do.
-    """
-    return pd.DataFrame(simulation.output_dataset.data.household)
-
-
 def _warn_on_mismatched_weights(
     baseline_data: pd.DataFrame, reform_data: pd.DataFrame, entity: str
 ) -> None: