Skip to content

Load certified annual projection datasets by year #523

Description

@MaxGhenis

Microcosm's static-aging operator now supports independent annual cross-sections (PolicyEngine/microcosm#957; design tracking in PolicyEngine/microcosm#333). The production data interface will publish one native single-year H5 per supported year.

The wrapper currently extracts future-year inputs from one base file and reuses derived cache files by dataset basename and year. It needs to select explicitly published annual artifacts and preserve their projection provenance.

Required behavior:

  • Validate producer metadata.dataset_years mappings during bundle certification and retain exact revision/hash pins.
  • Load the requested annual H5 and verify its embedded year, entity layout, and source identity.
  • Keep cache entries separate across data revisions and relevant model/runtime identities.
  • Prevent country auto-extension from substituting unpublished years; preserve genuine internal historical dependencies.
  • Load requested years plus the earlier published input years required for those dependencies, without fetching future years by default.
  • Preserve native-ID regional filtering and projected household weights.

This issue covers consumer integration. A separate certification change will select the published annual release and the model version containing the Medicaid/SNAP cycle fix (PolicyEngine/policyengine-us#9539).

Activity

  1. MaxGhenis commented on Oct 9, 2026

    @MaxGhenis
    ContributorAuthor

    A concrete UK case for "preserve genuine internal historical dependencies", from #556/#557.

    policyengine-uk's basic_state_pension, new_state_pension and additional_state_pension take the data year as min(simulation.dataset.years). They split state_pension_reported at that year against its legislated rates, then scale to the simulated year's rates.

    If a published UK annual H5 for 2026 were loaded on its own, 2026 would become the data year. The State Pension would then follow whatever uprating the producer applied to state_pension_reported (CPI in policyengine-uk's own projection) instead of the triple lock. That was the bug fixed in #557: −£6.2bn of State Pension in 2026, and rate reforms scored £0 under policyengine-uk 2.102.3.

    #557 handles this for wrapper-cut year files: each one keeps the observed year's tables, and run() rebuilds the chain from that year. Consumer integration of published UK annual files needs the same: load the observed base year with the requested year, or have the producer record the observed year and its state_pension_reported. The differential test pattern in tests/test_uk_year_file_data_year.py would catch a regression.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions