Use ACS-local data for US regional simulations

[anth-volk]

Fixes #717

Summary

• Read regional dataset path templates from the certified policyengine.py data_releases.{country}.region_datasets structure.
• Resolve every template to exactly one certified dataset and expose its identity in the runtime field region_dataset_identities.
• Select populace_us_2024_acs_local for US state, DC, and congressional-district simulations while retaining populace_us_2024 for national and multi-region simulations.
• Apply the same metadata-driven selection to direct and Stage 12 simulations, including annual and budget-window requests.
• Validate each runtime dataset against its declared URI, artifact revision, SHA-256 digest, and materialized population digest.
• Keep deployment precompute limited to the existing US national dataset for 2025, 2026, and 2027 and its distributed national baseline outputs. Do not prebuild ACS-local files.
• Prepare ACS-local datasets on demand in regional request workers, retaining the existing isolated dataset directories and provenance validation.
• Remove the additional Stage 12 precompute step and the regional artifact-manifest/image-download extensions. Keep the existing national executor precompute and manifest format unchanged.
• Allocate 64 GiB to US workers that can load ACS-local data; national segmented workers retain their existing allocation.
• Require and lock the published policyengine[models]==6.2.2 in both executor runtime dependency groups. Country-model and other dependency versions remain unchanged.

Dataset identity

The regional dataset resolves to revision populace-us-2024-buildo-acs-local-767312d60-20260923T074941Z with SHA-256 digest 769756c31f3ca646d12c272511744dec04c0e68870c6946dd945fbba65b6a7ec.

Dependency order

PolicyEngine/policyengine.py#552 merged and published 6.2.2 on PyPI. The executor lockfile now records that published release.

Before merging this PR, publish the temporary WIC fix in PolicyEngine/policyengine.py#566 and update both executor dependency groups and the frozen lockfile to that actual corrected release. The current 6.2.2 publication fails real ACS preparation. PolicyEngine/policyengine-api#3868 will also need its bundle pin aligned with that corrected release. Then merge and deploy this PR before the API PR, retaining the API's existing check that the selected simulation service supports its requested bundle.

Provisioning

This change adds no environment variables, secrets, service accounts, data stores, or deployment targets. The existing artifact bucket and deployment identities are reused.

Validation

Current local validation after installing the published PyPI release from the updated frozen lockfile:

• Executor focused precompute, artifact, image, deployment-script, bundle, and Stage 12 tests: 261 passed across the initial run and targeted reruns of two corrected test-fixture cases.
• On-demand dataset-selection and loading tests: 12 passed.
• Regression tests cover national-only precompute with and without forced recomputation, no additional Stage 12 precompute step, and no precomputed-artifact downloads in Stage 12 images.
• Gateway endpoint tests: 74 passed.
• Simulation-entry Stage 12 adapter tests: 18 passed.
• Shared-contract manifest tests: 8 passed.
• Direct packaged-bundle resolution confirms national US requests select populace_us_2024, while CA, state/DC, and congressional_district/CA-01 select populace_us_2024_acs_local.
• uv lock --check, relevant source formatting checks, and git diff --check pass.
• Ruff lint and formatting checks and Actionlint validation of the deployment workflow pass after the national-only correction. Earlier changed-module Pyright checks passed (0 type errors).

The complete executor suite, Docker builds, authenticated dataset checks, and live deployments were not run locally for this update. Pushing the branch triggers fresh PR checks.

Real ACS validation in PR CI

The executor image check now runs an ephemeral 8-CPU, 64-GiB Modal function that selects the certified Utah regional dataset, calls the real .py ensure_datasets API in a fresh temporary directory, and calculates a Utah baseline including WIC and household net income. Missing participation, wrong geography, invalid numeric outputs, preparation errors, and calculation errors fail the PR check. No prepared year or baseline result is reused.

This is invoked by the existing trusted-repository PR image workflow, not the local Docker integration command that excludes beta_only calculations. The source revision and hash are resolved from the installed bundle; no dataset identity, revision, package version, or hash is duplicated as a new configuration pin. The acceptance-condition tests use explicit small fixtures; the real Modal calculation does not mock its loader or model.

• Focused validation, image-check orchestration, and existing image-script unit tests: 30 passed.
• Changed-file Ruff lint and formatting and Actionlint: passed.
• Changed runtime modules pass scoped Pyright validation: 0 errors.
• The full executor suite and a separate manual paid Modal run were not run for this update. CI runs the real check after pushing; it is expected to fail while the affected 6.2.2 release remains pinned. Do not skip or mark that failure as expected.
• Existing national-only artifact precompute is unchanged. No new environment variable, secret, provisioned component, deployment, endpoint, schedule, GCS write, or database write is introduced.

The real check has a hard 30-minute function timeout, one container, and a 60-minute workflow timeout including image construction and the other checks. It reads the approximately 9.8-GB certified source before the calculation filters Utah, so this adds real data-loading and calculation cost to applicable PR image checks. See docs/us-regional-dataset-pr-validation.md.