Repository navigation
Add 2023 county-equivalent geography to US household ladders - #1376
Conversation
Adding puma_county_vintage to the seed protocol's direct kernel modules moves the seed-protocol and seed-map digests, the field-usage counts, the pointer inventory digest, the loader golden vector, the US pool-tool spec_sha256 and the regenerated us-f0-coverage.json. Each takes its observed value, per the documented re-pin procedure. The spine-blindness allowlist names the two county-vintage modules, and a changelog fragment records the new county_fips_2023 column. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
f46a8e3 to
b3dc8ad
Compare
Main's microcosm#454 re-pinned the same attestation values, so each conflict takes main's side and then the values this branch moves are recomputed on the merged tree: seed_map and seed_protocol, the field counts (42_218 configuration, 9_781 resolved bindings), the resolved_seed_protocol pin (827), the pointer inventory digest, the loader golden vector, the US pool-tool spec_sha256, and the regenerated us-f0-coverage.json (the coverage tool's --check passes, 41/41 inventory checks). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI on the merged head: the pool build tool now reaches county_vintage.py, puma_county_vintage.py and block_ladder_sources.py (which both import for CT_STATE_FIPS), all already classified, so the pinned graph is 78 modules. The added configuration field is compiler_semantic with no generation-0 effect, so those counts move to 27_718 and 3_687; every other asserted count matches the regenerated us-f0-coverage.json. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Integer tract GEOIDs become floats once pooled beside tractless ACS rows, and the 2023 county mapper rejected them. The donor tracts are already validated as integral, so they are normalized to 11-digit text before mapping. Regression: integer Tolland tracts still land in 09150 and 09110. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Fixed the round-2 review's nonblocking P3 (integer donor tracts) in 62ea8e8: pooled donor tracts, validated as integral by |
…dger bb18273 moved the configuration total to 42_218 and corrected the engine-contract coverage test, but this engine-free test still counted 27_717 compiler-semantic and 3_686 no-generation-0-effect fields, so both breakdowns summed to 42_217. The new county field is one of each. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Round-4 delta review (GPT-6.1 Sol, Subfleet 20261024-020103-lwi-r4-1376) APPROVE at a13de09, no findings. It confirmed both field-usage breakdowns now sum to 42_218, the coverage JSON regenerates byte for byte (41/41 inventory checks), and no other engine-free ledger pin disagrees. Round 3's P3 (integer-typed donor tracts) was fixed at 62ea8e8 and verified by mutation in round 3. Merging on green CI. |
The repinner moved the field-usage tests' mode and generation-0 counts by value transition, taking each old value from the committed coverage JSON. A merge can break that: in a trial merge of #1376 into this branch, git auto-merged #1376's compiler_semantic 27718 into the JSON while the conflicted test kept 27_717. The tool then saw no transition, left the stale literal, and --check passed while the test failed. The five asserted count dictionaries (source, mode and generation-0 counts in the engine-free field-usage test; mode and generation-0 counts in the coverage-tool test) are now rewritten by key from the ledger. The keys must match the observed ledger exactly. Totals still move by value transition from the module's own pins. On this branch --check is unchanged. On the trial merge the tool now flags and fixes the stale count, and both tests pass (21 passed). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1376 and this branch both moved the F0 spec pins and the pool tool's import graph. The conflicts resolve as follows: - agent-guide.md keeps both paragraphs, in the one running paragraph main uses. - The pool build tool's runtime graph is 80 modules: this branch's two crosswalk modules plus #1376's county_vintage, puma_county_vintage and block_ladder_sources. - Every other conflicted pin was regenerated by the documented sequence (repinner --legacy-only, bundle generator, repinner, coverage tool). The ledger is now 42,229 configuration fields, which is this branch's +11 plus #1376's +1, and coverage passes 41/41 inventory checks. The US bundle spec_sha256 is 53fb8de6. The repinner's keyed rewrite caught #1376's compiler_semantic 27718, which git had auto-merged into the coverage JSON but not into the conflicted tests. A final --check passes. 885 targeted engine-free and engine-contract tests pass, plus the pool tool's 7 constants-adapter tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round-4 review found that the identity test's `r"87 unclaimed normative field"` regex has no refresh path. Every field has exactly one primary claim, so dropping the selection claim leaves exactly its observed expansion (87) unclaimed. The repinner now rewrites that one expectation from the expansion it already computes. With a stale 86 planted, --check fails and the tool restores 87. The main merge also dropped #1376's note that the puma_county_vintage draw module joins the seed protocol's attested kernel modules; it is restored above the seed hashes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
US household county geography, 2023
Investigation for microcosm#922 item 3, against base
dec45a439.Engine and downstream contracts
uv.lock:1692–1707pins policyengine-us 2.2.1, with wheel SHA-2560993a6c73fcdfbe171a796aca9d302741bd8e00ab83388ad81c15a08740318c6.The locked wheel
was independently downloaded and hash-verified. Paths below refer to files in
that wheel; there is no matching GitHub version tag. Inspection without
importing the engine finds:
policyengine_us/variables/household/demographic/geographic/county/county_enum.py:315–322names Connecticut's eight legacy counties, rather than planning regions.
policyengine_us/variables/household/demographic/geographic/county/county.py:36–69joins
county_fipsto the engine county dataset, converts the result tothat enum, and falls back to the first county in the state on an unrecognized
code. Replacing Connecticut's legacy codes would therefore change rules
evaluation rather than merely relabel geography.
policyengine_us/tools/geography/county_helpers.py:12–23explicitly loads
county_fips_2020.csv.gz. Its Connecticut rows are09001,09003,09005,09007,09009,09011,09013,09015.policyengine_us/variables/input/geography.py:130–135declares
county_fipsas a string household input. The engine'spolicyengine_us/variables/gov/states/ct/oec/c4k/ct_c4k_region.py:22–30uses county enum names to approximate Care 4 Kids regions. The unrecognized
planning-region codes would fall back to Fairfield, changing the program's
regional computation.
packages/microcosm-frame/src/microcosm/frame/adapters/policyengine_us.py:670–682documents the engine's default county-based SPM measurement and its use of
stored
county_fips.packages/microcosm-build/src/microcosm/build/us_runtime/geography_ladder.py:443–447,500–525uses
county_fipsfor NYC shares and requires the legacy county to match theblock prefix.
puma_ladder.py:464–468,517–536also checks county/state and NYCgeography. These contracts must keep their current interpretation.
packages/microcosm-build/src/microcosm/build/us_runtime/base_pool.py:529–547checks donor
county_fipsagainst the legacy tract prefix during pooling.Decision: add the five-digit text household column
county_fips_2023.Preserve
county_fipsfor the locked engine, and preserve 2020 block/tract IDs.Register the new column as deliberately stored geography metadata, since it
is not a PE-US input (
packages/microcosm-data/src/microcosm/data/stored_inputs.py:320).Reusing the existing mapping
PR #1072 already supplies the correct route:
block_ladder_sources.py:225–275parses the 2020 BAF MCD block-to-town layer.block_ladder_sources.py:393–515parses the official town-to-planning-regioncrosswalk and joins blocks through their towns, refusing missing towns.
tools/build_us_block_ladder_artifact.py:67–82,285–310identifies the Censussource URLs and applies that join for CBSA assignment.
These paths are under
packages/microcosm-build/src/microcosm/build/us_runtime/unless prefixed with
tools/. Read-only inspection of the existing cachedsources finds 169 defined towns, nine planning regions, and 49,845 defined-town
blocks. All 880 corresponding 2020 tracts resolve to exactly one planning
region. Undefined coastal-water geography is outside that support. A compact
tract table can therefore reuse the existing block/town mapping without
deriving boundaries or guessing from legacy counties.
The PUMA ladder's default county-only path needs a population-weighted tract
draw conditional on the already drawn PUMA and legacy county. This draw runs
after the existing draws and supplies only the new county column. With
assign_tract=True, the new county comes directly from the assigned tract.Implementation
The compact crosswalk, original Census town crosswalk, and provenance JSON
are packaged under
us_runtime/data/. The loader checks SHA-256 digests.tools/build_us_county_vintage_crosswalk.pyreproduces them from the existingcached CT BAF, town crosswalk, and P.L. 94-171 archives using the maintained
parsers. Its normal CLI was run successfully without building a ladder.
Both household ladder assignments now write
county_fips_2023. PUMA'sconditional draw is isolated in
puma_county_vintage.py. The new stored-inputregister entry is bound to its producer by tests. Both geography gates require
the additional column and check completeness, text format, valid CT regions,
non-CT identity, and agreement with assigned block/tract geography when present.
Consequently, older artifacts without the new column fail these updated gates
and the default L0 export's required-geography check.
Validation
Engine-free checks cover national support of 3,144 county-equivalents in the
50 states plus DC, one region per CT town, non-CT identity, one non-null value
per household, and CT population conservation across county vintages. Hypothesis
checks generated mapping invariants and both PUMA tract modes are exercised.
The source-backed crosswalk conserves Connecticut's 3,605,944 people across
all 42,008 populated blocks, eight legacy counties and nine planning regions.
The primary engine-free run passed 275 tests, with one skip, serially
on Python 3.14.7. It covered the four new mapping/assignment/gate modules and
the existing block/PUMA ladder, stored-input register, L0 export, base-pool,
and block/PUMA source suites.
uv run ruff check .,python3 tools/ci_test_plan.py verify, andgit diff --checkpassed.Two additional seeded regression tests passed against outputs captured from
the original
dec45a439PUMA implementation: PUMA, CD, legacy county and optionaltract assignments remain unchanged. Across both invocations: 277 passed,
one skipped.
No full ladder NPZ is present in this workspace or the existing geography
cache. Use source-derived fixtures for CI and provide an optional read-only
check against an existing national ladder.
The skipped test is
test_us_county_vintage.py::test_existing_full_ladder_has_3144_counties_without_rebuilding.To run it, set
MICROCOSM_US_BLOCK_LADDER_ARTIFACTto an existing nationalblock-ladder NPZ and invoke that pytest node. The passing fixture check covers
the complete county support universe using 4,015 actual Census tracts; it does
not certify a full NPZ artifact. No ladder or release was built or published.
Review fixes (2026-10-24, a788815 and 012dd53)
An independent review found two problems; both are fixed.
base_pool._assign_pooled_puma_ladderrestored each preserved donor's legacycounty_fipsand district, but leftcounty_fips_2023as the sampled county. Reproduced: a donor in tract 09013890203 (Tolland, Northeastern region 09150) got 09110. Donor rows now takecounty_fips_2023from their retained tract viacounty_fips_2023_from_tracts.us_puma_ladder_gatenow validatescounty_fips_2023againsttract_geoidfor every row that carries a retained tract, even whenassign_tract=False. ACS rows have no tract and are unaffected.county_fips_2023is now in_PRODUCER_BOUND_ENTRIES, which repairs the two microcosm-data stored-input tests the register addition had broken.Seeded RNG order is unchanged. Local run at 012dd53:
test_us_base_pool.py,test_us_county_vintage_ladders.pyandpackages/microcosm-data/testsgave 1193 passed, 2 skipped; ruff check and format pass. A re-review is running.Release-tool follow-up
The task excludes edits to
tools/build_us_acs_local_release.pyandtools/us_acs_local_cd_surface.py. In particular,tools/build_us_acs_local_release.py:1203–1211has a lean household checkpointallowlist containing only
county_fips; that checkpoint will need to retaincounty_fips_2023before the local county-target lane can use the new geography.County calibration/target joins must explicitly select the 2023 column while
engine computation continues to use the legacy column.
axiom: n/a: data geography infrastructure