diff --git a/CHANGELOG.md b/CHANGELOG.md index 76f12fe..fae6ca7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,32 @@ each tagged release (`v`) is archived on Zenodo with its own DOI (see [Releasing](#releasing)). Plans and acceptance criteria for each round are in `PLAN_ROUND_.md`. +## 0.14.0 (Round 15): studies across releases, spatial coverage, 2020 Census + +- Confirmatory studies across releases: CPI conversion to a declared `dollar_year`, then + each release's own RPP (`rpp_years`); `"replicates": {"source": "census_vre"}` fetches + Census VRE tables (URLs and SHA-256 recorded). Overlapping releases are refused for any + replicate source; `replicate_scope: "covariates_only"` is an explicit, hashed exception + for outcomes without replicates. Example: MI/NY change in CPI- and RPP-adjusted mean + household income, 2015-2019 to 2020-2024: +$3,111 per county, 90% CI [2,709, 3,513]. +- Spatial coverage studies (info): `spatial_regression_coverage`, `local_morans_size`. + **Default changes (owner decisions):** the spatial lag model now defaults to GMM (ML rho + coverage under ACS sampling error was 0.48 at rho 0.3 and 0.003 at 0.6; GMM 0.89-0.90); + local Moran's I now reads p-values two-sided with 9999 permutations (per-area false + positive rate 0.20 to 0.10). Previous behaviour: `method="ml"`; + `sided="folded", permutations=999`. Caveats with measured numbers in both tools' + outputs (error-model lambda attenuation; false clusters under spatially patterned + sampling error). +- spreg's per-fit stdout print is suppressed at its source, so it can no longer corrupt + the MCP stdio stream. +- 2020 Decennial Census: P.L. 94-171 (`dec/pl`) and DHC (`dec/dhc`) through the existing + pipeline. Counts are "not sample-based (MOE undefined)", never MOE 0, with a + disclosure-avoidance advisory; sampling tests refuse decennial-only input; ACS vs + census comparisons treat the count as a fixed reference. New live check F11 + (invariants and block-to-tract/county hierarchy, 5,492 values); 11/11 checks pass live. +- Remote access decisions recorded (`docs/remote_access_decisions.md`); no networking + changes in this release. + ## 0.13.0 (Round 14): confirmatory multi-state analyses - Pre-registered manifests take several states (`geography.states`, 2-digit FIPS; not diff --git a/CITATION.cff b/CITATION.cff index f19acd1..f863694 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -12,7 +12,7 @@ authors: - alias: cafeTechne license: AGPL-3.0-or-later repository-code: "https://github.com/cafeTechne/US-Census-5-Year-ACS-Data-Batch-Downloader" -version: "0.13.0" +version: "0.14.0" doi: "10.5281/zenodo.23925301" # doi: Zenodo concept DOI, added after the first archived release (see CHANGELOG.md "Releasing"). date-released: "2026-11-08" diff --git a/PLAN_ROUND_15.md b/PLAN_ROUND_15.md new file mode 100644 index 0000000..c1c229a --- /dev/null +++ b/PLAN_ROUND_15.md @@ -0,0 +1,89 @@ +# Round 15: Studies Across Releases, Spatial Coverage, 2020 Census + +**Date:** 2026-11-08 +**Status:** Implemented (section 6); released as v0.14.0 +**Baseline:** v0.13.0 +**Companions:** [PLAN_ROUND_14.md](PLAN_ROUND_14.md), [docs/remote_access_decisions.md](docs/remote_access_decisions.md) + +## 0. Front-loaded findings and boundaries + +| ID | Source | Finding | Disposition | +| :--- | :--- | :--- | :--- | +| G1 | Round 14 deferrals | A manifest's RPP adjustment covers one release only, and `run_manifest` cannot fetch Census VRE replicates itself | Workstream A | +| G2 | `docs/audits/CHECK_MAP.md` | `spatial_regression` (lag/error) and `local_morans` have no coverage study; global Moran's I has only an info study | Workstream B | +| G3 | Owner gap analysis | ACS only. 2020 Decennial Census access exists only internally, as block counts for crosswalks (`census/block_weights.py`) | Workstream C | +| G4 | Scheduled CI | First weekly live run was due 2026-11-09 | Check it and triage as in Round 13 A1 | + +**Boundary (planner):** no networking changes in this round, and no HTTP, persistent service, job queue, custom transfer API or Data Commons adapter. Remote access is the next round (SSH, per `docs/remote_access_decisions.md`). + +As before: no tuning of tolerances, bands or thresholds to make something pass. + +## 1. Workstream A: confirmatory studies across releases (P0) + +- **A1. Several releases with price adjustment.** A manifest with several releases (for example non-overlapping 5-year periods, 2015-2019 vs 2020-2024, or 1-year releases) can declare RPP adjustment for each release's own year. Dollars are converted to one declared dollar year with the existing CPI machinery (R-CPI-U-RS default), *then* place-adjusted. The order is stated in the methods text. + - The declared `dollar_year` and per-release `rpp_year` are required, so the hash fixes them; `create_manifest` fills them as in Round 14. + - Existing overlap rules for comparisons of overlapping 5-year periods stay in force. +- **A2. Automatic replicate retrieval.** `"replicates": {"source": "census_vre"}` in a manifest makes `run_manifest` fetch the VRE tables for the manifest's variables, geography and releases, using the existing `census.replicate` code and its cache. + - The VRE file hashes and URLs go into the run record. + - Variables with no published replicates (for example B19013) are reported. Under the strict confirmatory policy they block, as today, unless the manifest says which analyses use only covariate replicates. The rule is explicit, never silent. +- **A3. Example study and tests.** + - An example manifest for a change study in Michigan and New York counties across two non-overlapping 5-year periods: change in RPP- and CPI-adjusted median household income, or another pre-registered contrast the existing methods support. + - Frozen-fixture offline test, plus a live test gated by `CENSUS_LIVE_TESTS=1`. + - Docs: the preregistration recipe and a short section in the case study. + +## 2. Workstream B: coverage for spatial regression and local Moran's I (P1) + +- **B1. `spatial_regression_coverage`.** A spatial lag and an error model on a lattice with ACS-like sampling error, for each estimator the tool offers (GMM/ML). Report coverage of beta, and of rho/lambda, for the tool's default intervals. Also report bias and the LM diagnostics' size under the null. +- **B2. `local_morans_size`.** Local Moran's I under spatial independence plus sampling noise. Report: + - per-area false-positive rate and family-wise or FDR control, for the tool's default inference and correction; + - power under one clustered alternative. +- **B3. Status.** Both run first as `info` studies. + - A study whose default intervals pass the band can be promoted to gated by owner decision (D15-1). + - If a default fails, the tool's outputs get a visible caveat, or the default changes (for example to a corrected method), with evidence. Never tune the band. +- Update `docs/audits/CHECK_MAP.md`. Keep each study's runtime reasonable under `--jobs`. + +## 3. Workstream C: 2020 Decennial Census (P1, provisional) + +- **C1. Retrieval.** P.L. 94-171 (`dec/pl`) and the Demographic and Housing Characteristics file (`dec/dhc`) through the existing API client, cache and output formats, with variable metadata and codebook. Reuse the block-count code where possible. +- **C2. Correct uncertainty semantics.** Decennial values are **published counts that are not sample-based but not error-free**: + - no sampling MOE; + - coverage error, processing error, and disclosure avoidance through the TopDown differential-privacy algorithm. + + Concretely: + - Outputs never carry a zero MOE or a CV of 0 presented as precision. The uncertainty fields are explicitly "not sample-based (MOE undefined)", with a disclosure-avoidance advisory. + - The advisory notes that small geographies and detailed cells are noisier, that only certain counts are invariant (state total population, block-level housing units and occupied group-quarters facility counts), and that block-level characteristics should be aggregated before use. + - Significance tools refuse decennial-only "sampling" tests, or say exactly what they test. + - Mixing ACS and decennial values in one comparison is allowed only with a stated note. +- **C3. Validation.** A new battery check against the Census Bureau's own invariants and hierarchy: + - state total population matches the published state totals; + - block-level housing-unit counts sum exactly to tract and county totals; + - tract and county totals from DHC agree with P.L. 94-171 where both publish the same item. + + Record live, freeze fixtures, keep fixtures small (Michigan, New York, Wyoming). +- **C4. Scope.** Retrieval, semantics and validation only. No new estimators. DHC tables used in the examples: P1, P12, H1, H3. The 2010 Census is out of scope unless needed. + +## 4. Release + +v0.14.0 with a Zenodo DOI: code review (`/code-review high`), fixes, full gate (`--jobs 0`), PR, CI, live dispatch, merge, tag, Zenodo confirmation. + +## 5. Defaults taken (owner may override) + +| ID | Default | +| :--- | :--- | +| D15-1 | New spatial studies start as `info`. Promotion to gated is an owner decision after the evidence is in | +| D15-2 | Multi-release dollar conversion: CPI to the declared dollar year first, then the RPP of each release's own year | +| D15-3 | Decennial geography levels: nation, state, county, tract, block group, block (block only for P.L. 94-171 and invariant counts), place | + +--- + +## 6. Implementation status (2026-11-08) + +| Slice | Result | +| :--- | :--- | +| A releases | Multi-release adjustments (CPI to dollar year, then per-release RPP) and `census_vre` replicate retrieval. Live MI/NY change study: +3,110.80 (90% CI [2,708.93, 3,512.67]), no deviations; hand recomputation from raw VRE files agrees to about 1e-12 | +| B spatial coverage | Info studies added to the full report. Lag ML rho coverage collapses under outcome sampling error (0.48, 0.003); GMM 0.89-0.90. LISA folded p-values doubled the false-positive rate. **Owner decisions:** lag default GMM; LISA two-sided, 9999 permutations. Error-model lambda attenuation and patterned-noise false clusters remain caveats | +| C 2020 Census | `dec/pl`, `dec/dhc` through the existing pipeline; MOE undefined, never 0; disclosure-avoidance advisory; sampling tests refuse; F11 passes live and offline | +| Review | `/code-review high`: 10 findings fixed with tests (`tests/test_review_fixes_r15.py`), including overlap refusal for every replicate source and explicit-evidence decennial detection | +| Gate | Full `--gate --jobs 0`: PASS, 19 min; existing coverage rows unchanged; new spatial rows are info | + +**Deferred:** 1-year releases and decennial data in manifests; aggregation with adjustment or `census_vre`; `covariates_only` beyond `fit_replicate`; a fix for error-model lambda attenuation and for patterned sampling error in LISA (no tested method yet). **Next:** minimum SSH remote access per `docs/remote_access_decisions.md`. diff --git a/README.md b/README.md index badaa03..fe97949 100644 --- a/README.md +++ b/README.md @@ -134,6 +134,7 @@ Each multi-year run writes `_concordance.csv` (every output column traced | `place` | 160 | `SSPPPPP` | place FIPS (required) | | `zcta` | 860 | `ZZZZZ` | ZCTAs; default all ZIPs in the state (`zip2fips.json`); requested in chunks of 50 | | `msa` | 310 | `MMMMM` | CBSA codes; default all metro and micro areas | +| `block` | 100 | `SSCCCTTTTTTBBBB` | 2020 Census only (`dec/pl`, or DHC `H1`): county FIPS; one request per county, default one per state | ### Datasets @@ -143,8 +144,10 @@ Each multi-year run writes `
_concordance.csv` (every output column traced | `profile` | `/acs/acs5/profile` | `DP02`–`DP05` | adds `PE`/`PM` percentages | | `subject` | `/acs/acs5/subject` | `S…` | `E`, `M` per column (`_C01_`, …) | | `cprofile` | `/acs/acs5/cprofile` | `CP02`–`CP05` | codes embed the period; `SS` significance flags | +| `dec/pl` | `/2020/dec/pl` | 2020 Census P.L. 94-171 redistricting data: `P1`–`P5`, `H1` | `N` (count), `NA` annotation; no MOE | +| `dec/dhc` | `/2020/dec/dhc` | 2020 Census Demographic and Housing Characteristics: e.g. `P1`, `P12`, `H1`, `H3` | `N` (count), `NA` annotation; no MOE | -The dataset is inferred from the table code. +The ACS dataset is inferred from the table code. The 2020 Census datasets are selected with `dataset="dec/pl"` / `"dec/dhc"` or `product="decennial"` (DHC); see [2020 Decennial Census](#2020-decennial-census). ### Outputs @@ -167,6 +170,7 @@ Every run also writes `
_extraction_metadata.json`. It records the request | `___` | Concept segment when a code was reused for different measures | | `_SE`, `_CV`, `_RELIABILITY` | Derived standard error, CV (%), and reliability flag | | `*_est`, `*_moe`, `*_pct`, `*_pct_moe`, `*_se`, `*_cv`, `*_rel`, `*_sig` | Panel slugs; `slug_style="code"` gives Stata-safe names | +| `N`, `*_count` | 2020 Census count (and its panel slug); no MOE, SE or CV columns exist | ## Methodology @@ -204,6 +208,7 @@ Every derived result records its `method` (for example `proportion`, `proportion | F6 | Prior-period dollars adjusted with R-CPI-U-RS | Comparison Profile values in current dollars | | F7 | Z-tests between periods | Comparison Profile significance flags | | F8 | Sums of counties | State estimates | +| F11 | 2020 Census state population; block housing units summed to tracts and counties; DHC vs P.L. 94-171 | Apportionment-release state totals; published tract and county totals (invariants and hierarchy) | ## Inference @@ -232,8 +237,8 @@ Built on GeoPandas, Shapely (GEOS), pyogrio (GDAL), pyproj, and PySAL. Outputs a | :--- | :--- | :--- | | Census boundaries matched to each ACS release | `census.boundaries.get_boundaries` / `get_boundaries` | Cartographic (1:500k) for maps, TIGER/Line for areal work; vintage mismatches are errors | | Journal maps that show uncertainty | `census.maps.choropleth` / `make_map` | Quantile classes by default; legend reports class separability (Sun and Wong 2010); low-reliability areas hatched; uncertainty and significance maps | -| Spatial autocorrelation | `census.spatial.morans_i`, `local_morans` / `spatial_autocorrelation` | Reproduces PySAL references; replicate SE of Moran's I is conservative (coverage 0.987) | -| Spatial regression | `census.spatial.spatial_regression` | Lag and error models with LM diagnostics (spreg) | +| Spatial autocorrelation | `census.spatial.morans_i`, `local_morans` / `spatial_autocorrelation` | Reproduces PySAL references; replicate SE of Moran's I is conservative (coverage 0.987); local clusters use two-sided permutation p-values (9999 permutations, BH): false-positive rates at nominal under randomness | +| Spatial regression | `census.spatial.spatial_regression` | Lag (GMM default) and error (ML default) models with LM diagnostics (spreg); simulated coverage under ACS outcome error: rho 0.89-0.90 (GMM lag), lambda attenuated | | Spatial Fay–Herriot | `census.sae_spatial.spatial_fay_herriot` / `spatial_small_area_estimates` | Reproduces the R `sae` grapes example; coverage 0.891 | | Population-weighted tract crosswalks | `census.block_weights`, `harmonize_panel(weight="auto")` | Decennial block weights; Delaware test: mean error 444 vs 624 people with land-area weights | | Block groups and custom polygons | `geo_level="block group"`, `census.custom_areas` / `custom_area_mapping` | Area or block-population allocation | @@ -251,12 +256,32 @@ Built on GeoPandas, Shapely (GEOS), pyogrio (GDAL), pyproj, and PySAL. Outputs a | Cost of living between places | `census.rpp.adjust_for_place` / `adjust_for_place` | BEA Regional Price Parities; reproduces BEA real personal income for all states (2008); never applied by default | | Interactive maps | `census.web_maps.web_choropleth` / `make_web_map` | Self-contained HTML, no external requests | +## 2020 Decennial Census + +The 2020 Census P.L. 94-171 redistricting data (`dec/pl`) and the Demographic and Housing Characteristics File (`dec/dhc`) use the same client, cache, extraction pipeline, output formats, codebook and replay as the ACS: pass `dataset="dec/dhc"` (or `"dec/pl"`), or `product="decennial"`, to `plan_extraction`, `batch_download`, `download_table`, `run_batch` or `python -m census extract --product decennial`, with `years=[2020]`. Geography levels are nation, state, county, tract, block group, place and block (block only for P.L. 94-171 and the invariant DHC `H1` housing-unit count). Variable metadata for both files is archived in `metadata/dec/`. + +```python +from census import ResearchOptions, build_entities, run_batch + +run_batch("P12", build_entities("county", "26"), [2020], "out", "panel", + ResearchOptions(include_codebook=True), dataset="dec/dhc") +``` + +**Uncertainty.** Census counts are not sample-based, so they have no sampling margin of error; they are also not error-free. Outputs never carry an MOE or CV of 0: no MOE, SE, CV or reliability columns are written (`compute_reliability` adds none), the codebook's `Uncertainty` column reads "not sample-based (MOE undefined)", and the extraction metadata has `moe_confidence_level: null` and an `uncertainty` record. Every decennial plan carries a disclosure-avoidance advisory: + +- the counts carry coverage error (measured by the [Post-Enumeration Survey](https://www.census.gov/newsroom/blogs/random-samplings/2021/12/post-enumeration-measuring-coverage-error.html)), processing error, and noise added by the TopDown differential-privacy algorithm; +- only three counts are invariant (published without noise): state total population, total housing units per block, and the number of occupied group-quarters facilities by type per block ([Census DAS update, 2020-11-25](https://www.census.gov/programs-surveys/decennial-census/decade/2020/planning-management/process/disclosure-avoidance/2020-das-updates/2020-11-25.html)); +- small geographies and detailed cells are noisier, and block-level characteristics should be aggregated before use ([fact sheet](https://www.census.gov/content/dam/Census/library/factsheets/2021/protecting-the-confidentiality-of-the-2020-census-redistricting-data.pdf)). + +**Statistics.** `z_test`, `calculate_significance`, `batch_significance`, `calculate_cv` and `acs_overlap_years` refuse decennial-only inputs (`product="decennial"`), since a sampling test says nothing about disclosure-avoidance noise. An ACS estimate can be tested against a census count with `product1="5-year", product2="decennial"`: the count is a fixed reference, and the result's `comparison` and `notes` say that only ACS sampling error is measured. `derive_estimate(..., product="decennial")` returns the point estimate with MOE, SE and CV undefined. `build_analysis_frame` gives census counts NaN `_se`/`_cv`, method `not_sample_based`, release `dec_2020` (`acs5_+dec_2020` with a note when mixed with ACS variables). No new estimators are provided for decennial data; validation check F11 compares the files with the Census Bureau's invariants and hierarchy. + ## Advisories and known limitations - **Boundary vintages.** Requests spanning 2019 to 2020 at tract, ZCTA, place, county, or CBSA level produce an advisory. Tract panels can be harmonized with `harmonize_tracts_to=2010|2020` (land-area weights from the Census relationship file, or NHGIS-style population weights you supply); apportioned MOEs assume independent components and exact weights, and boundary slivers produce tiny weights. Other levels are not harmonized. - **Medians from coarse brackets.** Household-income medians rebuilt from B19001 are usually within 1% of the published B19013, but can differ by more than 2% because Census interpolates a finer internal distribution. - **Home values.** Before the 2022 releases, home values were not on the period end-year dollar basis (ACS user note 2023-07), so deflating them is approximate. Plans include an advisory. - **Subject tables.** Their 2013 and 2017 redesigns changed thousands of labels. Multi-year subject pulls usually need `variables=` and reviewed overrides. +- **2020 Census counts.** No margins of error exist; disclosure-avoidance noise dominates blocks and detailed cells. Aggregate before use; see [2020 Decennial Census](#2020-decennial-census). Cross-release concordance and continuity checks do not apply (one census year), and the 2010 Census is not offered. - **ZIP codes vs. ZCTAs.** `zip2fips.json` maps each ZIP to one primary county; PO-box-only ZIPs have no ZCTA. ## Repository layout diff --git a/census/__init__.py b/census/__init__.py index 0c1f8d6..59ef3df 100644 --- a/census/__init__.py +++ b/census/__init__.py @@ -1,7 +1,7 @@ # SPDX-License-Identifier: AGPL-3.0-or-later """Headless Census ACS data client and processing library.""" -__version__ = "0.13.0" +__version__ = "0.14.0" from census.api import ( DATASETS, diff --git a/census/__main__.py b/census/__main__.py index f0dfc68..42f5760 100644 --- a/census/__main__.py +++ b/census/__main__.py @@ -124,18 +124,22 @@ def _options(args: argparse.Namespace) -> ResearchOptions: allow_unverified_concordance=args.allow_unverified_concordance, allow_provisional_index=args.allow_provisional_index, harmonize_tracts_to=args.harmonize_tracts_to, + **({"product": args.product} if getattr(args, "product", None) else {}), ) def _add_extract_args(p: argparse.ArgumentParser) -> None: p.add_argument("--table", required=True) p.add_argument("--years", required=True, help="e.g. 2015-2022 or 2018,2020") - p.add_argument("--geo", default="state", help="us, state, county, tract, place, zcta, msa") + p.add_argument("--geo", default="state", + help="us, state, county, tract, place, zcta, msa; 2020 Census also 'block group', block") p.add_argument("--state", help="2-digit state FIPS, or 'all' (50 states + DC)") p.add_argument("--include-pr", action="store_true", help="with --state all, add Puerto Rico") p.add_argument("--codes", help="comma-separated place/county/ZCTA/CBSA codes") p.add_argument("--variables", help="comma-separated codes, YEAR:CODE, or concept IDs") - p.add_argument("--dataset", help="acs5, profile, subject, cprofile (inferred)") + p.add_argument("--dataset", help="acs5, profile, subject, cprofile (inferred); 2020 Census: dec/dhc, dec/pl") + p.add_argument("--product", choices=("5-year", "1-year", "decennial"), + help="release family (default 5-year; 'decennial' = 2020 Census DHC unless --dataset dec/pl)") p.add_argument("--format", default="panel", choices=OUTPUT_FORMATS) p.add_argument("--concordance-mode", default="fail", choices=CONCORDANCE_MODES) p.add_argument("--allow-unverified-concordance", action="store_true") @@ -262,7 +266,7 @@ def cmd_derive(args: argparse.Namespace) -> int: return _emit( workflows.derive_estimate( args.operation, _floats(args.estimates), _floats(args.moes), args.percent, - args.year_current, args.year_prior, + args.year_current, args.year_prior, product=args.product, ), args.json, ) @@ -452,10 +456,12 @@ def build_parser() -> argparse.ArgumentParser: p = sub.add_parser("derive", help="derived estimate and MOE (Census approximation formulas)") p.add_argument("operation", choices=("proportion", "ratio", "product", "difference", "percent_change", "sum")) p.add_argument("--estimates", required=True, help="comma-separated, e.g. numerator,denominator") - p.add_argument("--moes", required=True, help="comma-separated 90%% MOEs") + p.add_argument("--moes", default="", help="comma-separated 90%% MOEs (omit with --product decennial)") p.add_argument("--percent", action="store_true", help="0-100 scale for proportion/ratio") p.add_argument("--year-current", type=int) p.add_argument("--year-prior", type=int) + p.add_argument("--product", choices=("5-year", "1-year", "decennial"), + help="'decennial': 2020 Census counts, point estimate only (MOE undefined)") p.add_argument("--json", action="store_true") p.set_defaults(func=cmd_derive) @@ -492,8 +498,9 @@ def build_parser() -> argparse.ArgumentParser: p.add_argument("--year2", type=int) p.add_argument("--confidence", type=float, default=0.90) p.add_argument("--correction", default="holm", choices=("holm", "bonferroni", "bh", "by")) - p.add_argument("--product", default="5-year", choices=("5-year", "1-year"), - help="ACS product of both estimates (1-year releases do not overlap)") + p.add_argument("--product", default="5-year", choices=("5-year", "1-year", "decennial"), + help="ACS product of both estimates (1-year releases do not overlap); " + "'decennial' is refused: census counts are not sample-based") p.add_argument("--out") p.add_argument("--json", action="store_true") p.set_defaults(func=cmd_significance) diff --git a/census/analysis.py b/census/analysis.py index 01334f3..b218232 100644 --- a/census/analysis.py +++ b/census/analysis.py @@ -22,8 +22,11 @@ (zero replicate denominator) are set to 0 as in VRE documentation Example 7. """ +import glob +import json import os import re +from functools import lru_cache from typing import Any, Dict, List, Mapping, Optional, Sequence, Tuple, Union import numpy as np @@ -59,18 +62,75 @@ def _read(source: Union[str, pd.DataFrame]) -> pd.DataFrame: path = str(source) ext = os.path.splitext(path)[1].lower() if ext == ".parquet": - return pd.read_parquet(path) - if ext in (".xlsx", ".xls"): - return pd.read_excel(path, dtype={"GEOID": str, "GEO_ID": str}) - return pd.read_csv(path, dtype={"GEOID": str, "GEO_ID": str}, low_memory=False) + df = pd.read_parquet(path) # keeps the attrs written by the extraction (decennial evidence) + elif ext in (".xlsx", ".xls"): + df = pd.read_excel(path, dtype={"GEOID": str, "GEO_ID": str}) + else: + df = pd.read_csv(path, dtype={"GEOID": str, "GEO_ID": str}, low_memory=False) + if not _attrs_decennial(df.attrs): + df.attrs.update(_extraction_attrs(path)) + return df + + +def _attrs_decennial(attrs: Mapping[str, Any]) -> bool: + """True when frame attrs say the data are 2020 Census counts (set by the decennial extraction).""" + from .decennial import DECENNIAL_DATASETS, PRODUCT_DECENNIAL + return bool(attrs.get("decennial") is True or attrs.get("product") == PRODUCT_DECENNIAL + or attrs.get("dataset") in DECENNIAL_DATASETS) + + +def _extraction_attrs(path: str) -> Dict[str, Any]: + """Decennial attrs for a file written by a decennial run_batch: its ``*_extraction_metadata.json`` + (in the file's folder or the one above) lists the file among its outputs and names a decennial + product; the codebook maps the panel slugs to their codes. Empty for anything else.""" + from .decennial import DECENNIAL_DATASETS, PRODUCT_DECENNIAL + target = os.path.abspath(path) + folder = os.path.dirname(target) + for d in (folder, os.path.dirname(folder)): + for meta_path in sorted(glob.glob(os.path.join(glob.escape(d), "*_extraction_metadata.json"))): + try: + with open(meta_path, encoding="utf-8") as f: + meta = json.load(f) + except (OSError, ValueError): + continue + rel = os.path.relpath(target, d).replace(os.sep, "/") + if not isinstance(meta, dict) or rel not in (meta.get("output_files") or {}): + continue + if meta.get("product") != PRODUCT_DECENNIAL and meta.get("dataset") not in DECENNIAL_DATASETS: + return {} + out: Dict[str, Any] = {"product": PRODUCT_DECENNIAL, "dataset": meta.get("dataset")} + book = os.path.join(d, os.path.basename(str((meta.get("outputs") or {}).get("codebook") or ""))) + if os.path.isfile(book): + try: + cb = pd.read_csv(book, dtype=str) + out["variable_codes"] = {s: c for s, c in zip(cb["Variable_Slug"], cb["Variable_Code"]) + if isinstance(s, str) and isinstance(c, str) + and (parse_variable_code(c) or ("", ""))[1] == "N"} + except (OSError, ValueError, KeyError): + pass + return out + return {} def load_source(source: Source) -> pd.DataFrame: - """Read one or more wide/panel files or frames and stack them.""" + """Read one or more wide/panel files or frames and stack them. + + ``attrs["decennial_columns"]`` lists the columns that came from 2020 Census + sources (frames or files carrying the decennial extraction's attrs or + metadata) and ``attrs["decennial_codes"]`` maps such columns to their + variable codes (``total_count`` -> ``P1_001N``).""" items = [source] if isinstance(source, (str, pd.DataFrame)) else list(source) if not items: raise ValueError("no source given") - df = pd.concat([_read(s) for s in items], ignore_index=True, sort=False) + frames = [_read(s) for s in items] + dec_columns: List[str] = [] + dec_codes: Dict[str, str] = {} + for f in frames: + if _attrs_decennial(f.attrs): + dec_columns += [str(c) for c in f.columns if str(c) not in dec_columns] + dec_codes.update({str(k): str(v) for k, v in (f.attrs.get("variable_codes") or {}).items()}) + df = pd.concat(frames, ignore_index=True, sort=False) + df.attrs = {"decennial_columns": dec_columns, "decennial_codes": dec_codes} if "GEOID" not in df.columns: if "GEO_ID" in df.columns: df["GEOID"] = df["GEO_ID"] @@ -82,13 +142,16 @@ def load_source(source: Source) -> pd.DataFrame: return df -def _resolve(df: pd.DataFrame, ref: Any) -> Tuple[str, Optional[str]]: +def _resolve(df: pd.DataFrame, ref: Any, decennial: Optional[bool] = None) -> Tuple[str, Optional[str]]: """(estimate column, MOE column or None) for a variable reference. Accepts a code (``B17001_002E``/``..PE``), a base (``B17001_002``), a slugged panel stem (```` with ``_est``/``_moe`` or ``_pct``/``_pct_moe``), or an explicit ``(est, moe)`` pair - (``moe`` may be None for a variable without sampling error). + (``moe`` may be None for a variable without sampling error). In a frame + identified as decennial (:func:`is_decennial_count`) a base such as + ``P1_001`` also resolves to its count column ``P1_001N`` or that column's + panel slug (``total_count``). """ if isinstance(ref, (tuple, list)): est, moe = ref @@ -103,6 +166,13 @@ def _resolve(df: pd.DataFrame, ref: Any) -> Tuple[str, Optional[str]]: return e, m if ref in df.columns: return ref, None + if decennial is not False: + if f"{ref}N" in df.columns and is_decennial_count(f"{ref}N", df, decennial): + return f"{ref}N", None + codes = (getattr(df, "attrs", None) or {}).get("decennial_codes") or {} + for col, code in codes.items(): + if code in (ref, f"{ref}N") and col in df.columns and is_decennial_count(col, df, decennial): + return col, None raise KeyError(f"no estimate/MOE columns found for {ref!r}") @@ -117,7 +187,7 @@ def _default_name(ref: Any) -> str: return str(ref) -def _default_variables(df: pd.DataFrame) -> Dict[str, Any]: +def _default_variables(df: pd.DataFrame, decennial: Optional[bool] = None) -> Dict[str, Any]: out: Dict[str, Any] = {} for key, est, moe in pair_estimates_and_moes(df.columns): out[key] = (est, moe) @@ -129,9 +199,59 @@ def _default_variables(df: pd.DataFrame) -> Dict[str, Any]: moe = f"{stem}_moe" if col.endswith("_est") else f"{stem}_pct_moe" if moe in df.columns: out[stem if col.endswith("_est") else f"{stem}_pct"] = (col, moe) + for col in df.columns: # 2020 Census counts (no MOE), in frames identified as decennial + col = str(col) + if col in ID_COLUMNS or not is_decennial_count(col, df, decennial): + continue + parsed = parse_variable_code(col) + out.setdefault(parsed[0] if parsed else col, (col, None)) return out +METHOD_NOT_SAMPLE_BASED = "not_sample_based" + + +@lru_cache(maxsize=1) +def _decennial_archive_codes() -> frozenset: + """Count codes (``P1_001N``) of the archived 2020 dec/pl and dec/dhc metadata (offline).""" + from .decennial import DECENNIAL_DATASETS, DECENNIAL_YEAR + from .metadata import archived_vintages, vintage_variables + codes: set = set() + for ds in DECENNIAL_DATASETS: + try: + if DECENNIAL_YEAR in archived_vintages(ds): + codes |= {str(c) for c in vintage_variables(ds, DECENNIAL_YEAR)[0] if str(c).endswith("N")} + except Exception: # noqa: BLE001 - an unreadable archive gives no code evidence + continue + return frozenset(codes) + + +def is_decennial_count(column: str, frame: Optional[pd.DataFrame] = None, + decennial: Optional[bool] = None) -> bool: + """True for a 2020 Census count column, on explicit evidence only. + + A raw code ``N`` found in the archived 2020 dec/pl or dec/dhc metadata + is decennial. Other ``N`` codes and panel slugs ``*_count`` are + decennial only when ``decennial`` is True or ``frame`` comes from a decennial + extraction (its attrs: ``product``/``dataset`` set by run_batch, or + ``decennial_columns`` from :func:`load_source`). ``decennial=False`` turns + detection off. An ordinary column such as ``crime_count`` is therefore not a + census count.""" + if decennial is False: + return False + col = str(column) + parsed = parse_variable_code(col) + is_n = bool(parsed and parsed[1] == "N") + if is_n and col in _decennial_archive_codes(): + return True + if not (is_n or (parsed is None and col.endswith("_count"))): + return False + if decennial: + return True + attrs = getattr(frame, "attrs", None) or {} + return col in (attrs.get("decennial_columns") or ()) or _attrs_decennial(attrs) + + def _clean(est: pd.Series, moe: Optional[pd.Series]) -> Tuple[np.ndarray, np.ndarray]: sentinels = np.array(sorted(CENSUS_SENTINELS), dtype=float) e = pd.to_numeric(est, errors="coerce").to_numpy(float) @@ -170,6 +290,7 @@ def analysis_frame( keep_columns: Sequence[str] = (), concept_ids: Optional[Mapping[str, str]] = None, release: Optional[str] = None, + decennial: Optional[bool] = None, ) -> pd.DataFrame: """Build a model-ready frame (one row per GEOID, or per GEOID and Year). @@ -183,10 +304,23 @@ def analysis_frame( Provenance: ``release`` column (``acs5_`` when Year is present, or the ``release`` argument), and ``frame.attrs["variables"]`` recording each variable's source columns, method and concept ID. + + 2020 Census counts are not sample-based: their ``_se`` and ``_cv`` are NaN + (undefined, never 0) and their method is ``not_sample_based``; release is + ``dec_2020``, or ``acs5_+dec_2020`` with a note in + ``frame.attrs["notes"]`` when ACS and decennial variables are mixed. Counts + are recognized only from explicit evidence (:func:`is_decennial_count`): + archived 2020 Census codes (``P1_001N``), sources from a decennial + extraction (frame attrs, Parquet attrs, or the extraction metadata next to a + file, with their ``*_count`` slugs), or ``decennial=True``; + ``decennial=False`` turns recognition off. Other columns without an MOE + (e.g. ``crime_count``) get SE 0 and method ``no_sampling_error``. In + decennial frames counts are default variables and resolve by base code + (``P1_001``). """ df = load_source(source) if variables is None: - var_map: Dict[str, Any] = _default_variables(df) + var_map: Dict[str, Any] = _default_variables(df, decennial) elif isinstance(variables, Mapping): var_map = dict(variables) else: @@ -197,18 +331,20 @@ def analysis_frame( if df.duplicated([c for c in UNIT_COLUMNS if c in df.columns]).any(): raise ValueError("source has duplicate GEOID (and Year) rows; one row per unit is required") out = df[[c for c in ID_COLUMNS if c in df.columns]].copy() - if "Year" in out.columns: - out["release"] = "acs5_" + out["Year"].astype(str) - elif release: - out["release"] = release info: Dict[str, Dict[str, Any]] = {} work = df.copy() + n_decennial = 0 for name, ref in var_map.items(): - est_col, moe_col = _resolve(df, ref) + est_col, moe_col = _resolve(df, ref, decennial) e, se = _clean(df[est_col], df[moe_col] if moe_col else None) - _put(out, name, e, se, METHOD_PUBLISHED if moe_col else "no_sampling_error") + method = METHOD_PUBLISHED if moe_col else "no_sampling_error" + if moe_col is None and is_decennial_count(est_col, df, decennial): + se = np.full_like(e, np.nan) # census count: sampling SE undefined, never 0 + method = METHOD_NOT_SAMPLE_BASED + n_decennial += 1 + _put(out, name, e, se, method) info[name] = {"estimate_column": est_col, "moe_column": moe_col, - "method": METHOD_PUBLISHED if moe_col else "no_sampling_error", + "method": method, "concept_id": (concept_ids or {}).get(name)} # expose cleaned values under the analysis name for derive specs work[f"{name}_estimate"] = e @@ -228,8 +364,25 @@ def analysis_frame( "concept_id": (concept_ids or {}).get(name)} for col in keep_columns: out[col] = df[col].to_numpy() + notes: List[str] = [] + if n_decennial and n_decennial < len(var_map): + from .decennial import MIXED_NOTE + + notes.append(MIXED_NOTE) + decennial_only = bool(n_decennial) and n_decennial == len(var_map) + if "Year" in out.columns or release or decennial_only: + from .decennial import DECENNIAL_YEAR + + if decennial_only and ("Year" in out.columns or not release): + label: Any = f"dec_{DECENNIAL_YEAR}" # the 2020 Census, whatever the Year column holds + elif "Year" not in out.columns: + label = release + else: + label = "acs5_" + out["Year"].astype(str) + ("+dec_2020" if n_decennial else "") + out.insert(len([c for c in ID_COLUMNS if c in out.columns]), "release", label) out = out.reset_index(drop=True) out.attrs["variables"] = info + out.attrs["notes"] = notes out.attrs["derive"] = [dict(e) for e in (derive or [])] return out diff --git a/census/api.py b/census/api.py index be21060..2fc5df7 100644 --- a/census/api.py +++ b/census/api.py @@ -10,6 +10,13 @@ from urllib3.util import Retry from census.cache import QueryCache +from census.decennial import ( + DECENNIAL_DATASETS, + DECENNIAL_DHC, + PRODUCT_DECENNIAL, + SAMPLING_REFUSAL, + is_decennial, +) load_dotenv() @@ -28,9 +35,15 @@ "acs1/profile": "acs/acs1/profile", "acs1/subject": "acs/acs1/subject", "acs1/cprofile": "acs/acs1/cprofile", + # 2020 Decennial Census (census.decennial): P.L. 94-171 redistricting data + # and the Demographic and Housing Characteristics File. Not sample-based. + "dec/pl": "dec/pl", + "dec/dhc": "dec/dhc", } -# ACS products. Every dataset key belongs to exactly one product. +# ACS products. Every ACS dataset key belongs to exactly one product. The +# decennial datasets form the separate product PRODUCT_DECENNIAL, which is not +# in PRODUCTS: ACS-only tools (overlap, replicates, PUMS) never accept it. PRODUCT_5YR = "5-year" PRODUCT_1YR = "1-year" PRODUCTS: Tuple[str, ...] = (PRODUCT_5YR, PRODUCT_1YR) @@ -75,6 +88,8 @@ "place": ("place", "160"), "msa": ("metropolitan statistical area/micropolitan statistical area", "310"), "zcta": ("zip code tabulation area", "860"), + # Decennial datasets only (census.decennial): GEOIDs are 15 digits. + "block": ("block", "100"), } # ZCTAs were nested within states (summary level 871) through the 2019 ACS @@ -120,12 +135,21 @@ def normalize_product(product: Optional[str]) -> str: def infer_dataset(table_code: str, product: str = PRODUCT_5YR) -> str: - """Pick the endpoint for a table code prefix within a product (5-year by default).""" + """Pick the endpoint for a table code prefix within a product (5-year by default). + + ``product='decennial'`` selects the 2020 DHC ('dec/dhc'); pass + ``dataset='dec/pl'`` explicitly for P.L. 94-171 tables (table IDs are + product-specific: P.L. 94-171 P1 is Race, DHC P1 is Total Population). + """ + if is_decennial(product): + return product if product in DECENNIAL_DATASETS else DECENNIAL_DHC return PRODUCT_DATASETS[normalize_product(product)][_table_type(table_code)] def dataset_product(dataset: str) -> str: - """'5-year' or '1-year' for a dataset key (ValueError when unknown).""" + """'5-year', '1-year' or 'decennial' for a dataset key (ValueError when unknown).""" + if dataset in DECENNIAL_DATASETS: + return PRODUCT_DECENNIAL try: return _DATASET_TABLE_TYPE[dataset][0] except KeyError: @@ -141,8 +165,12 @@ def dataset_for_product(dataset: str, product: str) -> str: def require_same_product(*products_or_datasets: str) -> str: """Return the shared product; raise ValueError (Census guidance) when 1-year and 5-year mix. - Accepts product names or dataset keys. + Accepts product names or dataset keys. A decennial product or dataset + raises ValueError (census.decennial.SAMPLING_REFUSAL): callers are ACS + sampling tools, which do not apply to census counts. """ + if any(is_decennial(p) for p in products_or_datasets): + raise ValueError(SAMPLING_REFUSAL) found = set() for p in products_or_datasets: if p is None: @@ -173,6 +201,15 @@ def build_geo_params( """ params: Dict[str, str] = {} geo_level = normalize_geo_level(geo_level) + if geo_level == "block": + if not state_code or "*" in state_code or "," in state_code: + raise ValueError("Block queries need exactly one state; request one state at a time.") + params["for"] = "block:*" + inn = f"state:{state_code} county:{county_code or '*'}" + if tract_code: + inn += f" tract:{tract_code}" + params["in"] = inn + return params if geo_level == BLOCK_GROUP: if not state_code or "*" in state_code or "," in state_code: raise ValueError("Block-group queries need exactly one state; request one state at a time.") @@ -353,8 +390,9 @@ def fetch( Geography levels: 'us', 'state', 'county', 'tract', 'block group' (one county per request; 2013+ releases, acs5 only), 'place', 'zcta', 'msa'. Dataset ('acs5', 'profile', 'subject', 'cprofile'; 1-year: 'acs1', - 'acs1/profile', 'acs1/subject', 'acs1/cprofile') is inferred from the - table code (5-year) when not given. + 'acs1/profile', 'acs1/subject', 'acs1/cprofile'; 2020 Census: 'dec/pl', + 'dec/dhc', which also accept 'block') is inferred from the table code + (5-year) when not given. """ endpoint, params = self.build_request( year=year, diff --git a/census/concordance.py b/census/concordance.py index 6b2efae..0fbd82e 100644 --- a/census/concordance.py +++ b/census/concordance.py @@ -20,12 +20,12 @@ import os import re from dataclasses import asdict, dataclass, field -from typing import Any, Dict, Iterable, List, Optional, Sequence, Set, Tuple +from typing import Any, Dict, Iterable, List, Mapping, Optional, Sequence, Set, Tuple from urllib.parse import urlparse from census.geo import get_resource_path from census.metadata import SOURCE_MISSING, archived_vintages, table_universe, table_variables -from census.stats import parse_variable_code +from census.stats import PRIMARY_SUFFIXES, parse_variable_code NORMALIZATION_VERSION = 2 RELABEL_SIMILARITY = 0.80 # same code, labels at least this similar -> needs review @@ -267,7 +267,7 @@ def representative_metadata(self) -> Dict[str, Dict[str, str]]: for cid, c in self.concepts.items(): year = max(c.codes) base = c.codes[year] - for suffix in ("E", "PE", "SS"): + for suffix in PRIMARY_SUFFIXES: meta = self.year_metadata.get(year, {}).get(f"{base}{suffix}") if meta: out[f"{reps[cid]}{suffix}"] = dict(meta) @@ -364,7 +364,7 @@ def to_rows( for year in c.years: base = c.codes[year] meta = self.year_metadata.get(year, {}) - label = (meta.get(f"{base}E") or meta.get(f"{base}PE") or meta.get(f"{base}SS") or {}).get("label", "") + label = _primary_meta(meta, base).get("label", "") rows.append( { "concept_id": cid, @@ -418,6 +418,15 @@ def summary(self) -> Dict[str, Any]: # --------------------------------------------------------------------------- +def _primary_meta(variables: Mapping[str, Any], base: str) -> Dict[str, str]: + """Metadata of a base's primary code (E, PE, SS, or a 2020 Census count N).""" + for suffix in PRIMARY_SUFFIXES: + meta = variables.get(f"{base}{suffix}") + if meta: + return meta + return {} + + def _units_for_year( variables: Dict[str, Dict[str, str]], dataset: str, year: int ) -> Dict[str, Tuple[str, str, Dict[str, str]]]: @@ -431,7 +440,7 @@ def _units_for_year( by_base.setdefault(base, {})[suffix] = meta units: Dict[str, Tuple[str, str, Dict[str, str]]] = {} for base, metas in by_base.items(): - primary = metas.get("E") or metas.get("PE") or metas.get("SS") + primary = next((metas[s] for s in PRIMARY_SUFFIXES if s in metas), None) if primary is None: continue units[base] = (line_role(base, dataset, year), canonical_label(primary["label"]), primary) @@ -697,7 +706,7 @@ def build_evidence( records = [] for year, base in sorted((int(y), b) for y, b in entries): variables, source = table_variables(table_code, dataset, year, client) - meta = variables.get(f"{base}E") or variables.get(f"{base}PE") or variables.get(f"{base}SS") or {} + meta = _primary_meta(variables, base) records.append( { "year": year, @@ -789,7 +798,7 @@ def validate_crosswalk_row( parsed = parse_variable_code(code) base = parsed[0] if parsed else code variables, _ = table_variables(table, dataset, year, client) - if not any(f"{base}{s}" in variables for s in ("E", "PE", "SS")): + if not any(f"{base}{s}" in variables for s in PRIMARY_SUFFIXES): errors.append(f"{code} does not exist in {dataset} {table} {year} metadata") if decision == "merge": diff --git a/census/decennial.py b/census/decennial.py new file mode 100644 index 0000000..29dcd50 --- /dev/null +++ b/census/decennial.py @@ -0,0 +1,146 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +"""2020 Decennial Census datasets and their uncertainty semantics. + +Two products are available through the same client, cache, pipeline and +output formats as the ACS (``dataset='dec/pl'`` or ``'dec/dhc'``, or +``product='decennial'``, which selects the DHC): + +* ``dec/pl``: the 2020 Census Redistricting Data (P.L. 94-171) Summary File + (tables P1-P5, H1; down to census blocks). +* ``dec/dhc``: the 2020 Census Demographic and Housing Characteristics File + (e.g. P1 total population, P12 sex by age, H1 housing units, H3 occupancy + status). Note that table IDs are product-specific: P.L. 94-171 P1 is + "Race", DHC P1 is "Total Population". + +Uncertainty semantics (Round 15, C2). Decennial values are published counts. +They are **not sample-based**, so they have no sampling margin of error, and +they are **not error-free**: they carry coverage error (measured by the +Post-Enumeration Survey), processing error, and noise that the TopDown +differential-privacy algorithm of the Disclosure Avoidance System adds to +every count except the invariants. Outputs therefore never carry an MOE of 0 +or a CV of 0: sampling MOE, SE and CV are undefined, and every extraction +records :data:`DISCLOSURE_AVOIDANCE_ADVISORY`. + +Invariants (published without noise), per the Census Bureau's DAS update of +2020-11-25 (:data:`INVARIANTS_URL`) and the redistricting-data fact sheet +(:data:`DAS_FACT_SHEET_URL`): total population of each state (and DC, Puerto +Rico); total housing units per block; the number of (occupied) group-quarters +facilities by type per block. Everything else, including block, tract and +county population, is noisy. +""" + +from typing import Any, Dict, Optional + +DECENNIAL_PL = "dec/pl" +DECENNIAL_DHC = "dec/dhc" +DECENNIAL_DATASETS = (DECENNIAL_PL, DECENNIAL_DHC) +PRODUCT_DECENNIAL = "decennial" +# The 2020 Census only (the 2010 Census is out of scope, PLAN_ROUND_15 C4). +DECENNIAL_YEARS = (2020,) +DECENNIAL_YEAR = 2020 +# PLAN_ROUND_15 D15-3. Blocks only for P.L. 94-171 and for invariant counts. +DECENNIAL_GEO_LEVELS = ("us", "state", "county", "tract", "block group", "block", "place") +# DHC tables allowed at block level: H1 (total housing units, a block invariant). +DHC_BLOCK_TABLES = ("H1",) + +INVARIANTS_URL = ( + "https://www.census.gov/programs-surveys/decennial-census/decade/2020/planning-management/process/" + "disclosure-avoidance/2020-das-updates/2020-11-25.html" +) +DAS_FACT_SHEET_URL = ( + "https://www.census.gov/content/dam/Census/library/factsheets/2021/" + "protecting-the-confidentiality-of-the-2020-census-redistricting-data.pdf" +) +PPMF_URL = "https://www.census.gov/data/tables/2024/dec/2020-census-ppmf.html" +PES_URL = "https://www.census.gov/newsroom/blogs/random-samplings/2021/12/post-enumeration-measuring-coverage-error.html" +APPORTIONMENT_URL = "https://www.census.gov/data/tables/2020/dec/2020-apportionment-data.html" + +UNCERTAINTY_LABEL = "not sample-based (MOE undefined)" +INVARIANTS = ( + "total population of each state (and DC, Puerto Rico)", + "total housing units per census block", + "number of occupied group-quarters facilities by type per census block", +) + +DISCLOSURE_AVOIDANCE_ADVISORY = ( + "DECENNIAL CENSUS (2020): values are published counts, not sample-based: there is no sampling " + "margin of error, so MOE, standard error, CV and reliability are undefined (never 0). The counts are " + "not error-free: they carry coverage and processing error (Post-Enumeration Survey, " + f"{PES_URL}) and disclosure-avoidance noise added by the TopDown differential-privacy algorithm. " + "Only these counts are invariant (published without noise): " + "; ".join(INVARIANTS) + + f" ({INVARIANTS_URL}). Small geographies and detailed cells are noisier; block-level " + f"characteristics are noisy and should be aggregated before use ({DAS_FACT_SHEET_URL})." +) + +BLOCK_ADVISORY = ( + "Census blocks: only total housing units (and occupied group-quarters facilities) are exact at block " + "level; block population and characteristics carry disclosure-avoidance noise that does not shrink " + "with block size. Aggregate blocks (block groups, tracts or custom areas) before analysis, and do not " + "subtract one table from another at block level." +) + +SAMPLING_REFUSAL = ( + "Decennial census counts (dec/pl, dec/dhc) are not sample-based: they have no sampling margin of " + "error, so a sampling significance test, standard error, CV or reliability rating does not apply, " + "and treating the MOE as 0 would overstate their precision. Their error comes from coverage and " + "processing error and from disclosure-avoidance noise (TopDown differential privacy), which these " + "tools do not measure. Compare decennial counts directly (aggregating small areas first), or test an " + "ACS estimate against a decennial count used as a fixed reference (product1='5-year', " + "product2='decennial')." +) + +MIXED_NOTE = ( + "ACS vs decennial comparison: the test accounts only for the ACS sampling error; the decennial " + "count is treated as a fixed reference with no sampling error. It does not measure the decennial " + "count's disclosure-avoidance noise or coverage error, nor differences in concept and timing (ACS " + "estimates describe a period and use the ACS residence rule; the census counts usual residence on " + "April 1, 2020)." +) + +# Explicit names only: generic words such as "census" or "dec" are not taken to mean the 2020 Census. +_PRODUCT_ALIASES = (PRODUCT_DECENNIAL, "decennial census", "2020 census") + DECENNIAL_DATASETS + + +def is_decennial(value: Optional[Any]) -> bool: + """True for a decennial dataset key ('dec/pl', 'dec/dhc'), the 'decennial' product, or its + spelled-out names ('decennial census', '2020 census').""" + if value is None: + return False + return str(value).strip().lower() in _PRODUCT_ALIASES + + +def refuse_sampling(*products: Optional[Any]) -> None: + """Raise ValueError (:data:`SAMPLING_REFUSAL`) when any product/dataset is decennial.""" + if any(is_decennial(p) for p in products): + raise ValueError(SAMPLING_REFUSAL) + + +def uncertainty_record(geo_level: Optional[str] = None) -> Dict[str, Any]: + """Machine-readable uncertainty statement written into decennial extraction metadata.""" + record: Dict[str, Any] = { + "sample_based": False, + "sampling_moe": None, + "sampling_moe_status": UNCERTAINTY_LABEL, + "error_sources": [ + "coverage error (Post-Enumeration Survey)", + "processing error", + "disclosure-avoidance noise (TopDown differential privacy)", + ], + "invariants": list(INVARIANTS), + "advisory": DISCLOSURE_AVOIDANCE_ADVISORY, + "sources": [INVARIANTS_URL, DAS_FACT_SHEET_URL, PES_URL], + } + if geo_level == "block": + record["block_advisory"] = BLOCK_ADVISORY + return record + + +def codebook_uncertainty(kind: str) -> str: + """Codebook ``Uncertainty`` cell for a decennial column of ``kind`` (classify_column kind).""" + if kind == "N": + return ( + f"{UNCERTAINTY_LABEL}; subject to disclosure-avoidance noise (TopDown differential privacy), " + "coverage and processing error. Invariant only for state total population, block housing " + "units and occupied group-quarters facilities." + ) + return "" diff --git a/census/geo.py b/census/geo.py index c27a360..f7ad184 100644 --- a/census/geo.py +++ b/census/geo.py @@ -190,6 +190,8 @@ def load_places_from_local(path: str) -> List[Tuple[str, str, str, str, str]]: "county": [("state", 2), ("county", 3)], "tract": [("state", 2), ("county", 3), ("tract", 6)], "block group": [("state", 2), ("county", 3), ("tract", 6), ("block group", 1)], + # 2020 Census blocks (decennial datasets only): SS CCC TTTTTT BBBB (15) + "block": [("state", 2), ("county", 3), ("tract", 6), ("block", 4)], "place": [("state", 2), ("place", 5)], "zcta": [(ZCTA_COLUMN, 5)], "msa": [(MSA_COLUMN, 5)], @@ -205,6 +207,7 @@ def load_places_from_local(path: str) -> List[Tuple[str, str, str, str, str]]: "county", "tract", "block group", + "block", "place", ZCTA_COLUMN, MSA_COLUMN, @@ -214,6 +217,8 @@ def load_places_from_local(path: str) -> List[Tuple[str, str, str, str, str]]: def detect_geo_level(columns: Iterable[str]) -> Optional[str]: """Infer the geography level from API response columns (most specific first).""" cols = set(columns) + if "block" in cols: + return "block" if "block group" in cols: return "block group" if "tract" in cols: @@ -236,7 +241,7 @@ def detect_geo_level(columns: Iterable[str]) -> Optional[str]: def build_geoid(components: Mapping[str, object], geo_level: str) -> str: """Build a standard Census GEOID string for one entity. - Block group: SSCCCTTTTTTB (12), Tract: SSCCCTTTTTT (11), County: SSCCC (5), Place: SSPPPPP (7), + Block: SSCCCTTTTTTBBBB (15), Block group: SSCCCTTTTTTB (12), Tract: SSCCCTTTTTT (11), County: SSCCC (5), Place: SSPPPPP (7), ZCTA: ZZZZZ (5), CBSA: MMMMM (5), State: SS (2), Nation: 'US'. """ if geo_level == "us": diff --git a/census/manifest.py b/census/manifest.py index 58fd463..5c9a6d7 100644 --- a/census/manifest.py +++ b/census/manifest.py @@ -20,6 +20,9 @@ "derivations": [{"name": "rate", "op": "proportion", "num": "a", "den": "b"}], "adjustments": [{"type": "rpp", "variables": ["income"], # optional; after exclusions "rpp_year": 2019, "areas": "county_metro" | "state"}], + # several releases: {"type": "rpp", "variables": [...], "dollar_year": 2024, + # "price_index": "r-cpi-u-rs", "rpp_years": {"2019": 2019, "2024": 2024}} + "replicates": {"source": "census_vre"}, # optional; run_manifest fetches VRE files "exclusions": [{"rule": "cv_screen", "threshold": 30, "variables": ["income"], "action": "flag" | "exclude"}, {"rule": "geoid_exclude", "geoids": ["..."]}], @@ -27,7 +30,8 @@ "confidence_level": 0.90, "correction": {"method": "holm", "family": "all" | "by_family"}, "analyses": [{"id": "m1", "method": "fit_replicate", "formula": "income ~ pop", - "args": {"model": "ols"}}], + "args": {"model": "ols"}, + "replicate_scope": "all" | "covariates_only"}], # optional, default "all" "hypotheses": [{"id": "H1", "statement": "...", "analysis": "m1", "statistic": "coefficient", "term": "pop", "direction": "two-sided" | "greater" | "less", "family": "primary"}], @@ -54,7 +58,46 @@ year in the RPP table when it is omitted, before hashing); the year used, every RPP area and the RPP table's SHA-256 (bundled checksum, or the custom table's path and hash) are recorded in ``run.json`` and ``bundle.json`` (``adjustments``). An RPP year -other than the release end year is a note, not a deviation. One release only. +other than the release end year is a note, not a deviation. + +Several releases (PLAN_ROUND_15 A1, decision D15-2): an RPP adjustment then also +requires ``dollar_year``, ``price_index`` (one of census.inflation.PRICE_INDEXES; +``"r-cpi-u-rs"``, the index the Census Bureau uses for ACS dollars, is the default +create_manifest fills in) and ``rpp_years`` (``{"": RPP year}`` for every +release, instead of ``rpp_year``; filled with each release's own year). Each +release's dollars are first converted to ``dollar_year`` dollars with the price +index (CPI(dollar_year) / CPI(release)), then divided by the RPP of the release's +``rpp_years`` entry; estimates, SEs and replicates of that release alike. With one +release a ``dollar_year`` (plus ``price_index``) is optional. Not combinable with +``aggregation``. The run records the CPI values and factors, the price-index +table's SHA-256 and every release's RPP areas. + +``replicates`` (optional, PLAN_ROUND_15 A2): ``{"source": "census_vre"}`` makes +run_manifest fetch the Census VRE files for the manifest's tables, geography and +releases (:func:`census.replicate.fetch_vre` and its cache), as the replicate +frame of every replicate analysis; ``run.json`` and ``bundle.json`` record each +file's URL and SHA-256, and ``replicates.unavailable`` lists variables without +published replicates (e.g. B19013 medians). Under the strict confirmatory policy a +replicate analysis that uses such a variable blocks (``replicates_incomplete``) +unless the analysis declares ``"replicate_scope": "covariates_only"`` +(fit_replicate only; target ``population`` or ``conservative``): its outcome is +then held at its estimate in every refit, and the declaration is in the hash and +the methods text. The declaration is allowed whenever it is well-formed: an +outcome the VRE registry lists can still lack replicates for the analysed units, +so create_manifest only notes such an outcome, and a run that finds replicates +for it still holds it fixed as declared (the hashed declaration governs) and +records that (``replicates.covariates_only_outcome_had_replicates``). Census VRE +files of tract and block-group geographies are per state, so those levels need +``geography.state`` or ``states``. Caller-supplied ``replicates`` are an error +with census_vre. + +Releases whose ACS 5-year periods overlap (shared sample years) are never +combined through replicates (the trend module's overlap rule: published +replicates do not carry the covariance of overlapping periods): validation +refuses census_vre or an analysis declaring ``"replicates": "required"`` with +overlapping releases, and caller-supplied replicates spanning overlapping +releases stop every replicate analysis with a blocking +``replicates_overlapping_releases`` deviation. When replicates are supplied and the manifest has derivations, each derivation absent from the replicate frame is evaluated on the replicates of @@ -124,7 +167,8 @@ BLOCKING_DEVIATIONS = ("blocked_variable", "extraction_failed", "variable_missing_from_extraction", "aggregation_failed", "analysis_frame_failed", "analysis_not_run", "replicates_unavailable", "analysis_failed", "replicates_incomplete", - "hypothesis_not_tested", "failed_calls", "data_source_mismatch", "adjustment_failed") + "hypothesis_not_tested", "failed_calls", "data_source_mismatch", "adjustment_failed", + "replicates_overlapping_releases") DATA_SOURCE_KEYS = ("synthetic", "description", "generator") STATUS_EXPLORATORY = "exploratory" @@ -142,7 +186,12 @@ STATISTICS = ("coefficient", "correlation", "trend_slope", "z_difference") EXCLUSION_RULES = ("cv_screen", "geoid_exclude") ADJUSTMENT_TYPES = ("rpp",) -RPP_ADJUSTMENT_KEYS = ("type", "variables", "rpp_year", "areas") +RPP_ADJUSTMENT_KEYS = ("type", "variables", "rpp_year", "areas", "rpp_years", "dollar_year", "price_index") +REPLICATE_SOURCES = ("census_vre",) +REPLICATES_KEYS = ("source",) +REPLICATE_SCOPES = ("all", "covariates_only") +# frame identifier columns a formula may name (C(Year) for a change across releases, C(GEOID...) terms) +FORMULA_ID_COLUMNS = ("GEOID", "Year") # geography levels whose GEOIDs start with the state FIPS (state RPP areas; usable with # geography.states). ZCTA GEOIDs are not state-prefixed and ZCTAs can cross state lines. _STATE_PREFIXED_LEVELS = ("state", "county", "tract", "block group", "place") @@ -309,8 +358,7 @@ def _check_adjustments(manifest: Mapping[str, Any], names: List[str], errors: Li level = normalize_geo_level(geo.get("level", "")) except Exception: # noqa: BLE001 - geography errors are reported by _check_geography level = geo.get("level") - if adjustments and len(manifest.get("releases") or []) != 1: - errors.append("adjustments: an RPP adjustment needs exactly one release (dollars of one end year)") + releases = manifest_releases(manifest) if adjustments and manifest.get("aggregation") is not None: errors.append("adjustments cannot be combined with aggregation (RPP areas are assigned per GEOID)") adjusted: List[str] = [] @@ -341,7 +389,6 @@ def _check_adjustments(manifest: Mapping[str, Any], names: List[str], errors: Li elif areas == "state" and level not in _STATE_PREFIXED_LEVELS: errors.append(f"adjustments[{i}]: areas 'state' needs a state-based geography level " f"{_STATE_PREFIXED_LEVELS}") - year = adj.get("rpp_year") try: from . import rpp years = rpp._years() @@ -350,15 +397,188 @@ def _check_adjustments(manifest: Mapping[str, Any], names: List[str], errors: Li continue if not years: errors.append(f"adjustments[{i}]: no RPP data available") - elif year is None: + continue + _check_dollar_year(adj, i, releases, errors) + if len(releases) > 1 or "rpp_years" in adj: + _check_rpp_years(adj, i, releases, years, errors) + continue + year = adj.get("rpp_year") + if year is None: errors.append(f"adjustments[{i}]: rpp_year is required (it fixes the price level in the " f"pre-registration); the latest year in the RPP table is {years[-1]}. " "create_manifest fills it in when omitted") - elif not isinstance(year, int) or isinstance(year, bool) or year not in years: + elif not _is_int(year) or year not in years: errors.append(f"adjustments[{i}]: rpp_year must be an integer year in the RPP table " f"({years[0]}-{years[-1]})") +def _is_int(v: Any) -> bool: + return isinstance(v, int) and not isinstance(v, bool) + + +def manifest_releases(manifest: Mapping[str, Any]) -> List[int]: + """The manifest's integer releases, in order (anything else is reported by validate_manifest).""" + releases = manifest.get("releases") + return [r for r in releases if _is_int(r)] if isinstance(releases, list) else [] + + +def release_overlap_message(releases: List[int], subject: str) -> Optional[str]: + """None when the ACS 5-year periods of ``releases`` share no sample years, else why replicate + SEs across them are refused (the trend module's overlap rule, census.trends.overlap_design_check).""" + if len(set(releases)) < 2: + return None + from .trends import overlap_design_check + design = overlap_design_check(releases) + if not design["overlapping"]: + return None + return (f"{subject} with overlapping 5-year periods: releases " + f"{[(p['year1'], p['year2']) for p in design['overlapping_pairs']]} share sample years, and their " + "replicates do not carry that covariance, so replicate SEs of analyses that combine or contrast them " + f"would be wrong; use non-overlapping releases {design['suggested_nonoverlapping']}") + + +def _check_rpp_years(adj: Mapping[str, Any], i: int, releases: List[int], years: List[int], + errors: List[str]) -> None: + """``rpp_years`` ({release: RPP year}, one entry per release) replaces ``rpp_year`` when a + manifest has several releases (each release's dollars get their own year's RPP, D15-2).""" + if "rpp_year" in adj: + errors.append(f"adjustments[{i}]: give rpp_years (one RPP year per release), not rpp_year, " + "when the manifest has several releases" if len(releases) > 1 + else f"adjustments[{i}]: give either rpp_year or rpp_years, not both") + return + ry = adj.get("rpp_years") + if ry is None: + errors.append(f"adjustments[{i}]: rpp_years is required with several releases, e.g. " + + json.dumps({str(r): r for r in releases}) + + " (each release's own year, D15-2; it fixes the price levels in the pre-registration). " + "create_manifest fills it in when omitted") + return + if not isinstance(ry, Mapping) or sorted(str(k) for k in ry) != sorted(str(r) for r in releases): + errors.append(f"adjustments[{i}]: rpp_years must map every release (as a string key) to an RPP year, " + "e.g. " + json.dumps({str(r): r for r in releases})) + return + for k, v in ry.items(): + if not _is_int(v) or v not in years: + errors.append(f"adjustments[{i}]: rpp_years[{k!r}] must be an integer year in the RPP table " + f"({years[0]}-{years[-1]})") + + +def _check_dollar_year(adj: Mapping[str, Any], i: int, releases: List[int], errors: List[str]) -> None: + """``dollar_year`` + ``price_index``: every release's dollars are converted to the dollar year + with the price index (CPI first, then the RPP; D15-2). Required with several releases.""" + from .inflation import PRICE_INDEXES, load_price_index + dy, index = adj.get("dollar_year"), adj.get("price_index") + if dy is None: + if len(releases) > 1: + errors.append(f"adjustments[{i}]: dollar_year is required with several releases (dollars of " + f"different years are converted to one year with the price index before the RPP); " + "create_manifest fills in the latest release") + if index is not None: + errors.append(f"adjustments[{i}]: price_index needs dollar_year") + return + if not _is_int(dy): + errors.append(f"adjustments[{i}]: dollar_year must be an integer year") + return + if index is None: + errors.append(f"adjustments[{i}]: price_index is required with dollar_year (one of {PRICE_INDEXES}; " + "create_manifest fills in the default r-cpi-u-rs)") + return + if index not in PRICE_INDEXES: + errors.append(f"adjustments[{i}]: price_index must be one of {PRICE_INDEXES}") + return + try: + table = load_price_index(index) + except Exception as exc: # noqa: BLE001 - e.g. a bundled-data integrity failure + errors.append(f"adjustments[{i}]: price index table unreadable ({type(exc).__name__}: {exc})") + return + missing = sorted({int(dy), *releases} - set(table)) + if missing: + errors.append(f"adjustments[{i}]: no complete {index} value for {missing} (available " + f"{min(table) if table else '-'}-{max(table) if table else '-'})") + + +def _vre_level(level: Any) -> str: + """A manifest geography level as census.replicate names it (``block group`` -> ``block_group``).""" + from .pipeline import normalize_geo_level + return normalize_geo_level(str(level)).replace(" ", "_") + + +def census_vre_source(manifest: Mapping[str, Any]) -> bool: + """True when the manifest declares ``"replicates": {"source": "census_vre"}``.""" + reps = manifest.get("replicates") + return isinstance(reps, Mapping) and reps.get("source") == "census_vre" + + +def _check_replicates(manifest: Mapping[str, Any], errors: List[str]) -> None: + """``replicates`` (optional): ``{"source": "census_vre"}`` makes run_manifest fetch the Census + VRE files for the manifest's tables, geography and releases (census.replicate and its cache).""" + reps = manifest.get("replicates") + if reps is None: + return + if not isinstance(reps, Mapping): + errors.append("replicates must be an object, e.g. {\"source\": \"census_vre\"}") + return + for k in sorted(set(reps) - set(REPLICATES_KEYS)): + errors.append(f"replicates: unknown key {k!r}; allowed {REPLICATES_KEYS}") + if reps.get("source") not in REPLICATE_SOURCES: + errors.append(f"replicates.source must be one of {REPLICATE_SOURCES}") + return + from . import replicate as vre + geo = manifest.get("geography") if isinstance(manifest.get("geography"), Mapping) else {} + try: + sumlev = vre.sumlev_for(_vre_level(geo.get("level", ""))) + except Exception as exc: # noqa: BLE001 - unsupported level for VRE files + errors.append(f"replicates.source census_vre: {exc}") + sumlev = None + if sumlev in vre.PER_STATE_SUMLEVS and not geo.get("state") and not geo.get("states"): + errors.append(f"replicates.source census_vre at level {geo.get('level')!r}: Census VRE files of summary " + f"level {sumlev} are published per state; give geography.state or geography.states") + releases = manifest_releases(manifest) + bad = [r for r in releases if r not in vre.VRE_LAYOUTS] + if bad: + errors.append(f"replicates.source census_vre: no Census VRE release for {bad} (5-year releases " + f"{vre.FIRST_VRE_YEAR}-{vre.LAST_KNOWN_VRE_YEAR})") + if manifest.get("aggregation") is not None: + errors.append("replicates.source census_vre cannot be combined with aggregation (VRE replicates are " + "per GEOID and are not aggregated by run_manifest)") + overlap = release_overlap_message(releases, "replicates.source census_vre") + if overlap: + errors.append(overlap) + + +def _variable_tables(manifest: Mapping[str, Any]) -> Dict[str, List[str]]: + """Manifest variable or derivation name -> the tables it is built from.""" + out: Dict[str, List[str]] = {} + variables = manifest.get("variables") + for v in variables if isinstance(variables, list) else []: + if isinstance(v, Mapping) and v.get("name") is not None and _table_of(v): # invalid entries: reported + out[str(v["name"])] = [str(_table_of(v))] + derivations = manifest.get("derivations") + for d in derivations if isinstance(derivations, list) else []: + if not isinstance(d, Mapping): # reported by validate_manifest + continue + refs = [r for part in ("num", "den") + for r in (d.get(part) if isinstance(d.get(part), list) else [d.get(part)]) if r is not None] + out[str(d.get("name"))] = sorted({t for r in refs for t in out.get(str(r), [])}) + return out + + +def vre_unavailable(manifest: Mapping[str, Any]) -> List[Dict[str, Any]]: + """Variables and derivations without published Census VRE replicates for some release + (offline, from the census.replicate registry): [{"variable", "table", "release", "reason"}].""" + from . import replicate as vre + level = _vre_level((manifest.get("geography") or {}).get("level", "")) + out = [] + releases = manifest_releases(manifest) + for name, tables in _variable_tables(manifest).items(): + for table in tables: + for year in releases: + info = vre.vre_availability(table, year, level) + if not info["available"]: + out.append({"variable": name, "table": table, "release": year, "reason": info["reason"]}) + return out + + def tables_and_refs(manifest: Mapping[str, Any]) -> Dict[str, List[str]]: """table code -> variable refs (in manifest order).""" out: Dict[str, List[str]] = {} @@ -398,11 +618,13 @@ def _duplicate_spec_keys(a: Mapping[str, Any]) -> List[str]: def _formula_names(formula: str) -> List[str]: - """Identifiers in a patsy formula that are not function calls.""" + """Identifiers in a patsy formula that are not function calls or frame identifiers + (GEOID, Year: e.g. ``C(Year)`` for a change between releases).""" names = [] for m in re.finditer(r"[A-Za-z_][A-Za-z0-9_.]*", str(formula)): tail = str(formula)[m.end():].lstrip() - if tail.startswith("(") or m.group(0) in FORMULA_FUNCTIONS or "." in m.group(0): + if tail.startswith("(") or m.group(0) in FORMULA_FUNCTIONS or "." in m.group(0) \ + or m.group(0) in FORMULA_ID_COLUMNS: continue names.append(m.group(0)) return names @@ -415,6 +637,9 @@ def _check_variables(manifest: Mapping[str, Any], errors: List[str], resolve: bo if not variables: errors.append("variables: at least one variable is required") return + if not isinstance(variables, list): + errors.append("variables must be a list of {name, ref} objects") + return by_table: Dict[str, List[str]] = {} for i, v in enumerate(variables): if not isinstance(v, Mapping) or "name" not in v or "ref" not in v: @@ -484,7 +709,7 @@ def validate_manifest(manifest: Mapping[str, Any], resolve: bool = True, client: except ValueError: errors.append("created must be an ISO 8601 timestamp") releases = manifest.get("releases") or [] - if not releases or not all(isinstance(r, int) for r in releases): + if not isinstance(releases, list) or not releases or not all(_is_int(r) for r in releases): errors.append("releases must be a non-empty list of ACS 5-year end years (integers)") _check_geography(manifest.get("geography") or {}, errors) level = manifest.get("confidence_level", 0.90) @@ -514,8 +739,13 @@ def validate_manifest(manifest: Mapping[str, Any], resolve: bool = True, client: errors.append("missing: only 'complete_case' is supported") _check_variables(manifest, errors, resolve, client) - names = [str(v.get("name")) for v in manifest.get("variables", []) if isinstance(v, Mapping)] - for d in manifest.get("derivations") or []: + variables = manifest.get("variables") + names = [str(v.get("name")) for v in (variables if isinstance(variables, list) else []) if isinstance(v, Mapping)] + derivations = manifest.get("derivations") or [] + for i, d in enumerate(derivations if isinstance(derivations, list) else [derivations]): + if not isinstance(d, Mapping): + errors.append(f"derivations[{i}] must be an object {{name, op, num, den}}") + continue names.append(str(d.get("name"))) for part in ("num", "den"): refs = d.get(part) @@ -531,6 +761,7 @@ def validate_manifest(manifest: Mapping[str, Any], resolve: bool = True, client: if agg is not None and not isinstance(agg.get("mapping"), Mapping): errors.append("aggregation.mapping must be an object GEOID -> group id") _check_adjustments(manifest, names, errors) + _check_replicates(manifest, errors) for i, ex in enumerate(manifest.get("exclusions") or []): if ex.get("rule") not in EXCLUSION_RULES: @@ -545,6 +776,7 @@ def validate_manifest(manifest: Mapping[str, Any], resolve: bool = True, client: if not analyses: errors.append("analyses: at least one analysis is required") ids = {} + needs_replicates: List[str] = [] # replicate analyses declaring "replicates": "required" for i, a in enumerate(analyses): for k in _duplicate_spec_keys(a): errors.append(f"analyses[{i}]: {k!r} given both at the top level and in args; give it once") @@ -556,6 +788,8 @@ def validate_manifest(manifest: Mapping[str, Any], resolve: bool = True, client: if a.get("id") in ids or not a.get("id"): errors.append(f"analyses[{i}]: id must be present and unique") ids[a.get("id")] = method + if method in REPLICATE_METHODS and a.get("replicates") == "required": + needs_replicates.append(str(a.get("id"))) for req in ANALYSIS_METHODS[method][0]: if req not in a: errors.append(f"analyses[{i}] ({method}): {req!r} is required") @@ -568,6 +802,29 @@ def validate_manifest(manifest: Mapping[str, Any], resolve: bool = True, client: for u in used: if u not in names: errors.append(f"analyses[{i}] ({a.get('id')}): {u!r} is not a manifest variable") + scope = a.get("replicate_scope", "all") + if scope not in REPLICATE_SCOPES: + errors.append(f"analyses[{i}] ({a.get('id')}): replicate_scope must be one of {REPLICATE_SCOPES}") + elif scope == "covariates_only": + outcome = _formula_names(a["formula"].split("~")[0]) if "~" in str(a.get("formula", "")) else [] + if method != "fit_replicate": + errors.append(f"analyses[{i}] ({a.get('id')}): replicate_scope 'covariates_only' applies to " + "fit_replicate only") + elif (a.get("args") or {}).get("target", "population") == "these_areas": + errors.append(f"analyses[{i}] ({a.get('id')}): replicate_scope 'covariates_only' cannot use " + "target 'these_areas' (its interval would rest on a sampling SE without the " + "outcome's sampling error); use 'population' or 'conservative'") + elif len(outcome) != 1: + errors.append(f"analyses[{i}] ({a.get('id')}): replicate_scope 'covariates_only' needs a formula " + "whose outcome is one manifest variable") + # An outcome the VRE registry lists may still lack replicates for the analysed units (a line + # missing from the file), so the declaration is allowed; create_manifest notes it and the run + # records whether the outcome had replicates (it is held fixed either way: the hash governs). + if needs_replicates and not census_vre_source(manifest): # census_vre: _check_replicates + overlap = release_overlap_message(manifest_releases(manifest), + f"analyses {needs_replicates} require replicates") + if overlap: + errors.append(overlap) hyps = manifest.get("hypotheses") or [] if not hyps: errors.append("hypotheses: at least one pre-registered hypothesis is required") @@ -593,6 +850,66 @@ def validate_manifest(manifest: Mapping[str, Any], resolve: bool = True, client: return errors +def _declared_covariates_only(manifest: Mapping[str, Any]) -> Dict[str, List[str]]: + """{analysis id: outcome names} of analyses declaring ``replicate_scope: "covariates_only"``.""" + out: Dict[str, List[str]] = {} + for a in manifest.get("analyses") or []: + if not isinstance(a, Mapping): + continue + a = _normalize_analysis(a) + if a.get("replicate_scope") == "covariates_only" and "~" in str(a.get("formula", "")): + out[str(a.get("id"))] = _formula_names(str(a["formula"]).split("~")[0]) + return out + + +_FILL_REASONS = { + "rpp_year": "the latest year in the RPP table", + "dollar_year": "the latest release", + "price_index": "the default index (R-CPI-U-RS, the index the Census Bureau uses for ACS dollars)", + "rpp_years": "each release's own year (D15-2)", +} + + +def _fill_adjustment_defaults(manifest: Dict[str, Any]) -> List[Dict[str, Any]]: + """Fill omitted required adjustment fields (in place) so the hash fixes them; returns + one ``{"adjustment": i, : value}`` entry per filled field. + + One release: ``rpp_year`` (the latest RPP year; Round 14). Several releases (or a declared + ``dollar_year``): ``dollar_year`` (the latest release), ``price_index`` (R-CPI-U-RS) and + ``rpp_years`` (each release's own year, when the RPP table has it; otherwise left out and + validation asks for it).""" + filled: List[Dict[str, Any]] = [] + adjustments = manifest.get("adjustments") + if not isinstance(adjustments, list): + return filled + releases = manifest_releases(manifest) + multi = len(releases) > 1 + try: + from . import rpp + rpp_years, latest = rpp._years(), latest_rpp_year() + except Exception: # noqa: BLE001 - an unreadable RPP table is reported by validation + rpp_years = latest = None + for i, adj in enumerate(adjustments): + if not (isinstance(adj, dict) and adj.get("type") == "rpp"): + continue + if multi and adj.get("dollar_year") is None and releases: + adj["dollar_year"] = max(releases) + filled.append({"adjustment": i, "dollar_year": max(releases)}) + if adj.get("dollar_year") is not None and adj.get("price_index") is None: + from .inflation import DEFAULT_PRICE_INDEX + adj["price_index"] = DEFAULT_PRICE_INDEX + filled.append({"adjustment": i, "price_index": DEFAULT_PRICE_INDEX}) + if multi or "rpp_years" in adj: + if adj.get("rpp_years") is None and "rpp_year" not in adj and rpp_years \ + and all(r in rpp_years for r in releases): + adj["rpp_years"] = {str(r): int(r) for r in releases} + filled.append({"adjustment": i, "rpp_years": dict(adj["rpp_years"])}) + elif adj.get("rpp_year") is None and latest is not None: + adj["rpp_year"] = int(latest) + filled.append({"adjustment": i, "rpp_year": int(latest)}) + return filled + + def create_manifest( path_or_dict: Union[str, Path, Mapping[str, Any]], path: Optional[Union[str, Path]] = None, client: Any = None, resolve: bool = True, @@ -604,8 +921,9 @@ def create_manifest( to the repository to fix the pre-registration (PLAN_ROUND_8 decision 3). An RPP adjustment without ``rpp_year`` gets the latest year in the active RPP - table before validation and hashing, so the hash fixes the applied year; the - result says so (``filled``, ``notes``) and the completed manifest is written to + table before validation and hashing, so the hash fixes the applied year (with + several releases: ``dollar_year``, ``price_index`` and ``rpp_years`` get their + defaults, :func:`_fill_adjustment_defaults`); the result says so (``filled``, ``notes``) and the completed manifest is written to ``path``, or, when no ``path`` is given, to a new sibling file ``.completed.json`` (the input file is never overwritten; a dict input without ``path`` gets it in ``manifest``). Commit and run the completed file. @@ -613,19 +931,7 @@ def create_manifest( ``"manifest"`` when something was filled in). """ manifest = load_manifest(path_or_dict) - filled: List[Dict[str, Any]] = [] - adjustments = manifest.get("adjustments") - if isinstance(adjustments, list): - year = None - for i, adj in enumerate(adjustments): - if isinstance(adj, dict) and adj.get("type") == "rpp" and adj.get("rpp_year") is None: - try: - year = year or latest_rpp_year() - except Exception: # noqa: BLE001 - an unreadable RPP table is reported by validation - year = None - if year is not None: - adj["rpp_year"] = int(year) - filled.append({"adjustment": i, "rpp_year": int(year)}) + filled = _fill_adjustment_defaults(manifest) errors = validate_manifest(manifest, resolve=resolve, client=client) out_path = str(path) if path else (str(path_or_dict) if not isinstance(path_or_dict, Mapping) else None) write_to = path @@ -637,10 +943,27 @@ def create_manifest( Path(write_to).parent.mkdir(parents=True, exist_ok=True) Path(write_to).write_text(json.dumps(manifest, indent=2, sort_keys=True, ensure_ascii=False) + "\n", encoding="utf-8", newline="\n") - notes = [f"adjustments[{f['adjustment']}]: rpp_year was omitted; filled in with {f['rpp_year']}, the latest " - "year in the RPP table, before hashing" + (f" (written to {write_to})" if write_to and not errors - else "; commit the completed manifest") - for f in filled] + where = f" (written to {write_to})" if write_to and not errors else "; commit the completed manifest" + notes = [f"adjustments[{f['adjustment']}]: {k} was omitted; filled in with {json.dumps(v)}, " + + _FILL_REASONS[k] + ", before hashing" + where + for f in filled for k, v in f.items() if k != "adjustment"] + if census_vre_source(manifest): + missing = vre_unavailable(manifest) + if missing: + notes.append("replicates (census_vre): no published Census VRE replicates for " + + "; ".join(f"{u['variable']} ({u['table']}, {u['release']})" for u in missing) + + ". Under the strict confirmatory policy a replicate analysis that uses them blocks " + "(replicates_incomplete) unless it declares \"replicate_scope\": \"covariates_only\" " + "(outcome without replicates, held at its estimate in the refits)") + no_vre = {u["variable"] for u in missing} + for aid, held in _declared_covariates_only(manifest).items(): + listed = [v for v in held if v not in no_vre] + if listed: + notes.append(f"analyses {aid}: replicate_scope 'covariates_only' declared, but the Census VRE " + f"registry lists replicates for the outcome {listed} in every release. The outcome " + "will be held at its estimate in every refit as declared (the hashed declaration " + "governs; the run records whether its replicates were available); remove the " + "declaration before hashing to use the outcome's replicates") out = {"hash": manifest_hash(manifest), "path": out_path, "errors": errors, "valid": not errors, "notes": notes, "filled": filled} if filled: @@ -735,6 +1058,8 @@ def _run_analysis(a: Mapping[str, Any], frame: pd.DataFrame, level: float, repli # the failure is recorded as a "replicates_incomplete" deviation. args["replicate_policy"] = CONFIRMATORY_REPLICATE_POLICY if method == "fit_replicate": + if a.get("replicate_scope") == "covariates_only": # pre-registered: the outcome is held fixed + args["fixed_variables"] = _formula_names(a["formula"].split("~")[0]) return models.fit_replicate(a["formula"], frame, replicates=replicates, confidence_level=level, **args) if method in ("regression_calibration", "simex"): return getattr(models, method)(a["formula"], frame, confidence_level=level, **args) @@ -823,6 +1148,11 @@ def run_manifest( (:func:`replicates_sha256`); ``replicates_synthetic`` (default: the data source's synthetic flag, or ``replicates.attrs["synthetic"]``) marks them synthetic in the methods text. + A manifest with ``"replicates": {"source": "census_vre"}`` fetches the + Census VRE files itself (:func:`census.replicate.fetch_vre` and its cache) + for its tables, geography and releases; the run records each file's URL and + SHA-256 and the variables without published replicates + (``replicates.unavailable``). Passing ``replicates`` as well is an error. ``analysis_status`` is ``"confirmatory"`` only without blocking deviations, else ``"confirmatory_incomplete"`` (see the module docstring). Never raises on data problems; returns ``success``, ``deviations``, @@ -836,6 +1166,9 @@ def run_manifest( m = load_manifest(manifest) errors = validate_manifest(m, resolve=True, client=client) + if census_vre_source(m) and replicates is not None: + errors.append("replicates: the manifest declares replicates.source 'census_vre' (fetched by run_manifest); " + "do not also supply replicates (replicates_path)") h = manifest_hash(m) out = Path(output_dir) if errors: @@ -861,6 +1194,7 @@ def run_manifest( panels: List[pd.DataFrame] = [] extraction: Dict[str, Any] = {} name_of = {str(v["ref"]): str(v["name"]) for v in m["variables"]} + codes_by_year: Dict[str, Dict[int, str]] = {} # manifest name -> {release: ACS base code} (VRE rows) for table, refs in tables_and_refs(m).items(): plan = plan_extraction(table, entities, releases, "panel", opts, variables=refs, client=client) plan_pub = {k: v for k, v in plan.items() if not k.startswith("_")} @@ -887,6 +1221,11 @@ def run_manifest( continue codebook = pd.read_csv(res["codebook_path"], dtype=str) cid_of = {s["input"]: s["concept_id"] for s in plan.get("selection", [])} + conc = plan.get("_concordance") + for ref in refs: + concept = getattr(conc, "concepts", {}).get(cid_of.get(ref, ref)) + if concept is not None: + codes_by_year[name_of[ref]] = {int(y): str(c) for y, c in concept.codes.items()} tcols = {} for ref in refs: cols = _codebook_columns(codebook, cid_of.get(ref, ref)) @@ -928,6 +1267,17 @@ def run_manifest( except Exception as exc: deviations.append({"kind": "analysis_frame_failed", "detail": str(exc)}) + # 3a. Census VRE replicates fetched for the manifest (replicates.source census_vre) + vre_info: Optional[Dict[str, Any]] = None + if census_vre_source(m) and frame is not None: + try: + replicates, vre_info = _census_vre_replicates(m, frame, codes_by_year, client) + except Exception as exc: # noqa: BLE001 - recorded as a blocking deviation, never substituted + deviations.append({"kind": "replicates_unavailable", + "detail": f"Census VRE replicates could not be fetched: {type(exc).__name__}: {exc}"}) + vre_info = {"source": "census_vre", "error": f"{type(exc).__name__}: {exc}"} + replicates = None + # 3b. derivations on supplied replicates work_reps = replicates derived_on_replicates: Dict[str, Dict[str, int]] = {} @@ -968,7 +1318,7 @@ def run_manifest( # 4b. pre-registered adjustments (a unit still without an RPP area blocks: adjustment_failed) adjustment_log: List[Dict[str, Any]] = [] if frame is not None and m.get("adjustments"): - frame, work_reps, adjustment_log, failure = _apply_adjustments(frame, work_reps, m, releases[0]) + frame, work_reps, adjustment_log, failure = _apply_adjustments(frame, work_reps, m, releases) if failure: deviations.append({"kind": "adjustment_failed", "detail": failure}) frame = None @@ -979,6 +1329,10 @@ def run_manifest( # 5. analyses available = set(columns) | {str(d["name"]) for d in m.get("derivations") or []} results: Dict[str, Any] = {} + # caller-supplied replicates of overlapping releases: refused (census_vre is refused at validation) + rep_overlap = _replicate_overlap(replicates) if replicates is not None and not census_vre_source(m) else None + declared_fixed = _declared_covariates_only(m) + fixed_had_replicates: Dict[str, List[str]] = {} for a in m["analyses"]: a = _normalize_analysis(a) aid = a["id"] @@ -995,6 +1349,19 @@ def run_manifest( deviations.append({"kind": "replicates_unavailable", "analysis": aid, "detail": "manifest requires replicate SEs; none supplied, analysis not run"}) continue + if census_vre_source(m) and replicates is None and a["method"] in REPLICATE_METHODS: + deviations.append({"kind": "replicates_unavailable", "analysis": aid, + "detail": "manifest declares Census VRE replicates; none were fetched or published " + "for its variables (see replicates.unavailable), analysis not run"}) + continue + if rep_overlap and a["method"] in REPLICATE_METHODS: + deviations.append({"kind": "replicates_overlapping_releases", "analysis": aid, + "detail": rep_overlap + "; analysis not run"}) + continue + if aid in declared_fixed and work_reps is not None: + had = [v for v in declared_fixed[aid] if _has_replicate_rows(work_reps, v)] + if had: + fixed_had_replicates[aid] = had try: results[aid] = _run_analysis(a, frame, level, work_reps, corr_method) except Exception as exc: @@ -1035,7 +1402,20 @@ def run_manifest( complete = not any(d["kind"] in BLOCKING_DEVIATIONS for d in deviations) status = STATUS_CONFIRMATORY if complete else STATUS_CONFIRMATORY_INCOMPLETE rep_info = None - if replicates is not None: + if vre_info is not None: + rep_info = dict(vre_info) + if replicates is not None: + digest, reason = replicates_digest(replicates) + rep_info.update(rows=int(len(replicates)), sha256=digest) + if digest is None: + rep_info["sha256_unavailable"] = reason + if derived_on_replicates: + rep_info["derived_on_replicates"] = list(derived_on_replicates) + rep_info["derived_on_replicates_rows"] = derived_on_replicates + scaled = [v for a in adjustment_log for v in a["variables"]] + if scaled and replicates is not None: + rep_info["scaled_by_adjustments"] = scaled + elif replicates is not None: if replicates_synthetic is None: attrs = getattr(replicates, "attrs", {}) or {} replicates_synthetic = bool(attrs.get("synthetic", source["synthetic"])) @@ -1050,6 +1430,16 @@ def run_manifest( scaled = [v for a in adjustment_log for v in a["variables"]] if scaled: rep_info["scaled_by_adjustments"] = scaled + if rep_info is not None and declared_fixed: + rep_info["declared_covariates_only"] = declared_fixed + if fixed_had_replicates: + # the hashed declaration governs: the outcome stays fixed although replicates were available + rep_info["covariates_only_outcome_had_replicates"] = fixed_had_replicates + rep_info.setdefault("notes", []).extend( + f"analysis {aid}: replicate_scope 'covariates_only' was declared and the outcome {names} was held " + "at its estimate in every refit as pre-registered, although replicates were available for it in " + "this run; se_sampling excludes the outcome's sampling error" + for aid, names in fixed_had_replicates.items()) tagged = {aid: _tag(r, h, aid, status) for aid, r in results.items()} run_info = {"manifest_hash": h, "analysis_status": status, "deviations": deviations, "exclusions": exclusion_log, "extraction": extraction, "n_units": None if frame is None @@ -1070,6 +1460,25 @@ def run_manifest( return out_res +def _replicate_overlap(replicates: Any) -> Optional[str]: + """Why caller-supplied replicates cannot be used (their releases' 5-year periods overlap), or None.""" + if isinstance(replicates, Mapping): # {release: frame} + years = [int(y) for y in replicates if str(y).isdigit()] + elif isinstance(replicates, pd.DataFrame) and "Year" in replicates.columns: + years = sorted({int(y) for y in pd.to_numeric(replicates["Year"], errors="coerce").dropna()}) + else: + return None + return release_overlap_message(years, "caller-supplied replicates") + + +def _has_replicate_rows(replicates: Any, name: str) -> bool: + """True when the replicate long frame has rows for manifest variable ``name``.""" + try: + return bool((replicates["variable"].astype(str) == str(name)).any()) + except Exception: # noqa: BLE001 - malformed replicates are reported by the analysis itself + return False + + def _replicate_rows(frame: pd.DataFrame, name: str) -> Dict[str, int]: """{"rows_replicate": rows whose ``name`` SE is a replicate SE, "rows": all rows}.""" col = frame.get(f"{name}_method") @@ -1131,39 +1540,210 @@ def refs_of(d: Mapping[str, Any]) -> List[str]: return out, combined, {name: counts[name] for name in derived} +def _census_vre_replicates(m: Mapping[str, Any], frame: pd.DataFrame, codes_by_year: Mapping[str, Mapping[int, str]], + client: Any = None) -> Tuple[Optional[pd.DataFrame], Dict[str, Any]]: + """Fetch the Census VRE files for the manifest's variables, geography and releases with + :func:`census.replicate.fetch_vre` (and its cache) and return (replicate long frame, record). + + The long frame has GEOID, Year, variable (the manifest variable name), estimate and the 80 + replicates, for the analysis frame's units only. The record lists every file (URL, SHA-256 of + the file as downloaded, release, summary level, published-MOE self-check) and every variable + without published replicates (``unavailable``: table not on the release's VRE list, or the + line missing from the file). Raises when a published file cannot be fetched.""" + from . import replicate as vre + from .analysis import REPLICATE_COLUMNS + from .pipeline import expand_states + geo = m["geography"] + level = _vre_level(geo["level"]) + sumlev = vre.sumlev_for(level) + states: List[Optional[str]] = [] + for st in manifest_states(geo): + states += list(expand_states(st)) if st else [] + calls: List[Optional[str]] = sorted(set(states)) if sumlev in vre.PER_STATE_SUMLEVS else [None] + if not calls: # per-state files and no state: nothing could be fetched (validation requires a state) + raise ValueError(f"Census VRE files of summary level {sumlev} ({geo['level']}) are published per state, " + "and the manifest geography names no state (geography.state or geography.states), so " + "no VRE file could be requested") + units = frame[["GEOID", "Year"]].astype({"GEOID": str, "Year": "int64"}) + releases = [int(r) for r in m["releases"]] + names_by_table: Dict[str, List[str]] = {} + for v in m["variables"]: + names_by_table.setdefault(str(_table_of(v)), []).append(str(v["name"])) + files: List[Dict[str, Any]] = [] + unavailable: List[Dict[str, Any]] = [] + parts: List[pd.DataFrame] = [] + covered: Dict[str, List[int]] = {} + for table, names in names_by_table.items(): + for year in releases: + info = vre.vre_availability(table, year, level) + if not info["available"]: + unavailable += [{"variable": n, "table": table, "release": year, "reason": info["reason"]} + for n in names] + continue + want = set(units.loc[units["Year"] == year, "GEOID"]) + frames = [] + for st in calls: + f = vre.fetch_vre(table, year, level, state_code=st, client=client) + files.append({"table": table, "release": year, "summary_level": f.attrs.get("summary_level"), + "state_code": f.attrs.get("state_code"), "url": f.attrs.get("source_url"), + "sha256": f.attrs.get("sha256"), "moe_check": f.attrs.get("moe_check")}) + frames.append(f[f["GEOID"].astype(str).isin(want)]) + rows = pd.concat(frames, ignore_index=True) if frames else pd.DataFrame() + for n in names: + code = (codes_by_year.get(n) or {}).get(year) + sub = rows[rows["variable"] == code] if code is not None and len(rows) else rows.iloc[0:0] + if sub.empty: + unavailable.append({"variable": n, "table": table, "release": year, + "reason": f"{code or 'no code'} has no rows in the {table} VRE file for " + "the analysed units"}) + continue + part = sub[["GEOID", "estimate"] + REPLICATE_COLUMNS].copy() + part.insert(1, "Year", int(year)) + part.insert(2, "variable", n) + parts.append(part) + covered.setdefault(n, []).append(int(year)) + reps = pd.concat(parts, ignore_index=True) if parts else None + if reps is not None: + reps.attrs["synthetic"] = False + record = {"source": "census_vre", "synthetic": False, "files": files, "variables": covered, + "unavailable": unavailable, "documentation": vre.VRE_DOCUMENTATION} + return reps, record + + +def _scale_rows(frame: pd.DataFrame, replicates: Any, variables: List[str], rows: np.ndarray, factor: float, + year: Optional[int]) -> Tuple[pd.DataFrame, Any]: + """Multiply estimates and SEs of ``variables`` on ``rows`` (and their replicates of release + ``year``) by ``factor`` (the CPI conversion to the dollar year).""" + from .analysis import REPLICATE_COLUMNS + frame = frame.copy() + for v in variables: + for col in (v, f"{v}_se"): + vals = pd.to_numeric(frame[col], errors="coerce").to_numpy(float).copy() + vals[rows] = vals[rows] * factor + frame[col] = vals + if replicates is not None: + reps = replicates.copy() + sel = reps["variable"].isin(variables).to_numpy(copy=True) + if year is not None: + sel = sel & (pd.to_numeric(reps["Year"]).astype(int) == int(year)).to_numpy() + cols = ["estimate"] + [c for c in REPLICATE_COLUMNS if c in reps.columns] + reps[cols] = reps[cols].astype(float) + reps.loc[sel, cols] = reps.loc[sel, cols].to_numpy(float) * factor + replicates = reps + return frame, replicates + + def _apply_adjustments(frame: pd.DataFrame, replicates: Any, m: Mapping[str, Any], - release: int) -> Tuple[Optional[pd.DataFrame], Any, List[Dict[str, Any]], Optional[str]]: + releases: List[int]) -> Tuple[Optional[pd.DataFrame], Any, List[Dict[str, Any]], Optional[str]]: """Run the manifest's ``adjustments`` in order with the place_adjust_frame core. - Returns (frame, replicates, log, failure message or None).""" + + Per release (D15-2): with a declared ``dollar_year``, estimates, SEs and replicates of the + release are first converted to dollars of that year with the declared price index + (CPI(dollar_year) / CPI(release); R-CPI-U-RS by default, the index the Census Bureau uses for + ACS dollars), then divided by the RPP / 100 of the release's own ``rpp_years`` entry (one + release: ``rpp_year``). Returns (frame, replicates, log, failure message or None).""" from . import rpp + from .inflation import INDEX_DISPLAY, get_cpi, price_index_info from .pipeline import normalize_geo_level from .workflows_microdata import place_adjust level = normalize_geo_level(m["geography"]["level"]) log: List[Dict[str, Any]] = [] + multi = len(releases) > 1 + if frame.empty: # e.g. exclusions removed every unit: nothing to scale; the analyses report the outcome + return frame, replicates, log, None for i, adj in enumerate(m.get("adjustments") or []): variables = [str(v) for v in adj["variables"]] areas = adj.get("areas", default_rpp_areas(level)) - rpp_year = int(adj["rpp_year"]) # required by validation, so the hash fixes it + dollar_year = adj.get("dollar_year") + index = adj.get("price_index") + rpp_years = {int(k): int(v) for k, v in (adj.get("rpp_years") or {}).items()} \ + or {int(releases[0]): int(adj["rpp_year"])} # required by validation, so the hash fixes them + per_release: List[Dict[str, Any]] = [] + parts: List[pd.DataFrame] = [] + rep_parts: List[pd.DataFrame] = [] try: - res = place_adjust(frame, variables, release, areas=areas, replicates=replicates, rpp_year=rpp_year) + if multi and replicates is not None and "Year" not in replicates.columns: + raise ValueError("replicates need a Year column when the manifest has several releases") + # one release: the whole frame (as in Round 14); several: the rows of each release + years_col = pd.to_numeric(frame["Year"]).astype(int).to_numpy() if multi \ + else np.full(len(frame), int(releases[0])) + work, wreps = frame, replicates + cpis: Dict[int, Optional[Dict[str, Any]]] = {} + for release in releases: # 1. price index: release dollars -> dollar_year dollars + cpis[release] = None + if dollar_year is not None: + c_from, c_to = get_cpi(int(release), index=index), get_cpi(int(dollar_year), index=index) + cpis[release] = {"index": index, "from_year": int(release), "to_year": int(dollar_year), + "index_from": c_from, "index_to": c_to, "factor": c_to / c_from} + work, wreps = _scale_rows(work, wreps, variables, years_col == int(release), c_to / c_from, + int(release) if multi else None) + done: List[int] = [] + for release in releases: # 2. RPP of the release's own rpp_years entry + rows = years_col == int(release) + if not rows.any(): + continue + sub_reps = wreps + if wreps is not None and multi: + sub_reps = wreps[pd.to_numeric(wreps["Year"]).astype(int).to_numpy() == int(release)] + res = place_adjust(work[rows], variables, int(release), areas=areas, replicates=sub_reps, + rpp_year=rpp_years[int(release)]) + parts.append(res["frame"]) + if sub_reps is not None: + rep_parts.append(res["replicates"]) + done.append(int(release)) + cpi = cpis[release] + notes = ([f"PRICE INDEX: release {release} dollars converted to {cpi['to_year']} dollars with " + f"{INDEX_DISPLAY.get(index, index)} (factor {cpi['factor']:.6f}) before the RPP division " + "below."] if cpi else []) + list(res["notes"]) + per_release.append({"release": int(release), "rpp_year": res["rpp_year"], "cpi": cpi, + "rpp_year_matches_release": res["rpp_year"] == int(release), + "rpp_areas": res["rpp_areas"], "boundary_source": res["boundary_source"], + "rows": int(len(res["frame"])), "notes": notes, + "advisories": res["advisories"], "custom_table": res["custom_table"]}) + if not per_release: + raise ValueError("no rows to adjust") except Exception as exc: # noqa: BLE001 - recorded as a blocking deviation, never substituted return None, replicates, log, f"adjustments[{i}] ({adj.get('type')}): {type(exc).__name__}: {exc}" - frame = res["frame"] - if replicates is not None: - replicates = res["replicates"] - notes = list(res["notes"]) + if multi: + frame = pd.concat(parts).loc[frame.index[np.isin(years_col, done)]] + if wreps is not None: + rest = wreps[~np.isin(pd.to_numeric(wreps["Year"]).astype(int).to_numpy(), done)] + replicates = pd.concat([rest] + rep_parts, ignore_index=True) + else: + frame = parts[0] + if wreps is not None: + replicates = rep_parts[0] info = rpp.price_parity_info() - table: Dict[str, Any] = {"source": "custom" if res["custom_table"] else "bundled"} - if res["custom_table"]: - table.update(res["custom_table"]) + custom = next((r["custom_table"] for r in per_release if r["custom_table"]), None) + table: Dict[str, Any] = {"source": "custom" if custom else "bundled"} + if custom: + table.update(custom) else: table["sha256"] = (info.get("csv_sha256") or {}).get("rpp.csv") - log.append({"type": adj["type"], "variables": variables, "areas": areas, "dollar_year": int(release), - "rpp_year": res["rpp_year"], "rpp_year_source": "manifest", - "rpp_year_matches_release": res["rpp_year"] == int(release), - "rpp_areas": res["rpp_areas"], "boundary_source": res["boundary_source"], - "rpp_table": table, "rows": int(len(frame)), "notes": notes, - "advisories": res["advisories"], "applied_by_default": False}) + first = per_release[0] + entry: Dict[str, Any] = { + "type": adj["type"], "variables": variables, "areas": areas, + "dollar_year": int(dollar_year) if dollar_year is not None else int(releases[0]), + "rpp_year": first["rpp_year"] if not multi else None, "rpp_year_source": "manifest", + "rpp_year_matches_release": all(r["rpp_year_matches_release"] for r in per_release), + "rpp_areas": first["rpp_areas"] if not multi else + [dict(a, release=r["release"]) for r in per_release for a in r["rpp_areas"]], + "boundary_source": first["boundary_source"], "rpp_table": table, + "rows": int(len(frame)), "notes": [n for r in per_release for n in r["notes"]], + "advisories": sorted({a for r in per_release for a in r["advisories"]}), "applied_by_default": False} + if dollar_year is not None: + pinfo = price_index_info(index) + entry["price_index"] = {"index": index, "name": INDEX_DISPLAY.get(index, index), + "sha256": pinfo.get("custom_table", {}).get("sha256") or pinfo.get("csv_sha256"), + "source": "custom" if pinfo.get("custom_table") else "bundled"} + entry["order"] = "price index to dollar_year first, then the RPP of each release's rpp_year" + if multi or adj.get("rpp_years") is not None: + entry["rpp_years"] = {str(r["release"]): r["rpp_year"] for r in per_release} + entry["releases"] = [{k: r[k] for k in ("release", "rpp_year", "cpi", "rpp_year_matches_release", + "boundary_source", "rows")} for r in per_release] + elif dollar_year is not None: + entry["releases"] = [{k: r[k] for k in ("release", "rpp_year", "cpi")} for r in per_release] + log.append(entry) return frame, replicates, log, None @@ -1336,7 +1916,9 @@ def _write_bundle(folder: Path, m, h, results, hyp_df, run_info, citation) -> Di if "adjustments" in run_info: # RPP provenance: years, areas, table hash provenance["adjustments"] = [{k: a[k] for k in ("type", "variables", "areas", "dollar_year", "rpp_year", "rpp_year_source", "rpp_areas", "boundary_source", - "rpp_table")} for a in run_info["adjustments"]] + "rpp_table", "rpp_years", "price_index", "order", + "releases") if k in a} + for a in run_info["adjustments"]] return results_bundle(folder, tables=tables, figures=figures_, methods=methods_section(m, run_info), results=payload, provenance=provenance, citation=citation) @@ -1370,6 +1952,41 @@ def _write_bundle(folder: Path, m, h, results, hyp_df, run_info, citation) -> Di } +def _multi_release_adjustment_text(m: Mapping[str, Any], adj: Mapping[str, Any], + done: Optional[Mapping[str, Any]]) -> str: + """Methods text of an adjustment with a declared dollar year (CPI, then RPP; D15-2).""" + from .inflation import INDEX_DISPLAY + index = INDEX_DISPLAY.get(str(adj.get("price_index")), str(adj.get("price_index"))) + rpp_years = adj.get("rpp_years") or {str(m.get("releases", [None])[0]): adj.get("rpp_year")} + areas = (done or {}).get("areas") or adj.get("areas") or "default" + text = (f"- Price adjustment{' (after the exclusions)' if m.get('exclusions') else ''}, in this order: " + f"(1) the values of {', '.join(adj.get('variables', []))} from each release were converted from the release's " + f"end-year dollars to {adj['dollar_year']} dollars with the {index} (annual averages; factor " + f"{index}({adj['dollar_year']}) / {index}(release end year)); (2) the converted values were divided by " + "the BEA Regional Price Parity (U.S. = 100) / 100 of each unit's area for the release's own RPP year (" + + ", ".join(f"release {k}: RPP {v}" for k, v in sorted(rpp_years.items())) + ")" + + (" (the county's metropolitan statistical area, else the state's nonmetropolitan portion; CBSA " + "membership from Census boundaries of the OMB 23-01 delineation)" if areas == "county_metro" + else " (its state)" if areas == "state" else "") + + ". Estimates, SEs and any replicates are scaled by the same factors and CVs are unchanged. Both " + "factors are multiplicative, so the order does not change the values; it is stated so the price " + "concepts are clear (national inflation first, then place). The price index and RPPs are treated " + "as fixed: their own estimation error is not in any SE or interval.") + if done: + for r in done.get("releases") or []: + cpi = r.get("cpi") or {} + if cpi: + text += (f"\n Release {r['release']}: {index} {cpi['index_from']} -> {cpi['index_to']} (factor " + f"{cpi['factor']:.6f}); RPP year {r['rpp_year']}.") + pidx = done.get("price_index") or {} + text += (f"\n {len(done.get('rpp_areas') or [])} release-by-RPP-area entries were used; RPP table " + + (f"custom {done['rpp_table'].get('path')} (SHA-256 {done['rpp_table'].get('sha256')}, not verified)" + if done["rpp_table"].get("source") == "custom" + else f"bundled (SHA-256 {done['rpp_table'].get('sha256')})") + + f"; price index table {pidx.get('source', 'bundled')} (SHA-256 {pidx.get('sha256')}).") + return text + + def methods_section(manifest: Mapping[str, Any], results: Optional[Mapping[str, Any]] = None) -> str: """Markdown methods text matching what the manifest specifies (and, with ``results`` = the run info of :func:`run_manifest`, what actually ran).""" @@ -1380,7 +1997,19 @@ def methods_section(manifest: Mapping[str, Any], results: Optional[Mapping[str, run = results or {} source = run.get("data_source") or effective_data_source(m) rep_info = run.get("replicates") - if rep_info: + if rep_info and rep_info.get("source") == "census_vre": + files = rep_info.get("files") or [] + tabs = sorted({f["table"] for f in files}) + rep_source = ("the 80 replicate estimates of the Census Bureau's variance replicate estimate (VRE) files " + f"fetched by the toolkit ({len(files)} file(s): tables {', '.join(tabs) or 'none'}; URLs and " + "SHA-256 of each file in run.json" + + (f"; canonical SHA-256 of the replicates used `{rep_info['sha256']}`" if rep_info.get("sha256") + else "") + ")") + rep_short = " and replicate-based standard errors from Census VRE files" + elif results is None and census_vre_source(m): + rep_source = "the 80 replicate estimates of the Census VRE files fetched for the manifest's tables" + rep_short = " and replicate-based standard errors from Census VRE files" + elif rep_info: kind = "synthetic caller-supplied" if rep_info.get("synthetic") else "caller-supplied" rep_source = (f"the {kind} replicate estimates (not fetched from Census variance replicate files by " + (f"the toolkit; canonical SHA-256 `{rep_info['sha256']}`)" if rep_info.get("sha256") @@ -1419,9 +2048,13 @@ def methods_section(manifest: Mapping[str, Any], results: Optional[Mapping[str, "rather than being substituted):") lines += [f"- `{v['name']}`: {v['ref']}" + (f" (table {v['table']})" if v.get("table") else "") for v in m.get("variables", [])] - if len(m.get("releases", [])) > 1: - lines.append("Multiple releases were linked through the concordance; overlapping 5-year periods " - "share sample years and are not independent.") + if len(manifest_releases(m)) > 1: + if release_overlap_message(manifest_releases(m), "releases") is not None: + lines.append("Multiple releases were linked through the concordance; overlapping 5-year periods " + "share sample years and are not independent.") + else: + lines.append("Multiple releases were linked through the concordance; the 5-year periods do not " + "overlap (no shared sample years), so their estimates come from independent samples.") if m.get("aggregation"): lines.append(f"Geographies were aggregated into {len(set(m['aggregation']['mapping'].values()))} groups; " "aggregate MOEs use replicate variance where available, else the ACS approximation " @@ -1450,6 +2083,9 @@ def methods_section(manifest: Mapping[str, Any], results: Optional[Mapping[str, run_adj = {i: a for i, a in enumerate(run.get("adjustments") or [])} for i, adj in enumerate(m.get("adjustments") or []): done = run_adj.get(i) + if adj.get("dollar_year") is not None: + lines.append(_multi_release_adjustment_text(m, adj, done)) + continue year = (done or {}).get("rpp_year") or adj.get("rpp_year") areas = (done or {}).get("areas") or adj.get("areas") or "default" lines.append(f"- Price adjustment{' (after the exclusions)' if m.get('exclusions') else ''}: " @@ -1470,6 +2106,11 @@ def methods_section(manifest: Mapping[str, Any], results: Optional[Mapping[str, + (f"custom {done['rpp_table'].get('path')} (SHA-256 {done['rpp_table'].get('sha256')}, " "not verified)" if done["rpp_table"].get("source") == "custom" else f"bundled (SHA-256 {done['rpp_table'].get('sha256')})") + ".") + if rep_info and rep_info.get("source") == "census_vre" and rep_info.get("unavailable"): + miss = sorted({f"{u['variable']} ({u['table']})" for u in rep_info["unavailable"]}) + lines.append(f"No Census VRE replicates are published for {', '.join(miss)}; replicate analyses using " + "them as variables with sampling error stop under the strict confirmatory policy unless " + "the manifest declares otherwise (below).") lines.append("Rows with missing or undefined estimates were dropped analysis by analysis (complete cases).") lines += ["", "### Analyses", ""] for a in m.get("analyses", []): @@ -1488,6 +2129,25 @@ def methods_section(manifest: Mapping[str, Any], results: Optional[Mapping[str, "(se_sampling)", "se_total": "combined SE (se_total)"}[info["basis_column"]], cov=", ".join(a.get("error_covariates") or args.get("error_covariates") or []), rep=rep_short if (a.get("replicates") or rep_info) else "", rep_source=rep_source) + if a.get("replicate_scope") == "covariates_only": + held = _formula_names(str(a.get("formula", "")).split("~")[0]) + had = ((rep_info or {}).get("covariates_only_outcome_had_replicates") or {}).get(a["id"]) + text += (f" Pre-registered replicate scope `covariates_only`: the outcome ({', '.join(held)}) was held " + "at its estimate in every refit" + + (" as declared, although replicates were available for it in this run" if had + else " (declared because it has no published replicates)") + + ", so se_sampling covers the covariates' sampling error only; the stated interval does not " + "use se_sampling alone.") + if a.get("formula") and "C(Year)" in str(a["formula"]) and len(manifest_releases(m)) > 1: + if release_overlap_message(manifest_releases(m), "releases") is None: + text += (" `C(Year)` contrasts the releases; replicate r of each release enters refit r together. " + "The ACS 5-year periods of these releases share no sample years, so their replicates come " + "from independent samples: the cross-release products have expectation zero and the " + "replicate variance of the contrast estimates the sum of the releases' sampling variances.") + else: + text += (" `C(Year)` contrasts releases whose ACS 5-year periods overlap (shared sample years): " + "their estimates are not independent, and no standard error reported here accounts for " + "that covariance (replicate SEs across overlapping releases are refused).") target = a.get("formula") or a.get("variables") or a.get("variable") lines.append(f"- **{a['id']}** (`{target}`): {text}") cites += refs diff --git a/census/mcp_profiles.py b/census/mcp_profiles.py index 90473ec..c3e70f7 100644 --- a/census/mcp_profiles.py +++ b/census/mcp_profiles.py @@ -76,7 +76,8 @@ DESCRIPTION_OVERRIDES: Dict[str, str] = { "get_recipe": ( "Return numbered steps for a standard analysis. Call this first when unsure what to do.\n" - "name: compare_counties | tract_trend | regression_sampling_error | small_area | preregistered; " + "name: compare_counties | tract_trend | regression_sampling_error | small_area | preregistered | " + "decennial_counts; " "omit to list recipes.\n" "Example: get_recipe(name=\"compare_counties\")" ), @@ -102,6 +103,7 @@ "table_code: e.g. 'B17001'. geography: us|state|county|tract|place|zcta|msa, optional " "':codes' e.g. 'county:453,201'.\n" "state_code: 2-digit FIPS. years: releases (default latest). variables: codes used.\n" + "dataset: 'dec/dhc' or 'dec/pl' for 2020 Census counts (no MOE).\n" "Example: download_table(table_code=\"B17001\", geography=\"county:453\", state_code=\"48\", " "years=[2023], variables=[\"B17001_001\", \"B17001_002\"])" ), @@ -312,5 +314,7 @@ def build_profile_server(base: Any, profile: str) -> Any: description = DESCRIPTION_OVERRIDES.get(name, tool.description) else: fn, description = tool.fn, tool.description - server.add_tool(fn, name=name, description=description) + # Python 3.13+ strips docstring indentation at compile time; older versions do not. + # Normalise so every Python sends the same (shorter) text and the budget holds. + server.add_tool(fn, name=name, description=inspect.cleandoc(description) if description else description) return server diff --git a/census/metadata.py b/census/metadata.py index 4b4e92e..00db743 100644 --- a/census/metadata.py +++ b/census/metadata.py @@ -1,5 +1,5 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -"""Per-vintage ACS variable and table metadata. +"""Per-vintage ACS (and 2020 Decennial Census) variable and table metadata. Source order for one (dataset, year): @@ -32,6 +32,9 @@ "acs1/profile": "acs/acs1/profile", "acs1/subject": "acs/acs1/subject", "acs1/cprofile": "acs/acs1/cprofile", + # 2020 Decennial Census (census.decennial), archived under metadata/dec//2020/ + "dec/pl": "dec/pl", + "dec/dhc": "dec/dhc", } SOURCE_ARCHIVE = "archive" @@ -103,7 +106,8 @@ def _normalize_variables(payload: Dict[str, Any]) -> Dict[str, VariableMeta]: if not isinstance(d, dict): continue out[code] = { - "label": str(d.get("label", "")).strip(), + # 2020 Census labels start with an empty part (" !!Total:"): drop it + "label": str(d.get("label", "")).strip().removeprefix("!!"), "concept": str(d.get("concept", "")).strip(), "group": str(d.get("group", "")), "predicateType": str(d.get("predicateType", "")), @@ -199,9 +203,19 @@ def latest_vintage(dataset: str) -> Optional[int]: def describe_variable( code: str, year: Optional[int] = None, dataset: Optional[str] = None, client: Any = None ) -> Dict[str, Any]: - """Label, concept, table and universe for one variable code in one vintage.""" + """Label, concept, table and universe for one variable code in one vintage. + + 2020 Census count codes (``P12_002N``) default to the DHC (``dec/dhc``); + pass ``dataset='dec/pl'`` for P.L. 94-171 codes. Decennial results carry + an ``uncertainty`` statement (not sample-based; MOE undefined). + """ from census.api import infer_dataset # local import avoids a cycle + from census.decennial import DECENNIAL_DHC, UNCERTAINTY_LABEL, is_decennial + from census.stats import parse_variable_code + parsed = parse_variable_code(str(code)) + if dataset is None and parsed and parsed[1] in ("N", "NA"): + dataset = DECENNIAL_DHC dataset = dataset or infer_dataset(code) year = year or latest_vintage(dataset) if year is None: @@ -227,4 +241,5 @@ def describe_variable( "universe": table_universe(meta["group"], dataset, year, client), "predicate_type": meta["predicateType"], "metadata_source": source, + **({"uncertainty": UNCERTAINTY_LABEL} if is_decennial(dataset) else {}), } diff --git a/census/models.py b/census/models.py index 347f6ba..8d30852 100644 --- a/census/models.py +++ b/census/models.py @@ -294,6 +294,7 @@ def fit_replicate( max_failed_replicates: int = 0, target: str = "population", replicate_policy: str = "flag", + fixed_variables: Sequence[str] = (), ) -> ModelResult: """Fit a statsmodels formula model with ACS replicate sampling SEs. @@ -314,6 +315,12 @@ def fit_replicate( and records diagnostics["replicates_incomplete"]; "strict" (confirmatory, used by census.manifest) raises ValueError instead. Replicates do not correct attenuation from noisy covariates; use regression_calibration for that. + ``fixed_variables``: variables declared in advance (a manifest's + ``replicate_scope: "covariates_only"``) to be held at their estimates in + every refit, whether or not replicates exist; they are not counted as + incomplete replicates, and the result records them + (diagnostics["replicates_declared_fixed"], a note): se_sampling then excludes + their sampling error. """ model = model.lower() check_replicate_policy(replicate_policy) @@ -337,10 +344,14 @@ def fit_replicate( method = "full_sample_fit" if replicates is not None and len(replicates): - mats, fixed = _replicate_matrices(data, replicates, used) + declared = [v for v in used if v in set(fixed_variables)] + if declared and target == "these_areas": + raise ValueError("fixed_variables cannot be combined with target='these_areas': its interval would " + f"use a sampling SE without the sampling error of {declared}") + mats, fixed = _replicate_matrices(data, replicates, [v for v in used if v not in declared]) if not mats: raise ValueError("no formula variable has replicates in the replicate frame") - missing = [v for v in used if v not in mats] + missing = [v for v in used if v not in mats and v not in declared] stoch_missing = stochastic_variables(frame, missing) precheck_replicates(replicate_policy, rows_held_fixed=fixed, variables_without_replicates=stoch_missing, context="fit_replicate") @@ -376,6 +387,11 @@ def fit_replicate( } diagnostics["replicate_failures"] = failed provenance["replicate_variables"] = sorted(mats) + if declared: + diagnostics["replicates_declared_fixed"] = declared + notes.append(f"DECLARED COVARIATE-ONLY REPLICATES: {declared} held at their estimates in every refit " + "as pre-declared; se_sampling covers the sampling error of " + f"{sorted(mats)} only, not of {declared}") notes.append("se_total = sqrt(se_model^2 + se_sampling^2) is reported but conservative (validated " "coverage 0.961 at nominal 0.90): the model residual already absorbs outcome sampling error") else: diff --git a/census/multitest.py b/census/multitest.py index da3f0c8..82f3499 100644 --- a/census/multitest.py +++ b/census/multitest.py @@ -172,6 +172,7 @@ def batch_z_test( suggested for exploratory screening) or ``by``. product: ``'5-year'`` (default) or ``'1-year'``. 1-year releases do not share sample (no overlap adjustment); same release = independent. + A decennial product raises ValueError (census counts are not sample-based). Returns: DataFrame indexed like ``df`` (``GEOID``/``NAME`` carried when diff --git a/census/panel.py b/census/panel.py index 8ade542..d27a109 100644 --- a/census/panel.py +++ b/census/panel.py @@ -31,6 +31,7 @@ SLUG_SUFFIXES: Dict[str, str] = { "E": "est", + "N": "count", "M": "moe", "PE": "pct", "PM": "pct_moe", @@ -53,7 +54,8 @@ def classify_column(col: str) -> Optional[Tuple[str, str, bool]]: """Return (variable_base, kind, is_percent) for a value/derived column. - kind is one of E, M, PE, PM, ANN, SE, CV, RELIABILITY, CONTROLLED, NOTE(:). + kind is one of E, N (2020 Census count), M, PE, PM, ANN, SE, CV, RELIABILITY, + CONTROLLED, NOTE(:). """ if col.endswith("_NOTE"): inner = classify_column(col[: -len("_NOTE")]) @@ -80,8 +82,10 @@ def _label_of(code: str, metadata: Metadata) -> str: return "" # Labels come from a single metadata vintage; the year in "(in 2021 # inflation-adjusted dollars)" would be wrong for other periods. The - # codebook Unit column records the actual dollar basis. - return _DOLLAR_IN_LABEL_RE.sub(" (inflation-adjusted dollars)", str(entry.get("label", ""))) + # codebook Unit column records the actual dollar basis. 2020 Census + # labels start with an empty part (" !!Total:"); it is dropped. + label = re.sub(r"^\s*!!", "", str(entry.get("label", ""))) + return _DOLLAR_IN_LABEL_RE.sub(" (inflation-adjusted dollars)", label) def base_label(base: str, is_percent: bool, metadata: Metadata) -> str: @@ -92,6 +96,7 @@ def base_label(base: str, is_percent: bool, metadata: Metadata) -> str: _label_of(f"{base}E", metadata) or _label_of(f"{base}PE", metadata) or _label_of(f"{base}SS", metadata) + or _label_of(f"{base}N", metadata) ) @@ -107,7 +112,7 @@ def describe_column(col: str, metadata: Metadata) -> str: return "" base, kind, pct = info found = base_label(base, pct, metadata) - if not found and kind in ("E", "PE"): + if not found and kind in ("E", "PE", "N"): return "" # no label known: keep the raw code rather than invent one label = found or f"{base}{'PE' if pct else 'E'}" stripped = _LABEL_PREFIX_RE.sub("", label) @@ -116,6 +121,7 @@ def describe_column(col: str, metadata: Metadata) -> str: return f"Census footnote for {inner}" return { "E": label, + "N": label, "PE": label, "M": f"Margin of Error (90%)!!{stripped}", "PM": f"Percent Margin of Error (90%)!!{stripped}", @@ -136,6 +142,7 @@ def _annotation_kind(col: str) -> str: "MA": "Margin of Error", "PEA": "Percent Estimate", "PMA": "Percent Margin of Error", + "NA": "Count", }.get(suffix, "") @@ -160,6 +167,7 @@ def measurement_type(col: str) -> str: return "Annotation" return { "E": "Estimate", + "N": "Count (not sample-based)", "M": "Margin of Error", "PE": "Percentage", "PM": "Margin of Error", @@ -189,7 +197,7 @@ def infer_unit( if kind == "CV" or pct: return "Percent" label = base_label(base, pct, metadata) - entry = metadata.get(f"{base}E") or {} + entry = metadata.get(f"{base}E") or metadata.get(f"{base}N") or {} if is_dollar_variable(str(entry.get("label", "")), str(entry.get("concept", ""))): if inflation_base_year: return f"Dollars (constant {inflation_base_year} dollars, {price_index})" @@ -334,13 +342,19 @@ def generate_codebook( slug_style: str = "label", price_index: str = "R-CPI-U-RS", concordance: Optional[Any] = None, + decennial: bool = False, ) -> pd.DataFrame: """Build a data dictionary for the given output columns. With a ``census.concordance.Concordance`` whose representative bases name the columns, adds Concept_ID, Years_Available, Source_Codes_By_Year and - Concordance_Status. + Concordance_Status. ``decennial`` (2020 Census datasets) adds an + ``Uncertainty`` column stating that counts are not sample-based (MOE + undefined) and carry disclosure-avoidance noise, and describes ``Year`` + as the census year. """ + from census.decennial import codebook_uncertainty # local: keep panel light + rep_to_cid: Dict[str, str] = {} if concordance is not None: rep_to_cid = {b: cid for cid, b in concordance.representative_bases().items()} @@ -354,7 +368,8 @@ def generate_codebook( concept = _DOLLAR_IN_LABEL_RE.sub( " (INFLATION-ADJUSTED DOLLARS)", str( - (metadata.get(f"{info[0]}E") or metadata.get(f"{info[0]}PE") or {}).get( + (metadata.get(f"{info[0]}E") or metadata.get(f"{info[0]}PE") + or metadata.get(f"{info[0]}N") or {}).get( "concept", "" ) ), @@ -370,6 +385,10 @@ def generate_codebook( "Unit": infer_unit(col, metadata, inflation_base_year, price_index), } ) + if decennial: + if col == "Year": + rows[-1]["Full_Description"] = "Census year (2020 Census; counts as of April 1, 2020)" + rows[-1]["Uncertainty"] = codebook_uncertainty(info[1]) if info else "" if concordance is not None: cid = rep_to_cid.get(info[0], "") if info else "" c = concordance.concepts.get(cid) if cid else None @@ -381,7 +400,8 @@ def generate_codebook( "Concordance_Status": ";".join(c.statuses) if c else "", } ) - cols = CODEBOOK_COLUMNS + (CONCORDANCE_CODEBOOK_COLUMNS if concordance is not None else []) + cols = (CODEBOOK_COLUMNS + (["Uncertainty"] if decennial else []) + + (CONCORDANCE_CODEBOOK_COLUMNS if concordance is not None else [])) return pd.DataFrame(rows, columns=cols) @@ -430,7 +450,7 @@ def build_tidy_panel( panel = combined[id_cols + value_cols].copy() for col in value_cols: kind = classify_column(str(col))[1] # type: ignore[index] - if kind in ("E", "M", "PE", "PM", "SE", "CV"): + if kind in ("E", "N", "M", "PE", "PM", "SE", "CV"): panel[col] = pd.to_numeric(panel[col], errors="coerce") else: panel[col] = panel[col].fillna("").astype(str) diff --git a/census/pipeline.py b/census/pipeline.py index 9d31cbc..92e7040 100644 --- a/census/pipeline.py +++ b/census/pipeline.py @@ -54,6 +54,19 @@ ) from census.continuity import continuity_check from census.data_integrity import BundledDataIntegrityError, integrity_error_result +from census.decennial import ( + BLOCK_ADVISORY, + DECENNIAL_DATASETS, + DECENNIAL_DHC, + DECENNIAL_GEO_LEVELS, + DECENNIAL_PL, + DECENNIAL_YEARS, + DHC_BLOCK_TABLES, + DISCLOSURE_AVOIDANCE_ADVISORY, + PRODUCT_DECENNIAL, + is_decennial, + uncertainty_record, +) from census.export import ( process_data_and_save, sanitize_filename, @@ -90,7 +103,7 @@ from census.stats import add_reliability_columns, clean_census_sentinels, parse_variable_code from census.tables import load_variable_metadata -PACKAGE_VERSION = "0.13.0" +PACKAGE_VERSION = "0.14.0" CONCORDANCE_MODES = ("fail", "strict", "code") MAX_ISSUES_IN_RESULT = 100 @@ -99,7 +112,7 @@ PANEL_FORMATS = ("panel", "parquet") OUTPUT_FORMATS = LEGACY_FORMATS + PANEL_FORMATS -GEO_LEVEL_NAMES = ("us", "state", "county", "tract", "block group", "place", "zcta", "msa") +GEO_LEVEL_NAMES = ("us", "state", "county", "tract", "block group", "block", "place", "zcta", "msa") # Release whose county list enumerates block-group requests when no counties # are given (one API request per county). BLOCK_GROUP_COUNTY_LIST_YEAR = 2023 @@ -159,7 +172,9 @@ class ResearchOptions: continuity_check: bool = True allow_unverified_concordance: bool = False # ACS product: "5-year" (default) or "1-year" (acs1 datasets; areas of - # 65,000+ only; 1-year and 5-year estimates are never compared). + # 65,000+ only; 1-year and 5-year estimates are never compared), or + # "decennial" (2020 Census, dec/dhc by default or dataset='dec/pl'; counts + # that are not sample-based, census.decennial). product: str = PRODUCT_5YR # 1-year only: the 2020 1-year release is unavailable (Census published only # experimental estimates, outside the API). True drops 2020 from the request @@ -314,6 +329,8 @@ def _nationwide_entities( 5-year vintage; ``state:*`` would also return Puerto Rico. Tracts cannot use a state wildcard, so they are one request per state, sharing one group. """ + if geo_level == "block": + raise ValueError("Census blocks are requested one state at a time; give a state_code (and county codes).") states = expand_states(NATIONWIDE, include_pr) joined = ",".join(states) label = NATIONWIDE_LABEL + ("_and_PR" if include_pr else "") @@ -365,6 +382,8 @@ def build_entities( block group (aliases block_group, bg, 150) -> county FIPS codes; one request per county (default: every county, listed with ``client``); GEOIDs are 12 digits + block -> 2020 Census only: county FIPS codes (one request each; + default one state-wide request); GEOIDs are 15 digits zcta -> 5-digit ZCTAs (default: ZIPs in state via zip2fips.json, or all ZCTAs nationally when no state is given) msa -> CBSA codes (default: all metro/micro areas) @@ -393,6 +412,23 @@ def build_entities( for st in bg_states: bg.extend(_block_group_entities(st, codes, names, client, years=years)) return bg + if geo_level == "block": + # 2020 Census only: one request per county given, else one per state (county:*). + blk_states = expand_states(state_code) + if len(blk_states) > 1 and codes: + raise ValueError("block county codes need a single state_code.") + blocks: List[Entity] = [] + for st in blk_states: + st_name = STATE_MAPPING.get(st, f"State_{st}") + if not codes: + blocks.append(Entity(f"{st_name}_All_Blocks", "block", st, county_code="*")) + continue + blocks.extend( + Entity(f"{names.get(c, st_name + '_County_' + c)}_Blocks", "block", st, + county_code=str(c).strip().zfill(3), group=f"{st_name}_Blocks") + for c in codes + ) + return blocks if geo_level == "place": if not codes: @@ -595,6 +631,43 @@ def _errata_summary(table_code: str, years: Sequence[int]) -> Any: return {"error": f"errata lookup unavailable: {exc}"} +def _decennial_plan_error( + table_code: str, dataset: str, years: Sequence[int], entities: Sequence[Entity], options: ResearchOptions +) -> Optional[Dict[str, Any]]: + """Refusals specific to the 2020 Census datasets (None when the request is valid).""" + other = [y for y in years if y not in DECENNIAL_YEARS] + if other: + return errors.error_result( + errors.VINTAGE_UNAVAILABLE, + f"The decennial datasets here cover the 2020 Census only ({dataset} {list(DECENNIAL_YEARS)}); " + f"not available for {other}. Request years=[2020].", + unavailable_years=other, + ) + level = normalize_geo_level(entities[0].geo_level) + if level not in DECENNIAL_GEO_LEVELS: + return errors.error_result( + errors.INVALID_GEO_LEVEL, + f"2020 Census extractions support {list(DECENNIAL_GEO_LEVELS)}; not '{level}'.", + ) + if level == "block" and dataset != DECENNIAL_PL and table_code not in DHC_BLOCK_TABLES: + return errors.error_result( + errors.INVALID_GEO_LEVEL, + f"Blocks are offered for P.L. 94-171 tables (dataset='{DECENNIAL_PL}') and for the invariant " + f"housing-unit count (DHC {list(DHC_BLOCK_TABLES)}), not {dataset} {table_code}: block-level " + "characteristics are dominated by disclosure-avoidance noise. Request block groups or tracts.", + ) + if options.harmonize_tracts_to is not None: + return errors.error_result( + errors.INVALID_INPUT, + "harmonize_tracts_to does not apply to the 2020 Census (it is tabulated on 2020 tracts).", + ) + if options.inflation_base_year: + return errors.error_result( + errors.INVALID_INPUT, "The 2020 Census P.L. 94-171 and DHC files publish no dollar values." + ) + return None + + def plan_extraction( table_code: str, entities: Sequence[Entity], @@ -608,8 +681,8 @@ def plan_extraction( ) -> Dict[str, Any]: """Dry run: validate, build the concordance, check continuity, count calls. - ``product`` ('5-year' default, or '1-year'; overrides ``options.product``) - selects the ACS release family; a generic ``dataset`` ('acs5', 'profile', + ``product`` ('5-year' default, '1-year', or 'decennial' for the 2020 + Census; overrides ``options.product``) selects the release family; a generic ``dataset`` ('acs5', 'profile', ...) is mapped to the product's endpoint, and an explicit 1-year dataset ('acs1', 'acs1/subject', ...) implies '1-year'. 1-year plans probe which requested areas each release publishes (one cached listing call per year @@ -633,11 +706,33 @@ def plan_extraction( if not entities: return errors.error_result(errors.INVALID_INPUT, "No geographic entities requested.") explicit_product = product is not None or options.product != PRODUCT_5YR - try: - product = normalize_product(product if product is not None else options.product) - except ValueError as exc: - return errors.error_result(errors.INVALID_INPUT, str(exc)) - if dataset and dataset in DATASETS: + requested_product = product if product is not None else options.product + decennial = is_decennial(dataset) or is_decennial(requested_product) + if decennial: + if dataset and dataset not in DECENNIAL_DATASETS: + return errors.error_result( + errors.INVALID_DATASET, + (f"Dataset '{dataset}' is an ACS dataset but product='decennial' was requested; " + if dataset in DATASETS else f"Dataset '{dataset}' is not a dataset key; ") + + f"use dataset {list(DECENNIAL_DATASETS)} for the 2020 Census.", + ) + if explicit_product and not is_decennial(requested_product): + return errors.error_result( + errors.INVALID_DATASET, + f"Dataset '{dataset}' is the 2020 Census but product='{requested_product}' was requested. " + "Decennial counts and ACS estimates are separate products.", + ) + product = PRODUCT_DECENNIAL + dataset = dataset if dataset in DECENNIAL_DATASETS else DECENNIAL_DHC + decennial_error = _decennial_plan_error(table_code, dataset, years, entities, options) + if decennial_error: + return decennial_error + else: + try: + product = normalize_product(requested_product) + except ValueError as exc: + return errors.error_result(errors.INVALID_INPUT, str(exc)) + if dataset and dataset in DATASETS and not decennial: ds_product = dataset_product(dataset) if ds_product != product: if not explicit_product: @@ -652,7 +747,12 @@ def plan_extraction( ) dataset = dataset or infer_dataset(table_code, product) if dataset not in DATASETS: - return errors.error_result(errors.INVALID_DATASET, f"Unsupported dataset '{dataset}'.") + return errors.error_result( + errors.INVALID_DATASET, + f"Unsupported dataset '{dataset}'." + + (f" For the 2020 Census use dataset {list(DECENNIAL_DATASETS)} (or product='decennial')." + if str(dataset).strip().lower().startswith(("dec", "census")) else ""), + ) options = replace(options, product=product) if options.concordance_mode not in CONCORDANCE_MODES: return errors.error_result( @@ -666,6 +766,12 @@ def plan_extraction( client = CensusAPIClient(cache=get_default_cache() if options.use_cache else None) geo_level = entities[0].geo_level + if geo_level == "block" and not decennial: + return errors.error_result( + errors.INVALID_GEO_LEVEL, + "Census blocks are published only in the 2020 Census datasets (dataset='dec/pl', or " + "'dec/dhc' for H1 housing units), not in the ACS.", + ) excluded_years: List[int] = [] acs1_advisories: List[str] = [] if product == PRODUCT_1YR: @@ -723,7 +829,7 @@ def plan_extraction( f"harmonize_tracts_to requires a panel output format {sorted(PANEL_FORMATS)}.", ) - if geo_level == BLOCK_GROUP: + if geo_level == BLOCK_GROUP and not decennial: if dataset != "acs5": return errors.error_result( errors.INVALID_GEO_LEVEL, @@ -795,6 +901,20 @@ def plan_extraction( ) advisories.extend(acs1_advisories) + if decennial: + advisories.append(DISCLOSURE_AVOIDANCE_ADVISORY) + if geo_level == "block": + advisories.append(BLOCK_ADVISORY) + advisories.append( + "Single census release: cross-release concordance and continuity checks do not apply; the " + "concordance file only maps this release's codes to labels. Do not splice 2020 Census counts " + "into an ACS series as if they were ACS estimates." + ) + if options.compute_reliability: + advisories.append( + "compute_reliability: no SE, CV or reliability columns are added for 2020 Census counts " + "(not sample-based; sampling MOE undefined, never 0)." + ) geography_availability: Dict[str, Any] = {} if product == PRODUCT_1YR: geography_availability = probe_acs1_geographies(client, dataset, years, entities) @@ -817,7 +937,7 @@ def plan_extraction( ) block_group_availability: Dict[str, Any] = {} - if geo_level == BLOCK_GROUP: + if geo_level == BLOCK_GROUP and not decennial: block_group_availability = probe_block_group_availability(client, table_code, years, entities) missing = block_group_availability["unpublished_years"] if missing: @@ -887,7 +1007,8 @@ def plan_extraction( "would_block": would_block, "block_reason": block_reason, "advisories": advisories, - "errata": _errata_summary(table_code, years), + "uncertainty": uncertainty_record(geo_level) if decennial else None, + "errata": [] if decennial else _errata_summary(table_code, years), "overrides": override_report, "continuity": { "checked_concepts": len(continuity_report), @@ -1308,6 +1429,10 @@ def run_batch( columns_out = list(panel.columns) slug_map = make_slug_map(columns_out, label_metadata, options.slug_style) panel = panel.rename(columns=slug_map) + if is_decennial(dataset): # explicit evidence for census.analysis (kept in Parquet files) + panel.attrs.update(product=PRODUCT_DECENNIAL, dataset=dataset, variable_codes={ + slug: col for col, slug in slug_map.items() + if (parse_variable_code(col) or ("", ""))[1] == "N"}) if duplicates_dropped: emit(f"Dropped {duplicates_dropped} duplicate (GEOID, Year) row(s).") ext = "parquet" if output_format == "parquet" else "csv" @@ -1343,6 +1468,7 @@ def run_batch( slug_style=options.slug_style, price_index=INDEX_DISPLAY.get(options.price_index, options.price_index), concordance=conc if use_concordance else None, + decennial=is_decennial(dataset), ) codebook_path = os.path.join(output_dir, f"{table_code}_codebook.csv") codebook.to_csv(codebook_path, index=False, encoding="utf-8") @@ -1404,7 +1530,9 @@ def run_batch( "rules_applied": getattr(conc, "rules_applied", []), "metadata_vintages": _vintage_provenance(conc), }, - "moe_confidence_level": 0.90, + # 2020 Census counts are not sample-based: no MOE confidence level (never 0 MOEs). + "moe_confidence_level": None if is_decennial(dataset) else 0.90, + "uncertainty": plan.get("uncertainty"), "advisories": plan["advisories"], "successful_calls": successful_calls, "failed_calls": failed_calls, diff --git a/census/recipes.py b/census/recipes.py index 2192b24..a9c9183 100644 --- a/census/recipes.py +++ b/census/recipes.py @@ -14,10 +14,12 @@ "regression_sampling_error": "Regression whose SEs include ACS sampling error", "small_area": "Small-area (Fay-Herriot) estimates for noisy tracts or counties", "preregistered": "Pre-registered (manifest) confirmatory analysis", + "decennial_counts": "Use 2020 Census counts (P.L. 94-171, DHC) without inventing margins of error", } RecipeName = Literal[ - "compare_counties", "tract_trend", "regression_sampling_error", "small_area", "preregistered" + "compare_counties", "tract_trend", "regression_sampling_error", "small_area", "preregistered", + "decennial_counts", ] @@ -26,7 +28,7 @@ def get_recipe(name: Optional[RecipeName] = None) -> Dict[str, Any]: Args: name: compare_counties, tract_trend, regression_sampling_error, - small_area, or preregistered. + small_area, preregistered, or decennial_counts. Example: get_recipe(name="compare_counties") """ diff --git a/census/simulation.py b/census/simulation.py index bd67188..6fa52b6 100644 --- a/census/simulation.py +++ b/census/simulation.py @@ -408,8 +408,10 @@ def fast_config(**overrides: Any) -> SimConfig: # output is unchanged; use spatial_config() to run them, e.g. # run_study(spatial_config(fast=True), studies=list(SPATIAL_STUDIES)). -SPATIAL_REPS_FULL = {"spatial_fay_herriot": 1000, "morans_i_replicate_se": 2000} -SPATIAL_REPS_FAST = {"spatial_fay_herriot": 40, "morans_i_replicate_se": 150} +SPATIAL_REPS_FULL = {"spatial_fay_herriot": 1000, "morans_i_replicate_se": 2000, + "spatial_regression_coverage": 1000, "local_morans_size": 1000} +SPATIAL_REPS_FAST = {"spatial_fay_herriot": 40, "morans_i_replicate_se": 150, + "spatial_regression_coverage": 6, "local_morans_size": 30} SAR_RHO = 0.5 # SAR(1) autocorrelation of area effects (spatial FH study) MORAN_RHO = 0.6 # SAR(1) autocorrelation of the true-value surface (Moran study) MORAN_SIGNAL_SD = 15.0 # SD of the SAR innovations (true-value surface, mean 50) @@ -539,9 +541,260 @@ def study_morans_i_replicate_se(cfg: SimConfig, reps: int) -> List[Dict[str, Any covered_true, {"attenuation": extra["attenuation"]}, gated=False)] +# Round 15 (PLAN_ROUND_15 workstream B, decision D15-1): info rows only, never gated. +SPREG_SIDE = 12 # 12 x 12 rook lattice (144 areas, close to cfg.n_areas) +SPREG_RHOS = (0.0, 0.3, 0.6) # rho (lag DGP) / lambda (error DGP) +SPREG_SECONDARY_SHARE = 0.5 # share of reps for the reference (no sampling error) and noisy-x cells +SPREG_NOISY_X_RHO = 0.3 +LM_SIZE_ALPHA = 0.05 # level of lm_diagnostics' decision rule +LISA_SIDE = 12 +LISA_MU = 50.0 # mean of the true values +LISA_TAU = 10.0 # SD of the independent true values +LISA_BLOCK = 4 # clustered alternative: a 4 x 4 hot spot ... +LISA_SHIFT = 1.5 * LISA_TAU # ... raised by 1.5 true-value SDs + + +def _rate_row(method: str, target: str, rates: Sequence[float], nominal_rate: float, + extra: Optional[Dict[str, Any]] = None) -> Dict[str, Any]: + """Info row for an error rate: coverage = 1 - mean rate, size = mean rate. + + ``rates``: one rate per repetition (0/1 for a family-wise event, a share + for a per-area rate); the MC SE is the between-repetition SE, since areas + in one repetition share a map. + """ + v = np.asarray(rates, float) + r = int(v.size) + rate = float(v.mean()) if r else float("nan") + row = {"method": method, "target": target, "reps": r, "nominal": 1 - nominal_rate, "coverage": 1 - rate, + "mc_se": float(v.std(ddof=1) / math.sqrt(r)) if r > 1 else float("nan"), "status": "info", + "size": rate, "nominal_size": nominal_rate} + row.update(extra or {}) + return row + + +def study_spatial_regression_coverage(cfg: SimConfig, reps: int, + rhos: Sequence[float] = SPREG_RHOS) -> List[Dict[str, Any]]: + """Default intervals of census.spatial.spatial_regression under ACS-like outcome error (info only). + + 12 x 12 rook lattice (row-standardized). Each repetition draws + x ~ N(cfg.x_mean, cfg.x_sd^2) and e ~ N(0, cfg.sigma_u^2), then the true + outcome from a spatial lag DGP, y* = (I - rho W)^-1 (10 + beta x + e), or + a spatial error DGP, y* = 10 + beta x + (I - lambda W)^-1 e, at each + rho/lambda in ``rhos``. The published outcome adds independent sampling + error with SE = tract-profile CV x the mean of y* (the outcome is + measured with error; the model has no term for it). Every estimator the + tool offers (ML and GMM; the tool default, marked "[tool default]" and + ``tool_default``, is GMM for lag and ML for error) is fitted to the same + draws through spatial_regression with its other defaults (population + target: se_model, 90% intervals, ml_method 'full'); reported: coverage of beta and of + rho/lambda, and bias. Secondary cells (SPREG_SECONDARY_SHARE of the reps): + the same DGPs without sampling error (rho > 0; separates estimator + behavior from the measurement-error effect), and a noisy covariate (x + also measured with tract-profile error) at rho = SPREG_NOISY_X_RHO. + LM diagnostics: rejection rates at 0.05 and the share of fits where the + decision rule suggests a spatial model, pooled over the two rho = 0 + cells with sampling error (no spatial dependence). Each (model, rho, + scenario) cell has its own seed, so cells do not depend on run order. + """ + from .spatial import DEFAULT_METHODS, SPATIAL_METHODS, SPATIAL_MODELS, spatial_regression + side = SPREG_SIDE + m = side * side + g = _geoids(m) + w = lattice_weights(side, g) + W = w.full()[0] + n_sec = max(int(round(reps * SPREG_SECONDARY_SHARE)), 1) + rows: List[Dict[str, Any]] = [] + lm_names = ("lm_error", "lm_lag", "robust_lm_error", "robust_lm_lag", "lm_sarma") + lm_rej: Dict[str, List[bool]] = {k: [] for k in lm_names} + lm_spatial: List[bool] = [] + scenarios = (("acs", "ACS-like outcome error"), ("none", "reference: no sampling error"), + ("noisy_x", "ACS-like outcome and covariate error")) + for mi, model in enumerate(SPATIAL_MODELS): + sp = "rho" if model == "lag" else "lambda" + for ri, rho in enumerate(rhos): + Binv = sar_inverse(W, rho) + mean_y = (10.0 + cfg.beta * cfg.x_mean) / (1 - rho) if model == "lag" else 10.0 + cfg.beta * cfg.x_mean + for si, (scen, scen_label) in enumerate(scenarios): + if (scen == "none" and rho == 0) or (scen == "noisy_x" and rho != SPREG_NOISY_X_RHO): + continue + n_reps = reps if scen == "acs" else n_sec + rng = np.random.default_rng(cfg.seed + 400 + 100 * mi + 10 * ri + si) + est = {k: {"x": [], sp: []} for k in SPATIAL_METHODS} + cov = {k: {"x": [], sp: []} for k in SPATIAL_METHODS} + failures = {k: 0 for k in SPATIAL_METHODS} + for _ in range(n_reps): + x = cfg.x_mean + cfg.x_sd * rng.standard_normal(m) + e = cfg.sigma_u * rng.standard_normal(m) + ystar = Binv @ (10.0 + cfg.beta * x + e) if model == "lag" else 10.0 + cfg.beta * x + Binv @ e + ses: Dict[str, np.ndarray] = {} + y, xo = ystar, x + if scen != "none": + ses["y"] = draw_cv(m, "tract", rng) / 100 * mean_y + y = ystar + ses["y"] * rng.standard_normal(m) + if scen == "noisy_x": + ses["x"] = draw_cv(m, "tract", rng) / 100 * cfg.x_mean + xo = x + ses["x"] * rng.standard_normal(m) + frame = acs_frame(g, {"y": y, "x": xo}, ses) + for k in SPATIAL_METHODS: + try: + res = spatial_regression("y ~ x", frame, w, model=model, method=k) + except Exception: # noqa: BLE001 - counted and reported + failures[k] += 1 + continue + tab = res.coefficients.set_index("term") + for term, truth in (("x", cfg.beta), (sp, rho)): + est[k][term].append(float(tab.loc[term, "estimate"])) + cov[k][term].append(bool(tab.loc[term, "ci_lower"] <= truth <= tab.loc[term, "ci_upper"])) + if scen == "acs" and rho == 0 and k == SPATIAL_METHODS[0]: + lm = res.diagnostics["lm_tests"] + for name in lm_names: + lm_rej[name].append(lm[name]["p_value"] < LM_SIZE_ALPHA) + lm_spatial.append(not lm["suggested_model"].startswith("ols")) + for k in SPATIAL_METHODS: + for term, truth in (("x", cfg.beta), (sp, rho)): + e_k = np.asarray(est[k][term], float) + dflt = k == DEFAULT_METHODS[model] + extra = {"model": model, "estimator": k, "rho": rho, "scenario": scen, "term": term, + "tool_default": dflt, + "bias": float(e_k.mean() - truth) if e_k.size else float("nan"), + "fit_failures": failures[k]} + if term == "x": + extra["rel_bias"] = extra["bias"] / cfg.beta + rows.append(coverage_summary( + f"spatial_regression {model} {'ML' if k == 'ml' else 'GMM'}" + f"{' [tool default]' if dflt else ''} (default intervals), " + f"{side}x{side} rook lattice, {sp}={rho}, {scen_label}", + f"{'beta' if term == 'x' else sp} ({truth})", cov[k][term], extra, gated=False)) + for name in lm_names: + rows.append(_rate_row(f"lm_diagnostics {name}, no spatial dependence, ACS-like outcome error", + f"size of the {LM_SIZE_ALPHA:g} test (coverage = 1 - size)", lm_rej[name], + LM_SIZE_ALPHA, {"term": name})) + rows.append(_rate_row("lm_diagnostics decision rule, no spatial dependence, ACS-like outcome error", + "share of fits suggesting a spatial model (coverage = 1 - share)", lm_spatial, + float("nan"), {"term": "suggested_model"})) + return rows + + +def _lisa_gradient_rank(side: int) -> np.ndarray: + """Rank (0 = lowest) of each lattice cell along the diagonal, for spatially clustered SEs.""" + r, c = np.divmod(np.arange(side * side), side) + return np.argsort(np.argsort(r + c + 1e-3 * r, kind="mergesort"), kind="mergesort") + + +def study_local_morans_size(cfg: SimConfig, reps: int) -> List[Dict[str, Any]]: + """False positives and power of census.spatial.local_morans with its defaults (info only). + + 12 x 12 rook lattice. True values are independent N(LISA_MU, LISA_TAU^2); + the published values add independent sampling error with SE = + tract-profile CV x LISA_MU. Scenarios: (a) CVs assigned at random (the + published values are exchangeable: spatial randomness holds); (b) the + same CVs sorted along the lattice diagonal (sampling noise, not the + values, is spatially clustered, like rural/urban CV gradients); + (c) clustered alternative: a LISA_BLOCK x LISA_BLOCK block of true + values raised by LISA_SHIFT, CVs at random. The permutation seed is + drawn from the cell's generator. + + Each repetition calls local_morans twice on the same map: with its + defaults (two-sided p_value = min(1, 2 p_sim), 9999 permutations, BH, + alpha 0.10; the ``default`` reading) and with the pre-Round 15 defaults + (permutations=999, sided='folded'; the ``previous_default`` reading). + Two more readings are derived from the same calls: ``folded_9999`` (BH + on the folded p_sim of the 9999-permutation call) and ``two_sided_999`` + (BH on min(1, 2 p_sim) of the 999-permutation call). + + Reported under the null (a, b): per-area rate of the uncorrected p-value + <= alpha, per-area and family-wise rate of significant areas after the + correction (under the complete null every discovery is false, so the + family-wise rate equals the FDR). Under the alternative (c): power + (share of hot-spot areas flagged significant HH) and the false discovery + proportion (discoveries outside the hot spot / discoveries, 0 when + none), averaged over repetitions. + """ + from .multitest import adjust_pvalues + from .spatial import local_morans + side = LISA_SIDE + m = side * side + g = _geoids(m) + w = lattice_weights(side, g) + r_, c_ = np.divmod(np.arange(m), side) + lo = (side - LISA_BLOCK) // 2 + block = (r_ >= lo) & (r_ < lo + LISA_BLOCK) & (c_ >= lo) & (c_ < lo + LISA_BLOCK) + grad = _lisa_gradient_rank(side) + alpha = 0.10 + # key, call, two-sided?, permutations, label + readings = (("default", "new", True, 9999, + "[tool default] two-sided p_value = min(1, 2 p_sim), 9999 permutations, BH, alpha 0.10"), + ("folded_9999", "new", False, 9999, "folded p_sim, 9999 permutations, BH, alpha 0.10"), + ("previous_default", "old", False, 999, + "[previous default] folded p_sim, 999 permutations, BH, alpha 0.10"), + ("two_sided_999", "old", True, 999, "two-sided p_value = min(1, 2 p_sim), 999 permutations, BH, " + "alpha 0.10")) + rows: List[Dict[str, Any]] = [] + scenarios = (("null_random_se", "spatial independence, ACS-like SEs at random"), + ("null_clustered_se", "spatial independence, SEs clustered along a gradient"), + ("hot_spot", f"{LISA_BLOCK}x{LISA_BLOCK} hot spot (+{LISA_SHIFT:g}), ACS-like SEs at random")) + for si, (scen, label) in enumerate(scenarios): + rng = np.random.default_rng(cfg.seed + 500 + si) + stats: Dict[str, Dict[str, List[float]]] = {k[0]: {"raw": [], "per_area": [], "fwer": [], "power": [], + "fdp": []} for k in readings} + for _ in range(reps): + theta = LISA_MU + LISA_TAU * rng.standard_normal(m) + (LISA_SHIFT * block if scen == "hot_spot" else 0.0) + cv = draw_cv(m, "tract", rng) + if scen == "null_clustered_se": + cv = np.sort(cv)[grad] + se = cv / 100 * LISA_MU + y = theta + se * rng.standard_normal(m) + frame = acs_frame(g, {"y": y}, {"y": se}) + seed = int(rng.integers(2 ** 31 - 1)) + outs = {"new": local_morans(frame, "y", w, seed=seed), + "old": local_morans(frame, "y", w, permutations=999, sided="folded", seed=seed)} + for key, call, two, _, _ in readings: + out = outs[call] + if key in ("default", "previous_default"): # the tool's own columns + p, sig = out["p_value"].to_numpy(float), out["significant"].to_numpy(bool) + else: + ps = out["p_sim"].to_numpy(float) + p = np.minimum(1.0, 2 * ps) if two else ps + sig = np.asarray(adjust_pvalues(p, "bh"), float) <= alpha + st = stats[key] + st["raw"].append(float(np.mean(p <= alpha))) + st["per_area"].append(float(np.mean(sig))) + st["fwer"].append(float(sig.any())) + if scen == "hot_spot": + hh = out["quadrant"].to_numpy() == "HH" + st["power"].append(float(np.mean(sig[block] & hh[block]))) + st["fdp"].append(float((sig & ~block).sum() / max(int(sig.sum()), 1))) + for key, _, two, perms, how in readings: + st = stats[key] + meth = f"local_morans {how}, {side}x{side} rook lattice, {label}" + extra = {"scenario": scen, "reading": key, "permutations": perms, "tool_default": key == "default"} + if scen == "hot_spot": + pw = np.asarray(st["power"]) + rows.append({"method": meth, "target": f"power: share of the {int(block.sum())} hot-spot areas " + "flagged significant HH (coverage column = power)", "reps": int(pw.size), + "nominal": float("nan"), "coverage": float(pw.mean()), + "mc_se": float(pw.std(ddof=1) / math.sqrt(pw.size)) if pw.size > 1 else float("nan"), + "status": "info", "power": float(pw.mean()), **extra}) + rows.append(_rate_row(meth, "FDR: discoveries outside the hot spot / discoveries (coverage = 1 - FDR)", + st["fdp"], alpha, {**extra, "fdr": float(np.mean(st["fdp"]))})) + continue + pname = "two-sided p_value" if two else "folded p_sim" + rows += [ + _rate_row(meth, f"per-area false-positive rate, uncorrected {pname} <= 0.10 (coverage = 1 - rate)", + st["raw"], alpha, extra), + _rate_row(meth, "per-area false-positive rate after correction (coverage = 1 - rate)", + st["per_area"], alpha, extra), + _rate_row(meth, "family-wise error rate = FDR under the complete null (coverage = 1 - rate)", + st["fwer"], alpha, extra), + ] + return rows + + SPATIAL_STUDIES: Dict[str, Callable[[SimConfig, int], List[Dict[str, Any]]]] = { "spatial_fay_herriot": study_spatial_fay_herriot, "morans_i_replicate_se": study_morans_i_replicate_se, + "spatial_regression_coverage": study_spatial_regression_coverage, # info only (D15-1) + "local_morans_size": study_local_morans_size, # info only (D15-1) } STUDIES.update(SPATIAL_STUDIES) diff --git a/census/spatial.py b/census/spatial.py index ae4647b..c4fffdc 100644 --- a/census/spatial.py +++ b/census/spatial.py @@ -18,10 +18,18 @@ * :func:`local_morans` - LISA (Anselin 1995) with conditional permutation p-values corrected for multiple comparisons (census.multitest; BH by default), plus the share of replicate maps that keep each area's quadrant. + Default: two-sided p-values (2 x esda's folded p_sim) from 9999 + permutations. Simulation ('local_morans_size'): per-area and family-wise + rates at nominal under randomness; little power on ~150 areas; spatially + patterned sampling error creates spurious clusters. * :func:`spatial_regression` - spatial lag / spatial error models by ML or GMM (spreg), returned as :class:`census.results.ModelResult` with the OLS Lagrange-multiplier diagnostics that suggest which model fits, and replicate - refits (80) for ACS sampling SEs. + refits (80) for ACS sampling SEs. Default estimator: GMM for the lag model, + ML for the error model. Simulation ('spatial_regression_coverage'): under + ACS outcome error ML lag intervals for rho fail (GMM lag holds), lambda is + attenuated by both estimators, beta intervals hold; LM tests keep their + 0.05 size. References: Moran (1950) Biometrika 37:17-23; Anselin (1995) Geographical Analysis 27:93-115; Anselin, Bera, Florax & Yoon (1996) Regional Science and @@ -29,8 +37,10 @@ PySAL; ACS Variance Replicate Estimate Tables documentation. """ +import builtins import hashlib import math +import sys import os import warnings from dataclasses import dataclass, field @@ -47,6 +57,46 @@ WEIGHT_KINDS = ("queen", "rook", "knn", "distance") SPATIAL_MODELS = ("lag", "error") SPATIAL_METHODS = ("ml", "gm") +# Default estimator per model (PLAN_ROUND_15 B, owner decision after the +# 'spatial_regression_coverage' study): GMM for the lag model (ML lag rho +# intervals fail under ACS outcome error, GMM's hold); ML for the error model +# (lambda is attenuated by both estimators and GMM is not better: coverage +# 0.558 vs 0.581 at lambda 0.3, 0.162 vs 0.192 at 0.6; beta alike). +DEFAULT_METHODS = {"lag": "gm", "error": "ml"} +LISA_PERMUTATIONS = 9999 +LISA_SIDED = ("two", "folded") + +# Measured caveats (PLAN_ROUND_15 workstream B; census.simulation studies +# 'spatial_regression_coverage' and 'local_morans_size', full configuration, +# 12 x 12 rook lattice, tract-profile ACS CVs, nominal 90% / 0.10). Reported, +# not tuned. +LAG_ML_CAVEAT = ( + "Caveat (simulation 'spatial_regression_coverage'): with ACS sampling error in the outcome, ML lag " + "intervals for rho covered 0.48 (true rho 0.3) and 0.003 (rho 0.6) at nominal 0.90; rho is biased toward " + "0 (-0.16, -0.41) because the outcome's sampling error enters W y. GMM (method='gm', spatial 2SLS, the " + "default for the lag model) covered 0.90 and 0.89 for rho. beta intervals covered 0.90 (rho 0.3) " + "and 0.84-0.85 (rho 0.6) for both estimators") +LAG_GM_NOTE = ( + "GMM lag (spatial 2SLS, the default for the lag model): simulation 'spatial_regression_coverage' found rho " + "intervals near nominal (0.89-0.90 at rho 0, 0.3, 0.6) under ACS outcome error, where ML lag fails " + "(0.48 and 0.003 at rho 0.3 and 0.6); beta intervals covered 0.89-0.90, but 0.84 at rho 0.6") +ERROR_CAVEAT = ( + "Caveat (simulation 'spatial_regression_coverage'): with ACS sampling error in the outcome, lambda describes " + "the spatial correlation of the observed residual, sampling noise included, and understates the true lambda " + "(bias -0.16 at 0.3, -0.26 at 0.6; 90% intervals covered 0.56-0.58 and 0.16-0.19, ML and GMM alike); " + "beta intervals covered 0.88-0.91") +NOISY_COVARIATE_CAVEAT = ( + "Covariates with sampling error attenuate beta (simulation: bias about -20%, 90% interval coverage " + "0.19-0.40); spatial_regression has no measurement-error correction") +LOCAL_MORAN_CAVEAT = ( + "Caveat (simulation 'local_morans_size', 144-area lattice): the default reading (two-sided p_value = " + "min(1, 2 p_sim), 9999 permutations, BH at alpha 0.10) kept the per-area rate of p_value <= 0.10 at " + "0.099 and the family-wise rate (= FDR) at 0.088 under spatial randomness. It has little power: 1.1% of " + "areas in a 4x4 hot spot raised by 1.5 SD were flagged, so no significant cluster is not evidence of no " + "clustering. Spatially patterned sampling error (CVs varying smoothly across the map) produced spurious " + "clusters in 38% of maps; check replicate_quadrant_share or rerun on Fay-Herriot estimates. p_sim is " + "esda's folded (one-sided, observed-direction) pseudo p-value; sided='folded' with permutations=999 is " + "the pre-Round 15 default (per-area rate of p_sim <= 0.10: 0.196)") _CACHE: Dict[str, Any] = {} @@ -366,9 +416,10 @@ def local_morans( var: str, weights, *, - permutations: int = 999, + permutations: int = LISA_PERMUTATIONS, correction: str = "bh", alpha: float = 0.10, + sided: str = "two", replicates: Optional[pd.DataFrame] = None, geoid_col: str = "GEOID", seed: Optional[int] = 12345, @@ -376,16 +427,30 @@ def local_morans( """Local Moran's I (LISA) with multiple-comparison-corrected permutation p-values. Returns one row per area: GEOID, Ii, quadrant (HH/LH/LL/HL), p_sim - (conditional permutation, esda), p_adjusted (census.multitest - ``correction``: bh default, by, holm, bonferroni), significant + (conditional permutation, esda: folded, i.e. one-sided in the observed + direction), p_value (``sided='two'``, the default: min(1, 2 p_sim); + ``sided='folded'``: p_sim), p_adjusted (p_value adjusted with + census.multitest ``correction``: bh default, by, holm, bonferroni), significant (p_adjusted <= alpha), cluster (quadrant if significant else 'ns') and, with replicates, ``replicate_quadrant_share``: the share of the 80 replicate maps on which the area keeps its quadrant (a sampling-stability check; low values mean the label may be noise). ``df.attrs`` holds notes. + + Defaults (PLAN_ROUND_15 B, owner decision after the 'local_morans_size' + study; :data:`LOCAL_MORAN_CAVEAT` is added to the notes): two-sided + p-values and 9999 permutations, so the smallest p_value (0.0002) can pass + BH's first threshold alpha/n up to n = 500. Under spatial randomness the + per-area rate of p_value <= 0.10 and the family-wise rate are at nominal + (the folded 999-permutation reading used before: per-area 0.196, and a + family-wise rate of 0.056 only because of its coarse p-value grid). + Power is low on ~150 areas, and spatially patterned sampling error + produces spurious clusters. About 0.6 s for 3,000 areas. """ from esda.moran import Moran_Local from .multitest import adjust_pvalues + if sided not in LISA_SIDED: + raise ValueError(f"sided must be one of {LISA_SIDED}") d = _aligned(data, var, weights, geoid_col) y = d[var].to_numpy(float) with warnings.catch_warnings(): @@ -393,16 +458,19 @@ def local_morans( lm = Moran_Local(y, weights, transformation="r", permutations=int(permutations), seed=seed, keep_simulations=False) p = np.asarray(lm.p_sim, float) - padj = np.asarray(adjust_pvalues(p, correction), float) + pv = np.minimum(1.0, 2 * p) if sided == "two" else p + padj = np.asarray(adjust_pvalues(pv, correction), float) quad = pd.Series(lm.q).map(QUADRANTS).to_numpy() sig = padj <= alpha out = pd.DataFrame({ geoid_col: list(d.index), "Ii": np.asarray(lm.Is, float), "quadrant": quad, - "p_sim": p, "p_adjusted": padj, "significant": sig, + "p_sim": p, "p_value": pv, "p_adjusted": padj, "significant": sig, "cluster": np.where(sig, quad, "ns"), }) - notes = [f"permutation p-values ({int(permutations)} conditional permutations, minimum attainable " - f"{1 / (permutations + 1):.4g}) adjusted with '{correction}' across {len(y)} local tests", + notes = [f"{'two-sided p_value = min(1, 2 p_sim)' if sided == 'two' else 'folded (one-sided) p_value = p_sim'}" + f" from {int(permutations)} conditional permutations (minimum attainable p_sim " + f"{1 / (permutations + 1):.4g}), adjusted with '{correction}' across {len(y)} local tests, " + f"significant at p_adjusted <= {alpha:g}", "islands (no neighbors) have Ii = 0 and are never significant"] if permutations else [] if replicates is not None and len(replicates): R = _replicate_matrix(replicates, var, list(d.index), geoid_col, _release_year(data)) @@ -413,6 +481,7 @@ def local_morans( q = np.where(Z > 0, np.where(L > 0, 1, 4), np.where(L > 0, 2, 3)) out["replicate_quadrant_share"] = (q == np.asarray(lm.q)[:, None]).mean(axis=1) notes.append("replicate_quadrant_share: share of the 80 ACS replicate maps with the same quadrant") + notes.append(LOCAL_MORAN_CAVEAT) out.attrs["notes"] = notes return out @@ -440,9 +509,31 @@ def _scalar(x) -> float: return float(np.asarray(x, dtype=float).reshape(-1)[0]) +def _spreg_print(*args: Any, **kwargs: Any) -> None: + """``print`` for the spreg.output module: drops what would reach stdout, forwards other files.""" + target = kwargs.get("file") + if target is not None and target is not sys.stdout and target is not sys.__stdout__: + builtins.print(*args, **kwargs) + + +def _silence_spreg_output() -> None: + """spreg 1.9 prints the estimator's class name (``print(_reg.__class__.__name__)`` in + ``spreg.output.out_part_top``) on every ML/GMM fit, and stdout carries the MCP stdio protocol. + Shadow ``print`` in that module only, once: no process-wide stdout redirection, so concurrent + fits and other threads' output are unaffected (contextlib.redirect_stdout swaps sys.stdout for + every thread).""" + import importlib + + # the module, not the spreg.output function that spreg/__init__ re-exports under the same name + spreg_output = importlib.import_module("spreg.output") + if spreg_output.__dict__.get("print") is not _spreg_print: + spreg_output.print = _spreg_print # type: ignore[attr-defined] + + def _spreg_fit(y, X, w, model: str, method: str, names: List[str], yname: str, ml_method: str, kwargs): import spreg + _silence_spreg_output() with warnings.catch_warnings(): warnings.simplefilter("ignore") if model == "lag" and method == "ml": @@ -502,7 +593,7 @@ def spatial_regression( weights, *, model: str = "lag", - method: str = "ml", + method: Optional[str] = None, replicates: Optional[pd.DataFrame] = None, target: str = "population", confidence_level: float = 0.90, @@ -517,7 +608,9 @@ def spatial_regression( ``model``: ``lag`` (y = rho W y + X b + e) or ``error`` (u = lambda W u + e). ``method``: ``ml`` (ML_Lag / ML_Error, ``ml_method`` full/ord/LU) or ``gm`` (GM_Lag: spatial 2SLS with W X, W^2 X instruments; GM_Error_Het: - Kelejian-Prucha GMM robust to heteroskedasticity). Coefficients include + Kelejian-Prucha GMM robust to heteroskedasticity). Default + (:data:`DEFAULT_METHODS`): ``gm`` for the lag model, ``ml`` for the error + model. Coefficients include the spatial parameter as term ``rho`` or ``lambda``. ``diagnostics`` carries the OLS LM tests (``lm_tests``) with a suggested model. @@ -526,7 +619,15 @@ def spatial_regression( sqrt(4/80 sum (b_r - b)^2) as in census.models.fit_replicate, and ``target`` (population / these_areas / conservative) picks the SE for intervals. ML refits cost O(n^3) each with ml_method='full'; use - ml_method='LU' or method='gm' for large n. ``replicate_policy``: "flag" + ml_method='LU' or method='gm' for large n. + + Measured behavior (census.simulation 'spatial_regression_coverage', added + to the notes): with ACS sampling error in the outcome the ML lag + estimator's rho is biased toward 0 and its intervals under-cover badly + (0.48 at rho 0.3, 0.003 at 0.6); GMM lag, the default since Round 15, + covers rho near nominal (0.89-0.90). In the error model lambda is + attenuated by either estimator (ML, the default, is not worse), while + beta intervals stay near nominal. ``replicate_policy``: "flag" (default) records failed refits or stochastic variables without replicates in diagnostics["replicates_incomplete"]; "strict" raises ValueError (census.results.replicate_completeness, PLAN_ROUND_11 Q2/D7). @@ -536,9 +637,10 @@ def spatial_regression( check_replicate_policy(replicate_policy) - model, method = model.lower(), method.lower() + model = model.lower() if model not in SPATIAL_MODELS: raise ValueError(f"model must be one of {SPATIAL_MODELS}") + method = DEFAULT_METHODS[model] if method is None else method.lower() if method not in SPATIAL_METHODS: raise ValueError(f"method must be one of {SPATIAL_METHODS}") if target not in INTERVAL_TARGETS: @@ -610,6 +712,8 @@ def spatial_regression( level = confidence_level if confidence_level <= 1 else confidence_level / 100 table = _table(params, se_model, se_sampling, level, target) notes.append(f"intervals and p-values use the '{target}' target") + notes.append(ERROR_CAVEAT if model == "error" else LAG_ML_CAVEAT if method == "ml" else LAG_GM_NOTE) + notes.append(NOISY_COVARIATE_CAVEAT) return ModelResult( model_type=f"spatial_{model}", formula=formula, n_obs=int(len(y)), coefficients=table, method=est_method, uncertainty_sources=sources, confidence_level=level, notes=notes, diff --git a/census/stats.py b/census/stats.py index e346df5..6ecdcca 100644 --- a/census/stats.py +++ b/census/stats.py @@ -72,11 +72,12 @@ def moe_to_se(moe: Any, z: Optional[float] = None) -> np.ndarray: # Variable code anatomy:
[_C| _]_ # B19013_001E, B01001A_003M, DP03_0062PE, S1701_C01_001E, B19013_001EA, -# CP03_2017_001E, CP03_2022to2017_001SS (comparison-profile significance) +# CP03_2017_001E, CP03_2022to2017_001SS (comparison-profile significance), +# P1_001N, P12_002NA (2020 Census count and its annotation; census.decennial) # Optional "___" marks a concept segment (a reused code split by year). BASE_PATTERN = r"[A-Z]+\d+[A-Z]*(?:_C\d+|_\d{4}(?:to\d{4})?)?_\d+(?:__\d{4}_\d{4})?" VARIABLE_CODE_RE = re.compile( - rf"^(?P{BASE_PATTERN})(?PPEA|PMA|PE|PM|EA|MA|SS|E|M)$" + rf"^(?P{BASE_PATTERN})(?PPEA|PMA|PE|PM|EA|MA|NA|SS|E|M|N)$" ) MEASUREMENT_TYPES: Dict[str, str] = { @@ -89,8 +90,14 @@ def moe_to_se(moe: Any, z: Optional[float] = None) -> np.ndarray: "PEA": "Annotation", "PMA": "Annotation", "SS": "Significance", + # 2020 Decennial Census: a published count, not sample-based (no MOE pair) + "N": "Count", + "NA": "Annotation", } +# Suffixes that carry a variable's primary value (labels, concordance keys) +PRIMARY_SUFFIXES: Tuple[str, ...] = ("E", "PE", "SS", "N") + # Estimate suffix -> paired MOE suffix ESTIMATE_MOE_PAIRS: Dict[str, str] = {"E": "M", "PE": "PM"} @@ -178,8 +185,16 @@ def reliability_flag(cv: float) -> str: return RELIABILITY_LOW -def calculate_cv(estimate: float, moe: float) -> Dict[str, object]: - """Return SE, CV and reliability flag for a single estimate/MOE pair.""" +def calculate_cv(estimate: float, moe: float, product: Optional[str] = None) -> Dict[str, object]: + """Return SE, CV and reliability flag for a single estimate/MOE pair. + + ``product='decennial'`` (or a 'dec/pl' / 'dec/dhc' key) raises + ``ValueError``: a census count has no sampling MOE, so its CV is undefined + (census.decennial.SAMPLING_REFUSAL). + """ + from census.decennial import refuse_sampling + + refuse_sampling(product) se = standard_error(moe) cv = coefficient_of_variation(estimate, moe) # Interval half-width from the decoded SE: 0 for a controlled estimate, @@ -255,6 +270,7 @@ def acs_overlap_years( """ from census.api import PRODUCT_1YR, require_same_product # local: keep stats light + # decennial products raise here: census counts have no sample periods to overlap product = require_same_product(product1, product2 if product2 is not None else product1) if product == PRODUCT_1YR: return 1 if int(year1) == int(year2) else 0 @@ -283,12 +299,39 @@ def z_test( share C = overlap_years / period uses the product's period length (5 or 1), so a same-release 1-year comparison (overlap 1 of 1) is treated as independent (factor 1.0), not as 1 of 5 shared years. + + 2020 Census counts (product 'decennial' or a 'dec/pl' / 'dec/dhc' key) are + not sample-based: two decennial values raise ``ValueError`` + (census.decennial.SAMPLING_REFUSAL). One ACS estimate against one decennial + count is tested with the decennial count as a fixed reference (its MOE + must be None, NaN or 0; SE 0): the result says so in ``notes`` and + ``comparison`` and measures ACS sampling error only. """ + from census.decennial import MIXED_NOTE, SAMPLING_REFUSAL, is_decennial + period = 5 if product is not None: product1 = product if product1 is None else product1 product2 = product if product2 is None else product2 - if product1 is not None or product2 is not None: + dec1, dec2 = is_decennial(product1), is_decennial(product2) + if dec1 and dec2: + raise ValueError(SAMPLING_REFUSAL) + if dec1 or dec2: + from census.api import PRODUCT_PERIOD_YEARS, require_same_product + + period = PRODUCT_PERIOD_YEARS[require_same_product(product2 if dec1 else product1)] + if overlap_years: + raise ValueError("overlap_years does not apply to an ACS vs decennial comparison; pass 0.") + ref_moe = moe1 if dec1 else moe2 + if ref_moe is not None and not _isnan(ref_moe) and float(ref_moe) != 0: + raise ValueError( + "The decennial value has no sampling margin of error; pass its MOE as None (or 0). " + MIXED_NOTE + ) + if dec1: + moe1 = 0.0 + else: + moe2 = 0.0 + elif product1 is not None or product2 is not None: from census.api import PRODUCT_PERIOD_YEARS, require_same_product period = PRODUCT_PERIOD_YEARS[require_same_product(product1, product2)] @@ -312,6 +355,9 @@ def _clean_moe(moe: float, which: str) -> float: return float(moe) m1, m2 = _clean_moe(moe1, "first"), _clean_moe(moe2, "second") + comparison = "acs_vs_decennial_reference" if (dec1 or dec2) else "acs_vs_acs" + if dec1 or dec2: + notes.append(MIXED_NOTE) diff = float(est1) - float(est2) if any(_isnan(v) for v in (est1, est2)) or any( int(float(v)) in CENSUS_SENTINELS for v in (est1, est2) if not _isnan(v) @@ -329,6 +375,7 @@ def _clean_moe(moe: float, which: str) -> float: "significant": None, "significant_at": {f"{int(lv * 100)}%": None for lv in Z_CRITICAL}, "overlap_years": overlap_years, + "comparison": comparison, "notes": notes, "verdict": "Not tested: " + "; ".join(notes) + ".", } @@ -355,11 +402,16 @@ def _clean_moe(moe: float, which: str) -> float: f"{int(lv * 100)}%": bool(z >= zc) for lv, zc in Z_CRITICAL.items() }, "overlap_years": overlap_years, + "comparison": comparison, "notes": notes, "verdict": ( - f"Statistically significant difference at the {label} confidence level." - if significant - else f"No statistically significant difference at the {label} confidence level." + ("ACS estimate vs decennial count as a fixed reference (ACS sampling error only): " + if comparison != "acs_vs_acs" else "") + + ( + f"Statistically significant difference at the {label} confidence level." + if significant + else f"No statistically significant difference at the {label} confidence level." + ) ), } diff --git a/census/validation.py b/census/validation.py index 94c3e97..ceddfd6 100644 --- a/census/validation.py +++ b/census/validation.py @@ -19,6 +19,10 @@ F8 sum of counties (aggregate.aggregate_geographies) vs state totals F9 Round 6 national continuity check (from the F1 extraction) F10 table-based Summary File cells (www2.census.gov) == API cells +F11 2020 Census invariants and hierarchy: state population (P.L. 94-171, + DHC) == published apportionment-release state totals; block housing + units sum exactly to tract and county totals; DHC == P.L. 94-171 for + the items both publish (tract, county) ====== ================================================================== Data come from a :class:`CensusSource`. Offline (the default) it reads frozen @@ -36,6 +40,7 @@ import datetime as _dt import gzip import hashlib +import io import json import math import os @@ -56,6 +61,7 @@ from census import medians as _medians from census import metadata as _md from census import replicate as _replicate +from census import decennial as _dec from census.api import BASE_URL, DATASETS, CensusAPIClient from census.stats import CENSUS_SENTINELS, acs5_overlap_years, parse_variable_code, z_test @@ -64,7 +70,7 @@ DEFAULT_REPORT_DIR = os.path.join(REPO_ROOT, "docs", "audits") REPORT_BASENAME = "validation_report" -CHECK_IDS: Tuple[str, ...] = ("F1", "F2", "F3", "F4", "F5", "F6", "F7", "F8", "F9", "F10") +CHECK_IDS: Tuple[str, ...] = ("F1", "F2", "F3", "F4", "F5", "F6", "F7", "F8", "F9", "F10", "F11") # Census API: at most 50 variables per request (NAME counts). MAX_GET_VARIABLES = 48 @@ -88,6 +94,11 @@ "2023_ACSSubjectDefinitions.pdf" ) CP_USER_NOTES = "https://www.census.gov/programs-surveys/acs/guidance/comparing-acs-data.html" +# 2020 Census Apportionment Results, Table 2: resident population of the 50 states, +# DC and Puerto Rico (released 2021-04-26, before the P.L. 94-171 and DHC files). +APPORTIONMENT_TABLE2_URL = ( + "https://www2.census.gov/programs-surveys/decennial/2020/data/apportionment/apportionment-2020-table02.xlsx" +) CPI_GUIDANCE = ( "https://www.census.gov/topics/income-poverty/income/guidance/current-vs-constant-dollars.html" ) @@ -127,6 +138,10 @@ class Battery: f4_county_constructions: Tuple[str, ...] = ("owner_occupied",) sf_release: int = 2023 sf_tables: Tuple[str, ...] = ("B01003", "B25003", "B17001", "B19001") + # F11 (2020 Census): Michigan, New York, Wyoming; blocks and tracts in one + # county each (Keweenaw MI, Hamilton NY, Albany WY) to keep fixtures small. + f11_states: Tuple[str, ...] = ("26", "36", "56") + f11_block_counties: Tuple[Tuple[str, str], ...] = (("26", "083"), ("36", "041"), ("56", "001")) def state_list(self) -> str: return ",".join(self.states) @@ -657,6 +672,26 @@ def vre_frame( self._memo[key] = frame return frame + def www2_file(self, name: str, url: str, kind: str) -> bytes: + """A small census.gov file, byte for byte (fixture ``name`` offline; recorded whole).""" + if url not in self.urls: + self.urls.append(url) + if not self.live: + return self.read_fixture(name) + import requests # local: keeps the module importable offline without network deps + + self._www2_wait() + resp = requests.get(url, headers={"User-Agent": USER_AGENT}, timeout=120) + body = resp.content or b"" + if b"Request Rejected" in body[:2000]: + raise SourceError(f"census.gov rejected the request for {url} (firewall page); retry later.") + if resp.status_code != 200: + raise SourceError(f"HTTP {resp.status_code} for {url}") + if self.record: + self.write_fixture(name, body, {"kind": kind, "source_url": url, + "transformation": "none (file stored byte for byte)"}) + return body + def summary_file_rows(self, year: int, table: str, geo_ids: Sequence[str]) -> Dict[str, Any]: """Header and rows (by GEO_ID) of a table-based Summary File, read by byte range. @@ -2064,6 +2099,159 @@ def check_f10_summary_file( return res.finish(n_failed == 0 and n > 0) +# --------------------------------------------------------------------------- +# F11: 2020 Census invariants and hierarchy +# --------------------------------------------------------------------------- + +# Items both 2020 Census files publish: (P.L. 94-171 variable, DHC variable, item). +F11_SHARED_ITEMS: Tuple[Tuple[str, str, str], ...] = ( + ("P1_001N", "P1_001N", "total population"), + ("H1_001N", "H1_001N", "housing units (block invariant)"), + ("H1_001N", "H3_001N", "housing units, DHC occupancy-status table"), + ("H1_002N", "H3_002N", "occupied housing units"), + ("H1_003N", "H3_003N", "vacant housing units"), +) + + +def _apportionment_state_totals(raw: bytes) -> Dict[str, int]: + """State FIPS -> resident population from 2020 Apportionment Results Table 2 (xlsx).""" + from census.geo import STATE_MAPPING + + by_name = {v.lower(): k for k, v in STATE_MAPPING.items()} + table = pd.read_excel(io.BytesIO(raw), header=None, dtype=object) + out: Dict[str, int] = {} + for _, row in table.iterrows(): + name, value = str(row.iloc[0]).strip().lower(), row.iloc[1] + if name in by_name and isinstance(value, (int, float, np.integer)) and not pd.isna(value): + out[by_name[name]] = int(value) + return out + + +def _f11_frame(source: CensusSource, dataset: str, variables: Sequence[str], geo: Mapping[str, str]) -> pd.DataFrame: + frame = source.api_frame(_dec.DECENNIAL_YEAR, dataset, variables, geo) + for v in variables: + frame[v] = pd.to_numeric(frame[v], errors="coerce") + return frame.set_index("GEOID") + + +def _f11_blocks(source: CensusSource, dataset: str, state: str, county: str) -> pd.DataFrame: + """Block housing units (``get=H1_001N`` only, no NAME: the smallest response).""" + endpoint = f"{BASE_URL}/{_dec.DECENNIAL_YEAR}/{DATASETS[dataset]}" + payload = source.get_json(endpoint, {"get": "H1_001N", "for": "block:*", "in": f"state:{state} county:{county}"}) + df = pd.DataFrame(payload[1:], columns=payload[0]) + df["H1_001N"] = pd.to_numeric(df["H1_001N"], errors="coerce") + df["tract_geoid"] = df["state"] + df["county"] + df["tract"] + df["GEOID"] = df["tract_geoid"] + df["block"] + return df + + +def check_f11_decennial_invariants( + source: CensusSource, battery: Battery = DEFAULT_BATTERY, ctx: Optional[Dict[str, Any]] = None +) -> ValidationResult: + b = battery + states = ",".join(b.f11_states) + counties = ", ".join(f"{s}{c}" for s, c in b.f11_block_counties) + res = ValidationResult( + "F11", + "2020 Census invariants and hierarchy (census.decennial; dec/pl and dec/dhc through the toolkit's " + f"API layer), states {states}: (a) state total population in P.L. 94-171 and DHC vs the resident " + "population published in the 2020 Apportionment Results (Table 2); (b) block housing units (H1_001N, " + f"a block invariant) summed to tracts and counties vs published tract and county totals, counties " + f"{counties}, and P.L. 94-171 vs DHC block by block; (c) DHC vs P.L. 94-171 for the items both " + f"publish ({', '.join(i[2] for i in F11_SHARED_ITEMS)}) at every county of the states and every " + "tract of those counties.", + tolerance="exact (integer equality); these are published counts with no sampling error", + ) + res.census_source += [_dec.APPORTIONMENT_URL, _dec.INVARIANTS_URL, _dec.PPMF_URL, + f"{BASE_URL}/2020/dec/pl", f"{BASE_URL}/2020/dec/dhc"] + failures: List[Dict[str, Any]] = [] + parts: Dict[str, Dict[str, int]] = {} + abs_e: List[float] = [] + + def compare(part: str, ours: float, ref: float, **where: Any) -> None: + stats = parts.setdefault(part, {"compared": 0, "failed": 0}) + stats["compared"] += 1 + diff = abs(float(ours) - float(ref)) if np.isfinite(ours) and np.isfinite(ref) else math.inf + abs_e.append(diff) + if diff != 0: + stats["failed"] += 1 + if len(failures) < MAX_FAILURE_SAMPLES: + failures.append({"part": part, **where, "value": ours, "reference": ref}) + + # (a) state totals vs the apportionment release + published = _apportionment_state_totals( + source.www2_file("dec2020_apportionment_table02.xlsx", APPORTIONMENT_TABLE2_URL, "apportionment_table") + ) + pl_vars = sorted({i[0] for i in F11_SHARED_ITEMS}) + dhc_vars = sorted({i[1] for i in F11_SHARED_ITEMS}) + pl_states = _f11_frame(source, _dec.DECENNIAL_PL, pl_vars, {"for": f"state:{states}"}) + dhc_states = _f11_frame(source, _dec.DECENNIAL_DHC, dhc_vars, {"for": f"state:{states}"}) + for st in b.f11_states: + ref = float(published.get(st, math.nan)) + compare("a_state_population", float(pl_states.at[st, "P1_001N"]), ref, geography=st, dataset="dec/pl") + compare("a_state_population", float(dhc_states.at[st, "P1_001N"]), ref, geography=st, dataset="dec/dhc") + + # (c) DHC vs P.L. 94-171, counties of the states and tracts of the block counties + geos = [("county", {"for": "county:*", "in": f"state:{states}"})] + [ + ("tract", {"for": "tract:*", "in": f"state:{s} county:{c}"}) for s, c in b.f11_block_counties + ] + tract_totals: Dict[str, Tuple[pd.DataFrame, pd.DataFrame]] = {} + county_totals: Optional[Tuple[pd.DataFrame, pd.DataFrame]] = None + for level, geo in geos: + pl = _f11_frame(source, _dec.DECENNIAL_PL, pl_vars, geo) + dhc = _f11_frame(source, _dec.DECENNIAL_DHC, dhc_vars, geo) + if sorted(pl.index) != sorted(dhc.index): + compare(f"c_{level}_geographies", len(pl.index), len(dhc.index), geography=geo.get("in", "")) + for g in pl.index.intersection(dhc.index): + for pv, dv, item in F11_SHARED_ITEMS: + compare(f"c_{level}_dhc_vs_pl", float(dhc.at[g, dv]), float(pl.at[g, pv]), + geography=g, item=f"{dv} vs {pv} ({item})") + if level == "county": + county_totals = (pl, dhc) + else: + tract_totals[geo["in"]] = (pl, dhc) + + # (b) block housing units -> tract and county totals; P.L. 94-171 vs DHC per block + block_counts: Dict[str, int] = {} + for st, co in b.f11_block_counties: + pl_blocks = _f11_blocks(source, _dec.DECENNIAL_PL, st, co) + dhc_blocks = _f11_blocks(source, _dec.DECENNIAL_DHC, st, co).set_index("GEOID") + block_counts[st + co] = len(pl_blocks) + for g, value in pl_blocks.set_index("GEOID")["H1_001N"].items(): + compare("b_block_pl_vs_dhc", float(dhc_blocks["H1_001N"].get(g, math.nan)), float(value), geography=g) + sums = pl_blocks.groupby("tract_geoid")["H1_001N"].sum() + pl_tracts, dhc_tracts = tract_totals[f"state:{st} county:{co}"] + for frame, ds in ((pl_tracts, "dec/pl"), (dhc_tracts, "dec/dhc")): + for g in frame.index.union(sums.index): + compare("b_blocks_to_tracts", float(sums.get(g, math.nan)), + float(frame["H1_001N"].get(g, math.nan)), geography=g, dataset=ds) + assert county_totals is not None + for frame, ds in zip(county_totals, ("dec/pl", "dec/dhc")): + compare("b_blocks_to_county", float(pl_blocks["H1_001N"].sum()), + float(frame.at[st + co, "H1_001N"]), geography=st + co, dataset=ds) + + res.n_compared = sum(p["compared"] for p in parts.values()) + res.n_failed = sum(p["failed"] for p in parts.values()) + res.max_abs_error = _max(abs_e) + res.failures = failures + res.details = {"parts": parts, "blocks_per_county": block_counts, + "published_state_population": {st: published.get(st) for st in b.f11_states}} + res.notes += [ + "Invariants (published without disclosure-avoidance noise): state total population, block total " + "housing units and occupied group-quarters facilities by type (Census DAS update 2020-11-25, " + f"{_dec.INVARIANTS_URL}; fact sheet {_dec.DAS_FACT_SHEET_URL}). Part (a) compares against an " + "independent publication (the April 2021 apportionment release), part (b) tests the block invariant " + "and the tabulation hierarchy.", + "Part (c) items other than housing units are not invariants: they agree because every 2020 product " + "is tabulated from the same privacy-protected microdata (\"Tabulations from the PPMF are entirely " + f"consistent with tables from these data products\", {_dec.PPMF_URL}). Agreement shows the toolkit " + "reads both files consistently; it does not show the counts are free of disclosure-avoidance noise.", + "There is no sampling error to compare: decennial counts have no MOE, so the check asserts exact " + "equality only.", + ] + return res.finish(res.n_failed == 0 and res.n_compared > 0) + + # --------------------------------------------------------------------------- # Battery and reports # --------------------------------------------------------------------------- @@ -2079,6 +2267,7 @@ def check_f10_summary_file( "F8": check_f8_county_sums, "F9": check_f9_continuity, "F10": check_f10_summary_file, + "F11": check_f11_decennial_invariants, } diff --git a/census/workflows.py b/census/workflows.py index b9ab956..d02992e 100644 --- a/census/workflows.py +++ b/census/workflows.py @@ -20,6 +20,9 @@ from census.aggregate import aggregate_geographies as _aggregate_geographies from census.api import CensusAPIClient, infer_dataset from census.cache import get_default_cache, import_frozen, read_frozen +from census.decennial import SAMPLING_REFUSAL as DECENNIAL_SAMPLING_REFUSAL +from census.decennial import UNCERTAINTY_LABEL as DECENNIAL_UNCERTAINTY_LABEL +from census.decennial import is_decennial from census.derived import difference as derive_difference from census.derived import percent_change as derive_percent_change from census.derived import product as derive_product @@ -190,10 +193,11 @@ def _derived_result(result: Any) -> Dict[str, Any]: def derive_estimate( operation: str, estimates: List[float], - moes: List[float], + moes: Optional[List[float]], as_percent: bool = False, year_current: Optional[int] = None, year_prior: Optional[int] = None, + product: Optional[str] = None, ) -> Dict[str, Any]: """Derive an estimate and its 90% MOE with the Census approximation formulas (ACS General Handbook 2020, ch. 8). @@ -209,6 +213,9 @@ def derive_estimate( as_percent: Return proportion/ratio on a 0-100 scale. year_current, year_prior: For percent_change between ACS releases (adds the overlapping-period note). + product: 'decennial' when the inputs are 2020 Census counts: returns the + point estimate only; MOE, SE and CV are undefined (None), because census + counts are not sample-based (pass moes as zeros or omit them). Returns estimate, moe, se, cv_percent, method, and notes. method records the formula used, e.g. 'proportion_ratio_fallback' when the proportion radicand @@ -220,10 +227,20 @@ def derive_estimate( op = operation.lower().strip() if op not in _DERIVE_OPS: return errors.error_result(errors.INVALID_INPUT, f"operation must be one of {_DERIVE_OPS}.") - if len(estimates) != len(moes) or not estimates: + decennial = is_decennial(product) + if decennial: + if any(m is not None and not (isinstance(m, float) and math.isnan(m)) and float(m) != 0 for m in moes or []): + return errors.error_result( + errors.INVALID_INPUT, + "2020 Census counts have no sampling margin of error; pass moes as zeros or omit them. " + + DECENNIAL_SAMPLING_REFUSAL, + ) + moes = [0.0] * len(estimates) + if moes is None or len(estimates) != len(moes) or not estimates: return errors.error_result(errors.INVALID_INPUT, "estimates and moes must be non-empty and aligned.") if op == "sum": - return _derived_result(sum_estimates(estimates, moes)) + return _decennial_point(sum_estimates(estimates, moes)) if decennial else _derived_result( + sum_estimates(estimates, moes)) if len(estimates) != 2: return errors.error_result(errors.INVALID_INPUT, f"'{op}' takes exactly two estimates.") (x, y), (mx, my) = estimates, moes @@ -237,7 +254,22 @@ def derive_estimate( res = derive_difference(x, mx, y, my) else: res = derive_percent_change(x, mx, y, my, year_current=year_current, year_prior=year_prior) - return _derived_result(res) + return _decennial_point(res) if decennial else _derived_result(res) + + +def _decennial_point(res: Any) -> Dict[str, Any]: + """A derived value from 2020 Census counts: point estimate, uncertainty undefined (never 0).""" + res.moe, res.se, res.cv_percent = math.nan, math.nan, math.nan + res.components = {k: (None if "moe" in str(k).lower() else v) for k, v in (res.components or {}).items()} + res.method = f"{res.method} (decennial counts: point estimate only)" + res.notes = [n for n in res.notes if "MOE" not in n] + [ + "2020 Census counts are not sample-based: the sampling MOE, SE and CV of this value are " + "undefined (not 0). The counts carry disclosure-avoidance noise and coverage error that " + "these formulas do not measure; aggregate small areas before deriving rates." + ] + out = _derived_result(res) + out["uncertainty"] = DECENNIAL_UNCERTAINTY_LABEL + return out @_guided( @@ -476,7 +508,9 @@ def batch_significance( correction: 'holm' (default; family-wise error), 'bonferroni', 'bh' (false discovery rate, for exploratory screening), or 'by'. product: '5-year' (default) or '1-year'. 1-year releases never share - sample, so no overlap adjustment is applied. + sample, so no overlap adjustment is applied. 'decennial' is refused: 2020 + Census counts are not sample-based (use calculate_significance with + product2='decennial' to test an ACS estimate against a census count). Testing 3,000 tracts at 90% yields about 300 false positives with no correction. Report significant_adjusted, not significant_raw. @@ -685,6 +719,7 @@ def download_table( years: Optional[List[int]] = None, variables: Optional[List[str]] = None, output_dir: Optional[str] = None, + dataset: Optional[str] = None, ) -> Dict[str, Any]: """Download one ACS 5-year table to a tidy panel (CSV) with codebook, SE/CV and reliability columns. Plans first and stops if the plan would block. @@ -698,6 +733,8 @@ def download_table( variables: Codes the analysis uses, e.g. ['B17001_001', 'B17001_002'] (default: the whole table, which blocks more often across years). output_dir: Destination folder (default downloads/). + dataset: 'dec/dhc' or 'dec/pl' for the 2020 Census (years [2020]; counts + are not sample-based, so no SE/CV columns; see `uncertainty`). Tract releases spanning 2019 and 2020 are apportioned to 2020 tracts. For any other option use batch_download (full profile). @@ -709,16 +746,18 @@ def download_table( except ValueError as exc: return errors.error_result(errors.INVALID_GEO_LEVEL, str(exc)) table = str(table_code).strip().upper() - year_list = sorted({int(y) for y in years}) if years else [_default_year(infer_dataset(table))] - options = ResearchOptions(compute_reliability=True, include_codebook=True) - if level == "tract" and year_list[0] <= 2019 < year_list[-1]: + decennial = is_decennial(dataset) + year_list = sorted({int(y) for y in years}) if years else ( + [2020] if decennial else [_default_year(infer_dataset(table))]) + options = ResearchOptions(compute_reliability=not decennial, include_codebook=True) + if level == "tract" and year_list[0] <= 2019 < year_list[-1] and not decennial: options.harmonize_tracts_to = 2020 # releases span the 2010/2020 tract change client = _client() try: entities = build_entities(level, state_code, codes, client=client, years=year_list) except ValueError as exc: return errors.error_result(errors.INVALID_GEO_LEVEL, str(exc)) - plan = plan_extraction(table, entities, year_list, "panel", options, None, variables, client) + plan = plan_extraction(table, entities, year_list, "panel", options, dataset, variables, client) if not plan.get("success"): return _jsonable(plan) if plan.get("would_block"): @@ -733,8 +772,10 @@ def download_table( "next_step": "name fewer variables (variables=[...]) or fewer years, or use check_variable_concordance", } base_dir = output_dir or os.path.join(ROOT, "downloads") - out = run_batch(table, entities, year_list, base_dir, "panel", options, None, client, variables=variables, + out = run_batch(table, entities, year_list, base_dir, "panel", options, dataset, client, variables=variables, plan=plan) + if isinstance(out, dict) and decennial: + out["uncertainty"] = plan.get("uncertainty") out = _jsonable(out) if isinstance(out, dict) and out.get("success"): out["years_requested"] = year_list diff --git a/census/workflows_geo.py b/census/workflows_geo.py index a60e710..293f02b 100644 --- a/census/workflows_geo.py +++ b/census/workflows_geo.py @@ -185,7 +185,11 @@ def spatial_autocorrelation( Args: weights: "queen" (default), "rook", or "knn". - local: Also return local clusters with multiple-comparison correction. + local: Also return local clusters with multiple-comparison correction + (two-sided p_value = 2 x p_sim from 9999 permutations, BH, + alpha 0.10). Simulation: rates at nominal under randomness, + little power on ~150 areas, and spatially patterned sampling + error creates spurious clusters (see local_notes). replicates_path: Census replicates (build_analysis_frame) for a design-based SE of I. That SE is conservative (validated coverage 0.987 for nominal 0.90). @@ -203,6 +207,7 @@ def spatial_autocorrelation( lisa = spatial.local_morans(gdf, column, w, replicates=reps) payload["local_output_path"] = _write_frame(lisa, output_path, "local_morans.csv") payload["local_preview"] = _records(lisa, 10) + payload["local_notes"] = list(lisa.attrs.get("notes", [])) except _GEO_ERRORS as exc: return _fail(errors.INVALID_INPUT, str(exc)) i_value = payload["global"].get("I") if isinstance(payload["global"], dict) else None @@ -218,17 +223,26 @@ def spatial_regression( model: str = "lag", weights: str = "queen", replicates_path: Optional[str] = None, + method: Optional[str] = None, ) -> Dict[str, Any]: """Spatial lag or spatial error regression (PySAL spreg) with Lagrange multiplier diagnostics; replicate SEs when replicates are given. + Args: + method: "gm" or "ml". Default: "gm" (GMM) for the lag model, "ml" + for the error model. Simulation with ACS sampling error in the + outcome: GMM lag intervals for rho covered 0.90 and 0.89 (rho + 0.3, 0.6) for nominal 0.90, ML lag only 0.48 and 0.003. Error + model: lambda is attenuated (either method); beta intervals + hold. See the result notes. + Example: spatial_regression("frame.parquet", "poverty_rate ~ median_rent", "tract", 2023, model="error") """ from census import spatial try: gdf, w = _weights_for(_with_boundaries(_load_frame(input_path), level, year), weights) - res = spatial.spatial_regression(formula, gdf, w, model=model, replicates=_release_replicates(_read_replicates(replicates_path), year)) + res = spatial.spatial_regression(formula, gdf, w, model=model, method=method, replicates=_release_replicates(_read_replicates(replicates_path), year)) except _GEO_ERRORS as exc: return _fail(errors.INVALID_INPUT, str(exc)) out = res.to_dict() diff --git a/census/workflows_inference.py b/census/workflows_inference.py index 93b8c3b..6637eaa 100644 --- a/census/workflows_inference.py +++ b/census/workflows_inference.py @@ -702,6 +702,10 @@ def create_manifest(manifest_path: str) -> Dict[str, Any]: (rpp_year is required; when omitted, the latest RPP year is filled in before hashing and the completed manifest is written to .completed.json, never over the input; commit and run that file, as the result's notes say). + Several releases: the adjustment also needs "dollar_year", "price_index" + and "rpp_years" ({"2019": 2019, "2024": 2024}); omitted ones are filled the same + way. "replicates": {"source": "census_vre"} makes run_manifest fetch Census + replicates; the notes list variables that have none. The schema is documented in census/manifest.py and examples/manifests/. Example: create_manifest("examples/manifests/income_population_tracts.json") @@ -726,8 +730,8 @@ def run_manifest(manifest_path: str, output_dir: Optional[str] = None, Args: manifest_path: the manifest JSON. - replicates_path: optional CSV/Parquet of replicate estimates (long format: - GEOID, Year, variable, rep_1..rep_80), needed for "replicates": "required". + replicates_path: optional CSV/Parquet of replicates (long: GEOID, Year, + variable, rep_1..rep_80) for "replicates": "required"; not with census_vre. replicates_synthetic: mark supplied replicates synthetic. Returns analysis_status, data_source, replicates, deviations, hypotheses; diff --git a/docs/agent_protocol.md b/docs/agent_protocol.md index beeeb08..d9ca733 100644 --- a/docs/agent_protocol.md +++ b/docs/agent_protocol.md @@ -155,7 +155,7 @@ Every derived result carries `estimate`, `moe` (90%), `se`, `cv_percent`, `metho ## 11. Validation against Census-published values -`run_validation` (CLI: `validate`; script: `scripts/validate_against_census.py`) recomputes values the Census Bureau publishes independently and compares: pipeline cells against raw API cells, Data Profile percentages from detailed tables, replicate MOEs against published MOEs, medians from distributions, Comparison Profile inflation adjustments and significance flags, and county sums against states. Offline checks run against frozen fixtures in every test run; CI runs the live battery weekly. If your analysis relies on a calculation path, check that the corresponding validation check passes in `docs/audits/validation_report.md`. +`run_validation` (CLI: `validate`; script: `scripts/validate_against_census.py`) recomputes values the Census Bureau publishes independently and compares: pipeline cells against raw API cells, Data Profile percentages from detailed tables, replicate MOEs against published MOEs, medians from distributions, Comparison Profile inflation adjustments and significance flags, county sums against states, and (F11) 2020 Census counts against the Census Bureau's invariants and hierarchy. Offline checks run against frozen fixtures in every test run; CI runs the live battery weekly. If your analysis relies on a calculation path, check that the corresponding validation check passes in `docs/audits/validation_report.md`. ## 12. Inference (models, small areas, correlations, figures) @@ -193,3 +193,9 @@ Validated in `docs/audits/inference_validation.md` by simulation with known trut - **Multilevel models and difference-in-differences** (`multilevel_model`, `difference_in_differences`): pass the outcome's SE column so sampling error is included. DiD needs non-overlapping releases and a pre-treatment reference period; with three or more pre-periods, report the parallel-trends test. - **Cost of living** (`adjust_for_place`): only on request; say in results that regional price parities were applied and that they carry their own error. +## 15. 2020 Decennial Census (P.L. 94-171, DHC) + +- **Access:** the same tools as the ACS with `dataset="dec/dhc"` (DHC: `P1` total population, `P12` sex by age, `H1` housing units, `H3` occupancy status) or `dataset="dec/pl"` (P.L. 94-171; its `P1` is race), or `product="decennial"`; `years=[2020]` only. Levels: `us`, `state`, `county`, `tract`, `block group`, `place`, and `block` (P.L. 94-171, or DHC `H1`). Variable codes end in `N` (count). Recipe: `get_recipe(name="decennial_counts")`. +- **Uncertainty:** counts are not sample-based: MOE, SE and CV are undefined, never 0, and no such columns are written. They are not error-free: coverage and processing error, and TopDown differential-privacy noise on everything except state population, block housing units and occupied group-quarters facilities. Read the plan's `uncertainty` record and advisories, aggregate blocks and small areas before computing rates, and say in write-ups that 2020 Census figures carry disclosure-avoidance noise. +- **Tests:** never run a significance test between two census counts (tools refuse `product="decennial"`). Testing an ACS estimate against a census count (`calculate_significance(..., product="5-year", product2="decennial")`) measures ACS sampling error only; report the note it returns. `derive_estimate(..., product="decennial")` gives point estimates only. +- **Do not** splice 2020 Census counts into an ACS series, or treat a DHC table code as the same table in P.L. 94-171. diff --git a/docs/audits/CHECK_MAP.md b/docs/audits/CHECK_MAP.md index c86d016..c9d3f0e 100644 --- a/docs/audits/CHECK_MAP.md +++ b/docs/audits/CHECK_MAP.md @@ -18,6 +18,8 @@ All commands use `.venv/bin/python` from the repository root. | Correlation, Fisher z (`census.correlation`) | `-m pytest -q tests/test_correlation.py` | `correlation` (gated observed-value row; true-value row info) | | Linear trend test (`census.trends`) | `-m pytest -q tests/test_trends.py` | `linear_trend` (gated) | | Moran's I replicate SE (`census.spatial`) | `-m pytest -q tests/test_spatial.py` | `morans_i_replicate_se` (info) | +| Spatial lag / error regression, LM diagnostics (`census.spatial.spatial_regression`) | `-m pytest -q tests/test_spatial.py tests/test_workflows_geo.py` | `spatial_regression_coverage` (info, D15-1) | +| Local Moran's I / LISA (`census.spatial.local_morans`) | `-m pytest -q tests/test_spatial.py tests/test_workflows_geo.py` | `local_morans_size` (info, D15-1) | | Multilevel, known variance (`census.multilevel`) | `-m pytest -q tests/test_multilevel.py` (includes the dense fixed-variance reference `TestKnownVarianceDenseReference`) | `multilevel_known_variance`, `multilevel_random_slope`, `multilevel_replicate` | | Difference-in-differences (`census.did`) | `-m pytest -q tests/test_did.py` (includes the hand-built differenced reference `TestTwoGroupDenseReference`) | `did_two_group`, `did_event`, `did_replicate`; serial correlation (info only): `did_ar1_sensitivity`, `did_serial` | | Release gate itself (`scripts/inference_validation.py`) | `-m pytest -q tests/test_validation_gate.py` | n/a | @@ -64,6 +66,38 @@ Release studies: found coverage 0.882-0.914 at every rho for both designs, so no degradation was observed in these short (4-5 period) panels. Results and scope note: `census.did` docstring. +- Spatial regression and local Moran's I (PLAN_ROUND_15 workstream B, + decision D15-1): `spatial_regression_coverage` and `local_morans_size` + are in the spatial group, so they run in every full report (and with + `--studies NAME`), but every row is `info`: they are not in + `REQUIRED_STUDIES` and never fail the gate. Promotion to gated is an + owner decision. Single-process runtime (full configuration, 16 cores + shared): `spatial_regression_coverage` 5.2 min, `local_morans_size` + about 3.6 min (4.4 min under load), both below the `--jobs 0` floor. + Design: 12x12 rook lattice, tract-profile ACS CVs. + `spatial_regression_coverage`: lag and error DGPs at rho/lambda 0, 0.3, + 0.6 with sampling error in the outcome, ML and GMM through + `spatial_regression` (1000 reps per cell; rows of the tool default are + marked `[tool default]`), plus reference cells without sampling error + and a noisy-covariate cell (500 reps), and the LM tests' size at rho = 0 + (2000 fits). `local_morans_size` (1000 reps per scenario): spatial + randomness with random or spatially patterned SEs, and a 4x4 hot spot + (+1.5 SD); each map is analysed with the tool default and the previous + default, plus the two cross readings (folded at 9999, two-sided at 999). +- Defaults changed after these studies (owner decision, Round 15): + `spatial_regression` uses GMM for the lag model (ML lag rho intervals + covered 0.48 at rho 0.3 and 0.003 at 0.6 under outcome sampling error; + GMM lag 0.899 and 0.889) and keeps ML for the error model (lambda is + attenuated by both estimators and GMM is not better: 0.558 vs 0.581 at + 0.3, 0.162 vs 0.192 at 0.6; beta 0.88-0.91 for both). `local_morans` + uses two-sided p-values (min(1, 2 p_sim); esda's p_sim is folded) from + 9999 permutations with BH at alpha 0.10: per-area rate 0.099 (previous + default 0.196) and family-wise rate 0.088 (MC SE 0.009) under + randomness; power for the hot spot 0.011; spatially patterned SEs still + give spurious clusters in 38% of maps (previous default 26%). The + previous behavior stays available (`method="ml"`; + `permutations=999, sided="folded"`). LM tests' size 0.046-0.056 at + 0.05. Results and caveat text: `census.spatial`. - Runtime (PLAN_ROUND_13 C3): the script pins BLAS/OpenMP to one thread per process, and `--jobs N` (0 = one per CPU) runs studies in parallel processes with output identical to a serial run (except the `seconds` diff --git a/docs/audits/inference_validation.md b/docs/audits/inference_validation.md index d4282cb..f26bed4 100644 --- a/docs/audits/inference_validation.md +++ b/docs/audits/inference_validation.md @@ -1,6 +1,6 @@ # Inference validation (PLAN_ROUND_8 workstream F1) -Generated by `scripts/inference_validation.py` (full configuration) on 2026-11-08; Python 3.14.4; 17.3 min. Seed 20261102. Do not edit by hand. +Generated by `scripts/inference_validation.py` (full configuration) on 2026-11-08; Python 3.14.4; 19.6 min. Seed 20261102. Do not edit by hand. Nominal interval level 90%; acceptance band 88%-92% (PLAN_ROUND_8 sec. 14.3). Monte Carlo SE = sqrt(c(1-c)/R) (Fay-Herriot: between-replication SE of the per-replication coverage, since areas share a fit). Rows with status `info` are comparisons (naive fits, targets a method does not claim) and are not gated. @@ -30,6 +30,92 @@ Nominal interval level 90%; acceptance band 88%-92% (PLAN_ROUND_8 sec. 14.3). Mo | spatial_fay_herriot | fay_herriot EBLUP (non-spatial) on the same data | area mean theta_i, SAR rho=0.5 | 100000 | 0.895 | 0.0010 | info | | | | morans_i_replicate_se | morans_i replicate SE (80 replicate maps), 10x10 rook lattice | E[Moran's I of ACS estimates | true surface] (0.277) | 2000 | 0.973 | 0.0036 | info | | | | morans_i_replicate_se | morans_i replicate SE (80 replicate maps), 10x10 rook lattice | Moran's I of true values (0.406): attenuated, not a target | 2000 | 0.390 | 0.0109 | info | | | +| spatial_regression_coverage | spatial_regression lag ML (default intervals), 12x12 rook lattice, rho=0.0, ACS-like outcome error | beta (0.5) | 1000 | 0.888 | 0.0100 | info | -0.001 | rel. bias -0.003 | +| spatial_regression_coverage | spatial_regression lag ML (default intervals), 12x12 rook lattice, rho=0.0, ACS-like outcome error | rho (0.0) | 1000 | 0.897 | 0.0096 | info | -0.005 | | +| spatial_regression_coverage | spatial_regression lag GMM [tool default] (default intervals), 12x12 rook lattice, rho=0.0, ACS-like outcome error | beta (0.5) | 1000 | 0.887 | 0.0100 | info | -0.003 | rel. bias -0.007 | +| spatial_regression_coverage | spatial_regression lag GMM [tool default] (default intervals), 12x12 rook lattice, rho=0.0, ACS-like outcome error | rho (0.0) | 1000 | 0.903 | 0.0094 | info | 0.005 | | +| spatial_regression_coverage | spatial_regression lag ML (default intervals), 12x12 rook lattice, rho=0.3, ACS-like outcome error | beta (0.5) | 1000 | 0.903 | 0.0094 | info | 0.006 | rel. bias 0.012 | +| spatial_regression_coverage | spatial_regression lag ML (default intervals), 12x12 rook lattice, rho=0.3, ACS-like outcome error | rho (0.3) | 1000 | 0.482 | 0.0158 | info | -0.162 | | +| spatial_regression_coverage | spatial_regression lag GMM [tool default] (default intervals), 12x12 rook lattice, rho=0.3, ACS-like outcome error | beta (0.5) | 1000 | 0.903 | 0.0094 | info | -0.005 | rel. bias -0.009 | +| spatial_regression_coverage | spatial_regression lag GMM [tool default] (default intervals), 12x12 rook lattice, rho=0.3, ACS-like outcome error | rho (0.3) | 1000 | 0.899 | 0.0095 | info | 0.007 | | +| spatial_regression_coverage | spatial_regression lag ML (default intervals), 12x12 rook lattice, rho=0.3, reference: no sampling error | beta (0.5) | 500 | 0.904 | 0.0132 | info | -0.004 | rel. bias -0.009 | +| spatial_regression_coverage | spatial_regression lag ML (default intervals), 12x12 rook lattice, rho=0.3, reference: no sampling error | rho (0.3) | 500 | 0.878 | 0.0146 | info | -0.010 | | +| spatial_regression_coverage | spatial_regression lag GMM [tool default] (default intervals), 12x12 rook lattice, rho=0.3, reference: no sampling error | beta (0.5) | 500 | 0.892 | 0.0139 | info | -0.006 | rel. bias -0.011 | +| spatial_regression_coverage | spatial_regression lag GMM [tool default] (default intervals), 12x12 rook lattice, rho=0.3, reference: no sampling error | rho (0.3) | 500 | 0.892 | 0.0139 | info | -0.001 | | +| spatial_regression_coverage | spatial_regression lag ML (default intervals), 12x12 rook lattice, rho=0.3, ACS-like outcome and covariate error | beta (0.5) | 500 | 0.402 | 0.0219 | info | -0.096 | rel. bias -0.191 | +| spatial_regression_coverage | spatial_regression lag ML (default intervals), 12x12 rook lattice, rho=0.3, ACS-like outcome and covariate error | rho (0.3) | 500 | 0.578 | 0.0221 | info | -0.143 | | +| spatial_regression_coverage | spatial_regression lag GMM [tool default] (default intervals), 12x12 rook lattice, rho=0.3, ACS-like outcome and covariate error | beta (0.5) | 500 | 0.362 | 0.0215 | info | -0.104 | rel. bias -0.209 | +| spatial_regression_coverage | spatial_regression lag GMM [tool default] (default intervals), 12x12 rook lattice, rho=0.3, ACS-like outcome and covariate error | rho (0.3) | 500 | 0.900 | 0.0134 | info | 0.010 | | +| spatial_regression_coverage | spatial_regression lag ML (default intervals), 12x12 rook lattice, rho=0.6, ACS-like outcome error | beta (0.5) | 1000 | 0.854 | 0.0112 | info | 0.036 | rel. bias 0.071 | +| spatial_regression_coverage | spatial_regression lag ML (default intervals), 12x12 rook lattice, rho=0.6, ACS-like outcome error | rho (0.6) | 1000 | 0.003 | 0.0017 | info | -0.409 | | +| spatial_regression_coverage | spatial_regression lag GMM [tool default] (default intervals), 12x12 rook lattice, rho=0.6, ACS-like outcome error | beta (0.5) | 1000 | 0.844 | 0.0115 | info | -0.014 | rel. bias -0.028 | +| spatial_regression_coverage | spatial_regression lag GMM [tool default] (default intervals), 12x12 rook lattice, rho=0.6, ACS-like outcome error | rho (0.6) | 1000 | 0.889 | 0.0099 | info | 0.013 | | +| spatial_regression_coverage | spatial_regression lag ML (default intervals), 12x12 rook lattice, rho=0.6, reference: no sampling error | beta (0.5) | 500 | 0.874 | 0.0148 | info | -0.001 | rel. bias -0.001 | +| spatial_regression_coverage | spatial_regression lag ML (default intervals), 12x12 rook lattice, rho=0.6, reference: no sampling error | rho (0.6) | 500 | 0.878 | 0.0146 | info | -0.012 | | +| spatial_regression_coverage | spatial_regression lag GMM [tool default] (default intervals), 12x12 rook lattice, rho=0.6, reference: no sampling error | beta (0.5) | 500 | 0.866 | 0.0152 | info | -0.002 | rel. bias -0.005 | +| spatial_regression_coverage | spatial_regression lag GMM [tool default] (default intervals), 12x12 rook lattice, rho=0.6, reference: no sampling error | rho (0.6) | 500 | 0.876 | 0.0147 | info | -0.002 | | +| spatial_regression_coverage | spatial_regression error ML [tool default] (default intervals), 12x12 rook lattice, lambda=0.0, ACS-like outcome error | beta (0.5) | 1000 | 0.890 | 0.0099 | info | 0.000 | rel. bias 0.000 | +| spatial_regression_coverage | spatial_regression error ML [tool default] (default intervals), 12x12 rook lattice, lambda=0.0, ACS-like outcome error | lambda (0.0) | 1000 | 0.880 | 0.0103 | info | -0.014 | | +| spatial_regression_coverage | spatial_regression error GMM (default intervals), 12x12 rook lattice, lambda=0.0, ACS-like outcome error | beta (0.5) | 1000 | 0.896 | 0.0097 | info | 0.000 | rel. bias 0.000 | +| spatial_regression_coverage | spatial_regression error GMM (default intervals), 12x12 rook lattice, lambda=0.0, ACS-like outcome error | lambda (0.0) | 1000 | 0.872 | 0.0106 | info | -0.013 | | +| spatial_regression_coverage | spatial_regression error ML [tool default] (default intervals), 12x12 rook lattice, lambda=0.3, ACS-like outcome error | beta (0.5) | 1000 | 0.909 | 0.0091 | info | -0.000 | rel. bias -0.001 | +| spatial_regression_coverage | spatial_regression error ML [tool default] (default intervals), 12x12 rook lattice, lambda=0.3, ACS-like outcome error | lambda (0.3) | 1000 | 0.581 | 0.0156 | info | -0.162 | | +| spatial_regression_coverage | spatial_regression error GMM (default intervals), 12x12 rook lattice, lambda=0.3, ACS-like outcome error | beta (0.5) | 1000 | 0.892 | 0.0098 | info | -0.000 | rel. bias -0.001 | +| spatial_regression_coverage | spatial_regression error GMM (default intervals), 12x12 rook lattice, lambda=0.3, ACS-like outcome error | lambda (0.3) | 1000 | 0.558 | 0.0157 | info | -0.162 | | +| spatial_regression_coverage | spatial_regression error ML [tool default] (default intervals), 12x12 rook lattice, lambda=0.3, reference: no sampling error | beta (0.5) | 500 | 0.870 | 0.0150 | info | -0.001 | rel. bias -0.002 | +| spatial_regression_coverage | spatial_regression error ML [tool default] (default intervals), 12x12 rook lattice, lambda=0.3, reference: no sampling error | lambda (0.3) | 500 | 0.894 | 0.0138 | info | -0.026 | | +| spatial_regression_coverage | spatial_regression error GMM (default intervals), 12x12 rook lattice, lambda=0.3, reference: no sampling error | beta (0.5) | 500 | 0.870 | 0.0150 | info | -0.001 | rel. bias -0.002 | +| spatial_regression_coverage | spatial_regression error GMM (default intervals), 12x12 rook lattice, lambda=0.3, reference: no sampling error | lambda (0.3) | 500 | 0.898 | 0.0135 | info | -0.025 | | +| spatial_regression_coverage | spatial_regression error ML [tool default] (default intervals), 12x12 rook lattice, lambda=0.3, ACS-like outcome and covariate error | beta (0.5) | 500 | 0.186 | 0.0174 | info | -0.103 | rel. bias -0.207 | +| spatial_regression_coverage | spatial_regression error ML [tool default] (default intervals), 12x12 rook lattice, lambda=0.3, ACS-like outcome and covariate error | lambda (0.3) | 500 | 0.450 | 0.0222 | info | -0.199 | | +| spatial_regression_coverage | spatial_regression error GMM (default intervals), 12x12 rook lattice, lambda=0.3, ACS-like outcome and covariate error | beta (0.5) | 500 | 0.236 | 0.0190 | info | -0.103 | rel. bias -0.207 | +| spatial_regression_coverage | spatial_regression error GMM (default intervals), 12x12 rook lattice, lambda=0.3, ACS-like outcome and covariate error | lambda (0.3) | 500 | 0.452 | 0.0223 | info | -0.198 | | +| spatial_regression_coverage | spatial_regression error ML [tool default] (default intervals), 12x12 rook lattice, lambda=0.6, ACS-like outcome error | beta (0.5) | 1000 | 0.881 | 0.0102 | info | 0.002 | rel. bias 0.005 | +| spatial_regression_coverage | spatial_regression error ML [tool default] (default intervals), 12x12 rook lattice, lambda=0.6, ACS-like outcome error | lambda (0.6) | 1000 | 0.192 | 0.0125 | info | -0.264 | | +| spatial_regression_coverage | spatial_regression error GMM (default intervals), 12x12 rook lattice, lambda=0.6, ACS-like outcome error | beta (0.5) | 1000 | 0.879 | 0.0103 | info | 0.002 | rel. bias 0.005 | +| spatial_regression_coverage | spatial_regression error GMM (default intervals), 12x12 rook lattice, lambda=0.6, ACS-like outcome error | lambda (0.6) | 1000 | 0.162 | 0.0117 | info | -0.265 | | +| spatial_regression_coverage | spatial_regression error ML [tool default] (default intervals), 12x12 rook lattice, lambda=0.6, reference: no sampling error | beta (0.5) | 500 | 0.904 | 0.0132 | info | -0.000 | rel. bias -0.000 | +| spatial_regression_coverage | spatial_regression error ML [tool default] (default intervals), 12x12 rook lattice, lambda=0.6, reference: no sampling error | lambda (0.6) | 500 | 0.880 | 0.0145 | info | -0.025 | | +| spatial_regression_coverage | spatial_regression error GMM (default intervals), 12x12 rook lattice, lambda=0.6, reference: no sampling error | beta (0.5) | 500 | 0.878 | 0.0146 | info | -0.000 | rel. bias -0.000 | +| spatial_regression_coverage | spatial_regression error GMM (default intervals), 12x12 rook lattice, lambda=0.6, reference: no sampling error | lambda (0.6) | 500 | 0.884 | 0.0143 | info | -0.021 | | +| spatial_regression_coverage | lm_diagnostics lm_error, no spatial dependence, ACS-like outcome error | size of the 0.05 test (coverage = 1 - size) | 2000 | 0.952 | 0.0048 | info | | test size 0.049 | +| spatial_regression_coverage | lm_diagnostics lm_lag, no spatial dependence, ACS-like outcome error | size of the 0.05 test (coverage = 1 - size) | 2000 | 0.945 | 0.0051 | info | | test size 0.056 | +| spatial_regression_coverage | lm_diagnostics robust_lm_error, no spatial dependence, ACS-like outcome error | size of the 0.05 test (coverage = 1 - size) | 2000 | 0.954 | 0.0047 | info | | test size 0.046 | +| spatial_regression_coverage | lm_diagnostics robust_lm_lag, no spatial dependence, ACS-like outcome error | size of the 0.05 test (coverage = 1 - size) | 2000 | 0.947 | 0.0050 | info | | test size 0.053 | +| spatial_regression_coverage | lm_diagnostics lm_sarma, no spatial dependence, ACS-like outcome error | size of the 0.05 test (coverage = 1 - size) | 2000 | 0.950 | 0.0049 | info | | test size 0.051 | +| spatial_regression_coverage | lm_diagnostics decision rule, no spatial dependence, ACS-like outcome error | share of fits suggesting a spatial model (coverage = 1 - share) | 2000 | 0.920 | 0.0061 | info | | test size 0.080 | +| local_morans_size | local_morans [tool default] two-sided p_value = min(1, 2 p_sim), 9999 permutations, BH, alpha 0.10, 12x12 rook lattice, spatial independence, ACS-like SEs at random | per-area false-positive rate, uncorrected two-sided p_value <= 0.10 (coverage = 1 - rate) | 1000 | 0.901 | 0.0008 | info | | test size 0.099 | +| local_morans_size | local_morans [tool default] two-sided p_value = min(1, 2 p_sim), 9999 permutations, BH, alpha 0.10, 12x12 rook lattice, spatial independence, ACS-like SEs at random | per-area false-positive rate after correction (coverage = 1 - rate) | 1000 | 0.999 | 0.0001 | info | | test size 0.001 | +| local_morans_size | local_morans [tool default] two-sided p_value = min(1, 2 p_sim), 9999 permutations, BH, alpha 0.10, 12x12 rook lattice, spatial independence, ACS-like SEs at random | family-wise error rate = FDR under the complete null (coverage = 1 - rate) | 1000 | 0.912 | 0.0090 | info | | test size 0.088 | +| local_morans_size | local_morans folded p_sim, 9999 permutations, BH, alpha 0.10, 12x12 rook lattice, spatial independence, ACS-like SEs at random | per-area false-positive rate, uncorrected folded p_sim <= 0.10 (coverage = 1 - rate) | 1000 | 0.802 | 0.0010 | info | | test size 0.198 | +| local_morans_size | local_morans folded p_sim, 9999 permutations, BH, alpha 0.10, 12x12 rook lattice, spatial independence, ACS-like SEs at random | per-area false-positive rate after correction (coverage = 1 - rate) | 1000 | 0.998 | 0.0002 | info | | test size 0.002 | +| local_morans_size | local_morans folded p_sim, 9999 permutations, BH, alpha 0.10, 12x12 rook lattice, spatial independence, ACS-like SEs at random | family-wise error rate = FDR under the complete null (coverage = 1 - rate) | 1000 | 0.834 | 0.0118 | info | | test size 0.166 | +| local_morans_size | local_morans [previous default] folded p_sim, 999 permutations, BH, alpha 0.10, 12x12 rook lattice, spatial independence, ACS-like SEs at random | per-area false-positive rate, uncorrected folded p_sim <= 0.10 (coverage = 1 - rate) | 1000 | 0.804 | 0.0011 | info | | test size 0.196 | +| local_morans_size | local_morans [previous default] folded p_sim, 999 permutations, BH, alpha 0.10, 12x12 rook lattice, spatial independence, ACS-like SEs at random | per-area false-positive rate after correction (coverage = 1 - rate) | 1000 | 0.999 | 0.0001 | info | | test size 0.001 | +| local_morans_size | local_morans [previous default] folded p_sim, 999 permutations, BH, alpha 0.10, 12x12 rook lattice, spatial independence, ACS-like SEs at random | family-wise error rate = FDR under the complete null (coverage = 1 - rate) | 1000 | 0.944 | 0.0073 | info | | test size 0.056 | +| local_morans_size | local_morans two-sided p_value = min(1, 2 p_sim), 999 permutations, BH, alpha 0.10, 12x12 rook lattice, spatial independence, ACS-like SEs at random | per-area false-positive rate, uncorrected two-sided p_value <= 0.10 (coverage = 1 - rate) | 1000 | 0.903 | 0.0008 | info | | test size 0.097 | +| local_morans_size | local_morans two-sided p_value = min(1, 2 p_sim), 999 permutations, BH, alpha 0.10, 12x12 rook lattice, spatial independence, ACS-like SEs at random | per-area false-positive rate after correction (coverage = 1 - rate) | 1000 | 1.000 | 0.0000 | info | | test size 0.000 | +| local_morans_size | local_morans two-sided p_value = min(1, 2 p_sim), 999 permutations, BH, alpha 0.10, 12x12 rook lattice, spatial independence, ACS-like SEs at random | family-wise error rate = FDR under the complete null (coverage = 1 - rate) | 1000 | 0.995 | 0.0022 | info | | test size 0.005 | +| local_morans_size | local_morans [tool default] two-sided p_value = min(1, 2 p_sim), 9999 permutations, BH, alpha 0.10, 12x12 rook lattice, spatial independence, SEs clustered along a gradient | per-area false-positive rate, uncorrected two-sided p_value <= 0.10 (coverage = 1 - rate) | 1000 | 0.910 | 0.0007 | info | | test size 0.090 | +| local_morans_size | local_morans [tool default] two-sided p_value = min(1, 2 p_sim), 9999 permutations, BH, alpha 0.10, 12x12 rook lattice, spatial independence, SEs clustered along a gradient | per-area false-positive rate after correction (coverage = 1 - rate) | 1000 | 0.995 | 0.0002 | info | | test size 0.005 | +| local_morans_size | local_morans [tool default] two-sided p_value = min(1, 2 p_sim), 9999 permutations, BH, alpha 0.10, 12x12 rook lattice, spatial independence, SEs clustered along a gradient | family-wise error rate = FDR under the complete null (coverage = 1 - rate) | 1000 | 0.618 | 0.0154 | info | | test size 0.382 | +| local_morans_size | local_morans folded p_sim, 9999 permutations, BH, alpha 0.10, 12x12 rook lattice, spatial independence, SEs clustered along a gradient | per-area false-positive rate, uncorrected folded p_sim <= 0.10 (coverage = 1 - rate) | 1000 | 0.827 | 0.0010 | info | | test size 0.173 | +| local_morans_size | local_morans folded p_sim, 9999 permutations, BH, alpha 0.10, 12x12 rook lattice, spatial independence, SEs clustered along a gradient | per-area false-positive rate after correction (coverage = 1 - rate) | 1000 | 0.991 | 0.0004 | info | | test size 0.009 | +| local_morans_size | local_morans folded p_sim, 9999 permutations, BH, alpha 0.10, 12x12 rook lattice, spatial independence, SEs clustered along a gradient | family-wise error rate = FDR under the complete null (coverage = 1 - rate) | 1000 | 0.477 | 0.0158 | info | | test size 0.523 | +| local_morans_size | local_morans [previous default] folded p_sim, 999 permutations, BH, alpha 0.10, 12x12 rook lattice, spatial independence, SEs clustered along a gradient | per-area false-positive rate, uncorrected folded p_sim <= 0.10 (coverage = 1 - rate) | 1000 | 0.828 | 0.0010 | info | | test size 0.172 | +| local_morans_size | local_morans [previous default] folded p_sim, 999 permutations, BH, alpha 0.10, 12x12 rook lattice, spatial independence, SEs clustered along a gradient | per-area false-positive rate after correction (coverage = 1 - rate) | 1000 | 0.994 | 0.0003 | info | | test size 0.006 | +| local_morans_size | local_morans [previous default] folded p_sim, 999 permutations, BH, alpha 0.10, 12x12 rook lattice, spatial independence, SEs clustered along a gradient | family-wise error rate = FDR under the complete null (coverage = 1 - rate) | 1000 | 0.740 | 0.0139 | info | | test size 0.260 | +| local_morans_size | local_morans two-sided p_value = min(1, 2 p_sim), 999 permutations, BH, alpha 0.10, 12x12 rook lattice, spatial independence, SEs clustered along a gradient | per-area false-positive rate, uncorrected two-sided p_value <= 0.10 (coverage = 1 - rate) | 1000 | 0.912 | 0.0007 | info | | test size 0.088 | +| local_morans_size | local_morans two-sided p_value = min(1, 2 p_sim), 999 permutations, BH, alpha 0.10, 12x12 rook lattice, spatial independence, SEs clustered along a gradient | per-area false-positive rate after correction (coverage = 1 - rate) | 1000 | 0.998 | 0.0002 | info | | test size 0.002 | +| local_morans_size | local_morans two-sided p_value = min(1, 2 p_sim), 999 permutations, BH, alpha 0.10, 12x12 rook lattice, spatial independence, SEs clustered along a gradient | family-wise error rate = FDR under the complete null (coverage = 1 - rate) | 1000 | 0.936 | 0.0077 | info | | test size 0.064 | +| local_morans_size | local_morans [tool default] two-sided p_value = min(1, 2 p_sim), 9999 permutations, BH, alpha 0.10, 12x12 rook lattice, 4x4 hot spot (+15), ACS-like SEs at random | power: share of the 16 hot-spot areas flagged significant HH (coverage column = power) | 1000 | 0.011 | 0.0011 | info | | power 0.011 | +| local_morans_size | local_morans [tool default] two-sided p_value = min(1, 2 p_sim), 9999 permutations, BH, alpha 0.10, 12x12 rook lattice, 4x4 hot spot (+15), ACS-like SEs at random | FDR: discoveries outside the hot spot / discoveries (coverage = 1 - FDR) | 1000 | 0.932 | 0.0076 | info | | test size 0.068; FDR 0.068 | +| local_morans_size | local_morans folded p_sim, 9999 permutations, BH, alpha 0.10, 12x12 rook lattice, 4x4 hot spot (+15), ACS-like SEs at random | power: share of the 16 hot-spot areas flagged significant HH (coverage column = power) | 1000 | 0.024 | 0.0018 | info | | power 0.024 | +| local_morans_size | local_morans folded p_sim, 9999 permutations, BH, alpha 0.10, 12x12 rook lattice, 4x4 hot spot (+15), ACS-like SEs at random | FDR: discoveries outside the hot spot / discoveries (coverage = 1 - FDR) | 1000 | 0.867 | 0.0096 | info | | test size 0.133; FDR 0.133 | +| local_morans_size | local_morans [previous default] folded p_sim, 999 permutations, BH, alpha 0.10, 12x12 rook lattice, 4x4 hot spot (+15), ACS-like SEs at random | power: share of the 16 hot-spot areas flagged significant HH (coverage column = power) | 1000 | 0.018 | 0.0017 | info | | power 0.018 | +| local_morans_size | local_morans [previous default] folded p_sim, 999 permutations, BH, alpha 0.10, 12x12 rook lattice, 4x4 hot spot (+15), ACS-like SEs at random | FDR: discoveries outside the hot spot / discoveries (coverage = 1 - FDR) | 1000 | 0.937 | 0.0064 | info | | test size 0.063; FDR 0.063 | +| local_morans_size | local_morans two-sided p_value = min(1, 2 p_sim), 999 permutations, BH, alpha 0.10, 12x12 rook lattice, 4x4 hot spot (+15), ACS-like SEs at random | power: share of the 16 hot-spot areas flagged significant HH (coverage column = power) | 1000 | 0.003 | 0.0007 | info | | power 0.003 | +| local_morans_size | local_morans two-sided p_value = min(1, 2 p_sim), 999 permutations, BH, alpha 0.10, 12x12 rook lattice, 4x4 hot spot (+15), ACS-like SEs at random | FDR: discoveries outside the hot spot / discoveries (coverage = 1 - FDR) | 1000 | 0.991 | 0.0026 | info | | test size 0.009; FDR 0.009 | | multilevel_known_variance | fit_multilevel known variance (REML), 154 tracts in 27 counties, random intercept | superpopulation slope beta (population) | 1000 | 0.891 | 0.0099 | pass | 0.000 | | | multilevel_known_variance | fit_multilevel MixedLM ignoring sampling variance (same data) | superpopulation slope beta (population) | 1000 | 0.907 | 0.0092 | info | | | | multilevel_random_slope | fit_multilevel known variance, random intercept + slope, 25 counties | superpopulation mean slope beta (population) | 500 | 0.890 | 0.0140 | pass | | | diff --git a/docs/audits/validation_report.json b/docs/audits/validation_report.json index b70db99..f1a98c0 100644 --- a/docs/audits/validation_report.json +++ b/docs/audits/validation_report.json @@ -1,5 +1,5 @@ { - "generated_at": "2026-11-08T04:40:37+00:00", + "generated_at": "2026-11-08T16:08:40+00:00", "mode": "live", "command": "python scripts/validate_against_census.py --live", "summary": { @@ -13,7 +13,8 @@ "F7", "F8", "F9", - "F10" + "F10", + "F11" ], "passed": [ "F1", @@ -25,7 +26,8 @@ "F7", "F8", "F9", - "F10" + "F10", + "F11" ], "failed": [], "errors": [], @@ -83,7 +85,7 @@ ] }, "mode": "live", - "seconds": 11.6 + "seconds": 13.01 }, { "check_id": "F2", @@ -686,7 +688,7 @@ ] }, "mode": "live", - "seconds": 8.49 + "seconds": 16.15 }, { "check_id": "F3", @@ -962,7 +964,7 @@ "api_cell_mismatches": 0 }, "mode": "live", - "seconds": 9.5 + "seconds": 48.09 }, { "check_id": "F4", @@ -1124,7 +1126,7 @@ "pairings": "same constructions and Census definitions as F2 (bachelors_or_higher excluded: B15003 has no VRE file)" }, "mode": "live", - "seconds": 0.74 + "seconds": 1.94 }, { "check_id": "F5", @@ -1340,7 +1342,7 @@ } }, "mode": "live", - "seconds": 3.0 + "seconds": 3.92 }, { "check_id": "F6", @@ -1484,7 +1486,7 @@ "median_items_outside_tolerance_not_asserted": 3 }, "mode": "live", - "seconds": 7.06 + "seconds": 7.95 }, { "check_id": "F7", @@ -2029,7 +2031,7 @@ "census_flag_encoding": "'*' = statistically significant at 90%; null/empty = not significant (variable label 'Statistical Significance!!...', predicateType string)" }, "mode": "live", - "seconds": 2.99 + "seconds": 1.97 }, { "check_id": "F8", @@ -2153,7 +2155,7 @@ } }, "mode": "live", - "seconds": 2.51 + "seconds": 2.85 }, { "check_id": "F9", @@ -2206,7 +2208,7 @@ } }, "mode": "live", - "seconds": 2.44 + "seconds": 3.59 }, { "check_id": "F10", @@ -2258,7 +2260,71 @@ } }, "mode": "live", - "seconds": 9.25 + "seconds": 9.14 + }, + { + "check_id": "F11", + "description": "2020 Census invariants and hierarchy (census.decennial; dec/pl and dec/dhc through the toolkit's API layer), states 26,36,56: (a) state total population in P.L. 94-171 and DHC vs the resident population published in the 2020 Apportionment Results (Table 2); (b) block housing units (H1_001N, a block invariant) summed to tracts and counties vs published tract and county totals, counties 26083, 36041, 56001, and P.L. 94-171 vs DHC block by block; (c) DHC vs P.L. 94-171 for the items both publish (total population, housing units (block invariant), housing units, DHC occupancy-status table, occupied housing units, vacant housing units) at every county of the states and every tract of those counties.", + "n_compared": 5492, + "n_failed": 0, + "max_abs_error": 0.0, + "max_rel_error": null, + "tolerance": "exact (integer equality); these are published counts with no sampling error", + "failures": [], + "census_source": [ + "https://www.census.gov/data/tables/2020/dec/2020-apportionment-data.html", + "https://www.census.gov/programs-surveys/decennial-census/decade/2020/planning-management/process/disclosure-avoidance/2020-das-updates/2020-11-25.html", + "https://www.census.gov/data/tables/2024/dec/2020-census-ppmf.html", + "https://api.census.gov/data/2020/dec/pl", + "https://api.census.gov/data/2020/dec/dhc" + ], + "passed": true, + "notes": [ + "Invariants (published without disclosure-avoidance noise): state total population, block total housing units and occupied group-quarters facilities by type (Census DAS update 2020-11-25, https://www.census.gov/programs-surveys/decennial-census/decade/2020/planning-management/process/disclosure-avoidance/2020-das-updates/2020-11-25.html; fact sheet https://www.census.gov/content/dam/Census/library/factsheets/2021/protecting-the-confidentiality-of-the-2020-census-redistricting-data.pdf). Part (a) compares against an independent publication (the April 2021 apportionment release), part (b) tests the block invariant and the tabulation hierarchy.", + "Part (c) items other than housing units are not invariants: they agree because every 2020 product is tabulated from the same privacy-protected microdata (\"Tabulations from the PPMF are entirely consistent with tables from these data products\", https://www.census.gov/data/tables/2024/dec/2020-census-ppmf.html). Agreement shows the toolkit reads both files consistently; it does not show the counts are free of disclosure-avoidance noise.", + "There is no sampling error to compare: decennial counts have no MOE, so the check asserts exact equality only." + ], + "status": "passed", + "details": { + "parts": { + "a_state_population": { + "compared": 6, + "failed": 0 + }, + "c_county_dhc_vs_pl": { + "compared": 840, + "failed": 0 + }, + "c_tract_dhc_vs_pl": { + "compared": 90, + "failed": 0 + }, + "b_block_pl_vs_dhc": { + "compared": 4514, + "failed": 0 + }, + "b_blocks_to_tracts": { + "compared": 36, + "failed": 0 + }, + "b_blocks_to_county": { + "compared": 6, + "failed": 0 + } + }, + "blocks_per_county": { + "26083": 496, + "36041": 912, + "56001": 3106 + }, + "published_state_population": { + "26": 10077331, + "36": 20201249, + "56": 576851 + } + }, + "mode": "live", + "seconds": 22.95 } ] } diff --git a/docs/audits/validation_report.md b/docs/audits/validation_report.md index d325f23..6ef726f 100644 --- a/docs/audits/validation_report.md +++ b/docs/audits/validation_report.md @@ -1,12 +1,12 @@ # Validation against Census-published values -Generated: 2026-11-08T04:40:37+00:00 +Generated: 2026-11-08T16:08:40+00:00 Mode: live Command: `python scripts/validate_against_census.py --live` Each check recomputes a number the Census Bureau computed independently and compares (PLAN_ROUND_7.md section 8). Tolerances and their reasons are stated per check; a widened tolerance is a visible decision. -**Overall: PASS** (passed 10, failed 0, errors 0, skipped 0) +**Overall: PASS** (passed 11, failed 0, errors 0, skipped 0) | Check | Status | Compared | Failed | Max abs error | Max rel error | Tolerance | | :--- | :--- | :--- | :--- | :--- | :--- | :--- | @@ -20,6 +20,7 @@ Each check recomputes a number the Census Bureau computed independently and comp | F8 | PASSED | 132 | 0 | 0 | - | estimate exact; MOE ratio (approximate sum MOE / published state MOE) reported, not asserted | | F9 | PASSED | 48 | 0 | - | - | as Round 6: no concept flagged as both significantly (z-test, 90%) and implausibly (>25% relative or >10 points) different at the national level | | F10 | PASSED | 1120 | 0 | 0 | - | exact (numeric equality of every estimate and MOE cell) | +| F11 | PASSED | 5492 | 0 | 0 | - | exact (integer equality); these are published counts with no sampling error | ### Known findings @@ -34,7 +35,7 @@ Each check recomputes a number the Census Bureau computed independently and comp Every cell of run_batch output (B01001, county level, releases 2019, 2023, concordance on; tidy CSV panel and Parquet) equals the raw API cell for the same GEOID, year and source code (Census sentinels -> missing, except controlled MOEs -> 0 with _CONTROLLED = True). - Tolerance: exact (float equality); sentinel codes and non-numeric cells must be missing in the output; a controlled MOE (-555555555) must be 0 and flagged in its _CONTROLLED column -- Compared: 9800; failed: 0; max abs error: 0; max rel error: -; time: 11.6 s +- Compared: 9800; failed: 0; max abs error: 0; max rel error: -; time: 13.01 s - Responses pass through run_batch unchanged via a client whose _get_json is served by the same source as the comparison, so the check isolates parsing, sentinel cleaning, concordance renaming, panel stacking, slug renaming and file export.
Details @@ -85,7 +86,7 @@ Sources: Data Profile percentages and percent MOEs rebuilt from detailed-table counts with derived.derive_columns (sum_estimates + proportion, Handbook 2020 ch. 8 formulas 1 and 6), states 06,36,48,17,56,26 and all their counties, releases 2019, 2023. - Tolerance: estimate: |ours - DP| <= 0.1 percentage point (DP is rounded to 0.1). MOE: our approximate MOE must equal an independent evaluation of the Handbook formulas (1e-9); for single-cell numerators (simple subset proportions) it must also be within max(0.2 point, 25%) of the DP MOE, unless the exact replicate MOE for the same geography reproduces the DP MOE within 0.1 point (then the gap is the approximation's ignored covariance and is counted as explained); for multi-cell numerators the ratio ours/DP is reported, not asserted -- Compared: 5880; failed: 0; max abs error: 0.05; max rel error: 0.04712; time: 8.49 s +- Compared: 5880; failed: 0; max abs error: 0.05; max rel error: 0.04712; time: 16.15 s - Census computes Data Profile MOEs from replicate weights, which capture the covariance between numerator and denominator; the Handbook approximation ignores it, so it can over- or understate the MOE (ACS General Handbook 2020, ch. 8). Where the approximate MOE misses the band, the exact replicate MOE for the same geography is computed: if it reproduces the published MOE, the toolkit's formula is right and the gap is the documented approximation error.
Details @@ -687,7 +688,7 @@ Sources: MOEs recomputed from the 80 variance replicates (replicate.verify_published_moe, variance.replicate_moe) vs the MOE published in the same VRE file; VRE estimates and MOEs vs API estimates and MOEs for the same cells. Tables B17001, B27010, B25003, B01001, B01003, B11001, B19001, B25063, state level, releases 2019, 2023, plus the committed county/state VRE fixtures. - Tolerance: replicate MOE within 0.5 of the published (integer) MOE, i.e. one rounding unit; VRE cell == API cell exactly (an API MOE of -555555555 matches a controlled VRE row) -- Compared: 30496; failed: 0; max abs error: 0.5; max rel error: -; time: 9.5 s +- Compared: 30496; failed: 0; max abs error: 0.5; max rel error: -; time: 48.09 s - Controlled rows (CME '*****', published MOE 0) and model-based rows (all 80 replicates equal the estimate, published MOE from the VRE documentation's eqs. 3-5) cannot be reproduced from replicates and are counted separately, as the VRE documentation prescribes.
Details @@ -962,7 +963,7 @@ Sources: Exact replicate proportions (replicate.replicate_proportion on VRE files) vs Data Profile percents and percent MOEs: states 06,36,48,17,56,26 for every construction with a VRE table, plus all their counties for owner_occupied; releases 2019, 2023. - Tolerance: |ours - DP| <= 0.1 percentage point for both the estimate and the MOE (DP rounded to 0.1) -- Compared: 1212; failed: 0; max abs error: 0.09838; max rel error: -; time: 0.74 s +- Compared: 1212; failed: 0; max abs error: 0.09838; max rel error: -; time: 1.94 s - published_moe_at_display_floor counts rows where the DP shows a percent MOE of 0.1 while the exact replicate MOE is below 0.05 (it would round to 0.0). This occurs for the 65+ share at the state level, which is nearly controlled to the population estimates by age; Census does not display a 0.0 percent MOE for an uncontrolled item, so 0.1 acts as a floor. These rows pass only because the tolerance is 0.1; they are listed so the margin is visible.
Details @@ -1123,7 +1124,7 @@ Sources: Medians interpolated from published distribution tables (medians.median_for_rows, linear interpolation) vs published medians, all counties of states 06,36,48,17,56,26, releases 2019, 2023. - Tolerance: gross rent: every county within max($1, 0.1%) of B25064. Household income: median |relative error| < 1% and >= 98% of counties within the published B19013 MOE (share within 2% reported, not asserted: B19001 is coarser than the distribution Census interpolates) -- Compared: 2315; failed: 0; max abs error: 1.623e+04; max rel error: 0.1985; time: 3.0 s +- Compared: 2315; failed: 0; max abs error: 1.623e+04; max rel error: 0.1985; time: 3.92 s - n_failed counts gross-rent counties outside the per-county bound plus household-income releases whose distribution criterion is not met (median relative error < 1% and >= 98% of counties within the published MOE; counties outside 2% are listed as counties_outside_2pct). Published medians at a top or bottom coding bound and medians we top/bottom-code are excluded.
Details @@ -1262,7 +1263,7 @@ Sources: Comparison Profile (CP03) earlier-period dollar values, which Census publishes in current-period dollars, vs the earlier release's DP03 value adjusted with inflation.adjust_inflation (R-CPI-U-RS, the default), states 06,36,48,17,56,26, CP releases 2019 (vs 2014), 2023 (vs 2018). - Tolerance: |ours - CP| <= max($1, 0.5% of CP), asserted for every item Census converts with a single factor (means, aggregates, per-capita); medians reported (see notes) -- Compared: 150; failed: 0; max abs error: 275.6; max rel error: 0.005741; time: 7.06 s +- Compared: 150; failed: 0; max abs error: 275.6; max rel error: 0.005741; time: 7.95 s - Asserted: means, aggregates and per-capita values, which Census converts with a single factor (implied factor constant to 5 digits across items and states). Median items are compared and reported but not asserted: in the 2019 Comparison Profile their implied factors scatter (1.0747-1.0838 against 1.08097 for means), showing that Census re-derived those medians from inflation-adjusted distributions rather than scaling the published median, which no deflator can reproduce from published data; in the 2023 CP medians scale with the same constant factor and all pass. Median misses are listed with asserted=false. - Round 6 acceptance criterion 6: the Census-implied factor (median over mean/aggregate/per-capita items) is compared with our R-CPI-U-RS factor in details.factors.*.implied_minus_ours_percent. @@ -1379,7 +1380,7 @@ Sources: Comparison Profile significance flags (CP03 '..._SS' variables: '*' = the two periods differ at the 90% level, empty = not significant) vs stats.z_test on the CP estimates with the Data Profile MOEs of each release, states 06,36,48,17,56,26, CP releases 2019 (vs 2014), 2023 (vs 2018). - Tolerance: agreement on >= 98% of tests that are decidable from the published (rounded) values; every disagreement listed with its z statistic and its z range under rounding -- Compared: 522; failed: 0; max abs error: -; max rel error: -; time: 2.99 s +- Compared: 522; failed: 0; max abs error: -; max rel error: -; time: 1.97 s - The API's cprofile endpoint publishes estimates (E), annotations (EA) and significance (SS) only, no MOEs; MOEs come from each release's Data Profile (PM for percent items, M for others). For dollar items the earlier MOE is scaled by the same R-CPI-U-RS factor as the estimate. - Census tests unrounded estimates and standard errors; we test the published (rounded) values, so disagreements are expected only for z close to the 1.645 critical value. - A test is 'decidable' when every combination of unrounded values consistent with the published rounding (estimates and MOEs +/- half a unit; a published percent MOE of 0.1 may be any value in (0, 0.15)) gives z on the same side of 1.645. Only decidable tests can disagree with Census because of a calculation error; agreement is asserted on them and reported for all tests. Failures list decidable disagreements first, then undecidable ones. @@ -1477,7 +1478,7 @@ Sources: Sum of all counties (aggregate.aggregate_geographies, approximate MOE) vs the published state estimate for additive counts (B01003_001, B25001_001, B11001_001, B11001_002, B11001_003, B11001_004, B11001_005, B11001_006, B11001_007, B11001_008, B11001_009), states 06,36,48,17,56,26, releases 2019, 2023. - Tolerance: estimate exact; MOE ratio (approximate sum MOE / published state MOE) reported, not asserted -- Compared: 132; failed: 0; max abs error: 0; max rel error: -; time: 2.51 s +- Compared: 132; failed: 0; max abs error: 0; max rel error: -; time: 2.85 s - B01003_001 is controlled (published MOE '*****') at county and state level, so no MOE ratio is formed for it. The approximate sum MOE ignores covariance between counties (Handbook 2020 ch. 8), so the ratio scatters around 1; it is reported, not asserted, as the plan specifies.
Details @@ -1600,7 +1601,7 @@ Sources: Round 6 national continuity check (census.continuity) for every multi-vintage concept of the F1 extraction (B01001, 2019, 2023). - Tolerance: as Round 6: no concept flagged as both significantly (z-test, 90%) and implausibly (>25% relative or >10 points) different at the national level -- Compared: 48; failed: 0; max abs error: -; max rel error: -; time: 2.44 s +- Compared: 48; failed: 0; max abs error: -; max rel error: -; time: 3.59 s - A pass shows no national value jumps implausibly between vintages; it does not prove the codes measure the same thing (census/continuity.py docstring). - Informational (not asserted): the Round 6 check flags concepts of for 2019-2023 although every concept keeps its code and label. The flagged series show real change over four years (income brackets in nominal dollars, poverty and insurance by age as the population ages). census/continuity.py's thresholds (25% relative, 10 points) are documented for adjacent, overlapping releases but are applied to any requested pair, so such extractions block in the default concordance mode. A false positive of the Round 6 check, reported for the module owner; it is not a calculation error. @@ -1652,7 +1653,7 @@ Sources: ACS 2019-2023 table-based Summary File cells (bulk delivery on www2.census.gov) vs API cells for tables B01003, B25003, B17001, B19001, the nation and states 06,36,48,17,56,26. - Tolerance: exact (numeric equality of every estimate and MOE cell) -- Compared: 1120; failed: 0; max abs error: 0; max rel error: -; time: 9.25 s +- Compared: 1120; failed: 0; max abs error: 0; max rel error: -; time: 9.14 s - Only the first bytes of each national per-table file are requested (HTTP Range); its rows are sorted by GEO_ID, so the nation and state rows precede the county, tract and block-group rows. The table-based Summary File exists from the 2021 releases on (summary_file/2021/table-based-SF/); summary_file/2019/ holds only data/ (the sequence-based files, one archive per state holding every table), documentation/ and prototype/ (checked 2026-11-02), so 2019 is not compared here.
Details @@ -1697,3 +1698,66 @@ Sources: - https://www2.census.gov/programs-surveys/acs/summary_file/2023/table-based-SF/data/5YRData/acsdt5y2023-b25003.dat - https://www2.census.gov/programs-surveys/acs/summary_file/2023/table-based-SF/data/5YRData/acsdt5y2023-b17001.dat - https://www2.census.gov/programs-surveys/acs/summary_file/2023/table-based-SF/data/5YRData/acsdt5y2023-b19001.dat + +## F11: PASSED + +2020 Census invariants and hierarchy (census.decennial; dec/pl and dec/dhc through the toolkit's API layer), states 26,36,56: (a) state total population in P.L. 94-171 and DHC vs the resident population published in the 2020 Apportionment Results (Table 2); (b) block housing units (H1_001N, a block invariant) summed to tracts and counties vs published tract and county totals, counties 26083, 36041, 56001, and P.L. 94-171 vs DHC block by block; (c) DHC vs P.L. 94-171 for the items both publish (total population, housing units (block invariant), housing units, DHC occupancy-status table, occupied housing units, vacant housing units) at every county of the states and every tract of those counties. + +- Tolerance: exact (integer equality); these are published counts with no sampling error +- Compared: 5492; failed: 0; max abs error: 0; max rel error: -; time: 22.95 s +- Invariants (published without disclosure-avoidance noise): state total population, block total housing units and occupied group-quarters facilities by type (Census DAS update 2020-11-25, https://www.census.gov/programs-surveys/decennial-census/decade/2020/planning-management/process/disclosure-avoidance/2020-das-updates/2020-11-25.html; fact sheet https://www.census.gov/content/dam/Census/library/factsheets/2021/protecting-the-confidentiality-of-the-2020-census-redistricting-data.pdf). Part (a) compares against an independent publication (the April 2021 apportionment release), part (b) tests the block invariant and the tabulation hierarchy. +- Part (c) items other than housing units are not invariants: they agree because every 2020 product is tabulated from the same privacy-protected microdata ("Tabulations from the PPMF are entirely consistent with tables from these data products", https://www.census.gov/data/tables/2024/dec/2020-census-ppmf.html). Agreement shows the toolkit reads both files consistently; it does not show the counts are free of disclosure-avoidance noise. +- There is no sampling error to compare: decennial counts have no MOE, so the check asserts exact equality only. + +
Details + +```json +{ + "parts": { + "a_state_population": { + "compared": 6, + "failed": 0 + }, + "c_county_dhc_vs_pl": { + "compared": 840, + "failed": 0 + }, + "c_tract_dhc_vs_pl": { + "compared": 90, + "failed": 0 + }, + "b_block_pl_vs_dhc": { + "compared": 4514, + "failed": 0 + }, + "b_blocks_to_tracts": { + "compared": 36, + "failed": 0 + }, + "b_blocks_to_county": { + "compared": 6, + "failed": 0 + } + }, + "blocks_per_county": { + "26083": 496, + "36041": 912, + "56001": 3106 + }, + "published_state_population": { + "26": 10077331, + "36": 20201249, + "56": 576851 + } +} +``` + +
+ +Sources: + +- https://www.census.gov/data/tables/2020/dec/2020-apportionment-data.html +- https://www.census.gov/programs-surveys/decennial-census/decade/2020/planning-management/process/disclosure-avoidance/2020-das-updates/2020-11-25.html +- https://www.census.gov/data/tables/2024/dec/2020-census-ppmf.html +- https://api.census.gov/data/2020/dec/pl +- https://api.census.gov/data/2020/dec/dhc diff --git a/docs/case_study_mi_ny.md b/docs/case_study_mi_ny.md index f9101a2..38bbfbc 100644 --- a/docs/case_study_mi_ny.md +++ b/docs/case_study_mi_ny.md @@ -361,6 +361,70 @@ printed precision as long as the API, the VRE file (cached under `.cache/vre` wi SHA-256) and the boundary files (`.cache/geo`, SHA-256 sidecars) are unchanged. Volatile fields: timestamps (`generated_at`), absolute paths, the git commit, and figure bytes. +## 8. Confirmatory change across releases: 2015-2019 vs 2020-2024 + +A second pre-registered study (PLAN_ROUND_15 A3) compares two **non-overlapping** 5-year +periods in the same 145 counties: +[`examples/manifests/mi_ny_income_change.json`](../examples/manifests/mi_ny_income_change.json). + +```bash +python -m census tool create_manifest --arg manifest_path=examples/manifests/mi_ny_income_change.json --json +python -m census tool run_manifest --arg manifest_path=examples/manifests/mi_ny_income_change.json --arg output_dir=case_study_output/change --json +``` + +Expected from `create_manifest`: `"valid": true`, `"notes": []` and +`"hash": "c420fcf08fcb0c758b050fea18229ffdbff6b0ec22335914c00b69558c923f8f"`. + +| Manifest entry | Value | +| :--- | :--- | +| `releases` | `[2019, 2024]`: no shared sample years, so the two periods are independent samples | +| `variables`, `derivations` | `mean_hh_income` = aggregate household income `B19025_001` / households `B19001_001` | +| `adjustments` | `{"type": "rpp", "variables": ["mean_hh_income"], "areas": "county_metro", "dollar_year": 2024, "price_index": "r-cpi-u-rs", "rpp_years": {"2019": 2019, "2024": 2024}}` | +| `replicates` | `{"source": "census_vre"}`: `run_manifest` fetches the four Census VRE county files itself | +| `analyses` | m1: `fit_replicate`, OLS `mean_hh_income ~ C(Year)`, target `these_areas` (interval from the replicate SE) | +| `hypotheses` | H1: coefficient of `C(Year)[T.2024]` > 0 (one-sided) | + +**Why mean, not median, household income.** The Census Bureau publishes no variance +replicates for `B19013` (section 4), so a change in median income could have no exact +sampling SE. Aggregate income (`B19025`) and households (`B19001`) are both on the VRE list, +so the mean (the Census definition of mean household income: aggregate over households), its +CPI and RPP adjustment and the change between periods are all evaluated on the 80 +replicates. The change is the unweighted mean over counties of each county's change, the +`C(Year)` coefficient (the panel is balanced). The `these_areas` target fits the question: +the 145 counties are all counties of the two states, so the uncertainty is ACS sampling +error, which the replicates measure. + +**Price adjustment, in order (D15-2).** Each release's dollars are converted to 2024 dollars +with the R-CPI-U-RS (2019: factor 462.511 / 375.813 = 1.2307; 2024: 1), then divided by the +BEA RPP / 100 of the county's area **for the release's own year** (2019 RPPs for 2015-2019, +2024 RPPs for 2020-2024). Replicates are scaled by the same factors. `run.json` records the +CPI values and factors, the price-index and RPP table SHA-256s, and the RPP areas per release. + +Expected: `"analysis_status": "confirmatory"`, `"deviations": []`, and for H1 (live, +2026-11-08): + +| Term | Estimate | se_sampling (VRE) | 90% CI | One-sided p (Holm) | +| :--- | ---: | ---: | :--- | ---: | +| C(Year)[T.2024] | 3,110.80 | 244.32 | [2,708.93, 3,512.67] | 1.9e-37 | + +So price-adjusted mean household income (2024 dollars, national-average purchasing power) +rose by about $3,100 per county on average between 2015-2019 and 2020-2024. Every one of the 80 +replicate refits gives a positive change. `run.json` lists under `replicates` the four VRE +files (`B19025` and `B19001`, 2019 and 2024, county) with URL and SHA-256, the published-MOE +self-check of each file, `unavailable: []`, and `mean_hh_income` evaluated on the replicates +for all 290 county-periods. The intercept (95,522) is the 2015-2019 county mean in the same +units. Limits: the RPP and CPI errors are not in the SE; county means weight every county +equally (not households); the 2019 RPPs use BEA's current MSA delineation (OMB 23-01), as for +2024. + +A variable without published replicates (for example `B19013` as the outcome) is listed in +`replicates.unavailable` and stops a replicate analysis under the strict confirmatory policy, +unless the analysis declares `"replicate_scope": "covariates_only"` (outcome held at its +estimate in every refit; `population` or `conservative` target only). The offline test +`tests/test_manifest_releases.py` runs both commands on frozen copies of the API responses and +of the four VRE files (trimmed to the Michigan and New York county rows of the two lines used) +with network access refused; with `CENSUS_LIVE_TESTS=1` it runs them live. + ## Limitations - **RPP error is not in any uncertainty measure.** RPPs are BEA estimates with their own @@ -407,6 +471,14 @@ Closed in Round 14 (PLAN_ROUND_14 A1-A4, offline tests in `tests/test_manifest_m (`{"type": "rpp", ...}`) that reuses the `place_adjust_frame` code and records RPP provenance. Derivations are evaluated on supplied replicates. Section 1 is the result. +Closed in Round 15 (PLAN_ROUND_15 A1-A3, offline tests in `tests/test_manifest_releases.py`): + +- A manifest's price adjustment covered one release only, and `run_manifest` could not fetch + Census replicates. Adjustments now take several releases (`dollar_year`, `price_index`, + `rpp_years`: CPI first, then each release's own RPP), and `"replicates": {"source": + "census_vre"}` fetches the VRE files and records their URLs and SHA-256s. Section 8 is the + result. + Open (documented, not closed): - Census publishes no replicates for `B19013` (and other published medians not on the VRE diff --git a/docs/recipes/decennial_counts.md b/docs/recipes/decennial_counts.md new file mode 100644 index 0000000..c129971 --- /dev/null +++ b/docs/recipes/decennial_counts.md @@ -0,0 +1,11 @@ +# Use 2020 Census counts (P.L. 94-171, DHC) + +Goal: total population, sex by age, housing units or occupancy from the 2020 Census, and use them correctly. + +1. `search_tables(query="occupancy", dataset="dec/dhc")` (DHC: `P1` total population, `P12` sex by age, `H1` housing units, `H3` occupancy status). P.L. 94-171 tables use `dataset="dec/pl"`; table IDs differ by file (P.L. 94-171 `P1` is race). +2. `describe_variable(code="H3_002N")` to confirm the label. Census counts end in `N`, not `E`/`M`. +3. `download_table(table_code="H3", geography="county", state_code="56", dataset="dec/dhc")` (years default to [2020]; the 2020 Census is the only census year here). Blocks: only `dataset="dec/pl"`, or DHC `H1`. +4. Read `uncertainty` and the advisories: the counts are not sample-based, so there is no MOE, SE or CV (they are undefined, never 0). They still carry disclosure-avoidance noise (TopDown differential privacy), coverage and processing error. Only state population, block housing units and occupied group-quarters facilities are exact. +5. Rates from counts: `derive_estimate(operation="proportion", estimates=[occupied, total], product="decennial")` returns the point estimate only. Aggregate blocks or small areas before computing rates. +6. Do not run significance tests between two census counts (`calculate_significance` refuses `product="decennial"`). To compare an ACS estimate with a census count, use `calculate_significance(est1=acs, moe1=acs_moe, est2=census, moe2=0, product="5-year", product2="decennial")` and report its note: only ACS sampling error is measured. +7. Report counts as "2020 Census (DHC/P.L. 94-171), April 1, 2020", without margins of error, and state the disclosure-avoidance caveat. diff --git a/docs/recipes/preregistered.md b/docs/recipes/preregistered.md index bc43811..fa89dc3 100644 --- a/docs/recipes/preregistered.md +++ b/docs/recipes/preregistered.md @@ -4,7 +4,9 @@ Goal: a confirmatory analysis fixed before the data are pulled. 1. Write the manifest (JSON or YAML): research question, table, variables (codes or `YEAR:CODE`), geography, releases, derivations, tests or model formula, multiple-comparison correction, and reliability exclusions. - Several states: `"geography": {"level": "county", "states": ["26", "36"]}` (instead of `"state"`; no `codes`; exactly 2-digit FIPS strings, `"06"` not `"6"`; not for `zcta`, whose GEOIDs are not state-prefixed). States are stacked; add a state term to the formula yourself, e.g. `C(GEOID.str[:2])`. - - Price adjustment: `"adjustments": [{"type": "rpp", "variables": ["income"], "rpp_year": 2024}]` divides dollar variables (and their SEs and replicates) by the BEA Regional Price Parity of each county's metro area or the state's nonmetro portion. `rpp_year` is required so the hash fixes it; when it is omitted, `create_manifest` fills in the latest RPP year, says so in its result, and writes the completed manifest to a new `.completed.json` (the input file is never overwritten); commit and run that file. Exclusions run before the adjustment, so a pre-registered `geoid_exclude` can drop a unit without an RPP area; any remaining unit without one blocks the run (`adjustment_failed`). One release only. + - Price adjustment: `"adjustments": [{"type": "rpp", "variables": ["income"], "rpp_year": 2024}]` divides dollar variables (and their SEs and replicates) by the BEA Regional Price Parity of each county's metro area or the state's nonmetro portion. `rpp_year` is required so the hash fixes it; when it is omitted, `create_manifest` fills in the latest RPP year, says so in its result, and writes the completed manifest to a new `.completed.json` (the input file is never overwritten); commit and run that file. Exclusions run before the adjustment, so a pre-registered `geoid_exclude` can drop a unit without an RPP area; any remaining unit without one blocks the run (`adjustment_failed`). Not combinable with `aggregation`. + - Several releases (non-overlapping 5-year periods, e.g. `"releases": [2019, 2024]`): the adjustment also needs `"dollar_year": 2024`, `"price_index": "r-cpi-u-rs"` and `"rpp_years": {"2019": 2019, "2024": 2024}` (instead of `rpp_year`). Each release's dollars are converted to the dollar year with the price index first, then divided by the RPP of that release's own year; the methods text states the order. `create_manifest` fills omitted fields (latest release, R-CPI-U-RS, each release's own year) into `.completed.json`, as above. Contrast the releases with `C(Year)` in a formula, e.g. `"income ~ C(Year)"`. + - Census replicates: `"replicates": {"source": "census_vre"}` makes `run_manifest` fetch the Census VRE files for the manifest's tables, geography and releases (cached under `.cache/vre`); `run.json` records each file's URL and SHA-256. Do not pass `replicates_path` as well (an error). Tract and block-group VRE files are per state, so give `geography.state` or `states`. Releases must not overlap for any replicate source: validation refuses `census_vre` or an analysis declaring `"replicates": "required"` with overlapping 5-year periods, and caller-supplied replicates (`replicates_path`) spanning overlapping releases stop every replicate analysis (`replicates_overlapping_releases`, blocking). Variables without published replicates (e.g. `B19013`) are listed by `create_manifest` and in `run.json` (`replicates.unavailable`); a replicate analysis that uses them stops (`replicates_incomplete`) unless that analysis declares `"replicate_scope": "covariates_only"` (`fit_replicate` only; the outcome is held at its estimate in every refit; target `population` or `conservative`). The declaration is also allowed for an outcome the VRE registry lists (its line can still be missing for your units): `create_manifest` notes it, and if the run finds replicates for the outcome it still holds it fixed as declared and records `replicates.covariates_only_outcome_had_replicates`. 2. Validate it with `create_manifest` (full profile); it checks every variable resolves and records the manifest's SHA-256. Commit the manifest to the repository before running. 3. Run it with `run_manifest` and the manifest path. It runs `plan_extraction`, the download, derivations and models exactly as written. 4. Do not change the manifest after seeing results. Anything you run outside it on the same data is exploratory; label it so. @@ -12,4 +14,6 @@ Goal: a confirmatory analysis fixed before the data are pulled. Worked example on real data (two states, RPP-adjusted): docs/case_study_mi_ny.md section 1 runs examples/manifests/mi_ny_income_education.json. +Worked example across releases (two states, CPI- and RPP-adjusted, Census VRE replicates): docs/case_study_mi_ny.md section 8 runs examples/manifests/mi_ny_income_change.json. + Worked example (synthetic, offline, executable): docs/reference_workflow.md runs examples/manifests/income_population_tracts.json end to end, including a rerun check and a blocked incomplete-replicates case. diff --git a/docs/remote_access_decisions.md b/docs/remote_access_decisions.md new file mode 100644 index 0000000..1cccd05 --- /dev/null +++ b/docs/remote_access_decisions.md @@ -0,0 +1,46 @@ +# Remote MCP access: planner decisions (2026-11-08) + +Answers to [remote_access_questions.md](remote_access_questions.md) from the owner's project +planner. They are provisional unless stated otherwise. Remote access is implemented +**after Round 15**; Round 15 makes no networking changes. + +## Summary + +| Area | Decision | +| :--- | :--- | +| Transport | MCP stdio over SSH, server started on demand by the SSH session. No Streamable HTTP, persistent service, or TLS layer unless a chosen client requires it | +| Network | Trusted home LAN first; WireGuard later for access away from home; nothing publicly exposed | +| Host setup | OpenSSH, key-only login, a dedicated key per device, a restricted account or forced command, narrowly scoped `ufw` rule. Verify key login before disabling passwords | +| Profiles | `core` by default, `review`/`full` as appropriate. Profiles are not authorization | +| Concurrency | One active research session; no scheduler | +| Census API key | Remote clients may use the desktop key. Requests logged; configurable soft usage limits | +| Authority | Existing safeguards preserved. Destructive operations, concordance overrides and methodological exceptions need explicit human authorization; remote agents cannot approve their own exceptions | +| Audit | Local JSONL: client identity, tool, timestamp, sanitized arguments; 90-day retention; never credentials | +| Files | The desktop is authoritative. Small results and artifact metadata go over MCP; large outputs and all uploads use `scp`/`rsync` over SSH. One workspace per study under a configured root; paths confined (symlinks, traversal). No automatic transfer of raw datasets, PUMS or boundaries | +| Artifact references | Study ID, artifact ID, relative path, content type, checksum | +| Provenance | Record the declared client name, model and version, kept distinct from the authenticated device identity; no unnecessary personal identifiers. Manifests in a dedicated study repository with commit hashes (local commits are not independent timestamps). Exact engine commit and environment recorded; warn or refuse when a study pins a version | +| Operations | Async/resumable jobs only when real workloads need them. Desktop unavailable when asleep. Owner-controlled updates; a lightweight info mechanism reports version, commit and profile | +| Acceptance | From the laptop: run the Michigan/New York confirmatory study through MCP, retrieve maps and bundles, verify provenance, confirm the firewall restrictions remain intact | + +## Larger platform + +- The engine is one independent scientific component of an interdisciplinary research platform. The platform may also include: + - Google Data Commons MCP; + - Semantic Scholar, OpenScholar, PaperQA2; + - local models and other domain engines. +- Interactive assistants (Claude Code, Codex, Gemini) offer critique when needed. Human oversight remains authoritative. No paid model API integrations are planned. +- Preferred formats: Parquet/CSV, GeoPackage, SVG/PDF, Markdown, BibTeX/CSL JSON. +- The platform owns combined-study provenance. This engine owns its own methodological records and validation. +- **Google Data Commons** is approved provisionally as a separate engine, connected through its own MCP service. Don't duplicate its retrieval here. Add interoperability tools only after testing shows a concrete need. No Data Commons work in Round 15. +- Owner-only server. Public source code does not mean public access; published users bring their own credentials. + +## Implementation boundary + +After Round 15, implement the minimum secure SSH-based remote access needed for one real laptop-to-desktop research workflow. Without a demonstrated requirement, do not build: +- Streamable HTTP; +- a persistent service; +- custom file-transfer APIs; +- job queues; +- Data Commons adapters. + +Document unresolved questions rather than resolving them speculatively. diff --git a/docs/remote_access_questions.md b/docs/remote_access_questions.md new file mode 100644 index 0000000..e135e27 --- /dev/null +++ b/docs/remote_access_questions.md @@ -0,0 +1,75 @@ +# Remote MCP access: questions for the project planner + +**Answered 2026-11-08:** see [remote_access_decisions.md](remote_access_decisions.md). + +**Context.** The Census Research Engine (this repository) runs on the owner's Linux desktop (Ubuntu, 128 GB RAM, wired LAN `192.168.1.0/24`). It exposes its tools through an MCP server (`mcp_server.py`, Python `mcp` 1.30, FastMCP). Today the server runs over **stdio only**, so only a client on the desktop can use it. The plan is for a **laptop** running other models and tools to call this server over the network. + +**Current host facts (checked 2026-11-08):** +- `ufw` is active and denies incoming connections. +- No SSH server is installed. +- The engine holds the owner's Census API key in a local `.env`. + +The engine never prints or ships the key; published users bring their own. + +Tool profiles exist: `core` (small local models), `review`, and `full`. + +**Sequencing (owner decision):** remote access comes after Round 15, which covers multi-release confirmatory studies, coverage studies for spatial regression and local Moran's I, and possibly another Census program. Cross-device testing time is limited, so the design should need little iteration. + +Please answer what you can. "Don't know yet" is fine; the engine side will take the default in brackets. + +## A. Clients and models + +1. **Laptop MCP clients.** Which MCP clients will the laptop run? Examples: LM Studio, Ollama with an MCP-capable front end (Open WebUI, Continue, a custom agent), Claude Desktop, Claude Code, or your own agent framework. Which transports does each support: stdio-over-SSH, streamable HTTP, SSE? [default: support both stdio-over-SSH and streamable HTTP] +2. **Clients beyond the laptop.** Will anything other than the laptop connect, such as phones, a second desktop, cloud agents, or scheduled jobs? [default: the laptop only, on the home LAN] +3. **Local-model sizes.** What model sizes will the laptop run? This decides whether the `core` profile stays the default for remote clients. [default: `core` for local models, `full` on request] +4. **Concurrency.** Will more than one client or agent call the server at the same time, for example parallel agents during a study? [default: a few concurrent sessions; long jobs queued] + +## B. Network and access + +5. **Network scope.** Is home-LAN access enough, or must it also work away from home: travel, another network, mobile hotspot? If away from home: a VPN such as Tailscale or WireGuard, or nothing public? [default: LAN only; no port exposed to the internet] +6. **SSH.** Is installing `openssh-server` on the desktop acceptable, with key-only login and a `ufw` rule allowing port 22 from `192.168.1.0/24` only? The laptop would then reach the MCP server through an SSH tunnel, and the MCP port would never be opened. [default: yes; this is the recommended route] +7. **Direct HTTP.** If not SSH, is a `ufw` rule for one HTTP port from the LAN subnet (or one laptop IP) acceptable? Should the laptop get a fixed IP or DHCP reservation so the rule can name it? [default: SSH first; HTTP bound to localhost behind the tunnel] +8. **TLS.** Is it required on the LAN, given that the SSH tunnel already encrypts? [default: none inside the tunnel; required for any non-tunnelled HTTP] +9. **Uptime.** Should the server always be running (a systemd user service started at boot), or started on demand per session? Must it survive reboots and logouts? [default: on demand, plus an optional systemd unit] + +## C. Authentication and authority + +10. **Authentication.** SSH keys only, or also a bearer token for HTTP clients? One shared credential, or one per client so access can be revoked separately? [default: per-client SSH keys; per-client tokens if HTTP is used] +11. **API key spending.** Remote clients spend the owner's Census API key. Is that acceptable for every authenticated client? Should there be a daily call or download budget, or a per-client cap? [default: allowed; a soft daily cap with logging] +12. **Tool restrictions for remote clients.** Should anything be off-limits remotely? Candidates: + - writing outside a workspace + - recording concordance overrides (`record_override`, `record_rule_review`) + - deleting caches + - running long jobs such as full validation or PUMS downloads + + Are overrides human-only, so the owner must approve them? [default: remote clients may draft overrides but not record them; destructive and long actions need an explicit flag] +13. **Audit log.** Who called which tool with which arguments, and when? How long is it kept? [default: a local JSONL log with the client name, kept 90 days, with secrets redacted] + +## D. Files and outputs (the main design question) + +Most tools take input paths and write outputs (maps, figures, tables, bundles, manifests) on the desktop's filesystem. A remote client can neither send its own files nor open the returned paths. + +14. **Where analysis files live.** On the desktop only, with the laptop treating them as remote? Synced, for example through a shared folder, Syncthing or Nextcloud? Or a mix? [default: a desktop-side workspace, fetched on demand] +15. **Getting outputs to the laptop.** Through the MCP itself (MCP resources, or base64 content in tool results, both with size limits), through a separate download path such as HTTP file links inside the tunnel or `scp`, or a shared folder? [default: MCP resources for small artifacts (≤ 10 MB); `scp`/`rsync` over the same SSH for bundles] +16. **Uploads.** Will the laptop upload its own inputs: manifests, custom crosswalks, its own CSVs, replicate files? Through MCP, `scp`, or a shared folder? [default: manifests and small CSVs through an MCP upload tool into the workspace; larger files by `scp`] +17. **Workspaces.** One per client, per study, or one shared? Should tool paths be confined to a workspace root, rejecting any path outside it? [default: one workspace per study under a configured root; paths confined] +18. **Size ceilings.** What are the largest outputs to transfer? PUMS extracts and national boundary files can be hundreds of MB. [default: never transfer PUMS or boundaries; send derived outputs only] + +## E. Reproducibility and provenance + +19. **Recording the client.** Should manifests and results bundles record which client or model ran the analysis: model name and version, client name, laptop host? Is the agent identity in `authors` (for example `agent:`) enough? [default: record the client name and model if the client sends them; never record the laptop's username or IP] +20. **Pre-registration timing.** When a laptop agent pre-registers a study, where is the manifest committed so the time-stamp is trustworthy: the desktop's git repository, a study repository, or both? [default: a study repository on the desktop; the engine reports the hash] +21. **Engine version for a study.** Should remote clients be able to pin it, for example refusing to run if the desktop has moved to a newer release mid-study? [default: report the version and warn on change] + +## F. Operations + +22. **Long jobs.** Full validation, PUMS downloads, big map batches: should they be asynchronous, returning a job id to poll, or block the call? What timeout do the laptop clients use? [default: anything expected over about 60 s becomes a job] +23. **Desktop availability.** If the desktop is asleep or off when the laptop wants it, is wake-on-LAN wanted, or is it simply unavailable? [default: unavailable] +24. **Updates.** When the engine updates, who restarts the server? Should the laptop be told about new tools? [default: the owner restarts; a `server_info` tool reports the version and profile] +25. **Acceptance test.** What counts as "remote access done"? Proposed: from the laptop, with the chosen client, run the Michigan/New York confirmatory case study end to end and retrieve its maps and bundle, with the firewall still denying everything except the agreed rule. + +## G. Larger-project fit + +26. **The larger project.** What else does it contain that this engine must interoperate with: other MCP servers, a study notebook or writing tool, a reference manager, an orchestration agent? Which output formats do they expect: Parquet, CSV, GeoPackage, PDF or SVG figures, LaTeX or Markdown tables, BibTeX citations? +27. **Studies outside the engine.** Will complete studies combine this engine's outputs with non-Census data? If so, which side owns provenance and the combined manifest? +28. **Other users.** Will anyone other than the owner use this server: collaborators, students? This changes authentication, audit and API key policy (published users bring their own key). diff --git a/examples/manifests/mi_ny_income_change.json b/examples/manifests/mi_ny_income_change.json new file mode 100644 index 0000000..a961a41 --- /dev/null +++ b/examples/manifests/mi_ny_income_change.json @@ -0,0 +1,34 @@ +{ + "schema_version": 1, + "title": "Change in price-adjusted county mean household income, Michigan and New York, ACS 2015-2019 vs 2020-2024", + "authors": ["human:example-researcher", "agent:claude"], + "created": "2026-11-08T00:00:00Z", + "releases": [2019, 2024], + "geography": {"level": "county", "states": ["26", "36"]}, + "variables": [ + {"name": "agg_income", "ref": "2024:B19025_001"}, + {"name": "households", "ref": "2024:B19001_001"} + ], + "derivations": [ + {"name": "mean_hh_income", "op": "ratio", "num": "agg_income", "den": "households"} + ], + "adjustments": [ + {"type": "rpp", "variables": ["mean_hh_income"], "areas": "county_metro", + "dollar_year": 2024, "price_index": "r-cpi-u-rs", "rpp_years": {"2019": 2019, "2024": 2024}} + ], + "replicates": {"source": "census_vre"}, + "exclusions": [ + {"rule": "cv_screen", "threshold": 30, "variables": ["mean_hh_income"], "action": "flag"} + ], + "missing": "complete_case", + "confidence_level": 0.90, + "correction": {"method": "holm", "family": "all"}, + "analyses": [ + {"id": "m1", "method": "fit_replicate", "formula": "mean_hh_income ~ C(Year)", + "args": {"model": "ols", "target": "these_areas"}} + ], + "hypotheses": [ + {"id": "H1", "statement": "Across all Michigan and New York counties, mean household income in 2024 dollars, adjusted for each county's price level (RPP of the release's own year), was higher in 2020-2024 than in 2015-2019 (unweighted mean of the county changes).", + "analysis": "m1", "statistic": "coefficient", "term": "C(Year)[T.2024]", "direction": "greater"} + ] +} diff --git a/mcp_server.py b/mcp_server.py index 7cdee75..617eab3 100644 --- a/mcp_server.py +++ b/mcp_server.py @@ -57,6 +57,7 @@ from census import generate_codebook as _generate_codebook from census.concordance import BLOCKING_KINDS from census.data_integrity import BundledDataIntegrityError, integrity_error_result +from census.decennial import DISCLOSURE_AVOIDANCE_ADVISORY, is_decennial, uncertainty_record from census.metadata import table_universe from census.panel import IDENTIFIER_COLUMNS, describe_columns from census.review import draft_override, record_override @@ -108,6 +109,12 @@ - Nationwide: state_code='all'. Reproducibility: replay_extraction. - run_validation checks the toolkit against Census-published values. +2020 Census: dataset='dec/dhc' (DHC: P1, P12, H1, H3, ...) or 'dec/pl' (P.L. 94-171, +down to blocks), years 2020 only, through the same plan_extraction / batch_download. +Census counts are not sample-based: MOE/CV are undefined (never 0); significance tools +refuse decennial-only tests (product='decennial'); an ACS estimate can be tested against +a census count with calculate_significance(product='5-year', product2='decennial'). + Inflation: default index is R-CPI-U-RS (the Census Bureau's ACS index); 'c-cpi-u' matches CPS ASEC income reports. Never pass guessed values via cpi_table. Branch on error_code in failed results. @@ -145,7 +152,7 @@ def _entities_for( places_file = None if is_nationwide(state_code) else get_local_places_file(state_code or "") if places_file and os.path.exists(places_file): names = {p[1]: p[2] for p in load_places_from_local(places_file)} - elif geo_level in ("county", "tract"): + elif geo_level in ("county", "tract", "block"): codes = county_codes elif geo_level == "zcta": codes = zcta_codes @@ -342,7 +349,9 @@ def list_zctas( @mcp.tool() -def search_tables(query: str, limit: int = 25, year: Optional[int] = None) -> List[Dict[str, str]]: +def search_tables( + query: str, limit: int = 25, year: Optional[int] = None, dataset: Optional[str] = None +) -> List[Dict[str, str]]: """Search ACS 5-year tables (detailed, data profile, subject, comparison profile) by code or keyword. @@ -350,16 +359,17 @@ def search_tables(query: str, limit: int = 25, year: Optional[int] = None) -> Li query: Table code (e.g. 'B01001', 'DP03', 'S1701') or keyword (e.g. 'poverty'). limit: Maximum number of matching tables to return (default: 25). year: Release to search (default: latest archived release). + dataset: Search one dataset only, e.g. 'dec/dhc' or 'dec/pl' (2020 Census). """ all_tables: Dict[str, str] = {} - for dataset in ("acs5", "profile", "subject", "cprofile"): - catalog = load_table_catalog(dataset, year) or DATASET_TABLES.get(dataset, {}) + for ds in (dataset,) if dataset else ("acs5", "profile", "subject", "cprofile"): + catalog = load_table_catalog(ds, year) or DATASET_TABLES.get(ds, {}) all_tables.update(catalog) q = query.strip().lower() matches: List[Dict[str, str]] = [] for code, desc in sorted(all_tables.items()): if q in code.lower() or q in desc.lower(): - matches.append({"table_code": code, "description": desc, "dataset": infer_dataset(code)}) + matches.append({"table_code": code, "description": desc, "dataset": dataset or infer_dataset(code)}) if len(matches) >= limit: break return matches @@ -367,17 +377,19 @@ def search_tables(query: str, limit: int = 25, year: Optional[int] = None) -> Li @mcp.tool() def list_variables( - table_code: str, year: Optional[int] = None, limit: int = 50 + table_code: str, year: Optional[int] = None, limit: int = 50, dataset: Optional[str] = None ) -> Dict[str, Any]: """List a table's variables and labels for one release. Args: - table_code: ACS table code (e.g. 'B01001', 'DP02'). + table_code: ACS table code (e.g. 'B01001', 'DP02'), or a 2020 Census table + with dataset='dec/dhc' / 'dec/pl' (e.g. 'P12'). year: Release year (default: latest archived). Labels and codes differ between releases. limit: Maximum variables returned (default 50). + dataset: Override the inferred endpoint (e.g. 'dec/dhc'). """ - dataset = infer_dataset(table_code) + dataset = dataset or infer_dataset(table_code) year = year or _default_year(dataset) variables, source = table_variables(table_code, dataset, year, _client()) items = sorted(variables.items()) @@ -394,16 +406,19 @@ def list_variables( @mcp.tool() -def describe_variable(code: str, year: Optional[int] = None) -> Dict[str, Any]: +def describe_variable(code: str, year: Optional[int] = None, dataset: Optional[str] = None) -> Dict[str, Any]: """Label, concept, table, and universe for one variable code in one release. Use this to confirm what a code means in each year before pooling years. Args: - code: Variable code (e.g. 'DP02_0016PE', 'B19013_001E'). + code: Variable code (e.g. 'DP02_0016PE', 'B19013_001E', or a 2020 Census + count such as 'P12_002N'). year: Release year (default: latest archived). + dataset: Override the inferred endpoint ('dec/dhc' or 'dec/pl' for the 2020 + Census; 'N' codes default to 'dec/dhc'). """ - return _describe_variable(code, year, client=_client()) + return _describe_variable(code, year, dataset=dataset, client=_client()) # --------------------------------------------------------------------------- @@ -438,13 +453,15 @@ def fetch_table_data( state_code: 2-digit state FIPS code (e.g. '36'); not needed for 'us'/'msa'. table_code: Detailed ('B01001'), data profile ('DP03'), subject ('S1701') or comparison profile ('CP03') table code. - geo_level: 'us', 'state', 'county', 'tract', 'place', 'zcta', or 'msa'. + geo_level: 'us', 'state', 'county', 'tract', 'place', 'zcta', or 'msa' ('block' + with dataset='dec/pl', one state and county). place_code: Required if geo_level is 'place' (e.g. '51000' for New York city). county_code: County FIPS for 'county'/'tract' levels (default '*' = all). tract_code: Optional 6-digit tract code (default all tracts). zcta_code: ZCTA(s) for 'zcta' level, comma-separated (default '*'). msa_code: CBSA code(s) for 'msa' level (default '*'). - dataset: 'acs5', 'profile', 'subject' or 'cprofile' (inferred if omitted). + dataset: 'acs5', 'profile', 'subject' or 'cprofile' (inferred if omitted), or the + 2020 Census 'dec/dhc' / 'dec/pl' (counts: no MOE; see `uncertainty`). compute_reliability: Add standard error, CV % and reliability columns. clean_sentinels: Replace Census jam values (-666666666 etc.) with null. use_cache: Read/write the local SQLite query cache. @@ -494,6 +511,7 @@ def fetch_table_data( formatted_df = df.rename(columns=rename) formatted_df = formatted_df.astype(object).where(pd.notna(formatted_df), None) + decennial = is_decennial(dataset) return { "success": True, "year": year, @@ -501,6 +519,8 @@ def fetch_table_data( "table_code": table_code, "dataset": dataset, "geo_level": geo_level, + "uncertainty": uncertainty_record(geo_level) if decennial else None, + "advisories": [DISCLOSURE_AVOIDANCE_ADVISORY] if decennial else [], "row_count": len(formatted_df), "columns": list(formatted_df.columns), "data": formatted_df.head(max_rows).to_dict(orient="records"), @@ -663,7 +683,8 @@ def batch_download( table_code: Table code ('B01001', 'DP03', 'S1701', 'CP03'). start_year: Starting ACS 5-year end-year (e.g. 2015). end_year: Ending year (e.g. 2022). - geo_level: 'us', 'state', 'county', 'tract', 'place', 'zcta', or 'msa'. + geo_level: 'us', 'state', 'county', 'tract', 'place', 'zcta', or 'msa'; the 2020 + Census also offers 'block group' and 'block' (block: dec/pl, or dec/dhc H1). variables: Codes the analysis uses ('B19013_001', '2018:DP02_0002', or concept IDs). Omitted = whole table, so any drift in the table can block. place_codes: Place FIPS codes (required for 'place'). @@ -673,7 +694,8 @@ def batch_download( county_codes: Counties for 'county' or 'tract' (omitted = all in state). zcta_codes: ZCTAs for 'zcta' (omitted = all ZIPs in state via zip2fips.json). msa_codes: CBSA codes for 'msa' (omitted = all metro/micro areas). - dataset: 'acs5', 'profile', 'subject', 'cprofile' (inferred if omitted). + dataset: 'acs5', 'profile', 'subject', 'cprofile' (inferred if omitted), or the + 2020 Census 'dec/dhc' (DHC) / 'dec/pl' (P.L. 94-171); years 2020 only. concordance_mode: 'fail' (default), 'strict' (exploration), or 'code' (legacy). continuity_check: Check national continuity of multi-year concepts. allow_unverified_concordance: Required for multi-year 'code' mode. @@ -691,7 +713,10 @@ def batch_download( harmonize_tracts_to: 2010 or 2020: apportion a tract panel spanning the 2010/2020 tract change to one tract vintage (panel/parquet only). product: "5-year" (default) or "1-year" (areas of 65,000+ people only; - timelier, non-overlapping releases; never mix with 5-year in one comparison). + timelier, non-overlapping releases; never mix with 5-year in one comparison), + or "decennial" (2020 Census DHC unless dataset='dec/pl'; counts are not + sample-based: no MOE/SE/CV columns, a disclosure-avoidance advisory and an + `uncertainty` record in the metadata). allow_experimental_2020: 1-year only: Census published no standard 2020 1-year estimates; opt in to acknowledge the gap (2020 is skipped with an advisory). """ @@ -877,7 +902,7 @@ def validate_crosswalks() -> Dict[str, Any]: @mcp.tool() -def calculate_cv(estimate: float, moe: float) -> Dict[str, Any]: +def calculate_cv(estimate: float, moe: float, product: Optional[str] = None) -> Dict[str, Any]: """Standard error, coefficient of variation and reliability for an ACS estimate. SE = MOE / 1.645; CV = SE / estimate x 100. @@ -886,8 +911,12 @@ def calculate_cv(estimate: float, moe: float) -> Dict[str, Any]: Args: estimate: The ACS estimate. moe: Its published 90% margin of error. + product: 'decennial' is refused: 2020 Census counts have no sampling MOE. """ - return _calculate_cv(estimate, moe) + try: + return _calculate_cv(estimate, moe, product=product) + except ValueError as exc: + return errors.error_result(errors.INVALID_INPUT, str(exc)) @mcp.tool() @@ -900,6 +929,7 @@ def calculate_significance( year1: Optional[int] = None, year2: Optional[int] = None, product: str = "5-year", + product2: Optional[str] = None, ) -> Dict[str, Any]: """Two-tailed Z-test for whether two ACS estimates differ significantly. @@ -913,11 +943,17 @@ def calculate_significance( confidence_level: 0.90, 0.95 or 0.99 (default 0.90, the ACS standard). year1, year2: Optional ACS end years for the overlap adjustment. product: '5-year' (default) or '1-year' (1-year releases never overlap). + 'decennial' alone is refused (census counts are not sample-based). + product2: Product of the second value when it differs: 'decennial' tests the + ACS estimate against a 2020 Census count as a fixed reference (moe2=0); + the result notes that only ACS sampling error is measured. """ + second = product2 or product try: - overlap = acs_overlap_years(year1, year2, product) if year1 and year2 else 0 + both_acs = not (is_decennial(product) or is_decennial(second)) + overlap = acs_overlap_years(year1, year2, product, second) if year1 and year2 and both_acs else 0 return z_test(est1, moe1, est2, moe2, confidence_level, overlap_years=overlap, - product1=product, product2=product) + product1=product, product2=second) except ValueError as exc: return errors.error_result(errors.INVALID_INPUT, str(exc)) @@ -985,7 +1021,8 @@ def generate_codebook( Args: table_code: e.g. 'B19013', 'DP03', 'S1701'. year: Release year (default: latest archived). - dataset: Override endpoint ('acs5', 'profile', 'subject', 'cprofile'). + dataset: Override endpoint ('acs5', 'profile', 'subject', 'cprofile', or the 2020 + Census 'dec/dhc' / 'dec/pl': counts only, with an Uncertainty column). include_moe: Include margin-of-error variables. include_reliability: Include derived SE / CV / reliability columns. output_path: Optional CSV path to also write the codebook to. @@ -1000,6 +1037,9 @@ def generate_codebook( columns: List[str] = [] for code in sorted(metadata): + if is_decennial(dataset) and code.endswith("N"): + columns.append(code) # 2020 Census count: no MOE or reliability columns exist + continue if code.endswith("PE"): base, moe, pct = code[:-2], f"{code[:-2]}PM", True elif code.endswith("E") and not code.endswith("EA"): @@ -1014,7 +1054,8 @@ def generate_codebook( columns += [f"{key}_SE", f"{key}_CV", f"{key}_RELIABILITY"] codebook = _generate_codebook( - columns, metadata, universe=table_universe(table_code, dataset, year) + columns, metadata, universe=table_universe(table_code, dataset, year), + decennial=is_decennial(dataset), ) if output_path: codebook.to_csv(output_path, index=False, encoding="utf-8") diff --git a/metadata/dec/dhc/2020/groups.json.gz b/metadata/dec/dhc/2020/groups.json.gz new file mode 100644 index 0000000..66de6c3 Binary files /dev/null and b/metadata/dec/dhc/2020/groups.json.gz differ diff --git a/metadata/dec/dhc/2020/variables.json.gz b/metadata/dec/dhc/2020/variables.json.gz new file mode 100644 index 0000000..748708b Binary files /dev/null and b/metadata/dec/dhc/2020/variables.json.gz differ diff --git a/metadata/dec/pl/2020/groups.json.gz b/metadata/dec/pl/2020/groups.json.gz new file mode 100644 index 0000000..bbd827d Binary files /dev/null and b/metadata/dec/pl/2020/groups.json.gz differ diff --git a/metadata/dec/pl/2020/variables.json.gz b/metadata/dec/pl/2020/variables.json.gz new file mode 100644 index 0000000..c363da1 Binary files /dev/null and b/metadata/dec/pl/2020/variables.json.gz differ diff --git a/metadata/manifest.json b/metadata/manifest.json index 357420c..46e9cc9 100644 --- a/metadata/manifest.json +++ b/metadata/manifest.json @@ -1,5 +1,5 @@ { - "description": "Per-vintage Census API metadata for ACS 5-year and 1-year datasets. Paths are //.gz; sha256 and bytes refer to the raw (uncompressed) API response.", + "description": "Per-vintage Census API metadata for ACS 5-year and 1-year datasets and the 2020 Decennial Census (dec/pl, dec/dhc). Paths are //.gz; sha256 and bytes refer to the raw (uncompressed) API response.", "files": { "acs1/2005/groups.json.gz": { "url": "https://api.census.gov/data/2005/acs/acs1/groups.json", @@ -1153,6 +1153,30 @@ "bytes": 490033, "sha256": "2ef6cfc2f82415cefff5931f6742c4f006e95f2d760f082e58f662380f143544" }, + "dec/dhc/2020/groups.json.gz": { + "url": "https://api.census.gov/data/2020/dec/dhc/groups.json", + "retrieved_at": "2026-11-08T15:50:37+00:00", + "bytes": 71485, + "sha256": "68ba7390233d0507103120af8b49890085c45708c4bd8ea5355c8b63f4386907" + }, + "dec/dhc/2020/variables.json.gz": { + "url": "https://api.census.gov/data/2020/dec/dhc/variables.json", + "retrieved_at": "2026-11-08T15:50:36+00:00", + "bytes": 2689500, + "sha256": "8e14e443ea2b9049d46964db2f20630d05ffac229ffb09193bf0ad646da617c6" + }, + "dec/pl/2020/groups.json.gz": { + "url": "https://api.census.gov/data/2020/dec/pl/groups.json", + "retrieved_at": "2026-11-08T15:50:36+00:00", + "bytes": 1292, + "sha256": "400a68d5b1b702b19e47b5c0afe6e4d7aa1509dc26d1d6ac4a8f5d68b3cd6d92" + }, + "dec/pl/2020/variables.json.gz": { + "url": "https://api.census.gov/data/2020/dec/pl/variables.json", + "retrieved_at": "2026-11-08T15:50:36+00:00", + "bytes": 108496, + "sha256": "45798ba51fe0e6d97c3efdec6b04af455ff01a4d948843576c41a5d15d0db07a" + }, "profile/2009/groups.json.gz": { "url": "https://api.census.gov/data/2009/acs/acs5/profile/groups.json", "retrieved_at": "2026-11-02T02:38:43+00:00", @@ -1670,6 +1694,12 @@ 2023, 2024, 2025 + ], + "dec/pl": [ + 2020 + ], + "dec/dhc": [ + 2020 ] } } diff --git a/scripts/fetch_metadata_archive.py b/scripts/fetch_metadata_archive.py index bf629c5..a127da6 100644 --- a/scripts/fetch_metadata_archive.py +++ b/scripts/fetch_metadata_archive.py @@ -1,5 +1,5 @@ # SPDX-License-Identifier: AGPL-3.0-or-later -"""Download per-vintage ACS 5-year and 1-year metadata into metadata/ (offline reference archive). +"""Download per-vintage ACS 5-year, 1-year and 2020 Census metadata into metadata/ (offline reference archive). For each dataset and each vintage listed in the Census API catalog, saves ``variables.json`` and ``groups.json`` gzipped, and records source URL, @@ -9,6 +9,7 @@ Usage: python scripts/fetch_metadata_archive.py # fetch missing files python scripts/fetch_metadata_archive.py --refresh # re-download and report changes + python scripts/fetch_metadata_archive.py --datasets dec/pl,dec/dhc # only these datasets Published vintages are expected to be immutable; --refresh reports any file whose checksum changed instead of silently replacing history. @@ -44,7 +45,12 @@ "acs1/profile": "acs/acs1/profile", "acs1/subject": "acs/acs1/subject", "acs1/cprofile": "acs/acs1/cprofile", + # 2020 Decennial Census: P.L. 94-171 and DHC (2020 only; see VINTAGE_FILTER) + "dec/pl": "dec/pl", + "dec/dhc": "dec/dhc", } +# Datasets archived for some vintages only (the catalog also lists 2000/2010 dec/pl). +VINTAGE_FILTER: Dict[str, List[int]] = {"dec/pl": [2020], "dec/dhc": [2020]} FILES = ("variables.json", "groups.json") @@ -57,7 +63,10 @@ def discover_vintages(session: requests.Session) -> Dict[str, List[int]]: year = entry.get("c_vintage") if path in paths and year: found[paths[path]].append(int(year)) - return {k: sorted(set(v)) for k, v in found.items()} + return { + k: sorted(y for y in set(v) if k not in VINTAGE_FILTER or y in VINTAGE_FILTER[k]) + for k, v in found.items() + } def load_manifest() -> Dict: @@ -70,19 +79,29 @@ def load_manifest() -> Dict: def main() -> int: parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) parser.add_argument("--refresh", action="store_true", help="re-download existing files") + parser.add_argument("--datasets", help="comma-separated dataset keys to fetch (default: all)") args = parser.parse_args() + only = [d.strip() for d in args.datasets.split(",")] if args.datasets else None + unknown = [d for d in only or [] if d not in DATASETS] + if unknown: + parser.error(f"unknown dataset(s) {unknown}; choose from {sorted(DATASETS)}") session = requests.Session() manifest = load_manifest() manifest["description"] = ( - "Per-vintage Census API metadata for ACS 5-year and 1-year datasets. Paths are " + "Per-vintage Census API metadata for ACS 5-year and 1-year datasets and the 2020 " + "Decennial Census (dec/pl, dec/dhc). Paths are " "//.gz; sha256 and bytes refer to the raw " "(uncompressed) API response." ) manifest["catalog_url"] = CATALOG_URL files = manifest.setdefault("files", {}) vintages = discover_vintages(session) - manifest["vintages"] = vintages + if only is not None: + vintages = {k: v for k, v in vintages.items() if k in only} + manifest["vintages"] = {**manifest.get("vintages", {}), **vintages} + else: + manifest["vintages"] = vintages changed: List[str] = [] for dataset, years in vintages.items(): diff --git a/scripts/inference_validation.py b/scripts/inference_validation.py index 3995200..d9652b8 100644 --- a/scripts/inference_validation.py +++ b/scripts/inference_validation.py @@ -18,8 +18,11 @@ regression_calibration (~22 min). Summed per-study times are ~83 min, the serial estimate. spatial_fay_herriot takes ~3.5 min for 1000 reps (was 56 min in v0.11.0, when BLAS ran multithreaded on its 100 x 100 matrices). ---fast --gate takes ~50 s serial. --jobs output is identical to the serial -output at the same seed except the "seconds" column (wall time per study). +--fast --gate takes ~50 s serial. The Round 15 info studies in the spatial +group add 5.2 min (spatial_regression_coverage) and ~3.6 min +(local_morans_size) of single-process time, below the --jobs 0 floor. +--jobs output is identical to the serial output at the same seed except the +"seconds" column (wall time per study). Report versus gate (PLAN_ROUND_12 decision P12-3): by default the command is a report generator and always exits 0. With --gate it exits 1 when any gated @@ -71,7 +74,8 @@ OUT_MD = os.path.join(ROOT, "docs", "audits", "inference_validation.md") # Studies whose gated rows are release claims (the default run of this script). -# Other studies (fit_replicate_noisy_x, simex, *_low_rel, morans_i_replicate_se) +# Other studies (fit_replicate_noisy_x, simex, *_low_rel, morans_i_replicate_se, +# and the Round 15 spatial_regression_coverage and local_morans_size, D15-1) # report info rows only. Fixed here, not derived from the results, so a study # that silently stops producing gated rows is reported as missing. # Default-method design studies (Round 10) run in every full report and count @@ -195,7 +199,8 @@ def render(df, cfg, label: str, seconds: float) -> str: ("mse_rel_bias_mean", "MSE rel. bias (mean over areas)"), ("mse_rel_bias_abs_max", "max abs MSE rel. bias"), ("eblup_mse_vs_direct", "EBLUP MSE / direct variance"), - ("min_area_coverage", "min per-area coverage"), ("size", "test size")): + ("min_area_coverage", "min per-area coverage"), ("size", "test size"), + ("power", "power"), ("fdr", "FDR")): if k in r and r[k] == r[k] and r[k] is not None: notes.append(f"{lab} {_f(r[k])}") lines.append(f"| {r['study']} | {r['method']} | {r['target']} | {r['reps']} | {_f(r['coverage'])} | " diff --git a/tests/fixtures/mi_ny_change/frozen_api_responses_2019_26.json b/tests/fixtures/mi_ny_change/frozen_api_responses_2019_26.json new file mode 100644 index 0000000..c894f07 --- /dev/null +++ b/tests/fixtures/mi_ny_change/frozen_api_responses_2019_26.json @@ -0,0 +1 @@ +{"_about":"REAL Census Data API responses (2019 acs5, all counties of state 26, get=NAME,group(
)), recorded by `python -m tests.test_manifest_releases --record` for examples/manifests/mi_ny_income_change.json; columns trimmed to the manifest's variables (estimate, MOE and annotations) plus GEO_ID, NAME, state, county.","request":{"year":2019,"dataset":"acs5","geo_level":"county","state_code":"26","county_code":"*"},"requests":{"B19025":[["NAME","B19025_001E","B19025_001EA","B19025_001M","B19025_001MA","GEO_ID","NAME","state","county"],["Allegan County, Michigan","3315158900",null,"72132788",null,"0500000US26005","Allegan County, Michigan","26","005"],["Alger County, Michigan","167414600",null,"11453661",null,"0500000US26003","Alger County, Michigan","26","003"],["Oceana County, Michigan","633535300",null,"28150773",null,"0500000US26127","Oceana County, Michigan","26","127"],["Clare County, Michigan","654276400",null,"31075091",null,"0500000US26035","Clare County, Michigan","26","035"],["Isabella County, Michigan","1574305800",null,"58459757",null,"0500000US26073","Isabella County, Michigan","26","073"],["Tuscola County, Michigan","1364732800",null,"37329590",null,"0500000US26157","Tuscola County, Michigan","26","157"],["Alpena County, Michigan","732576600",null,"30184713",null,"0500000US26007","Alpena County, Michigan","26","007"],["Midland County, Michigan","2972250000",null,"99379584",null,"0500000US26111","Midland County, Michigan","26","111"],["Oscoda County, Michigan","209255600",null,"17114154",null,"0500000US26135","Oscoda County, Michigan","26","135"],["Clinton County, Michigan","2634301500",null,"73667327",null,"0500000US26037","Clinton County, Michigan","26","037"],["Gratiot County, Michigan","952001100",null,"38961857",null,"0500000US26057","Gratiot County, Michigan","26","057"],["Lenawee County, Michigan","2691252100",null,"77786240",null,"0500000US26091","Lenawee County, Michigan","26","091"],["Montcalm County, Michigan","1479452000",null,"43346911",null,"0500000US26117","Montcalm County, Michigan","26","117"],["St. Clair County, Michigan","4838696400",null,"97768415",null,"0500000US26147","St. Clair County, Michigan","26","147"],["Washtenaw County, Michigan","14376886000",null,"239398946",null,"0500000US26161","Washtenaw County, Michigan","26","161"],["Cass County, Michigan","1590199200",null,"77674707",null,"0500000US26027","Cass County, Michigan","26","027"],["Roscommon County, Michigan","600784100",null,"35761864",null,"0500000US26143","Roscommon County, Michigan","26","143"],["Branch County, Michigan","1079671200",null,"38402359",null,"0500000US26023","Branch County, Michigan","26","023"],["Jackson County, Michigan","4361320600",null,"105894301",null,"0500000US26075","Jackson County, Michigan","26","075"],["Macomb County, Michigan","27625731500",null,"294043217",null,"0500000US26099","Macomb County, Michigan","26","099"],["Baraga County, Michigan","188588000",null,"14639286",null,"0500000US26013","Baraga County, Michigan","26","013"],["Eaton County, Michigan","3586724400",null,"97765099",null,"0500000US26045","Eaton County, Michigan","26","045"],["Gogebic County, Michigan","375608200",null,"30626511",null,"0500000US26053","Gogebic County, Michigan","26","053"],["Chippewa County, Michigan","887134000",null,"74770022",null,"0500000US26033","Chippewa County, Michigan","26","033"],["Monroe County, Michigan","4656633200",null,"123619587",null,"0500000US26115","Monroe County, Michigan","26","115"],["Oakland County, Michigan","54875948000",null,"563314318",null,"0500000US26125","Oakland County, Michigan","26","125"],["Hillsdale County, Michigan","1152260900",null,"35556109",null,"0500000US26059","Hillsdale County, Michigan","26","059"],["Muskegon County, Michigan","4318932100",null,"95555332",null,"0500000US26121","Muskegon County, Michigan","26","121"],["Leelanau County, Michigan","823959000",null,"34220821",null,"0500000US26089","Leelanau County, Michigan","26","089"],["Lapeer County, Michigan","2634179500",null,"79432076",null,"0500000US26087","Lapeer County, Michigan","26","087"],["Sanilac County, Michigan","1063846200",null,"31184290",null,"0500000US26151","Sanilac County, Michigan","26","151"],["Keweenaw County, Michigan","76488400",null,"7677590",null,"0500000US26083","Keweenaw County, Michigan","26","083"],["Ontonagon County, Michigan","143081600",null,"7515806",null,"0500000US26131","Ontonagon County, Michigan","26","131"],["Ottawa County, Michigan","8969169900",null,"159609568",null,"0500000US26139","Ottawa County, Michigan","26","139"],["Alcona County, Michigan","261547700",null,"12317445",null,"0500000US26001","Alcona County, Michigan","26","001"],["Ionia County, Michigan","1554175400",null,"40008554",null,"0500000US26067","Ionia County, Michigan","26","067"],["Kalkaska County, Michigan","420577400",null,"19597813",null,"0500000US26079","Kalkaska County, Michigan","26","079"],["Berrien County, Michigan","4674824000",null,"128110292",null,"0500000US26021","Berrien County, Michigan","26","021"],["Ogemaw County, Michigan","494042100",null,"21107070",null,"0500000US26129","Ogemaw County, Michigan","26","129"],["Wexford County, Michigan","814911200",null,"47600440",null,"0500000US26165","Wexford County, Michigan","26","165"],["Crawford County, Michigan","361865400",null,"19718030",null,"0500000US26039","Crawford County, Michigan","26","039"],["Dickinson County, Michigan","745023500",null,"45816563",null,"0500000US26043","Dickinson County, Michigan","26","043"],["Huron County, Michigan","862127300",null,"25634974",null,"0500000US26063","Huron County, Michigan","26","063"],["Ingham County, Michigan","8201401700",null,"135738723",null,"0500000US26065","Ingham County, Michigan","26","065"],["Mason County, Michigan","838078800",null,"54840464",null,"0500000US26105","Mason County, Michigan","26","105"],["Menominee County, Michigan","636581700",null,"40855391",null,"0500000US26109","Menominee County, Michigan","26","109"],["Missaukee County, Michigan","356974600",null,"12290237",null,"0500000US26113","Missaukee County, Michigan","26","113"],["St. Joseph County, Michigan","1588891100",null,"55156383",null,"0500000US26149","St. Joseph County, Michigan","26","149"],["Van Buren County, Michigan","2099426400",null,"79635941",null,"0500000US26159","Van Buren County, Michigan","26","159"],["Arenac County, Michigan","357799900",null,"17344953",null,"0500000US26011","Arenac County, Michigan","26","011"],["Barry County, Michigan","1932428400",null,"74923417",null,"0500000US26015","Barry County, Michigan","26","015"],["Iron County, Michigan","279541000",null,"15610275",null,"0500000US26071","Iron County, Michigan","26","071"],["Benzie County, Michigan","475895200",null,"22595514",null,"0500000US26019","Benzie County, Michigan","26","019"],["Schoolcraft County, Michigan","192543500",null,"10386217",null,"0500000US26153","Schoolcraft County, Michigan","26","153"],["Wayne County, Michigan","46331691700",null,"365376903",null,"0500000US26163","Wayne County, Michigan","26","163"],["Grand Traverse County, Michigan","3187227600",null,"130011445",null,"0500000US26055","Grand Traverse County, Michigan","26","055"],["Mackinac County, Michigan","315400700",null,"21833216",null,"0500000US26097","Mackinac County, Michigan","26","097"],["Genesee County, Michigan","11008352900",null,"154626389",null,"0500000US26049","Genesee County, Michigan","26","049"],["Antrim County, Michigan","744638800",null,"33258054",null,"0500000US26009","Antrim County, Michigan","26","009"],["Houghton County, Michigan","791861400",null,"31562314",null,"0500000US26061","Houghton County, Michigan","26","061"],["Livingston County, Michigan","7490044500",null,"124416846",null,"0500000US26093","Livingston County, Michigan","26","093"],["Iosco County, Michigan","630601700",null,"27697577",null,"0500000US26069","Iosco County, Michigan","26","069"],["Lake County, Michigan","218489800",null,"11301715",null,"0500000US26085","Lake County, Michigan","26","085"],["Manistee County, Michigan","609632700",null,"27969031",null,"0500000US26101","Manistee County, Michigan","26","101"],["Montmorency County, Michigan","221918600",null,"11764832",null,"0500000US26119","Montmorency County, Michigan","26","119"],["Presque Isle County, Michigan","356448100",null,"19873798",null,"0500000US26141","Presque Isle County, Michigan","26","141"],["Gladwin County, Michigan","642160100",null,"21792863",null,"0500000US26051","Gladwin County, Michigan","26","051"],["Kalamazoo County, Michigan","8192263400",null,"137103685",null,"0500000US26077","Kalamazoo County, Michigan","26","077"],["Mecosta County, Michigan","947913600",null,"42927375",null,"0500000US26107","Mecosta County, Michigan","26","107"],["Marquette County, Michigan","1791959800",null,"52301606",null,"0500000US26103","Marquette County, Michigan","26","103"],["Newaygo County, Michigan","1232633100",null,"37892735",null,"0500000US26123","Newaygo County, Michigan","26","123"],["Osceola County, Michigan","510962200",null,"16559884",null,"0500000US26133","Osceola County, Michigan","26","133"],["Otsego County, Michigan","658302200",null,"35894213",null,"0500000US26137","Otsego County, Michigan","26","137"],["Calhoun County, Michigan","3492296300",null,"88371532",null,"0500000US26025","Calhoun County, Michigan","26","025"],["Cheboygan County, Michigan","660067500",null,"30877466",null,"0500000US26031","Cheboygan County, Michigan","26","031"],["Luce County, Michigan","129268200",null,"11642172",null,"0500000US26095","Luce County, Michigan","26","095"],["Kent County, Michigan","20561505300",null,"270889118",null,"0500000US26081","Kent County, Michigan","26","081"],["Saginaw County, Michigan","5305962400",null,"131992425",null,"0500000US26145","Saginaw County, Michigan","26","145"],["Shiawassee County, Michigan","1915910600",null,"60702227",null,"0500000US26155","Shiawassee County, Michigan","26","155"],["Bay County, Michigan","2811752600",null,"80343982",null,"0500000US26017","Bay County, Michigan","26","017"],["Charlevoix County, Michigan","874184800",null,"29188607",null,"0500000US26029","Charlevoix County, Michigan","26","029"],["Delta County, Michigan","1006176900",null,"44230838",null,"0500000US26041","Delta County, Michigan","26","041"],["Emmet County, Michigan","1113483400",null,"46045796",null,"0500000US26047","Emmet County, Michigan","26","047"]],"B19001":[["NAME","B19001_001E","B19001_001EA","B19001_001M","B19001_001MA","GEO_ID","NAME","state","county"],["Allegan County, Michigan","43416",null,"583",null,"0500000US26005","Allegan County, Michigan","26","005"],["Alger County, Michigan","3007",null,"180",null,"0500000US26003","Alger County, Michigan","26","003"],["Oceana County, Michigan","10156",null,"232",null,"0500000US26127","Oceana County, Michigan","26","127"],["Clare County, Michigan","12199",null,"320",null,"0500000US26035","Clare County, Michigan","26","035"],["Isabella County, Michigan","24739",null,"430",null,"0500000US26073","Isabella County, Michigan","26","073"],["Tuscola County, Michigan","21777",null,"334",null,"0500000US26157","Tuscola County, Michigan","26","157"],["Alpena County, Michigan","12752",null,"283",null,"0500000US26007","Alpena County, Michigan","26","007"],["Midland County, Michigan","34199",null,"413",null,"0500000US26111","Midland County, Michigan","26","111"],["Oscoda County, Michigan","3806",null,"171",null,"0500000US26135","Oscoda County, Michigan","26","135"],["Clinton County, Michigan","29728",null,"298",null,"0500000US26037","Clinton County, Michigan","26","037"],["Gratiot County, Michigan","15035",null,"289",null,"0500000US26057","Gratiot County, Michigan","26","057"],["Lenawee County, Michigan","38345",null,"523",null,"0500000US26091","Lenawee County, Michigan","26","091"],["Montcalm County, Michigan","23913",null,"361",null,"0500000US26117","Montcalm County, Michigan","26","117"],["St. Clair County, Michigan","64850",null,"525",null,"0500000US26147","St. Clair County, Michigan","26","147"],["Washtenaw County, Michigan","141245",null,"943",null,"0500000US26161","Washtenaw County, Michigan","26","161"],["Cass County, Michigan","21019",null,"328",null,"0500000US26027","Cass County, Michigan","26","027"],["Roscommon County, Michigan","11139",null,"356",null,"0500000US26143","Roscommon County, Michigan","26","143"],["Branch County, Michigan","16650",null,"293",null,"0500000US26023","Branch County, Michigan","26","023"],["Jackson County, Michigan","61805",null,"707",null,"0500000US26075","Jackson County, Michigan","26","075"],["Macomb County, Michigan","346402",null,"1325",null,"0500000US26099","Macomb County, Michigan","26","099"],["Baraga County, Michigan","3107",null,"152",null,"0500000US26013","Baraga County, Michigan","26","013"],["Eaton County, Michigan","44480",null,"409",null,"0500000US26045","Eaton County, Michigan","26","045"],["Gogebic County, Michigan","6744",null,"257",null,"0500000US26053","Gogebic County, Michigan","26","053"],["Chippewa County, Michigan","13999",null,"337",null,"0500000US26033","Chippewa County, Michigan","26","033"],["Monroe County, Michigan","59929",null,"547",null,"0500000US26115","Monroe County, Michigan","26","115"],["Oakland County, Michigan","504585",null,"1837",null,"0500000US26125","Oakland County, Michigan","26","125"],["Hillsdale County, Michigan","18107",null,"280",null,"0500000US26059","Hillsdale County, Michigan","26","059"],["Muskegon County, Michigan","65939",null,"653",null,"0500000US26121","Muskegon County, Michigan","26","121"],["Leelanau County, Michigan","9139",null,"217",null,"0500000US26089","Leelanau County, Michigan","26","089"],["Lapeer County, Michigan","33700",null,"386",null,"0500000US26087","Lapeer County, Michigan","26","087"],["Sanilac County, Michigan","17499",null,"286",null,"0500000US26151","Sanilac County, Michigan","26","151"],["Keweenaw County, Michigan","1081",null,"74",null,"0500000US26083","Keweenaw County, Michigan","26","083"],["Ontonagon County, Michigan","2793",null,"103",null,"0500000US26131","Ontonagon County, Michigan","26","131"],["Ottawa County, Michigan","102610",null,"669",null,"0500000US26139","Ottawa County, Michigan","26","139"],["Alcona County, Michigan","4988",null,"139",null,"0500000US26001","Alcona County, Michigan","26","001"],["Ionia County, Michigan","22964",null,"410",null,"0500000US26067","Ionia County, Michigan","26","067"],["Kalkaska County, Michigan","7145",null,"212",null,"0500000US26079","Kalkaska County, Michigan","26","079"],["Berrien County, Michigan","63665",null,"506",null,"0500000US26021","Berrien County, Michigan","26","021"],["Ogemaw County, Michigan","9184",null,"237",null,"0500000US26129","Ogemaw County, Michigan","26","129"],["Wexford County, Michigan","12963",null,"291",null,"0500000US26165","Wexford County, Michigan","26","165"],["Crawford County, Michigan","6141",null,"261",null,"0500000US26039","Crawford County, Michigan","26","039"],["Dickinson County, Michigan","11231",null,"311",null,"0500000US26043","Dickinson County, Michigan","26","043"],["Huron County, Michigan","13847",null,"273",null,"0500000US26063","Huron County, Michigan","26","063"],["Ingham County, Michigan","112840",null,"865",null,"0500000US26065","Ingham County, Michigan","26","065"],["Mason County, Michigan","12186",null,"295",null,"0500000US26105","Mason County, Michigan","26","105"],["Menominee County, Michigan","10627",null,"268",null,"0500000US26109","Menominee County, Michigan","26","109"],["Missaukee County, Michigan","6055",null,"149",null,"0500000US26113","Missaukee County, Michigan","26","113"],["St. Joseph County, Michigan","24150",null,"319",null,"0500000US26149","St. Joseph County, Michigan","26","149"],["Van Buren County, Michigan","29411",null,"528",null,"0500000US26159","Van Buren County, Michigan","26","159"],["Arenac County, Michigan","6571",null,"180",null,"0500000US26011","Arenac County, Michigan","26","011"],["Barry County, Michigan","24296",null,"400",null,"0500000US26015","Barry County, Michigan","26","015"],["Iron County, Michigan","5225",null,"162",null,"0500000US26071","Iron County, Michigan","26","071"],["Benzie County, Michigan","6792",null,"196",null,"0500000US26019","Benzie County, Michigan","26","019"],["Schoolcraft County, Michigan","3468",null,"139",null,"0500000US26153","Schoolcraft County, Michigan","26","153"],["Wayne County, Michigan","682282",null,"2593",null,"0500000US26163","Wayne County, Michigan","26","163"],["Grand Traverse County, Michigan","37319",null,"597",null,"0500000US26055","Grand Traverse County, Michigan","26","055"],["Mackinac County, Michigan","5269",null,"250",null,"0500000US26097","Mackinac County, Michigan","26","097"],["Genesee County, Michigan","167902",null,"1068",null,"0500000US26049","Genesee County, Michigan","26","049"],["Antrim County, Michigan","9899",null,"265",null,"0500000US26009","Antrim County, Michigan","26","009"],["Houghton County, Michigan","13386",null,"312",null,"0500000US26061","Houghton County, Michigan","26","061"],["Livingston County, Michigan","72000",null,"450",null,"0500000US26093","Livingston County, Michigan","26","093"],["Iosco County, Michigan","11669",null,"265",null,"0500000US26069","Iosco County, Michigan","26","069"],["Lake County, Michigan","4631",null,"116",null,"0500000US26085","Lake County, Michigan","26","085"],["Manistee County, Michigan","9426",null,"275",null,"0500000US26101","Manistee County, Michigan","26","101"],["Montmorency County, Michigan","4452",null,"174",null,"0500000US26119","Montmorency County, Michigan","26","119"],["Presque Isle County, Michigan","5797",null,"134",null,"0500000US26141","Presque Isle County, Michigan","26","141"],["Gladwin County, Michigan","11047",null,"206",null,"0500000US26051","Gladwin County, Michigan","26","051"],["Kalamazoo County, Michigan","103445",null,"875",null,"0500000US26077","Kalamazoo County, Michigan","26","077"],["Mecosta County, Michigan","15808",null,"405",null,"0500000US26107","Mecosta County, Michigan","26","107"],["Marquette County, Michigan","26552",null,"426",null,"0500000US26103","Marquette County, Michigan","26","103"],["Newaygo County, Michigan","19161",null,"275",null,"0500000US26123","Newaygo County, Michigan","26","123"],["Osceola County, Michigan","9181",null,"162",null,"0500000US26133","Osceola County, Michigan","26","133"],["Otsego County, Michigan","9944",null,"241",null,"0500000US26137","Otsego County, Michigan","26","137"],["Calhoun County, Michigan","53991",null,"612",null,"0500000US26025","Calhoun County, Michigan","26","025"],["Cheboygan County, Michigan","11195",null,"309",null,"0500000US26031","Cheboygan County, Michigan","26","031"],["Luce County, Michigan","2231",null,"160",null,"0500000US26095","Luce County, Michigan","26","095"],["Kent County, Michigan","241746",null,"975",null,"0500000US26081","Kent County, Michigan","26","081"],["Saginaw County, Michigan","78933",null,"654",null,"0500000US26145","Saginaw County, Michigan","26","145"],["Shiawassee County, Michigan","27593",null,"295",null,"0500000US26155","Shiawassee County, Michigan","26","155"],["Bay County, Michigan","44339",null,"476",null,"0500000US26017","Bay County, Michigan","26","017"],["Charlevoix County, Michigan","11503",null,"245",null,"0500000US26029","Charlevoix County, Michigan","26","029"],["Delta County, Michigan","16234",null,"256",null,"0500000US26041","Delta County, Michigan","26","041"],["Emmet County, Michigan","14463",null,"326",null,"0500000US26047","Emmet County, Michigan","26","047"]]}} diff --git a/tests/fixtures/mi_ny_change/frozen_api_responses_2019_36.json b/tests/fixtures/mi_ny_change/frozen_api_responses_2019_36.json new file mode 100644 index 0000000..50afe58 --- /dev/null +++ b/tests/fixtures/mi_ny_change/frozen_api_responses_2019_36.json @@ -0,0 +1 @@ +{"_about":"REAL Census Data API responses (2019 acs5, all counties of state 36, get=NAME,group(
)), recorded by `python -m tests.test_manifest_releases --record` for examples/manifests/mi_ny_income_change.json; columns trimmed to the manifest's variables (estimate, MOE and annotations) plus GEO_ID, NAME, state, county.","request":{"year":2019,"dataset":"acs5","geo_level":"county","state_code":"36","county_code":"*"},"requests":{"B19025":[["NAME","B19025_001E","B19025_001EA","B19025_001M","B19025_001MA","GEO_ID","NAME","state","county"],["Schoharie County, New York","910637300",null,"43019909",null,"0500000US36095","Schoharie County, New York","36","095"],["Fulton County, New York","1451703700",null,"57953004",null,"0500000US36035","Fulton County, New York","36","035"],["Rensselaer County, New York","5571198200",null,"109161071",null,"0500000US36083","Rensselaer County, New York","36","083"],["Franklin County, New York","1250088300",null,"48436319",null,"0500000US36033","Franklin County, New York","36","033"],["Queens County, New York","70042528000",null,"661727873",null,"0500000US36081","Queens County, New York","36","081"],["Washington County, New York","1710388600",null,"71069528",null,"0500000US36115","Washington County, New York","36","115"],["New York County, New York","121187596900",null,"1728227853",null,"0500000US36061","New York County, New York","36","061"],["Cayuga County, New York","2343686000",null,"68209598",null,"0500000US36011","Cayuga County, New York","36","011"],["Rockland County, New York","12422152100",null,"234382049",null,"0500000US36087","Rockland County, New York","36","087"],["Niagara County, New York","6416233800",null,"133941096",null,"0500000US36063","Niagara County, New York","36","063"],["Essex County, New York","1165711500",null,"52173271",null,"0500000US36031","Essex County, New York","36","031"],["Nassau County, New York","68291348500",null,"596781517",null,"0500000US36059","Nassau County, New York","36","059"],["Schuyler County, New York","477818500",null,"27692558",null,"0500000US36097","Schuyler County, New York","36","097"],["Chenango County, New York","1354171300",null,"48060122",null,"0500000US36017","Chenango County, New York","36","017"],["Jefferson County, New York","2817558400",null,"78361317",null,"0500000US36045","Jefferson County, New York","36","045"],["Suffolk County, New York","63213626500",null,"554742938",null,"0500000US36103","Suffolk County, New York","36","103"],["Wyoming County, New York","1107239400",null,"40231366",null,"0500000US36121","Wyoming County, New York","36","121"],["Oswego County, New York","3300732300",null,"118444858",null,"0500000US36075","Oswego County, New York","36","075"],["Wayne County, New York","2775674700",null,"89460663",null,"0500000US36117","Wayne County, New York","36","117"],["Broome County, New York","5458481000",null,"124241173",null,"0500000US36007","Broome County, New York","36","007"],["Erie County, New York","30390997600",null,"299298217",null,"0500000US36029","Erie County, New York","36","029"],["Sullivan County, New York","2175712000",null,"98202550",null,"0500000US36105","Sullivan County, New York","36","105"],["Orange County, New York","12924205800",null,"216490662",null,"0500000US36071","Orange County, New York","36","071"],["Bronx County, New York","29594741700",null,"457700840",null,"0500000US36005","Bronx County, New York","36","005"],["Onondaga County, New York","15301927400",null,"206080832",null,"0500000US36067","Onondaga County, New York","36","067"],["Clinton County, New York","2192580100",null,"88172025",null,"0500000US36019","Clinton County, New York","36","019"],["Seneca County, New York","969771500",null,"40060145",null,"0500000US36099","Seneca County, New York","36","099"],["Putnam County, New York","4569211200",null,"180359431",null,"0500000US36079","Putnam County, New York","36","079"],["Herkimer County, New York","1656149500",null,"54395534",null,"0500000US36043","Herkimer County, New York","36","043"],["Ontario County, New York","3952691800",null,"117125910",null,"0500000US36069","Ontario County, New York","36","069"],["Steuben County, New York","2850683100",null,"83323520",null,"0500000US36101","Steuben County, New York","36","101"],["Hamilton County, New York","79750600",null,"8857723",null,"0500000US36041","Hamilton County, New York","36","041"],["Chautauqua County, New York","3279239000",null,"79420390",null,"0500000US36013","Chautauqua County, New York","36","013"],["Kings County, New York","87814147800",null,"813871938",null,"0500000US36047","Kings County, New York","36","047"],["Tioga County, New York","1537788200",null,"59231072",null,"0500000US36107","Tioga County, New York","36","107"],["Orleans County, New York","1104352400",null,"53765850",null,"0500000US36073","Orleans County, New York","36","073"],["Saratoga County, New York","9833795000",null,"213163134",null,"0500000US36091","Saratoga County, New York","36","091"],["Columbia County, New York","2259625400",null,"85155644",null,"0500000US36021","Columbia County, New York","36","021"],["Lewis County, New York","689682600",null,"27119027",null,"0500000US36049","Lewis County, New York","36","049"],["Westchester County, New York","53970073700",null,"651799130",null,"0500000US36119","Westchester County, New York","36","119"],["Delaware County, New York","1202886300",null,"47990293",null,"0500000US36025","Delaware County, New York","36","025"],["Montgomery County, New York","1276093300",null,"45149327",null,"0500000US36057","Montgomery County, New York","36","057"],["Richmond County, New York","17394183200",null,"315115336",null,"0500000US36085","Richmond County, New York","36","085"],["St. Lawrence County, New York","2732358500",null,"66453078",null,"0500000US36089","St. Lawrence County, New York","36","089"],["Tompkins County, New York","3325929300",null,"118347079",null,"0500000US36109","Tompkins County, New York","36","109"],["Warren County, New York","2242084900",null,"81871185",null,"0500000US36113","Warren County, New York","36","113"],["Cortland County, New York","1315832700",null,"82397634",null,"0500000US36023","Cortland County, New York","36","023"],["Dutchess County, New York","11387031800",null,"207586777",null,"0500000US36027","Dutchess County, New York","36","027"],["Monroe County, New York","24362520700",null,"279157835",null,"0500000US36055","Monroe County, New York","36","055"],["Oneida County, New York","6607825300",null,"156919494",null,"0500000US36065","Oneida County, New York","36","065"],["Ulster County, New York","5996168400",null,"179842082",null,"0500000US36111","Ulster County, New York","36","111"],["Chemung County, New York","2402253200",null,"92199999",null,"0500000US36015","Chemung County, New York","36","015"],["Livingston County, New York","1771562000",null,"69288380",null,"0500000US36051","Livingston County, New York","36","051"],["Otsego County, New York","1648118000",null,"65351499",null,"0500000US36077","Otsego County, New York","36","077"],["Greene County, New York","1207818200",null,"51945557",null,"0500000US36039","Greene County, New York","36","039"],["Albany County, New York","11295896400",null,"208838708",null,"0500000US36001","Albany County, New York","36","001"],["Allegany County, New York","1127324500",null,"85760991",null,"0500000US36003","Allegany County, New York","36","003"],["Cattaraugus County, New York","1950224200",null,"61942203",null,"0500000US36009","Cattaraugus County, New York","36","009"],["Madison County, New York","2034100300",null,"72326017",null,"0500000US36053","Madison County, New York","36","053"],["Schenectady County, New York","4508393400",null,"114545566",null,"0500000US36093","Schenectady County, New York","36","093"],["Genesee County, New York","1731891500",null,"45299698",null,"0500000US36037","Genesee County, New York","36","037"],["Yates County, New York","667824700",null,"37059002",null,"0500000US36123","Yates County, New York","36","123"]],"B19001":[["NAME","B19001_001E","B19001_001EA","B19001_001M","B19001_001MA","GEO_ID","NAME","state","county"],["Schoharie County, New York","12559",null,"322",null,"0500000US36095","Schoharie County, New York","36","095"],["Fulton County, New York","22557",null,"505",null,"0500000US36035","Fulton County, New York","36","035"],["Rensselaer County, New York","64906",null,"574",null,"0500000US36083","Rensselaer County, New York","36","083"],["Franklin County, New York","19015",null,"375",null,"0500000US36033","Franklin County, New York","36","033"],["Queens County, New York","778932",null,"2366",null,"0500000US36081","Queens County, New York","36","081"],["Washington County, New York","24014",null,"435",null,"0500000US36115","Washington County, New York","36","115"],["New York County, New York","759460",null,"4629",null,"0500000US36061","New York County, New York","36","061"],["Cayuga County, New York","31221",null,"485",null,"0500000US36011","Cayuga County, New York","36","011"],["Rockland County, New York","100438",null,"598",null,"0500000US36087","Rockland County, New York","36","087"],["Niagara County, New York","88519",null,"733",null,"0500000US36063","Niagara County, New York","36","063"],["Essex County, New York","15790",null,"448",null,"0500000US36031","Essex County, New York","36","031"],["Nassau County, New York","446977",null,"1301",null,"0500000US36059","Nassau County, New York","36","059"],["Schuyler County, New York","7324",null,"214",null,"0500000US36097","Schuyler County, New York","36","097"],["Chenango County, New York","20697",null,"344",null,"0500000US36017","Chenango County, New York","36","017"],["Jefferson County, New York","42734",null,"711",null,"0500000US36045","Jefferson County, New York","36","045"],["Suffolk County, New York","489301",null,"2144",null,"0500000US36103","Suffolk County, New York","36","103"],["Wyoming County, New York","15917",null,"245",null,"0500000US36121","Wyoming County, New York","36","121"],["Oswego County, New York","46079",null,"691",null,"0500000US36075","Oswego County, New York","36","075"],["Wayne County, New York","36720",null,"454",null,"0500000US36117","Wayne County, New York","36","117"],["Broome County, New York","78549",null,"784",null,"0500000US36007","Broome County, New York","36","007"],["Erie County, New York","389585",null,"1634",null,"0500000US36029","Erie County, New York","36","029"],["Sullivan County, New York","28184",null,"635",null,"0500000US36105","Sullivan County, New York","36","105"],["Orange County, New York","128016",null,"950",null,"0500000US36071","Orange County, New York","36","071"],["Bronx County, New York","503829",null,"1325",null,"0500000US36005","Bronx County, New York","36","005"],["Onondaga County, New York","185324",null,"1191",null,"0500000US36067","Onondaga County, New York","36","067"],["Clinton County, New York","31301",null,"553",null,"0500000US36019","Clinton County, New York","36","019"],["Seneca County, New York","13564",null,"289",null,"0500000US36099","Seneca County, New York","36","099"],["Putnam County, New York","34854",null,"500",null,"0500000US36079","Putnam County, New York","36","079"],["Herkimer County, New York","24524",null,"431",null,"0500000US36043","Herkimer County, New York","36","043"],["Ontario County, New York","44787",null,"541",null,"0500000US36069","Ontario County, New York","36","069"],["Steuben County, New York","39861",null,"451",null,"0500000US36101","Steuben County, New York","36","101"],["Hamilton County, New York","1157",null,"105",null,"0500000US36041","Hamilton County, New York","36","041"],["Chautauqua County, New York","53075",null,"619",null,"0500000US36013","Chautauqua County, New York","36","013"],["Kings County, New York","958567",null,"2485",null,"0500000US36047","Kings County, New York","36","047"],["Tioga County, New York","20030",null,"356",null,"0500000US36107","Tioga County, New York","36","107"],["Orleans County, New York","16563",null,"340",null,"0500000US36073","Orleans County, New York","36","073"],["Saratoga County, New York","94406",null,"839",null,"0500000US36091","Saratoga County, New York","36","091"],["Columbia County, New York","24966",null,"497",null,"0500000US36021","Columbia County, New York","36","021"],["Lewis County, New York","10247",null,"231",null,"0500000US36049","Lewis County, New York","36","049"],["Westchester County, New York","349292",null,"1512",null,"0500000US36119","Westchester County, New York","36","119"],["Delaware County, New York","18968",null,"485",null,"0500000US36025","Delaware County, New York","36","025"],["Montgomery County, New York","19660",null,"291",null,"0500000US36057","Montgomery County, New York","36","057"],["Richmond County, New York","166246",null,"1095",null,"0500000US36085","Richmond County, New York","36","085"],["St. Lawrence County, New York","41933",null,"533",null,"0500000US36089","St. Lawrence County, New York","36","089"],["Tompkins County, New York","39618",null,"531",null,"0500000US36109","Tompkins County, New York","36","109"],["Warren County, New York","28015",null,"428",null,"0500000US36113","Warren County, New York","36","113"],["Cortland County, New York","17745",null,"397",null,"0500000US36023","Cortland County, New York","36","023"],["Dutchess County, New York","108413",null,"900",null,"0500000US36027","Dutchess County, New York","36","027"],["Monroe County, New York","301948",null,"1547",null,"0500000US36055","Monroe County, New York","36","055"],["Oneida County, New York","89729",null,"797",null,"0500000US36065","Oneida County, New York","36","065"],["Ulster County, New York","69320",null,"753",null,"0500000US36111","Ulster County, New York","36","111"],["Chemung County, New York","33989",null,"525",null,"0500000US36015","Chemung County, New York","36","015"],["Livingston County, New York","24176",null,"400",null,"0500000US36051","Livingston County, New York","36","051"],["Otsego County, New York","23409",null,"458",null,"0500000US36077","Otsego County, New York","36","077"],["Greene County, New York","17100",null,"457",null,"0500000US36039","Greene County, New York","36","039"],["Albany County, New York","126540",null,"951",null,"0500000US36001","Albany County, New York","36","001"],["Allegany County, New York","17948",null,"351",null,"0500000US36003","Allegany County, New York","36","003"],["Cattaraugus County, New York","31779",null,"318",null,"0500000US36009","Cattaraugus County, New York","36","009"],["Madison County, New York","25877",null,"574",null,"0500000US36053","Madison County, New York","36","053"],["Schenectady County, New York","54302",null,"825",null,"0500000US36093","Schenectady County, New York","36","093"],["Genesee County, New York","23759",null,"339",null,"0500000US36037","Genesee County, New York","36","037"],["Yates County, New York","8919",null,"351",null,"0500000US36123","Yates County, New York","36","123"]]}} diff --git a/tests/fixtures/mi_ny_change/frozen_api_responses_2019_us.json b/tests/fixtures/mi_ny_change/frozen_api_responses_2019_us.json new file mode 100644 index 0000000..1800877 --- /dev/null +++ b/tests/fixtures/mi_ny_change/frozen_api_responses_2019_us.json @@ -0,0 +1 @@ +{"_about":"REAL Census Data API responses (2019 acs5, nation, get=NAME,group(
)), untrimmed, read by the continuity check of a multi-release extraction; recorded by `python -m tests.test_manifest_releases --record`.","request":{"year":2019,"dataset":"acs5","geo_level":"us","state_code":null},"requests":{"B19025":[["NAME","B19025_001E","B19025_001EA","B19025_001M","B19025_001MA","GEO_ID","NAME","us"],["United States","10699792935700",null,"43081188883",null,"0100000US","United States","1"]],"B19001":[["NAME","B19001_001E","B19001_001EA","B19001_001M","B19001_001MA","B19001_002E","B19001_002EA","B19001_002M","B19001_002MA","B19001_003E","B19001_003EA","B19001_003M","B19001_003MA","B19001_004E","B19001_004EA","B19001_004M","B19001_004MA","B19001_005E","B19001_005EA","B19001_005M","B19001_005MA","B19001_006E","B19001_006EA","B19001_006M","B19001_006MA","B19001_007E","B19001_007EA","B19001_007M","B19001_007MA","B19001_008E","B19001_008EA","B19001_008M","B19001_008MA","B19001_009E","B19001_009EA","B19001_009M","B19001_009MA","B19001_010E","B19001_010EA","B19001_010M","B19001_010MA","B19001_011E","B19001_011EA","B19001_011M","B19001_011MA","B19001_012E","B19001_012EA","B19001_012M","B19001_012MA","B19001_013E","B19001_013EA","B19001_013M","B19001_013MA","B19001_014E","B19001_014EA","B19001_014M","B19001_014MA","B19001_015E","B19001_015EA","B19001_015M","B19001_015MA","B19001_016E","B19001_016EA","B19001_016M","B19001_016MA","B19001_017E","B19001_017EA","B19001_017M","B19001_017MA","GEO_ID","NAME","us"],["United States","120756048",null,"236892",null,"7302871",null,"18046",null,"5189583",null,"15224",null,"5300624",null,"15337",null,"5459520",null,"16244",null,"5285952",null,"16670",null,"5506182",null,"18565",null,"5093804",null,"14176",null,"5081209",null,"16048",null,"4647032",null,"14255",null,"9090005",null,"22946",null,"11699885",null,"30719",null,"15374617",null,"51444",null,"11019036",null,"49867",null,"7267775",null,"40108",null,"8173563",null,"49077",null,"9264390",null,"54033",null,"0100000US","United States","1"]]}} diff --git a/tests/fixtures/mi_ny_change/frozen_api_responses_2024_26.json b/tests/fixtures/mi_ny_change/frozen_api_responses_2024_26.json new file mode 100644 index 0000000..d8c7394 --- /dev/null +++ b/tests/fixtures/mi_ny_change/frozen_api_responses_2024_26.json @@ -0,0 +1 @@ +{"_about":"REAL Census Data API responses (2024 acs5, all counties of state 26, get=NAME,group(
)), recorded by `python -m tests.test_manifest_releases --record` for examples/manifests/mi_ny_income_change.json; columns trimmed to the manifest's variables (estimate, MOE and annotations) plus GEO_ID, NAME, state, county.","request":{"year":2024,"dataset":"acs5","geo_level":"county","state_code":"26","county_code":"*"},"requests":{"B19025":[["NAME","B19025_001E","B19025_001EA","B19025_001M","B19025_001MA","GEO_ID","NAME","state","county"],["Alcona County, Michigan","388217000",null,"21962228",null,"0500000US26001","Alcona County, Michigan","26","001"],["Alger County, Michigan","258474200",null,"19818815",null,"0500000US26003","Alger County, Michigan","26","003"],["Allegan County, Michigan","4729246200",null,"133857111",null,"0500000US26005","Allegan County, Michigan","26","005"],["Alpena County, Michigan","968852900",null,"69991462",null,"0500000US26007","Alpena County, Michigan","26","007"],["Antrim County, Michigan","1039872900",null,"37468595",null,"0500000US26009","Antrim County, Michigan","26","009"],["Arenac County, Michigan","473174100",null,"19467059",null,"0500000US26011","Arenac County, Michigan","26","011"],["Baraga County, Michigan","246916100",null,"21701666",null,"0500000US26013","Baraga County, Michigan","26","013"],["Barry County, Michigan","2520168600",null,"159273831",null,"0500000US26015","Barry County, Michigan","26","015"],["Bay County, Michigan","3570029300",null,"107176064",null,"0500000US26017","Bay County, Michigan","26","017"],["Benzie County, Michigan","760384000",null,"41349784",null,"0500000US26019","Benzie County, Michigan","26","019"],["Berrien County, Michigan","5748771000",null,"127948647",null,"0500000US26021","Berrien County, Michigan","26","021"],["Branch County, Michigan","1351912600",null,"42920884",null,"0500000US26023","Branch County, Michigan","26","023"],["Calhoun County, Michigan","4447339400",null,"132626151",null,"0500000US26025","Calhoun County, Michigan","26","025"],["Cass County, Michigan","2032697100",null,"109814081",null,"0500000US26027","Cass County, Michigan","26","027"],["Charlevoix County, Michigan","1287246400",null,"65842898",null,"0500000US26029","Charlevoix County, Michigan","26","029"],["Cheboygan County, Michigan","919551300",null,"47178308",null,"0500000US26031","Cheboygan County, Michigan","26","031"],["Chippewa County, Michigan","1077164100",null,"43976009",null,"0500000US26033","Chippewa County, Michigan","26","033"],["Clare County, Michigan","855380000",null,"54129694",null,"0500000US26035","Clare County, Michigan","26","035"],["Clinton County, Michigan","3565400100",null,"140270135",null,"0500000US26037","Clinton County, Michigan","26","037"],["Crawford County, Michigan","428432200",null,"25170663",null,"0500000US26039","Crawford County, Michigan","26","039"],["Delta County, Michigan","1300713400",null,"88290069",null,"0500000US26041","Delta County, Michigan","26","041"],["Dickinson County, Michigan","967016400",null,"52105269",null,"0500000US26043","Dickinson County, Michigan","26","043"],["Eaton County, Michigan","4405144000",null,"122111889",null,"0500000US26045","Eaton County, Michigan","26","045"],["Emmet County, Michigan","1667375000",null,"105243123",null,"0500000US26047","Emmet County, Michigan","26","047"],["Genesee County, Michigan","14066474800",null,"346497599",null,"0500000US26049","Genesee County, Michigan","26","049"],["Gladwin County, Michigan","835367400",null,"35087053",null,"0500000US26051","Gladwin County, Michigan","26","051"],["Gogebic County, Michigan","514581900",null,"40724340",null,"0500000US26053","Gogebic County, Michigan","26","053"],["Grand Traverse County, Michigan","4242212300",null,"149600617",null,"0500000US26055","Grand Traverse County, Michigan","26","055"],["Gratiot County, Michigan","1226486500",null,"67596733",null,"0500000US26057","Gratiot County, Michigan","26","057"],["Hillsdale County, Michigan","1461238000",null,"62472692",null,"0500000US26059","Hillsdale County, Michigan","26","059"],["Houghton County, Michigan","1157255100",null,"55572056",null,"0500000US26061","Houghton County, Michigan","26","061"],["Huron County, Michigan","1084234300",null,"35891414",null,"0500000US26063","Huron County, Michigan","26","063"],["Ingham County, Michigan","10429124400",null,"262236682",null,"0500000US26065","Ingham County, Michigan","26","065"],["Ionia County, Michigan","2068394900",null,"117980396",null,"0500000US26067","Ionia County, Michigan","26","067"],["Iosco County, Michigan","786879300",null,"35529437",null,"0500000US26069","Iosco County, Michigan","26","069"],["Iron County, Michigan","397895000",null,"27180602",null,"0500000US26071","Iron County, Michigan","26","071"],["Isabella County, Michigan","1849026500",null,"72226619",null,"0500000US26073","Isabella County, Michigan","26","073"],["Jackson County, Michigan","5544337400",null,"180169007",null,"0500000US26075","Jackson County, Michigan","26","075"],["Kalamazoo County, Michigan","10731709300",null,"302712457",null,"0500000US26077","Kalamazoo County, Michigan","26","077"],["Kalkaska County, Michigan","636266400",null,"41169697",null,"0500000US26079","Kalkaska County, Michigan","26","079"],["Kent County, Michigan","28131805400",null,"466413860",null,"0500000US26081","Kent County, Michigan","26","081"],["Keweenaw County, Michigan","93204600",null,"12505273",null,"0500000US26083","Keweenaw County, Michigan","26","083"],["Lake County, Michigan","320107000",null,"22304321",null,"0500000US26085","Lake County, Michigan","26","085"],["Lapeer County, Michigan","3367866500",null,"128981314",null,"0500000US26087","Lapeer County, Michigan","26","087"],["Leelanau County, Michigan","1319605500",null,"87216254",null,"0500000US26089","Leelanau County, Michigan","26","089"],["Lenawee County, Michigan","3369490800",null,"100749795",null,"0500000US26091","Lenawee County, Michigan","26","091"],["Livingston County, Michigan","9930275000",null,"180312580",null,"0500000US26093","Livingston County, Michigan","26","093"],["Luce County, Michigan","164781600",null,"13941846",null,"0500000US26095","Luce County, Michigan","26","095"],["Mackinac County, Michigan","420495400",null,"30563702",null,"0500000US26097","Mackinac County, Michigan","26","097"],["Macomb County, Michigan","35454519300",null,"476512386",null,"0500000US26099","Macomb County, Michigan","26","099"],["Manistee County, Michigan","888978800",null,"52192891",null,"0500000US26101","Manistee County, Michigan","26","101"],["Marquette County, Michigan","2318540800",null,"67683942",null,"0500000US26103","Marquette County, Michigan","26","103"],["Mason County, Michigan","1097073600",null,"59327573",null,"0500000US26105","Mason County, Michigan","26","105"],["Mecosta County, Michigan","1240426200",null,"42146196",null,"0500000US26107","Mecosta County, Michigan","26","107"],["Menominee County, Michigan","760472500",null,"35360429",null,"0500000US26109","Menominee County, Michigan","26","109"],["Midland County, Michigan","3644884400",null,"133487732",null,"0500000US26111","Midland County, Michigan","26","111"],["Missaukee County, Michigan","490893100",null,"45061167",null,"0500000US26113","Missaukee County, Michigan","26","113"],["Monroe County, Michigan","6183593200",null,"184997623",null,"0500000US26115","Monroe County, Michigan","26","115"],["Montcalm County, Michigan","1954264100",null,"60600855",null,"0500000US26117","Montcalm County, Michigan","26","117"],["Montmorency County, Michigan","311565200",null,"17581894",null,"0500000US26119","Montmorency County, Michigan","26","119"],["Muskegon County, Michigan","5550660300",null,"141395811",null,"0500000US26121","Muskegon County, Michigan","26","121"],["Newaygo County, Michigan","1552484800",null,"49187064",null,"0500000US26123","Newaygo County, Michigan","26","123"],["Oakland County, Michigan","71143129800",null,"676932606",null,"0500000US26125","Oakland County, Michigan","26","125"],["Oceana County, Michigan","859320400",null,"38050775",null,"0500000US26127","Oceana County, Michigan","26","127"],["Ogemaw County, Michigan","650050300",null,"33031867",null,"0500000US26129","Ogemaw County, Michigan","26","129"],["Ontonagon County, Michigan","212926100",null,"13206401",null,"0500000US26131","Ontonagon County, Michigan","26","131"],["Osceola County, Michigan","707076400",null,"26605401",null,"0500000US26133","Osceola County, Michigan","26","133"],["Oscoda County, Michigan","232636800",null,"17163595",null,"0500000US26135","Oscoda County, Michigan","26","135"],["Otsego County, Michigan","995050900",null,"62975124",null,"0500000US26137","Otsego County, Michigan","26","137"],["Ottawa County, Michigan","12843332000",null,"341138991",null,"0500000US26139","Ottawa County, Michigan","26","139"],["Presque Isle County, Michigan","513527100",null,"37104665",null,"0500000US26141","Presque Isle County, Michigan","26","141"],["Roscommon County, Michigan","902417600",null,"60878618",null,"0500000US26143","Roscommon County, Michigan","26","143"],["Saginaw County, Michigan","6435032700",null,"168383740",null,"0500000US26145","Saginaw County, Michigan","26","145"],["St. Clair County, Michigan","6147486100",null,"189961811",null,"0500000US26147","St. Clair County, Michigan","26","147"],["St. Joseph County, Michigan","1923458100",null,"72595642",null,"0500000US26149","St. Joseph County, Michigan","26","149"],["Sanilac County, Michigan","1364117600",null,"38984498",null,"0500000US26151","Sanilac County, Michigan","26","151"],["Schoolcraft County, Michigan","303071100",null,"23243549",null,"0500000US26153","Schoolcraft County, Michigan","26","153"],["Shiawassee County, Michigan","2383899300",null,"122626702",null,"0500000US26155","Shiawassee County, Michigan","26","155"],["Tuscola County, Michigan","1658196900",null,"54392466",null,"0500000US26157","Tuscola County, Michigan","26","157"],["Van Buren County, Michigan","2630691200",null,"77956663",null,"0500000US26159","Van Buren County, Michigan","26","159"],["Washtenaw County, Michigan","18775357000",null,"317276598",null,"0500000US26161","Washtenaw County, Michigan","26","161"],["Wayne County, Michigan","59973454700",null,"564698043",null,"0500000US26163","Wayne County, Michigan","26","163"],["Wexford County, Michigan","1077922400",null,"50227140",null,"0500000US26165","Wexford County, Michigan","26","165"]],"B19001":[["NAME","B19001_001E","B19001_001EA","B19001_001M","B19001_001MA","GEO_ID","NAME","state","county"],["Alcona County, Michigan","5305",null,"165",null,"0500000US26001","Alcona County, Michigan","26","001"],["Alger County, Michigan","3531",null,"166",null,"0500000US26003","Alger County, Michigan","26","003"],["Allegan County, Michigan","45522",null,"633",null,"0500000US26005","Allegan County, Michigan","26","005"],["Alpena County, Michigan","13019",null,"276",null,"0500000US26007","Alpena County, Michigan","26","007"],["Antrim County, Michigan","11100",null,"298",null,"0500000US26009","Antrim County, Michigan","26","009"],["Arenac County, Michigan","6552",null,"168",null,"0500000US26011","Arenac County, Michigan","26","011"],["Baraga County, Michigan","3433",null,"137",null,"0500000US26013","Baraga County, Michigan","26","013"],["Barry County, Michigan","24480",null,"367",null,"0500000US26015","Barry County, Michigan","26","015"],["Bay County, Michigan","45103",null,"512",null,"0500000US26017","Bay County, Michigan","26","017"],["Benzie County, Michigan","7708",null,"237",null,"0500000US26019","Benzie County, Michigan","26","019"],["Berrien County, Michigan","63999",null,"596",null,"0500000US26021","Berrien County, Michigan","26","021"],["Branch County, Michigan","16906",null,"286",null,"0500000US26023","Branch County, Michigan","26","023"],["Calhoun County, Michigan","53333",null,"559",null,"0500000US26025","Calhoun County, Michigan","26","025"],["Cass County, Michigan","21207",null,"325",null,"0500000US26027","Cass County, Michigan","26","027"],["Charlevoix County, Michigan","12267",null,"327",null,"0500000US26029","Charlevoix County, Michigan","26","029"],["Cheboygan County, Michigan","11159",null,"274",null,"0500000US26031","Cheboygan County, Michigan","26","031"],["Chippewa County, Michigan","14080",null,"360",null,"0500000US26033","Chippewa County, Michigan","26","033"],["Clare County, Michigan","12521",null,"408",null,"0500000US26035","Clare County, Michigan","26","035"],["Clinton County, Michigan","31639",null,"284",null,"0500000US26037","Clinton County, Michigan","26","037"],["Crawford County, Michigan","6231",null,"239",null,"0500000US26039","Crawford County, Michigan","26","039"],["Delta County, Michigan","15971",null,"316",null,"0500000US26041","Delta County, Michigan","26","041"],["Dickinson County, Michigan","11905",null,"254",null,"0500000US26043","Dickinson County, Michigan","26","043"],["Eaton County, Michigan","45428",null,"420",null,"0500000US26045","Eaton County, Michigan","26","045"],["Emmet County, Michigan","15190",null,"413",null,"0500000US26047","Emmet County, Michigan","26","047"],["Genesee County, Michigan","167334",null,"940",null,"0500000US26049","Genesee County, Michigan","26","049"],["Gladwin County, Michigan","11280",null,"293",null,"0500000US26051","Gladwin County, Michigan","26","051"],["Gogebic County, Michigan","6989",null,"265",null,"0500000US26053","Gogebic County, Michigan","26","053"],["Grand Traverse County, Michigan","40287",null,"624",null,"0500000US26055","Grand Traverse County, Michigan","26","055"],["Gratiot County, Michigan","14775",null,"214",null,"0500000US26057","Gratiot County, Michigan","26","057"],["Hillsdale County, Michigan","17860",null,"312",null,"0500000US26059","Hillsdale County, Michigan","26","059"],["Houghton County, Michigan","14305",null,"372",null,"0500000US26061","Houghton County, Michigan","26","061"],["Huron County, Michigan","14150",null,"280",null,"0500000US26063","Huron County, Michigan","26","063"],["Ingham County, Michigan","117096",null,"839",null,"0500000US26065","Ingham County, Michigan","26","065"],["Ionia County, Michigan","23355",null,"286",null,"0500000US26067","Ionia County, Michigan","26","067"],["Iosco County, Michigan","11713",null,"324",null,"0500000US26069","Iosco County, Michigan","26","069"],["Iron County, Michigan","5547",null,"190",null,"0500000US26071","Iron County, Michigan","26","071"],["Isabella County, Michigan","24914",null,"426",null,"0500000US26073","Isabella County, Michigan","26","073"],["Jackson County, Michigan","62540",null,"601",null,"0500000US26075","Jackson County, Michigan","26","075"],["Kalamazoo County, Michigan","107110",null,"708",null,"0500000US26077","Kalamazoo County, Michigan","26","077"],["Kalkaska County, Michigan","7789",null,"210",null,"0500000US26079","Kalkaska County, Michigan","26","079"],["Kent County, Michigan","256648",null,"994",null,"0500000US26081","Kent County, Michigan","26","081"],["Keweenaw County, Michigan","1135",null,"79",null,"0500000US26083","Keweenaw County, Michigan","26","083"],["Lake County, Michigan","5190",null,"170",null,"0500000US26085","Lake County, Michigan","26","085"],["Lapeer County, Michigan","34836",null,"350",null,"0500000US26087","Lapeer County, Michigan","26","087"],["Leelanau County, Michigan","9645",null,"290",null,"0500000US26089","Leelanau County, Michigan","26","089"],["Lenawee County, Michigan","38871",null,"467",null,"0500000US26091","Lenawee County, Michigan","26","091"],["Livingston County, Michigan","76445",null,"481",null,"0500000US26093","Livingston County, Michigan","26","093"],["Luce County, Michigan","2455",null,"139",null,"0500000US26095","Luce County, Michigan","26","095"],["Mackinac County, Michigan","5199",null,"186",null,"0500000US26097","Mackinac County, Michigan","26","097"],["Macomb County, Michigan","358668",null,"1153",null,"0500000US26099","Macomb County, Michigan","26","099"],["Manistee County, Michigan","10561",null,"280",null,"0500000US26101","Manistee County, Michigan","26","101"],["Marquette County, Michigan","28255",null,"346",null,"0500000US26103","Marquette County, Michigan","26","103"],["Mason County, Michigan","12801",null,"336",null,"0500000US26105","Mason County, Michigan","26","105"],["Mecosta County, Michigan","16708",null,"368",null,"0500000US26107","Mecosta County, Michigan","26","107"],["Menominee County, Michigan","10361",null,"208",null,"0500000US26109","Menominee County, Michigan","26","109"],["Midland County, Michigan","34993",null,"412",null,"0500000US26111","Midland County, Michigan","26","111"],["Missaukee County, Michigan","5909",null,"180",null,"0500000US26113","Missaukee County, Michigan","26","113"],["Monroe County, Michigan","63029",null,"496",null,"0500000US26115","Monroe County, Michigan","26","115"],["Montcalm County, Michigan","24511",null,"338",null,"0500000US26117","Montcalm County, Michigan","26","117"],["Montmorency County, Michigan","4921",null,"197",null,"0500000US26119","Montmorency County, Michigan","26","119"],["Muskegon County, Michigan","67732",null,"648",null,"0500000US26121","Muskegon County, Michigan","26","121"],["Newaygo County, Michigan","19651",null,"344",null,"0500000US26123","Newaygo County, Michigan","26","123"],["Oakland County, Michigan","530460",null,"1718",null,"0500000US26125","Oakland County, Michigan","26","125"],["Oceana County, Michigan","10419",null,"202",null,"0500000US26127","Oceana County, Michigan","26","127"],["Ogemaw County, Michigan","9370",null,"285",null,"0500000US26129","Ogemaw County, Michigan","26","129"],["Ontonagon County, Michigan","3149",null,"131",null,"0500000US26131","Ontonagon County, Michigan","26","131"],["Osceola County, Michigan","9356",null,"205",null,"0500000US26133","Osceola County, Michigan","26","133"],["Oscoda County, Michigan","3788",null,"167",null,"0500000US26135","Oscoda County, Michigan","26","135"],["Otsego County, Michigan","11231",null,"358",null,"0500000US26137","Otsego County, Michigan","26","137"],["Ottawa County, Michigan","111322",null,"737",null,"0500000US26139","Ottawa County, Michigan","26","139"],["Presque Isle County, Michigan","6655",null,"302",null,"0500000US26141","Presque Isle County, Michigan","26","141"],["Roscommon County, Michigan","11579",null,"352",null,"0500000US26143","Roscommon County, Michigan","26","143"],["Saginaw County, Michigan","77958",null,"704",null,"0500000US26145","Saginaw County, Michigan","26","145"],["St. Clair County, Michigan","67816",null,"583",null,"0500000US26147","St. Clair County, Michigan","26","147"],["St. Joseph County, Michigan","23620",null,"344",null,"0500000US26149","St. Joseph County, Michigan","26","149"],["Sanilac County, Michigan","17516",null,"266",null,"0500000US26151","Sanilac County, Michigan","26","151"],["Schoolcraft County, Michigan","3890",null,"163",null,"0500000US26153","Schoolcraft County, Michigan","26","153"],["Shiawassee County, Michigan","28126",null,"371",null,"0500000US26155","Shiawassee County, Michigan","26","155"],["Tuscola County, Michigan","21498",null,"287",null,"0500000US26157","Tuscola County, Michigan","26","157"],["Van Buren County, Michigan","30622",null,"511",null,"0500000US26159","Van Buren County, Michigan","26","159"],["Washtenaw County, Michigan","150550",null,"825",null,"0500000US26161","Washtenaw County, Michigan","26","161"],["Wayne County, Michigan","700591",null,"1709",null,"0500000US26163","Wayne County, Michigan","26","163"],["Wexford County, Michigan","13696",null,"314",null,"0500000US26165","Wexford County, Michigan","26","165"]]}} diff --git a/tests/fixtures/mi_ny_change/frozen_api_responses_2024_36.json b/tests/fixtures/mi_ny_change/frozen_api_responses_2024_36.json new file mode 100644 index 0000000..ff17e8b --- /dev/null +++ b/tests/fixtures/mi_ny_change/frozen_api_responses_2024_36.json @@ -0,0 +1 @@ +{"_about":"REAL Census Data API responses (2024 acs5, all counties of state 36, get=NAME,group(
)), recorded by `python -m tests.test_manifest_releases --record` for examples/manifests/mi_ny_income_change.json; columns trimmed to the manifest's variables (estimate, MOE and annotations) plus GEO_ID, NAME, state, county.","request":{"year":2024,"dataset":"acs5","geo_level":"county","state_code":"36","county_code":"*"},"requests":{"B19025":[["NAME","B19025_001E","B19025_001EA","B19025_001M","B19025_001MA","GEO_ID","NAME","state","county"],["Albany County, New York","15021910500",null,"326739709",null,"0500000US36001","Albany County, New York","36","001"],["Allegany County, New York","1354463100",null,"43066793",null,"0500000US36003","Allegany County, New York","36","003"],["Bronx County, New York","38330615000",null,"639870605",null,"0500000US36005","Bronx County, New York","36","005"],["Broome County, New York","7062900600",null,"214021727",null,"0500000US36007","Broome County, New York","36","007"],["Cattaraugus County, New York","2405667200",null,"59021526",null,"0500000US36009","Cattaraugus County, New York","36","009"],["Cayuga County, New York","2729443500",null,"86008057",null,"0500000US36011","Cayuga County, New York","36","011"],["Chautauqua County, New York","4057882300",null,"111543613",null,"0500000US36013","Chautauqua County, New York","36","013"],["Chemung County, New York","3045724700",null,"131542696",null,"0500000US36015","Chemung County, New York","36","015"],["Chenango County, New York","1557796600",null,"57394398",null,"0500000US36017","Chenango County, New York","36","017"],["Clinton County, New York","3062118200",null,"136495426",null,"0500000US36019","Clinton County, New York","36","019"],["Columbia County, New York","3312243800",null,"144942383",null,"0500000US36021","Columbia County, New York","36","021"],["Cortland County, New York","1621320600",null,"59355426",null,"0500000US36023","Cortland County, New York","36","023"],["Delaware County, New York","1643153500",null,"74109747",null,"0500000US36025","Delaware County, New York","36","025"],["Dutchess County, New York","15582112200",null,"439882136",null,"0500000US36027","Dutchess County, New York","36","027"],["Erie County, New York","40039828500",null,"524501258",null,"0500000US36029","Erie County, New York","36","029"],["Essex County, New York","1561593800",null,"94057900",null,"0500000US36031","Essex County, New York","36","031"],["Franklin County, New York","1563554100",null,"66557302",null,"0500000US36033","Franklin County, New York","36","033"],["Fulton County, New York","1973902300",null,"142713157",null,"0500000US36035","Fulton County, New York","36","035"],["Genesee County, New York","2192461700",null,"65661673",null,"0500000US36037","Genesee County, New York","36","037"],["Greene County, New York","1952266800",null,"112160914",null,"0500000US36039","Greene County, New York","36","039"],["Hamilton County, New York","217719200",null,"18788544",null,"0500000US36041","Hamilton County, New York","36","041"],["Herkimer County, New York","2225219900",null,"100408018",null,"0500000US36043","Herkimer County, New York","36","043"],["Jefferson County, New York","4104111400",null,"142423499",null,"0500000US36045","Jefferson County, New York","36","045"],["Kings County, New York","124575230900",null,"1314298422",null,"0500000US36047","Kings County, New York","36","047"],["Lewis County, New York","898806300",null,"36000208",null,"0500000US36049","Lewis County, New York","36","049"],["Livingston County, New York","2281851800",null,"86883354",null,"0500000US36051","Livingston County, New York","36","051"],["Madison County, New York","2549623300",null,"74887857",null,"0500000US36053","Madison County, New York","36","053"],["Monroe County, New York","32688613000",null,"448528229",null,"0500000US36055","Monroe County, New York","36","055"],["Montgomery County, New York","1607489500",null,"56968598",null,"0500000US36057","Montgomery County, New York","36","057"],["Nassau County, New York","87832341700",null,"971408164",null,"0500000US36059","Nassau County, New York","36","059"],["New York County, New York","151076814100",null,"2372140897",null,"0500000US36061","New York County, New York","36","061"],["Niagara County, New York","8192279400",null,"197181883",null,"0500000US36063","Niagara County, New York","36","063"],["Oneida County, New York","8670919900",null,"179443100",null,"0500000US36065","Oneida County, New York","36","065"],["Onondaga County, New York","20188896100",null,"333795439",null,"0500000US36067","Onondaga County, New York","36","067"],["Ontario County, New York","5219594400",null,"128089495",null,"0500000US36069","Ontario County, New York","36","069"],["Orange County, New York","17194772800",null,"340347384",null,"0500000US36071","Orange County, New York","36","071"],["Orleans County, New York","1363518600",null,"89898000",null,"0500000US36073","Orleans County, New York","36","073"],["Oswego County, New York","4212950100",null,"138310174",null,"0500000US36075","Oswego County, New York","36","075"],["Otsego County, New York","2111933400",null,"77195904",null,"0500000US36077","Otsego County, New York","36","077"],["Putnam County, New York","5468784000",null,"171756375",null,"0500000US36079","Putnam County, New York","36","079"],["Queens County, New York","97045279200",null,"1348426366",null,"0500000US36081","Queens County, New York","36","081"],["Rensselaer County, New York","7141125300",null,"200649476",null,"0500000US36083","Rensselaer County, New York","36","083"],["Richmond County, New York","21304615000",null,"385320655",null,"0500000US36085","Richmond County, New York","36","085"],["Rockland County, New York","15381883200",null,"342909806",null,"0500000US36087","Rockland County, New York","36","087"],["St. Lawrence County, New York","3444268500",null,"142023098",null,"0500000US36089","St. Lawrence County, New York","36","089"],["Saratoga County, New York","13559213000",null,"358002416",null,"0500000US36091","Saratoga County, New York","36","091"],["Schenectady County, New York","6886626500",null,"221120253",null,"0500000US36093","Schenectady County, New York","36","093"],["Schoharie County, New York","1165092300",null,"58412376",null,"0500000US36095","Schoharie County, New York","36","095"],["Schuyler County, New York","660000500",null,"42117402",null,"0500000US36097","Schuyler County, New York","36","097"],["Seneca County, New York","1222819300",null,"69784687",null,"0500000US36099","Seneca County, New York","36","099"],["Steuben County, New York","3445484500",null,"100991326",null,"0500000US36101","Steuben County, New York","36","101"],["Suffolk County, New York","85198737400",null,"1016473428",null,"0500000US36103","Suffolk County, New York","36","103"],["Sullivan County, New York","3322677000",null,"166756526",null,"0500000US36105","Sullivan County, New York","36","105"],["Tioga County, New York","1987097200",null,"86763599",null,"0500000US36107","Tioga County, New York","36","107"],["Tompkins County, New York","4443887700",null,"153086841",null,"0500000US36109","Tompkins County, New York","36","109"],["Ulster County, New York","8716739800",null,"278880643",null,"0500000US36111","Ulster County, New York","36","111"],["Warren County, New York","3177616500",null,"152177200",null,"0500000US36113","Warren County, New York","36","113"],["Washington County, New York","2178415400",null,"84102001",null,"0500000US36115","Washington County, New York","36","115"],["Wayne County, New York","3447869100",null,"108447173",null,"0500000US36117","Wayne County, New York","36","117"],["Westchester County, New York","71608726600",null,"1103975311",null,"0500000US36119","Westchester County, New York","36","119"],["Wyoming County, New York","1405797800",null,"51872153",null,"0500000US36121","Wyoming County, New York","36","121"],["Yates County, New York","912853400",null,"48365318",null,"0500000US36123","Yates County, New York","36","123"]],"B19001":[["NAME","B19001_001E","B19001_001EA","B19001_001M","B19001_001MA","GEO_ID","NAME","state","county"],["Albany County, New York","134703",null,"1000",null,"0500000US36001","Albany County, New York","36","001"],["Allegany County, New York","17710",null,"319",null,"0500000US36003","Allegany County, New York","36","003"],["Bronx County, New York","534100",null,"1505",null,"0500000US36005","Bronx County, New York","36","005"],["Broome County, New York","82005",null,"806",null,"0500000US36007","Broome County, New York","36","007"],["Cattaraugus County, New York","31895",null,"379",null,"0500000US36009","Cattaraugus County, New York","36","009"],["Cayuga County, New York","31524",null,"438",null,"0500000US36011","Cayuga County, New York","36","011"],["Chautauqua County, New York","53393",null,"785",null,"0500000US36013","Chautauqua County, New York","36","013"],["Chemung County, New York","35262",null,"439",null,"0500000US36015","Chemung County, New York","36","015"],["Chenango County, New York","19890",null,"381",null,"0500000US36017","Chenango County, New York","36","017"],["Clinton County, New York","33556",null,"442",null,"0500000US36019","Clinton County, New York","36","019"],["Columbia County, New York","26627",null,"558",null,"0500000US36021","Columbia County, New York","36","021"],["Cortland County, New York","18999",null,"384",null,"0500000US36023","Cortland County, New York","36","023"],["Delaware County, New York","19203",null,"485",null,"0500000US36025","Delaware County, New York","36","025"],["Dutchess County, New York","116275",null,"524",null,"0500000US36027","Dutchess County, New York","36","027"],["Erie County, New York","409665",null,"1635",null,"0500000US36029","Erie County, New York","36","029"],["Essex County, New York","16342",null,"375",null,"0500000US36031","Essex County, New York","36","031"],["Franklin County, New York","19514",null,"405",null,"0500000US36033","Franklin County, New York","36","033"],["Fulton County, New York","22866",null,"476",null,"0500000US36035","Fulton County, New York","36","035"],["Genesee County, New York","24873",null,"296",null,"0500000US36037","Genesee County, New York","36","037"],["Greene County, New York","18674",null,"510",null,"0500000US36039","Greene County, New York","36","039"],["Hamilton County, New York","2423",null,"155",null,"0500000US36041","Hamilton County, New York","36","041"],["Herkimer County, New York","25741",null,"353",null,"0500000US36043","Herkimer County, New York","36","043"],["Jefferson County, New York","47606",null,"645",null,"0500000US36045","Jefferson County, New York","36","045"],["Kings County, New York","1009927",null,"2338",null,"0500000US36047","Kings County, New York","36","047"],["Lewis County, New York","11110",null,"274",null,"0500000US36049","Lewis County, New York","36","049"],["Livingston County, New York","24473",null,"401",null,"0500000US36051","Livingston County, New York","36","051"],["Madison County, New York","26252",null,"407",null,"0500000US36053","Madison County, New York","36","053"],["Monroe County, New York","318428",null,"1337",null,"0500000US36055","Monroe County, New York","36","055"],["Montgomery County, New York","19603",null,"348",null,"0500000US36057","Montgomery County, New York","36","057"],["Nassau County, New York","458166",null,"1227",null,"0500000US36059","Nassau County, New York","36","059"],["New York County, New York","778281",null,"4924",null,"0500000US36061","New York County, New York","36","061"],["Niagara County, New York","91884",null,"707",null,"0500000US36063","Niagara County, New York","36","063"],["Oneida County, New York","95220",null,"781",null,"0500000US36065","Oneida County, New York","36","065"],["Onondaga County, New York","197760",null,"1030",null,"0500000US36067","Onondaga County, New York","36","067"],["Ontario County, New York","48311",null,"558",null,"0500000US36069","Ontario County, New York","36","069"],["Orange County, New York","138608",null,"924",null,"0500000US36071","Orange County, New York","36","071"],["Orleans County, New York","16220",null,"282",null,"0500000US36073","Orleans County, New York","36","073"],["Oswego County, New York","47239",null,"606",null,"0500000US36075","Oswego County, New York","36","075"],["Otsego County, New York","23189",null,"482",null,"0500000US36077","Otsego County, New York","36","077"],["Putnam County, New York","35545",null,"459",null,"0500000US36079","Putnam County, New York","36","079"],["Queens County, New York","841003",null,"2217",null,"0500000US36081","Queens County, New York","36","081"],["Rensselaer County, New York","66686",null,"619",null,"0500000US36083","Rensselaer County, New York","36","083"],["Richmond County, New York","170777",null,"1007",null,"0500000US36085","Richmond County, New York","36","085"],["Rockland County, New York","103879",null,"590",null,"0500000US36087","Rockland County, New York","36","087"],["St. Lawrence County, New York","41666",null,"542",null,"0500000US36089","St. Lawrence County, New York","36","089"],["Saratoga County, New York","101911",null,"833",null,"0500000US36091","Saratoga County, New York","36","091"],["Schenectady County, New York","66902",null,"527",null,"0500000US36093","Schenectady County, New York","36","093"],["Schoharie County, New York","12861",null,"253",null,"0500000US36095","Schoharie County, New York","36","095"],["Schuyler County, New York","7355",null,"259",null,"0500000US36097","Schuyler County, New York","36","097"],["Seneca County, New York","13606",null,"256",null,"0500000US36099","Seneca County, New York","36","099"],["Steuben County, New York","39219",null,"465",null,"0500000US36101","Steuben County, New York","36","101"],["Suffolk County, New York","512565",null,"1647",null,"0500000US36103","Suffolk County, New York","36","103"],["Sullivan County, New York","31434",null,"846",null,"0500000US36105","Sullivan County, New York","36","105"],["Tioga County, New York","20347",null,"312",null,"0500000US36107","Tioga County, New York","36","107"],["Tompkins County, New York","43263",null,"611",null,"0500000US36109","Tompkins County, New York","36","109"],["Ulster County, New York","74278",null,"803",null,"0500000US36111","Ulster County, New York","36","111"],["Warren County, New York","30133",null,"454",null,"0500000US36113","Warren County, New York","36","113"],["Washington County, New York","24469",null,"424",null,"0500000US36115","Washington County, New York","36","115"],["Wayne County, New York","38404",null,"391",null,"0500000US36117","Wayne County, New York","36","117"],["Westchester County, New York","372883",null,"1455",null,"0500000US36119","Westchester County, New York","36","119"],["Wyoming County, New York","16453",null,"292",null,"0500000US36121","Wyoming County, New York","36","121"],["Yates County, New York","9490",null,"278",null,"0500000US36123","Yates County, New York","36","123"]]}} diff --git a/tests/fixtures/mi_ny_change/frozen_api_responses_2024_us.json b/tests/fixtures/mi_ny_change/frozen_api_responses_2024_us.json new file mode 100644 index 0000000..05f4703 --- /dev/null +++ b/tests/fixtures/mi_ny_change/frozen_api_responses_2024_us.json @@ -0,0 +1 @@ +{"_about":"REAL Census Data API responses (2024 acs5, nation, get=NAME,group(
)), untrimmed, read by the continuity check of a multi-release extraction; recorded by `python -m tests.test_manifest_releases --record`.","request":{"year":2024,"dataset":"acs5","geo_level":"us","state_code":null},"requests":{"B19025":[["NAME","B19025_001E","B19025_001EA","B19025_001M","B19025_001MA","GEO_ID","NAME","us"],["United States","14671199681000",null,"49032297444",null,"0100000US","United States","1"]],"B19001":[["NAME","B19001_001E","B19001_001EA","B19001_001M","B19001_001MA","B19001_002E","B19001_002EA","B19001_002M","B19001_002MA","B19001_003E","B19001_003EA","B19001_003M","B19001_003MA","B19001_004E","B19001_004EA","B19001_004M","B19001_004MA","B19001_005E","B19001_005EA","B19001_005M","B19001_005MA","B19001_006E","B19001_006EA","B19001_006M","B19001_006MA","B19001_007E","B19001_007EA","B19001_007M","B19001_007MA","B19001_008E","B19001_008EA","B19001_008M","B19001_008MA","B19001_009E","B19001_009EA","B19001_009M","B19001_009MA","B19001_010E","B19001_010EA","B19001_010M","B19001_010MA","B19001_011E","B19001_011EA","B19001_011M","B19001_011MA","B19001_012E","B19001_012EA","B19001_012M","B19001_012MA","B19001_013E","B19001_013EA","B19001_013M","B19001_013MA","B19001_014E","B19001_014EA","B19001_014M","B19001_014MA","B19001_015E","B19001_015EA","B19001_015M","B19001_015MA","B19001_016E","B19001_016EA","B19001_016M","B19001_016MA","B19001_017E","B19001_017EA","B19001_017M","B19001_017MA","GEO_ID","NAME","us"],["United States","129227496",null,"209365",null,"6397599",null,"26113",null,"4406926",null,"20579",null,"3858602",null,"17470",null,"4245461",null,"17755",null,"4192539",null,"15949",null,"4358743",null,"17495",null,"4343918",null,"16698",null,"4433159",null,"18540",null,"4296543",null,"18669",null,"8280714",null,"22509",null,"11750969",null,"29181",null,"16319799",null,"42704",null,"13068961",null,"40040",null,"9601960",null,"42352",null,"12405370",null,"57556",null,"17266233",null,"82369",null,"0100000US","United States","1"]]}} diff --git a/tests/fixtures/mi_ny_change/vre/2019/050/B19001.csv.zip b/tests/fixtures/mi_ny_change/vre/2019/050/B19001.csv.zip new file mode 100644 index 0000000..435ae80 Binary files /dev/null and b/tests/fixtures/mi_ny_change/vre/2019/050/B19001.csv.zip differ diff --git a/tests/fixtures/mi_ny_change/vre/2019/050/B19001.csv.zip.json b/tests/fixtures/mi_ny_change/vre/2019/050/B19001.csv.zip.json new file mode 100644 index 0000000..a5e3abf --- /dev/null +++ b/tests/fixtures/mi_ny_change/vre/2019/050/B19001.csv.zip.json @@ -0,0 +1,8 @@ +{ + "source_url": "https://www2.census.gov/programs-surveys/acs/replicate_estimates/2019/data/5-year/050/B19001.csv.zip", + "sha256": "d68031d60482c7e63098a7e38329cdb311f710347f7d06e047852767ae4a3662", + "size_bytes": 32882, + "content_type": "application/zip", + "downloaded_at": "2026-11-08T00:00:00Z", + "_about": "TRIMMED copy of the Census VRE file at source_url (original SHA-256 56374faae955dd44c9eec5f45c7b2d145781b6271028d714eae6ebafc2f260d3): header, table-title rows, and the rows of the counties of states 26, 36 for line(s) [1]; recorded by `python -m tests.test_manifest_releases --record`." +} diff --git a/tests/fixtures/mi_ny_change/vre/2019/050/B19025.csv.zip b/tests/fixtures/mi_ny_change/vre/2019/050/B19025.csv.zip new file mode 100644 index 0000000..51171cc Binary files /dev/null and b/tests/fixtures/mi_ny_change/vre/2019/050/B19025.csv.zip differ diff --git a/tests/fixtures/mi_ny_change/vre/2019/050/B19025.csv.zip.json b/tests/fixtures/mi_ny_change/vre/2019/050/B19025.csv.zip.json new file mode 100644 index 0000000..20f0196 --- /dev/null +++ b/tests/fixtures/mi_ny_change/vre/2019/050/B19025.csv.zip.json @@ -0,0 +1,8 @@ +{ + "source_url": "https://www2.census.gov/programs-surveys/acs/replicate_estimates/2019/data/5-year/050/B19025.csv.zip", + "sha256": "84c429c967c23ac4e06b70e32bfccb3e38e9a3b797c9b1de235c5877af1279f4", + "size_bytes": 56998, + "content_type": "application/zip", + "downloaded_at": "2026-11-08T00:00:00Z", + "_about": "TRIMMED copy of the Census VRE file at source_url (original SHA-256 658f77ffc48e90bed0b48a8a404f8590e52cc0bda4f8e35c2273f3ff8dea8401): header, table-title rows, and the rows of the counties of states 26, 36 for line(s) [1]; recorded by `python -m tests.test_manifest_releases --record`." +} diff --git a/tests/fixtures/mi_ny_change/vre/2024/050/B19001.csv.zip b/tests/fixtures/mi_ny_change/vre/2024/050/B19001.csv.zip new file mode 100644 index 0000000..19fec74 Binary files /dev/null and b/tests/fixtures/mi_ny_change/vre/2024/050/B19001.csv.zip differ diff --git a/tests/fixtures/mi_ny_change/vre/2024/050/B19001.csv.zip.json b/tests/fixtures/mi_ny_change/vre/2024/050/B19001.csv.zip.json new file mode 100644 index 0000000..c9d491c --- /dev/null +++ b/tests/fixtures/mi_ny_change/vre/2024/050/B19001.csv.zip.json @@ -0,0 +1,8 @@ +{ + "source_url": "https://www2.census.gov/programs-surveys/acs/replicate_estimates/2024/data/5-year/050/B19001.csv.zip", + "sha256": "772c716639bfdd96bebd31797b93f110f1370e99419abd18338f56567ddcfe61", + "size_bytes": 32671, + "content_type": "application/zip", + "downloaded_at": "2026-11-08T00:00:00Z", + "_about": "TRIMMED copy of the Census VRE file at source_url (original SHA-256 95cb48eabcaef5c44fc68377f8b596f31cbb079d846bbea9c6bf7e83ba896d0f): header, table-title rows, and the rows of the counties of states 26, 36 for line(s) [1]; recorded by `python -m tests.test_manifest_releases --record`." +} diff --git a/tests/fixtures/mi_ny_change/vre/2024/050/B19025.csv.zip b/tests/fixtures/mi_ny_change/vre/2024/050/B19025.csv.zip new file mode 100644 index 0000000..f40b8dc Binary files /dev/null and b/tests/fixtures/mi_ny_change/vre/2024/050/B19025.csv.zip differ diff --git a/tests/fixtures/mi_ny_change/vre/2024/050/B19025.csv.zip.json b/tests/fixtures/mi_ny_change/vre/2024/050/B19025.csv.zip.json new file mode 100644 index 0000000..5bac423 --- /dev/null +++ b/tests/fixtures/mi_ny_change/vre/2024/050/B19025.csv.zip.json @@ -0,0 +1,8 @@ +{ + "source_url": "https://www2.census.gov/programs-surveys/acs/replicate_estimates/2024/data/5-year/050/B19025.csv.zip", + "sha256": "1d8ec7b7ecf41d85bd37eec0a20046eac44bfbebd9bcc27968aebd7c4c2d7a2b", + "size_bytes": 57940, + "content_type": "application/zip", + "downloaded_at": "2026-11-08T00:00:00Z", + "_about": "TRIMMED copy of the Census VRE file at source_url (original SHA-256 817bf4e7cf3844330daae9e6f8ba63163848cb0340faec970866ca8017671464): header, table-title rows, and the rows of the counties of states 26, 36 for line(s) [1]; recorded by `python -m tests.test_manifest_releases --record`." +} diff --git a/tests/fixtures/validation/api_2020_dec_dhc_594af93aed63.json.gz b/tests/fixtures/validation/api_2020_dec_dhc_594af93aed63.json.gz new file mode 100644 index 0000000..2e262ec Binary files /dev/null and b/tests/fixtures/validation/api_2020_dec_dhc_594af93aed63.json.gz differ diff --git a/tests/fixtures/validation/api_2020_dec_dhc_594af93aed63.json.gz.json b/tests/fixtures/validation/api_2020_dec_dhc_594af93aed63.json.gz.json new file mode 100644 index 0000000..1be5b4f --- /dev/null +++ b/tests/fixtures/validation/api_2020_dec_dhc_594af93aed63.json.gz.json @@ -0,0 +1,9 @@ +{ + "kind": "census_api_response", + "note": "Parsed API JSON re-serialized compactly (content identical) and gzip-compressed with mtime=0. The API key is not part of the URL.", + "retrieved": "2026-11-08", + "sha256": "9852433338dedeaad358a475f1730cd1ad8d28d73882ecaa163fa82d0f982a6c", + "size_bytes": 173, + "source_url": "https://api.census.gov/data/2020/dec/dhc?for=state%3A26%2C36%2C56&get=NAME%2CH1_001N%2CH3_001N%2CH3_002N%2CH3_003N%2CP1_001N", + "uncompressed_sha256": "28c1abffd0aa2e87063b655ebb7014c59d0cba0d6ee0e5ff2ce5e600010c7774" +} diff --git a/tests/fixtures/validation/api_2020_dec_dhc_6e1631a86459.json.gz b/tests/fixtures/validation/api_2020_dec_dhc_6e1631a86459.json.gz new file mode 100644 index 0000000..886e041 Binary files /dev/null and b/tests/fixtures/validation/api_2020_dec_dhc_6e1631a86459.json.gz differ diff --git a/tests/fixtures/validation/api_2020_dec_dhc_6e1631a86459.json.gz.json b/tests/fixtures/validation/api_2020_dec_dhc_6e1631a86459.json.gz.json new file mode 100644 index 0000000..aeb65d9 --- /dev/null +++ b/tests/fixtures/validation/api_2020_dec_dhc_6e1631a86459.json.gz.json @@ -0,0 +1,9 @@ +{ + "kind": "census_api_response", + "note": "Parsed API JSON re-serialized compactly (content identical) and gzip-compressed with mtime=0. The API key is not part of the URL.", + "retrieved": "2026-11-08", + "sha256": "2b9cf7fa2e706a4879063a8de0d8ab6b07d0fdfa6dea45f0e159f9092d628e93", + "size_bytes": 179, + "source_url": "https://api.census.gov/data/2020/dec/dhc?for=tract%3A%2A&get=NAME%2CH1_001N%2CH3_001N%2CH3_002N%2CH3_003N%2CP1_001N&in=state%3A26+county%3A083", + "uncompressed_sha256": "b9e7eecb58398532f4cc1b85a9c1c994804e4d26714666ee16e7366fde6caca4" +} diff --git a/tests/fixtures/validation/api_2020_dec_dhc_707869350f4c.json.gz b/tests/fixtures/validation/api_2020_dec_dhc_707869350f4c.json.gz new file mode 100644 index 0000000..06fa72d Binary files /dev/null and b/tests/fixtures/validation/api_2020_dec_dhc_707869350f4c.json.gz differ diff --git a/tests/fixtures/validation/api_2020_dec_dhc_707869350f4c.json.gz.json b/tests/fixtures/validation/api_2020_dec_dhc_707869350f4c.json.gz.json new file mode 100644 index 0000000..d368bb4 --- /dev/null +++ b/tests/fixtures/validation/api_2020_dec_dhc_707869350f4c.json.gz.json @@ -0,0 +1,9 @@ +{ + "kind": "census_api_response", + "note": "Parsed API JSON re-serialized compactly (content identical) and gzip-compressed with mtime=0. The API key is not part of the URL.", + "retrieved": "2026-11-08", + "sha256": "1c930dcf76c454a043d7ba6e871b60098fe8db2dd4812be679dba55b2f4d3a3a", + "size_bytes": 338, + "source_url": "https://api.census.gov/data/2020/dec/dhc?for=tract%3A%2A&get=NAME%2CH1_001N%2CH3_001N%2CH3_002N%2CH3_003N%2CP1_001N&in=state%3A56+county%3A001", + "uncompressed_sha256": "a2f8e0c8238e59d4e8beff7ce9efed5834c22b60f509410f039cbc5304302266" +} diff --git a/tests/fixtures/validation/api_2020_dec_dhc_a33e067da5e2.json.gz b/tests/fixtures/validation/api_2020_dec_dhc_a33e067da5e2.json.gz new file mode 100644 index 0000000..124a551 Binary files /dev/null and b/tests/fixtures/validation/api_2020_dec_dhc_a33e067da5e2.json.gz differ diff --git a/tests/fixtures/validation/api_2020_dec_dhc_a33e067da5e2.json.gz.json b/tests/fixtures/validation/api_2020_dec_dhc_a33e067da5e2.json.gz.json new file mode 100644 index 0000000..ad51243 --- /dev/null +++ b/tests/fixtures/validation/api_2020_dec_dhc_a33e067da5e2.json.gz.json @@ -0,0 +1,9 @@ +{ + "kind": "census_api_response", + "note": "Parsed API JSON re-serialized compactly (content identical) and gzip-compressed with mtime=0. The API key is not part of the URL.", + "retrieved": "2026-11-08", + "sha256": "89cad0754d2cd864f7612cd1287a53e3a7daf1f662cd96f2cc432dd1a174c864", + "size_bytes": 3163, + "source_url": "https://api.census.gov/data/2020/dec/dhc?for=block%3A%2A&get=H1_001N&in=state%3A36+county%3A041", + "uncompressed_sha256": "9b2ec4af00662c6d5a677208754cc7b66f6134db7d21c05330e52df3c3c4e53a" +} diff --git a/tests/fixtures/validation/api_2020_dec_dhc_ba5ae967e2c8.json.gz b/tests/fixtures/validation/api_2020_dec_dhc_ba5ae967e2c8.json.gz new file mode 100644 index 0000000..08d9d5d Binary files /dev/null and b/tests/fixtures/validation/api_2020_dec_dhc_ba5ae967e2c8.json.gz differ diff --git a/tests/fixtures/validation/api_2020_dec_dhc_ba5ae967e2c8.json.gz.json b/tests/fixtures/validation/api_2020_dec_dhc_ba5ae967e2c8.json.gz.json new file mode 100644 index 0000000..726723f --- /dev/null +++ b/tests/fixtures/validation/api_2020_dec_dhc_ba5ae967e2c8.json.gz.json @@ -0,0 +1,9 @@ +{ + "kind": "census_api_response", + "note": "Parsed API JSON re-serialized compactly (content identical) and gzip-compressed with mtime=0. The API key is not part of the URL.", + "retrieved": "2026-11-08", + "sha256": "4010d6b9b615d9d25b486c17a316ba235a50bd9ea6d4d87b89c2bcd7821e1e6a", + "size_bytes": 210, + "source_url": "https://api.census.gov/data/2020/dec/dhc?for=tract%3A%2A&get=NAME%2CH1_001N%2CH3_001N%2CH3_002N%2CH3_003N%2CP1_001N&in=state%3A36+county%3A041", + "uncompressed_sha256": "7ab0351ce62940824b39035cce8e209f2f1fc12696557ccdc72606ee0fb093d9" +} diff --git a/tests/fixtures/validation/api_2020_dec_dhc_c0e1ab8dfdec.json.gz b/tests/fixtures/validation/api_2020_dec_dhc_c0e1ab8dfdec.json.gz new file mode 100644 index 0000000..8d55191 Binary files /dev/null and b/tests/fixtures/validation/api_2020_dec_dhc_c0e1ab8dfdec.json.gz differ diff --git a/tests/fixtures/validation/api_2020_dec_dhc_c0e1ab8dfdec.json.gz.json b/tests/fixtures/validation/api_2020_dec_dhc_c0e1ab8dfdec.json.gz.json new file mode 100644 index 0000000..0c3de62 --- /dev/null +++ b/tests/fixtures/validation/api_2020_dec_dhc_c0e1ab8dfdec.json.gz.json @@ -0,0 +1,9 @@ +{ + "kind": "census_api_response", + "note": "Parsed API JSON re-serialized compactly (content identical) and gzip-compressed with mtime=0. The API key is not part of the URL.", + "retrieved": "2026-11-08", + "sha256": "70c7a31d11ceb978653cac3300c4cca9010c4c5dd6f2da926b35814f3f04714e", + "size_bytes": 10248, + "source_url": "https://api.census.gov/data/2020/dec/dhc?for=block%3A%2A&get=H1_001N&in=state%3A56+county%3A001", + "uncompressed_sha256": "5303de321c080bee256ea2e96554cca68f1afb6f8ebcf6f8dfcc68d4413fe4cd" +} diff --git a/tests/fixtures/validation/api_2020_dec_dhc_db81a965e8d0.json.gz b/tests/fixtures/validation/api_2020_dec_dhc_db81a965e8d0.json.gz new file mode 100644 index 0000000..b09c21d Binary files /dev/null and b/tests/fixtures/validation/api_2020_dec_dhc_db81a965e8d0.json.gz differ diff --git a/tests/fixtures/validation/api_2020_dec_dhc_db81a965e8d0.json.gz.json b/tests/fixtures/validation/api_2020_dec_dhc_db81a965e8d0.json.gz.json new file mode 100644 index 0000000..1f6709a --- /dev/null +++ b/tests/fixtures/validation/api_2020_dec_dhc_db81a965e8d0.json.gz.json @@ -0,0 +1,9 @@ +{ + "kind": "census_api_response", + "note": "Parsed API JSON re-serialized compactly (content identical) and gzip-compressed with mtime=0. The API key is not part of the URL.", + "retrieved": "2026-11-08", + "sha256": "0759702652c9f6b0b49d5e518a8e87c29962b57e53591bc7a496be0597245b5b", + "size_bytes": 1678, + "source_url": "https://api.census.gov/data/2020/dec/dhc?for=block%3A%2A&get=H1_001N&in=state%3A26+county%3A083", + "uncompressed_sha256": "13d9fd82f8e0aabbcea95d1088a9f6f0189a90258b1323dc4be1b26c42bd30c7" +} diff --git a/tests/fixtures/validation/api_2020_dec_dhc_e8ba05b05f0a.json.gz b/tests/fixtures/validation/api_2020_dec_dhc_e8ba05b05f0a.json.gz new file mode 100644 index 0000000..487feed Binary files /dev/null and b/tests/fixtures/validation/api_2020_dec_dhc_e8ba05b05f0a.json.gz differ diff --git a/tests/fixtures/validation/api_2020_dec_dhc_e8ba05b05f0a.json.gz.json b/tests/fixtures/validation/api_2020_dec_dhc_e8ba05b05f0a.json.gz.json new file mode 100644 index 0000000..5443f2e --- /dev/null +++ b/tests/fixtures/validation/api_2020_dec_dhc_e8ba05b05f0a.json.gz.json @@ -0,0 +1,9 @@ +{ + "kind": "census_api_response", + "note": "Parsed API JSON re-serialized compactly (content identical) and gzip-compressed with mtime=0. The API key is not part of the URL.", + "retrieved": "2026-11-08", + "sha256": "65120b7f5d9dbf0dbd2db8888ebe2337485f47eea31e20b9f6d85c016bf79242", + "size_bytes": 4266, + "source_url": "https://api.census.gov/data/2020/dec/dhc?for=county%3A%2A&get=NAME%2CH1_001N%2CH3_001N%2CH3_002N%2CH3_003N%2CP1_001N&in=state%3A26%2C36%2C56", + "uncompressed_sha256": "b609b2f43620352e0d780d7b21335ee8449e6b91070240a0aeee29326bac5525" +} diff --git a/tests/fixtures/validation/api_2020_dec_pl_1e47019798e2.json.gz b/tests/fixtures/validation/api_2020_dec_pl_1e47019798e2.json.gz new file mode 100644 index 0000000..150e536 Binary files /dev/null and b/tests/fixtures/validation/api_2020_dec_pl_1e47019798e2.json.gz differ diff --git a/tests/fixtures/validation/api_2020_dec_pl_1e47019798e2.json.gz.json b/tests/fixtures/validation/api_2020_dec_pl_1e47019798e2.json.gz.json new file mode 100644 index 0000000..584dbb1 --- /dev/null +++ b/tests/fixtures/validation/api_2020_dec_pl_1e47019798e2.json.gz.json @@ -0,0 +1,9 @@ +{ + "kind": "census_api_response", + "note": "Parsed API JSON re-serialized compactly (content identical) and gzip-compressed with mtime=0. The API key is not part of the URL.", + "retrieved": "2026-11-08", + "sha256": "5edec1997847dac6ae06c32e76784e51ca6eb808f494f90182cf1a7b4b1773d1", + "size_bytes": 168, + "source_url": "https://api.census.gov/data/2020/dec/pl?for=state%3A26%2C36%2C56&get=NAME%2CH1_001N%2CH1_002N%2CH1_003N%2CP1_001N", + "uncompressed_sha256": "a667f11b6ff0ca0f089d4fdf592733388c940e52162656360343ad7428ffbcaa" +} diff --git a/tests/fixtures/validation/api_2020_dec_pl_26213a609079.json.gz b/tests/fixtures/validation/api_2020_dec_pl_26213a609079.json.gz new file mode 100644 index 0000000..c863b7e Binary files /dev/null and b/tests/fixtures/validation/api_2020_dec_pl_26213a609079.json.gz differ diff --git a/tests/fixtures/validation/api_2020_dec_pl_26213a609079.json.gz.json b/tests/fixtures/validation/api_2020_dec_pl_26213a609079.json.gz.json new file mode 100644 index 0000000..ceacd32 --- /dev/null +++ b/tests/fixtures/validation/api_2020_dec_pl_26213a609079.json.gz.json @@ -0,0 +1,9 @@ +{ + "kind": "census_api_response", + "note": "Parsed API JSON re-serialized compactly (content identical) and gzip-compressed with mtime=0. The API key is not part of the URL.", + "retrieved": "2026-11-08", + "sha256": "35bb76ef8c75e43f698af0972d409b1ded3205d6575d6e74d25d56bcccc27c0d", + "size_bytes": 1675, + "source_url": "https://api.census.gov/data/2020/dec/pl?for=block%3A%2A&get=H1_001N&in=state%3A26+county%3A083", + "uncompressed_sha256": "882aa78592f6b0e79835f5cf0d8691fb56806693cf172a4f2c9920ae31a5519f" +} diff --git a/tests/fixtures/validation/api_2020_dec_pl_3617274ffd10.json.gz b/tests/fixtures/validation/api_2020_dec_pl_3617274ffd10.json.gz new file mode 100644 index 0000000..fed3652 Binary files /dev/null and b/tests/fixtures/validation/api_2020_dec_pl_3617274ffd10.json.gz differ diff --git a/tests/fixtures/validation/api_2020_dec_pl_3617274ffd10.json.gz.json b/tests/fixtures/validation/api_2020_dec_pl_3617274ffd10.json.gz.json new file mode 100644 index 0000000..ba99c9d --- /dev/null +++ b/tests/fixtures/validation/api_2020_dec_pl_3617274ffd10.json.gz.json @@ -0,0 +1,9 @@ +{ + "kind": "census_api_response", + "note": "Parsed API JSON re-serialized compactly (content identical) and gzip-compressed with mtime=0. The API key is not part of the URL.", + "retrieved": "2026-11-08", + "sha256": "fae15afa5f60832beebdccb75e8aebb1f71fac4a83fc3092cec64a57dce470b5", + "size_bytes": 3157, + "source_url": "https://api.census.gov/data/2020/dec/pl?for=block%3A%2A&get=H1_001N&in=state%3A36+county%3A041", + "uncompressed_sha256": "6842fe0c15b019e3e93e181112b4ae4cae4bea9f3ec76d6093ca2b8d5687fffa" +} diff --git a/tests/fixtures/validation/api_2020_dec_pl_3dfac96036fc.json.gz b/tests/fixtures/validation/api_2020_dec_pl_3dfac96036fc.json.gz new file mode 100644 index 0000000..b99f54a Binary files /dev/null and b/tests/fixtures/validation/api_2020_dec_pl_3dfac96036fc.json.gz differ diff --git a/tests/fixtures/validation/api_2020_dec_pl_3dfac96036fc.json.gz.json b/tests/fixtures/validation/api_2020_dec_pl_3dfac96036fc.json.gz.json new file mode 100644 index 0000000..43c191d --- /dev/null +++ b/tests/fixtures/validation/api_2020_dec_pl_3dfac96036fc.json.gz.json @@ -0,0 +1,9 @@ +{ + "kind": "census_api_response", + "note": "Parsed API JSON re-serialized compactly (content identical) and gzip-compressed with mtime=0. The API key is not part of the URL.", + "retrieved": "2026-11-08", + "sha256": "cbce6924d80c241ce711baa92f2124226ed63979679a63bf4802572a28e4cc91", + "size_bytes": 4039, + "source_url": "https://api.census.gov/data/2020/dec/pl?for=county%3A%2A&get=NAME%2CH1_001N%2CH1_002N%2CH1_003N%2CP1_001N&in=state%3A26%2C36%2C56", + "uncompressed_sha256": "709cc2b0acd26f85ac569563f3f414edd8cef55f641553350403649441d214c2" +} diff --git a/tests/fixtures/validation/api_2020_dec_pl_48aef1c10729.json.gz b/tests/fixtures/validation/api_2020_dec_pl_48aef1c10729.json.gz new file mode 100644 index 0000000..895cd69 Binary files /dev/null and b/tests/fixtures/validation/api_2020_dec_pl_48aef1c10729.json.gz differ diff --git a/tests/fixtures/validation/api_2020_dec_pl_48aef1c10729.json.gz.json b/tests/fixtures/validation/api_2020_dec_pl_48aef1c10729.json.gz.json new file mode 100644 index 0000000..6088c62 --- /dev/null +++ b/tests/fixtures/validation/api_2020_dec_pl_48aef1c10729.json.gz.json @@ -0,0 +1,9 @@ +{ + "kind": "census_api_response", + "note": "Parsed API JSON re-serialized compactly (content identical) and gzip-compressed with mtime=0. The API key is not part of the URL.", + "retrieved": "2026-11-08", + "sha256": "455aea122a753344d3e18a83add6b9207cf49affee6546eb003fcecf76a95065", + "size_bytes": 174, + "source_url": "https://api.census.gov/data/2020/dec/pl?for=tract%3A%2A&get=NAME%2CH1_001N%2CH1_002N%2CH1_003N%2CP1_001N&in=state%3A26+county%3A083", + "uncompressed_sha256": "e87e27914535707a39425362e9d6d0a562885f5a9684b1e9c477d43a105fd35c" +} diff --git a/tests/fixtures/validation/api_2020_dec_pl_a390672cc2d8.json.gz b/tests/fixtures/validation/api_2020_dec_pl_a390672cc2d8.json.gz new file mode 100644 index 0000000..899d04a Binary files /dev/null and b/tests/fixtures/validation/api_2020_dec_pl_a390672cc2d8.json.gz differ diff --git a/tests/fixtures/validation/api_2020_dec_pl_a390672cc2d8.json.gz.json b/tests/fixtures/validation/api_2020_dec_pl_a390672cc2d8.json.gz.json new file mode 100644 index 0000000..597f01c --- /dev/null +++ b/tests/fixtures/validation/api_2020_dec_pl_a390672cc2d8.json.gz.json @@ -0,0 +1,9 @@ +{ + "kind": "census_api_response", + "note": "Parsed API JSON re-serialized compactly (content identical) and gzip-compressed with mtime=0. The API key is not part of the URL.", + "retrieved": "2026-11-08", + "sha256": "d20f3dc2019ff4f63dbeb3cd3f546d8a6229207b7ac1e246461238e36887f2c5", + "size_bytes": 10175, + "source_url": "https://api.census.gov/data/2020/dec/pl?for=block%3A%2A&get=H1_001N&in=state%3A56+county%3A001", + "uncompressed_sha256": "19e3d504638f124f517875175ef30e449a6465525932aa233786c79c722600d4" +} diff --git a/tests/fixtures/validation/api_2020_dec_pl_bf4e410a4614.json.gz b/tests/fixtures/validation/api_2020_dec_pl_bf4e410a4614.json.gz new file mode 100644 index 0000000..1548f20 Binary files /dev/null and b/tests/fixtures/validation/api_2020_dec_pl_bf4e410a4614.json.gz differ diff --git a/tests/fixtures/validation/api_2020_dec_pl_bf4e410a4614.json.gz.json b/tests/fixtures/validation/api_2020_dec_pl_bf4e410a4614.json.gz.json new file mode 100644 index 0000000..56eae35 --- /dev/null +++ b/tests/fixtures/validation/api_2020_dec_pl_bf4e410a4614.json.gz.json @@ -0,0 +1,9 @@ +{ + "kind": "census_api_response", + "note": "Parsed API JSON re-serialized compactly (content identical) and gzip-compressed with mtime=0. The API key is not part of the URL.", + "retrieved": "2026-11-08", + "sha256": "5f8a6b5cbef515a1feaa149bf1cf5aa6f17ca08803cf078809e53e6ea53e37d4", + "size_bytes": 327, + "source_url": "https://api.census.gov/data/2020/dec/pl?for=tract%3A%2A&get=NAME%2CH1_001N%2CH1_002N%2CH1_003N%2CP1_001N&in=state%3A56+county%3A001", + "uncompressed_sha256": "48834065f35300c5c57754057e63473dea5da48fcc9c9d3725fd26f97c163daf" +} diff --git a/tests/fixtures/validation/api_2020_dec_pl_f1c2dd2a3760.json.gz b/tests/fixtures/validation/api_2020_dec_pl_f1c2dd2a3760.json.gz new file mode 100644 index 0000000..6344511 Binary files /dev/null and b/tests/fixtures/validation/api_2020_dec_pl_f1c2dd2a3760.json.gz differ diff --git a/tests/fixtures/validation/api_2020_dec_pl_f1c2dd2a3760.json.gz.json b/tests/fixtures/validation/api_2020_dec_pl_f1c2dd2a3760.json.gz.json new file mode 100644 index 0000000..5f9329f --- /dev/null +++ b/tests/fixtures/validation/api_2020_dec_pl_f1c2dd2a3760.json.gz.json @@ -0,0 +1,9 @@ +{ + "kind": "census_api_response", + "note": "Parsed API JSON re-serialized compactly (content identical) and gzip-compressed with mtime=0. The API key is not part of the URL.", + "retrieved": "2026-11-08", + "sha256": "3b08591ac042779df1a34e0e7d16966140055d26fc23da74bee6e5abfc772ded", + "size_bytes": 205, + "source_url": "https://api.census.gov/data/2020/dec/pl?for=tract%3A%2A&get=NAME%2CH1_001N%2CH1_002N%2CH1_003N%2CP1_001N&in=state%3A36+county%3A041", + "uncompressed_sha256": "22e31e4917fa11c3a16b63e68e1069657c3178085c10b9b06d0cb92f59054e3d" +} diff --git a/tests/fixtures/validation/dec2020_apportionment_table02.xlsx b/tests/fixtures/validation/dec2020_apportionment_table02.xlsx new file mode 100644 index 0000000..f26e99b Binary files /dev/null and b/tests/fixtures/validation/dec2020_apportionment_table02.xlsx differ diff --git a/tests/fixtures/validation/dec2020_apportionment_table02.xlsx.json b/tests/fixtures/validation/dec2020_apportionment_table02.xlsx.json new file mode 100644 index 0000000..27d0652 --- /dev/null +++ b/tests/fixtures/validation/dec2020_apportionment_table02.xlsx.json @@ -0,0 +1,8 @@ +{ + "kind": "apportionment_table", + "retrieved": "2026-11-08", + "sha256": "70346bd7bb2019b48e7da57dcfc9ebdcbfcafdbec7650a5dd1d9638b1968a9e9", + "size_bytes": 13013, + "source_url": "https://www2.census.gov/programs-surveys/decennial/2020/data/apportionment/apportionment-2020-table02.xlsx", + "transformation": "none (file stored byte for byte)" +} diff --git a/tests/test_case_study_live.py b/tests/test_case_study_live.py index c44e8bf..b41249c 100644 --- a/tests/test_case_study_live.py +++ b/tests/test_case_study_live.py @@ -117,6 +117,19 @@ def test_confirmatory_manifest_matches_exploratory(self): self.assertEqual(prov["adjustments"][0]["rpp_year"], 2024) self.assertEqual(len(prov["adjustments"][0]["rpp_areas"]), 31) + def test_change_across_releases(self): + """PLAN_ROUND_15 A3 (section 8): two releases, CPI then RPP, Census VRE replicates fetched.""" + from tests.test_manifest_releases import LIVE_RESULT + + run = [o for n, o in self.outputs if n == "run_manifest"][1] + self.assertEqual(run["analysis_status"], "confirmatory") + self.assertEqual(run["deviations"], []) + self.assertEqual(run["replicates"]["source"], "census_vre") + self.assertEqual(len(run["replicates"]["files"]), 4) + h1 = run["hypotheses"][0] + for k, v in LIVE_RESULT.items(): + self.assertAlmostEqual(h1[k], v, places=2) + if __name__ == "__main__": unittest.main() diff --git a/tests/test_decennial.py b/tests/test_decennial.py new file mode 100644 index 0000000..046e993 --- /dev/null +++ b/tests/test_decennial.py @@ -0,0 +1,295 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +"""2020 Decennial Census retrieval and uncertainty semantics (PLAN_ROUND_15 C1-C3). + +Offline: canned API responses, the archived dec/pl and dec/dhc 2020 metadata +(metadata/dec/), and the frozen F11 fixtures. +""" + +import json +import math +import os +import tempfile +import unittest + +import pandas as pd + +from census import decennial as D +from census import validation as V +from census import workflows +from census.analysis import analysis_frame +from census.api import ( + DATASETS, + CensusAPIClient, + build_geo_params, + dataset_product, + infer_dataset, + require_same_product, +) +from census.metadata import archived_vintages, describe_variable, table_variables, verify_archive_file +from census.multitest import batch_z_test +from census.panel import classify_column, generate_codebook, measurement_type +from census.pipeline import Entity, ResearchOptions, build_entities, plan_extraction, replay_extraction, run_batch +from census.stats import acs_overlap_years, add_reliability_columns, calculate_cv, parse_variable_code, z_test + +P1_STATES = [ + ["NAME", "GEO_ID", "NAME", "P1_001N", "P1_001NA", "state"], + ["Michigan", "0400000US26", "Michigan", "10077331", None, "26"], + ["Wyoming", "0400000US56", "Wyoming", "576851", None, "56"], +] +H1_BLOCKS = [ + ["NAME", "H1_001N", "H1_002N", "H1_003N", "state", "county", "tract", "block"], + ["Block 1000", "3", "2", "1", "56", "001", "962700", "1000"], + ["Block 1001", "0", "0", "0", "56", "001", "962700", "1001"], +] + + +class FakeClient(CensusAPIClient): + def __init__(self): + super().__init__(api_key="", cache=None) + self.requests = [] + + def fetch(self, year, table_code, dataset=None, **kwargs): # type: ignore[override] + self.requests.append(self.build_request(year=year, table_code=table_code, dataset=dataset, **kwargs)) + self.last_error = None + return H1_BLOCKS if kwargs.get("geo_level") == "block" else P1_STATES + + +def _opts(**kw): + return ResearchOptions(use_cache=False, **kw) + + +class TestDatasets(unittest.TestCase): + def test_dataset_keys_and_products(self): + self.assertEqual(DATASETS["dec/pl"], "dec/pl") + self.assertEqual(DATASETS["dec/dhc"], "dec/dhc") + self.assertEqual(dataset_product("dec/dhc"), "decennial") + self.assertEqual(infer_dataset("P12", "decennial"), "dec/dhc") + self.assertEqual(infer_dataset("P1", "dec/pl"), "dec/pl") + self.assertEqual(infer_dataset("B01001"), "acs5") # ACS default unchanged + url = CensusAPIClient(api_key="").endpoint(2020, "dec/pl") + self.assertEqual(url, "https://api.census.gov/data/2020/dec/pl") + + def test_example_tables_exist_in_archived_dhc(self): + self.assertEqual(archived_vintages("dec/dhc"), [2020]) + self.assertTrue(verify_archive_file("dec/dhc", 2020)) + self.assertTrue(verify_archive_file("dec/pl", 2020)) + for table, first in (("P1", "P1_001N"), ("P12", "P12_002N"), ("H1", "H1_001N"), ("H3", "H3_002N")): + variables, source = table_variables(table, "dec/dhc", 2020) + self.assertEqual(source, "archive") + self.assertIn(first, variables, table) + self.assertEqual(table_variables("H3", "dec/dhc", 2020)[0]["H3_002N"]["label"], "Total:!!Occupied") + + def test_block_geo_params(self): + self.assertEqual(build_geo_params("block", state_code="56", county_code="001"), + {"for": "block:*", "in": "state:56 county:001"}) + with self.assertRaises(ValueError): + build_geo_params("block", state_code="26,56") + ents = build_entities("block", "56", ["001", "3"]) + self.assertEqual([e.county_code for e in ents], ["001", "003"]) + with self.assertRaises(ValueError): + build_entities("block", "all") + + def test_variable_codes(self): + self.assertEqual(parse_variable_code("P12_002N"), ("P12_002", "N")) + self.assertEqual(parse_variable_code("P12_002NA"), ("P12_002", "NA")) + self.assertEqual(classify_column("P1_001N"), ("P1_001", "N", False)) + self.assertEqual(classify_column("P1_001NA")[1], "ANN") + self.assertEqual(measurement_type("P1_001N"), "Count (not sample-based)") + # ACS codes are unaffected + self.assertEqual(parse_variable_code("B01001_001E"), ("B01001_001", "E")) + + def test_describe_variable_defaults_to_dhc(self): + d = describe_variable("P12_002N") + self.assertTrue(d["found"]) + self.assertEqual((d["dataset"], d["year"], d["table"]), ("dec/dhc", 2020, "P12")) + self.assertEqual(d["uncertainty"], D.UNCERTAINTY_LABEL) + self.assertEqual(describe_variable("P1_001N", dataset="dec/pl")["concept"], "RACE") + + +class TestPlanAndRun(unittest.TestCase): + def test_plan_refusals(self): + state = [Entity("WY", "state", "56")] + cases = [ + (dict(entities=state, years=[2019, 2020], dataset="dec/dhc"), "VINTAGE_UNAVAILABLE"), + (dict(entities=[Entity("Z", "zcta", zcta_code="82001")], years=[2020], dataset="dec/dhc"), + "INVALID_GEO_LEVEL"), + (dict(entities=build_entities("block", "56", ["001"]), years=[2020], dataset="dec/dhc", + table="P12"), "INVALID_GEO_LEVEL"), + (dict(entities=build_entities("block", "56", ["001"]), years=[2020], dataset="acs5", + table="B01001"), "INVALID_GEO_LEVEL"), + (dict(entities=state, years=[2020], dataset="dec/dhc", product="1-year"), "INVALID_DATASET"), + (dict(entities=state, years=[2020], dataset="acs5", product="decennial"), "INVALID_DATASET"), + (dict(entities=state, years=[2020], dataset="dec/dhc", options=_opts(inflation_base_year=2023)), + "INVALID_INPUT"), + ] + for kw, code in cases: + with self.subTest(kw=kw): + plan = plan_extraction(kw.pop("table", "P1"), kw.pop("entities"), kw.pop("years"), + options=kw.pop("options", _opts()), client=FakeClient(), **kw) + self.assertFalse(plan["success"]) + self.assertEqual(plan["error_code"], code) + + def test_plan_product_decennial(self): + plan = plan_extraction("P1", [Entity("WY", "state", "56")], [2020], + options=_opts(compute_reliability=True), client=FakeClient(), product="decennial") + self.assertTrue(plan["success"], plan.get("message")) + self.assertEqual((plan["dataset"], plan["product"]), ("dec/dhc", "decennial")) + self.assertIn(D.DISCLOSURE_AVOIDANCE_ADVISORY, plan["advisories"]) + self.assertTrue(any("compute_reliability" in a for a in plan["advisories"])) + self.assertFalse(plan["uncertainty"]["sample_based"]) + self.assertIsNone(plan["uncertainty"]["sampling_moe"]) + self.assertEqual(plan["errata"], []) + + def test_run_batch_outputs_never_carry_zero_moe(self): + with tempfile.TemporaryDirectory() as d: + res = run_batch("P1", [Entity("MI_WY", "state", "26,56")], [2020], d, "panel", + _opts(include_codebook=True, compute_reliability=True), "dec/dhc", FakeClient()) + self.assertTrue(res["success"], res.get("message")) + panel = pd.read_csv(res["panel_path"], dtype={"geoid": str}) + self.assertEqual(list(panel.columns), ["geoid", "year", "name", "total_count"]) + self.assertEqual(panel["total_count"].tolist(), [10077331, 576851]) + cb = pd.read_csv(res["codebook_path"]) + row = cb[cb["Variable_Code"] == "P1_001N"].iloc[0] + self.assertEqual(row["Measurement_Type"], "Count (not sample-based)") + self.assertIn("MOE undefined", row["Uncertainty"]) + self.assertIn("disclosure-avoidance", row["Uncertainty"]) + self.assertIn("Census year", cb[cb["Variable_Code"] == "Year"].iloc[0]["Full_Description"]) + with open(res["metadata_path"], encoding="utf-8") as f: + meta = json.load(f) + self.assertEqual(meta["product"], "decennial") + self.assertIsNone(meta["moe_confidence_level"]) + self.assertEqual(meta["uncertainty"]["sampling_moe_status"], D.UNCERTAINTY_LABEL) + self.assertIn(D.INVARIANTS_URL, meta["uncertainty"]["sources"]) + replay = replay_extraction(res["metadata_path"], client=FakeClient()) + self.assertTrue(replay["success"] and replay["verified"], replay) + + def test_run_batch_blocks(self): + client = FakeClient() + with tempfile.TemporaryDirectory() as d: + res = run_batch("H1", build_entities("block", "56", ["001"]), [2020], d, "panel", _opts(), + "dec/pl", client) + self.assertTrue(res["success"], res.get("message")) + panel = pd.read_csv(res["panel_path"], dtype={"geoid": str}) + self.assertEqual(panel["geoid"].tolist(), ["560019627001000", "560019627001001"]) + self.assertIn(D.BLOCK_ADVISORY, res["advisories"]) + self.assertEqual(client.requests[0][1]["for"], "block:*") + self.assertEqual(client.requests[0][1]["in"], "state:56 county:001") + + def test_reliability_columns_not_added(self): + df = pd.DataFrame({"P1_001N": [10.0], "P1_001NA": [None]}) + self.assertEqual(list(add_reliability_columns(df).columns), ["P1_001N", "P1_001NA"]) + + def test_codebook_uncertainty_column_only_for_decennial(self): + meta = {"P1_001N": {"label": "Total", "concept": "TOTAL POPULATION"}} + acs = generate_codebook(["B01001_001E"], {}) + self.assertNotIn("Uncertainty", acs.columns) + dec = generate_codebook(["GEOID", "P1_001N"], meta, decennial=True) + self.assertEqual(dec["Uncertainty"].iloc[0], "") + self.assertTrue(dec["Uncertainty"].iloc[1].startswith(D.UNCERTAINTY_LABEL)) + + +class TestSignificanceSemantics(unittest.TestCase): + def test_decennial_only_tests_refused(self): + for call in ( + lambda: z_test(100, 0, 90, 0, product="decennial"), + lambda: z_test(100, None, 90, None, product1="dec/pl", product2="dec/dhc"), + lambda: calculate_cv(100, 0, product="decennial"), + lambda: acs_overlap_years(2020, 2020, "decennial"), + lambda: require_same_product("dec/dhc"), + lambda: batch_z_test(pd.DataFrame({"a": [1.0], "b": [0.0], "c": [2.0], "d": [0.0]}), + "a", "b", "c", "d", product="decennial"), + ): + with self.assertRaises(ValueError) as cm: + call() + self.assertIn("not sample-based", str(cm.exception)) + + def test_acs_vs_decennial_reference(self): + r = z_test(1050, 30, 1000, None, product1="5-year", product2="decennial") + self.assertEqual(r["comparison"], "acs_vs_decennial_reference") + self.assertIn(D.MIXED_NOTE, r["notes"]) + self.assertAlmostEqual(r["se_difference"], 30 / 1.645, places=6) + self.assertTrue(r["verdict"].startswith("ACS estimate vs decennial count")) + with self.assertRaises(ValueError): # a decennial MOE is not accepted + z_test(1050, 30, 1000, 12, product1="5-year", product2="decennial") + with self.assertRaises(ValueError): # no overlap between a survey and a census + z_test(1050, 30, 1000, None, overlap_years=3, product1="5-year", product2="decennial") + acs = z_test(1050, 30, 1000, 30, product="5-year") + self.assertEqual(acs["comparison"], "acs_vs_acs") + + def test_derive_estimate_decennial_point_only(self): + out = workflows.derive_estimate("proportion", [19137, 37066], None, product="decennial") + self.assertTrue(out["success"]) + self.assertAlmostEqual(out["estimate"], 19137 / 37066) + self.assertIsNone(out["moe"]) + self.assertIsNone(out["se"]) + self.assertIsNone(out["cv_percent"]) + self.assertTrue(all(v is None for k, v in out["components"].items() if "moe" in k)) + self.assertEqual(out["uncertainty"], D.UNCERTAINTY_LABEL) + bad = workflows.derive_estimate("sum", [1, 2], [1, 1], product="decennial") + self.assertFalse(bad["success"]) + + def test_batch_significance_refuses_decennial(self): + with tempfile.TemporaryDirectory() as d: + path = os.path.join(d, "x.csv") + pd.DataFrame({"GEOID": ["1"], "a": [1.0], "b": [0.0], "c": [2.0], "d": [0.0]}).to_csv(path, index=False) + out = workflows.batch_significance(path, "a", "b", "c", "d", product="decennial") + self.assertFalse(out["success"]) + self.assertIn("not sample-based", out["error"]) + + def test_mcp_tools(self): + import mcp_server + + self.assertFalse(mcp_server.calculate_cv(100, 0, product="decennial")["success"]) + self.assertFalse(mcp_server.calculate_significance(1, 0, 2, 0, product="decennial")["success"]) + r = mcp_server.calculate_significance(1050, 30, 1000, 0, product="5-year", product2="decennial", + year1=2020, year2=2020) + self.assertEqual(r["comparison"], "acs_vs_decennial_reference") + self.assertEqual(r["overlap_years"], 0) + + def test_analysis_frame_counts_have_undefined_se(self): + df = pd.DataFrame({"GEOID": ["56001", "56003"], "Year": [2020, 2020], + "total_count": [37066, 11521], "total_male_count": [19137, 5865]}) + df.attrs.update(product="decennial", dataset="dec/dhc") # as the decennial extraction sets them + f = analysis_frame(df, variables=["total_count", "total_male_count"]) + self.assertTrue(f["total_count_se"].isna().all()) + self.assertTrue(f["total_count_cv"].isna().all()) + self.assertEqual(set(f["total_count_method"]), {"not_sample_based"}) + self.assertEqual(set(f["release"]), {"dec_2020"}) + mixed = df.assign(B01003_001E=[38000, 11800], B01003_001M=[0, 0]) + g = analysis_frame(mixed, variables=["total_count", "B01003_001"]) + self.assertEqual(set(g["release"]), {"acs5_2020+dec_2020"}) + self.assertIn(D.MIXED_NOTE, g.attrs["notes"]) + + +class _Tampered(V.CensusSource): + """Offline source that changes one block's housing-unit count in P.L. 94-171.""" + + def get_json(self, endpoint, params): + payload = super().get_json(endpoint, params) + if endpoint.endswith("/dec/pl") and params.get("for") == "block:*" and "county:001" in params.get("in", ""): + payload = [list(r) for r in payload] + payload[1][0] = str(int(payload[1][0]) + 1) + return payload + + +class TestF11(unittest.TestCase): + def test_offline_passes(self): + r = V.run_battery("F11")[0] + self.assertEqual(r.status, V.STATUS_PASSED, f"{r.notes} {r.failures[:3]}") + self.assertEqual(r.n_failed, 0) + parts = r.details["parts"] + for part in ("a_state_population", "b_block_pl_vs_dhc", "b_blocks_to_tracts", "b_blocks_to_county", + "c_county_dhc_vs_pl", "c_tract_dhc_vs_pl"): + self.assertGreater(parts[part]["compared"], 0, part) + self.assertEqual(r.details["published_state_population"], {"26": 10077331, "36": 20201249, "56": 576851}) + + def test_detects_a_changed_block(self): + r = V.run_battery("F11", source=_Tampered(live=False))[0] + self.assertEqual(r.status, V.STATUS_FAILED) + failed = {p for p, s in r.details["parts"].items() if s["failed"]} + self.assertEqual(failed, {"b_block_pl_vs_dhc", "b_blocks_to_tracts", "b_blocks_to_county"}) + self.assertTrue(math.isfinite(r.max_abs_error) and r.max_abs_error == 1) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_manifest_multistate.py b/tests/test_manifest_multistate.py index 3f83509..256c2e3 100644 --- a/tests/test_manifest_multistate.py +++ b/tests/test_manifest_multistate.py @@ -178,7 +178,10 @@ def adj(**kw): {"type": "rpp", "variables": ["income"]}]), "adjusted twice") self.check(two_state_manifest(releases=[2023, 2024], variables=[ {"name": "income", "ref": "2024:B19013_001"}, {"name": "total", "ref": "2024:B15002_001"}, - {"name": "college", "ref": "2024:B15002_015"}]), "exactly one release") + {"name": "college", "ref": "2024:B15002_015"}]), "give rpp_years") # several releases: Round 15 + self.check(two_state_manifest(releases=[2023, 2024], variables=[ + {"name": "income", "ref": "2024:B19013_001"}, {"name": "total", "ref": "2024:B15002_001"}, + {"name": "college", "ref": "2024:B15002_015"}]), "dollar_year is required") self.check(two_state_manifest(aggregation={"mapping": {"26001": "a"}}), "aggregation") tracts = two_state_manifest(geography={"level": "tract", "state": "26"}, adjustments=[{"type": "rpp", "variables": ["income"], "areas": "county_metro"}]) diff --git a/tests/test_manifest_releases.py b/tests/test_manifest_releases.py new file mode 100644 index 0000000..9a71a63 --- /dev/null +++ b/tests/test_manifest_releases.py @@ -0,0 +1,676 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +"""Manifests across releases with price adjustment and Census VRE replicates +(PLAN_ROUND_15 workstream A). + +- A1: several releases with a declared ``dollar_year`` / ``price_index`` / + ``rpp_years`` (CPI to the dollar year first, then each release's own RPP; + D15-2); validation, ``create_manifest`` defaults, and the arithmetic on a + synthetic two-release, two-state fake client. +- A2: ``"replicates": {"source": "census_vre"}``: run_manifest fetches the VRE + files itself (``census.replicate.fetch_vre``, mocked here), records URLs and + SHA-256s, reports variables without published replicates, blocks them under + the strict policy unless an analysis declares ``replicate_scope: + "covariates_only"``; supplying replicates as well is an error; overlapping + periods are refused. +- A3: the whole route of examples/manifests/mi_ny_income_change.json offline: + ``cache import`` of frozen real Census API responses + (tests/fixtures/mi_ny_change/, trimmed to the manifest's variables), the + real Census VRE files trimmed to the Michigan and New York county rows of the + manifest's lines (tests/fixtures/mi_ny_change/vre/, used as the VRE cache), + and the frozen CBSA membership of tests/fixtures/mi_ny_manifest/, with + network access refused; then ``python -m census tool run_manifest`` and the + MCP tool. + +Re-record the fixtures (live; needs CENSUS_API_KEY; never prints the key):: + + python -m tests.test_manifest_releases --record +""" + +import contextlib +import copy +import csv +import io +import json +import os +import shutil +import sys +import tempfile +import unittest +import zipfile +from unittest import mock + +import numpy as np +import pandas as pd + +from census.api import CensusAPIClient +from census.manifest import (create_manifest, load_manifest, manifest_hash, methods_section, run_manifest, + validate_manifest) +from tests.test_manifest_multistate import patched_membership + +ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..")) +MANIFEST = os.path.join(ROOT, "examples", "manifests", "mi_ny_income_change.json") +MANIFEST_HASH = "c420fcf08fcb0c758b050fea18229ffdbff6b0ec22335914c00b69558c923f8f" +FIXTURES = os.path.join(ROOT, "tests", "fixtures", "mi_ny_change") +VRE_FIXTURES = os.path.join(FIXTURES, "vre") +STATES = ("26", "36") +RELEASES = (2019, 2024) +REP_COLS = [f"rep_{i}" for i in range(1, 81)] +LIVE = os.getenv("CENSUS_LIVE_TESTS") == "1" +# live result of the manifest (2026-11-08): change in RPP- and CPI-adjusted (2024 dollars) +# county mean household income, 2015-2019 -> 2020-2024, these_areas target (replicate SE) +LIVE_RESULT = {"estimate": 3110.80, "se": 244.32, "ci_lower": 2708.93, "ci_upper": 3512.67} + + +# --------------------------------------------------------------------------- +# synthetic two-release, two-state client and VRE files (no fixtures) +# --------------------------------------------------------------------------- + +N = 8 +VALUES = { # (table, code) -> function of (state, year) -> estimates + ("B19025", "B19025_001"): lambda st, y: np.linspace(1e8, 9e8, N) * (1.0 + 0.1 * (y - 2019) / 5) * ( + 1.1 if st == "36" else 1.0), + ("B19001", "B19001_001"): lambda st, y: np.linspace(2000, 15000, N).round(), + ("B19013", "B19013_001"): lambda st, y: np.linspace(40000, 80000, N).round(), +} + + +def _payload(table, state, year): + cols = {code: fn(state, year) for (t, code), fn in VALUES.items() if t == table} + header = ["GEO_ID", "NAME"] + [f"{c}{s}" for c in cols for s in ("E", "EA", "M", "MA")] + ["state", "county"] + rows = [header] + for i in range(N): + county = f"{2 * i + 1:03d}" + vals = [] + for est in cols.values(): + vals += [f"{est[i]:.0f}", None, f"{max(est[i] * 0.03, 1):.0f}", None] + rows.append([f"0500000US{state}{county}", f"County {i} {state}"] + vals + [state, county]) + return rows + + +class TwoReleaseClient(CensusAPIClient): + def __init__(self): + super().__init__(api_key="", cache=None) + self.calls = [] + + def fetch(self, year, state_code, table_code, geo_level="place", **kwargs): # type: ignore[override] + self.calls.append((table_code, state_code, year)) + self.last_error = None + return _payload(table_code, state_code, year) + + def fetch_variable_metadata(self, year, table_code, dataset=None): # type: ignore[override] + return {} + + +def fake_vre(table, year, geo_level, state_code=None, client=None, **kw): + """Synthetic replicate frame in the fetch_vre format for both states.""" + rng = np.random.default_rng(year + len(table)) + parts = [] + for st in STATES: + payload = pd.DataFrame(_payload(table, st, year)[1:], columns=_payload(table, st, year)[0]) + for (t, code), _ in VALUES.items(): + if t != table: + continue + est = pd.to_numeric(payload[f"{code}E"]).to_numpy(float) + reps = est[:, None] * (1 + rng.normal(0, 0.02, (len(est), 80))) + part = pd.DataFrame({"GEOID": payload["state"] + payload["county"], "NAME": payload["NAME"], + "variable": code, "estimate": est, "moe": est * 0.03}) + parts.append(pd.concat([part, pd.DataFrame(reps, columns=REP_COLS)], axis=1)) + out = pd.concat(parts, ignore_index=True) + out.attrs.update(table=table, year=year, summary_level="050", state_code=state_code, + source_url=f"https://example.invalid/{year}/050/{table}.csv.zip", sha256=f"sha-{table}-{year}", + moe_check={"passed": True}) + return out + + +def release_manifest(**changes): + m = { + "schema_version": 1, "title": "Two-release synthetic test", "authors": ["agent:test"], + "created": "2026-11-08T00:00:00Z", "releases": [2019, 2024], + "geography": {"level": "county", "states": ["26", "36"]}, + "variables": [{"name": "agg", "ref": "2024:B19025_001"}, {"name": "hh", "ref": "2024:B19001_001"}], + "derivations": [{"name": "mean_inc", "op": "ratio", "num": "agg", "den": "hh"}], + "adjustments": [{"type": "rpp", "variables": ["mean_inc"], "areas": "state", "dollar_year": 2024, + "price_index": "r-cpi-u-rs", "rpp_years": {"2019": 2019, "2024": 2024}}], + "replicates": {"source": "census_vre"}, + "analyses": [{"id": "m1", "method": "fit_replicate", "formula": "mean_inc ~ C(Year)", + "args": {"target": "these_areas"}}], + "hypotheses": [{"id": "H1", "analysis": "m1", "statistic": "coefficient", "term": "C(Year)[T.2024]", + "direction": "greater"}], + } + m.update(changes) + return m + + +def covariate_manifest(**changes): + """Outcome B19013 (no published VRE) with a covariate that has replicates.""" + return release_manifest( + variables=[{"name": "income", "ref": "2024:B19013_001"}, {"name": "agg", "ref": "2024:B19025_001"}, + {"name": "hh", "ref": "2024:B19001_001"}], + adjustments=[{"type": "rpp", "variables": ["income"], "areas": "state", "dollar_year": 2024, + "price_index": "r-cpi-u-rs", "rpp_years": {"2019": 2019, "2024": 2024}}], + analyses=[{"id": "m1", "method": "fit_replicate", "formula": "income ~ hh"}], + hypotheses=[{"id": "H1", "analysis": "m1", "statistic": "coefficient", "term": "hh"}], **changes) + + +class TestValidation(unittest.TestCase): + def check(self, m, fragment): + errs = validate_manifest(m, resolve=False) + self.assertTrue(any(fragment in e for e in errs), errs) + + def test_example_valid_and_hash(self): + m = load_manifest(MANIFEST) + self.assertEqual(validate_manifest(m), []) + self.assertEqual(manifest_hash(m), MANIFEST_HASH) + + def test_round14_manifests_unchanged(self): + from tests.test_manifest_multistate import MANIFEST as MINY, MANIFEST_HASH as MINY_HASH + from tests.test_reference_workflow import MANIFEST as SINGLE, MANIFEST_HASH as SINGLE_HASH + + self.assertEqual(manifest_hash(load_manifest(MINY)), MINY_HASH) + self.assertEqual(manifest_hash(load_manifest(SINGLE)), SINGLE_HASH) + self.assertEqual(validate_manifest(load_manifest(MINY)), []) + + def test_multi_release_adjustment_errors(self): + def adj(**kw): + base = {"type": "rpp", "variables": ["mean_inc"], "areas": "state", "dollar_year": 2024, + "price_index": "r-cpi-u-rs", "rpp_years": {"2019": 2019, "2024": 2024}} + base.update(kw) + return release_manifest(adjustments=[{k: v for k, v in base.items() if v is not None}]) + + self.assertEqual(validate_manifest(adj(), resolve=False), []) + self.assertEqual(validate_manifest(adj(rpp_years={"2019": 2024, "2024": 2024}), resolve=False), []) + self.check(adj(dollar_year=None, price_index=None), "dollar_year is required") + self.check(adj(price_index=None), "price_index is required") + self.check(adj(price_index="ppi"), "price_index must be") + self.check(adj(dollar_year="2024"), "dollar_year must be an integer") + self.check(adj(dollar_year=1990), "no complete r-cpi-u-rs value") + self.check(adj(rpp_years=None), "rpp_years is required") + self.check(adj(rpp_years={"2019": 2019}), "map every release") + self.check(adj(rpp_years={"2019": 2019, "2024": 1990}), "rpp_years['2024']") + self.check(adj(rpp_years=None, rpp_year=2024), "give rpp_years") + self.check(release_manifest(aggregation={"mapping": {"26001": "a"}}), "aggregation") + one = release_manifest(releases=[2024], adjustments=[{"type": "rpp", "variables": ["mean_inc"], + "rpp_year": 2024, "rpp_years": {"2024": 2024}}]) + self.check(one, "not both") + # one release with a dollar year: CPI conversion of that release (price_index required) + one = release_manifest(releases=[2024], adjustments=[{"type": "rpp", "variables": ["mean_inc"], + "rpp_year": 2024, "dollar_year": 2023}]) + self.check(one, "price_index is required") + + def test_replicates_errors(self): + self.assertEqual(validate_manifest(release_manifest(), resolve=False), []) + self.check(release_manifest(replicates="census_vre"), "must be an object") + self.check(release_manifest(replicates={"source": "api"}), "replicates.source must be") + self.check(release_manifest(replicates={"source": "census_vre", "year": 1}), "unknown key") + self.check(release_manifest(releases=[2012, 2024], variables=[ + {"name": "agg", "ref": "2024:B19025_001"}, {"name": "hh", "ref": "2024:B19001_001"}], + adjustments=None), "no Census VRE release") + self.check(release_manifest(geography={"level": "school district (unified)", "state": "26"}, + adjustments=None), "replicates.source census_vre: Unsupported geo_level") + bg = release_manifest(geography={"level": "block group", "state": "26"}, adjustments=None) + self.assertFalse([e for e in validate_manifest(bg, resolve=False) if "replicates" in e]) + # the overlap rule: periods sharing sample years are refused, with the non-overlapping suggestion + over = release_manifest(releases=[2021, 2024], adjustments=None) + errs = validate_manifest(over, resolve=False) + self.assertTrue(any("overlapping 5-year periods" in e and "[2024]" in e for e in errs), errs) + + def test_overlap_rule_of_trend_tests_still_enforced(self): + from census.trends import linear_trend_test + + with self.assertRaisesRegex(ValueError, "periods overlap"): + linear_trend_test([2020, 2022, 2024], [1.0, 2.0, 3.0], ses=[0.1, 0.1, 0.1]) + + def test_covariates_only_declaration(self): + m = covariate_manifest() + self.assertEqual(validate_manifest(m, resolve=False), []) # declared nothing: valid, blocks at run + ok = copy.deepcopy(m) + ok["analyses"][0]["replicate_scope"] = "covariates_only" + self.assertEqual(validate_manifest(ok, resolve=False), []) + bad = copy.deepcopy(ok) + bad["analyses"][0]["replicate_scope"] = "outcome" + self.check(bad, "replicate_scope must be") + bad = copy.deepcopy(ok) + bad["analyses"][0]["args"] = {"target": "these_areas"} + self.check(bad, "cannot use target 'these_areas'") + bad = copy.deepcopy(ok) + listed = copy.deepcopy(ok) + listed["analyses"][0]["formula"] = "hh ~ income" # outcome hh is on the VRE list: allowed, noted + listed["hypotheses"][0]["term"] = "income" + self.assertEqual(validate_manifest(listed, resolve=False), []) + bad = copy.deepcopy(ok) + bad["analyses"] = [{"id": "m1", "method": "correlation_matrix", "variables": ["income", "hh"], + "replicate_scope": "covariates_only"}] + bad["hypotheses"] = [{"id": "H1", "analysis": "m1", "statistic": "correlation", "variables": ["income", "hh"]}] + self.check(bad, "fit_replicate only") + + def test_year_and_geoid_allowed_in_formulas(self): + m = release_manifest(analyses=[{"id": "m1", "method": "fit_replicate", + "formula": "mean_inc ~ C(Year) + C(GEOID)"}]) + self.assertEqual(validate_manifest(m, resolve=False), []) + + +class TestCreateManifestDefaults(unittest.TestCase): + def setUp(self): + self.dir = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, self.dir, True) + + def test_defaults_filled_into_completed_file(self): + m = release_manifest() + for k in ("dollar_year", "price_index", "rpp_years"): + del m["adjustments"][0][k] + path = os.path.join(self.dir, "study.json") + with open(path, "w", encoding="utf-8") as f: + json.dump(m, f) + res = create_manifest(path, resolve=False) + self.assertTrue(res["valid"], res["errors"]) + self.assertEqual(res["filled"], [{"adjustment": 0, "dollar_year": 2024}, + {"adjustment": 0, "price_index": "r-cpi-u-rs"}, + {"adjustment": 0, "rpp_years": {"2019": 2019, "2024": 2024}}]) + self.assertEqual(res["path"], os.path.join(self.dir, "study.completed.json")) + self.assertEqual(res["hash"], manifest_hash(release_manifest())) # same as declaring them + with open(path, encoding="utf-8") as f: + self.assertEqual(json.load(f), m) # input never overwritten + self.assertEqual(manifest_hash(load_manifest(res["path"])), res["hash"]) + self.assertTrue(any("each release's own year" in n for n in res["notes"])) + + def test_vre_note_lists_variables_without_replicates(self): + res = create_manifest(covariate_manifest(), resolve=False) + self.assertTrue(res["valid"], res["errors"]) + note = [n for n in res["notes"] if n.startswith("replicates (census_vre)")] + self.assertEqual(len(note), 1) + self.assertIn("income (B19013, 2019)", note[0]) + self.assertIn("covariates_only", note[0]) + + +class TestTwoReleaseRun(unittest.TestCase): + def setUp(self): + self.dir = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, self.dir, True) + + def run_m(self, m, vre=fake_vre, **kw): + with mock.patch("census.replicate.fetch_vre", side_effect=vre): + return run_manifest(m, os.path.join(self.dir, "run"), client=TwoReleaseClient(), use_cache=False, **kw) + + def test_cpi_then_rpp_and_replicates(self): + from census import rpp + from census.inflation import get_cpi + + res = self.run_m(release_manifest()) + self.assertEqual(res["analysis_status"], "confirmatory", res["deviations"]) + frame = res["frame"].set_index(["GEOID", "Year"]) + self.assertEqual(len(frame), 2 * 2 * N) + cpi = get_cpi(2024) / get_cpi(2019) + for (g, y), row in frame.iterrows(): + st, i = g[:2], (int(g[2:]) - 1) // 2 + raw = VALUES[("B19025", "B19025_001")](st, y)[i] / VALUES[("B19001", "B19001_001")](st, y)[i] + expected = raw * (cpi if y == 2019 else 1.0) * 100.0 / rpp.place_parity(st, y)["rpp"] + self.assertAlmostEqual(row["mean_inc"] / expected, 1.0, places=6) # API values are rounded + self.assertEqual(row["mean_inc_method"], "replicate") + adj = res["adjustments"][0] + self.assertEqual((adj["dollar_year"], adj["rpp_years"]), (2024, {"2019": 2019, "2024": 2024})) + self.assertAlmostEqual(adj["releases"][0]["cpi"]["factor"], cpi) + self.assertEqual(adj["releases"][1]["cpi"]["factor"], 1.0) + self.assertEqual(adj["price_index"]["index"], "r-cpi-u-rs") + self.assertTrue(adj["notes"][0].startswith("PRICE INDEX: release 2019")) + self.assertEqual(sorted({a["release"] for a in adj["rpp_areas"]}), [2019, 2024]) + # the replicate SE of the contrast, from the fetched replicates with the same CPI and RPP factors + reps = {} + for y in RELEASES: + vals = [] + for st in STATES: + a, h = (fake_vre(t, y, "county") for t in ("B19025", "B19001")) + a, h = (f[f["GEOID"].str.startswith(st)] for f in (a, h)) + cols = ["estimate"] + REP_COLS + r = a[cols].to_numpy(float) / h[cols].to_numpy(float) + vals.append(r * (cpi if y == 2019 else 1.0) * 100.0 / rpp.place_parity(st, y)["rpp"]) + reps[y] = np.vstack(vals) + d = (reps[2024] - reps[2019]).mean(axis=0) + h1 = res["hypotheses"].iloc[0] + self.assertAlmostEqual(h1["estimate"], d[0], places=4) + self.assertAlmostEqual(h1["se"], np.sqrt(4 / 80 * ((d[1:] - d[0]) ** 2).sum()), places=6) + self.assertEqual(h1["se_basis"], "se_sampling") + info = res["replicates"] + self.assertEqual(info["source"], "census_vre") + self.assertEqual(sorted((f["table"], f["release"]) for f in info["files"]), + [("B19001", 2019), ("B19001", 2024), ("B19025", 2019), ("B19025", 2024)]) + self.assertEqual(info["files"][0]["url"], "https://example.invalid/2019/050/B19025.csv.zip") + self.assertEqual(info["unavailable"], []) + self.assertEqual(info["derived_on_replicates"], ["mean_inc"]) + self.assertEqual(info["scaled_by_adjustments"], ["mean_inc"]) + with open(os.path.join(self.dir, "run", "bundle", "bundle.json"), encoding="utf-8") as f: + prov = json.load(f)["provenance"] + self.assertEqual(prov["replicates"]["files"][1]["sha256"], "sha-B19025-2024") + self.assertEqual(prov["adjustments"][0]["rpp_years"], {"2019": 2019, "2024": 2024}) + with open(os.path.join(self.dir, "run", "bundle", "methods.md"), encoding="utf-8") as f: + text = f.read() + self.assertIn("in this order: (1)", text) + self.assertIn("variance replicate estimate (VRE) files fetched by the toolkit", text) + self.assertIn("do not overlap", text) + + def test_one_release_dollar_year(self): + """One release with a declared dollar year: CPI conversion, then the RPP (Round 14 form otherwise).""" + from census import rpp + from census.inflation import get_cpi + + m = release_manifest(releases=[2019], variables=[{"name": "agg", "ref": "2019:B19025_001"}, + {"name": "hh", "ref": "2019:B19001_001"}], + adjustments=[{"type": "rpp", "variables": ["mean_inc"], "areas": "state", + "rpp_year": 2019, "dollar_year": 2024, "price_index": "r-cpi-u-rs"}], + analyses=[{"id": "m1", "method": "fit_replicate", "formula": "mean_inc ~ hh"}], + hypotheses=[{"id": "H1", "analysis": "m1", "statistic": "coefficient", "term": "hh"}]) + res = self.run_m(m) + self.assertEqual(res["analysis_status"], "confirmatory", res["deviations"]) + row = res["frame"].set_index("GEOID").loc["36003"] + raw = VALUES[("B19025", "B19025_001")]("36", 2019)[1] / VALUES[("B19001", "B19001_001")]("36", 2019)[1] + self.assertAlmostEqual(row["mean_inc"] / (raw * get_cpi(2024) / get_cpi(2019) * 100.0 + / rpp.place_parity("36", 2019)["rpp"]), 1.0, places=6) + self.assertEqual(res["adjustments"][0]["rpp_year"], 2019) + + def test_supplying_replicates_too_is_an_error(self): + res = self.run_m(release_manifest(), replicates=fake_vre("B19025", 2019, "county")) + self.assertFalse(res["success"]) + self.assertTrue(any("do not also supply replicates" in e for e in res["errors"]), res["errors"]) + + def test_cli_replicates_path_and_census_vre_is_invalid_input(self): + from census.workflows_inference import run_manifest as tool + + path = os.path.join(self.dir, "m.json") + with open(path, "w", encoding="utf-8") as f: + json.dump(release_manifest(), f) + reps = os.path.join(self.dir, "reps.csv") + fake_vre("B19025", 2019, "county").to_csv(reps, index=False) + with mock.patch("census.manifest.validate_manifest", return_value=[]): + out = tool(path, os.path.join(self.dir, "out"), replicates_path=reps) + self.assertFalse(out["success"]) + self.assertIn("do not also supply replicates", out["error"]) + + def test_outcome_without_vre_blocks_unless_declared(self): + res = self.run_m(covariate_manifest()) + self.assertEqual(res["analysis_status"], "confirmatory_incomplete") + kinds = [d["kind"] for d in res["deviations"]] + self.assertIn("replicates_incomplete", kinds) + missing = res["replicates"]["unavailable"] + self.assertEqual(sorted((u["variable"], u["release"]) for u in missing), [("income", 2019), ("income", 2024)]) + self.assertIn("not in the", missing[0]["reason"]) + with open(os.path.join(self.dir, "run", "bundle", "methods.md"), encoding="utf-8") as f: + self.assertIn("No Census VRE replicates are published for income (B19013)", f.read()) + + m = covariate_manifest() + m["analyses"][0]["replicate_scope"] = "covariates_only" + res = self.run_m(m) + self.assertEqual(res["analysis_status"], "confirmatory", res["deviations"]) + m1 = res["results"]["m1"] + self.assertEqual(m1.diagnostics["replicates_declared_fixed"], ["income"]) + self.assertEqual(m1.provenance["replicate_variables"], ["hh"]) + self.assertTrue(any(n.startswith("DECLARED COVARIATE-ONLY REPLICATES") for n in m1.notes)) + self.assertFalse(m1.diagnostics.get("replicates_incomplete")) + self.assertEqual(res["replicates"]["declared_covariates_only"], {"m1": ["income"]}) + with open(os.path.join(self.dir, "run", "bundle", "methods.md"), encoding="utf-8") as f: + self.assertIn("replicate scope `covariates_only`", f.read()) + + def test_fixed_variables_refuse_these_areas(self): + from census.models import fit_replicate + + frame = pd.DataFrame({"GEOID": ["1", "2", "3", "4"], "y": [1.0, 2.0, 3.5, 4.0], "y_se": 0.1, + "x": [1.0, 2.0, 3.0, 4.0], "x_se": 0.1}) + reps = pd.DataFrame({"GEOID": frame["GEOID"], "variable": "x", "estimate": frame["x"]}) + reps = pd.concat([reps, pd.DataFrame(np.outer(frame["x"], np.ones(80)) * 1.01, columns=REP_COLS)], axis=1) + with self.assertRaisesRegex(ValueError, "these_areas"): + fit_replicate("y ~ x", frame, reps, target="these_areas", fixed_variables=["y"]) + res = fit_replicate("y ~ x", frame, reps, replicate_policy="strict", fixed_variables=["y"]) + self.assertEqual(res.diagnostics["replicates_declared_fixed"], ["y"]) + + def test_fetch_failure_blocks(self): + from census.replicate import VREDownloadError + + res = self.run_m(release_manifest(), vre=mock.Mock(side_effect=VREDownloadError("HTTP 503"))) + self.assertEqual(res["analysis_status"], "confirmatory_incomplete") + kinds = [d["kind"] for d in res["deviations"]] + self.assertIn("replicates_unavailable", kinds) + self.assertIn("hypothesis_not_tested", kinds) + self.assertIn("HTTP 503", res["replicates"]["error"]) + + def test_caller_supplied_replicates_without_year_fail_multi_release_adjustment(self): + m = release_manifest(replicates=None) + del m["replicates"] + reps = fake_vre("B19025", 2019, "county") + reps["variable"] = "agg" + res = self.run_m(m, replicates=reps) + self.assertIn("adjustment_failed", [d["kind"] for d in res["deviations"]]) + + +# --------------------------------------------------------------------------- +# frozen real-data route (examples/manifests/mi_ny_income_change.json) +# --------------------------------------------------------------------------- + +def run_cli(argv): + from census.__main__ import main + + buf = io.StringIO() + with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(io.StringIO()): + code = main(argv) + return code, json.loads(buf.getvalue()) + + +def vre_fixture_files(): + out = [] + for dirpath, _, files in os.walk(VRE_FIXTURES): + out += [os.path.join(dirpath, f) for f in files if f.endswith(".zip")] + return sorted(out) + + +class TestFrozenChangeRoute(unittest.TestCase): + @classmethod + def setUpClass(cls): + from tests.test_reference_workflow import offline + + cls.dir = tempfile.mkdtemp(prefix="mi_ny_change_") + vre_dir = os.path.join(cls.dir, "vre") + shutil.copytree(VRE_FIXTURES, vre_dir) + cls.env = mock.patch.dict(os.environ, {"CENSUS_CACHE_PATH": os.path.join(cls.dir, "cache.db"), + "CENSUS_VRE_DIR": vre_dir}) + cls.env.start() + with offline(), patched_membership(): + code, cls.imported = run_cli(["cache", "import", FIXTURES, "--json"]) + assert code == 0, cls.imported + code, cls.created = run_cli(["tool", "create_manifest", "--arg", f"manifest_path={MANIFEST}", "--json"]) + assert code == 0, cls.created + code, cls.cli = run_cli(["tool", "run_manifest", "--arg", f"manifest_path={MANIFEST}", + "--arg", f"output_dir={os.path.join(cls.dir, 'cli')}", "--json"]) + cls.cli_code = code + + @classmethod + def tearDownClass(cls): + cls.env.stop() + shutil.rmtree(cls.dir, ignore_errors=True) + + def test_inputs(self): + self.assertEqual(self.imported["imported"], 12) # 2 tables x (2 states + nation) x 2 releases + self.assertFalse(self.imported["synthetic"]) + self.assertEqual(self.created["hash"], MANIFEST_HASH) + self.assertEqual(self.created["notes"], []) + self.assertEqual(len(vre_fixture_files()), 4) + + def test_confirmatory_result(self): + out = self.cli + self.assertEqual(self.cli_code, 0, out.get("error")) + self.assertEqual(out["analysis_status"], "confirmatory") + self.assertEqual(out["deviations"], []) + self.assertEqual(out["manifest_hash"], MANIFEST_HASH) + self.assertFalse(out["data_source"]["synthetic"]) + h1 = out["hypotheses"][0] + self.assertEqual((h1["id"], h1["status"], h1["se_basis"]), ("H1", "tested", "se_sampling")) + for k, v in LIVE_RESULT.items(): + self.assertAlmostEqual(h1[k], v, places=2) + self.assertTrue(h1["significant"]) + + def test_vre_provenance(self): + rep = self.cli["replicates"] + self.assertEqual(rep["source"], "census_vre") + self.assertEqual(rep["unavailable"], []) + self.assertEqual(rep["variables"], {"agg_income": [2019, 2024], "households": [2019, 2024]}) + sidecars = {} + for path in vre_fixture_files(): + with open(path + ".json", encoding="utf-8") as f: + meta = json.load(f) + sidecars[meta["source_url"]] = meta["sha256"] + self.assertEqual({f["url"]: f["sha256"] for f in rep["files"]}, sidecars) + self.assertTrue(all(u.startswith("https://www2.census.gov/programs-surveys/acs/replicate_estimates/") + for u in sidecars)) + self.assertTrue(all(f["moe_check"]["passed"] for f in rep["files"])) + with open(os.path.join(self.dir, "cli", "bundle", "results", "run.json"), encoding="utf-8") as f: + run = json.load(f) + self.assertEqual(run["n_units"], 290) + self.assertEqual(run["replicates"]["derived_on_replicates_rows"], + {"mean_hh_income": {"rows_replicate": 290, "rows": 290}}) + adj = run["adjustments"][0] + self.assertEqual(adj["rpp_years"], {"2019": 2019, "2024": 2024}) + self.assertAlmostEqual(adj["releases"][0]["cpi"]["factor"], 462.511 / 375.813) + with open(os.path.join(self.dir, "cli", "bundle", "bundle.json"), encoding="utf-8") as f: + prov = json.load(f)["provenance"] + self.assertEqual(prov["replicates"]["files"], run["replicates"]["files"]) + self.assertEqual(prov["adjustments"][0]["price_index"]["index"], "r-cpi-u-rs") + + def test_mcp_route_agrees(self): + from tests.test_reference_workflow import _mcp_call, offline + + import mcp_server + + with offline(), patched_membership(): + out = _mcp_call(mcp_server, "run_manifest", {"manifest_path": MANIFEST, + "output_dir": os.path.join(self.dir, "mcp")}) + self.assertEqual(out["analysis_status"], "confirmatory", out.get("error")) + a, b = out["hypotheses"][0], self.cli["hypotheses"][0] + for k in ("estimate", "se", "ci_lower", "ci_upper", "p_value"): + self.assertAlmostEqual(a[k], b[k], places=9) + + +@unittest.skipUnless(LIVE, "set CENSUS_LIVE_TESTS=1 to run live Census API tests") +class TestLiveChangeRoute(unittest.TestCase): + """The same route on the live Census API, VRE files and boundary files (default caches).""" + + def test_live_confirmatory_run(self): + d = tempfile.mkdtemp(prefix="mi_ny_change_live_") + self.addCleanup(shutil.rmtree, d, True) + code, out = run_cli(["tool", "run_manifest", "--arg", f"manifest_path={MANIFEST}", + "--arg", f"output_dir={d}", "--json"]) + self.assertEqual(code, 0, out.get("error")) + self.assertEqual(out["analysis_status"], "confirmatory") + self.assertEqual(out["data_source"]["synthetic"], False) + h1 = out["hypotheses"][0] + for k, v in LIVE_RESULT.items(): + self.assertAlmostEqual(h1[k], v, places=2) + self.assertEqual(len(out["replicates"]["files"]), 4) + + +# --------------------------------------------------------------------------- +# recording (live) +# --------------------------------------------------------------------------- + +def _trim_vre(raw, geoid_prefixes, orders): + """CSV bytes of a VRE download with the header, the table-title rows and only the rows of + the given GEOID prefixes (after the summary-level prefix) and line ORDER values.""" + from census.replicate import _decompress, normalize_vre_geoid + + text = _decompress(raw).decode("latin-1") + lines = text.splitlines(keepends=True) + out = [lines[0]] + for line in lines[1:]: + row = next(csv.reader([line])) + geoid = row[1].strip() + if not geoid: + out.append(line) + elif normalize_vre_geoid(geoid)[:2] in geoid_prefixes and row[3].strip().isdigit() \ + and int(row[3]) in orders: + out.append(line) + return "".join(out).encode("latin-1") + + +def record(): + """Fetch the eight API responses (2019 and 2024 acs5, all counties of 26 and 36; B19025 and + B19001) through the toolkit client, trim each to the manifest's variables, and write trimmed + copies of the four Census VRE county files (Michigan and New York rows of line 001) with + sidecars. Never writes or prints the API key.""" + from census import replicate as vre + from census.manifest import tables_and_refs + + m = load_manifest(MANIFEST) + keep = {t: [r.split(":")[1] for r in refs] for t, refs in tables_and_refs(m).items()} + client = CensusAPIClient() + os.makedirs(FIXTURES, exist_ok=True) + for year in RELEASES: + for st in STATES: + request = {"year": year, "dataset": "acs5", "geo_level": "county", "state_code": st, + "county_code": "*"} + doc = {"_about": (f"REAL Census Data API responses ({year} acs5, all counties of state {st}, " + "get=NAME,group(
)), recorded by `python -m tests.test_manifest_releases " + "--record` for examples/manifests/mi_ny_income_change.json; columns trimmed to the " + "manifest's variables (estimate, MOE and annotations) plus GEO_ID, NAME, state, county."), + "request": request, "requests": {}} + for table, codes in keep.items(): + rows = client.fetch(table_code=table, **request) + if rows is None: + raise SystemExit(f"fetch failed for {table} {year} state {st}") + header = rows[0] + cols = [i for i, h in enumerate(header) + if h in ("GEO_ID", "NAME", "state", "county") + or any(h == f"{c}{s}" for c in codes for s in ("E", "EA", "M", "MA"))] + doc["requests"][table] = [[r[i] for i in cols] for r in rows] + with open(os.path.join(FIXTURES, f"frozen_api_responses_{year}_{st}.json"), "w", encoding="utf-8", + newline="\n") as f: + json.dump(doc, f, separators=(",", ":")) + f.write("\n") + for year in RELEASES: # national rows of each table: the cross-release continuity check reads them + request = {"year": year, "dataset": "acs5", "geo_level": "us", "state_code": None} + doc = {"_about": (f"REAL Census Data API responses ({year} acs5, nation, get=NAME,group(
)), " + "untrimmed, read by the continuity check of a multi-release extraction; recorded by " + "`python -m tests.test_manifest_releases --record`."), + "request": request, "requests": {}} + for table in keep: + rows = client.fetch(table_code=table, **request) + if rows is None: + raise SystemExit(f"fetch failed for {table} {year} nation") + doc["requests"][table] = rows + with open(os.path.join(FIXTURES, f"frozen_api_responses_{year}_us.json"), "w", encoding="utf-8", + newline="\n") as f: + json.dump(doc, f, separators=(",", ":")) + f.write("\n") + for table, codes in keep.items(): + orders = {int(c.split("_")[1]) for c in codes} + for year in RELEASES: + frame = vre.fetch_vre(table, year, "county") # downloads into the VRE cache if needed + with open(frame.attrs["cache_path"], "rb") as f: + raw = f.read() + data = _trim_vre(raw, STATES, orders) + out_dir = os.path.join(VRE_FIXTURES, str(year), frame.attrs["summary_level"]) + os.makedirs(out_dir, exist_ok=True) + name = os.path.basename(frame.attrs["cache_path"]) + member = name[: -len(".zip")] if name.endswith(".zip") else name + buf = io.BytesIO() + with zipfile.ZipFile(buf, "w", zipfile.ZIP_DEFLATED) as zf: + info = zipfile.ZipInfo(member, date_time=(2026, 11, 8, 0, 0, 0)) + info.compress_type = zipfile.ZIP_DEFLATED + zf.writestr(info, data) + trimmed = buf.getvalue() + with open(os.path.join(out_dir, name), "wb") as f: + f.write(trimmed) + import hashlib + + meta = {"source_url": frame.attrs["source_url"], "sha256": hashlib.sha256(trimmed).hexdigest(), + "size_bytes": len(trimmed), "content_type": "application/zip", + "downloaded_at": "2026-11-08T00:00:00Z", + "_about": (f"TRIMMED copy of the Census VRE file at source_url (original SHA-256 " + f"{frame.attrs['sha256']}): header, table-title rows, and the rows of the counties " + f"of states {', '.join(STATES)} for line(s) {sorted(orders)}; recorded by " + "`python -m tests.test_manifest_releases --record`.")} + with open(os.path.join(out_dir, name + ".json"), "w", encoding="utf-8", newline="\n") as f: + json.dump(meta, f, indent=2) + f.write("\n") + print(f"recorded fixtures into {FIXTURES}") + + +if __name__ == "__main__": + if "--record" in sys.argv: + record() + else: + unittest.main() diff --git a/tests/test_review_fixes_r15.py b/tests/test_review_fixes_r15.py new file mode 100644 index 0000000..7b20406 --- /dev/null +++ b/tests/test_review_fixes_r15.py @@ -0,0 +1,355 @@ +# SPDX-License-Identifier: AGPL-3.0-or-later +"""Regression tests for the Round 15 code-review findings (offline). + +1. Overlapping releases are refused for any replicate source (validation for a + declared need; a blocking deviation for caller-supplied replicates); the + methods text claims independence only for non-overlapping releases. +2. validate_manifest / create_manifest never raise on malformed variables. +3. census_vre at per-state VRE levels needs explicit states; an empty call list + is a clear error. +4. replicate_scope 'covariates_only' is allowed for registry-listed outcomes; + the run holds the outcome fixed and records that replicates were available. +5. / 6. Decennial counts are recognized only from explicit evidence. +7. An empty frame passes through the adjustments. +8. Generic product aliases ("census", "dec") are not decennial. +9. _fill_adjustment_defaults uses latest_rpp_year; releases helper. +10. spreg fits print nothing to stdout, under one lock. +""" + +import os +import shutil +import tempfile +import unittest +from unittest import mock + +import numpy as np +import pandas as pd + +from census.manifest import (create_manifest, methods_section, run_manifest, validate_manifest) +from tests.test_manifest_releases import TwoReleaseClient, covariate_manifest, fake_vre, release_manifest + + +def _errs(m): + return validate_manifest(m, resolve=False) + + +def _caller_reps(years, names=(("B19025", "B19025_001", "agg"), ("B19001", "B19001_001", "hh"))): + parts = [] + for y in years: + for table, code, name in names: + f = fake_vre(table, y, "county") + f = f[f["variable"] == code].copy() + f["variable"] = name + f.insert(1, "Year", y) + parts.append(f.drop(columns=["NAME", "moe"])) + return pd.concat(parts, ignore_index=True) + + +class _Run(unittest.TestCase): + def setUp(self): + self.dir = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, self.dir, True) + + def run_m(self, m, vre=fake_vre, **kw): + with mock.patch("census.replicate.fetch_vre", side_effect=vre): + return run_manifest(m, os.path.join(self.dir, "run"), client=TwoReleaseClient(), use_cache=False, **kw) + + def methods(self): + with open(os.path.join(self.dir, "run", "bundle", "methods.md"), encoding="utf-8") as f: + return f.read() + + +def _no_vre_manifest(releases, required=False, formula="hh ~ agg"): + m = release_manifest(releases=releases, adjustments=None, + variables=[{"name": "agg", "ref": f"{releases[-1]}:B19025_001"}, + {"name": "hh", "ref": f"{releases[-1]}:B19001_001"}], + derivations=None, + analyses=[{"id": "m1", "method": "fit_replicate", "formula": formula}], + hypotheses=[{"id": "H1", "analysis": "m1", "statistic": "coefficient", "term": "agg"}]) + for k in ("replicates", "adjustments", "derivations"): + m.pop(k) + if required: + m["analyses"][0]["replicates"] = "required" + return m + + +class TestOverlapAnyReplicateSource(_Run): + def test_validation_refuses_required_replicates_across_overlapping_releases(self): + errs = _errs(_no_vre_manifest([2021, 2024], required=True)) + self.assertTrue(any("require replicates with overlapping 5-year periods" in e for e in errs), errs) + self.assertEqual(_errs(_no_vre_manifest([2019, 2024], required=True)), []) + self.assertEqual(_errs(_no_vre_manifest([2021, 2024])), []) # nothing declared: checked at run time + + def test_caller_supplied_overlapping_replicates_block(self): + res = self.run_m(_no_vre_manifest([2021, 2024]), replicates=_caller_reps([2021, 2024])) + self.assertEqual(res["analysis_status"], "confirmatory_incomplete") + dev = [d for d in res["deviations"] if d["kind"] == "replicates_overlapping_releases"] + self.assertEqual(len(dev), 1, res["deviations"]) + self.assertIn("share sample years", dev[0]["detail"]) + self.assertNotIn("m1", res["results"]) + + def test_caller_supplied_non_overlapping_replicates_run(self): + res = self.run_m(_no_vre_manifest([2019, 2024]), replicates=_caller_reps([2019, 2024])) + self.assertFalse([d for d in res["deviations"] if d["kind"] == "replicates_overlapping_releases"]) + self.assertIn("m1", res["results"]) + + def test_methods_independence_only_without_overlap(self): + over = _no_vre_manifest([2021, 2024], formula="hh ~ agg + C(Year)") + text = methods_section(over) + self.assertNotIn("independent samples", text) + self.assertIn("overlap (shared sample years)", text) + ok = methods_section(_no_vre_manifest([2019, 2024], formula="hh ~ agg + C(Year)")) + self.assertIn("share no sample years", ok) + self.assertIn("independent samples", ok) + + +class TestMalformedVariablesNeverRaise(unittest.TestCase): + def test_variable_without_name(self): + m = release_manifest() + m["variables"][0] = {"ref": "2024:B19025_001"} + errs = _errs(m) + self.assertTrue(any("needs 'name' and 'ref'" in e for e in errs), errs) + res = create_manifest(m, resolve=False) # vre_unavailable runs for the notes + self.assertFalse(res["valid"]) + + def test_malformed_derivation_and_variables(self): + m = release_manifest(derivations=["mean_inc"]) + self.assertTrue(any("derivations[0] must be an object" in e for e in _errs(m))) + self.assertFalse(create_manifest(m, resolve=False)["valid"]) + m = release_manifest(variables=5) + self.assertTrue(any("variables must be a list" in e for e in _errs(m))) + self.assertFalse(create_manifest(m, resolve=False)["valid"]) + + +class TestPerStateVreLevels(unittest.TestCase): + def test_validation_requires_states(self): + m = release_manifest(geography={"level": "tract"}, adjustments=None) + self.assertTrue(any("published per state" in e for e in _errs(m)), _errs(m)) + ok = release_manifest(geography={"level": "tract", "state": "26"}, adjustments=None) + self.assertFalse([e for e in _errs(ok) if "per state" in e]) + + def test_empty_call_list_is_a_clear_error(self): + from census.manifest import _census_vre_replicates + + m = release_manifest(geography={"level": "tract"}, adjustments=None) + frame = pd.DataFrame({"GEOID": ["26001000100"], "Year": [2024]}) + with mock.patch("census.replicate.fetch_vre", side_effect=AssertionError("must not fetch")): + with self.assertRaisesRegex(ValueError, "published per state.*names no state"): + _census_vre_replicates(m, frame, {"agg": {2024: "B19025_001"}}) + + +class TestCovariatesOnlyListedOutcome(_Run): + def manifest(self): + m = release_manifest(adjustments=None, derivations=None, + analyses=[{"id": "m1", "method": "fit_replicate", "formula": "hh ~ agg", + "replicate_scope": "covariates_only"}], + hypotheses=[{"id": "H1", "analysis": "m1", "statistic": "coefficient", "term": "agg"}]) + for k in ("adjustments", "derivations"): + m.pop(k) + return m + + def test_declaration_allowed_and_noted(self): + m = self.manifest() + self.assertEqual(_errs(m), []) + res = create_manifest(m, resolve=False) + self.assertTrue(res["valid"], res["errors"]) + note = [n for n in res["notes"] if "covariates_only" in n] + self.assertEqual(len(note), 1, res["notes"]) + self.assertIn("held at its estimate", note[0]) + + def test_run_holds_outcome_fixed_and_records_it(self): + res = self.run_m(self.manifest()) + self.assertEqual(res["analysis_status"], "confirmatory", res["deviations"]) + self.assertEqual(res["results"]["m1"].diagnostics["replicates_declared_fixed"], ["hh"]) + self.assertEqual(res["replicates"]["covariates_only_outcome_had_replicates"], {"m1": ["hh"]}) + self.assertEqual(res["replicates"]["declared_covariates_only"], {"m1": ["hh"]}) + self.assertIn("although replicates were available", self.methods()) + + def test_outcome_without_replicates_unchanged(self): + m = covariate_manifest() + m["analyses"][0]["replicate_scope"] = "covariates_only" + res = self.run_m(m) + self.assertEqual(res["analysis_status"], "confirmatory", res["deviations"]) + self.assertNotIn("covariates_only_outcome_had_replicates", res["replicates"]) + self.assertIn("declared because it has no published replicates", self.methods()) + + +class TestEmptyFrameAdjustments(_Run): + def test_exclusions_remove_everything(self): + geoids = [f"{st}{2 * i + 1:03d}" for st in ("26", "36") for i in range(8)] + m = release_manifest(exclusions=[{"rule": "geoid_exclude", "geoids": geoids}]) + res = self.run_m(m) + kinds = [d["kind"] for d in res["deviations"]] + self.assertNotIn("adjustment_failed", kinds) + self.assertIn("analysis_failed", kinds) + self.assertEqual(res["adjustments"], []) + + def test_direct(self): + from census.manifest import _apply_adjustments + + m = release_manifest() + frame = pd.DataFrame(columns=["GEOID", "Year", "mean_inc", "mean_inc_se"]) + out, _, log, failure = _apply_adjustments(frame, None, m, [2019, 2024]) + self.assertIsNone(failure) + self.assertTrue(out.empty) + + +class TestCleanup(unittest.TestCase): + def test_fill_defaults_uses_latest_rpp_year(self): + m = release_manifest(releases=[2024], adjustments=[{"type": "rpp", "variables": ["mean_inc"], + "areas": "state"}]) + with mock.patch("census.manifest.latest_rpp_year", return_value=2021): + res = create_manifest(m, resolve=False) + self.assertEqual(res["filled"], [{"adjustment": 0, "rpp_year": 2021}]) + + def test_bool_release_rejected(self): + self.assertTrue(any("releases must be" in e for e in _errs(release_manifest(releases=[2019, True])))) + + def test_releases_helper(self): + from census.manifest import manifest_releases + + self.assertEqual(manifest_releases({"releases": [2019, True, "2020", 2024]}), [2019, 2024]) + self.assertEqual(manifest_releases({"releases": 2019}), []) + + +class TestDecennialEvidence(unittest.TestCase): + """Findings 5 and 6: decennial counts only from explicit evidence; defaults and base codes.""" + + def test_user_count_column_is_not_decennial(self): + from census.analysis import analysis_frame, is_decennial_count + from census.decennial import MIXED_NOTE + + df = pd.DataFrame({"GEOID": ["01001", "01003"], "Year": [2020, 2020], "crime_count": [12, 30], + "B01003_001E": [55000, 220000], "B01003_001M": [0, 0]}) + f = analysis_frame(df, variables=["crime_count", "B01003_001"]) + self.assertEqual(set(f["crime_count_method"]), {"no_sampling_error"}) + self.assertEqual(f["crime_count_se"].tolist(), [0.0, 0.0]) + self.assertNotIn(MIXED_NOTE, f.attrs["notes"]) + self.assertEqual(set(f["release"]), {"acs5_2020"}) + self.assertFalse(is_decennial_count("crime_count")) + self.assertTrue(is_decennial_count("crime_count", decennial=True)) + self.assertFalse(is_decennial_count("P1_001N", decennial=False)) + + def test_archived_code_is_decennial_without_attrs(self): + from census.analysis import analysis_frame, is_decennial_count + + self.assertTrue(is_decennial_count("P1_001N")) + df = pd.DataFrame({"GEOID": ["56001", "56003"], "P1_001N": [37066, 11521]}) + f = analysis_frame(df) # default variables: the count, named by its base code + self.assertEqual(list(f.attrs["variables"]), ["P1_001"]) + self.assertTrue(f["P1_001_se"].isna().all()) + self.assertEqual(set(f["P1_001_method"]), {"not_sample_based"}) + self.assertEqual(set(f["release"]), {"dec_2020"}) + g = analysis_frame(df, variables=["P1_001"]) # by base code + self.assertEqual(g.attrs["variables"]["P1_001"]["estimate_column"], "P1_001N") + + def test_slug_frame_with_attrs_defaults_and_base_code(self): + from census.analysis import analysis_frame + + df = pd.DataFrame({"GEOID": ["56001", "56003"], "Year": [2020, 2020], "total_count": [37066, 11521]}) + df.attrs.update(product="decennial", dataset="dec/dhc", variable_codes={"total_count": "P1_001N"}) + f = analysis_frame(df) + self.assertEqual(list(f.attrs["variables"]), ["total_count"]) + self.assertEqual(set(f["release"]), {"dec_2020"}) + g = analysis_frame(df, variables={"pop": "P1_001"}) + self.assertEqual(g.attrs["variables"]["pop"]["estimate_column"], "total_count") + self.assertEqual(set(g["pop_method"]), {"not_sample_based"}) + with self.assertRaises(KeyError): # not a decennial frame: no base-code lookup + analysis_frame(df.copy().pipe(lambda d: setattr(d, "attrs", {}) or d), variables={"pop": "P1_001"}) + + def test_extraction_outputs_carry_evidence(self): + from census.analysis import _read, analysis_frame + from census.pipeline import Entity, ResearchOptions, run_batch + from tests.test_decennial import FakeClient + + with tempfile.TemporaryDirectory() as d: + for fmt in ("panel", "parquet"): + res = run_batch("P1", [Entity("MI_WY", "state", "26,56")], [2020], os.path.join(d, fmt), fmt, + ResearchOptions(use_cache=False, include_codebook=True), "dec/dhc", FakeClient()) + self.assertTrue(res["success"], res.get("message")) + raw = _read(res["panel_path"]) # CSV: extraction metadata; Parquet: attrs in the file + self.assertEqual(raw.attrs.get("product"), "decennial", fmt) + self.assertEqual(raw.attrs.get("variable_codes"), {"total_count": "P1_001N"}, fmt) + frame = raw.rename(columns={"geoid": "GEOID", "year": "Year"}) + f = analysis_frame(frame, variables={"pop": "P1_001"}) + self.assertEqual(set(f["pop_method"]), {"not_sample_based"}) + self.assertEqual(set(f["release"]), {"dec_2020"}) + # an ACS-like CSV elsewhere carries no decennial evidence + path = os.path.join(d, "mine.csv") + pd.DataFrame({"GEOID": ["1"], "crime_count": [3]}).to_csv(path, index=False) + self.assertEqual(_read(path).attrs, {}) + + +class TestDecennialAliases(unittest.TestCase): + def test_generic_words_not_decennial(self): + from census.decennial import is_decennial + + for word in ("census", "dec", "Census"): + self.assertFalse(is_decennial(word), word) + for word in ("decennial", "dec/pl", "dec/dhc", "decennial census", "2020 census"): + self.assertTrue(is_decennial(word), word) + + def test_plan_message_for_dec(self): + from census.pipeline import Entity, ResearchOptions, plan_extraction + from tests.test_decennial import FakeClient + + for ds, fragment in (("dec", "Unsupported dataset 'dec'"), ("decennial", "is not a dataset key")): + plan = plan_extraction("P1", [Entity("WY", "state", "56")], [2020], dataset=ds, + options=ResearchOptions(use_cache=False), client=FakeClient()) + self.assertFalse(plan["success"]) + self.assertIn(fragment, plan["error"]) + self.assertIn("dec/pl", plan["error"]) + self.assertNotIn("is an ACS dataset", plan["error"]) + + +class TestSpregStdout(unittest.TestCase): + """Finding 10: spreg's class-name print is silenced at its source, without swapping sys.stdout.""" + + @classmethod + def setUpClass(cls): + from census.simulation import lattice_weights, make_replicates, replicate_long, sar_field + + rng = np.random.default_rng(3) + cls.w = lattice_weights(5) + ids = cls.w.id_order + x = rng.uniform(0, 10, 25) + y = 5 + 2 * x + sar_field(cls.w.full()[0], 0.4, 2.0, rng) + cls.df = pd.DataFrame({"GEOID": ids, "y": y, "x": x}) + cls.reps = replicate_long(ids, ["y"], y[:, None], make_replicates(y[:, None], np.ones((25, 1)), rng)) + + def test_no_stdout_from_any_estimator_or_refit(self): + import contextlib + import io + + from census.spatial import spatial_regression + + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + for model in ("lag", "error"): + for method in ("gm", "ml"): + spatial_regression("y ~ x", self.df, self.w, model=model, method=method, replicates=self.reps) + self.assertEqual(buf.getvalue(), "") + + def test_fit_does_not_swap_process_stdout(self): + import sys + + import spreg + + from census.spatial import spatial_regression + + seen = [] + real = spreg.GM_Lag + + def spy(*a, **k): + seen.append(sys.stdout) + return real(*a, **k) + + outer = sys.stdout + with mock.patch.object(spreg, "GM_Lag", side_effect=spy): + spatial_regression("y ~ x", self.df, self.w, model="lag", method="gm") + self.assertTrue(seen) + self.assertTrue(all(s is outer for s in seen)) # other threads' output is never redirected + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_spatial.py b/tests/test_spatial.py index 8a52716..2de0aef 100644 --- a/tests/test_spatial.py +++ b/tests/test_spatial.py @@ -178,12 +178,31 @@ def test_local_morans_correction(self): from census.spatial import local_morans out = local_morans(self.df, "y", self.w, permutations=199, correction="bh", replicates=self.reps) self.assertEqual(len(out), 64) - self.assertTrue((out["p_adjusted"] >= out["p_sim"] - 1e-15).all()) + self.assertTrue((out["p_adjusted"] >= out["p_value"] - 1e-15).all()) self.assertTrue(set(out["quadrant"]) <= {"HH", "LH", "LL", "HL"}) self.assertTrue(((out["cluster"] == "ns") | out["significant"]).all()) self.assertTrue(out["replicate_quadrant_share"].between(0, 1).all()) from census.multitest import adjust_pvalues - np.testing.assert_allclose(out["p_adjusted"], adjust_pvalues(out["p_sim"].to_numpy(), "bh")) + # default: two-sided p_value = min(1, 2 p_sim), BH-adjusted (PLAN_ROUND_15 B, owner decision) + np.testing.assert_allclose(out["p_value"], np.minimum(1.0, 2 * out["p_sim"])) + np.testing.assert_allclose(out["p_adjusted"], adjust_pvalues(out["p_value"].to_numpy(), "bh")) + np.testing.assert_array_equal(out["significant"], out["p_adjusted"] <= 0.10) + self.assertIn("two-sided", out.attrs["notes"][0]) + old = local_morans(self.df, "y", self.w, permutations=199, sided="folded") + np.testing.assert_allclose(old["p_value"], old["p_sim"]) + np.testing.assert_allclose(old["p_adjusted"], adjust_pvalues(old["p_sim"].to_numpy(), "bh")) + with self.assertRaises(ValueError): + local_morans(self.df, "y", self.w, permutations=19, sided="one") + + def test_local_morans_defaults(self): + import inspect + + from census.spatial import local_morans + sig = inspect.signature(local_morans).parameters + self.assertEqual(sig["permutations"].default, 9999) + self.assertEqual(sig["sided"].default, "two") + self.assertEqual(sig["correction"].default, "bh") + self.assertEqual(sig["alpha"].default, 0.10) @unittest.skipUnless(HAVE_PYSAL, "PySAL stack not installed") @@ -261,6 +280,28 @@ def test_replicate_refits(self): with self.assertRaises(ValueError): spatial_regression("y ~ x", df, w, target="these_areas") + def test_default_method_per_model(self): + # PLAN_ROUND_15 B (owner decision): GMM for lag, ML for error; ML lag stays available with its caveat + from census.simulation import lattice_weights, sar_field + from census.spatial import DEFAULT_METHODS, spatial_regression + self.assertEqual(DEFAULT_METHODS, {"lag": "gm", "error": "ml"}) + rng = np.random.default_rng(5) + w = lattice_weights(7) + x = rng.uniform(0, 10, 49) + df = pd.DataFrame({"GEOID": w.id_order, "x": x, "y": 5 + 2 * x + sar_field(w.full()[0], 0.4, 2.0, rng)}) + lag = spatial_regression("y ~ x", df, w) + self.assertIn("GMM", lag.method) + self.assertIsNone(lag.provenance["ml_method"]) + self.assertIn("the default for the lag model", " ".join(lag.notes)) + explicit = spatial_regression("y ~ x", df, w, method="gm") + np.testing.assert_allclose(lag.coefficients["estimate"], explicit.coefficients["estimate"]) + ml = spatial_regression("y ~ x", df, w, method="ml") + self.assertIn("ML", ml.method) + self.assertIn("ML lag intervals for rho covered 0.48", " ".join(ml.notes)) + err = spatial_regression("y ~ x", df, w, model="error") + self.assertIn("ML", err.method) + self.assertIn("understates the true lambda", " ".join(err.notes)) + @unittest.skipUnless(HAVE_PYSAL, "PySAL stack not installed") class TestMoranReplicateSimulation(unittest.TestCase): diff --git a/tests/test_validation.py b/tests/test_validation.py index 138d746..40f9613 100644 --- a/tests/test_validation.py +++ b/tests/test_validation.py @@ -1,7 +1,7 @@ # SPDX-License-Identifier: AGPL-3.0-or-later """Offline validation battery (PLAN_ROUND_7.md section 8) against frozen Census responses. -Every check F1-F10 runs on tests/fixtures/validation/ (captured from +Every check F1-F11 runs on tests/fixtures/validation/ (captured from api.census.gov and www2.census.gov, each with a .json sidecar: source URL without the API key, retrieval date, SHA-256). No network access. @@ -104,7 +104,7 @@ def test_f7_cp_significance(self): self.assertGreater(d["decidable_tests"], 400) def test_f8_f9_f10(self): - for cid in ("F8", "F9", "F10"): + for cid in ("F8", "F9", "F10", "F11"): self.assertPassed(cid) self.assertEqual(self.results["F8"].max_abs_error, 0.0) @@ -178,7 +178,7 @@ def test_parse_checks(self): self.assertEqual(V.parse_checks(None), list(V.CHECK_IDS)) self.assertEqual(V.parse_checks("f4, F1,F4"), ["F1", "F4"]) with self.assertRaises(ValueError): - V.parse_checks("F11") + V.parse_checks("F12") def test_z_rounding_range(self): # 63.7 vs 63.8 with MOEs 0.1: published values cannot decide significance. diff --git a/tests/test_validation_gate.py b/tests/test_validation_gate.py index 988aed4..3d13b8a 100644 --- a/tests/test_validation_gate.py +++ b/tests/test_validation_gate.py @@ -186,3 +186,53 @@ def test_parallel_matches_serial(self): a = pd.concat(serial, ignore_index=True).drop(columns="seconds") b = pd.concat(par, ignore_index=True).drop(columns="seconds") self.assertEqual(a.to_csv(index=False), b.to_csv(index=False)) + + +class TestRound15SpatialStudies(unittest.TestCase): + """PLAN_ROUND_15 B3 (D15-1): spatial regression / local Moran's I studies are info only.""" + + NAMES = ("spatial_regression_coverage", "local_morans_size") + + def test_registered_in_default_report_not_required(self): + from census.simulation import SPATIAL_REPS_FAST, SPATIAL_REPS_FULL, SPATIAL_STUDIES, STUDIES + for name in self.NAMES: + self.assertIn(name, STUDIES) # runnable with --studies + self.assertIn(name, SPATIAL_STUDIES) # and in the default full report (spatial group) + self.assertGreater(SPATIAL_REPS_FULL[name], 0) + self.assertGreater(SPATIAL_REPS_FAST[name], 0) + self.assertNotIn(name, iv.REQUIRED_STUDIES) + + def test_tiny_smoke_runs_are_info_only(self): + from census.simulation import FAST, study_local_morans_size, study_spatial_regression_coverage + reg = pd.DataFrame(study_spatial_regression_coverage(FAST, 2, rhos=(0.0, 0.3))) + lisa = pd.DataFrame(study_local_morans_size(FAST, 2)) + for df in (reg, lisa): + self.assertTrue(len(df)) + self.assertEqual(set(df["status"]), {"info"}) + self.assertTrue(df["coverage"].between(0, 1).all()) + # both estimators and models, both parameters; LM size rows at the null + self.assertEqual(set(reg["estimator"].dropna()), {"ml", "gm"}) + self.assertEqual(set(reg["model"].dropna()), {"lag", "error"}) + self.assertEqual(set(reg["term"].dropna()) >= {"x", "rho", "lambda", "lm_error", "lm_lag"}, True) + self.assertTrue(reg["target"].str.startswith("size of the 0.05 test").any()) + self.assertEqual(set(lisa["scenario"]), {"null_random_se", "null_clustered_se", "hot_spot"}) + self.assertTrue(lisa["power"].notna().any()) + # the tool-default rows are labelled: GMM lag, ML error; two-sided 9999-permutation LISA + dflt = reg[reg["tool_default"].fillna(False).astype(bool)] + self.assertEqual(set(zip(dflt["model"], dflt["estimator"])), {("lag", "gm"), ("error", "ml")}) + self.assertTrue(dflt["method"].str.contains(r"\[tool default\]").all()) + ld = lisa[lisa["reading"] == "default"] + self.assertTrue(ld["tool_default"].all() and (ld["permutations"] == 9999).all()) + self.assertTrue(ld["method"].str.contains(r"\[tool default\] two-sided").all()) + # info rows never fail the gate, and never satisfy a required study + both = pd.concat([reg, lisa]).assign(study="x") + self.assertTrue(iv.gate(both, ())[0]) + ok, problems = iv.gate(both, ("x",)) + self.assertFalse(ok) + self.assertIn("x: required study missing", problems[0]) + + def test_rows_depend_only_on_the_seed(self): + from census.simulation import FAST, study_local_morans_size + a = pd.DataFrame(study_local_morans_size(FAST, 2)) + b = pd.DataFrame(study_local_morans_size(FAST, 2)) + self.assertEqual(a.to_csv(index=False), b.to_csv(index=False)) diff --git a/tests/test_validation_live.py b/tests/test_validation_live.py index 97d18a3..42f21cf 100644 --- a/tests/test_validation_live.py +++ b/tests/test_validation_live.py @@ -28,7 +28,7 @@ def test_no_errors(self): def test_checks_pass(self): # F5 has a documented medians.py discrepancy (see tests/test_validation.py). - for cid in ("F1", "F2", "F3", "F4", "F6", "F7", "F8", "F9", "F10"): + for cid in ("F1", "F2", "F3", "F4", "F6", "F7", "F8", "F9", "F10", "F11"): with self.subTest(check=cid): r = self.results[cid] self.assertTrue(r.passed, f"{cid}: {r.notes} {r.failures[:3]}") diff --git a/tests/test_workflows_geo.py b/tests/test_workflows_geo.py index 426e164..a819053 100644 --- a/tests/test_workflows_geo.py +++ b/tests/test_workflows_geo.py @@ -52,9 +52,22 @@ def test_maps(self): def test_spatial(self): from census import workflows_geo as g - self.guided(g.spatial_autocorrelation(self.path, "rate", "tract", 2023, local=True, - output_path=os.path.join(self.dir, "lisa.csv"))) - self.guided(g.spatial_regression(self.path, "rate ~ x", "tract", 2023, model="error")) + res = g.spatial_autocorrelation(self.path, "rate", "tract", 2023, local=True, + output_path=os.path.join(self.dir, "lisa.csv")) + self.guided(res) + # PLAN_ROUND_15 B3: measured caveats reach the tool output + self.assertIn("local_morans_size", " ".join(res["local_notes"])) + res = g.spatial_regression(self.path, "rate ~ x", "tract", 2023, model="error") + self.guided(res) + self.assertIn("understates the true lambda", " ".join(res["notes"])) + res = g.spatial_regression(self.path, "rate ~ x", "tract", 2023) # lag default: GMM + self.guided(res) + self.assertIn("GMM", res["method"]) + self.assertIn("the default for the lag model", " ".join(res["notes"])) + res = g.spatial_regression(self.path, "rate ~ x", "tract", 2023, method="ml") + self.guided(res) + self.assertIn("ML lag intervals for rho covered 0.48", " ".join(res["notes"])) + self.assertFalse(g.spatial_regression(self.path, "rate ~ x", "tract", 2023, method="ols")["success"]) self.guided(g.spatial_small_area_estimates(self.path, "rate ~ x", "rate_se", "tract", 2023, output_path=os.path.join(self.dir, "sfh.csv"))) self.assertFalse(g.spatial_regression(self.path, "rate ~ nope", "tract", 2023)["success"])