Round 15: studies across releases, spatial coverage, 2020 Census (v0.14.0)

CHANGELOG.md

@@ -5,6 +5,32 @@ each tagged release (`v<version>`) is archived on Zenodo with its own DOI (see
 [Releasing](#releasing)). Plans and acceptance criteria for each round are in
 `PLAN_ROUND_<n>.md`.
 
+## 0.14.0 (Round 15): studies across releases, spatial coverage, 2020 Census
+
+- Confirmatory studies across releases: CPI conversion to a declared `dollar_year`, then
+  each release's own RPP (`rpp_years`); `"replicates": {"source": "census_vre"}` fetches
+  Census VRE tables (URLs and SHA-256 recorded). Overlapping releases are refused for any
+  replicate source; `replicate_scope: "covariates_only"` is an explicit, hashed exception
+  for outcomes without replicates. Example: MI/NY change in CPI- and RPP-adjusted mean
+  household income, 2015-2019 to 2020-2024: +$3,111 per county, 90% CI [2,709, 3,513].
+- Spatial coverage studies (info): `spatial_regression_coverage`, `local_morans_size`.
+  **Default changes (owner decisions):** the spatial lag model now defaults to GMM (ML rho
+  coverage under ACS sampling error was 0.48 at rho 0.3 and 0.003 at 0.6; GMM 0.89-0.90);
+  local Moran's I now reads p-values two-sided with 9999 permutations (per-area false
+  positive rate 0.20 to 0.10). Previous behaviour: `method="ml"`;
+  `sided="folded", permutations=999`. Caveats with measured numbers in both tools'
+  outputs (error-model lambda attenuation; false clusters under spatially patterned
+  sampling error).
+- spreg's per-fit stdout print is suppressed at its source, so it can no longer corrupt
+  the MCP stdio stream.
+- 2020 Decennial Census: P.L. 94-171 (`dec/pl`) and DHC (`dec/dhc`) through the existing
+  pipeline. Counts are "not sample-based (MOE undefined)", never MOE 0, with a
+  disclosure-avoidance advisory; sampling tests refuse decennial-only input; ACS vs
+  census comparisons treat the count as a fixed reference. New live check F11
+  (invariants and block-to-tract/county hierarchy, 5,492 values); 11/11 checks pass live.
+- Remote access decisions recorded (`docs/remote_access_decisions.md`); no networking
+  changes in this release.
+
 ## 0.13.0 (Round 14): confirmatory multi-state analyses
 
 - Pre-registered manifests take several states (`geography.states`, 2-digit FIPS; not

CITATION.cff

@@ -12,7 +12,7 @@ authors:
   - alias: cafeTechne
 license: AGPL-3.0-or-later
 repository-code: "https://github.com/cafeTechne/US-Census-5-Year-ACS-Data-Batch-Downloader"
-version: "0.13.0"
+version: "0.14.0"
 doi: "10.5281/zenodo.23925301"
 # doi: Zenodo concept DOI, added after the first archived release (see CHANGELOG.md "Releasing").
 date-released: "2026-11-08"

PLAN_ROUND_15.md

@@ -0,0 +1,89 @@
+# Round 15: Studies Across Releases, Spatial Coverage, 2020 Census
+
+**Date:** 2026-11-08
+**Status:** Implemented (section 6); released as v0.14.0
+**Baseline:** v0.13.0
+**Companions:** [PLAN_ROUND_14.md](PLAN_ROUND_14.md), [docs/remote_access_decisions.md](docs/remote_access_decisions.md)
+
+## 0. Front-loaded findings and boundaries
+
+| ID | Source | Finding | Disposition |
+| :--- | :--- | :--- | :--- |
+| G1 | Round 14 deferrals | A manifest's RPP adjustment covers one release only, and `run_manifest` cannot fetch Census VRE replicates itself | Workstream A |
+| G2 | `docs/audits/CHECK_MAP.md` | `spatial_regression` (lag/error) and `local_morans` have no coverage study; global Moran's I has only an info study | Workstream B |
+| G3 | Owner gap analysis | ACS only. 2020 Decennial Census access exists only internally, as block counts for crosswalks (`census/block_weights.py`) | Workstream C |
+| G4 | Scheduled CI | First weekly live run was due 2026-11-09 | Check it and triage as in Round 13 A1 |
+
+**Boundary (planner):** no networking changes in this round, and no HTTP, persistent service, job queue, custom transfer API or Data Commons adapter. Remote access is the next round (SSH, per `docs/remote_access_decisions.md`).
+
+As before: no tuning of tolerances, bands or thresholds to make something pass.
+
+## 1. Workstream A: confirmatory studies across releases (P0)
+
+- **A1. Several releases with price adjustment.** A manifest with several releases (for example non-overlapping 5-year periods, 2015-2019 vs 2020-2024, or 1-year releases) can declare RPP adjustment for each release's own year. Dollars are converted to one declared dollar year with the existing CPI machinery (R-CPI-U-RS default), *then* place-adjusted. The order is stated in the methods text.
+  - The declared `dollar_year` and per-release `rpp_year` are required, so the hash fixes them; `create_manifest` fills them as in Round 14.
+  - Existing overlap rules for comparisons of overlapping 5-year periods stay in force.
+- **A2. Automatic replicate retrieval.** `"replicates": {"source": "census_vre"}` in a manifest makes `run_manifest` fetch the VRE tables for the manifest's variables, geography and releases, using the existing `census.replicate` code and its cache.
+  - The VRE file hashes and URLs go into the run record.
+  - Variables with no published replicates (for example B19013) are reported. Under the strict confirmatory policy they block, as today, unless the manifest says which analyses use only covariate replicates. The rule is explicit, never silent.
+- **A3. Example study and tests.**
+  - An example manifest for a change study in Michigan and New York counties across two non-overlapping 5-year periods: change in RPP- and CPI-adjusted median household income, or another pre-registered contrast the existing methods support.
+  - Frozen-fixture offline test, plus a live test gated by `CENSUS_LIVE_TESTS=1`.
+  - Docs: the preregistration recipe and a short section in the case study.
+
+## 2. Workstream B: coverage for spatial regression and local Moran's I (P1)
+
+- **B1. `spatial_regression_coverage`.** A spatial lag and an error model on a lattice with ACS-like sampling error, for each estimator the tool offers (GMM/ML). Report coverage of beta, and of rho/lambda, for the tool's default intervals. Also report bias and the LM diagnostics' size under the null.
+- **B2. `local_morans_size`.** Local Moran's I under spatial independence plus sampling noise. Report:
+  - per-area false-positive rate and family-wise or FDR control, for the tool's default inference and correction;
+  - power under one clustered alternative.
+- **B3. Status.** Both run first as `info` studies.
+  - A study whose default intervals pass the band can be promoted to gated by owner decision (D15-1).
+  - If a default fails, the tool's outputs get a visible caveat, or the default changes (for example to a corrected method), with evidence. Never tune the band.
+- Update `docs/audits/CHECK_MAP.md`. Keep each study's runtime reasonable under `--jobs`.
+
+## 3. Workstream C: 2020 Decennial Census (P1, provisional)
+
+- **C1. Retrieval.** P.L. 94-171 (`dec/pl`) and the Demographic and Housing Characteristics file (`dec/dhc`) through the existing API client, cache and output formats, with variable metadata and codebook. Reuse the block-count code where possible.
+- **C2. Correct uncertainty semantics.** Decennial values are **published counts that are not sample-based but not error-free**:
+  - no sampling MOE;
+  - coverage error, processing error, and disclosure avoidance through the TopDown differential-privacy algorithm.
+
+  Concretely:
+  - Outputs never carry a zero MOE or a CV of 0 presented as precision. The uncertainty fields are explicitly "not sample-based (MOE undefined)", with a disclosure-avoidance advisory.
+  - The advisory notes that small geographies and detailed cells are noisier, that only certain counts are invariant (state total population, block-level housing units and occupied group-quarters facility counts), and that block-level characteristics should be aggregated before use.
+  - Significance tools refuse decennial-only "sampling" tests, or say exactly what they test.
+  - Mixing ACS and decennial values in one comparison is allowed only with a stated note.
+- **C3. Validation.** A new battery check against the Census Bureau's own invariants and hierarchy:
+  - state total population matches the published state totals;
+  - block-level housing-unit counts sum exactly to tract and county totals;
+  - tract and county totals from DHC agree with P.L. 94-171 where both publish the same item.
+
+  Record live, freeze fixtures, keep fixtures small (Michigan, New York, Wyoming).
+- **C4. Scope.** Retrieval, semantics and validation only. No new estimators. DHC tables used in the examples: P1, P12, H1, H3. The 2010 Census is out of scope unless needed.
+
+## 4. Release
+
+v0.14.0 with a Zenodo DOI: code review (`/code-review high`), fixes, full gate (`--jobs 0`), PR, CI, live dispatch, merge, tag, Zenodo confirmation.
+
+## 5. Defaults taken (owner may override)
+
+| ID | Default |
+| :--- | :--- |
+| D15-1 | New spatial studies start as `info`. Promotion to gated is an owner decision after the evidence is in |
+| D15-2 | Multi-release dollar conversion: CPI to the declared dollar year first, then the RPP of each release's own year |
+| D15-3 | Decennial geography levels: nation, state, county, tract, block group, block (block only for P.L. 94-171 and invariant counts), place |
+
+---
+
+## 6. Implementation status (2026-11-08)
+
+| Slice | Result |
+| :--- | :--- |
+| A releases | Multi-release adjustments (CPI to dollar year, then per-release RPP) and `census_vre` replicate retrieval. Live MI/NY change study: +3,110.80 (90% CI [2,708.93, 3,512.67]), no deviations; hand recomputation from raw VRE files agrees to about 1e-12 |
+| B spatial coverage | Info studies added to the full report. Lag ML rho coverage collapses under outcome sampling error (0.48, 0.003); GMM 0.89-0.90. LISA folded p-values doubled the false-positive rate. **Owner decisions:** lag default GMM; LISA two-sided, 9999 permutations. Error-model lambda attenuation and patterned-noise false clusters remain caveats |
+| C 2020 Census | `dec/pl`, `dec/dhc` through the existing pipeline; MOE undefined, never 0; disclosure-avoidance advisory; sampling tests refuse; F11 passes live and offline |
+| Review | `/code-review high`: 10 findings fixed with tests (`tests/test_review_fixes_r15.py`), including overlap refusal for every replicate source and explicit-evidence decennial detection |
+| Gate | Full `--gate --jobs 0`: PASS, 19 min; existing coverage rows unchanged; new spatial rows are info |
+
+**Deferred:** 1-year releases and decennial data in manifests; aggregation with adjustment or `census_vre`; `covariates_only` beyond `fit_replicate`; a fix for error-model lambda attenuation and for patterned sampling error in LISA (no tested method yet). **Next:** minimum SSH remote access per `docs/remote_access_decisions.md`.

README.md

@@ -134,6 +134,7 @@ Each multi-year run writes `<table>_concordance.csv` (every output column traced
 | `place` | 160 | `SSPPPPP` | place FIPS (required) |
 | `zcta` | 860 | `ZZZZZ` | ZCTAs; default all ZIPs in the state (`zip2fips.json`); requested in chunks of 50 |
 | `msa` | 310 | `MMMMM` | CBSA codes; default all metro and micro areas |
+| `block` | 100 | `SSCCCTTTTTTBBBB` | 2020 Census only (`dec/pl`, or DHC `H1`): county FIPS; one request per county, default one per state |
 
 ### Datasets
 
@@ -143,8 +144,10 @@ Each multi-year run writes `<table>_concordance.csv` (every output column traced
 | `profile` | `/acs/acs5/profile` | `DP02`–`DP05` | adds `PE`/`PM` percentages |
 | `subject` | `/acs/acs5/subject` | `S…` | `E`, `M` per column (`_C01_`, …) |
 | `cprofile` | `/acs/acs5/cprofile` | `CP02`–`CP05` | codes embed the period; `SS` significance flags |
+| `dec/pl` | `/2020/dec/pl` | 2020 Census P.L. 94-171 redistricting data: `P1`–`P5`, `H1` | `N` (count), `NA` annotation; no MOE |
+| `dec/dhc` | `/2020/dec/dhc` | 2020 Census Demographic and Housing Characteristics: e.g. `P1`, `P12`, `H1`, `H3` | `N` (count), `NA` annotation; no MOE |
 
-The dataset is inferred from the table code.
+The ACS dataset is inferred from the table code. The 2020 Census datasets are selected with `dataset="dec/pl"` / `"dec/dhc"` or `product="decennial"` (DHC); see [2020 Decennial Census](#2020-decennial-census).
 
 ### Outputs
 
@@ -167,6 +170,7 @@ Every run also writes `<table>_extraction_metadata.json`. It records the request
 | `<base>__<first>_<last>` | Concept segment when a code was reused for different measures |
 | `<base>_SE`, `_CV`, `_RELIABILITY` | Derived standard error, CV (%), and reliability flag |
 | `*_est`, `*_moe`, `*_pct`, `*_pct_moe`, `*_se`, `*_cv`, `*_rel`, `*_sig` | Panel slugs; `slug_style="code"` gives Stata-safe names |
+| `<base>N`, `*_count` | 2020 Census count (and its panel slug); no MOE, SE or CV columns exist |
 
 ## Methodology
 
@@ -204,6 +208,7 @@ Every derived result records its `method` (for example `proportion`, `proportion
 | F6 | Prior-period dollars adjusted with R-CPI-U-RS | Comparison Profile values in current dollars |
 | F7 | Z-tests between periods | Comparison Profile significance flags |
 | F8 | Sums of counties | State estimates |
+| F11 | 2020 Census state population; block housing units summed to tracts and counties; DHC vs P.L. 94-171 | Apportionment-release state totals; published tract and county totals (invariants and hierarchy) |
 
 ## Inference
 
@@ -232,8 +237,8 @@ Built on GeoPandas, Shapely (GEOS), pyogrio (GDAL), pyproj, and PySAL. Outputs a
 | :--- | :--- | :--- |
 | Census boundaries matched to each ACS release | `census.boundaries.get_boundaries` / `get_boundaries` | Cartographic (1:500k) for maps, TIGER/Line for areal work; vintage mismatches are errors |
 | Journal maps that show uncertainty | `census.maps.choropleth` / `make_map` | Quantile classes by default; legend reports class separability (Sun and Wong 2010); low-reliability areas hatched; uncertainty and significance maps |
-| Spatial autocorrelation | `census.spatial.morans_i`, `local_morans` / `spatial_autocorrelation` | Reproduces PySAL references; replicate SE of Moran's I is conservative (coverage 0.987) |
-| Spatial regression | `census.spatial.spatial_regression` | Lag and error models with LM diagnostics (spreg) |
+| Spatial autocorrelation | `census.spatial.morans_i`, `local_morans` / `spatial_autocorrelation` | Reproduces PySAL references; replicate SE of Moran's I is conservative (coverage 0.987); local clusters use two-sided permutation p-values (9999 permutations, BH): false-positive rates at nominal under randomness |
+| Spatial regression | `census.spatial.spatial_regression` | Lag (GMM default) and error (ML default) models with LM diagnostics (spreg); simulated coverage under ACS outcome error: rho 0.89-0.90 (GMM lag), lambda attenuated |
 | Spatial Fay–Herriot | `census.sae_spatial.spatial_fay_herriot` / `spatial_small_area_estimates` | Reproduces the R `sae` grapes example; coverage 0.891 |
 | Population-weighted tract crosswalks | `census.block_weights`, `harmonize_panel(weight="auto")` | Decennial block weights; Delaware test: mean error 444 vs 624 people with land-area weights |
 | Block groups and custom polygons | `geo_level="block group"`, `census.custom_areas` / `custom_area_mapping` | Area or block-population allocation |
@@ -251,12 +256,32 @@ Built on GeoPandas, Shapely (GEOS), pyogrio (GDAL), pyproj, and PySAL. Outputs a
 | Cost of living between places | `census.rpp.adjust_for_place` / `adjust_for_place` | BEA Regional Price Parities; reproduces BEA real personal income for all states (2008); never applied by default |
 | Interactive maps | `census.web_maps.web_choropleth` / `make_web_map` | Self-contained HTML, no external requests |
 
+## 2020 Decennial Census
+
+The 2020 Census P.L. 94-171 redistricting data (`dec/pl`) and the Demographic and Housing Characteristics File (`dec/dhc`) use the same client, cache, extraction pipeline, output formats, codebook and replay as the ACS: pass `dataset="dec/dhc"` (or `"dec/pl"`), or `product="decennial"`, to `plan_extraction`, `batch_download`, `download_table`, `run_batch` or `python -m census extract --product decennial`, with `years=[2020]`. Geography levels are nation, state, county, tract, block group, place and block (block only for P.L. 94-171 and the invariant DHC `H1` housing-unit count). Variable metadata for both files is archived in `metadata/dec/`.
+
+```python
+from census import ResearchOptions, build_entities, run_batch
+
+run_batch("P12", build_entities("county", "26"), [2020], "out", "panel",
+          ResearchOptions(include_codebook=True), dataset="dec/dhc")
+```
+
+**Uncertainty.** Census counts are not sample-based, so they have no sampling margin of error; they are also not error-free. Outputs never carry an MOE or CV of 0: no MOE, SE, CV or reliability columns are written (`compute_reliability` adds none), the codebook's `Uncertainty` column reads "not sample-based (MOE undefined)", and the extraction metadata has `moe_confidence_level: null` and an `uncertainty` record. Every decennial plan carries a disclosure-avoidance advisory:
+
+- the counts carry coverage error (measured by the [Post-Enumeration Survey](https://www.census.gov/newsroom/blogs/random-samplings/2021/12/post-enumeration-measuring-coverage-error.html)), processing error, and noise added by the TopDown differential-privacy algorithm;
+- only three counts are invariant (published without noise): state total population, total housing units per block, and the number of occupied group-quarters facilities by type per block ([Census DAS update, 2020-11-25](https://www.census.gov/programs-surveys/decennial-census/decade/2020/planning-management/process/disclosure-avoidance/2020-das-updates/2020-11-25.html));
+- small geographies and detailed cells are noisier, and block-level characteristics should be aggregated before use ([fact sheet](https://www.census.gov/content/dam/Census/library/factsheets/2021/protecting-the-confidentiality-of-the-2020-census-redistricting-data.pdf)).
+
+**Statistics.** `z_test`, `calculate_significance`, `batch_significance`, `calculate_cv` and `acs_overlap_years` refuse decennial-only inputs (`product="decennial"`), since a sampling test says nothing about disclosure-avoidance noise. An ACS estimate can be tested against a census count with `product1="5-year", product2="decennial"`: the count is a fixed reference, and the result's `comparison` and `notes` say that only ACS sampling error is measured. `derive_estimate(..., product="decennial")` returns the point estimate with MOE, SE and CV undefined. `build_analysis_frame` gives census counts NaN `_se`/`_cv`, method `not_sample_based`, release `dec_2020` (`acs5_<year>+dec_2020` with a note when mixed with ACS variables). No new estimators are provided for decennial data; validation check F11 compares the files with the Census Bureau's invariants and hierarchy.
+
 ## Advisories and known limitations
 
 - **Boundary vintages.** Requests spanning 2019 to 2020 at tract, ZCTA, place, county, or CBSA level produce an advisory. Tract panels can be harmonized with `harmonize_tracts_to=2010|2020` (land-area weights from the Census relationship file, or NHGIS-style population weights you supply); apportioned MOEs assume independent components and exact weights, and boundary slivers produce tiny weights. Other levels are not harmonized.
 - **Medians from coarse brackets.** Household-income medians rebuilt from B19001 are usually within 1% of the published B19013, but can differ by more than 2% because Census interpolates a finer internal distribution.
 - **Home values.** Before the 2022 releases, home values were not on the period end-year dollar basis (ACS user note 2023-07), so deflating them is approximate. Plans include an advisory.
 - **Subject tables.** Their 2013 and 2017 redesigns changed thousands of labels. Multi-year subject pulls usually need `variables=` and reviewed overrides.
+- **2020 Census counts.** No margins of error exist; disclosure-avoidance noise dominates blocks and detailed cells. Aggregate before use; see [2020 Decennial Census](#2020-decennial-census). Cross-release concordance and continuity checks do not apply (one census year), and the 2010 Census is not offered.
 - **ZIP codes vs. ZCTAs.** `zip2fips.json` maps each ZIP to one primary county; PO-box-only ZIPs have no ZCTA.
 
 ## Repository layout