Round 16: spatial methods validated under ACS sampling error (v0.15.0)

.github/workflows/ci.yml

@@ -5,8 +5,8 @@ on:
   push:
   pull_request:
   schedule:
-    # Weekly live validation against Census-published values (Mondays 06:00 UTC).
-    - cron: "0 6 * * 1"
+    # Weekly live validation against Census-published values (Mondays 06:17 UTC; off the hour, since GitHub may delay or drop top-of-hour schedules).
+    - cron: "17 6 * * 1"
   workflow_dispatch:
 
 permissions:

.gitignore

@@ -44,3 +44,6 @@ docs/audits/*.csv
 
 # Agent git worktrees (local only)
 .claude/worktrees/
+
+# Literature PDFs downloaded by hand (copyrighted; never committed)
+literature_pdfs/

.zenodo.json

@@ -1,5 +1,5 @@
 {
-  "title": "US Census ACS 5-Year Data Batch Downloader",
+  "title": "Census Research Engine",
   "description": "A Python toolkit and MCP server for extracting American Community Survey data from the US Census Bureau API into research-ready files, with vintage-safe variable concordance, margins of error for derived estimates, replicate-based variance, inference with sampling error, journal maps, and validation against Census-published values.",
   "upload_type": "software",
   "creators": [
@@ -8,6 +8,6 @@
   "license": "AGPL-3.0-or-later",
   "keywords": ["American Community Survey", "US Census Bureau", "margins of error", "survey statistics", "research data"],
   "related_identifiers": [
-    {"identifier": "https://github.com/cafeTechne/US-Census-5-Year-ACS-Data-Batch-Downloader", "relation": "isSupplementTo", "scheme": "url"}
+    {"identifier": "https://github.com/cafeTechne/Census-Research-Engine", "relation": "isSupplementTo", "scheme": "url"}
   ]
 }

CHANGELOG.md

@@ -5,6 +5,45 @@ each tagged release (`v<version>`) is archived on Zenodo with its own DOI (see
 [Releasing](#releasing)). Plans and acceptance criteria for each round are in
 `PLAN_ROUND_<n>.md`.
 
+## 0.15.0 (Round 16): spatial methods validated under ACS sampling error
+
+- Project renamed **Census Research Engine** (github.com/cafeTechne/Census-Research-Engine);
+  the distribution name `census-acs-toolkit` is unchanged.
+- **Lag-model effects:** `spatial_regression(model="lag")` reports direct, indirect and
+  total effects first (delta-method and replicate SEs; verified against the S(W) algebra
+  and spreg's multipliers), with notes that beta is not the marginal effect and that a
+  rho = 0 test is not a specification test. Coverage and a heavy-tail caveat at high rho
+  are measured (`docs/audits/spatial_regression_validation.md`).
+- **New option `spatial_regression(model="error", correct_outcome_error=True, se_col=...)`:**
+  corrects lambda for outcome sampling error using spatial Fay-Herriot (REML). Lambda
+  coverage is 0.86-0.93 against 0.003-0.85 uncorrected, measured on the shipped estimator.
+  - Assumes errors independent across areas. Warns when errors are correlated, and when
+    the outcome is a controlled total, where the sign of the bias depends on the release.
+  - Limited to 2,000 areas.
+- **ACS sampling-error covariance measured from VRE replicates** (Michigan and New York,
+  2015-2019 and 2020-2024; `docs/audits/acs_sampling_covariance.md`):
+  - neighbour error correlation is about 0 for rates, shares, medians and most counts;
+  - controlled totals are correlated within county, with a sign that can differ by
+    release.
+- **LISA validation:** under ACS error (`docs/audits/lisa_validation.md`), with no default
+  change.
+  - The Round 15 "38%" caveat is corrected: it counts maps with at least one significant
+    area, not contiguous false clusters.
+  - Replicate stability and Fay-Herriot reruns are shown not to be false-positive checks.
+- **Common-factor and specification studies:** LR size measured in Juhl's design; Koley &
+  Bera score tests shown not to identify SEM vs SDM. All new studies are info-only and
+  run on demand.
+- **Polite census.gov downloads:** one shared helper (`census/http.py`) with
+  - backoff and Retry-After on 429/5xx;
+  - per-host pacing;
+  - a clear error for the firewall "Request Rejected" page, with the support ID and the
+    cache path for a manual download;
+  - validation of manually placed files.
+- **Literature:** full-text reading notes with verification levels
+  (`docs/literature/full_text_reading_notes.md`). Remote-access planner decisions are
+  recorded in `docs/remote_access_decisions.md`.
+- **CI:** the weekly live-validation cron moved off the hour (Mondays 06:17 UTC).
+
 ## 0.14.0 (Round 15): studies across releases, spatial coverage, 2020 Census
 
 - Confirmatory studies across releases: CPI conversion to a declared `dollar_year`, then

CITATION.cff

@@ -1,6 +1,6 @@
 cff-version: 1.2.0
 message: "If you use this software in research, please cite it as below."
-title: "US Census ACS 5-Year Data Batch Downloader"
+title: "Census Research Engine"
 abstract: >-
   A Python toolkit and MCP server for extracting American Community Survey
   5-year data from the US Census Bureau API into research-ready files, with
@@ -11,11 +11,11 @@ type: software
 authors:
   - alias: cafeTechne
 license: AGPL-3.0-or-later
-repository-code: "https://github.com/cafeTechne/US-Census-5-Year-ACS-Data-Batch-Downloader"
-version: "0.14.0"
+repository-code: "https://github.com/cafeTechne/Census-Research-Engine"
+version: "0.15.0"
 doi: "10.5281/zenodo.23925301"
 # doi: Zenodo concept DOI, added after the first archived release (see CHANGELOG.md "Releasing").
-date-released: "2026-11-08"
+date-released: "2026-11-09"
 keywords:
   - American Community Survey
   - US Census Bureau

PLAN_ROUND_16.md

@@ -0,0 +1,173 @@
+# Round 16: Validating Spatial Methods Under ACS Sampling Error
+
+**Date:** 2026-11-08
+**Status:** Implemented (section 7); released as v0.15.0
+**Baseline:** v0.14.0 (spreg 1.9.1, esda 2.10.0, libpysal 4.15.1)
+**Companions:** [PLAN_ROUND_15.md](PLAN_ROUND_15.md) (spatial coverage studies), [docs/remote_access_decisions.md](docs/remote_access_decisions.md) (remote access follows this round)
+
+## 0. Principle and findings carried in
+
+**Validation first:** understand each failure, test competing explanations, and only then change an estimator. New estimators or substantial methods come in only when the experiments **and** accessible methodological sources support them. Otherwise findings are documented and an implementation round is proposed.
+
+| ID | Finding (Round 15) | Correct reading |
+| :--- | :--- | :--- |
+| F1 | Error-model λ coverage 0.16-0.58 under outcome sampling error | λ is spatial autocorrelation of the **disturbances**, not a substantive covariate spillover. The failure is **attenuation** of λ by measurement error in y (no-sampling-error reference covers 0.88-0.90). Earlier summaries calling λ a "spillover term" were imprecise |
+| F2 | Lag model reports β and ρ only | β is not the marginal effect in a lag model. There is no direct/indirect/total effects output, so users can misread β |
+| F3 | LISA false clusters in 38% of maps ("patterned" scenario) | That scenario has **spatially heterogeneous but independent** sampling errors (CVs sorted along a gradient). It is **not** evidence about spatially **correlated** sampling errors, which are untested |
+
+**Two different false-cluster situations**, kept distinct throughout:
+
+| Scenario | Definition |
+| :--- | :--- |
+| Heterogeneous variances | Sampling variances vary across space, errors independent |
+| Correlated errors | Sampling errors covary between areas (survey design, weighting controls) |
+
+A method that handles one may fail the other. FDR control does not repair p-values computed under a wrong null.
+
+## 1. Workstream A: spatial regression (P0)
+
+**A1. λ attenuation (error model).**
+- Establish, from the original source, the assumptions under which a measurement-error correction for λ is valid (candidate: Lorincz-Comi et al. 2021, *Spatial Statistics* 45:100523; others if accessible). In particular, does it need only the error variances, or their covariance?
+- Compare uncorrected and corrected estimators under known DGPs: bias, RMSE, interval coverage, numerical stability (non-convergence, boundary estimates).
+- Sensitivity to the weights matrix (rook, queen, kNN) and to measurement-error magnitude (CV profiles).
+- Run under independent and under spatially correlated errors separately. A correction validated under independence is not assumed valid under correlation.
+- Implement the correction in the tool only if it validates and its assumptions are verified from an accessible source; otherwise document and propose.
+
+**A2. Effects for the lag model (approved).**
+- First inspect spreg 1.9.1 for established impacts functionality (spatial multiplier, direct/indirect/total effects, inference) and use it if available.
+- Verify the reported effects against independently computed impacts from S(W) = (I - ρW)^-1 β: the direct effect is the mean of the diagonal, the total effect is the mean row sum, and indirect = total - direct.
+- Report the effects, with uncertainty, in the tool output and results. Make it impossible to mistake β for the marginal effect: the label and notes say β is not the marginal effect, and the effects table comes first.
+- Measure effects coverage in the simulation.
+
+**A3. DGP set (spatial Durbin model approved as a DGP only).**
+- Simulate:
+  - SEM (no substantive spillovers);
+  - SAR lag with known effects;
+  - SDM, y = ρWy + Xβ + WXθ + ε, with known effects;
+  - an omitted spatially correlated covariate.
+- Fit the tool's models to each. Report parameter bias and coverage, and recovery of direct/indirect/total effects where they are defined.
+- Separate the recovery of disturbance parameters from the recovery of substantive effects.
+- **No production SDM estimator in this round.**
+
+**A4. Common-factor test (SEM as SDM with θ = -ρβ).**
+- Evaluate a likelihood-ratio test only where both models can be fitted reliably. Confirm:
+  - nesting and the restriction;
+  - comparable likelihoods (same data, same Jacobian treatment);
+  - finite-sample size and power.
+- Asymptotic Wald tests are known to be parameterization-dependent (Political Analysis 2020), so use LR, with simulation or bootstrap critical values if the LR size is off.
+- Add the test to the tool only if it earns its place; no checklist additions.
+
+## 2. Workstream B: LISA under sampling error (P0)
+
+**B1. Five scenarios** (approved): no true clustering with
+1. independent homogeneous errors;
+2. heterogeneous independent errors;
+3. spatially correlated errors;
+
+plus true clusters with
+4. independent errors;
+5. correlated errors.
+
+Correlation structures in scenarios 3 and 5 are parameterized, with ranges informed by C1 and stated as assumptions.
+
+**B2. Four candidate methods**, all on the same simulated maps:
+1. conventional LISA (current default);
+2. `replicate_quadrant_share` (a stability diagnostic, not a calibrated probability);
+3. LISA on spatial Fay-Herriot EBLUPs. Explicitly test whether the smoothing **induces or alters** spatial patterns (false clusters created by smoothing; true clusters attenuated);
+4. a Wong (2021) threshold-first, uncertainty-aware approach, implemented only as far as the open-access paper specifies it.
+
+**B3. Metrics, defined in the plan and the report:**
+- **false-positive rate:** per-area share of null areas flagged;
+- **FDR:** the expected share of flagged areas that are null;
+- **false-cluster rate:** the share of maps with at least one flagged contiguous cluster, of size ≥ k, made only of null areas; k is stated;
+- **power:** the share of true-cluster areas flagged;
+- **cluster-location accuracy:** for example the Jaccard overlap of the flagged cluster with the true cluster;
+- **sensitivity to W.**
+
+Every metric carries a Monte Carlo SE.
+
+Decompose errors attributable to sampling uncertainty from errors attributable to misspecified spatial dependence (for example by comparing the same maps with and without sampling error, and under the correct vs a misspecified W).
+
+A preferred method is never chosen by minimizing false positives alone: power and smoothing bias are weighed.
+
+**B4. Three questions kept separate:**
+- stability of a classification across replicate maps;
+- statistical significance;
+- evidence of a genuine underlying pattern.
+
+## 3. Workstream C: ACS sampling-error covariance from replicates (P0, central question)
+
+**C1. Measure it, and measure how reliably it can be measured.**
+- Use the ACS VRE replicates for Michigan and New York tracts (and counties), for several variables of different types (a count, a proportion, a median if replicated).
+- Estimate the sampling covariance between estimates with the ACS replicate formula, Cov = (4/80) Σ_r (X_r - X)(Y_r - Y), consistent with the variance scaling.
+- Analyse:
+  - neighbouring-pair correlations against non-neighbour pairs, by distance or contiguity order, and inside vs across county lines (weighting controls);
+  - structured summaries instead of unrestricted matrices, since 80 replicates give a rank-limited covariance across hundreds of tracts: mean or median neighbour correlation, a fitted structured model (for example one correlation parameter per contiguity order), shrinkage or regularization where justified;
+  - the stability of those summaries: bootstrap or jackknife over replicates, and split-half agreement;
+  - the noise floor: what pairwise correlation magnitude 80 replicates can distinguish from zero.
+- Pairwise correlations are never treated as established individually.
+
+**C2. Report** whether the covariance structure is estimable well enough to use in a correction (A1) or a null model (B). If so, that is a methodological contribution to the engine; if not, the limit is documented with evidence. Results feed the B1 correlation parameters as stated assumptions.
+
+## 4. Workstream D: literature notes (P1)
+
+`docs/literature/spatial_methods.md` lists the references the owner's planner provided. Each entry records:
+- full citation and DOI/URL;
+- **verification level:**
+  - (1) verified from full text;
+  - (2) supported by abstract or accessible excerpt only;
+  - (3) a hypothesis or implementation suggestion still to verify;
+- what it is used for, and which experiment tests it.
+
+The planner's summaries are not substitutes for the sources. Where access is restricted, document it, and don't implement methods whose essential assumptions cannot be verified.
+
+References:
+- Lacombe & LeSage 2015
+- Koley & Bera 2024
+- Political Analysis 2020 (common-factor Wald test)
+- IJGIS 2018 (ACS spatial autocorrelation critique)
+- Wong 2021
+- SBC 2019
+- Bayesian areal ACS 2021
+- Sun et al. 2015
+- Arthur 2024
+- LeSage & Pace 2014
+- Lorincz-Comi et al. 2021
+- Rogerson 2024
+
+## 5. Scope
+
+- No unrelated architectural changes; preserve the existing safeguards, reproducibility contracts and working functionality.
+- Studies are `info`. Gating is an owner decision after the evidence is in.
+- Out of scope this round:
+  - hierarchical Bayesian measurement-error models (a later research extension);
+  - Arthur-style constrained resampling and spatial FDR procedures (Sun et al.), unless B shows the basic null-model problem is solved first;
+  - a production SDM estimator.
+- Remote laptop access follows this round.
+
+## 6. Release
+
+v0.15.0 with a Zenodo DOI, after code review, fixes, the full gate, and a test run on both Python 3.10 and 3.14 before the push.
+
+---
+
+## 7. Implementation status (2026-11-09)
+
+| Workstream | Result |
+| :--- | :--- |
+| A spatial regression | **λ attenuation is real and correctable:** spatial Fay-Herriot recovers λ (coverage 0.86-0.93, uncorrected 0.003-0.85) under independent errors, and fails under positively correlated errors. **Lag effects** were added to the tool. **SDM data** break default GMM lag (documented). **LR common-factor test:** size fine for n ≥ 100, power weak, so not added. **Koley & Bera tests** don't identify SEM vs SDM |
+| B LISA | No default change. "38%" corrected. Fay-Herriot smoothing induces clusters only under positively correlated errors. Stability ≠ evidence. Wong 2021 targets a different estimand |
+| C covariance | Rates/shares/medians: neighbour error r ≈ 0 (±0.003 tracts). Controlled totals are correlated within county, with a release-dependent sign (2020-2024 negative; 2015-2019 households positive). The single 2019 Michigan population file is rejected by the census.gov firewall (URL-specific, not rate limiting) |
+| D literature | 12 sources read in full (owner-supplied PDFs for the blocked ones); verification levels recorded |
+| Owner decisions | Include `correct_outcome_error` in v0.15.0 (approved 2026-11-09) |
+| Review | `/code-review high`: 10 findings fixed. The shipped λ estimator is now the one the studies run, with coverage unchanged within 0.007 |
+
+**Proposals for a later round:**
+- the spatial Durbin model via spreg `slx_lags`, plus the common-factor test (LR/bootstrap);
+- simulated percentile intervals for lag effects at high ρ;
+- adaptive (Storey) BH and cluster-level FDR for LISA, scored on the existing runs;
+- a corrected global Moran's I (Lorincz-Comi Eq. 4) using the replicate covariance, after its inference is verified;
+- posterior exceedance probabilities (Sun et al. 2015);
+- the LISA weights guidance (P-B5).
+
+**Next:** minimum SSH remote access (docs/remote_access_decisions.md).