Repository navigation
Round 16: spatial methods validated under ACS sampling error (v0.15.0) - #13
Merged
Merged
Conversation
…ound 16 B1-B3) census.lisa_validation: five scenarios on a 12x12 lattice (no true clustering with independent homogeneous, heterogeneous independent and SAR-correlated errors at mean neighbour correlation -0.2..0.5; a hot spot with independent and correlated errors), 80 replicate maps per map with the scenario's error covariance and ACS 4/80 scaling, and four methods on the same maps: tool-default local_morans, a replicate_quadrant_share stability filter, LISA on spatial Fay-Herriot EBLUPs, and the Wong (2021) heuristic hot/cold-spot procedure scored against its own target. Map-level metrics with between-map MC SEs; W sensitivity (rook, queen, kNN, misspecified). Registered as the on-demand 'lisa' group (--studies only; info rows; not required). No tool behavior changes. Also make a Round 15 test robust to pandas' object-dtype fillna FutureWarning on Python 3.10.
…-Research-Engine) Repository URL in citation, Zenodo, and package metadata; titles; HTTP user agents; personal Windows paths in setup docs replaced with placeholders. Distribution name census-acs-toolkit is unchanged.
…E replicates Structured summaries (class means with replicate-bootstrap and delete-one-group jackknife SEs), analytic and derangement noise floors, split-half reliability, aggregation variance ratio, Ledoit-Wolf intensity and held-out comparison of structured covariance models (PLAN_ROUND_16 C1). Tests on synthetic replicates with known covariance.
…ley & Bera (SSRN preprint)
… 16 A2) spatial_regression(model='lag') now returns ModelResult.effects: average direct, indirect and total effects (LeSage & Pace 2009) from the eigenvalues of W, with delta-method SEs from the fit's (beta, rho) covariance and, with replicates, replicate-refit SEs. Effects come first in to_dict(), the summary and notes say beta is not the marginal effect, and the coefficient table carries an interpretation column. report_tables.effects_table renders them; regression_table notes lag models. The rho/lambda p-value is labelled as a test of the parameter = 0, not a specification test (Juhl 2021 sec. 3.2). spreg 1.9.1 provides point impacts only (sp_multipliers), no inference.
…nfo only) census.simulation_spreg: lambda attenuation and the spatial Fay-Herriot correction under independent and correlated sampling errors (A1), DGP recovery incl. SDM and Juhl's omitted-variable DGP with effects coverage (A2/A3), and the common-factor LR test in Juhl's design with residual bootstrap, plus an ACS-error extension (A4). Registered in group 'spreg_validation'; run only when named with --studies.
…cterization (Round 16 B) docs/audits/lisa_validation.md: designs, metric definitions, results with MC SEs (400 maps per study), the smoothing-induced-pattern finding, what Wong (2021) specifies and what was implemented, conclusions and proposals. LOCAL_MORAN_CAVEAT: the Round 15 38% is the share of maps with at least one significant area under heterogeneous but independent errors, not spurious clusters, and says nothing about correlated errors; drop the unvalidated advice to check replicate_quadrant_share or rerun on Fay-Herriot estimates (neither is a false-positive check in this study). CHECK_MAP lists the on-demand studies.
…ial_dgp_recovery Lag fits expose diagnostics['coefficient_vcov']; the A3 study adds LeSage & Pace simulated percentile intervals for the effects next to the tool's delta-method intervals, and rejection rates of spreg's OLS-based RS tests (Koley & Bera 2024), none of which tests SEM vs SDM.
…ssel 2019, Rogerson 2025, Lacombe-LeSage 2015
…s caveat docs/audits/spatial_regression_validation.md: designs, results and proposals for A1-A4 (spatial Fay-Herriot validates as the lambda correction under independent outcome error and fails under correlated error; lag effects coverage; DGP recovery; LR common-factor size and power). The lag model notes carry the measured effects caveat. CHECK_MAP lists the new studies. Study code: record SEM/SDM estimates across gamma in common_factor_lr; silence the singular-information warning at the A = 0 boundary. Test fix: test_validation_gate no longer triggers pandas 2.3's fillna downcast FutureWarning (failed on Python 3.10 before this branch).
# Conflicts: # census/simulation.py # docs/audits/CHECK_MAP.md # scripts/inference_validation.py
…ak-free held-out comparison replicate_spectrum finds replicate-space directions along which every area's deviations nearly vanish (3 in 2020-2024 MI/NY tracts: about 77 effective replicates). holdout_model_comparison projects each half orthogonally to them, removing a cross-half dependence that made Ledoit-Wolf appear to beat independence.
…iance (PLAN_ROUND_16 C1-C2) Michigan and New York tracts and counties, 2020-2024 (and 2015-2019 counties): pairwise replicate correlations summarised by contiguity order, county, rook/queen and distance, with replicate-bootstrap and county-jackknife SEs, analytic/derangement/cross-state noise floors, split-half reliability, aggregation ratios and held-out structured models. Writes docs/audits/acs_sampling_covariance.md and .csv.
…ts and counties (PLAN_ROUND_16 C1-C2)
…el (PLAN_ROUND_16 A1) spatial_regression(model='error', correct_outcome_error=True, se_col=...) fits the error model with the outcome's known independent sampling variances, V = sigma^2 [(I - lambda W)'(I - lambda W)]^-1 + diag(SE^2), through the spatial Fay-Herriot machinery (census.sae_spatial; REML default, ML via method='ml'). Reports the corrected lambda with its information-based Wald interval and GLS beta intervals (the A1-validated methods), the uncorrected lambda for comparison, convergence/boundary flags, and notes stating the independence assumption with the A1 and C evidence, the 'diffusion of shocks' reading of lambda, the correlated-error caveat, and a warning for controlled totals (population/household/housing-unit counts, or controlled_total=True). Non-convergence raises RuntimeError (no fallback). Default behaviour unchanged; ERROR_CAVEAT points to the option. Exposed through workflows_geo.spatial_regression (MCP/CLI). Tests: the tool path equals the A1 study estimator (me_sem_fit) and the study cell's own rows on its seeded draws (REML and ML), a 20-draw coverage smoke check, option handling, warnings, boundary and non-convergence.
… protocol CHECK_MAP: new row for the corrected error model, mapping its offline test (tool path equals the A1 study estimator) to the spatial_error_attenuation REML rows as the release evidence; acceptance bands unchanged.
census/http.py polite_get: honest per-module User-Agent, process-wide minimum interval per host (CENSUS_WWW2_MIN_INTERVAL, 1 s), exponential backoff with jitter on 429/500/502/503/504 and connection errors or timeouts (4 attempts, base 2 s, cap 60 s), Retry-After honoured. The firewall 'Request Rejected' page is retried once after 30 s (CENSUS_WWW2_REJECTED_RETRY_DELAY), then raises the caller's error with the support ID and the exact cache path for a browser-downloaded copy; it never loops. adopt_manual_file records such a copy on first read (SHA-256, source 'manual download'); fetch_vre and get_boundaries expose it as attrs['acquisition']. Switched: replicate, boundaries, pums, block_weights, validation, errata link check, scripts/fetch_acs_errata.py and scripts/fetch_tract_relationships.py. The firewall page is URL-specific, not only a rate limit; misleading comments fixed. Offline tests run without waits (tests/conftest.py).
…ason A file census.gov will not serve (firewall support ID, HTTP status, or not published) is recorded with its URL and reason, listed in the report's Data section with the cache path for a browser-downloaded copy, and only the variables that need it are skipped. Findings 6 and the conclusion updated for the 2015-2019 tract comparison.
Live rerun with --tract-years 2024 2019. 2015-2019 tracts: uncontrolled variables and New York population stable across releases; tract households change sign (positive within county in 2015-2019, aggregation ratio 0.90 / 1.00). The Michigan 2019 tract B01003 file was refused by the census.gov firewall (support ID 13979921158196382640) on both tries and is listed as unavailable.
…gn (2015-2019 households positive)
…; pace outside the lock; close retried streams - census.http.payload_problem: refuses HTML/firewall pages, empty files and zips without the expected member; adopt_manual_file moves a refused file aside to <path>.invalid (no sidecar) and names both paths, so a bad manual file no longer blocks every later run (also for VRE and boundaries). - PUMS cached_download and block relationship files: manual files validated before adoption, downloads validated before caching (nothing written on failure), invalid cached content moved aside and fetched again. - _pace reserves the per-host slot under the lock and sleeps outside it. - Retryable (and firewall-page) responses are closed before the retry, so a streamed body does not hold its connection.
…idates the area for that ratio A replicate ratio with a zero denominator was set to 0 (the SE convention of VRE documentation Example 7), which fabricates a replicate deviation of minus the estimate. The area is now excluded for that variable and counted in the report. Rerun (2024 and 2019 tracts, counties): 1-2 tracts per ratio variable and state excluded (MI 2024 owner_share 1; NY poverty_rate 2 and owner_share 1 in 2024 and 2019); class means, intervals, findings and conclusion unchanged at the reported precision.
…, rho bounds, eigenvalue cache, island multipliers - error_model_known_variances (census.spatial) is the one estimator behind correct_outcome_error=True and the A1 studies' diagonal-Sigma rows; me_sem_fit stays only for the dense-Sigma oracle (and the LR bootstrap). - correct_outcome_error refuses more than CORRECTION_MAX_N = 2000 areas (measured, one BLAS thread: n=1024 40 s / 0.5 GB, n=2025 3.9 min / 0.9 GB, n=3025 12.8 min / 1.7 GB) with a county/subset suggestion; MCP next_step. - SE == 0 is refused with a clear message; a singular V at the optimum raises the documented RuntimeError instead of a raw LinAlgError. - Lag refits with |rho| >= 1 count as failed refits (reported, subject to max_failed_replicates) instead of NaN effects that se_total dropped; a main fit with |rho| >= 1 gets a note and undefined effects. - W eigenvalues cached on the weights (eigvalsh on the symmetric similar matrix for symmetric-pattern W); spreg impacts off in our lag fits. - diagnostics['effects_multipliers'] include the island adjustment of the effects table.
…stimator spatial_error_attenuation and spatial_error_attenuation_correlated, full configuration (--jobs 0, 51.7 min), SFH rows now from census.spatial.error_model_known_variances. Every SFH coverage/bias/RMSE moved by at most 0.007; non-convergence 0 of 8,100 fits per estimator (was 50 REML / 37 ML with me_sem_fit). Quoted ranges in the tool notes hold (REML lambda 0.86-0.93 at lambda > 0, 0.83-0.89 at 0; beta 0.85-0.93); correlated-error caveat: 0.65-0.74 at lambda 0.3 (was 0.64-0.74).
Full --gate --jobs 0: PASS, 19 min; coverage table unchanged. Full suite: Python 3.14 1187 passed; Python 3.10 1183 passed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See PLAN_ROUND_16.md section 7 and the CHANGELOG 0.15.0 entry.
Local: full suite 1187 passed (3.14), 1183 passed (3.10); full gate PASS (19 min).