Add survey_estimation: design-based standard errors for clustered surveys

[Varnasr]

What does this PR do?

EquityStack could weight a mean and could not put a standard error on it. For a stratified, clustered national survey that gap is not cosmetic: an independence standard error is too small, often by a factor of two, and every confidence interval and test built on it errs in the direction that flatters the result.

survey_estimation/design_based_estimates.py implements the Taylor linearisation (ultimate cluster) estimator, the same one behind Stata's svy: prefix and R's survey package. It is not tied to any survey; it needs a weight, a PSU and a stratum.

Three things it gets right that a hand-rolled version usually does not:

• Subgroups are domains, not subsets. Filtering before estimating discards the PSUs holding none of the subgroup. Those PSUs belong to the design, and dropping them understates the standard error.
• Proportions get a logit interval. A 2.5 percent outcome concentrated in one cluster gives a linear lower bound of -0.024, which is not a proportion.
• Intervals use t on clusters minus strata and say so. Every result carries ci_dist. Where scipy is missing the normal fallback warns rather than silently narrowing: on 40 clusters and 8 strata that is [17.1, 32.8] against [16.9, 33.2].

survey_estimation/dhs_stunting.py reproduces India's published NFHS-5 stunting table by wealth quintile from the raw children's recode. It consumes the CSV written by InsightStack's data_starters/dhs-south-asia loader, so the two repositories couple through a file rather than an import and neither needs the other installed.

The comparison step is the point. DHS published 35.5 percent nationally, 46.1 in the poorest wealth quintile and 22.9 in the richest (API indicator CN_NUTS_C_HA2, survey IA2020DHS, retrieved 2026-09-08). A run that misses those has a fault upstream, and the script names the three that account for nearly all of them.

scipy is added to requirements.txt.

Type of change

• Bug fix (broken link, layout, JS error)
• New content (course, case study, translation)
• New feature or tool
• Accessibility improvement
• Performance improvement
• Documentation update

Checklist

• 14 tests added in tests/test_survey_estimation.py, all passing locally
• Each test pinned to a known answer rather than to the estimator's own output: one unit per cluster must collapse to s / sqrt(n) exactly, scaling every weight by a million must not move the standard error, and perfect intra-cluster correlation must give a design effect of exactly (N - 1) / (n - 1)
• Cross-validated against R's survey 4.2.1 on R 4.3.3, the reference implementation. Estimates, standard errors and degrees of freedom agree to twelve significant figures. Those reference values are hardcoded into the suite against a dataset built with no random number generator, so the check is stable across library versions
• pytest tests/ green, which is what CI runs
• Browser testing not applicable, no UI in this change

Two defects surfaced during development and are fixed here. The scipy fallback hardcoded the 95 percent quantile, so conf=0.99 silently returned a 95 percent interval, and it applied the normal without saying so, which is the same class of quiet wrongness this module exists to catch. Both are now tested.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DMYhECx2Dzbqad6KJxWZ7t

---

Generated by Claude Code