diff --git a/README.md b/README.md index e4c8de7..b928454 100644 --- a/README.md +++ b/README.md @@ -1,168 +1,93 @@ # EquityStack -**Python scripts and Jupyter notebooks for development sector data workflows.** +Python for distributional analysis and design-based survey estimation in +development research. Part of [OpenStacks](https://openstacks.dev). Status: +Stable, per the family +[maintenance policy](https://github.com/Varnasr/OpenStacks-for-Change/blob/main/MAINTENANCE.md). -[![Part of OpenStacks](https://img.shields.io/badge/Part%20of-OpenStacks-blue)](https://openstacks.dev) -[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE) -[![Status: Stable](https://img.shields.io/badge/Status-Stable-0969da?style=flat-square)](https://github.com/Varnasr/OpenStacks-for-Change/blob/main/MAINTENANCE.md) +Site: [varnasr.github.io/EquityStack](https://varnasr.github.io/EquityStack/). -> Distributional analysis and survey estimation for health, gender, education and climate equity data. +## The two main packages -> **Status: Stable.** This repository works and is correct, but it is not under active -> development. Bug reports are welcome and issues stay open; new features are unlikely, -> and replies are measured in weeks rather than days. Dependencies are pinned deliberately -> so that a clone still runs years from now. See the [maintenance policy](https://github.com/Varnasr/OpenStacks-for-Change/blob/main/MAINTENANCE.md). +**`inequality/`** measures how unequally an outcome is spread and across whom. +Gini, Theil, Atkinson, Palma, the 80/20 ratio, Lorenz and concentration +curves, the concentration index with the Erreygers and Wagstaff corrections, +Theil within/between decomposition, Blinder-Oaxaca, and benefit incidence by +level of service. All survey-weighted. 54 tests. See +[`inequality/README.md`](inequality/README.md). ---- - -## What This Is - -EquityStack is a collection of **Python scripts, Jupyter notebooks, and sample data** for development sector analysis. It provides ready-to-use utilities for data cleaning, validation, modelling, and visualisation — with a focus on public health, gender equity, education, and climate resilience workflows. - -This is the **data pipeline layer** of [OpenStacks for Change](https://openstacks.dev) — an open ecosystem of tools for public interest research and evaluation. - -## The two modules to read first - -Most of this repository is technique on stand-in data. Two parts are not, and -they are what the name is about. - -**`inequality/`** answers *who has it*. The Gini says how unequally consumption -is spread; the concentration index says whether stunting falls on the poor, by -how much, and comparably across states whose prevalence differs. It also -decomposes: `theil_decomposition` splits national inequality exactly into a -within-state and a between-state part, and `oaxaca_blinder` splits a group gap -into endowments and returns. `benefit_incidence` asks who actually receives a -public budget, which for tertiary health and higher education is usually not the -people it was voted for. - -**`survey_estimation/`** answers *how sure are we*. Weighting is the half -everyone remembers; a national household survey is also clustered, and an -interval that ignores that is too narrow, often by half, erring in the direction -that flatters the result. - -Together they are the two halves of an equity finding: the gradient, and whether -it is real. +**`survey_estimation/`** gives proportions and means for stratified, clustered +surveys with standard errors that account for the design, the same estimator +as Stata's `svy:` and R's `survey`. Checked against R's `survey` 4.2.1 to +twelve significant figures. A worked example reproduces the published NFHS-5 +stunting table. See [`survey_estimation/README.md`](survey_estimation/README.md). ```python from inequality import concentration_index, theil_decomposition from survey_estimation import svy_prop_by -svy_prop_by(df, "stunted", by="wealth_quintile") # is the gap real? -concentration_index(df.stunted, rank_by=df.wealth_index) # how steep is it? -theil_decomposition(df.consumption, groups=df.state) # where does it sit? +svy_prop_by(df, "stunted", by="wealth_quintile") +concentration_index(df.stunted, rank_by=df.wealth_index) +theil_decomposition(df.consumption, groups=df.state) ``` -`inequality/README.md` has the full function-by-function guide, the four things -that are easy to get wrong, and how to bootstrap a standard error over PSUs. - -## What's Inside - -### Core Modules - -| Module | What It Does | Status | -|--------|-------------|--------| -| `cleaning/` | Column name standardisation, dtype conversion, missing value handling, outlier flagging, cleaning log generation | Ready | -| `impact_evaluation/` | Causal inference: Difference-in-Differences, Propensity Score Matching, Regression Discontinuity Design | Ready | -| `validation/` | Input validation with Pydantic models | Ready | -| `io_helpers/` | Chunked CSV reading, Stata/SPSS import, formatted Excel export | Ready | -| `modelling/` | Multicollinearity checks (VIF) | Ready | -| `visualisation/` | Annotated bar charts, district-level choropleth maps | Ready | -| `inequality/` | **Distributional analysis**: Gini, Theil, Atkinson, Palma, Lorenz and concentration curves, the concentration index with Erreygers and Wagstaff corrections, Theil within/between decomposition, Blinder-Oaxaca, benefit incidence | Ready | -| `social_sector/` | Composite indices with explicit direction, normalisation and weighting, plus a rank-sensitivity check | Ready | -| `survey_estimation/` | Design-based proportions and means for stratified, clustered surveys, with a worked NFHS-5 example | Ready | - -### Notebooks - -| Notebook | What It Does | -|----------|-------------| -| `sector_gender_summary.ipynb` | Gender-disaggregated analysis with sample data | -| `sector_wee_time_use.ipynb` | Women's economic empowerment time-use analysis | - -### Data and Testing +## Other modules -| Directory | What It Contains | -|-----------|-----------------| -| `sample_data/` | Gender sample and time-use sample datasets | -| `tests/` | 11 pytest test files covering all core modules | -| `scripts/` | Standalone export utilities | +| Module | What it does | +| --- | --- | +| `cleaning/` | Column names, types, missing values, outlier flags, a cleaning log | +| `impact_evaluation/` | Difference-in-differences, propensity score matching, regression discontinuity | +| `validation/` | Input validation with Pydantic models | +| `io_helpers/` | Chunked CSV reading, Stata and SPSS import, formatted Excel export | +| `modelling/` | Multicollinearity checks (VIF) | +| `visualisation/` | Annotated bar charts, district choropleth maps | +| `social_sector/` | Composite indices with stated direction, normalisation and weights, and a rank-sensitivity check | +| `notebooks/` | Two worked analyses: gender-disaggregated outcomes, women's time use | +| `sample_data/` | Small CSVs to try the code on | -## Getting Started +Most of these are short. `inequality/`, `survey_estimation/` and +`impact_evaluation/` hold most of the code. -### Prerequisites +## Install and test -- **Python 3.8+** -- **Jupyter** (notebook or lab) - -### Installation +Python 3.11. Versions in `requirements.txt` are pinned and were verified +together. ```bash git clone https://github.com/Varnasr/EquityStack.git cd EquityStack pip install -r requirements.txt -jupyter notebook +PYTHONPATH=$(pwd) pytest tests/ # 101 tests ``` -### Quick Start - -1. Open a notebook from `notebooks/` to see a working analysis -2. Load practice data from `sample_data/` -3. Use `cleaning/` to prepare your own data -4. Apply `modelling/` and `visualisation/` for analysis and outputs -5. Export with `io_helpers/` for Excel or dashboard-ready formats - -### Key Dependencies - -- pandas, numpy, scipy — data manipulation and statistics -- matplotlib, seaborn — visualisation -- statsmodels — modelling -- openpyxl, xlsxwriter — Excel I/O -- pyreadstat — Stata and SPSS I/O -- pydantic — data validation -- geopandas, folium — spatial mapping -- ydata-profiling — quick EDA reports - -Versions are pinned in `requirements.txt` and were resolved together and -verified on Python 3.11 with the suite green. - -## How It Connects - -EquityStack is one of several stacks in the [OpenStacks](https://openstacks.dev) ecosystem: +## The family -| Stack | Focus | -|-------|-------| -| [InsightStack](https://github.com/Varnasr/InsightStack) | MEL tools, calculators, research documentation, and loaders for real survey microdata. Stata, Python, R, SPSS | -| [FieldStack](https://github.com/Varnasr/FieldStack) | Field operations while a survey is being collected, and survey analysis after. R | -| **EquityStack** (this repo) | Equity from a development economics perspective: distributional analysis, the concentration index, and design-based survey estimation. Python | +| Repository | What it is for | Language | +| --- | --- | --- | +| [InsightStack](https://github.com/Varnasr/InsightStack) | MEL tools, calculators, research documentation, loaders for survey microdata | Stata, Python, R, SPSS | +| [FieldStack](https://github.com/Varnasr/FieldStack) | Field operations while a survey is in the field; sampling and weighted estimation after | R | +| **EquityStack** (this repository) | Inequality measurement and design-based survey estimation | Python | -[openstacks.dev](https://openstacks.dev) is the index for all of it. [SignalStack](https://github.com/Varnasr/SignalStack) is the companion archive for the [Research Rundown](https://varna.substack.com) newsletter, alongside the stacks rather than one of them. [PolicyStack](https://github.com/Varnasr/PolicyStack) is superseded by [PolicyDhara](https://github.com/Varnasr/PolicyDhara). RootStack, BridgeStack and ViewStack are archived. +[openstacks.dev](https://openstacks.dev) is the index. +[SignalStack](https://github.com/Varnasr/SignalStack) is the companion archive +for the [Research Rundown](https://varna.substack.com) newsletter, beside the +stacks rather than one of them. +[PolicyStack](https://github.com/Varnasr/PolicyStack) is superseded by +[PolicyDhara](https://github.com/Varnasr/PolicyDhara). RootStack, BridgeStack +and ViewStack are archived. -**Use EquityStack when** you work in Python/Jupyter. Use **FieldStack** for R-based equivalents. Use **InsightStack** for Stata tools and MEL calculators. +`survey_estimation/dhs_stunting.py` reads the CSV written by InsightStack's +DHS loader. FieldStack's `survey_tools/dhs_stunting.R` does the same in R, and +the two agree on the same data. -## Contributing - -Contributions welcome — especially from data practitioners in the development sector. See [contributing guidelines](https://github.com/Varnasr/.github/blob/main/CONTRIBUTING.md). - -High-impact areas: -- **EDA tools** — correlation matrices, group summaries, data profiling -- **Modelling** — logistic regression evaluation, OLS summary tables -- **Social sector** — climate risk flags, education outcomes, gender disaggregation, WEE analysis -- **Notebooks** — more worked examples with real analysis workflows -- **Visualisation** — categorical distributions, regression diagnostics, time series - -## Citation +## Citation and license ```bibtex @software{equitystack, author = {Sri Raman, Varna}, - title = {EquityStack: Python Workflows for Development Data}, + title = {EquityStack: inequality measurement and survey estimation in Python}, url = {https://github.com/Varnasr/EquityStack} } ``` -## License - -MIT — free to use, modify, and share. See [LICENSE](LICENSE). - ---- - -Part of [OpenStacks for Change](https://openstacks.dev). Created by [Varna Sri Raman](https://on-web.link/varna). +MIT. See [LICENSE](LICENSE). diff --git a/docs/run_ready_header.md b/docs/run_ready_header.md index a337deb..ccc418e 100644 --- a/docs/run_ready_header.md +++ b/docs/run_ready_header.md @@ -1,6 +1,6 @@ --- -## 🟢 Run-Ready Notebook +## Run-Ready Notebook This notebook is ready to run out-of-the-box. It uses data from the `/sample_data` folder and requires no additional configuration. diff --git a/docs/spatial_tools_snippet.md b/docs/spatial_tools_snippet.md index 0a44ef9..bf95cb8 100644 --- a/docs/spatial_tools_snippet.md +++ b/docs/spatial_tools_snippet.md @@ -1,5 +1,5 @@ -### 🗺 Spatial Tools +### Spatial Tools EquityStack uses: - `geopandas` for reading, transforming, and joining shapefiles diff --git a/index.html b/index.html index 3834b16..f46ec75 100644 --- a/index.html +++ b/index.html @@ -3,8 +3,8 @@ -EquityStack · Python for development sector data workflows - +EquityStack · Inequality measurement and survey estimation in Python + @@ -17,7 +17,7 @@
EquityStack
-

Varna Sri Raman. Built for reproducibility, and licensed so you can take it.

+

Varna Sri Raman. MIT licence.

diff --git a/inequality/README.md b/inequality/README.md index a8af0ae..6404a2e 100644 --- a/inequality/README.md +++ b/inequality/README.md @@ -1,13 +1,7 @@ # inequality -Distributional analysis for survey microdata: how unequally something is spread, -and across whom. - -The rest of this repository answers "what is the level, and how sure are we". -This package answers the question that follows: **who has it**. Those are -different questions and the second one is the harder to get right, because every -measure of it is a covariance between an outcome and a position in a -distribution, and positions are where the errors hide. +Distributional analysis for survey microdata: how unequally something is +spread, and across whom. Every function takes optional survey weights. ```python from inequality import gini, concentration_index, theil_decomposition @@ -17,79 +11,68 @@ concentration_index(df.stunted, rank_by=df.wealth_index, weights=df.child_weight theil_decomposition(df.consumption, groups=df.state, weights=df.hh_weight) ``` -## What to reach for +## Functions -**One variable, how unequal is it?** +**One variable: how unequal is it?** -| Function | Use when | -|---|---| -| `gini` | The headline. Comparable with almost every published figure. | -| `theil_t`, `theil_l` | You need the within/between split. Only GE(0) and GE(1) decompose exactly. | -| `atkinson` | You want the inequality aversion stated rather than implied. | -| `generalised_entropy` | You want to choose where the index is sensitive: GE(0) bottom, GE(1) neutral, GE(2) top. | -| `palma_ratio`, `ratio_80_20` | The audience is not technical. Both are ratios of shares. | -| `quantile_shares`, `lorenz_curve`, `share_of_top` | You want the distribution itself, not a summary of it. | +| Function | Use | +| --- | --- | +| `gini` | The standard summary; comparable with most published figures | +| `theil_t`, `theil_l` | Entropy measures; the two that decompose exactly into within and between | +| `atkinson` | Inequality with a stated aversion parameter | +| `generalised_entropy` | GE(α): sensitive to the bottom (0), neutral (1) or the top (2) | +| `palma_ratio`, `ratio_80_20` | Ratios of shares, for a non-technical audience | +| `quantile_shares`, `lorenz_curve`, `share_of_top` | The distribution itself | -**Two variables, unequal across whom?** +**Two variables: unequal across whom?** -| Function | Use when | -|---|---| -| `concentration_index` | The core: is this outcome concentrated among the poor? Negative means yes. | -| `erreygers_index` | Comparing the same indicator across states or years whose mean differs. | -| `wagstaff_index` | The same problem, relative rather than absolute. Pick one and say which. | -| `concentration_curve` | Plotting it, or checking dominance between two distributions. | -| `concentration_index_by` | A table: one gradient per state, per round, per sex. | -| `achievement_index` | One number combining level and distribution. Report it beside the mean, never instead. | +| Function | Use | +| --- | --- | +| `concentration_index` | Is the outcome concentrated among the poor? Negative means yes | +| `erreygers_index` | Comparing the same indicator across states or years with different means, absolute version | +| `wagstaff_index` | The same, relative version. Pick one and say which | +| `concentration_curve` | For plotting, or checking dominance between two distributions | +| `concentration_index_by` | One index per group: state, round, sex | +| `achievement_index` | Level and distribution in one number. Report beside the mean | **What explains it?** -| Function | Use when | -|---|---| -| `theil_decomposition` | Is national inequality *between* states or *within* them? | -| `oaxaca_blinder` | How much of a group gap is endowments, how much is returns? | +| Function | Use | +| --- | --- | +| `theil_decomposition` | Within-group and between-group parts of total inequality | +| `oaxaca_blinder` | A group gap split into endowments and returns | **Who gets the money?** -| Function | Use when | -|---|---| -| `benefit_incidence` | Allocating a budget across quintiles by who uses the service. | -| `benefit_incidence_by_level` | The same, split by primary / secondary / tertiary, which is usually where the finding is. | +| Function | Use | +| --- | --- | +| `benefit_incidence` | A budget allocated across quintiles by who uses the service | +| `benefit_incidence_by_level` | The same, split by primary, secondary and tertiary | -## Four things that are easy to get wrong +## Notes -**The concentration index ranks by living standards, not by the outcome.** -Reversing them returns a number in the right range with the wrong meaning, and -nothing errors. The signature makes it awkward to do by accident: outcome first, -then `rank_by=`. +The concentration index ranks by living standards, not by the outcome. The +signature is `concentration_index(outcome, rank_by=...)`. Reversing them +returns a number in the right range with the wrong meaning. -**Sign convention.** Negative means concentrated among the *poor*. For stunting -that is the expected direction and it is bad news; for institutional delivery it -is good news. The index has no opinion about which. +Negative means concentrated among the poor. For stunting that is the expected +direction; for institutional delivery it is the good direction. -**A raw concentration index is not comparable across different prevalences.** Its -theoretical range shrinks as a binary outcome's mean moves away from 0.5, so an -indicator at 8 per cent and the same indicator at 60 per cent cannot be put in -one table without a correction. Erreygers keeps absolute gradients comparable, -Wagstaff keeps relative ones, and they can disagree about the direction of -change over time. Choose before you look at the answer. +A raw concentration index is not comparable across different prevalences, +because its range shrinks as a binary outcome's mean moves away from 0.5. Use +`erreygers_index` for absolute comparisons and `wagstaff_index` for relative +ones. They can disagree about the direction of change; choose before looking +at the answer. -**Ties.** A wealth quintile is a legitimate ranking variable and this package -handles it the way the World Bank's own DHS equity work does: everyone sharing a -quintile shares the midpoint rank of the block that quintile occupies. The index -is attenuated relative to a continuous wealth measure, which is a property of the -coarser data and not a defect. `test_inequality.py` asserts the attenuation runs -in the expected direction. +Ties, such as a wealth quintile, share the midpoint rank of their block. This +is the World Bank's DHS convention. The index is smaller than it would be on a +continuous wealth measure; a test checks the attenuation runs the expected +way. ## Standard errors -None of these functions return one. The analytic variance of a concentration -index under a stratified clustered design is not something to write from memory, -and the convenient regression form people quote gives a standard error that -ignores the design entirely. - -Bootstrap over PSUs within strata instead, resampling clusters rather than -households, and use this repository's `survey_estimation` package for the design -object. Roughly: +None of these functions returns one. Bootstrap over PSUs within strata, +resampling clusters rather than households: ```python import numpy as np @@ -109,44 +92,32 @@ def boot_ci(df, reps=500, seed=0): return np.percentile(out, [2.5, 97.5]) ``` -## Verification - -`tests/test_inequality.py`, 54 checks. Every one is either an identity the -measure must satisfy or a figure worked out by hand from the definition, never -a number this code produced on a previous run. A distributional index cannot be -checked by eye: 0.31 and 0.34 are both plausible consumption Ginis, and a sign -error gives you a number in exactly the right range that says the opposite of -the truth. - -The identities that carry the most weight: - -- The Gini equals the mean absolute difference over twice the mean, on fifteen - random datasets, computed independently by brute force over all pairs. -- Twice the area between the Lorenz curve and the diagonal equals the Gini, and - the same for the concentration curve and the concentration index. -- The mean fractional rank is exactly 0.5 for any weights and any pattern of - ties. Three derivations here depend on it. -- `within + between == total` in the Theil decomposition, for both GE(0) and - GE(1), with weights. -- Oaxaca's components sum to the gap under all four reference choices. -- A weight of 2 gives the same answer as the row appearing twice. -- GE(2) equals half the squared coefficient of variation. +## Tests + +`tests/test_inequality.py`, 54 checks. Each is an identity the measure must +satisfy or a value worked out by hand: the Gini against the mean absolute +difference over twice the mean, computed by brute force over all pairs; twice +the area between the Lorenz curve and the diagonal against the Gini; the mean +fractional rank equal to 0.5 under any weights and ties; `within + between == +total` for GE(0) and GE(1); Oaxaca components summing to the gap under all four +reference choices; a weight of 2 equal to a row appearing twice; GE(2) equal to +half the squared coefficient of variation. ## Sources -Cowell, *Measuring Inequality*, 3rd edn, Oxford University Press 2011, chapters -2-3, for the entropy family and the axioms. +Cowell, *Measuring Inequality*, 3rd edn, Oxford University Press, 2011, +chapters 2 and 3. + +O'Donnell, van Doorslaer, Wagstaff and Lindelow, *Analyzing Health Equity +Using Household Survey Data*, World Bank, 2008, chapters 8 and 15. -O'Donnell, van Doorslaer, Wagstaff and Lindelow, *Analyzing Health Equity Using -Household Survey Data*, World Bank 2008, chapters 8 and 15, for the -concentration index and its corrections. +Erreygers, "Correcting the concentration index", *Journal of Health +Economics* 28(2), 2009, 504-515. -Erreygers, "Correcting the concentration index", *Journal of Health Economics* -28(2), 2009, 504-515. Wagstaff, "The bounds of the concentration index when the -variable of interest is binary", *Health Economics* 14(4), 2005, 429-432. +Wagstaff, "The bounds of the concentration index when the variable of +interest is binary", *Health Economics* 14(4), 2005, 429-432. -Demery, *Benefit Incidence: A Practitioner's Guide*, World Bank 2000. +Demery, *Benefit Incidence: A Practitioner's Guide*, World Bank, 2000. Fortin, Lemieux and Firpo, "Decomposition methods in economics", *Handbook of -Labor Economics* 4A, 2011, section 3, on what the unexplained component is and -is not. +Labor Economics* 4A, 2011, section 3. diff --git a/survey_estimation/README.md b/survey_estimation/README.md index 4f68f2b..e09383b 100644 --- a/survey_estimation/README.md +++ b/survey_estimation/README.md @@ -1,54 +1,35 @@ # Survey estimation -Design-based estimates from complex survey data: weighted proportions and means -with standard errors that account for stratification and clustering. - -Weights are the easy half and the half everyone remembers. A national household -survey is also stratified and clustered, so a standard error computed as though -the sample were independent is too small, often by a factor of two or more, and -every confidence interval and test built on it is wrong in the direction that -flatters the finding. - -`design_based_estimates.py` implements the Taylor linearisation (ultimate -cluster) variance estimator, the same one behind Stata's `svy:` prefix and R's -`survey` package. It is not tied to any particular survey. +Weighted proportions and means for stratified, clustered surveys, with +standard errors that account for the design. `design_based_estimates.py` +implements the Taylor linearisation (ultimate cluster) estimator used by +Stata's `svy:` and R's `survey`. ```python from survey_estimation import svy_prop_by svy_prop_by(df, "stunted", by="wealth_quintile") ``` -The DataFrame needs a weight, a PSU and a stratum column. Everything else is -optional. +The DataFrame needs a weight, a PSU and a stratum column. -## Two things it does that a hand-rolled version usually does not +## Behaviour -**Subgroups are estimated as domains, not subsets.** Filtering the data before -estimating a subgroup throws away the PSUs that contain none of its members. -Those PSUs are still part of the design and still count towards the stratum's -cluster total, so dropping them understates the standard error. Pass `domain=` -or use `svy_prop_by`, and they are kept. +Subgroups are estimated as domains. The PSUs that contain none of the +subgroup's members stay in the design, so the standard error is not +understated. Use `domain=` or `svy_prop_by`. -**Proportions get a logit interval.** Near zero or one, a linear interval runs -outside [0, 1] and reports something that is not a proportion. In the test -suite, a 2.5 percent outcome concentrated in one cluster produces a linear +Proportions get a logit interval, so a small proportion never has a negative +lower bound. In the tests, a 2.5 per cent outcome in one cluster has a linear lower bound of -0.024 and a logit lower bound of 0.003. -**The interval says which distribution produced it.** Degrees of freedom are -clusters minus strata, and with a few dozen clusters the t quantile is -noticeably larger than 1.96. Every result carries a `ci_dist` field. Without -`scipy` the module falls back to the normal quantile, which makes intervals -slightly too narrow, and it warns rather than doing so quietly: on a 40-cluster, -8-stratum example the lowest quintile's interval is [17.1, 32.8] under the -normal and [16.9, 33.2] under t on 32 degrees of freedom. +Intervals use t on clusters minus strata and report the distribution in +`ci_dist`. Without `scipy` the module uses the normal quantile and warns. ## Worked example -`dhs_stunting.py` reproduces India's published NFHS-5 stunting table from the -raw children's recode, by wealth quintile, with design-based intervals. It is -the second half of a chain that starts in -[InsightStack](https://github.com/Varnasr/InsightStack)'s -`data_starters/dhs-south-asia/`: +`dhs_stunting.py` reproduces India's published NFHS-5 stunting table by +wealth quintile from the raw children's recode. The input CSV comes from +InsightStack's DHS loader: ``` # in InsightStack @@ -58,15 +39,10 @@ python load_dhs.py IAKR7EFL.DTA --vars v190 v025 hw70 b5 --anthro --out children python -m survey_estimation.dhs_stunting children.csv ``` -The two repositories are coupled through a CSV rather than an import, so -neither needs the other installed. - -The comparison step is the point. DHS published NFHS-5 stunting at 35.5 percent -nationally, 46.1 in the poorest wealth quintile and 22.9 in the richest. A run -that does not land within a few tenths of that has a fault upstream, and the -script names the three that account for almost all of them: an unscaled weight, -a subgroup filtered before estimation, an anthropometry flag divided instead of -dropped. +DHS published 35.5 per cent nationally, 46.1 in the poorest quintile and 22.9 +in the richest. A run that does not land within a few tenths of that has a +fault upstream. The three usual ones: an unscaled weight, a subgroup filtered +before estimation, an anthropometry flag divided instead of dropped. ## Tests @@ -74,8 +50,8 @@ dropped. python -m pytest tests/test_survey_estimation.py ``` -Twelve checks, each pinned to a case with a known closed-form answer rather than -to the estimator's own output: one unit per cluster must reduce to `s / sqrt(n)`, -multiplying every weight by a million must leave the standard error unchanged, -and a sample where every unit inside a cluster is identical must produce a -design effect of exactly `(N - 1) / (n - 1)`. +Twelve checks against closed-form answers: one unit per cluster reduces to +`s / sqrt(n)`; scaling every weight by a million leaves the standard error +unchanged; identical units within each cluster give a design effect of +`(N - 1) / (n - 1)`. A regression test against R's `survey` 4.2.1 agrees to +twelve significant figures. diff --git a/tests/README.md b/tests/README.md index 8f9784d..34ad857 100644 --- a/tests/README.md +++ b/tests/README.md @@ -3,7 +3,7 @@ This folder contains automated tests for all core `.py` scripts in the repo. These are **sanity checks** to ensure that each function runs with expected inputs and does not raise errors. -## 🔧 How to Run +## How to Run 1. Install dependencies: @@ -18,7 +18,7 @@ pip install pytest pytest tests/ ``` -## 🧪 Coverage +## Coverage This suite tests: