diff --git a/CLAUDE.md b/CLAUDE.md index b693a18..ee45990 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -7,10 +7,18 @@ Python scripts and notebooks for development sector data workflows. Part of the ## Layout `cleaning/`, `eda/`, `modelling/`, `validation/`, `io_helpers/`, -`impact_evaluation/`, `social_sector/`, `visualisation/`, plus +`impact_evaluation/`, `social_sector/`, `visualisation/`, plus the two that +carry the repository: `inequality/` for distributional analysis and `survey_estimation/` for design-based estimates from complex surveys. Notebooks in `notebooks/`, small CSVs in `sample_data/`, tests in `tests/`. +Sizes, so nobody has to guess again: 101 tests, and `inequality/` (about 950 +lines) plus `survey_estimation/` (357) plus `impact_evaluation/` (220) are most +of the substance. The rest is technique taught against stand-in data, and some +of it is a handful of lines. That is a deliberate split, not neglect, but do not +describe a five-line snippet folder as a module in any user-facing copy: an +audit on 2026-09-08 found the landing page doing exactly that. + House style is numpydoc docstrings, one module per group of related functions, pytest with plain asserts. Match it. @@ -43,6 +51,38 @@ pip install -r requirements.txt PYTHONPATH=$(pwd) pytest tests/ ``` +## inequality + +The package that makes the repository's name true. Everything in it is a +covariance between an outcome and a position in a distribution, and positions +are where the errors hide, so three things are centralised rather than +reimplemented per function. + +- **Weighted fractional ranks** live in `inequality/ranks.py` and are shared by + the Gini, the concentration index and both curves, so those three agree by + construction. Ties take the block midpoint, which is what the World Bank's DHS + equity work does and what makes a wealth quintile a legitimate rank variable. +- **The mean fractional rank is exactly 0.5** for any weights and any pattern of + ties. Three derivations depend on it and a test asserts it. +- **Sign convention: negative means concentrated among the poor.** Reversing + `outcome` and `rank_by` returns a number in the right range with the opposite + meaning and nothing errors, which is why the signature is + `concentration_index(outcome, rank_by=...)`. + +A raw concentration index is not comparable across different prevalences, so +`erreygers_index` and `wagstaff_index` exist; they answer different normative +questions and can disagree about the direction of change over time. Pick one per +table and say which. + +**Do not add an analytic standard error to these.** The convenient regression +form people quote ignores the survey design entirely. Bootstrap over PSUs within +strata; `inequality/README.md` carries the recipe. + +Tests are pinned to closed-form identities, never to previous output: the Gini +against brute-force mean absolute difference on random data, curve areas against +their indices by trapezoid, `within + between == total` for both GE(0) and +GE(1), Oaxaca's components summing to the gap under all four reference choices. + ## survey_estimation The one part worth reading before touching. It implements Taylor linearisation diff --git a/README.md b/README.md index 2e34e52..798b0ee 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE) [![Status: Stable](https://img.shields.io/badge/Status-Stable-0969da?style=flat-square)](https://github.com/Varnasr/OpenStacks-for-Change/blob/main/MAINTENANCE.md) -> Plug-and-play templates for health, gender, education, and climate equity data — built for reproducibility. +> Distributional analysis and survey estimation for health, gender, education and climate equity data. > **Status: Stable.** This repository works and is correct, but it is not under active > development. Bug reports are welcome and issues stay open; new features are unlikely, @@ -21,6 +21,40 @@ EquityStack is a collection of **Python scripts, Jupyter notebooks, and sample d This is the **data pipeline layer** of [OpenStacks for Change](https://openstacks.dev) — an open ecosystem of tools for public interest research and evaluation. +## The two modules to read first + +Most of this repository is technique on stand-in data. Two parts are not, and +they are what the name is about. + +**`inequality/`** answers *who has it*. The Gini says how unequally consumption +is spread; the concentration index says whether stunting falls on the poor, by +how much, and comparably across states whose prevalence differs. It also +decomposes: `theil_decomposition` splits national inequality exactly into a +within-state and a between-state part, and `oaxaca_blinder` splits a group gap +into endowments and returns. `benefit_incidence` asks who actually receives a +public budget, which for tertiary health and higher education is usually not the +people it was voted for. + +**`survey_estimation/`** answers *how sure are we*. Weighting is the half +everyone remembers; a national household survey is also clustered, and an +interval that ignores that is too narrow, often by half, erring in the direction +that flatters the result. + +Together they are the two halves of an equity finding: the gradient, and whether +it is real. + +```python +from inequality import concentration_index, theil_decomposition +from survey_estimation import svy_prop_by + +svy_prop_by(df, "stunted", by="wealth_quintile") # is the gap real? +concentration_index(df.stunted, rank_by=df.wealth_index) # how steep is it? +theil_decomposition(df.consumption, groups=df.state) # where does it sit? +``` + +`inequality/README.md` has the full function-by-function guide, the four things +that are easy to get wrong, and how to bootstrap a standard error over PSUs. + ## What's Inside ### Core Modules @@ -33,7 +67,8 @@ This is the **data pipeline layer** of [OpenStacks for Change](https://openstack | `io_helpers/` | Chunked CSV reading, Stata/SPSS import, formatted Excel export | Ready | | `modelling/` | Multicollinearity checks (VIF) | Ready | | `visualisation/` | Annotated bar charts, district-level choropleth maps | Ready | -| `social_sector/` | Public health access index | Ready | +| `inequality/` | **Distributional analysis**: Gini, Theil, Atkinson, Palma, Lorenz and concentration curves, the concentration index with Erreygers and Wagstaff corrections, Theil within/between decomposition, Blinder-Oaxaca, benefit incidence | Ready | +| `social_sector/` | Composite indices with explicit direction, normalisation and weighting, plus a rank-sensitivity check | Ready | | `survey_estimation/` | Design-based proportions and means for stratified, clustered surveys, with a worked NFHS-5 example | Ready | ### Notebooks diff --git a/index.html b/index.html index 417e30e..3834b16 100644 --- a/index.html +++ b/index.html @@ -17,8 +17,9 @@
EquityStack
@@ -26,20 +27,72 @@

EquityStack

-

The Python you reach for on a development research project: getting messy data into - shape, asking whether a programme worked, and getting the confidence intervals right - when the survey is clustered. Part of OpenStacks.

+

The Python for a development research project, pointed at the distributional + question: not just what the average is, but who is carrying the bad outcome, how steep the + gradient runs, and whether the gap survives a clustered sample. + Part of OpenStacks.

-
9Modules
-
34Python files
-
2Notebooks
+
24Distributional measures
+
10Modules
4Impact evaluation methods
+
101Tests
-
+
+
+

Distributional analysis

+

Who is carrying it, and how steeply

+
+
+ +

inequality/

+

The concentration index

+

An average tells you the level. It cannot tell you whether stunting falls on the poor, + which is the question a programme budget answers to. The concentration index does, on a + scale where negative means concentrated among the poor, and it works off a wealth + quintile as readily as a continuous consumption measure.

+

Read the module →

+
+ +

Comparability

+

Why the raw index misleads

+

A concentration index is not comparable between two places whose prevalence differs, + because its possible range shrinks as the mean moves away from half. Comparing states or + years without correcting for that is a standard error in published work. Both fixes are + here, and they can disagree about which way things moved.

+

Read the guide →

+
+ +

Decomposition

+

Where the inequality sits

+

A national Gini of 0.35 is equally consistent with identical states that differ from each + other and with unequal states that look alike on average. Those want opposite responses. + The Theil split answers it exactly, and Blinder-Oaxaca separates how much of a group gap + is land and schooling from how much is the return on them.

+

Read the module →

+
+ +

Benefit incidence

+

Who receives the budget

+

Public tertiary hospitals and universities are free at the point of use and mostly used by + the better-off, so the spending is regressive whatever the intention. Splitting a budget + by level of service usually shows primary care progressive and tertiary regressive, the + two cancelling in any aggregate figure.

+

Read the module →

+
+
+

Twenty-four measures: Gini, Theil, Atkinson, Palma, the 80/20 ratio, Lorenz and + concentration curves, quantile shares, the Erreygers and Wagstaff corrections, within and + between decomposition, Blinder-Oaxaca, benefit incidence. All survey-weighted. Verified + against identities rather than against previous output, because 0.31 and 0.34 are both + plausible Ginis and a sign error gives a number in the right range that says the opposite of + the truth.

+
+ + -