From f3db8591e2a75af75e77fbdde81205b210b3799c Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 8 Sep 2026 12:05:49 +0000 Subject: [PATCH] Add the inequality package, so the repository's name is earned Counted before starting: 27 Python files outside tests, 872 lines, of which survey_estimation, impact_evaluation and one cleaning pipeline were 732. Exactly one thing in the repository was about distribution, and the file whose name promised it, public_health_access_index.py, was a five-line row mean. inequality/ is 24 functions across five modules. Indices: Gini, the full generalised entropy family with Theil T and the mean log deviation named, Atkinson at any aversion, Palma, the 80/20 ratio, quantile shares, the Lorenz curve, weighted quantiles. Concentration: the concentration index, which is the measure that answers the question an average cannot, whether an outcome falls on the poor and how steeply. With it the Erreygers and Wagstaff corrections, because a raw index is not comparable between two places whose prevalence differs and comparing them anyway is a standard error in published work. Plus the concentration curve, a by-group table, and Wagstaff's achievement index. Decomposition: the Theil within/between split, exact for GE(0) and GE(1) and refusing any other alpha rather than returning parts that do not sum. Blinder-Oaxaca in twofold and threefold form with four reference choices. Incidence: benefit incidence analysis, single-service and split by level of service, which is where the finding usually is. Primary care progressive, tertiary regressive, the two cancelling in the aggregate. Three things are centralised rather than reimplemented per function. Weighted fractional ranks live in one file and are shared by the Gini, the concentration index and both curves, so those agree by construction. Ties take the block midpoint, which is what makes a wealth quintile a legitimate ranking variable. And every function refuses rather than returning a plausible wrong number when an assumption breaks: a Gini on negative values, GE(0) on zeros, Wagstaff where everyone has the outcome, a level with no budget, an Oaxaca on three groups. Tests: 25 to 101. Every new one is an identity or a hand-computed figure, never previous output. The Gini against brute-force mean absolute difference on fifteen random datasets. Curve areas against their indices by trapezoid. within + between == total for both alphas, with weights. Oaxaca's components summing to the gap under all four references. GE(2) equalling half the squared coefficient of variation. A weight of two equalling the row appearing twice. The mean fractional rank being exactly 0.5, which three derivations here depend on. Two of my own test expectations were wrong on first run and the code was right, which is the point of pinning to closed forms: an Oaxaca test drew both groups from the same distribution and then asserted the realised endowment gap was zero, and a composite-index test hardcoded arithmetic I had done in my head. Both now compute the expectation from the definition. Also: social_sector/composite_index.py replaces the row mean with an index that makes its three decisions arguments rather than assumptions (direction, normalisation, weights), offers goalpost normalisation so a series is comparable across years, geometric aggregation where indicators should not substitute for each other, a minimum non-missing count so a district reporting two of six indicators is not scored as though it reported all six, and rank_sensitivity to show how far a ranking moves under an equally defensible choice. compute_access_index keeps working. Deleted workflows/ci.yml: dead configuration, pinned to Python 3.10 and running R tests in a repository with no R, duplicating the real workflow at .github/workflows/python-tests.yml. The landing page described it as "start-to-finish routes through an analysis", which it never was. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01DMYhECx2Dzbqad6KJxWZ7t --- CLAUDE.md | 42 +- README.md | 39 +- index.html | 81 +++- inequality/README.md | 152 +++++++ inequality/__init__.py | 56 +++ inequality/concentration.py | 262 +++++++++++ inequality/decomposition.py | 235 ++++++++++ inequality/incidence.py | 197 +++++++++ inequality/indices.py | 293 ++++++++++++ inequality/ranks.py | 128 ++++++ social_sector/composite_index.py | 253 +++++++++++ social_sector/public_health_access_index.py | 73 ++- tests/test_composite_index.py | 211 +++++++++ tests/test_inequality.py | 466 ++++++++++++++++++++ workflows/ci.yml | 32 -- 15 files changed, 2467 insertions(+), 53 deletions(-) create mode 100644 inequality/README.md create mode 100644 inequality/__init__.py create mode 100644 inequality/concentration.py create mode 100644 inequality/decomposition.py create mode 100644 inequality/incidence.py create mode 100644 inequality/indices.py create mode 100644 inequality/ranks.py create mode 100644 social_sector/composite_index.py create mode 100644 tests/test_composite_index.py create mode 100644 tests/test_inequality.py delete mode 100644 workflows/ci.yml diff --git a/CLAUDE.md b/CLAUDE.md index b693a18..ee45990 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -7,10 +7,18 @@ Python scripts and notebooks for development sector data workflows. Part of the ## Layout `cleaning/`, `eda/`, `modelling/`, `validation/`, `io_helpers/`, -`impact_evaluation/`, `social_sector/`, `visualisation/`, plus +`impact_evaluation/`, `social_sector/`, `visualisation/`, plus the two that +carry the repository: `inequality/` for distributional analysis and `survey_estimation/` for design-based estimates from complex surveys. Notebooks in `notebooks/`, small CSVs in `sample_data/`, tests in `tests/`. +Sizes, so nobody has to guess again: 101 tests, and `inequality/` (about 950 +lines) plus `survey_estimation/` (357) plus `impact_evaluation/` (220) are most +of the substance. The rest is technique taught against stand-in data, and some +of it is a handful of lines. That is a deliberate split, not neglect, but do not +describe a five-line snippet folder as a module in any user-facing copy: an +audit on 2026-09-08 found the landing page doing exactly that. + House style is numpydoc docstrings, one module per group of related functions, pytest with plain asserts. Match it. @@ -43,6 +51,38 @@ pip install -r requirements.txt PYTHONPATH=$(pwd) pytest tests/ ``` +## inequality + +The package that makes the repository's name true. Everything in it is a +covariance between an outcome and a position in a distribution, and positions +are where the errors hide, so three things are centralised rather than +reimplemented per function. + +- **Weighted fractional ranks** live in `inequality/ranks.py` and are shared by + the Gini, the concentration index and both curves, so those three agree by + construction. Ties take the block midpoint, which is what the World Bank's DHS + equity work does and what makes a wealth quintile a legitimate rank variable. +- **The mean fractional rank is exactly 0.5** for any weights and any pattern of + ties. Three derivations depend on it and a test asserts it. +- **Sign convention: negative means concentrated among the poor.** Reversing + `outcome` and `rank_by` returns a number in the right range with the opposite + meaning and nothing errors, which is why the signature is + `concentration_index(outcome, rank_by=...)`. + +A raw concentration index is not comparable across different prevalences, so +`erreygers_index` and `wagstaff_index` exist; they answer different normative +questions and can disagree about the direction of change over time. Pick one per +table and say which. + +**Do not add an analytic standard error to these.** The convenient regression +form people quote ignores the survey design entirely. Bootstrap over PSUs within +strata; `inequality/README.md` carries the recipe. + +Tests are pinned to closed-form identities, never to previous output: the Gini +against brute-force mean absolute difference on random data, curve areas against +their indices by trapezoid, `within + between == total` for both GE(0) and +GE(1), Oaxaca's components summing to the gap under all four reference choices. + ## survey_estimation The one part worth reading before touching. It implements Taylor linearisation diff --git a/README.md b/README.md index 2e34e52..798b0ee 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE) [![Status: Stable](https://img.shields.io/badge/Status-Stable-0969da?style=flat-square)](https://github.com/Varnasr/OpenStacks-for-Change/blob/main/MAINTENANCE.md) -> Plug-and-play templates for health, gender, education, and climate equity data — built for reproducibility. +> Distributional analysis and survey estimation for health, gender, education and climate equity data. > **Status: Stable.** This repository works and is correct, but it is not under active > development. Bug reports are welcome and issues stay open; new features are unlikely, @@ -21,6 +21,40 @@ EquityStack is a collection of **Python scripts, Jupyter notebooks, and sample d This is the **data pipeline layer** of [OpenStacks for Change](https://openstacks.dev) — an open ecosystem of tools for public interest research and evaluation. +## The two modules to read first + +Most of this repository is technique on stand-in data. Two parts are not, and +they are what the name is about. + +**`inequality/`** answers *who has it*. The Gini says how unequally consumption +is spread; the concentration index says whether stunting falls on the poor, by +how much, and comparably across states whose prevalence differs. It also +decomposes: `theil_decomposition` splits national inequality exactly into a +within-state and a between-state part, and `oaxaca_blinder` splits a group gap +into endowments and returns. `benefit_incidence` asks who actually receives a +public budget, which for tertiary health and higher education is usually not the +people it was voted for. + +**`survey_estimation/`** answers *how sure are we*. Weighting is the half +everyone remembers; a national household survey is also clustered, and an +interval that ignores that is too narrow, often by half, erring in the direction +that flatters the result. + +Together they are the two halves of an equity finding: the gradient, and whether +it is real. + +```python +from inequality import concentration_index, theil_decomposition +from survey_estimation import svy_prop_by + +svy_prop_by(df, "stunted", by="wealth_quintile") # is the gap real? +concentration_index(df.stunted, rank_by=df.wealth_index) # how steep is it? +theil_decomposition(df.consumption, groups=df.state) # where does it sit? +``` + +`inequality/README.md` has the full function-by-function guide, the four things +that are easy to get wrong, and how to bootstrap a standard error over PSUs. + ## What's Inside ### Core Modules @@ -33,7 +67,8 @@ This is the **data pipeline layer** of [OpenStacks for Change](https://openstack | `io_helpers/` | Chunked CSV reading, Stata/SPSS import, formatted Excel export | Ready | | `modelling/` | Multicollinearity checks (VIF) | Ready | | `visualisation/` | Annotated bar charts, district-level choropleth maps | Ready | -| `social_sector/` | Public health access index | Ready | +| `inequality/` | **Distributional analysis**: Gini, Theil, Atkinson, Palma, Lorenz and concentration curves, the concentration index with Erreygers and Wagstaff corrections, Theil within/between decomposition, Blinder-Oaxaca, benefit incidence | Ready | +| `social_sector/` | Composite indices with explicit direction, normalisation and weighting, plus a rank-sensitivity check | Ready | | `survey_estimation/` | Design-based proportions and means for stratified, clustered surveys, with a worked NFHS-5 example | Ready | ### Notebooks diff --git a/index.html b/index.html index 417e30e..3834b16 100644 --- a/index.html +++ b/index.html @@ -17,8 +17,9 @@ @@ -26,20 +27,72 @@

EquityStack

-

The Python you reach for on a development research project: getting messy data into - shape, asking whether a programme worked, and getting the confidence intervals right - when the survey is clustered. Part of OpenStacks.

+

The Python for a development research project, pointed at the distributional + question: not just what the average is, but who is carrying the bad outcome, how steep the + gradient runs, and whether the gap survives a clustered sample. + Part of OpenStacks.

-
9Modules
-
34Python files
-
2Notebooks
+
24Distributional measures
+
10Modules
4Impact evaluation methods
+
101Tests
-
+
+
+

Distributional analysis

+

Who is carrying it, and how steeply

+
+
+ +

inequality/

+

The concentration index

+

An average tells you the level. It cannot tell you whether stunting falls on the poor, + which is the question a programme budget answers to. The concentration index does, on a + scale where negative means concentrated among the poor, and it works off a wealth + quintile as readily as a continuous consumption measure.

+

Read the module →

+
+ +

Comparability

+

Why the raw index misleads

+

A concentration index is not comparable between two places whose prevalence differs, + because its possible range shrinks as the mean moves away from half. Comparing states or + years without correcting for that is a standard error in published work. Both fixes are + here, and they can disagree about which way things moved.

+

Read the guide →

+
+ +

Decomposition

+

Where the inequality sits

+

A national Gini of 0.35 is equally consistent with identical states that differ from each + other and with unequal states that look alike on average. Those want opposite responses. + The Theil split answers it exactly, and Blinder-Oaxaca separates how much of a group gap + is land and schooling from how much is the return on them.

+

Read the module →

+
+ +

Benefit incidence

+

Who receives the budget

+

Public tertiary hospitals and universities are free at the point of use and mostly used by + the better-off, so the spending is regressive whatever the intention. Splitting a budget + by level of service usually shows primary care progressive and tertiary regressive, the + two cancelling in any aggregate figure.

+

Read the module →

+
+
+

Twenty-four measures: Gini, Theil, Atkinson, Palma, the 80/20 ratio, Lorenz and + concentration curves, quantile shares, the Erreygers and Wagstaff corrections, within and + between decomposition, Blinder-Oaxaca, benefit incidence. All survey-weighted. Verified + against identities rather than against previous output, because 0.31 and 0.34 are both + plausible Ginis and a sign error gives a number in the right range that says the opposite of + the truth.

+
+ + -