Skip to content

Add the inequality package, so the repository's name is earned #24

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Varnasr merged 1 commit into from
Sep 8, 2026
Merged

Add the inequality package, so the repository's name is earned #24

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 41 additions & 1 deletion CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,18 @@ Python scripts and notebooks for development sector data workflows. Part of the
## Layout

`cleaning/`, `eda/`, `modelling/`, `validation/`, `io_helpers/`,
`impact_evaluation/`, `social_sector/`, `visualisation/`, plus
`impact_evaluation/`, `social_sector/`, `visualisation/`, plus the two that
carry the repository: `inequality/` for distributional analysis and
`survey_estimation/` for design-based estimates from complex surveys.
Notebooks in `notebooks/`, small CSVs in `sample_data/`, tests in `tests/`.

Sizes, so nobody has to guess again: 101 tests, and `inequality/` (about 950
lines) plus `survey_estimation/` (357) plus `impact_evaluation/` (220) are most
of the substance. The rest is technique taught against stand-in data, and some
of it is a handful of lines. That is a deliberate split, not neglect, but do not
describe a five-line snippet folder as a module in any user-facing copy: an
audit on 2026-09-08 found the landing page doing exactly that.

House style is numpydoc docstrings, one module per group of related functions,
pytest with plain asserts. Match it.

Expand Down Expand Up @@ -43,6 +51,38 @@ pip install -r requirements.txt
PYTHONPATH=$(pwd) pytest tests/
```

## inequality

The package that makes the repository's name true. Everything in it is a
covariance between an outcome and a position in a distribution, and positions
are where the errors hide, so three things are centralised rather than
reimplemented per function.

- **Weighted fractional ranks** live in `inequality/ranks.py` and are shared by
the Gini, the concentration index and both curves, so those three agree by
construction. Ties take the block midpoint, which is what the World Bank's DHS
equity work does and what makes a wealth quintile a legitimate rank variable.
- **The mean fractional rank is exactly 0.5** for any weights and any pattern of
ties. Three derivations depend on it and a test asserts it.
- **Sign convention: negative means concentrated among the poor.** Reversing
`outcome` and `rank_by` returns a number in the right range with the opposite
meaning and nothing errors, which is why the signature is
`concentration_index(outcome, rank_by=...)`.

A raw concentration index is not comparable across different prevalences, so
`erreygers_index` and `wagstaff_index` exist; they answer different normative
questions and can disagree about the direction of change over time. Pick one per
table and say which.

**Do not add an analytic standard error to these.** The convenient regression
form people quote ignores the survey design entirely. Bootstrap over PSUs within
strata; `inequality/README.md` carries the recipe.

Tests are pinned to closed-form identities, never to previous output: the Gini
against brute-force mean absolute difference on random data, curve areas against
their indices by trapezoid, `within + between == total` for both GE(0) and
GE(1), Oaxaca's components summing to the gap under all four reference choices.

## survey_estimation

The one part worth reading before touching. It implements Taylor linearisation
Expand Down
39 changes: 37 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
[![Status: Stable](https://img.shields.io/badge/Status-Stable-0969da?style=flat-square)](https://github.com/Varnasr/OpenStacks-for-Change/blob/main/MAINTENANCE.md)

> Plug-and-play templates for health, gender, education, and climate equity data — built for reproducibility.
> Distributional analysis and survey estimation for health, gender, education and climate equity data.

> **Status: Stable.** This repository works and is correct, but it is not under active
> development. Bug reports are welcome and issues stay open; new features are unlikely,
Expand All @@ -21,6 +21,40 @@ EquityStack is a collection of **Python scripts, Jupyter notebooks, and sample d

This is the **data pipeline layer** of [OpenStacks for Change](https://openstacks.dev) — an open ecosystem of tools for public interest research and evaluation.

## The two modules to read first

Most of this repository is technique on stand-in data. Two parts are not, and
they are what the name is about.

**`inequality/`** answers *who has it*. The Gini says how unequally consumption
is spread; the concentration index says whether stunting falls on the poor, by
how much, and comparably across states whose prevalence differs. It also
decomposes: `theil_decomposition` splits national inequality exactly into a
within-state and a between-state part, and `oaxaca_blinder` splits a group gap
into endowments and returns. `benefit_incidence` asks who actually receives a
public budget, which for tertiary health and higher education is usually not the
people it was voted for.

**`survey_estimation/`** answers *how sure are we*. Weighting is the half
everyone remembers; a national household survey is also clustered, and an
interval that ignores that is too narrow, often by half, erring in the direction
that flatters the result.

Together they are the two halves of an equity finding: the gradient, and whether
it is real.

```python
from inequality import concentration_index, theil_decomposition
from survey_estimation import svy_prop_by

svy_prop_by(df, "stunted", by="wealth_quintile") # is the gap real?
concentration_index(df.stunted, rank_by=df.wealth_index) # how steep is it?
theil_decomposition(df.consumption, groups=df.state) # where does it sit?
```

`inequality/README.md` has the full function-by-function guide, the four things
that are easy to get wrong, and how to bootstrap a standard error over PSUs.

## What's Inside

### Core Modules
Expand All @@ -33,7 +67,8 @@ This is the **data pipeline layer** of [OpenStacks for Change](https://openstack
| `io_helpers/` | Chunked CSV reading, Stata/SPSS import, formatted Excel export | Ready |
| `modelling/` | Multicollinearity checks (VIF) | Ready |
| `visualisation/` | Annotated bar charts, district-level choropleth maps | Ready |
| `social_sector/` | Public health access index | Ready |
| `inequality/` | **Distributional analysis**: Gini, Theil, Atkinson, Palma, Lorenz and concentration curves, the concentration index with Erreygers and Wagstaff corrections, Theil within/between decomposition, Blinder-Oaxaca, benefit incidence | Ready |
| `social_sector/` | Composite indices with explicit direction, normalisation and weighting, plus a rank-sensitivity check | Ready |
| `survey_estimation/` | Design-based proportions and means for stratified, clustered surveys, with a worked NFHS-5 example | Ready |

### Notebooks
Expand Down
81 changes: 67 additions & 14 deletions index.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,29 +17,82 @@
<div class="in">
<a class="brand" href="./"><span class="dot"></span> EquityStack</a>
<nav>
<a class="opt" href="#equity">Equity</a>
<a class="opt" href="#survey">Survey</a>
<a class="opt" href="#modules">Modules</a>
<a class="opt" href="#survey">Survey estimation</a>
<a href="https://github.com/Varnasr/EquityStack">GitHub</a>
</nav>
</div>
</header>

<section class="hero">
<h1 class="word">Equity<span class="s2">Stack</span></h1>
<p class="lede">The Python you reach for on a development research project: getting messy data into
shape, asking whether a programme worked, and getting the confidence intervals right
when the survey is clustered. Part of <a href="https://openstacks.dev">OpenStacks</a>.</p>
<p class="lede">The Python for a development research project, pointed at the distributional
question: not just what the average is, but who is carrying the bad outcome, how steep the
gradient runs, and whether the gap survives a clustered sample.
Part of <a href="https://openstacks.dev">OpenStacks</a>.</p>
<div class="counts">
<div class="hi"><b>9</b><span>Modules</span></div>
<div><b>34</b><span>Python files</span></div>
<div><b>2</b><span>Notebooks</span></div>
<div class="hi"><b>24</b><span>Distributional measures</span></div>
<div><b>10</b><span>Modules</span></div>
<div><b>4</b><span>Impact evaluation methods</span></div>
<div><b>101</b><span>Tests</span></div>
</div>
</section>

<main id="main">

<section class="band white" id="survey">
<section class="band white" id="equity">
<div class="band-h">
<h2>Distributional analysis</h2>
<p class="sub">Who is carrying it, and how steeply</p>
</div>
<div class="tiles">
<a class="tile" href="https://github.com/Varnasr/EquityStack/tree/main/inequality">
<p class="meta">inequality/</p>
<h3>The concentration index</h3>
<p>An average tells you the level. It cannot tell you whether stunting falls on the poor,
which is the question a programme budget answers to. The concentration index does, on a
scale where negative means concentrated among the poor, and it works off a wealth
quintile as readily as a continuous consumption measure.</p>
<p class="go">Read the module &rarr;</p>
</a>
<a class="tile" href="https://github.com/Varnasr/EquityStack/blob/main/inequality/README.md">
<p class="meta">Comparability</p>
<h3>Why the raw index misleads</h3>
<p>A concentration index is not comparable between two places whose prevalence differs,
because its possible range shrinks as the mean moves away from half. Comparing states or
years without correcting for that is a standard error in published work. Both fixes are
here, and they can disagree about which way things moved.</p>
<p class="go">Read the guide &rarr;</p>
</a>
<a class="tile" href="https://github.com/Varnasr/EquityStack/blob/main/inequality/decomposition.py">
<p class="meta">Decomposition</p>
<h3>Where the inequality sits</h3>
<p>A national Gini of 0.35 is equally consistent with identical states that differ from each
other and with unequal states that look alike on average. Those want opposite responses.
The Theil split answers it exactly, and Blinder-Oaxaca separates how much of a group gap
is land and schooling from how much is the return on them.</p>
<p class="go">Read the module &rarr;</p>
</a>
<a class="tile" href="https://github.com/Varnasr/EquityStack/blob/main/inequality/incidence.py">
<p class="meta">Benefit incidence</p>
<h3>Who receives the budget</h3>
<p>Public tertiary hospitals and universities are free at the point of use and mostly used by
the better-off, so the spending is regressive whatever the intention. Splitting a budget
by level of service usually shows primary care progressive and tertiary regressive, the
two cancelling in any aggregate figure.</p>
<p class="go">Read the module &rarr;</p>
</a>
</div>
<p class="note">Twenty-four measures: Gini, Theil, Atkinson, Palma, the 80/20 ratio, Lorenz and
concentration curves, quantile shares, the Erreygers and Wagstaff corrections, within and
between decomposition, Blinder-Oaxaca, benefit incidence. All survey-weighted. Verified
against identities rather than against previous output, because 0.31 and 0.34 are both
plausible Ginis and a sign error gives a number in the right range that says the opposite of
the truth.</p>
</section>

<section class="band navy" id="survey">
<div class="band-h">
<h2>Survey estimation</h2>
<p class="sub">The part worth reading first</p>
Expand Down Expand Up @@ -70,12 +123,15 @@ <h3>NFHS-5 stunting, reproduced</h3>
bound, and the interval widens correctly when you have few clusters.</p>
</section>

<section class="band navy" id="modules">
<section class="band white" id="modules">
<div class="band-h">
<h2>Modules</h2>
<p class="sub">Nine, plus notebooks and sample data</p>
<p class="sub">Ten, plus notebooks and sample data</p>
</div>
<div class="rows">
<a class="row" href="https://github.com/Varnasr/EquityStack/tree/main/inequality">
<span class="t"><b>Inequality</b><span>Python</span></span>
<span class="d">Gini, Theil, Atkinson, Palma, concentration indices, decompositions and benefit incidence, all survey-weighted.</span></a>
<a class="row" href="https://github.com/Varnasr/EquityStack/tree/main/cleaning">
<span class="t"><b>Cleaning</b><span>Python</span></span>
<span class="d">Get a messy export into shape, and keep a record of what you changed.</span></a>
Expand All @@ -99,7 +155,7 @@ <h2>Modules</h2>
<span class="d">Open a Stata or SPSS file, read a CSV too big for memory, hand a colleague a usable spreadsheet.</span></a>
<a class="row" href="https://github.com/Varnasr/EquityStack/tree/main/social_sector">
<span class="t"><b>Social sector</b><span>Python</span></span>
<span class="d">Build a health access index and compare districts on it.</span></a>
<span class="d">Build an access or vulnerability index, and check whether a district's rank survives a different but equally defensible normalisation.</span></a>
<a class="row" href="https://github.com/Varnasr/EquityStack/tree/main/visualisation">
<span class="t"><b>Visualisation</b><span>Python</span></span>
<span class="d">District maps and annotated charts that can go straight into a report.</span></a>
Expand All @@ -109,9 +165,6 @@ <h2>Modules</h2>
<a class="row" href="https://github.com/Varnasr/EquityStack/tree/main/sample_data">
<span class="t"><b>Sample data</b><span>CSV</span></span>
<span class="d">Small datasets to try any of the above on.</span></a>
<a class="row" href="https://github.com/Varnasr/EquityStack/tree/main/workflows">
<span class="t"><b>Workflows</b><span>Guides</span></span>
<span class="d">Start-to-finish routes through an analysis, so you know what comes next.</span></a>
</div>
<p class="note">These open on GitHub, where the code lives.</p>
</section>
Expand Down
Loading