From c142982e3cb5f3aabf8ba375245f09ffe5859828 Mon Sep 17 00:00:00 2001 From: vahid-ahmadi Date: Fri, 23 Oct 2026 13:21:06 +0100 Subject: [PATCH] Document building the FRS from a UK Data Service download Say which FRS release main supports, that the PyPI package is stale, how to build the base FRS from the UKDS tab files, how to run the result in policyengine.py, and which FRS codes changed since 2022-23. Co-Authored-By: Claude Opus 5.5 --- README.md | 91 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 91 insertions(+) diff --git a/README.md b/README.md index 4f6de4a32..00786240f 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,97 @@ If `../policyengine-uk` exists, you can run: make data-local ``` +## Building the FRS from your own UK Data Service download + +If you hold a UK Data Service (UKDS) licence for the Family Resources Survey, +you can build PolicyEngine's base FRS dataset from the raw tables yourself. + +**Supported release.** `main` builds FRS 2024-25 only (UKDS SN 9563, pinned as +`CURRENT_FRS_RELEASE` in `policyengine_uk_data/datasets/frs_release.py`). The +build selects columns and thresholds reported benefits by survey year, and no +test runs it on another release. + +**Install from GitHub, not PyPI.** At the time of writing the +`policyengine-uk-data` package on PyPI is version 1.11.1, from December 2024. +It knows FRS releases up to 2022-23 only and does not map codes that later +releases use. Clone this repository instead: + +```sh +git clone https://github.com/PolicyEngine/policyengine-uk-data +cd policyengine-uk-data +uv sync --extra dev # Python 3.13 +``` + +**Build.** + +1. Download the FRS 2024-25 TAB-format zip from the UKDS (SN 9563). +2. Check it is the release the build pins: `shasum -a 256` of the zip should + equal `ukds_tab_zip_sha256` in `frs_release.py`. A different digest means a + re-issued release, so check the "NEW FOR" sheet of its variable listing + before relying on the mappings. +3. Extract the `.tab` files under `UKDA-9563-tab/tab/` into + `policyengine_uk_data/storage/frs_2024_25/`. The folder name sets the survey + year, and `create_frs` refuses a `year` that does not match it. +4. Run `uv run python -m policyengine_uk_data.datasets.frs`. This writes + `policyengine_uk_data/storage/frs_2024_25.h5`. From Python: + + ```python + from policyengine_uk_data.datasets.frs import create_frs + + frs = create_frs( + "policyengine_uk_data/storage/frs_2024_25", year=2024, policy_year=2025 + ) + frs.save("frs_2024_25.h5") + ``` + +**What you get.** The base FRS with DWP's grossing weights. It has none of the +enhanced FRS imputations (wealth, consumption, VAT, top incomes from the SPI) +and no calibration to administrative totals, so its aggregates differ from the +figures PolicyEngine publishes, which use the enhanced FRS and Microcosm +datasets. UKDS licence holders can request access to those through the +[policyengine-uk getting-started guide](https://github.com/PolicyEngine/policyengine-uk/blob/main/docs/book/usage/getting-started.md#running-microsimulation-analyses). + +**Running it.** A local file bypasses `policyengine.py`'s certified release +bundle, so pass `allow_unmanaged=True`: + +```python +import policyengine as pe + +datasets = pe.uk.ensure_datasets( + datasets=["/path/to/frs_2024_25.h5"], + years=[2026], + data_folder="./data", + allow_unmanaged=True, +) +``` + +**FRS codes that changed since 2022-23.** The UKDS variable listings for +2022-23 (SN 9252), 2023-24 (SN 9367) and 2024-25 (SN 9563) give the same codes +for region (GVTREGNO), tenure (PTENTYP2), accommodation type (TYPEACC), marital +status (MARITAL), ILO employment status (EMPSTATI) and school type (TYPEED2). +The differences that matter here: + +- **Council tax band (CTBAND).** From 2024-25 DWP takes the band from Valuation + Office Agency and Scottish Assessors Association records rather than asking + the respondent. Code 10, now "Information not found" (before 2024-25 + "Household not valued separately"), gives no band, and the build assigns + band D, the policyengine-uk default. Code 9 is band I, now labelled "Wales + only". +- **Benefit codes (BENEFITS.BENEFIT).** Codes 69 and 70, recoveries and Social + Fund repayments, are new in 2024-25 and are not read. Scottish Adult and + Child Disability Payment (117, 118, 121, 122) are not read yet: see + [#500](https://github.com/PolicyEngine/policyengine-uk-data/issues/500). +- **Benefit receipt.** DWP now bases receipt of the main benefits on + administrative data, so reported receipt is higher than in earlier releases + ([FRS 2024-25 background information and methodology](https://www.gov.uk/government/statistics/family-resources-survey-financial-year-2024-to-2025/family-resources-survey-background-information-and-methodology)). +- **Highest qualification (EDUCQUAL).** Its labels were reworded in 2024-25 + with the same codes. The mapping to `highest_education` does not yet follow + the data dictionary: see + [#459](https://github.com/PolicyEngine/policyengine-uk-data/issues/459). +- **Scottish sewerage charge (CSEWAMT).** The column is empty in 2024-25, so + the build sets Scottish water and sewerage charges to zero: see + [#467](https://github.com/PolicyEngine/policyengine-uk-data/issues/467). + ## Public UK calibrated transfer dataset This repo now also includes a public calibrated microdata file: