Document building the FRS from a UK Data Service download

README.md

@@ -16,6 +16,97 @@ If `../policyengine-uk` exists, you can run:
 make data-local
 ```
 
+## Building the FRS from your own UK Data Service download
+
+If you hold a UK Data Service (UKDS) licence for the Family Resources Survey,
+you can build PolicyEngine's base FRS dataset from the raw tables yourself.
+
+**Supported release.** `main` builds FRS 2024-25 only (UKDS SN 9563, pinned as
+`CURRENT_FRS_RELEASE` in `policyengine_uk_data/datasets/frs_release.py`). The
+build selects columns and thresholds reported benefits by survey year, and no
+test runs it on another release.
+
+**Install from GitHub, not PyPI.** At the time of writing the
+`policyengine-uk-data` package on PyPI is version 1.11.1, from December 2024.
+It knows FRS releases up to 2022-23 only and does not map codes that later
+releases use. Clone this repository instead:
+
+```sh
+git clone https://github.com/PolicyEngine/policyengine-uk-data
+cd policyengine-uk-data
+uv sync --extra dev  # Python 3.13
+```
+
+**Build.**
+
+1. Download the FRS 2024-25 TAB-format zip from the UKDS (SN 9563).
+2. Check it is the release the build pins: `shasum -a 256` of the zip should
+   equal `ukds_tab_zip_sha256` in `frs_release.py`. A different digest means a
+   re-issued release, so check the "NEW FOR" sheet of its variable listing
+   before relying on the mappings.
+3. Extract the `.tab` files under `UKDA-9563-tab/tab/` into
+   `policyengine_uk_data/storage/frs_2024_25/`. The folder name sets the survey
+   year, and `create_frs` refuses a `year` that does not match it.
+4. Run `uv run python -m policyengine_uk_data.datasets.frs`. This writes
+   `policyengine_uk_data/storage/frs_2024_25.h5`. From Python:
+
+   ```python
+   from policyengine_uk_data.datasets.frs import create_frs
+
+   frs = create_frs(
+       "policyengine_uk_data/storage/frs_2024_25", year=2024, policy_year=2025
+   )
+   frs.save("frs_2024_25.h5")
+   ```
+
+**What you get.** The base FRS with DWP's grossing weights. It has none of the
+enhanced FRS imputations (wealth, consumption, VAT, top incomes from the SPI)
+and no calibration to administrative totals, so its aggregates differ from the
+figures PolicyEngine publishes, which use the enhanced FRS and Microcosm
+datasets. UKDS licence holders can request access to those through the
+[policyengine-uk getting-started guide](https://github.com/PolicyEngine/policyengine-uk/blob/main/docs/book/usage/getting-started.md#running-microsimulation-analyses).
+
+**Running it.** A local file bypasses `policyengine.py`'s certified release
+bundle, so pass `allow_unmanaged=True`:
+
+```python
+import policyengine as pe
+
+datasets = pe.uk.ensure_datasets(
+    datasets=["/path/to/frs_2024_25.h5"],
+    years=[2026],
+    data_folder="./data",
+    allow_unmanaged=True,
+)
+```
+
+**FRS codes that changed since 2022-23.** The UKDS variable listings for
+2022-23 (SN 9252), 2023-24 (SN 9367) and 2024-25 (SN 9563) give the same codes
+for region (GVTREGNO), tenure (PTENTYP2), accommodation type (TYPEACC), marital
+status (MARITAL), ILO employment status (EMPSTATI) and school type (TYPEED2).
+The differences that matter here:
+
+- **Council tax band (CTBAND).** From 2024-25 DWP takes the band from Valuation
+  Office Agency and Scottish Assessors Association records rather than asking
+  the respondent. Code 10, now "Information not found" (before 2024-25
+  "Household not valued separately"), gives no band, and the build assigns
+  band D, the policyengine-uk default. Code 9 is band I, now labelled "Wales
+  only".
+- **Benefit codes (BENEFITS.BENEFIT).** Codes 69 and 70, recoveries and Social
+  Fund repayments, are new in 2024-25 and are not read. Scottish Adult and
+  Child Disability Payment (117, 118, 121, 122) are not read yet: see
+  [#500](https://github.com/PolicyEngine/policyengine-uk-data/issues/500).
+- **Benefit receipt.** DWP now bases receipt of the main benefits on
+  administrative data, so reported receipt is higher than in earlier releases
+  ([FRS 2024-25 background information and methodology](https://www.gov.uk/government/statistics/family-resources-survey-financial-year-2024-to-2025/family-resources-survey-background-information-and-methodology)).
+- **Highest qualification (EDUCQUAL).** Its labels were reworded in 2024-25
+  with the same codes. The mapping to `highest_education` does not yet follow
+  the data dictionary: see
+  [#459](https://github.com/PolicyEngine/policyengine-uk-data/issues/459).
+- **Scottish sewerage charge (CSEWAMT).** The column is empty in 2024-25, so
+  the build sets Scottish water and sewerage charges to zero: see
+  [#467](https://github.com/PolicyEngine/policyengine-uk-data/issues/467).
+
 ## Public UK calibrated transfer dataset
 
 This repo now also includes a public calibrated microdata file: