Skip to content

Correct the default-dataset switch docs page (#1692) - #1718

Merged
vahid-ahmadi merged 7 commits into
mainfrom
vahid-ahmadi/issue-1692-mp30k-switch-plan
Oct 24, 2026
Merged

vahid-ahmadi merged 7 commits into
mainfrom
vahid-ahmadi/issue-1692-mp30k-switch-plan

Conversation

@vahid-ahmadi

@vahid-ahmadi vahid-ahmadi commented May 27, 2026 •

Copy link
Copy Markdown
Collaborator

Corrects the default-dataset switch and rollback planning page (docs/book/assumptions/mp30k-uk-switch-plan.md), written for #1692.

The page itself is already on main: it shipped in #2386 (merged 21 October 2026), and its original .added changelog entry was released in 2.143.0. This branch has merged main, dropped the released fragment and now carries only corrections to main's copy of the page.

What this PR corrects (from its changelog fragment):

Corrected the default-dataset switch plan (docs/book/assumptions/mp30k-uk-switch-plan.md): the tests now pin Enhanced FRS 2024-25 from policyengine-uk-data 1.58.1, policyengine.py 6.10.0 already switched its UK default to Microcosm UK, an inline situation takes precedence over any dataset, unknown enum names raise an error, and the VAT survey-coverage parameter is calibrated to Microcosm UK and must be re-derived on a switch or rollback.

The claim-by-claim review behind these corrections, with sources, is in the "Ready for hub review" comment on this PR.

  • Changelog fragment: changelog.d/1692-dataset-switch-page-corrections.changed.md (changed).
  • Docs and changelog only; no code changes.

🤖 Generated with Claude Code

@vahid-ahmadi
vahid-ahmadi force-pushed the vahid-ahmadi/issue-1692-mp30k-switch-plan branch from 4a46c65 to 4445482 Compare October 8, 2026 14:54
@vahid-ahmadi vahid-ahmadi changed the title Add mp-30k-uk switch and rollback planning doc (#1692) Add default-dataset switch and rollback planning doc (#1692) Oct 8, 2026
vahid-ahmadi and others added 3 commits October 12, 2026 11:02
#1692 asks for a documented model-side path to switch the default
dataset from the enhanced FRS to Microplex UK (mp-30k-uk), with an
explicit rollback path. Persist that plan in-repo at
docs/book/assumptions/mp30k-uk-switch-plan.md.

The page covers:

- the current state: the dataset URL is resolved at runtime
  (constructor arg -> POLICYENGINE_UK_DEFAULT_DATASET env var ->
  ValueError), so the model is already dataset-agnostic and the
  switch is a configuration concern rather than a code change,
- loader-side assumptions to audit before switching (HDF5 layout
  matching UKMultiYearDataset, advertised years, enum label parity),
- variable-side audit needs (input/* columns present in both
  datasets; reported-vs-derived benefit columns implying take-up
  recalibration),
- calibration-side implications for policyengine-uk-data targets,
- five gating criteria (aggregate parity, cross-section parity,
  reform parity, full test suite, documentation parity),
- a rollback path that requires no code change in this repo — just
  reverting the env var per deploy environment and keeping eFRS
  published in parallel during beta.
…elog fragment

Checked every variable, parameter, path and link against main in October
2026, updated the data pipeline references to Microcosm UK, and renamed the
changelog fragment to a Towncrier type.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop the unverified Microcosm UK release date and the Microplex archive
month (the repository is archived; the date is not recorded), keep the
national file name, which Microcosm's runbook and registry use. Note that
Microsimulation() here is deprecated in favour of policyengine.py's
managed bundle (README), so the default switch has to happen there too.
Say that _pre_encode_enum_columns lives in simulation.py, not
dataset_schema.py. Drop the leading bullet from the changelog fragment
so towncrier does not render it twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vahid-ahmadi
vahid-ahmadi force-pushed the vahid-ahmadi/issue-1692-mp30k-switch-plan branch from 4445482 to 118b1c9 Compare October 12, 2026 09:10
MaxGhenis and others added 4 commits October 23, 2026 02:03
Main already carries this page (added by #2386); the merge brings the
branch up to date before the page is corrected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- An inline situation is used before dataset= and the environment variable
  (simulation.py:349-362).
- Tests now pin enhanced_frs_2024_25.h5 from policyengine-uk-data 1.58.1
  by commit and SHA-256 (tests/certified_dataset.yaml), not the 2023-24
  1.40.3 file, so both datasets share the 2024-25 base year.
- policyengine.py 6.10.0 (21 October 2026) switched its UK default to
  Microcosm UK 2024-25; only this repository's tests remain on the eFRS.
- Enum columns are matched by member name and an unknown name raises a
  ValueError (core Enum.encode), rather than encoding incorrectly.
- The loader sets any dataset column named after a variable, so
  variables/input/ is not the whole data contract.
- Rollback is not code-free: gov.simulation.vat.survey_consumption_coverage
  is calibrated to Microcosm UK only and must be re-derived.
- Replaced the released .added fragment with a .changed fragment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The tests set the variable only when it is not already set (conftest.py).
- policyengine.py switched under its own certification, with a recorded
  UC caseload-floor exception; the page no longer implies the gates ran,
  and no longer claims nothing else downstream uses the eFRS.
- test_latest_data_smoke.py still loads enhanced_frs_2023_24.h5, and the
  three named tests do not exercise a new default; criterion 4 now names
  the certified pin and the reform regression fixtures.
- variables/input/ holds many, not most, of the dataset inputs.
- The VAT coverage parameter matches one Microcosm cut (20261002), not the
  cut policyengine.py now pins; 38% and 44% are two uk-data releases.
- Rollback: the tests' default is a committed pin, and policyengine.py's
  default comes from its manifest, not the environment variable.
- An hf:// path cannot load Microcosm UK, which sits in a dataset repo
  while core requests repo_type="model".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Second-round review: materialise() in utils/aggregate_impact/dataset.py
downloads the pinned file with repo_type="model" and load_pin() accepts no
repository-type key, so criterion 4 cannot pin a Microcosm file without a
loader change. Also corrected the description of the three named tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Collaborator

UK hub (Claude build session, for Max)
Ready for hub review

Head: 20cad2fc6d34b1b82c7b352b7294bc0414886a46

Context. This page already reached main, byte for byte, through #2386 (merged 21 October 2026; batch commit d50ee85), and its 1692.added.md entry is already in CHANGELOG.md under 2.143.0. I merged origin/main into this branch (no rebase, no force-push), deleted the released .added fragment, and corrected the page. The PR's net diff against main is now just those corrections plus changelog.d/1692-dataset-switch-page-corrections.changed.md. Suggested title: "Correct the default-dataset switch plan".

Is the plan still worth having? Partly overtaken: policyengine.py 6.10.0 (21 October 2026) already switched its UK default to Microcosm UK. The page now says so, and says it doesn't record whether these gates were run for that switch. The plan still applies to switching this repository's test pin, which stays on the Enhanced FRS. I recommend keeping it.

Review. I had a verifier extract and check every claim, checked each correction myself, and then ran independent adversarial reviews. Rounds 1 and 2 asked for changes, and round 3 approved; its one optional precision point is folded into this head.

Corrected (13)

  1. Resolution order: an inline situation is used before dataset= and the environment variable; otherwise get_default_dataset_url raises a ValueError (policyengine_uk/simulation.py:181-189,349-363).
  2. The test pin is enhanced_frs_2024_25.h5 from policyengine-uk-data 1.58.1, pinned by commit and SHA-256 in policyengine_uk/tests/certified_dataset.yaml, not the 2023-24 1.40.3 file.
  3. The tests set the environment variable only when a token is present and the variable isn't already set (policyengine_uk/tests/conftest.py:118-121).
  4. policyengine.py 6.10.0 switched its UK default to microcosm_uk_2024_25 (its CHANGELOG and manifest). It did so under its own certification, which records an accepted UC caseload-floor exception.
  5. Base years now match: both are 2024-25.
  6. String enum columns are matched by member name, and an unknown name raises a ValueError (policyengine-core enums/enum.py:87-107). Integer columns are not checked (simulation.py:215-219).
  7. variables/input/ holds many inputs, but the loader sets any column named after a variable (build_from_multi_year_dataset, simulation.py:1189-1197). Examples defined elsewhere: tenure_type, household_weight, would_claim_uc, *_reported.
  8. Criterion 4: test_latest_data_smoke.py:29-31 loads a hard-coded, unpinned enhanced_frs_2023_24.h5, and the other two named tests use inline situations or a hand-built dataset. The criterion now names the pin and the reform fixtures (tests/microsimulation/reforms_config.yaml).
  9. Both hf:// loading (core tools/hugging_face.py:139) and the pin's materialise (utils/aggregate_impact/dataset.py:155-159) request model repositories, and load_pin accepts no repository-type key. Microcosm sits in a dataset repo, so pinning it needs a loader change.
  10. Rollback is not code-free: gov.simulation.vat.survey_consumption_coverage is calibrated to one Microcosm cut (20261002T230158Z-5c6b3f68). On the Enhanced FRS it puts VAT 38% (uk-data 1.56.16) or 44% (1.58.0) above receipts (parameters/gov/simulation/vat/survey_consumption_coverage.yaml).
  11. Per-environment defaults: the env var for direct use, the committed pin for tests, and the bundle manifest for policyengine.py.
  12. Rollback steps and the Acceptance text now include re-pinning and re-deriving dataset-calibrated parameters.
  13. Replaced the released .added fragment with a .changed fragment.

Checked and correct (unchanged)

Uncertain

  • That hf:// cannot load a dataset-repo file is deduced from the code. I didn't run it.

@vahid-ahmadi vahid-ahmadi changed the title Add default-dataset switch and rollback planning doc (#1692) Correct the default-dataset switch docs page (#1692) Oct 23, 2026
@vahid-ahmadi
vahid-ahmadi added this pull request to the merge queue Oct 23, 2026
Merged via the queue into main with commit 3718578 Oct 24, 2026
10 checks passed
@vahid-ahmadi
vahid-ahmadi deleted the vahid-ahmadi/issue-1692-mp30k-switch-plan branch October 24, 2026 07:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants