Repository navigation
Align dataset group inputs with their identifiers - #2449
Merged
Merged
Conversation
vahid-ahmadi
left a comment
Collaborator
There was a problem hiding this comment.
Looks right, fine to approve.
@juaristi22, this fixes #2278 on every loader I traced: build_from_multi_year_dataset (so single-year datasets, local H5 files, hf:// and gs:// sources too), build_from_dataset for Core datasets, and the frozen-assumptions copy. The caller's tables stay unchanged. One hardening suggestion and a merge note.
- should,
policyengine_uk/simulation.py:1046-1050and:964-969: the alignment sorts each year's group table by its own IDs and assumes those IDs are the population's. If a later year lists a different set (for example household 4 where the base year has 3), the rows still shift silently, which is the same class of error as #2278. A check thatnp.sort(ids)equalsself.populations[entity].idsfor each year, raising aValueErrorif not, would turn the documented rule ("Multi-year tables must still describe the same population") into an enforced one. - nit,
docs/book/usage/simulations.md:274: #2273 adds a paragraph at the same spot, so whichever merges second has a small text conflict.
What I checked
- Read every
build_from_*path insimulation.pyto confirm all group inputs pass through one of the two changed loops.build_from_dataframecorrectly keepsnp.unique. - Ran my own case: three households given in the order 2, 3, 1, with benefit units in the order 3, 1, 2. Region and weight map to the right people in 2025 and 2027, under a reform, and when a new simulation is built from
sim.dataset. The caller's table order is unchanged. pytest test_dataset_group_order.py: 14 passed. With thesimulation.pychange reversed: 8 failed, 6 passed.test_dataset_sources.pyandtest_simulation_dataset_sources.py: 17 passed.ruff checkandruff format --checkon the two changed Python files: clean.- Several other open PRs touch
simulation.py, including #2193.git merge-treeshows only the #2273 docs conflict above in a file this PR changes. - Reproducer and axiom lines are present;
fixedfragment type is right. CI, including the documentation test, is green on the current head.
🤖 Generated with Claude Code
vahid-ahmadi
approved these changes
Oct 18, 2026
vahid-ahmadi
left a comment
Collaborator
There was a problem hiding this comment.
Approving, per my review above. The shoulds and nits can follow.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2278
Unsorted benefit-unit or household tables can assign a group's region, weight or capital to a different group. Align group inputs with the simulation's sorted identifiers, independently for each year, while preserving person order and the caller's table order. Cover UK dataset objects, local H5 files and Core datasets, including copied tables used when economic assumptions are disabled.
Preserve Core's scalar-input normalisation and population-length validation. Add regression cases for group identity, year-specific permutations, final Universal Credit outcomes under explicit synthetic parameters, scalar inputs and invalid multi-group scalars. Document the table-order contract. Related: #2273 addresses identifier storage separately.
Reproducer: policyengine_uk/tests/test_dataset_group_order.py::test_unsorted_group_tables_keep_their_inputs
Validation:
Local validation used Python 3.13 / Core 3.32.27; CI uses locked Core 3.33.2. The full local suite was not completed: its YAML phase was interrupted before the Python phase started. Private population calculations and the documentation build were not run locally.
axiom: n/a: dataset-loading identity and input-shape compatibility; tax-benefit rules and economic assumptions are unchanged.