Skip to content

Align dataset group inputs with their identifiers - #2449

Merged
juaristi22 merged 2 commits into
mainfrom
fix-2278-group-input-order
Oct 19, 2026
Merged

juaristi22 merged 2 commits into
mainfrom
fix-2278-group-input-order

Conversation

@juaristi22

Copy link
Copy Markdown
Collaborator

Fixes #2278

Unsorted benefit-unit or household tables can assign a group's region, weight or capital to a different group. Align group inputs with the simulation's sorted identifiers, independently for each year, while preserving person order and the caller's table order. Cover UK dataset objects, local H5 files and Core datasets, including copied tables used when economic assumptions are disabled.

Preserve Core's scalar-input normalisation and population-length validation. Add regression cases for group identity, year-specific permutations, final Universal Credit outcomes under explicit synthetic parameters, scalar inputs and invalid multi-group scalars. Document the table-order contract. Related: #2273 addresses identifier storage separately.

Reproducer: policyengine_uk/tests/test_dataset_group_order.py::test_unsorted_group_tables_keep_their_inputs

Validation:

  • 68 distinct focused cases passed across the implementation and repair gates, including the full changed regression module and relevant loader, source, fiscal-year, input-record and enum controls.
  • Final combined regression batch: 39 passed, including all four scalar compatibility and rejection cases.
  • make format, Ruff and git diff --check passed.

Local validation used Python 3.13 / Core 3.32.27; CI uses locked Core 3.33.2. The full local suite was not completed: its YAML phase was interrupted before the Python phase started. Private population calculations and the documentation build were not run locally.

axiom: n/a: dataset-loading identity and input-shape compatibility; tax-benefit rules and economic assumptions are unchanged.

@juaristi22 juaristi22 self-assigned this Oct 16, 2026

@vahid-ahmadi vahid-ahmadi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks right, fine to approve.

@juaristi22, this fixes #2278 on every loader I traced: build_from_multi_year_dataset (so single-year datasets, local H5 files, hf:// and gs:// sources too), build_from_dataset for Core datasets, and the frozen-assumptions copy. The caller's tables stay unchanged. One hardening suggestion and a merge note.

  1. should, policyengine_uk/simulation.py:1046-1050 and :964-969: the alignment sorts each year's group table by its own IDs and assumes those IDs are the population's. If a later year lists a different set (for example household 4 where the base year has 3), the rows still shift silently, which is the same class of error as #2278. A check that np.sort(ids) equals self.populations[entity].ids for each year, raising a ValueError if not, would turn the documented rule ("Multi-year tables must still describe the same population") into an enforced one.
  2. nit, docs/book/usage/simulations.md:274: #2273 adds a paragraph at the same spot, so whichever merges second has a small text conflict.

What I checked

  • Read every build_from_* path in simulation.py to confirm all group inputs pass through one of the two changed loops. build_from_dataframe correctly keeps np.unique.
  • Ran my own case: three households given in the order 2, 3, 1, with benefit units in the order 3, 1, 2. Region and weight map to the right people in 2025 and 2027, under a reform, and when a new simulation is built from sim.dataset. The caller's table order is unchanged.
  • pytest test_dataset_group_order.py: 14 passed. With the simulation.py change reversed: 8 failed, 6 passed. test_dataset_sources.py and test_simulation_dataset_sources.py: 17 passed.
  • ruff check and ruff format --check on the two changed Python files: clean.
  • Several other open PRs touch simulation.py, including #2193. git merge-tree shows only the #2273 docs conflict above in a file this PR changes.
  • Reproducer and axiom lines are present; fixed fragment type is right. CI, including the documentation test, is green on the current head.

🤖 Generated with Claude Code

@vahid-ahmadi vahid-ahmadi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving, per my review above. The shoulds and nits can follow.

@juaristi22
juaristi22 added this pull request to the merge queue Oct 19, 2026
Merged via the queue into main with commit 0ad26d3 Oct 19, 2026
15 checks passed
@juaristi22
juaristi22 deleted the fix-2278-group-input-order branch October 19, 2026 19:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unsorted benefit unit or household tables give each group another group's inputs

2 participants