Skip to content

Register weighted-microdata rules: weights aren't values #3937

Open

Description

@anth-volk

Problem

A weighted MicroSeries or MicroDataFrame weights every .sum(), .mean() and .count(). Aggregating a weight column through one weights it by itself, and returns the sum of squared weights.

That happened in the Simulation API (PolicyEngine/policyengine-sim-api#766, fixed by PolicyEngine/policyengine-sim-api#767). budget.households was about 10^4 times too high on every run, and its test missed it because it used plain pd.DataFrames.

Once the unused economy comparison module is removed, nothing in the Public API handles microdata. Nothing stops new code from making the same mistake, though.

Proposal

Phase 1 step 1.9. Register the rule that weights aren't values, plus AST guards for the shapes a source check can recognise, in rules/registry.toml:

  • api.microdata.weights-not-values (doc-only): weights, values, labels and counts each allow different operations.
  • api.microdata.weights-not-aggregated-as-values: a weight is never the value of a weighted MicroSeries, the variable of a weighted aggregator, or frame[<weight>] followed by a weighted reduction.
  • api.microdata.weight-stripping-confined: raw values (np.array, .values, .to_numpy() and so on) are read from weighted microdata only in a designated module. None is designated yet.
  • api.microdata.unweighted-series-are-labels: a MicroSeries built without weights= is only a grouping or filter key.

The aggregation rule's proof is the real pre-#767 Simulation API line. With no current code in scope, all three guarded rules start enforced, with no findings.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions