Skip to content

Add population-side transport graph kernels (NZ graph G5a) - #1159

Merged
MaxGhenis merged 4 commits into
mainfrom
transport/graph-population-kernels
Oct 9, 2026
Merged

MaxGhenis merged 4 commits into
mainfrom
transport/graph-population-kernels

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

G5a of the Microcosm NZ graph plan (Max's d962): the population-side transport kernels, country-neutral, as thin wrappers over the merged G3b transport functions (#1130) and G4 concept/unit functions (#1122).

What it adds (packages/microcosm-build/src/microcosm/build/transport/)

  • Codecs (codecs.py): populace-us-h5-v1, ledger-consumer-artifact-v1 and rulespec-tree-v1, registered by an explicit, idempotent register_transport_codecs(registry). The donor decodes to a donor-scale Frame; facts and RuleSpec trees stay bytes, as graph/codecs.py does.
  • Kernels, registered together in registry.py:
    • population (population_kernels.py): transport.create@1, transport.boundary@1, transport.currency@1, transport.quantile_map@1;
    • columns (column_kernels.py): transport.unit_attributes@1, concepts.encode@1, concepts.encode_groups@1, takeup.assign@1, transport.scenario_override@1;
    • geography (geography_kernels.py): geography.support_from_facts@1.
  • CREATE authenticates both donor pins, splits concepts, builds G4 benefit units, builds its schema from the unit rule (equal to NZ_SCHEMA for every input CREATE accepts, since it requires rule.entity == "family"), derives source-ID seeds, installs destination-scale design weights and support strata, and lists the columns split_for_transport drops in its receipt.
  • Boundary requires the declared person slice and the conserve policy.
  • takeup.assign@1 implements the contract the G4 input closure declares: code 0 refuses; receipt priorities, exclusion groups and positive-payment screens come from spec data.
  • Spec values and their resource SHA-256 travel in node params; those *_sha256 resource params move node keys and are format-checked only. Content is authenticated only for the donor pins (donor_sha256, donor_size), facts_sha256 and the concept-schema digest. implementation_hash = source_hash(...) binds the wrapper and the wrapped modules, never the whole-spec fingerprint. Capabilities are deterministic, with KEYED only for seed and draw derivation (transport.create@1 seeds, takeup.assign@1 draws): the three encode kernels refuse take_up_rates, so takeup.assign@1 owns every take-up draw. geography.support_from_facts@1 declares PLATFORM_BITWISE (its NPZ is a deflated zip). consumes_se=False.

Invariants (Hypothesis property tests, plus direct-function differentials)

  • A1 determinism: two fresh stores give identical keys and artifact bytes.
  • A2 memoization: a second run executes zero kernels.
  • A4 inert fields: editing description or citation moves no key.
  • A6 input identity: renaming a source path changes nothing; changing a byte invalidates exactly that source's consumers.
  • D2 mass: CREATE mass equals the declared total (±1e-6 relative), and the boundary FILTER conserves it.
  • CREATE columns equal the declaration; seeds lie in [0, 1) and are stable under row permutation; rewrites never change ids or row order; receipt flags imply eligibility.
  • run_graph rejects at run time a variant that owns a structural column, after compile_graph accepts it.
  • Differentials: each kernel's output equals a direct call of the wrapped G3b/G4 function on the same inputs (quantile aggregate and component maps, concept encoding, group encoding, unit construction). The household-aggregate differential uses a donor with two asset holders in one household, and band shares come from unequal facts, so aggregation and share mutants fail.
  • Geography: integer support is apportioned exactly, conserved and invariant to row permutation.

Tests (one file at a time)

At fd38688 (after the review fixes and the merge of main with #1158), rerun by the hub (population kernels again at 7ef990e: 27): test_transport_graph.py 15 (the source-scan rules: no "nz" word, no float literal in build/transport/*.py), test_transport_population_kernels.py 27, test_transport_column_kernels.py 17, test_transport_codecs_geography.py 22, test_transport_gate_bindings.py 110, microcosm-frame test_group_encode.py 81: all passed. tools/ci_test_plan.py verify: 641 files registered, 0 violations. Ruff check and format pass on the changed files.

Carried nits from earlier reviews

Not in this PR (activation, for G6/G8)

  • build/nz/benefit_unit_rule.json still has a null age_limit_years; the composer must pass a reviewed resolved rule.
  • build/nz/precal_references.json references are placeholders until G8.
  • The receipt contract (priorities, exclusion groups, payment columns, count references) is prose in axiom_input_closure.json; the composer needs a reviewed structured contract.
  • No donor record is labelled as New Zealand microdata.

Review

  • r1 (independent Opus 5.5 Subfleet review at 8c8df19): REQUEST_CHANGES, Needs Max: no. Blocking: the existing source-scan rules in test_transport_graph.py failed (an NZ_SCHEMA name and a 0.0 literal). Non-blocking: two vacuous differentials (household aggregation, band-fact shares), keyed draws in kernels declaring SeedSource.NONE, BITWISE on a deflated NPZ, unpinned string storage, unchecked band-fact units, a pre-filtered encode_groups mapping, test gaps, and resource-digest wording. No kernel correctness bug.
  • Fixes in 5447397 (in-session Opus 5.5): every finding addressed; CREATE builds the schema from the unit rule (equal to NZ_SCHEMA by value); declared string columns use the graph's pinned Python storage; band facts must be unfiltered counts on the ranked unit; encode kernels refuse take_up_rates; 15 failing test items (from 10 test functions) on 8c8df19, and 17 named mutations caught. Merge of main (Refuse unsigned ids int64 cannot hold; match concept ids as int64 #1158) in fd38688, clean.
  • Delta review of the fixes (independent in-session Opus 5.5) at fd38688: APPROVE_WITH_NITS, Needs Max: no. All 10 r1 findings resolved; every r1 survivor mutant and every fix reversion is now caught; the encode kernels' take_up_rates refusal is consistent with plan §2.3 (any unfiltered NZ encode is also refused, fail-closed); Refuse unsigned ids int64 cannot hold; match concept ids as int64 #1158's int64 matching changes nothing the kernels feed. Nits applied in 7ef990e: CREATE strata stored as object (an inferred pandas string dtype followed pyarrow), with a dtype assertion that fails on fd38688; PR-body wording. Not applied (optional): a test for the zero-eligible-mass receipt branch, which is equivalent to the old code.

Part of Microcosm NZ v0 (d962), package G5a of the NZ hub's graph plan. Built on a GPT-6.1 Sol Subfleet lane; verified, committed and opened by the NZ hub.

🤖 Generated with Claude Code

MaxGhenis and others added 3 commits October 9, 2026 02:17
- Register donor H5, Chronicle facts and RuleSpec tree codecs explicitly.
- Add ten neutral population kernels with destination-scale design weights,
  benefit units, fact-backed quantile maps and atomic geography support.
- Delegate concept encoding and scenario allocation to G4; assign receipt
  flags through determined eligibility, payment screens and spec priorities.
- Add synthetic H5 graph properties and wrapped-function differentials for
  mass, identity, memoization, seed stability, eligibility and runtime refusals.
- Clarify that ADR step 6 relies on benefit-unit construction.
- Pin the subnormal/large-weight gate example and describe refusal-only
  overflow behavior.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
- Build CREATE's schema from the household and the unit rule's entity
  instead of naming the country schema, and compute a zero eligible mass's
  receipt rate without a float literal, so the transport source rules pass.
- Pin graph string storage (dtype_for_token("string")) in CREATE and in the
  column kernels' casts, so pyarrow cannot change artifact dtypes.
- Require quantile band references to be unfiltered count facts on the
  ranked unit (the aggregate entity when aggregating).
- Refuse take_up_rates in concepts.encode@1, concepts.encode_groups@1 and
  transport.scenario_override@1; takeup.assign@1 owns the keyed draws, so
  SeedSource.NONE holds.
- Pass encode_groups the whole mapping so its state/concept name guard
  sees bindings outside the selected modules.
- Declare platform_bitwise numerics for geography.support_from_facts@1,
  like the atomic geography kernels.
- Say in docstrings that the *_sha256 resource params only move node keys;
  donor pins, facts_sha256 and the concept schema digest are checked.
- Tests: unequal band facts with shares derived from the facts; a donor with
  two asset holders in one household; band-unit, boundary, take-up, path
  filter, collision guard, target filter/entity, codec order, string storage
  and stale donor pin refusals.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis added a commit that referenced this pull request Oct 9, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The #1159 delta review found the CREATE frame's strata Series built
without a dtype, so pandas 3 inferred its `str` dtype, whose storage
follows whether pyarrow is installed (string_storage=pyarrow on this
machine). The graph pins Python storage so that an optional pyarrow
cannot change an artifact's physical dtype, and the donor codec already
stores strata as object. Build the strata (and the test-support
expectation) with dtype=object, and assert the dtype in the CREATE
differential; that assertion fails on fd38688.

tests: test_transport_population_kernels.py 27 passed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis merged commit 36557ca into main Oct 9, 2026
10 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Merged by the NZ hub (session local_e407a527) at 7ef990e with --merge. Gates: gh pr checks exit 0; MERGEABLE; not draft; no CHANGES_REQUESTED review. Reviewed head: 7ef990e. Independent review chain: r1 by an independent Opus 5.5 Subfleet review at 8c8df19 (build was GPT-6.1 Sol): REQUEST_CHANGES, Needs Max: no (jobs/review-microcosm-1159/REPORT-r1.md) — blocking source-scan rule failures plus 9 non-blocking findings, no kernel correctness bug. Fixes 5447397 (in-session Opus 5.5 builder) + main merge fd38688: delta review by a separate in-session Opus 5.5 reviewer APPROVE_WITH_NITS (delta/REPORT.md: all 10 findings resolved, r1 survivor mutants now caught). Strata dtype nit 7ef990e: same reviewer APPROVE, Needs Max: no (delta/REPORT-final.md). All 10 checks green, MERGEABLE, no CHANGES_REQUESTED.

@MaxGhenis
MaxGhenis deleted the transport/graph-population-kernels branch October 9, 2026 09:11
MaxGhenis added a commit that referenced this pull request Oct 9, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant