Skip to content

Add Stats NZ household net worth for the year ended June 2024 - #329

Merged
MaxGhenis merged 11 commits into
mainfrom
nzhub/b6nw
Oct 10, 2026
Merged

MaxGhenis merged 11 commits into
mainfrom
nzhub/b6nw

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Added for PolicyEngine/microcosm#592, outside #176's original 15 packages. This is B6 part 2 of the NZ v0 plan. Part 1 was the MBIE rents package, merged in #320.

What this adds

packages/stats_nz/household_net_worth_2024 holds 1,413 Household Economic Survey observations from Stats NZ's "Household net worth statistics: Year ended June 2024" (released 26 September 2025). The reference period is July 2023 to June 2024, fiscal_year:2023 under the opening-year convention.

It covers 14 tables:

  • 1.01: assets and liabilities by type, among households holding each;
  • 2.01–2.04: median, mean, total and households, by net-worth quintile;
  • 3.01–3.04: the same, by income quintile;
  • 5.01: household net worth by household characteristics;
  • 7.01–7.04: individual assets and liabilities by age group.

Across these tables, the package retains median, mean, total and count estimates, in $000s or in thousands of households or people. Tables 2.01–2.04, 3.01–3.04 and 7.01–7.04 each publish one statistic per sheet.

What it leaves out:

  • Sampling errors. Relative sampling errors stay in the source cells only: Chronicle's active fact, spec and consumer contract has no sampling-error field. The package README records this.
  • Percentage changes. These columns, and their sampling errors, are omitted.
  • Suppressed cells. Seven estimates marked S stay as raw text. They are never treated as facts or zeros.

Source

  • https://www.stats.govt.nz/assets/Uploads/Household-net-worth-statistics/Household-net-worth-statistics-Year-ended-June-2024/Download-data/household-net-worth-statistics-year-ended-june-2024.xlsx.
  • 386,546 bytes, SHA-256 2fb94232…54913f9e, fetched 2026-10-10.
  • Licence CC BY 4.0, per the Stats NZ copyright page, checked 2026-10-10.
  • The workbook is published to raw/nz/stats_nz/household_net_worth_2024/…. The hub verified it by streaming it back (SHA-256 matches).

Invariants and how they're checked

These are scoped to this package's 1,413 facts:

  • One cell per fact. Every fact equals one cached publisher cell, with no ranges, divisors, rounding or rescaling. The complete workbook bytes are preserved. The Contents sheet plus the 14 scoped sheets' used ranges come to 31,534 cells.

  • Holders versus all households. Table 1.01's asset-type means and medians cover only households holding that asset. The table's own population note is carried in each fact's dimensions. Two figures make the difference concrete:

    • Table 5.01 C37 (525, $000s) is the median across all households.
    • Table 1.01 C36 (529) is the holders-only figure.

    Both are kept as separate facts.

  • Published groupings only. Medians are cut-points, and quintiles are the publisher's groupings with their published counts. Nothing is derived, summed or converted between households and people.

  • Section boundaries follow the workbook. Each published_section value covers exactly the rows under the publisher's bold header, up to its bold subtotal. The test declares each section's member rows from the workbook's structure, not from the YAML, and checks both the facts and the exported consumer constraints.

  • Fixed reference period. Requesting another build year yields identical facts.

  • Labels. Every dimension and value is labelled (Dimension labels are only checked for the UK suite: 68 of 79 other packages fail the consumer_fact.v2 rules #265).

Regression tests reject 752 mutated publisher guard cells and 8 YAML mutations: scale, divisor, range, rounding, period, labels, population and median. Fact output is identical under PYTHONHASHSEED=1 and 2.

Tests

  • tests/test_chronicle_nz_household_net_worth.py: 795 passed. The fix lane ran it, and the hub re-ran it on the final tree.
  • validate-package and build-suite pass: 1,413 facts, lineage coverage 1.0.
  • The full-bundle node is left to CI.

Bundle pins

Built by a GPT-6.1 Sol lane from the NZ hub's brief. The hub verified the test runs, artifact hash and R2 object.

Chronicle governance

  • Agent role: ledger-source-ingestor. As in the other package PRs, the per-package files outside its allowed_paths are the alias entry in chronicle/source_package.py, db/data/**, docs/pe-nz-source-checklist.md, the changelog fragment and the bundle pins. There are no logic changes outside packages/.
  • Deterministic checks:
    • source-package validation: validate-package passes, with 0 errors and 0 warnings;
    • source-cell preservation: the complete workbook bytes are kept, and 31,534 cells across Contents and the 14 scoped sheets;
    • consumer-contract and raw-facts boundary validation: the package's build-suite acceptance gates pass inside the test file.
  • Judge questions:
    • ledger-source-fidelity: round 1, at 007d2451, recorded FAIL on the section leak. Round 2, at d287ab50, found the leak fixed: all 1,413 values equal their publisher cells, and an independent workbook-derived audit found zero grouping mismatches across the 14 sheets. Round 2 did not restate a named verdict, so PASS is the hub's reading of it.
    • ledger-boundary: round 1 recorded PASS. Round 2 found no arithmetic, period alignment or other boundary change.
    • ledger-contract: no schema or consumer-contract change.
  • Receipts. Both reviews note that the hub's fetch and R2 read-back are attested here, not shown by a receipt kept in the repo. Round 2 re-downloaded the workbook from Stats NZ and got the same bytes and SHA-256. A follow-up will add retained receipts for the three new NZ packages.

Review

An independent GPT-6.1 Sol review at 007d2451 returned REQUEST_CHANGES on one defect:

  • The defect. In Table 1.01, the "Shares and other equity" section ran past its subtotal (row 23) onto rows 24–27: mutual funds, pension funds, other household financial assets, and the household financial assets total. Sixteen facts carried the wrong section filter. Their values were right.
  • The fix (df575669). Rows 24–27 now sit under the publisher's "Assets" header (A9), and their lineage guard points at that header. All 1,413 values are unchanged: the sorted (record id, value) pairs hash identically before and after.
  • The audit. Every section in all 14 tables was checked against the workbook's bold headers and subtotals. This was the only leak. The README lists each section's member rows.
  • The regression. Two new tests fail when the leak is restored in memory, for the fact dimensions and for the exported consumer constraints.

The review's wording nits are fixed above: the cell count and the per-sheet measures. Its stale "R2 pending" line in the checklist is fixed in 335dbd69.

A second review at d287ab50 returned APPROVE_WITH_NITS with no blocking defect:

  • 795 tests passed.
  • The section query returns only rows 19–23.
  • Both grouping tests fail under the restored leak, and under a second grouping mutant on another table.
  • All values and selected coordinates are unchanged from round 1.
  • Its one nit is retained receipts, a follow-up.

🤖 Generated with Claude Code

MaxGhenis and others added 7 commits October 10, 2026 06:43
Register ird-wage-salary-distribution-2026, following the merged I1 package.
Preserve the complete 198,542-byte staged workbook unchanged and pin the hub
receipt: fetched 2026-10-10T08:31:07Z, SHA-256
d0526b2092f9fa2bec44e11984c0ef17fda788a742c1c7054d416f6a0cd9e750.
Use the stable publisher URL, CC BY 4.0 licence, and Chronicle build_r2_key.
R2 upload/remote checksum verification remain for the hub before merge.

Emit 510 administrative person observations for the ending 2026 tax year:
472 band facts (235 bands plus separate total, count and income), 19 decile
facts and 19 top-percentile facts. Preserve all 20,631 worksheet cells and
verbatim explanatory drawing notes and population footnotes. Historical and
revised columns remain source cells only. Rename checklist row 62 from
ird/wage_salary_distribution_2025 to ird/wage_salary_distribution_2026;
tick artifact/package boxes and leave R2 unchecked. Update the stale I1
sentence and add the changelog and source evidence note.

Invariants and checks:

- Every numeric fact reads one publisher cell. Audit all 510 unique selected
  coordinates against the unchanged workbook; direct-cell lineage/unit-scale
  regression and build-suite acceptance pass. All facts are administrative
  observations for persons, without PolicyEngine-computed values.
- Separately published totals remain independent BB242/BC242 facts, never sums
  of bands. Exact-cell and no-range-selector tests pass; total-kind YAML
  mutation fails its regression before the accepted YAML passes.
- Upper boundaries are NZD cut-points with aggregation quantile, never shares.
  Exact BB255/BB278 anchors and all boundary aggregation tests pass; changing
  quantile to sum fails its regression.
- Decile 10 BB264 and percentile 100 BB279 have no upper boundary. Both raw
  blanks remain source cells, both boundary facts stay absent, and their
  income facts remain present. Adding the blank percentile-100 boundary to
  scratch YAML fails the regression; no value is imputed.
- TY2026 means 1 April 2025 to 31 March 2026, the ending year. Header guards,
  coverage-date tests and build at bundle year 2023 retain tax_year:2026;
  changing the package period to 2025 fails its regression.
- $M cells retain exact cached publisher precision with scale 1000000 and no
  rounding; boundaries have scale 1. All 510 values equal the core's existing
  normalization of their cached source cells. Scale, rounding and divisor
  mutations each fail the regression. Source XML numeric lexemes remain in
  unchanged workbook bytes; no means, shares or uprated values are derived.
- Open bands retain literal publisher labels and categorical equality
  constraints; no endpoints are closed and no I1 bands are merged. Changing
  the top band label in scratch YAML fails the regression.
- Every dimension and value carries the explicit authored label. Complete
  consumer-contract/label tests pass; removing a dimension declaration fails.
- The drawing sentence naming revisions to 2022–2025 and row-2 R flags naming
  2023–2025 are both retained verbatim without reconciliation. Raw revision
  and explanatory-note tests pass; only New TY2026 facts are selected.
- Every required publisher guard has a regression: 273 independent coordinates
  cover all row/measure headers, years, New flag and population footnotes.
  Guard removal yields 274 expected failures; all 273 guards individually
  reject mutated source cells in the final green run.

Validation: validate-package CLI PASS (5 sets, 274 rows, 6 measures, 510 records,
5 regions, 0 errors/warnings). Targeted test file: 295 passed, 12 dependency
warnings; 10 scratch YAML red cases: 283 expected failures. Build-suite PASS,
all 12 acceptance gates true, 510 facts/records, 20631 cells, 510 constraints,
lineage coverage 1.0. Axiom validation was not run; acceptance records the
standard concept_alignment_validation_skipped warning. Ruff check/format
and git diff --check PASS. Local ledger-source-fidelity and ledger-boundary
reviews PASS. Full bundle/full-suite tests were not run; CI owns them.

Measured bundle delta (single-source build_bundle, year=2023):

- facts: +510; by_source ird: +510
- geographies: +0; new geography keys: []; by_geography country:NZ: +510
- publisher sources: +0 (ird already present)
- packages: +1 (ird-wage-salary-distribution-2026)
- tables: +1; by_source_table
  ird:Wage and salary distributions for individuals 2026: +510
- warnings: +0; errors: +0; skipped sources: +0
- period count: +0 (tax_year:2026 already present); by_period tax_year:2026: +510
- entity count: +0 (person already present); by_entity person: +510
- aggregate duplicate keys: +0; semantic duplicate keys: +0

Single-source standalone counts: 510 facts, 1 geography country:NZ, 1 source ird,
1 package, 1 table, 1 period tax_year:2026, 1 entity person, 0 warnings/duplicates.
Pins use baseline 36c0358 plus this measured delta: 1001258 facts, 319 packages,
314 tables, ird 4362, person 495562, country:NZ 4581; geography 20547, source 56,
period 511, entity 12, warning 117 counts unchanged. Hub can re-pin if parallel
packages land first. The 46,758,982-byte bundle output and pytest temp directory
were deleted; remaining scratch contains only small scripts and red evidence.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Preserve the unchanged September 2025 Stats NZ XLSX and emit 1,413 HES
observations across Table1.01, Tables2.01–2.04, Tables3.01–3.04, Table5.01
and Tables7.01–7.04. Added for microcosm#592; outside #176's 15.

Artifact: 386,546 bytes, fetched 2026-10-10T08:31:08Z, SHA-256
2fb942322e5131824a15007e7b0742c4f8220822c7e8f9adee83bae154913f9e.
Receipt URL and CC BY 4.0 / Stats NZ attribution are recorded in manifest.
R2 key uses chronicle.artifacts.build_r2_key and requires hub upload:
raw/nz/stats_nz/household_net_worth_2024/2024/2fb942322e5131824a15007e7b0742c4f8220822c7e8f9adee83bae154913f9e/household-net-worth-statistics-year-ended-june-2024.xlsx

Invariant checks:
- Every value is publisher-asserted: all 1,413 facts equal a distinct cached
  2024 source cell, with SHA/size/URL/R2 provenance and explicit label/note
  lineage. Whole workbook bytes (39 sheets) are unchanged; 31,534 source
  cells preserve Contents and all fourteen scoped worksheet used ranges.
  Tests: artifact_receipt_storage_and_licence_are_pinned,
  preserves_contents_and_complete_scoped_sheets,
  every_fact_is_one_unscaled_published_estimate, exact_publisher_cells.
- Medians are cut-points and quintiles are publisher groupings/counts:
  median aggregation, exact literal quintile headings, published count cells
  and every scoped asset/group cell are checked. No percentile bounds or
  counts are derived. Tests: quintiles_and_age_groups_remain_publisher_groupings,
  frozen_publisher_boundaries; changing median to sum is rejected in YAML.
- Table1.01 asset means/medians are among non-zero holders: literal A5
  population note is a fact dimension, concept evidence and guarded lineage.
  All-household Table5.01 C37=525 remains separate from holders-only
  Table1.01 C36=529. Tables2/3 .01/.02 and Tables7 .01/.02 retain their
  publisher-assumed zeros; count sheets retain holders notes. Total sheets
  without a population note get no inferred note. Tests:
  population_notes_distinguish_holders_and_all_households,
  every_publisher_note_is_verbatim_in_fact_lineage; population YAML mutant
  and every population-note source mutant reject.
- Separately published totals remain separate single-cell observations:
  no end_address, divisor, range, sum of rows or reconciliation. Tests:
  every_fact_is_one_unscaled_published_estimate, frozen_publisher_boundaries;
  mutated range/divisor YAML rejects.
- Stats NZ rounding and units are preserved: value_scale=1, no round_to;
  values remain NZD_thousands, household_thousands or person_thousands.
  All 1,413 numeric cells compare exactly; scale/rounding YAML mutants reject.
- Reference period is July 2023–June 2024: fiscal_year:2023 uses opening year,
  survey_reference coverage 2023-07-01..2024-06-30, artifact_year fixed 2024.
  Other requested build years yield identical facts; period YAML mutant
  rejects. Tests: requested_year_cannot_change_reference_period and
  frozen_publisher_boundaries. No uprating, aging or period alignment.
- Every dimension/value is labelled: bundle dimension-label report has zero
  errors/warnings; source spelling and footnote markers preserved. Chronicle
  display labels trim outer whitespace; source cells/layout remain verbatim.
  Tests: dimensions_values_and_consumer_contract_have_labels; missing-label
  YAML mutant rejects. Package-specific label dimensions avoid collisions.
- Suppression stays missing: seven S estimate cells and literal definitions
  stay raw text, with no facts, substitutions or implied zeros. All seven
  source mutations reject. Percentage changes and their errors are omitted;
  RSEs remain source cells because the active AggregateFact/spec/consumer
  contract has no sampling-error field (legacy SourceFact.margin_of_error
  does not apply). Tests: suppression_is_raw_evidence_and_never_an_estimate.
- Household/person quantities come only from their published tables; no
  conversions, cash/liquid-asset computation, imputation, donor handling,
  reconciliation, target activation or model-measurement bindings. Verified
  by complete one-cell coverage, frozen YAML and independent ledger-boundary
  judge. All facts assertion=observation, provenance_class=survey_aggregate,
  survey_instrument=Household Economic Survey.
- Determinism: source-cell, fact and consumer-row serialized SHA-256 outputs
  match in fresh interpreters under PYTHONHASHSEED=1 and 2.

Validation:
- validate-package: PASS, 82 record sets, 1,413 records, zero errors/warnings.
- build-suite: PASS, all agent-acceptance checks true; source-cell, facts,
  consumer-contract and raw-facts boundary gates pass; lineage coverage 1.0.
  Final single-source suite: 31,534 cells, 1,413 facts, 4,925 constraints,
  82 source regions, 3 source-label concept alignments, 1 raw artifact.
  Optional Axiom validation was skipped (normal nested concept/acceptance
  warning); source-only mappings remain source_label.
- First red: 752 independently enumerated mutated publisher guard coordinates
  rejected; 8 mutated YAML cases rejected (scale, divisor, range, rounding,
  period, labels, population, median). Green regressions follow each guard.
- pytest tests/test_chronicle_nz_household_net_worth.py -p no:cacheprovider
  --basetemp=.hub-scratch/pytest -o tmp_path_retention_policy=failed -q:
  793 passed in 75.11s; 12 dependency deprecation warnings. Basetemp deleted.
  Includes 16 exact-cell anchors and all 752 guard regressions.
- Ruff check + format check of changed Python files: PASS. git diff --check:
  PASS. Full-bundle test node and all full bundles left for CI.
- Independent ledger-source-fidelity: PASS; ledger-boundary: PASS.
  Final follow-up confirms no unresolved findings.

Exact measured single-source bundle contribution (build year 2023):
source alias=stats-nz-household-net-worth-2024; facts=1413; geographies=1;
geography keys/by_geography={country:NZ:1413}; sources=1 (stats_nz);
packages=1; tables=1; warnings=0; errors=0; periods=1;
by_period={fiscal_year:2023:1413}; entities=2;
by_entity={household:940,person:473}; by_source={stats_nz:1413}.
by_source_table={stats_nz:Household net worth statistics: Year ended June 2024, Tables 1.01, 2.01–2.04, 3.01–3.04, 5.01 and 7.01–7.04:1413}
Aggregate/semantic duplicate keys=0; skipped sources=0. Bundle deleted.

Default-bundle net delta relative to assigned base 36c0358:
+1413 facts; +0 geography keys (country:NZ already exists), +0 publisher
names (stats_nz already exists), +1 package, +1 source table, +0 warnings,
+0 period keys (fiscal_year:2023 already exists), +0 entity keys (household
and person exist), +0 duplicate keys, +0 skipped sources.
by_geography increments={country:NZ:+1413};
by_period increments={fiscal_year:2023:+1413};
by_entity increments={household:+940,person:+473};
by_source increments={stats_nz:+1413}; source-table increment is above.
Pins: facts1000748→1002161; packages318→319; tables313→314;
stats_nz85→1498; country:NZ4071→5484;
household153696→154636; person495052→495525;
geographies20547, periods511, entities12, sources56, warnings117 unchanged.
Hub can re-pin these additive counts after another NZ package lands.

No commits, pushes, PR edits or network calls performed; existing staged
hub sources preserved. R2 upload/remote verification remains with the hub.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
The hub published the manifest-declared objects with chronicle publish-raw and
verified each by streaming it back from ledger-raw (SHA-256 matches) on
2026-10-10. Tick the checklist's raw/nz column and replace the lane's
upload-pending sentences.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The hub published the manifest-declared objects with chronicle publish-raw and
verified each by streaming it back from ledger-raw (SHA-256 matches) on
2026-10-10. Tick the checklist's raw/nz column and replace the lane's
upload-pending sentences.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review r1 (nit 2): the fragment still said the upload was pending.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dditively

Stack chronicle#328 on #330 so both land without a second two-hour bundle
run. tests/test_chronicle_bundle.py pins are main 36c0358 plus both measured
single-source deltas: facts 1000748 + 510 (IRD I2) + 112 (Treasury) = 1001370;
packages 318 + 2 = 320; tables 313 + 2 = 315; country:NZ 4071 + 622 = 4693;
ird 4362, treasury 139, person 495562, government 3562. Git merged the
identical 318->319 and 313->314 edits silently; corrected to 320 and 315.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…erification

Review r1 (finding 2): the row ticked raw/nz verified but still said the key
awaited upload.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 2 commits October 10, 2026 14:00
Declare an independent 510-observation map from the committed workbook's
literal row ranges and fixed BB/BC columns. Check exact record IDs, sheet,
cell address, unit, scale and aggregation, plus emitted fact metadata.
Use the independent coordinates and scales for every value/lineage check.

Add four in-memory row-override regressions: count to income, boundary to
income, total to band, and wrong-year percentile column. All four mutants
fail the new positive test in its body, with no setup errors. Tracked YAML
and all 510 fact IDs, value types and exact values remain unchanged.

Validation: 300 targeted tests pass; Ruff check and format pass; git diff
--check passes. Only the wage/salary test file is changed. No full suite or
full-bundle node ran. Leave changes uncommitted for the NZ hub.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
MaxGhenis and others added 2 commits October 10, 2026 14:02
End Table1.01 Shares and other equity at its published row 23 subtotal.
Attach rows 24–27 to the publisher-backed Assets parent and A9 guard,
preserving all 1,413 numeric facts. Audit every section, quintile and age
group across all 14 scoped sheets; document exact publisher boundaries.

Add independent exact-membership tests for fact dimensions/constraints
and exported consumer universe_constraints. Existing independent XLSX
coordinate coverage remains intact. An in-memory original-defect mutant
kills both new tests; a constraint-only mutant also kills the facts test.
Coordinate mutants fail the existing completeness test, with 6 total
call-phase assertion failures.

Validation: 795 targeted tests pass; validate-package passes with no
errors/warnings; Ruff check/format and git diff --check pass. Before/after
record/value SHA256: 2fdaaf83226b148adb5922616e158c5e7b77ab952a5b7dd9fbaa77811a94cf0d.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
…tively

Stack chronicle#329 on #328 (which contains main at 3dce46a, including the
Budget 2026 Estimates package) so both land without another two-hour bundle
run each. tests/test_chronicle_bundle.py pins are that branch's plus this
package's measured single-source delta: facts 1001370 + 1413 = 1002783;
packages 321; tables 316; stats_nz 1498; country:NZ 4693 + 1413 = 6106;
household 154636; person 495562 + 473 = 496035; fiscal_year:2023 + 1413.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis merged commit 244a128 into main Oct 10, 2026
3 checks passed
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Merged by the NZ hub (session local_e407a527) at d287ab5 with --merge. Gates: gh pr checks exit 0; MERGEABLE; not draft; no CHANGES_REQUESTED review. Reviewed head: d287ab5. Independent GPT-6.1 Sol reviews: round 1 REQUEST_CHANGES at 007d245 (16 facts carried the wrong Table 1.01 section), round 2 APPROVE_WITH_NITS at d287ab5 (leak fixed; an independent workbook-derived audit found zero grouping mismatches across 14 sheets; all 1,413 values unchanged; 795 tests). The reviewers marked 'Needs Max: yes' because the PR adds publisher observations to a public repo. The hub is not holding on that: these are official statistics republished unchanged under CC BY 4.0 with attribution, the round-1 reviews of the same content judged it as not needing Max, and it matches the NZ source packages already merged under the same gates (#209, #211, #318-#321, #330). No model output, estimate or claim of ours is published. Landed after #328, on which it was stacked.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant