Skip to content

Add Stats NZ 2023 Census household and family counts by regional council - #335

Open
MaxGhenis wants to merge 2 commits into
mainfrom
nzhub/b5
Open

MaxGhenis wants to merge 2 commits into
mainfrom
nzhub/b5

Conversation

@MaxGhenis

Copy link
Copy Markdown
Contributor

Fills the two Census 2023 rows of the NZ source checklist (#176 packages 3–4): stats_nz/census_2023_households_by_region and stats_nz/census_2023_family_type. This is B5 of the NZ v0 plan and the last NZ source the population build was waiting on.

What this adds

288 published 2023 Census counts from Stats NZ's Place Summaries tool: 16 per place, for New Zealand, the 16 regional councils, and the publisher's separate "Area Outside Region" entry (code 99).

Section Categories Facts per place
Number of families and households Families, Households 2
Tenure of household Owned, Not owned, In family trust 3
Family type Couple Without Children, Couple With Child(ren), One Parent With Child(ren) 3
Number of usual residents in household One usual resident … Eight or more usual residents 8

Not emitted, but preserved in the pinned files: the Percent rows, the 2013 and 2018 periods, the Māori-ethnic-group sections and every other section.

Two things the plan asked for are not in this product, and nothing approximates them: a household-composition table (it publishes household size instead) and a social-renting split (tenure has three categories).

Source

  • Stats NZ's Place Summaries tool, "Download data", one CSV per place: https://tools.summaries.stats.govt.nz/api/export-csv/<classification identifier>/<code>/. Each manifest records its exact URL.
  • The endpoint generates the CSV on request, and the download filename carries a timestamp. The hub fetched the Auckland file twice and got identical bytes, so the content is pinned by SHA-256. Each manifest records the fetch time, HTTP status and the publisher's download filename.
  • 18 files, about 400 KB each, fetched 2026-10-10. .gitattributes marks them binary, so their CRLF line endings stay byte-exact.
  • Licence CC BY 4.0, per the Stats NZ copyright page, checked 2026-10-10.
  • All 18 objects are published under raw/nz/stats_nz/census_2023_place_summaries/2023/…. The hub verified each by streaming it back (SHA-256 matches).

Package structure, and a question for review

Chronicle's SourceArtifactSpec selects one manifest per declarative package, and this source is 18 files. The lane kept the checklist's two package ids as roots. Each root's YAML describes New Zealand, and each rc_XX/source_package.yaml child describes one regional-council entry: 36 package members in all. Both families read the same 18 files, committed once in db/data/stats_nz/census_2023_place_summaries/ with one manifest per place.

Costs of that shape, flagged for the reviewer:

  • Nested package members have no precedent in this repo. discover_source_package_dirs globs two levels, so nested members are covered by the alias map but not by discovery.
  • It adds 36 packages for 288 facts.
  • The two families give the same file different virtual sheet names, to keep their source-cell keys apart.

The alternative is 18 flat packages, one per place, each emitting all 16 facts: half the package count, the standard two-level layout, and no virtual sheet names, with both checklist rows pointing at the one family. The hub will restructure if review prefers that.

Invariants and how they're checked

These are scoped to this PR's 288 facts:

  • One CSV row per fact. Every fact reads one row's value, with no sums. The New Zealand rows are their own facts, never a sum of regions. Households is never the sum of its categories: nationally the three tenure counts sum to 1,780,101 against 1,780,527 households, and both stand.
  • Exact selection. The test declares every expected fact (place code, section, category and row identity) from literal lists, not from the package YAML, and requires the packages to select exactly that set.
  • Status lineage. confidentiality_indicator and final_indicator travel with each fact. A flagged row is never emitted as a number.
  • Labels as published, including Couple Without Children. Every dimension and value is labelled (Dimension labels are only checked for the UK suite: 68 of 79 other packages fail the consumer_fact.v2 rules #265).
  • Reference date 7 March 2023. The product does not state its regional boundary vintage, and the package records that it is unstated.
  • Area Outside Region is its own geography (region:99) and is folded into no region.

Mutants that fail the tests: a Percent row in place of a Count row, a 2018 row in place of 2023, one region's row read from another region's file, and a Māori-ethnic-group row in place of the total-population row. Byte-level CRLF mutants fail too.

Spot values, checked against the files:

  • New Zealand: Households 1,780,527; Families 1,294,503; Not owned 604,884; One Parent With Child(ren) 213,534.
  • Auckland Region: Households 539,490; Not owned 218,220.

Tests

  • tests/test_chronicle_nz_census_households.py: 186 passed. The build lane ran it, and the hub re-ran it on the final tree.
  • All 36 members pass validate-package and single-source build-suite.
  • The full-bundle node is left to CI.

Bundle pins

Measured against main 244a1281:

  • facts +288 (1,002,783 → 1,003,071);
  • packages +36 (321 → 357);
  • tables +1;
  • geographies +1 (the new region:99);
  • stats_nz +288, household +216, family +72, country:NZ +16, each of the 16 regions +16;
  • no new sources, periods, entities or warnings.

Chronicle governance

  • Agent role: ledger-source-ingestor. As in the other package PRs, the per-package files outside its allowed_paths are the alias entries in chronicle/source_package.py, db/data/**, docs/pe-nz-source-checklist.md, the changelog fragment and the bundle pins. chronicle/source_package.py changes by alias entries only.
  • Deterministic checks: validate-package and the build-suite acceptance gates (source-cell preservation, consumer contract, raw-facts boundary) pass for all 36 members inside the test file.
  • Judge verdicts: the build lane's own ledger-source-fidelity and ledger-boundary checks passed. The independent review that follows is asked to give both as named verdicts.
  • Receipts: the R2 read-back is the hub's attestation. chronicle verify-raw (Add chronicle verify-raw to read back R2 artifacts against manifests #334) will let anyone with read access regenerate it.

Built by a GPT-6.1 Sol lane from the NZ hub's brief. The hub verified the test run, the 18 artifact hashes and the 18 R2 objects. An independent review follows before merge.

🤖 Generated with Claude Code

MaxGhenis and others added 2 commits October 10, 2026 19:28
Keep the two requested package families, each with national and rc_XX members:
36 declarative source packages share 18 unchanged CSV artifacts and manifests.
Emit 288 publisher Count rows: 216 household and 72 family facts, 16 per place.
Area Outside Region is a separate publisher region:99, not an inferred residual.

Invariants and checks:
- Exactly one CSV row and O value cell per fact; independent literal expected
  map pins place, section, category, physical CSV row, file hash and selector.
  Atomic end_address=None, no divisor, no round_to, scale 1; every emitted
  value equals that row's publisher lexeme. aggregation=sum describes the
  published count; no source rows are summed to construct any fact.
- NZ totals read the national CSV. Households reads its own row 1655; the
  1,780,101 sum of tenure counts differs from published Households 1,780,527.
  Exact-cell and atomic-lineage regression checks preserve both unchanged.
- Only 2023 Census Count rows of the four named total-population sections;
  literal expected map and red mutants reject Percent, 2018, another region's
  file and the Māori-ethnic-group section. Full 2,030 rows in every CSV remain
  preserved, including other periods, percentages and all excluded sections.
- Census-night coverage is 2023-03-07, attributed to the hub source brief;
  files themselves say 2023 and Census. Off-build-year regression keeps the
  reference period 2023. Regional vintage is explicitly unstated; no rounding
  policy is asserted because no staged notes supply one.
- Confidentiality_indicator/label and final_indicator/label remain full-row
  lineage. Exact blank admission criteria reject confidential/unavailable
  rows even with numeric values. Every identity/status selection field and
  nonblank selector guard has independent coverage plus a mutated-source
  failure demonstration and regression. No imputed not-stated category.
- Category labels retain publisher spelling/case, including Couple Without
  Children; every dimension and value passes consumer label validation.
- Source-cell keys use distinct Chronicle virtual CSV sheets for households
  and families; cross-family identity regression prevents compact virtual
  coordinate collisions while retaining identical file SHA/bytes.
- 18 SHA-256/size checks match receipts (7,410,011 bytes); manifests retain
  URL/final URL, fetched_at, Content-Disposition filename, and CC BY 4.0.
  R2 metadata uses this worktree's chronicle.artifacts.build_r2_key with the
  country-aware raw/nz prefix. The hub still owns upload and verification.
- No reconciliation, aging, imputation, shares, target activation, measurement
  bindings or PolicyEngine-computed facts. Missing household composition and
  social-renting split are documented, without approximations.

Validation and measured bundle delta:
- Targeted checks: 186 NZ regressions + 36 single-source validate-package/
  build-suite/build_bundle cases + 18 unchanged-byte reruns = 240 passed
  executions / 222 unique cases. Every temporary single-source bundle deleted.
- Red proofs: 36 wrong-selection/identity/status mutants + 18 CRLF-normalized
  byte mutants = 54 deliberate failures. The 18 original artifacts then pass.
  '*.csv binary' in the artifact .gitattributes prevents Git CRLF normalization;
  each receipt SHA/size and the attribute are guarded independently.
- ledger-source-fidelity PASS; ledger-boundary PASS.
- Measured union: facts 288, geographies 18,
  sources 1 (stats_nz), packages 36, tables 1,
  periods 1 (calendar_year:2023 = 288), entities 2
  (household = 216, family = 72), bundle warnings 0, both duplicate counts 0.
  Table: stats_nz:2023 Census Place summaries: families and households.
- Additive to 244a128: facts +288
  (1002783 -> 1003071), geographies +1 (20547 -> 20548; new region:99),
  sources +0 (56; existing stats_nz facts 1498 -> 1786), packages +36
  (321 -> 357), tables +1 (316 -> 317; table above), warnings +0 (117),
  periods +0 (511; calendar_year:2023 facts +288), entities +0 (12;
  household 154636 -> 154852, family 1628 -> 1700). Duplicate deltas 0.
- Exact by_geography union and baseline additive counts:
  country:NZ: 16; additive +16; baseline 6106 -> 6122
  region:01: 16; additive +16; baseline 5 -> 21
  region:02: 16; additive +16; baseline 5 -> 21
  region:03: 16; additive +16; baseline 5 -> 21
  region:04: 16; additive +16; baseline 5 -> 21
  region:05: 16; additive +16; baseline 5 -> 21
  region:06: 16; additive +16; baseline 5 -> 21
  region:07: 16; additive +16; baseline 5 -> 21
  region:08: 16; additive +16; baseline 5 -> 21
  region:09: 16; additive +16; baseline 5 -> 21
  region:12: 16; additive +16; baseline 5 -> 21
  region:13: 16; additive +16; baseline 5 -> 21
  region:14: 16; additive +16; baseline 5 -> 21
  region:15: 16; additive +16; baseline 5 -> 21
  region:16: 16; additive +16; baseline 5 -> 21
  region:17: 16; additive +16; baseline 5 -> 21
  region:18: 16; additive +16; baseline 5 -> 21
  region:99: 16; additive +16; baseline absent -> 16
- Suite acceptance warnings (Axiom metadata CLI not configured): {"concept_alignment_validation_skipped": 36}.
  Concept-stage warnings: {"axiom_cli_not_configured": 36}. Bundle warnings remain 0.
- All three modified Python files pass lint; both modified tests pass format.
  source_package.py format checking finds a pre-existing legacy IRD alias style.
- Checklist package-valid cells are ticked; R2 stays unchecked until the hub
  uploads and checksum-verifies all 18 objects. No commit/push/PR performed.
No full-country/default bundle was built; CI owns the merged-contract test.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
The hub published the manifest-declared objects with chronicle publish-raw and
verified each by streaming it back from ledger-raw (SHA-256 matches) on
2026-10-10. Tick the checklist's raw/nz column for both rows and replace the
lane's upload-pending sentences.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant