Repository navigation
Add chronicle verify-raw to read back R2 artifacts against manifests - #334
Merged
Merged
Conversation
Add chronicle verify-raw with publish-raw's manifest scan and country-aware resolve_r2_prefix/build_r2_key routing. Stream declared R2 objects through Wrangler get --remote --pipe, hash binary bytes, count their length, and compare manifest and optional local measurements. Emit frozen reports and JSON with closed statuses and separate no_r2_location counts. Exit 1 for any unverified entry or manifest error, following artifact command conventions; missing R2 declarations fail without adding a skip flag. Declared buckets take precedence over the fallback --r2-bucket option. Read-only invariant: snapshot all root file bytes and recursive directory entries before and after success and every failure status; assert fake Wrangler receives only get calls and never put/delete. A write-sentinel mutant proves the snapshot checks detect writes. Verified invariant: require declared key == canonical key, remote SHA-256 == manifest SHA-256, and remote size == manifest size. Assert all three on the successful binary fixture; fake-remote corruption, truncation, missing-object and wrong-prefix tests exercise failures. Compare present local bytes too. Determinism invariant: identical fake remote yields identical complete JSON reports across runs; assert sorted manifest paths and manifest file insertion order explicitly. A reverse-scan mutant fails that ordering assertion. Payload property: Hypothesis is absent from pyproject.toml and existing tests; use seeded random bytes without a dependency. Cover empty and multi-chunk payloads, byte flips, truncation, missing objects and wrong-prefix keys. Validation: 25 tests passed in tests/test_chronicle_verify_raw.py using the assigned Python interpreter and required no-cache/basetemp flags. Ruff check and format check passed on all changed Python files; git diff --check passed. Six in-memory mutants ran in two one-file pytest invocations: 33 body failures and 17 passes, with every one of the 25 new cases killed and no setup errors. Exact failing IDs and E lines are in .hub-scratch/report.txt. Source-file hashes are unchanged across mutation runs; pytest scratch was removed after each run. publish-raw is unchanged. This checkout already rejects differing recorded keys before upload or manifest writes, so the incident's key rewrite is not a remaining publish-raw behavior in the code read for this change. Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Contributor
Author
|
Merged by the NZ hub (session local_e407a527) at 6b2a859 with --merge. Gates: |
This was referenced Oct 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #332.
What this adds
chronicle verify-rawis the read-only counterpart ofpublish-raw. It scans manifests under--rootthe same way. For each file entry with an R2 location, it:resolve_r2_prefix/build_r2_keypathpublish-rawuses, country segment included;wrangler r2 object get --remote --pipe;It never writes to manifests or to R2. The JSON report gives one status per entry:
verified,key_mismatch,digest_mismatch,size_mismatch,missing_remote,no_r2_location,fetch_errororinvalid_manifest. The command exits 0 only when every entry isverifiedand the scan reported no errors.That report is a receipt anyone with read access can regenerate. Five independent reviews of the NZ packages (#328, #329, #330) noted that the R2 upload and read-back were only attested in prose.
Live run (hub, with R2 read credentials, at this branch's code)
db/data/ird/wage_salary_distribution_2026verifieddb/data/stats_nz/household_net_worth_2024verifieddb/data/treasury/estimates_of_appropriations_2026_27(--r2-prefix raw/nz)verifieddb/data/ird/taxable_income_distribution_2025verifieddb/data/mbie/tenancy_bond_rents_tla_2026verifiedNegative checks ran on scratch copies of the IRD manifest:
nzsegment (raw/ird/…, where a stray copy of the object actually exists):key_mismatch, exit 1.missing_remote, exit 1. Wrangler's real "The specified key does not exist" message is recognised.invalid_manifest.No manifest changed during any run.
Invariants and how they're checked
get. There are snapshot tests, and a mutant that writes a sentinel file fails them.verifiedmeans the key, digest and size all match. A mutant forcing every status toverifiedfails 10 tests.publish-raw's scan order. A mutant reversing the order fails.raw/ird/…key reportsmissing_remote, and a manifest declaring the legacy key reportskey_mismatch.Tests
tests/test_chronicle_verify_raw.py: 25 passed. The build lane ran it, and the hub re-ran it.tests/test_chronicle_artifacts.pystill passes (36).publish-rawis unchanged. On current main it already refuses a declared key that differs from the recomputed one before uploading, so the incident in Add chronicle verify-raw to read back R2 artifacts against manifests #332 needed an older installedchronicle.Chronicle governance
This is repository tooling, not a source package. The closest approved role is
ledger-contract-maintainer, but this PR changes no schema, identity, provenance or consumer contract. It adds a read-only CLI command and its tests inchronicle/artifacts.py,chronicle/cli.pyandchronicle/harness.py, plus README and changelog lines. No facts or bundle pins change.Built by a GPT-6.1 Sol lane from the NZ hub's brief. The hub verified the tests and ran the live checks above. An independent review follows before merge.
🤖 Generated with Claude Code