From 269ccf1aa62baa0c2cf93fef1cdcde73a732ab21 Mon Sep 17 00:00:00 2001 From: Raymond Yee Date: Thu, 23 Jul 2026 10:02:18 -0700 Subject: [PATCH 1/6] CANONICAL.md: name the canonical data set; document the accidental history MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Answers RY's 2026-07-22 question ('can we simplify our stable of supplementary parquet files?'): the live set is minimal (17 files + search index); the accumulation is superseded suffix versions (each bump a real fix that left its predecessor served — the #326 class) and two orphan snapshot generations (202604/202606). Suffix-bump policy + post-grant attic plan included. Chunk 1 of the #334 v0 work. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Nohddj7oXLnkJ4P9Cibg8Y --- CANONICAL.md | 85 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 CANONICAL.md diff --git a/CANONICAL.md b/CANONICAL.md new file mode 100644 index 0000000..dee6af7 --- /dev/null +++ b/CANONICAL.md @@ -0,0 +1,85 @@ +# CANONICAL.md — which data files are real, and which are accidental history + +*Created 2026-07-23 (grant closing week) to answer: "can we simplify our stable of +supplementary parquet files?" Answer: the **live set is already minimal** — the +accumulation is superseded versions and orphan snapshots around it. This file +names the canonical set and documents every non-canonical survivor so the +accident is neutralized even where files remain on the bucket.* + +**Companion:** the machine-readable release manifest +(`tools/build_release_manifest.py` → `isamples_202608_release_manifest.json`) +enumerates the same canonical set with sizes/checksums; the Explorer cross-checks +it at boot (#334 v0). A file absent from the manifest is non-canonical **by +definition**. + +## 1. The canonical (live) set — 202608 + +Exactly what production `explorer.qmd` loads, all under `https://data.isamples.org/`: + +| File | Role (plain English) | +|---|---| +| `isamples_202608_wide.parquet` | Full sample detail, one row per entity — everything derives from this | +| `isamples_202608_samples_map_lite_v3.parquet` | Slim map/table columns (coords, label, place, date, h3 cells) | +| `isamples_202608_sample_facets_v4.parquet` | Per-sample facet URIs + search-text blob (the `?fts=off` scan target) | +| `isamples_202608_h3_summary_res{4,6,8}.parquet` | Pre-counted globe clusters at 3 zoom tiers | +| `isamples_202608_facet_summaries.parquet` | Facet checkbox counts, no filters active | +| `isamples_202608_facet_cross_filter.parquet` | Facet counts under single cross-filter | +| `isamples_202608_facet_tree_summaries.parquet` | Hierarchical (tree) facet counts | +| `isamples_202608_facet_tree_cross_filter.parquet` | Tree facet counts under cross-filter | +| `isamples_202608_sample_facet_membership.parquet` | Sample↔facet-tree membership | +| `isamples_202608_sample_facet_masks.parquet` | Bitmask substrate for multi-filter counts | +| `isamples_202608_facet_node_bits.parquet` | Facet-node bit assignments for the masks | +| `isamples_202608_sample_facet_index.parquet` | Per-sample facet index (multi-filter fast path) | +| `isamples_202608_sample_facet_index_meta.parquet` | ~1 KB trusted manifest for the above (#313/#317 boot-race fix) | +| `vocab_labels_202608.parquet` | URI → human label (539 entries; verified complete 2026-07-22) | +| `isamples_202608_search_index_v1/` (852 objects) | Sharded FTS index — the default search since 2026-07-17 (+ sidecars `hot_tokens.json`, `shard_sizes.json`, `df.parquet`, `build_stats.json`) | + +The ~9-file facet family looks baroque but is load-bearing: it is the price of +fast multi-filter counts with no server. See `EXPLORER_QUERIES.md` for how each +is queried and `DATA_PROVENANCE.md` for how each is built. + +## 2. Superseded versions (still served; do not use) + +Each suffix bump was a real fix that left its predecessor in place — the #326 +incident ("two sibling files silently diverged in vintage") is what happens when +nothing marks the old ones dead. That marking is this table. + +| Superseded file | Replaced by | Why (the bump's fix) | +|---|---|---| +| `isamples_202608_sample_facets.parquet` | `_v3` → `_v4` | v3: concept labels folded into search text (#277); v4: place_name folded in (#326) | +| `isamples_202608_sample_facets_v3.parquet` | `_v4` | place_name search (#326) — **the file whose staleness caused the #326 bug** | +| `isamples_202608_samples_map_lite.parquet` | `_v2` → `_v3` | v2: h3_res4/res6 columns for filtered clusters (#300); v3: corrected place_name (#311) | +| `isamples_202608_samples_map_lite_v2.parquet` | `_v3` | corrected place_name (#311) | + +**Policy going forward:** a suffix bump MUST add a row here and regenerate the +release manifest in the same change. (The manifest makes violations visible at +boot; this table makes them legible to humans.) + +## 3. Orphan snapshot generations (serve no one) + +| Generation | Status | +|---|---| +| `isamples_202512_*` | Narrow only — **archival** (in the Zenodo deposit; there is no 202601 narrow — known label mismatch, documented in the deposit README) | +| `isamples_202601_*` | **Archival** — the coherent snapshot in Zenodo draft 21288719 | +| `isamples_202604_*` | Orphan work-in-progress cut. Nothing references it. Post-grant: move to `/attic/` | +| `isamples_202606_*` | Orphan work-in-progress cut. Same disposition | +| `isamples_202608_*` | **Live** (production Explorer) | + +## 4. Conveniences (not load-bearing; fine to keep, labeled) + +| File | Note | +|---|---| +| `isamples_202601_wide_h3.parquet` | Analyst convenience (wide + precomputed H3 cells). In the Zenodo deposit; NOT loaded by the Explorer | +| `isamples_202601_oc_sidecar.parquet` | OpenContext supplemental fields experiment (#140 lineage) | +| `*.csv` twins of lite/H3 (~640 MB) | Convenience exports; parquet is authoritative; excluded from Zenodo by design | + +## 5. Post-grant simplification options (deliberately NOT done in-grant) + +1. Move §2 + §3-orphan objects to an `/attic/` prefix on R2 (don't delete — old + notebooks/links may reference them). Needs R2 write credentials. +2. Collapse the facet family only alongside a real pipeline rework — the current + 9 files trade storage (cheap) for browser CPU (scarce); collapsing them is an + engineering project, not cleanup. +3. If a fresher export ever lands (#320), cut ONE coherent generation and retire + 202608 wholesale via the manifest. + From cc58bb14c01659abbb19252c8b4eaa5ecebf5bfa Mon Sep 17 00:00:00 2001 From: Raymond Yee Date: Thu, 23 Jul 2026 10:04:21 -0700 Subject: [PATCH 2/6] release manifest builder + generated 202608 manifest (#334 v0, chunk 2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Machine-readable twin of CANONICAL.md: 20 canonical objects (16 parquet + 4 search-index sidecars), size/etag via 1-byte ranged GET against the live origin (the Worker 403s HEAD and the default Python UA — script identifies honestly). Fail-closed: any missing file aborts emission. Manifest is committed into the site repo so it deploys atomically with the code whose pinned URLs it describes; the boot cross-check (chunk 3) fetches it same-origin. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Nohddj7oXLnkJ4P9Cibg8Y --- isamples_202608_release_manifest.json | 114 +++++++++++++++++++++ tools/build_release_manifest.py | 138 ++++++++++++++++++++++++++ 2 files changed, 252 insertions(+) create mode 100644 isamples_202608_release_manifest.json create mode 100644 tools/build_release_manifest.py diff --git a/isamples_202608_release_manifest.json b/isamples_202608_release_manifest.json new file mode 100644 index 0000000..9ff2dd1 --- /dev/null +++ b/isamples_202608_release_manifest.json @@ -0,0 +1,114 @@ +{ + "schema_version": 1, + "release_id": "isamples_202608", + "generated_at_utc": "2026-07-23T17:04:01+00:00", + "base": "https://data.isamples.org", + "files": { + "isamples_202608_wide.parquet": { + "size_bytes": 300303095, + "etag": "1ef84eef3618a9f9d7f140adc42d543c", + "last_modified": null + }, + "isamples_202608_samples_map_lite_v3.parquet": { + "size_bytes": 62924115, + "etag": "36849864596413e56688378dc83582dc-2", + "last_modified": null + }, + "isamples_202608_sample_facets_v4.parquet": { + "size_bytes": 69384530, + "etag": "29b7fcff8683ac3a937db9550bdc8d37-9", + "last_modified": null + }, + "isamples_202608_h3_summary_res4.parquet": { + "size_bytes": 505651, + "etag": "e2364e810bc5fda6b613bdc8f3fb947f", + "last_modified": null + }, + "isamples_202608_h3_summary_res6.parquet": { + "size_bytes": 1322875, + "etag": "65200a3ce894d7e3ea16560784c5c9f6", + "last_modified": null + }, + "isamples_202608_h3_summary_res8.parquet": { + "size_bytes": 2009510, + "etag": "f6410fa139a75a48fc6db1d6f24d76a6", + "last_modified": null + }, + "isamples_202608_facet_summaries.parquet": { + "size_bytes": 1689, + "etag": "6498e8f60113f973a9519fe647860d54", + "last_modified": null + }, + "isamples_202608_facet_cross_filter.parquet": { + "size_bytes": 5862, + "etag": "da81a1eaaf97c109cef279128e1bbbd5", + "last_modified": null + }, + "isamples_202608_facet_tree_summaries.parquet": { + "size_bytes": 2773, + "etag": "8631b499654eb19b356f30bd1cbc10f7", + "last_modified": null + }, + "isamples_202608_facet_tree_cross_filter.parquet": { + "size_bytes": 7102, + "etag": "fe9155ec83548e873c29657ae569ae7e", + "last_modified": null + }, + "isamples_202608_sample_facet_membership.parquet": { + "size_bytes": 30908543, + "etag": "24d5021eda3c1b75060e265c18da36bd-4", + "last_modified": null + }, + "isamples_202608_sample_facet_masks.parquet": { + "size_bytes": 10138648, + "etag": "953b19a368d21660c9bca7219811cf3d", + "last_modified": null + }, + "isamples_202608_facet_node_bits.parquet": { + "size_bytes": 1839, + "etag": "687e2ee9e72184c1affb293aec861850", + "last_modified": null + }, + "isamples_202608_sample_facet_index.parquet": { + "size_bytes": 10152968, + "etag": "818b43b4c5ee5bbb06ffe1050a3bbcd7", + "last_modified": null + }, + "isamples_202608_sample_facet_index_meta.parquet": { + "size_bytes": 1154, + "etag": "2e7686fd6f008666cbde31dc047f344c", + "last_modified": null + }, + "vocab_labels_202608.parquet": { + "size_bytes": 58783, + "etag": "4db27051fc74301140795b66ff39ab08", + "last_modified": null + }, + "isamples_202608_search_index_v1/build_stats.json": { + "size_bytes": 1980, + "etag": "a44862c90b538fec751fd03d372e2e1d", + "last_modified": null + }, + "isamples_202608_search_index_v1/hot_tokens.json": { + "size_bytes": 17747, + "etag": "66c8fb0ea713915085dc12036d7e6110", + "last_modified": null + }, + "isamples_202608_search_index_v1/shard_sizes.json": { + "size_bytes": 7933, + "etag": "488bb3c2797b81d036e8d6431ee23cf0", + "last_modified": null + }, + "isamples_202608_search_index_v1/df.parquet": { + "size_bytes": 7990281, + "etag": "a42de35cafeeb8bfae58c91ef09023e9", + "last_modified": null + } + }, + "search_index": { + "path": "isamples_202608_search_index_v1/", + "object_count_expected": 852, + "note": "sidecars pinned above; shards validate via build_stats.json" + }, + "docs": "CANONICAL.md (human twin); #334 v0-detect" +} diff --git a/tools/build_release_manifest.py b/tools/build_release_manifest.py new file mode 100644 index 0000000..17dbf2d --- /dev/null +++ b/tools/build_release_manifest.py @@ -0,0 +1,138 @@ +#!/usr/bin/env python3 +"""build_release_manifest.py — emit the canonical release manifest (#334 v0-detect). + +The manifest is the machine-readable twin of CANONICAL.md: it enumerates exactly +the data files the Explorer is entitled to load for a release, with per-file +size/etag/last-modified gathered from HTTP HEAD against the live origin (no +downloads). The Explorer fetches this at boot and cross-checks its pinned URLs +(v0 = DETECT divergence loudly; deriving URLs from the manifest is v1, post-grant +— see #334). + +Usage: + python3 tools/build_release_manifest.py # print JSON + python3 tools/build_release_manifest.py --out isamples_202608_release_manifest.json + python3 tools/build_release_manifest.py --base https://data.isamples.org + +Fail-closed: any missing file or non-200 HEAD aborts with exit 1 — a manifest +must never be generated from a broken origin. (Policy learned 2026-07-18.) + +""" +import argparse +import datetime +import json +import sys +import urllib.request + +RELEASE_ID = "isamples_202608" + +# The canonical set. Keep in lockstep with CANONICAL.md §1 — a suffix bump edits +# BOTH in the same change (policy in CANONICAL.md §2). +CANONICAL_FILES = [ + "isamples_202608_wide.parquet", + "isamples_202608_samples_map_lite_v3.parquet", + "isamples_202608_sample_facets_v4.parquet", + "isamples_202608_h3_summary_res4.parquet", + "isamples_202608_h3_summary_res6.parquet", + "isamples_202608_h3_summary_res8.parquet", + "isamples_202608_facet_summaries.parquet", + "isamples_202608_facet_cross_filter.parquet", + "isamples_202608_facet_tree_summaries.parquet", + "isamples_202608_facet_tree_cross_filter.parquet", + "isamples_202608_sample_facet_membership.parquet", + "isamples_202608_sample_facet_masks.parquet", + "isamples_202608_facet_node_bits.parquet", + "isamples_202608_sample_facet_index.parquet", + "isamples_202608_sample_facet_index_meta.parquet", + "vocab_labels_202608.parquet", +] + +# The search index is a directory of ~852 objects; the manifest pins its +# self-describing sidecars (the index validates itself via build_stats). +SEARCH_INDEX_SIDECARS = [ + "isamples_202608_search_index_v1/build_stats.json", + "isamples_202608_search_index_v1/hot_tokens.json", + "isamples_202608_search_index_v1/shard_sizes.json", + "isamples_202608_search_index_v1/df.parquet", +] + + +def head(base, name): + """Probe a file without downloading it. + + The data.isamples.org Worker 403s HEAD, so use a 1-byte ranged GET (the + same primitive DuckDB-WASM relies on) and read the total from + Content-Range: "bytes 0-0/TOTAL". + """ + # data.isamples.org 403s the default Python-urllib User-Agent (UA-based bot + # filtering); identify honestly instead. + req = urllib.request.Request(f"{base}/{name}", headers={ + "Range": "bytes=0-0", + "User-Agent": "isamples-release-manifest-builder/1.0 (+https://github.com/isamplesorg/isamplesorg.github.io)", + }) + with urllib.request.urlopen(req, timeout=30) as r: + if r.status not in (200, 206): + raise RuntimeError(f"HTTP {r.status} for {name}") + h = r.headers + size = None + cr = h.get("Content-Range", "") + if "/" in cr: + tail = cr.rsplit("/", 1)[-1] + if tail.isdigit(): + size = int(tail) + if size is None: # origin ignored Range and returned the whole body header + cl = h.get("Content-Length") + if cl and cl.isdigit() and r.status == 200: + size = int(cl) + return { + "size_bytes": size, + "etag": (h.get("ETag") or "").strip('"') or None, + "last_modified": h.get("Last-Modified"), + } + + +def main(): + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap.add_argument("--base", default="https://data.isamples.org") + ap.add_argument("--out", metavar="FILE") + args = ap.parse_args() + + files, errors = {}, [] + for name in CANONICAL_FILES + SEARCH_INDEX_SIDECARS: + try: + meta = head(args.base, name) + if meta["size_bytes"] in (None, 0): + errors.append(f"{name}: missing/zero Content-Length") + files[name] = meta + except Exception as e: # noqa: BLE001 + errors.append(f"{name}: {e}") + + if errors: + print("REFUSING to emit manifest — origin problems:", file=sys.stderr) + for e in errors: + print(f" - {e}", file=sys.stderr) + sys.exit(1) + + manifest = { + "schema_version": 1, + "release_id": RELEASE_ID, + "generated_at_utc": datetime.datetime.now(datetime.timezone.utc) + .isoformat(timespec="seconds"), + "base": args.base, + "files": files, + "search_index": { + "path": "isamples_202608_search_index_v1/", + "object_count_expected": 852, + "note": "sidecars pinned above; shards validate via build_stats.json", + }, + "docs": "CANONICAL.md (human twin); #334 v0-detect", + } + text = json.dumps(manifest, indent=1) + print(text) + if args.out: + with open(args.out, "w") as f: + f.write(text + "\n") + print(f"wrote {args.out}", file=sys.stderr) + + +if __name__ == "__main__": + main() From 162a2531bbc89c8c4d95362e14612abac4dbdd6a Mon Sep 17 00:00:00 2001 From: Raymond Yee Date: Thu, 23 Jul 2026 10:05:45 -0700 Subject: [PATCH 3/6] #334 v0-detect: Explorer cross-checks pinned URLs against the release manifest New OJS cell (depends only on the URL cells; non-blocking, never throws): fetches the same-origin release manifest and verifies every pinned data URL appears in it. Mismatch -> console.error + dismissible amber banner ('data release could not be validated'); missing manifest -> console warning only. Converts the #326 silent-divergence class into caught-at-next-boot. window.__manifestCheck exposes the result for specs. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Nohddj7oXLnkJ4P9Cibg8Y --- explorer.qmd | 68 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) diff --git a/explorer.qmd b/explorer.qmd index 9050b79..4c1a2f7 100644 --- a/explorer.qmd +++ b/explorer.qmd @@ -912,6 +912,74 @@ index_url = `${R2_BASE}/isamples_202608_sample_facet_index.parquet` // runs. Always deployed paired with index_url (same build_id). index_meta_url = `${R2_BASE}/isamples_202608_sample_facet_index_meta.parquet` +// === #334 v0-detect: release-manifest boot cross-check === +// The release manifest (isamples_202608_release_manifest.json, committed in +// this repo and deployed with the site; generated by +// tools/build_release_manifest.py; human twin: CANONICAL.md) enumerates the +// canonical data objects for this release. This cell cross-checks every URL +// the page pins against that list and complains LOUDLY on divergence — +// converting the #326 failure class ("sibling data files silently diverged +// for weeks") into "caught at next boot". v0 = detect only; deriving URLs +// FROM the manifest is the post-grant v1 (see #334). +// Non-blocking by design: runs after boot-critical cells (depends only on the +// URL cells), never throws, and a missing manifest is a console warning, not +// a user-facing failure. +manifestCheck = { + const pinned = [ + h3_res4_url, h3_res6_url, h3_res8_url, lite_url, wide_url, facets_url, + facet_summaries_url, cross_filter_url, vocab_labels_url, facet_tree_url, + membership_url, tree_cross_filter_url, masks_url, node_bits_url, + index_url, index_meta_url, + `${R2_BASE}/isamples_202608_search_index_v1/build_stats.json`, + `${R2_BASE}/isamples_202608_search_index_v1/hot_tokens.json`, + `${R2_BASE}/isamples_202608_search_index_v1/shard_sizes.json`, + `${R2_BASE}/isamples_202608_search_index_v1/df.parquet`, + ]; + const prefix = `${R2_BASE}/`; + const pinnedNames = pinned.map(u => u.startsWith(prefix) ? u.slice(prefix.length) : u); + const result = { status: 'unchecked', missing: [], extra: [], release_id: null }; + try { + const resp = await fetch(new URL('isamples_202608_release_manifest.json', document.baseURI)); + if (!resp.ok) { + console.warn(`#334 manifest check: manifest fetch HTTP ${resp.status} — skipping`); + result.status = 'manifest-unavailable'; + } else { + const manifest = await resp.json(); + result.release_id = manifest.release_id; + const manifestNames = new Set(Object.keys(manifest.files || {})); + result.missing = pinnedNames.filter(n => !manifestNames.has(n)); + result.extra = [...manifestNames].filter(n => !pinnedNames.includes(n)); + result.status = result.missing.length ? 'MISMATCH' : 'ok'; + if (result.missing.length) { + console.error('#334 RELEASE-MANIFEST MISMATCH — the page pins data files ' + + 'absent from the release manifest (stale manifest, or an unvetted ' + + 'suffix bump — see CANONICAL.md policy):', result.missing); + const banner = document.createElement('div'); + banner.id = 'manifestMismatchBanner'; + banner.style.cssText = 'position:sticky;top:0;z-index:9999;background:#8a6d00;' + + 'color:#fff;padding:6px 12px;font:13px sans-serif;cursor:pointer;'; + banner.textContent = 'Data release could not be validated — some data files ' + + 'are not in the release manifest. Results may be inconsistent. ' + + '(Details in the browser console; click to dismiss.)'; + banner.onclick = () => banner.remove(); + document.body.prepend(banner); + } else if (result.extra.length) { + // Informational only: the manifest may legitimately list objects this + // page doesn't load (e.g. future additions). Not a divergence signal. + console.info('#334 manifest check ok; manifest lists unpinned entries:', result.extra); + } else { + console.info(`#334 manifest check ok — ${pinnedNames.length} pinned URLs ` + + `all present in release ${manifest.release_id}`); + } + } + } catch (e) { + console.warn('#334 manifest check failed (non-fatal):', e); + result.status = 'check-error'; + } + if (typeof window !== 'undefined') window.__manifestCheck = result; + return result; +} + // Canonical palette — see issue #113. Path-relative so this works under // both isamples.org (custom domain at root) and project-pages fork // previews (rdhyee.github.io/isamplesorg.github.io/...). From a4d843fe79c908615104a9a47ec1996fc003b962 Mon Sep 17 00:00:00 2001 From: Raymond Yee Date: Thu, 23 Jul 2026 10:31:08 -0700 Subject: [PATCH 4/6] #334 v0, Codex round 1: deploy the manifest; close the search-index gap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - _quarto.yml: manifest added to project.resources (it would never have deployed — detection dead on arrival) - search_index_base: single top-level cell shared by the runtime substrate path AND the manifest check; check also compares manifest.search_index.path so a _v1->_v2 bump with a stale manifest trips the banner - check pins hot_topk.parquet (runtime-loaded), drops df.parquet (offline-only per contract); audits hrefs from the DOM - builder: strict probe (206 + exact Content-Range + 1 byte + ETag; --permissive for dev), shard inventory derived from shard_sizes.json (256 shards, not the hard-coded 852) - banner idempotent across cell re-evaluations - CANONICAL.md: 202604 correctly marked as backing the live /current/ alias (NOT movable); 202606 as documented pipeline intermediate; suffix history corrected to actual commits; nonexistent oc_sidecar row removed Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Nohddj7oXLnkJ4P9Cibg8Y --- CANONICAL.md | 17 +++-- _quarto.yml | 1 + explorer.qmd | 39 +++++++++-- isamples_202608_release_manifest.json | 21 +++++- tools/build_release_manifest.py | 94 +++++++++++++++++++-------- 5 files changed, 125 insertions(+), 47 deletions(-) diff --git a/CANONICAL.md b/CANONICAL.md index dee6af7..97b01f7 100644 --- a/CANONICAL.md +++ b/CANONICAL.md @@ -32,7 +32,7 @@ Exactly what production `explorer.qmd` loads, all under `https://data.isamples.o | `isamples_202608_sample_facet_index.parquet` | Per-sample facet index (multi-filter fast path) | | `isamples_202608_sample_facet_index_meta.parquet` | ~1 KB trusted manifest for the above (#313/#317 boot-race fix) | | `vocab_labels_202608.parquet` | URI → human label (539 entries; verified complete 2026-07-22) | -| `isamples_202608_search_index_v1/` (852 objects) | Sharded FTS index — the default search since 2026-07-17 (+ sidecars `hot_tokens.json`, `shard_sizes.json`, `df.parquet`, `build_stats.json`) | +| `isamples_202608_search_index_v1/` | Sharded FTS index — the default search since 2026-07-17. Runtime loads: `build_stats.json`, `hot_tokens.json`, `shard_sizes.json`, `hot_topk.parquet`, + 256 token shards (inventory in `shard_sizes.json`); `df.parquet` is offline-only | The ~9-file facet family looks baroque but is load-bearing: it is the price of fast multi-filter counts with no server. See `EXPLORER_QUERIES.md` for how each @@ -46,23 +46,23 @@ nothing marks the old ones dead. That marking is this table. | Superseded file | Replaced by | Why (the bump's fix) | |---|---|---| -| `isamples_202608_sample_facets.parquet` | `_v3` → `_v4` | v3: concept labels folded into search text (#277); v4: place_name folded in (#326) | -| `isamples_202608_sample_facets_v3.parquet` | `_v4` | place_name search (#326) — **the file whose staleness caused the #326 bug** | -| `isamples_202608_samples_map_lite.parquet` | `_v2` → `_v3` | v2: h3_res4/res6 columns for filtered clusters (#300); v3: corrected place_name (#311) | -| `isamples_202608_samples_map_lite_v2.parquet` | `_v3` | corrected place_name (#311) | +| `isamples_202608_sample_facets_v2.parquet` | `_v3` | concept labels folded into search text (#277; v2 introduced in `10fd415`, bumped in `8a6ebf5`) | +| `isamples_202608_sample_facets_v3.parquet` | `_v4` | place_name folded into search text (#326, `7e900a0`) — **the file whose staleness caused the #326 bug** | +| `isamples_202608_samples_map_lite.parquet` | `_v2` | h3_res4/res6 columns for filtered clusters (#300, `1715142`) | +| `isamples_202608_samples_map_lite_v2.parquet` | `_v3` | corrected place_name + Array.isArray fix (#311, `3f2f20a`) | **Policy going forward:** a suffix bump MUST add a row here and regenerate the release manifest in the same change. (The manifest makes violations visible at boot; this table makes them legible to humans.) -## 3. Orphan snapshot generations (serve no one) +## 3. Snapshot generations (who serves whom) | Generation | Status | |---|---| | `isamples_202512_*` | Narrow only — **archival** (in the Zenodo deposit; there is no 202601 narrow — known label mismatch, documented in the deposit README) | | `isamples_202601_*` | **Archival** — the coherent snapshot in Zenodo draft 21288719 | -| `isamples_202604_*` | Orphan work-in-progress cut. Nothing references it. Post-grant: move to `/attic/` | -| `isamples_202606_*` | Orphan work-in-progress cut. Same disposition | +| `isamples_202604_*` | **NOT orphan — backs the stable `/current/wide.parquet` alias** used by data.qmd and multiple tutorials (see SERIALIZATIONS.md). Must NOT be moved until the alias is repointed and consumers audited | +| `isamples_202606_*` | Pipeline intermediate (the #272 OC-concept-enriched wide that fed 202608 — see DATA_PROVENANCE.md step 3b). Not loaded by anything at runtime; safe to attic post-grant, lineage documented | | `isamples_202608_*` | **Live** (production Explorer) | ## 4. Conveniences (not load-bearing; fine to keep, labeled) @@ -70,7 +70,6 @@ boot; this table makes them legible to humans.) | File | Note | |---|---| | `isamples_202601_wide_h3.parquet` | Analyst convenience (wide + precomputed H3 cells). In the Zenodo deposit; NOT loaded by the Explorer | -| `isamples_202601_oc_sidecar.parquet` | OpenContext supplemental fields experiment (#140 lineage) | | `*.csv` twins of lite/H3 (~640 MB) | Convenience exports; parquet is authoritative; excluded from Zenodo by design | ## 5. Post-grant simplification options (deliberately NOT done in-grant) diff --git a/_quarto.yml b/_quarto.yml index adfe6c9..b881bc5 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -2,6 +2,7 @@ project: type: website output-dir: docs resources: + - isamples_202608_release_manifest.json - assets/js/source-palette.js - assets/js/sql-builders.js - assets/js/explorer-utils.js diff --git a/explorer.qmd b/explorer.qmd index 4c1a2f7..681bfda 100644 --- a/explorer.qmd +++ b/explorer.qmd @@ -912,6 +912,11 @@ index_url = `${R2_BASE}/isamples_202608_sample_facet_index.parquet` // runs. Always deployed paired with index_url (same build_id). index_meta_url = `${R2_BASE}/isamples_202608_sample_facet_index_meta.parquet` +// Single source of truth for the search-index location (#334 round-1 review): +// the runtime substrate path and the manifest check MUST share this cell, so a +// version bump can't diverge them silently. +search_index_base = `${R2_BASE}/isamples_202608_search_index_v1` + // === #334 v0-detect: release-manifest boot cross-check === // The release manifest (isamples_202608_release_manifest.json, committed in // this repo and deployed with the site; generated by @@ -930,13 +935,24 @@ manifestCheck = { facet_summaries_url, cross_filter_url, vocab_labels_url, facet_tree_url, membership_url, tree_cross_filter_url, masks_url, node_bits_url, index_url, index_meta_url, - `${R2_BASE}/isamples_202608_search_index_v1/build_stats.json`, - `${R2_BASE}/isamples_202608_search_index_v1/hot_tokens.json`, - `${R2_BASE}/isamples_202608_search_index_v1/shard_sizes.json`, - `${R2_BASE}/isamples_202608_search_index_v1/df.parquet`, + // Runtime search-index loads, via the SHARED base cell (df.parquet is + // offline-only per SEARCH_INDEX_V1.md and deliberately not pinned here): + `${search_index_base}/build_stats.json`, + `${search_index_base}/hot_tokens.json`, + `${search_index_base}/shard_sizes.json`, + `${search_index_base}/hot_topk.parquet`, ]; + // Hard-coded hrefs in the page header are a separate + // divergence surface (round-1 P2): audit whatever is actually in the DOM. + for (const l of document.querySelectorAll('link[rel="preload"][as="fetch"]')) { + if (l.href && l.href.includes('data.isamples.org')) pinned.push(l.href); + } const prefix = `${R2_BASE}/`; - const pinnedNames = pinned.map(u => u.startsWith(prefix) ? u.slice(prefix.length) : u); + const pinnedNames = [...new Set(pinned.map(u => { + if (u.startsWith(prefix)) return u.slice(prefix.length); + const m = u.match(/data\.isamples\.org\/(.+)$/); + return m ? m[1] : u; + }))]; const result = { status: 'unchecked', missing: [], extra: [], release_id: null }; try { const resp = await fetch(new URL('isamples_202608_release_manifest.json', document.baseURI)); @@ -947,9 +963,18 @@ manifestCheck = { const manifest = await resp.json(); result.release_id = manifest.release_id; const manifestNames = new Set(Object.keys(manifest.files || {})); - result.missing = pinnedNames.filter(n => !manifestNames.has(n)); + // The search-index PATH must also agree (a SUBSTRATE_BASE _v1 -> _v2 bump + // with a stale manifest must trip the check, not slide by). + const mPath = ((manifest.search_index || {}).path || '').replace(/\/$/, ''); + const localPath = search_index_base.startsWith(prefix) + ? search_index_base.slice(prefix.length) : search_index_base; + if (mPath && mPath !== localPath) result.missing.push(`search_index path: page=${localPath} manifest=${mPath}`); + result.missing = result.missing.concat(pinnedNames.filter(n => !manifestNames.has(n))); result.extra = [...manifestNames].filter(n => !pinnedNames.includes(n)); result.status = result.missing.length ? 'MISMATCH' : 'ok'; + // Idempotency (round-1 P2): re-evaluations must not stack banners, and a + // clean re-run must clear a stale one. + document.getElementById('manifestMismatchBanner')?.remove(); if (result.missing.length) { console.error('#334 RELEASE-MANIFEST MISMATCH — the page pins data files ' + 'absent from the release manifest (stale manifest, or an unvetted ' @@ -5983,7 +6008,7 @@ zoomWatcher = { // SME A/B ranking comparison (see #172); `?fts=v1` remains accepted so // pre-flip opt-in links behave identically. §7 end-to-end budgets stay // open in #172 as the pin-overlay / downstream-latency track. - const SUBSTRATE_BASE = `${R2_BASE}/isamples_202608_search_index_v1`; + const SUBSTRATE_BASE = search_index_base; // shared cell — see #334 manifest check let _substrateMeta = null; function substrateEnabled() { diff --git a/isamples_202608_release_manifest.json b/isamples_202608_release_manifest.json index 9ff2dd1..c161a71 100644 --- a/isamples_202608_release_manifest.json +++ b/isamples_202608_release_manifest.json @@ -1,7 +1,7 @@ { "schema_version": 1, "release_id": "isamples_202608", - "generated_at_utc": "2026-07-23T17:04:01+00:00", + "generated_at_utc": "2026-07-23T17:29:53+00:00", "base": "https://data.isamples.org", "files": { "isamples_202608_wide.parquet": { @@ -99,6 +99,11 @@ "etag": "488bb3c2797b81d036e8d6431ee23cf0", "last_modified": null }, + "isamples_202608_search_index_v1/hot_topk.parquet": { + "size_bytes": 690750, + "etag": "a88d1228baa0b4c2ce7bcb9c1dede7bb", + "last_modified": null + }, "isamples_202608_search_index_v1/df.parquet": { "size_bytes": 7990281, "etag": "a42de35cafeeb8bfae58c91ef09023e9", @@ -107,8 +112,18 @@ }, "search_index": { "path": "isamples_202608_search_index_v1/", - "object_count_expected": 852, - "note": "sidecars pinned above; shards validate via build_stats.json" + "shard_count": 256, + "shard_bytes_total": 551211162, + "runtime_sidecars": [ + "build_stats.json", + "hot_tokens.json", + "shard_sizes.json", + "hot_topk.parquet" + ], + "offline_only": [ + "df.parquet" + ], + "note": "shard inventory derived from shard_sizes.json at generation time" }, "docs": "CANONICAL.md (human twin); #334 v0-detect" } diff --git a/tools/build_release_manifest.py b/tools/build_release_manifest.py index 17dbf2d..972f846 100644 --- a/tools/build_release_manifest.py +++ b/tools/build_release_manifest.py @@ -52,60 +52,94 @@ "isamples_202608_search_index_v1/build_stats.json", "isamples_202608_search_index_v1/hot_tokens.json", "isamples_202608_search_index_v1/shard_sizes.json", + "isamples_202608_search_index_v1/hot_topk.parquet", # loaded by the topk query path + # df.parquet is OFFLINE-ONLY per SEARCH_INDEX_V1.md — archived, not runtime; + # it is probed for existence but excluded from the Explorer's runtime check. "isamples_202608_search_index_v1/df.parquet", ] -def head(base, name): - """Probe a file without downloading it. +UA = "isamples-release-manifest-builder/1.0 (+https://github.com/isamplesorg/isamplesorg.github.io)" - The data.isamples.org Worker 403s HEAD, so use a 1-byte ranged GET (the - same primitive DuckDB-WASM relies on) and read the total from - Content-Range: "bytes 0-0/TOTAL". + +def head(base, name, permissive=False): + """Probe a file without downloading it (1-byte ranged GET). + + The data.isamples.org Worker 403s HEAD and the default Python UA, so we use + a ranged GET with an honest UA — the same primitive DuckDB-WASM relies on. + STRICT by default (Codex P1): require 206 + exact Content-Range + one byte + + an ETag. Anything else (200-with-HTML proxy page, broken Range support — + the very capability the Explorer needs, missing integrity metadata) refuses. + --permissive relaxes for dev mirrors. """ - # data.isamples.org 403s the default Python-urllib User-Agent (UA-based bot - # filtering); identify honestly instead. req = urllib.request.Request(f"{base}/{name}", headers={ - "Range": "bytes=0-0", - "User-Agent": "isamples-release-manifest-builder/1.0 (+https://github.com/isamplesorg/isamplesorg.github.io)", + "Range": "bytes=0-0", "User-Agent": UA, }) with urllib.request.urlopen(req, timeout=30) as r: - if r.status not in (200, 206): - raise RuntimeError(f"HTTP {r.status} for {name}") + body = r.read() h = r.headers - size = None + etag = (h.get("ETag") or "").strip('"') or None cr = h.get("Content-Range", "") - if "/" in cr: - tail = cr.rsplit("/", 1)[-1] - if tail.isdigit(): - size = int(tail) - if size is None: # origin ignored Range and returned the whole body header + size = None + m = cr.split("/")[-1] if "/" in cr else "" + if m.isdigit(): + size = int(m) + if not permissive: + if r.status != 206: + raise RuntimeError(f"{name}: expected 206, got {r.status} (Range support broken?)") + if not cr.startswith("bytes 0-0/") or size is None: + raise RuntimeError(f"{name}: bad Content-Range {cr!r}") + if len(body) != 1: + raise RuntimeError(f"{name}: expected 1 byte, got {len(body)}") + if not etag: + raise RuntimeError(f"{name}: missing ETag (no integrity metadata)") + elif size is None: cl = h.get("Content-Length") - if cl and cl.isdigit() and r.status == 200: - size = int(cl) - return { - "size_bytes": size, - "etag": (h.get("ETag") or "").strip('"') or None, - "last_modified": h.get("Last-Modified"), - } + size = int(cl) if cl and cl.isdigit() and r.status == 200 else None + return {"size_bytes": size, "etag": etag, "last_modified": h.get("Last-Modified")} + + +def fetch_json(base, name): + req = urllib.request.Request(f"{base}/{name}", headers={"User-Agent": UA}) + with urllib.request.urlopen(req, timeout=30) as r: + if r.status != 200: + raise RuntimeError(f"HTTP {r.status} for {name}") + return json.loads(r.read().decode()) def main(): ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) ap.add_argument("--base", default="https://data.isamples.org") ap.add_argument("--out", metavar="FILE") + ap.add_argument("--permissive", action="store_true", + help="relax probe strictness for dev mirrors") args = ap.parse_args() files, errors = {}, [] for name in CANONICAL_FILES + SEARCH_INDEX_SIDECARS: try: - meta = head(args.base, name) + meta = head(args.base, name, permissive=args.permissive) if meta["size_bytes"] in (None, 0): - errors.append(f"{name}: missing/zero Content-Length") + errors.append(f"{name}: missing/zero size") files[name] = meta except Exception as e: # noqa: BLE001 errors.append(f"{name}: {e}") + # Shard inventory from the index's own shard_sizes.json (Codex P1: the + # object count must come from data, not a hard-coded 852). + shard_count, shard_bytes = None, None + try: + sizes = fetch_json(args.base, "isamples_202608_search_index_v1/shard_sizes.json") + entries = sizes if isinstance(sizes, dict) else {} + if not entries: + errors.append("shard_sizes.json: empty/unexpected shape") + else: + shard_count = len(entries) + vals = list(entries.values()) + shard_bytes = sum(v for v in vals if isinstance(v, int)) or None + except Exception as e: # noqa: BLE001 + errors.append(f"shard_sizes.json inventory: {e}") + if errors: print("REFUSING to emit manifest — origin problems:", file=sys.stderr) for e in errors: @@ -121,8 +155,12 @@ def main(): "files": files, "search_index": { "path": "isamples_202608_search_index_v1/", - "object_count_expected": 852, - "note": "sidecars pinned above; shards validate via build_stats.json", + "shard_count": shard_count, + "shard_bytes_total": shard_bytes, + "runtime_sidecars": ["build_stats.json", "hot_tokens.json", + "shard_sizes.json", "hot_topk.parquet"], + "offline_only": ["df.parquet"], + "note": "shard inventory derived from shard_sizes.json at generation time", }, "docs": "CANONICAL.md (human twin); #334 v0-detect", } From 1741ffaedb78330c8104bd0f2e9bd23b31efab89 Mon Sep 17 00:00:00 2001 From: Raymond Yee Date: Thu, 23 Jul 2026 10:43:12 -0700 Subject: [PATCH 5/6] #334 v0, Codex round 2: full shard inventory + probe order + doc accuracy - Shard inventory: base=256 (shard_sizes.json) + hot sub-files=591 (build_stats.json shard_files=847 physical vs shard_count=256 logical - the distinction round 2 surfaced) = 852 objects with sidecars; the builder cross-validates logical count between both artifacts - Probe validates status + Content-Range (strict fullmatch) BEFORE reading the body (bounded read(2)) so a Range-ignoring origin can't force a 300 MB download - Header text now matches reality (ranged GET, not HEAD); CANONICAL's attic sentence names its targets instead of a stale section ref Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Nohddj7oXLnkJ4P9Cibg8Y --- CANONICAL.md | 4 +- isamples_202608_release_manifest.json | 10 +++-- tools/build_release_manifest.py | 61 ++++++++++++++++++--------- 3 files changed, 50 insertions(+), 25 deletions(-) diff --git a/CANONICAL.md b/CANONICAL.md index 97b01f7..e03a544 100644 --- a/CANONICAL.md +++ b/CANONICAL.md @@ -32,7 +32,7 @@ Exactly what production `explorer.qmd` loads, all under `https://data.isamples.o | `isamples_202608_sample_facet_index.parquet` | Per-sample facet index (multi-filter fast path) | | `isamples_202608_sample_facet_index_meta.parquet` | ~1 KB trusted manifest for the above (#313/#317 boot-race fix) | | `vocab_labels_202608.parquet` | URI → human label (539 entries; verified complete 2026-07-22) | -| `isamples_202608_search_index_v1/` | Sharded FTS index — the default search since 2026-07-17. Runtime loads: `build_stats.json`, `hot_tokens.json`, `shard_sizes.json`, `hot_topk.parquet`, + 256 token shards (inventory in `shard_sizes.json`); `df.parquet` is offline-only | +| `isamples_202608_search_index_v1/` | Sharded FTS index — the default search since 2026-07-17. Runtime loads: `build_stats.json`, `hot_tokens.json`, `shard_sizes.json`, `hot_topk.parquet`, + 256 base token shards and ~591 hot-token sub-files (852 objects total incl. sidecars; base inventory in `shard_sizes.json`, totals in `build_stats.json`); `df.parquet` is offline-only | The ~9-file facet family looks baroque but is load-bearing: it is the price of fast multi-filter counts with no server. See `EXPLORER_QUERIES.md` for how each @@ -74,7 +74,7 @@ boot; this table makes them legible to humans.) ## 5. Post-grant simplification options (deliberately NOT done in-grant) -1. Move §2 + §3-orphan objects to an `/attic/` prefix on R2 (don't delete — old +1. Move §2's superseded versions and the 202606 pipeline intermediate to an `/attic/` prefix on R2 (don't delete — old notebooks/links may reference them). Needs R2 write credentials. 2. Collapse the facet family only alongside a real pipeline rework — the current 9 files trade storage (cheap) for browser CPU (scarce); collapsing them is an diff --git a/isamples_202608_release_manifest.json b/isamples_202608_release_manifest.json index c161a71..3f0e88f 100644 --- a/isamples_202608_release_manifest.json +++ b/isamples_202608_release_manifest.json @@ -1,7 +1,7 @@ { "schema_version": 1, "release_id": "isamples_202608", - "generated_at_utc": "2026-07-23T17:29:53+00:00", + "generated_at_utc": "2026-07-23T17:43:12+00:00", "base": "https://data.isamples.org", "files": { "isamples_202608_wide.parquet": { @@ -112,8 +112,10 @@ }, "search_index": { "path": "isamples_202608_search_index_v1/", - "shard_count": 256, - "shard_bytes_total": 551211162, + "base_shard_count": 256, + "base_shard_bytes": 551211162, + "total_shard_files": 847, + "hot_shard_files": 591, "runtime_sidecars": [ "build_stats.json", "hot_tokens.json", @@ -123,7 +125,7 @@ "offline_only": [ "df.parquet" ], - "note": "shard inventory derived from shard_sizes.json at generation time" + "note": "base shards inventoried by shard_sizes.json; hot-token sub-files counted via build_stats.json shard_count" }, "docs": "CANONICAL.md (human twin); #334 v0-detect" } diff --git a/tools/build_release_manifest.py b/tools/build_release_manifest.py index 972f846..1e294f9 100644 --- a/tools/build_release_manifest.py +++ b/tools/build_release_manifest.py @@ -13,8 +13,8 @@ python3 tools/build_release_manifest.py --out isamples_202608_release_manifest.json python3 tools/build_release_manifest.py --base https://data.isamples.org -Fail-closed: any missing file or non-200 HEAD aborts with exit 1 — a manifest -must never be generated from a broken origin. (Policy learned 2026-07-18.) +Fail-closed: any missing file or non-strict probe result aborts with exit 1 — a +manifest must never be generated from a broken origin. (Policy learned 2026-07-18.) """ import argparse @@ -72,27 +72,28 @@ def head(base, name, permissive=False): the very capability the Explorer needs, missing integrity metadata) refuses. --permissive relaxes for dev mirrors. """ + import re as _re req = urllib.request.Request(f"{base}/{name}", headers={ "Range": "bytes=0-0", "User-Agent": UA, }) with urllib.request.urlopen(req, timeout=30) as r: - body = r.read() h = r.headers etag = (h.get("ETag") or "").strip('"') or None cr = h.get("Content-Range", "") - size = None - m = cr.split("/")[-1] if "/" in cr else "" - if m.isdigit(): - size = int(m) + m = _re.fullmatch(r"bytes 0-0/(\d+)", cr) + size = int(m.group(1)) if m else None if not permissive: + # Validate status + headers BEFORE touching the body, so a + # Range-ignoring origin can't make us download a 300 MB object. if r.status != 206: raise RuntimeError(f"{name}: expected 206, got {r.status} (Range support broken?)") - if not cr.startswith("bytes 0-0/") or size is None: + if size is None or size <= 0: raise RuntimeError(f"{name}: bad Content-Range {cr!r}") - if len(body) != 1: - raise RuntimeError(f"{name}: expected 1 byte, got {len(body)}") if not etag: raise RuntimeError(f"{name}: missing ETag (no integrity metadata)") + body = r.read(2) # bounded: expect exactly the 1 requested byte + if len(body) != 1: + raise RuntimeError(f"{name}: expected 1 byte, got {len(body)}") elif size is None: cl = h.get("Content-Length") size = int(cl) if cl and cl.isdigit() and r.status == 200 else None @@ -125,20 +126,42 @@ def main(): except Exception as e: # noqa: BLE001 errors.append(f"{name}: {e}") - # Shard inventory from the index's own shard_sizes.json (Codex P1: the - # object count must come from data, not a hard-coded 852). - shard_count, shard_bytes = None, None + # Shard inventory, cross-validated between the index's own artifacts + # (Codex rounds 1-2: counts come from data; shard_sizes.json covers only + # the BASE shards — hot-token sub-files are counted in build_stats.json). + inv = {"base_shard_count": None, "base_shard_bytes": None, + "total_shard_files": None, "hot_shard_files": None} try: sizes = fetch_json(args.base, "isamples_202608_search_index_v1/shard_sizes.json") entries = sizes if isinstance(sizes, dict) else {} if not entries: errors.append("shard_sizes.json: empty/unexpected shape") else: - shard_count = len(entries) - vals = list(entries.values()) - shard_bytes = sum(v for v in vals if isinstance(v, int)) or None + inv["base_shard_count"] = len(entries) + inv["base_shard_bytes"] = sum(v for v in entries.values() if isinstance(v, int)) or None except Exception as e: # noqa: BLE001 errors.append(f"shard_sizes.json inventory: {e}") + try: + stats = fetch_json(args.base, "isamples_202608_search_index_v1/build_stats.json") + # build_stats: shard_count = LOGICAL shards (256, matches shard_sizes); + # shard_files = PHYSICAL files (base + hot-token sub-files). We want files. + tot = stats.get("shard_files") + if isinstance(tot, int) and tot > 0: + inv["total_shard_files"] = tot + if inv["base_shard_count"]: + inv["hot_shard_files"] = tot - inv["base_shard_count"] + else: + errors.append("build_stats.json: missing/invalid shard_files") + logical = stats.get("shard_count") + if isinstance(logical, int) and inv["base_shard_count"] and logical != inv["base_shard_count"]: + errors.append(f"inventory contradiction: build_stats logical shards {logical} " + f"!= shard_sizes entries {inv['base_shard_count']}") + if (inv["base_shard_count"] and isinstance(tot, int) + and tot < inv["base_shard_count"]): + errors.append(f"inventory contradiction: build_stats shard_count {tot} " + f"< base shards {inv['base_shard_count']}") + except Exception as e: # noqa: BLE001 + errors.append(f"build_stats.json inventory: {e}") if errors: print("REFUSING to emit manifest — origin problems:", file=sys.stderr) @@ -155,12 +178,12 @@ def main(): "files": files, "search_index": { "path": "isamples_202608_search_index_v1/", - "shard_count": shard_count, - "shard_bytes_total": shard_bytes, + **inv, "runtime_sidecars": ["build_stats.json", "hot_tokens.json", "shard_sizes.json", "hot_topk.parquet"], "offline_only": ["df.parquet"], - "note": "shard inventory derived from shard_sizes.json at generation time", + "note": "base shards inventoried by shard_sizes.json; hot-token " + "sub-files counted via build_stats.json shard_count", }, "docs": "CANONICAL.md (human twin); #334 v0-detect", } From a2d084ca8072ef3ce492295122175c5f8c9c82c9 Mon Sep 17 00:00:00 2001 From: Raymond Yee Date: Thu, 23 Jul 2026 10:51:33 -0700 Subject: [PATCH 6/6] #334 v0, Codex round 3: hot files enumerated from source; counts hard-required - hot_shard_files now summed from hot_tokens.json tokens[*].sub_files (their authoritative declaration), not derived by subtraction - three-way cross-check: shard_sizes base + hot_tokens hot == build_stats shard_files, any disagreement or missing count refuses emission - last stale HEAD wording + shard_count/shard_files confusion in note and error text corrected Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Nohddj7oXLnkJ4P9Cibg8Y --- isamples_202608_release_manifest.json | 5 +-- tools/build_release_manifest.py | 52 +++++++++++++++++++-------- 2 files changed, 40 insertions(+), 17 deletions(-) diff --git a/isamples_202608_release_manifest.json b/isamples_202608_release_manifest.json index 3f0e88f..cdb15a7 100644 --- a/isamples_202608_release_manifest.json +++ b/isamples_202608_release_manifest.json @@ -1,7 +1,7 @@ { "schema_version": 1, "release_id": "isamples_202608", - "generated_at_utc": "2026-07-23T17:43:12+00:00", + "generated_at_utc": "2026-07-23T17:51:11+00:00", "base": "https://data.isamples.org", "files": { "isamples_202608_wide.parquet": { @@ -116,6 +116,7 @@ "base_shard_bytes": 551211162, "total_shard_files": 847, "hot_shard_files": 591, + "hot_token_count": 129, "runtime_sidecars": [ "build_stats.json", "hot_tokens.json", @@ -125,7 +126,7 @@ "offline_only": [ "df.parquet" ], - "note": "base shards inventoried by shard_sizes.json; hot-token sub-files counted via build_stats.json shard_count" + "note": "base shards from shard_sizes.json; hot sub-files enumerated from hot_tokens.json; totals cross-checked vs build_stats shard_files" }, "docs": "CANONICAL.md (human twin); #334 v0-detect" } diff --git a/tools/build_release_manifest.py b/tools/build_release_manifest.py index 1e294f9..de9f38e 100644 --- a/tools/build_release_manifest.py +++ b/tools/build_release_manifest.py @@ -3,7 +3,7 @@ The manifest is the machine-readable twin of CANONICAL.md: it enumerates exactly the data files the Explorer is entitled to load for a release, with per-file -size/etag/last-modified gathered from HTTP HEAD against the live origin (no +size/etag/last-modified gathered via strict ranged GETs against the live origin (no downloads). The Explorer fetches this at boot and cross-checks its pinned URLs (v0 = DETECT divergence loudly; deriving URLs from the manifest is v1, post-grant — see #334). @@ -144,24 +144,46 @@ def main(): try: stats = fetch_json(args.base, "isamples_202608_search_index_v1/build_stats.json") # build_stats: shard_count = LOGICAL shards (256, matches shard_sizes); - # shard_files = PHYSICAL files (base + hot-token sub-files). We want files. + # shard_files = PHYSICAL files (base + hot-token sub-files). tot = stats.get("shard_files") - if isinstance(tot, int) and tot > 0: - inv["total_shard_files"] = tot - if inv["base_shard_count"]: - inv["hot_shard_files"] = tot - inv["base_shard_count"] - else: - errors.append("build_stats.json: missing/invalid shard_files") logical = stats.get("shard_count") - if isinstance(logical, int) and inv["base_shard_count"] and logical != inv["base_shard_count"]: + if not (isinstance(tot, int) and tot > 0): + errors.append("build_stats.json: missing/invalid shard_files") + tot = None + if not (isinstance(logical, int) and logical > 0): + # Hard requirement (Codex round 3): absence must fail, not slide by. + errors.append("build_stats.json: missing/invalid shard_count (logical)") + elif inv["base_shard_count"] and logical != inv["base_shard_count"]: errors.append(f"inventory contradiction: build_stats logical shards {logical} " f"!= shard_sizes entries {inv['base_shard_count']}") - if (inv["base_shard_count"] and isinstance(tot, int) - and tot < inv["base_shard_count"]): - errors.append(f"inventory contradiction: build_stats shard_count {tot} " - f"< base shards {inv['base_shard_count']}") + inv["total_shard_files"] = tot except Exception as e: # noqa: BLE001 errors.append(f"build_stats.json inventory: {e}") + try: + # Hot sub-files enumerated from their authoritative source: each + # hot_tokens.json entry declares its physical sub_files count. + ht = fetch_json(args.base, "isamples_202608_search_index_v1/hot_tokens.json") + toks = ht.get("tokens") + if not isinstance(toks, dict) or not toks: + errors.append("hot_tokens.json: missing/empty tokens map") + else: + bad = [t for t, v in toks.items() + if not isinstance(v, dict) or not isinstance(v.get("sub_files"), int) + or v["sub_files"] < 1] + if bad: + errors.append(f"hot_tokens.json: malformed entries: {bad[:5]}") + else: + inv["hot_shard_files"] = sum(v["sub_files"] for v in toks.values()) + inv["hot_token_count"] = len(toks) + # Three-way cross-check: base (shard_sizes) + hot (hot_tokens) must + # equal physical shard_files (build_stats). Any disagreement refuses. + if (inv["base_shard_count"] and inv.get("hot_shard_files") is not None + and inv["total_shard_files"] is not None + and inv["base_shard_count"] + inv["hot_shard_files"] != inv["total_shard_files"]): + errors.append(f"inventory contradiction: base {inv['base_shard_count']} + " + f"hot {inv['hot_shard_files']} != shard_files {inv['total_shard_files']}") + except Exception as e: # noqa: BLE001 + errors.append(f"hot_tokens.json inventory: {e}") if errors: print("REFUSING to emit manifest — origin problems:", file=sys.stderr) @@ -182,8 +204,8 @@ def main(): "runtime_sidecars": ["build_stats.json", "hot_tokens.json", "shard_sizes.json", "hot_topk.parquet"], "offline_only": ["df.parquet"], - "note": "base shards inventoried by shard_sizes.json; hot-token " - "sub-files counted via build_stats.json shard_count", + "note": "base shards from shard_sizes.json; hot sub-files enumerated " + "from hot_tokens.json; totals cross-checked vs build_stats shard_files", }, "docs": "CANONICAL.md (human twin); #334 v0-detect", }