diff --git a/CANONICAL.md b/CANONICAL.md new file mode 100644 index 0000000..e03a544 --- /dev/null +++ b/CANONICAL.md @@ -0,0 +1,84 @@ +# CANONICAL.md — which data files are real, and which are accidental history + +*Created 2026-07-23 (grant closing week) to answer: "can we simplify our stable of +supplementary parquet files?" Answer: the **live set is already minimal** — the +accumulation is superseded versions and orphan snapshots around it. This file +names the canonical set and documents every non-canonical survivor so the +accident is neutralized even where files remain on the bucket.* + +**Companion:** the machine-readable release manifest +(`tools/build_release_manifest.py` → `isamples_202608_release_manifest.json`) +enumerates the same canonical set with sizes/checksums; the Explorer cross-checks +it at boot (#334 v0). A file absent from the manifest is non-canonical **by +definition**. + +## 1. The canonical (live) set — 202608 + +Exactly what production `explorer.qmd` loads, all under `https://data.isamples.org/`: + +| File | Role (plain English) | +|---|---| +| `isamples_202608_wide.parquet` | Full sample detail, one row per entity — everything derives from this | +| `isamples_202608_samples_map_lite_v3.parquet` | Slim map/table columns (coords, label, place, date, h3 cells) | +| `isamples_202608_sample_facets_v4.parquet` | Per-sample facet URIs + search-text blob (the `?fts=off` scan target) | +| `isamples_202608_h3_summary_res{4,6,8}.parquet` | Pre-counted globe clusters at 3 zoom tiers | +| `isamples_202608_facet_summaries.parquet` | Facet checkbox counts, no filters active | +| `isamples_202608_facet_cross_filter.parquet` | Facet counts under single cross-filter | +| `isamples_202608_facet_tree_summaries.parquet` | Hierarchical (tree) facet counts | +| `isamples_202608_facet_tree_cross_filter.parquet` | Tree facet counts under cross-filter | +| `isamples_202608_sample_facet_membership.parquet` | Sample↔facet-tree membership | +| `isamples_202608_sample_facet_masks.parquet` | Bitmask substrate for multi-filter counts | +| `isamples_202608_facet_node_bits.parquet` | Facet-node bit assignments for the masks | +| `isamples_202608_sample_facet_index.parquet` | Per-sample facet index (multi-filter fast path) | +| `isamples_202608_sample_facet_index_meta.parquet` | ~1 KB trusted manifest for the above (#313/#317 boot-race fix) | +| `vocab_labels_202608.parquet` | URI → human label (539 entries; verified complete 2026-07-22) | +| `isamples_202608_search_index_v1/` | Sharded FTS index — the default search since 2026-07-17. Runtime loads: `build_stats.json`, `hot_tokens.json`, `shard_sizes.json`, `hot_topk.parquet`, + 256 base token shards and ~591 hot-token sub-files (852 objects total incl. sidecars; base inventory in `shard_sizes.json`, totals in `build_stats.json`); `df.parquet` is offline-only | + +The ~9-file facet family looks baroque but is load-bearing: it is the price of +fast multi-filter counts with no server. See `EXPLORER_QUERIES.md` for how each +is queried and `DATA_PROVENANCE.md` for how each is built. + +## 2. Superseded versions (still served; do not use) + +Each suffix bump was a real fix that left its predecessor in place — the #326 +incident ("two sibling files silently diverged in vintage") is what happens when +nothing marks the old ones dead. That marking is this table. + +| Superseded file | Replaced by | Why (the bump's fix) | +|---|---|---| +| `isamples_202608_sample_facets_v2.parquet` | `_v3` | concept labels folded into search text (#277; v2 introduced in `10fd415`, bumped in `8a6ebf5`) | +| `isamples_202608_sample_facets_v3.parquet` | `_v4` | place_name folded into search text (#326, `7e900a0`) — **the file whose staleness caused the #326 bug** | +| `isamples_202608_samples_map_lite.parquet` | `_v2` | h3_res4/res6 columns for filtered clusters (#300, `1715142`) | +| `isamples_202608_samples_map_lite_v2.parquet` | `_v3` | corrected place_name + Array.isArray fix (#311, `3f2f20a`) | + +**Policy going forward:** a suffix bump MUST add a row here and regenerate the +release manifest in the same change. (The manifest makes violations visible at +boot; this table makes them legible to humans.) + +## 3. Snapshot generations (who serves whom) + +| Generation | Status | +|---|---| +| `isamples_202512_*` | Narrow only — **archival** (in the Zenodo deposit; there is no 202601 narrow — known label mismatch, documented in the deposit README) | +| `isamples_202601_*` | **Archival** — the coherent snapshot in Zenodo draft 21288719 | +| `isamples_202604_*` | **NOT orphan — backs the stable `/current/wide.parquet` alias** used by data.qmd and multiple tutorials (see SERIALIZATIONS.md). Must NOT be moved until the alias is repointed and consumers audited | +| `isamples_202606_*` | Pipeline intermediate (the #272 OC-concept-enriched wide that fed 202608 — see DATA_PROVENANCE.md step 3b). Not loaded by anything at runtime; safe to attic post-grant, lineage documented | +| `isamples_202608_*` | **Live** (production Explorer) | + +## 4. Conveniences (not load-bearing; fine to keep, labeled) + +| File | Note | +|---|---| +| `isamples_202601_wide_h3.parquet` | Analyst convenience (wide + precomputed H3 cells). In the Zenodo deposit; NOT loaded by the Explorer | +| `*.csv` twins of lite/H3 (~640 MB) | Convenience exports; parquet is authoritative; excluded from Zenodo by design | + +## 5. Post-grant simplification options (deliberately NOT done in-grant) + +1. Move §2's superseded versions and the 202606 pipeline intermediate to an `/attic/` prefix on R2 (don't delete — old + notebooks/links may reference them). Needs R2 write credentials. +2. Collapse the facet family only alongside a real pipeline rework — the current + 9 files trade storage (cheap) for browser CPU (scarce); collapsing them is an + engineering project, not cleanup. +3. If a fresher export ever lands (#320), cut ONE coherent generation and retire + 202608 wholesale via the manifest. + diff --git a/_quarto.yml b/_quarto.yml index adfe6c9..b881bc5 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -2,6 +2,7 @@ project: type: website output-dir: docs resources: + - isamples_202608_release_manifest.json - assets/js/source-palette.js - assets/js/sql-builders.js - assets/js/explorer-utils.js diff --git a/explorer.qmd b/explorer.qmd index 9050b79..681bfda 100644 --- a/explorer.qmd +++ b/explorer.qmd @@ -912,6 +912,99 @@ index_url = `${R2_BASE}/isamples_202608_sample_facet_index.parquet` // runs. Always deployed paired with index_url (same build_id). index_meta_url = `${R2_BASE}/isamples_202608_sample_facet_index_meta.parquet` +// Single source of truth for the search-index location (#334 round-1 review): +// the runtime substrate path and the manifest check MUST share this cell, so a +// version bump can't diverge them silently. +search_index_base = `${R2_BASE}/isamples_202608_search_index_v1` + +// === #334 v0-detect: release-manifest boot cross-check === +// The release manifest (isamples_202608_release_manifest.json, committed in +// this repo and deployed with the site; generated by +// tools/build_release_manifest.py; human twin: CANONICAL.md) enumerates the +// canonical data objects for this release. This cell cross-checks every URL +// the page pins against that list and complains LOUDLY on divergence — +// converting the #326 failure class ("sibling data files silently diverged +// for weeks") into "caught at next boot". v0 = detect only; deriving URLs +// FROM the manifest is the post-grant v1 (see #334). +// Non-blocking by design: runs after boot-critical cells (depends only on the +// URL cells), never throws, and a missing manifest is a console warning, not +// a user-facing failure. +manifestCheck = { + const pinned = [ + h3_res4_url, h3_res6_url, h3_res8_url, lite_url, wide_url, facets_url, + facet_summaries_url, cross_filter_url, vocab_labels_url, facet_tree_url, + membership_url, tree_cross_filter_url, masks_url, node_bits_url, + index_url, index_meta_url, + // Runtime search-index loads, via the SHARED base cell (df.parquet is + // offline-only per SEARCH_INDEX_V1.md and deliberately not pinned here): + `${search_index_base}/build_stats.json`, + `${search_index_base}/hot_tokens.json`, + `${search_index_base}/shard_sizes.json`, + `${search_index_base}/hot_topk.parquet`, + ]; + // Hard-coded hrefs in the page header are a separate + // divergence surface (round-1 P2): audit whatever is actually in the DOM. + for (const l of document.querySelectorAll('link[rel="preload"][as="fetch"]')) { + if (l.href && l.href.includes('data.isamples.org')) pinned.push(l.href); + } + const prefix = `${R2_BASE}/`; + const pinnedNames = [...new Set(pinned.map(u => { + if (u.startsWith(prefix)) return u.slice(prefix.length); + const m = u.match(/data\.isamples\.org\/(.+)$/); + return m ? m[1] : u; + }))]; + const result = { status: 'unchecked', missing: [], extra: [], release_id: null }; + try { + const resp = await fetch(new URL('isamples_202608_release_manifest.json', document.baseURI)); + if (!resp.ok) { + console.warn(`#334 manifest check: manifest fetch HTTP ${resp.status} — skipping`); + result.status = 'manifest-unavailable'; + } else { + const manifest = await resp.json(); + result.release_id = manifest.release_id; + const manifestNames = new Set(Object.keys(manifest.files || {})); + // The search-index PATH must also agree (a SUBSTRATE_BASE _v1 -> _v2 bump + // with a stale manifest must trip the check, not slide by). + const mPath = ((manifest.search_index || {}).path || '').replace(/\/$/, ''); + const localPath = search_index_base.startsWith(prefix) + ? search_index_base.slice(prefix.length) : search_index_base; + if (mPath && mPath !== localPath) result.missing.push(`search_index path: page=${localPath} manifest=${mPath}`); + result.missing = result.missing.concat(pinnedNames.filter(n => !manifestNames.has(n))); + result.extra = [...manifestNames].filter(n => !pinnedNames.includes(n)); + result.status = result.missing.length ? 'MISMATCH' : 'ok'; + // Idempotency (round-1 P2): re-evaluations must not stack banners, and a + // clean re-run must clear a stale one. + document.getElementById('manifestMismatchBanner')?.remove(); + if (result.missing.length) { + console.error('#334 RELEASE-MANIFEST MISMATCH — the page pins data files ' + + 'absent from the release manifest (stale manifest, or an unvetted ' + + 'suffix bump — see CANONICAL.md policy):', result.missing); + const banner = document.createElement('div'); + banner.id = 'manifestMismatchBanner'; + banner.style.cssText = 'position:sticky;top:0;z-index:9999;background:#8a6d00;' + + 'color:#fff;padding:6px 12px;font:13px sans-serif;cursor:pointer;'; + banner.textContent = 'Data release could not be validated — some data files ' + + 'are not in the release manifest. Results may be inconsistent. ' + + '(Details in the browser console; click to dismiss.)'; + banner.onclick = () => banner.remove(); + document.body.prepend(banner); + } else if (result.extra.length) { + // Informational only: the manifest may legitimately list objects this + // page doesn't load (e.g. future additions). Not a divergence signal. + console.info('#334 manifest check ok; manifest lists unpinned entries:', result.extra); + } else { + console.info(`#334 manifest check ok — ${pinnedNames.length} pinned URLs ` + + `all present in release ${manifest.release_id}`); + } + } + } catch (e) { + console.warn('#334 manifest check failed (non-fatal):', e); + result.status = 'check-error'; + } + if (typeof window !== 'undefined') window.__manifestCheck = result; + return result; +} + // Canonical palette — see issue #113. Path-relative so this works under // both isamples.org (custom domain at root) and project-pages fork // previews (rdhyee.github.io/isamplesorg.github.io/...). @@ -5915,7 +6008,7 @@ zoomWatcher = { // SME A/B ranking comparison (see #172); `?fts=v1` remains accepted so // pre-flip opt-in links behave identically. §7 end-to-end budgets stay // open in #172 as the pin-overlay / downstream-latency track. - const SUBSTRATE_BASE = `${R2_BASE}/isamples_202608_search_index_v1`; + const SUBSTRATE_BASE = search_index_base; // shared cell — see #334 manifest check let _substrateMeta = null; function substrateEnabled() { diff --git a/isamples_202608_release_manifest.json b/isamples_202608_release_manifest.json new file mode 100644 index 0000000..cdb15a7 --- /dev/null +++ b/isamples_202608_release_manifest.json @@ -0,0 +1,132 @@ +{ + "schema_version": 1, + "release_id": "isamples_202608", + "generated_at_utc": "2026-07-23T17:51:11+00:00", + "base": "https://data.isamples.org", + "files": { + "isamples_202608_wide.parquet": { + "size_bytes": 300303095, + "etag": "1ef84eef3618a9f9d7f140adc42d543c", + "last_modified": null + }, + "isamples_202608_samples_map_lite_v3.parquet": { + "size_bytes": 62924115, + "etag": "36849864596413e56688378dc83582dc-2", + "last_modified": null + }, + "isamples_202608_sample_facets_v4.parquet": { + "size_bytes": 69384530, + "etag": "29b7fcff8683ac3a937db9550bdc8d37-9", + "last_modified": null + }, + "isamples_202608_h3_summary_res4.parquet": { + "size_bytes": 505651, + "etag": "e2364e810bc5fda6b613bdc8f3fb947f", + "last_modified": null + }, + "isamples_202608_h3_summary_res6.parquet": { + "size_bytes": 1322875, + "etag": "65200a3ce894d7e3ea16560784c5c9f6", + "last_modified": null + }, + "isamples_202608_h3_summary_res8.parquet": { + "size_bytes": 2009510, + "etag": "f6410fa139a75a48fc6db1d6f24d76a6", + "last_modified": null + }, + "isamples_202608_facet_summaries.parquet": { + "size_bytes": 1689, + "etag": "6498e8f60113f973a9519fe647860d54", + "last_modified": null + }, + "isamples_202608_facet_cross_filter.parquet": { + "size_bytes": 5862, + "etag": "da81a1eaaf97c109cef279128e1bbbd5", + "last_modified": null + }, + "isamples_202608_facet_tree_summaries.parquet": { + "size_bytes": 2773, + "etag": "8631b499654eb19b356f30bd1cbc10f7", + "last_modified": null + }, + "isamples_202608_facet_tree_cross_filter.parquet": { + "size_bytes": 7102, + "etag": "fe9155ec83548e873c29657ae569ae7e", + "last_modified": null + }, + "isamples_202608_sample_facet_membership.parquet": { + "size_bytes": 30908543, + "etag": "24d5021eda3c1b75060e265c18da36bd-4", + "last_modified": null + }, + "isamples_202608_sample_facet_masks.parquet": { + "size_bytes": 10138648, + "etag": "953b19a368d21660c9bca7219811cf3d", + "last_modified": null + }, + "isamples_202608_facet_node_bits.parquet": { + "size_bytes": 1839, + "etag": "687e2ee9e72184c1affb293aec861850", + "last_modified": null + }, + "isamples_202608_sample_facet_index.parquet": { + "size_bytes": 10152968, + "etag": "818b43b4c5ee5bbb06ffe1050a3bbcd7", + "last_modified": null + }, + "isamples_202608_sample_facet_index_meta.parquet": { + "size_bytes": 1154, + "etag": "2e7686fd6f008666cbde31dc047f344c", + "last_modified": null + }, + "vocab_labels_202608.parquet": { + "size_bytes": 58783, + "etag": "4db27051fc74301140795b66ff39ab08", + "last_modified": null + }, + "isamples_202608_search_index_v1/build_stats.json": { + "size_bytes": 1980, + "etag": "a44862c90b538fec751fd03d372e2e1d", + "last_modified": null + }, + "isamples_202608_search_index_v1/hot_tokens.json": { + "size_bytes": 17747, + "etag": "66c8fb0ea713915085dc12036d7e6110", + "last_modified": null + }, + "isamples_202608_search_index_v1/shard_sizes.json": { + "size_bytes": 7933, + "etag": "488bb3c2797b81d036e8d6431ee23cf0", + "last_modified": null + }, + "isamples_202608_search_index_v1/hot_topk.parquet": { + "size_bytes": 690750, + "etag": "a88d1228baa0b4c2ce7bcb9c1dede7bb", + "last_modified": null + }, + "isamples_202608_search_index_v1/df.parquet": { + "size_bytes": 7990281, + "etag": "a42de35cafeeb8bfae58c91ef09023e9", + "last_modified": null + } + }, + "search_index": { + "path": "isamples_202608_search_index_v1/", + "base_shard_count": 256, + "base_shard_bytes": 551211162, + "total_shard_files": 847, + "hot_shard_files": 591, + "hot_token_count": 129, + "runtime_sidecars": [ + "build_stats.json", + "hot_tokens.json", + "shard_sizes.json", + "hot_topk.parquet" + ], + "offline_only": [ + "df.parquet" + ], + "note": "base shards from shard_sizes.json; hot sub-files enumerated from hot_tokens.json; totals cross-checked vs build_stats shard_files" + }, + "docs": "CANONICAL.md (human twin); #334 v0-detect" +} diff --git a/tools/build_release_manifest.py b/tools/build_release_manifest.py new file mode 100644 index 0000000..de9f38e --- /dev/null +++ b/tools/build_release_manifest.py @@ -0,0 +1,221 @@ +#!/usr/bin/env python3 +"""build_release_manifest.py — emit the canonical release manifest (#334 v0-detect). + +The manifest is the machine-readable twin of CANONICAL.md: it enumerates exactly +the data files the Explorer is entitled to load for a release, with per-file +size/etag/last-modified gathered via strict ranged GETs against the live origin (no +downloads). The Explorer fetches this at boot and cross-checks its pinned URLs +(v0 = DETECT divergence loudly; deriving URLs from the manifest is v1, post-grant +— see #334). + +Usage: + python3 tools/build_release_manifest.py # print JSON + python3 tools/build_release_manifest.py --out isamples_202608_release_manifest.json + python3 tools/build_release_manifest.py --base https://data.isamples.org + +Fail-closed: any missing file or non-strict probe result aborts with exit 1 — a +manifest must never be generated from a broken origin. (Policy learned 2026-07-18.) + +""" +import argparse +import datetime +import json +import sys +import urllib.request + +RELEASE_ID = "isamples_202608" + +# The canonical set. Keep in lockstep with CANONICAL.md §1 — a suffix bump edits +# BOTH in the same change (policy in CANONICAL.md §2). +CANONICAL_FILES = [ + "isamples_202608_wide.parquet", + "isamples_202608_samples_map_lite_v3.parquet", + "isamples_202608_sample_facets_v4.parquet", + "isamples_202608_h3_summary_res4.parquet", + "isamples_202608_h3_summary_res6.parquet", + "isamples_202608_h3_summary_res8.parquet", + "isamples_202608_facet_summaries.parquet", + "isamples_202608_facet_cross_filter.parquet", + "isamples_202608_facet_tree_summaries.parquet", + "isamples_202608_facet_tree_cross_filter.parquet", + "isamples_202608_sample_facet_membership.parquet", + "isamples_202608_sample_facet_masks.parquet", + "isamples_202608_facet_node_bits.parquet", + "isamples_202608_sample_facet_index.parquet", + "isamples_202608_sample_facet_index_meta.parquet", + "vocab_labels_202608.parquet", +] + +# The search index is a directory of ~852 objects; the manifest pins its +# self-describing sidecars (the index validates itself via build_stats). +SEARCH_INDEX_SIDECARS = [ + "isamples_202608_search_index_v1/build_stats.json", + "isamples_202608_search_index_v1/hot_tokens.json", + "isamples_202608_search_index_v1/shard_sizes.json", + "isamples_202608_search_index_v1/hot_topk.parquet", # loaded by the topk query path + # df.parquet is OFFLINE-ONLY per SEARCH_INDEX_V1.md — archived, not runtime; + # it is probed for existence but excluded from the Explorer's runtime check. + "isamples_202608_search_index_v1/df.parquet", +] + + +UA = "isamples-release-manifest-builder/1.0 (+https://github.com/isamplesorg/isamplesorg.github.io)" + + +def head(base, name, permissive=False): + """Probe a file without downloading it (1-byte ranged GET). + + The data.isamples.org Worker 403s HEAD and the default Python UA, so we use + a ranged GET with an honest UA — the same primitive DuckDB-WASM relies on. + STRICT by default (Codex P1): require 206 + exact Content-Range + one byte + + an ETag. Anything else (200-with-HTML proxy page, broken Range support — + the very capability the Explorer needs, missing integrity metadata) refuses. + --permissive relaxes for dev mirrors. + """ + import re as _re + req = urllib.request.Request(f"{base}/{name}", headers={ + "Range": "bytes=0-0", "User-Agent": UA, + }) + with urllib.request.urlopen(req, timeout=30) as r: + h = r.headers + etag = (h.get("ETag") or "").strip('"') or None + cr = h.get("Content-Range", "") + m = _re.fullmatch(r"bytes 0-0/(\d+)", cr) + size = int(m.group(1)) if m else None + if not permissive: + # Validate status + headers BEFORE touching the body, so a + # Range-ignoring origin can't make us download a 300 MB object. + if r.status != 206: + raise RuntimeError(f"{name}: expected 206, got {r.status} (Range support broken?)") + if size is None or size <= 0: + raise RuntimeError(f"{name}: bad Content-Range {cr!r}") + if not etag: + raise RuntimeError(f"{name}: missing ETag (no integrity metadata)") + body = r.read(2) # bounded: expect exactly the 1 requested byte + if len(body) != 1: + raise RuntimeError(f"{name}: expected 1 byte, got {len(body)}") + elif size is None: + cl = h.get("Content-Length") + size = int(cl) if cl and cl.isdigit() and r.status == 200 else None + return {"size_bytes": size, "etag": etag, "last_modified": h.get("Last-Modified")} + + +def fetch_json(base, name): + req = urllib.request.Request(f"{base}/{name}", headers={"User-Agent": UA}) + with urllib.request.urlopen(req, timeout=30) as r: + if r.status != 200: + raise RuntimeError(f"HTTP {r.status} for {name}") + return json.loads(r.read().decode()) + + +def main(): + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap.add_argument("--base", default="https://data.isamples.org") + ap.add_argument("--out", metavar="FILE") + ap.add_argument("--permissive", action="store_true", + help="relax probe strictness for dev mirrors") + args = ap.parse_args() + + files, errors = {}, [] + for name in CANONICAL_FILES + SEARCH_INDEX_SIDECARS: + try: + meta = head(args.base, name, permissive=args.permissive) + if meta["size_bytes"] in (None, 0): + errors.append(f"{name}: missing/zero size") + files[name] = meta + except Exception as e: # noqa: BLE001 + errors.append(f"{name}: {e}") + + # Shard inventory, cross-validated between the index's own artifacts + # (Codex rounds 1-2: counts come from data; shard_sizes.json covers only + # the BASE shards — hot-token sub-files are counted in build_stats.json). + inv = {"base_shard_count": None, "base_shard_bytes": None, + "total_shard_files": None, "hot_shard_files": None} + try: + sizes = fetch_json(args.base, "isamples_202608_search_index_v1/shard_sizes.json") + entries = sizes if isinstance(sizes, dict) else {} + if not entries: + errors.append("shard_sizes.json: empty/unexpected shape") + else: + inv["base_shard_count"] = len(entries) + inv["base_shard_bytes"] = sum(v for v in entries.values() if isinstance(v, int)) or None + except Exception as e: # noqa: BLE001 + errors.append(f"shard_sizes.json inventory: {e}") + try: + stats = fetch_json(args.base, "isamples_202608_search_index_v1/build_stats.json") + # build_stats: shard_count = LOGICAL shards (256, matches shard_sizes); + # shard_files = PHYSICAL files (base + hot-token sub-files). + tot = stats.get("shard_files") + logical = stats.get("shard_count") + if not (isinstance(tot, int) and tot > 0): + errors.append("build_stats.json: missing/invalid shard_files") + tot = None + if not (isinstance(logical, int) and logical > 0): + # Hard requirement (Codex round 3): absence must fail, not slide by. + errors.append("build_stats.json: missing/invalid shard_count (logical)") + elif inv["base_shard_count"] and logical != inv["base_shard_count"]: + errors.append(f"inventory contradiction: build_stats logical shards {logical} " + f"!= shard_sizes entries {inv['base_shard_count']}") + inv["total_shard_files"] = tot + except Exception as e: # noqa: BLE001 + errors.append(f"build_stats.json inventory: {e}") + try: + # Hot sub-files enumerated from their authoritative source: each + # hot_tokens.json entry declares its physical sub_files count. + ht = fetch_json(args.base, "isamples_202608_search_index_v1/hot_tokens.json") + toks = ht.get("tokens") + if not isinstance(toks, dict) or not toks: + errors.append("hot_tokens.json: missing/empty tokens map") + else: + bad = [t for t, v in toks.items() + if not isinstance(v, dict) or not isinstance(v.get("sub_files"), int) + or v["sub_files"] < 1] + if bad: + errors.append(f"hot_tokens.json: malformed entries: {bad[:5]}") + else: + inv["hot_shard_files"] = sum(v["sub_files"] for v in toks.values()) + inv["hot_token_count"] = len(toks) + # Three-way cross-check: base (shard_sizes) + hot (hot_tokens) must + # equal physical shard_files (build_stats). Any disagreement refuses. + if (inv["base_shard_count"] and inv.get("hot_shard_files") is not None + and inv["total_shard_files"] is not None + and inv["base_shard_count"] + inv["hot_shard_files"] != inv["total_shard_files"]): + errors.append(f"inventory contradiction: base {inv['base_shard_count']} + " + f"hot {inv['hot_shard_files']} != shard_files {inv['total_shard_files']}") + except Exception as e: # noqa: BLE001 + errors.append(f"hot_tokens.json inventory: {e}") + + if errors: + print("REFUSING to emit manifest — origin problems:", file=sys.stderr) + for e in errors: + print(f" - {e}", file=sys.stderr) + sys.exit(1) + + manifest = { + "schema_version": 1, + "release_id": RELEASE_ID, + "generated_at_utc": datetime.datetime.now(datetime.timezone.utc) + .isoformat(timespec="seconds"), + "base": args.base, + "files": files, + "search_index": { + "path": "isamples_202608_search_index_v1/", + **inv, + "runtime_sidecars": ["build_stats.json", "hot_tokens.json", + "shard_sizes.json", "hot_topk.parquet"], + "offline_only": ["df.parquet"], + "note": "base shards from shard_sizes.json; hot sub-files enumerated " + "from hot_tokens.json; totals cross-checked vs build_stats shard_files", + }, + "docs": "CANONICAL.md (human twin); #334 v0-detect", + } + text = json.dumps(manifest, indent=1) + print(text) + if args.out: + with open(args.out, "w") as f: + f.write(text + "\n") + print(f"wrote {args.out}", file=sys.stderr) + + +if __name__ == "__main__": + main()