#334 v0-detect: canonical data inventory + release manifest + boot cross-check

CANONICAL.md

@@ -0,0 +1,84 @@
+# CANONICAL.md — which data files are real, and which are accidental history
+
+*Created 2026-07-23 (grant closing week) to answer: "can we simplify our stable of
+supplementary parquet files?" Answer: the **live set is already minimal** — the
+accumulation is superseded versions and orphan snapshots around it. This file
+names the canonical set and documents every non-canonical survivor so the
+accident is neutralized even where files remain on the bucket.*
+
+**Companion:** the machine-readable release manifest
+(`tools/build_release_manifest.py` → `isamples_202608_release_manifest.json`)
+enumerates the same canonical set with sizes/checksums; the Explorer cross-checks
+it at boot (#334 v0). A file absent from the manifest is non-canonical **by
+definition**.
+
+## 1. The canonical (live) set — 202608
+
+Exactly what production `explorer.qmd` loads, all under `https://data.isamples.org/`:
+
+| File | Role (plain English) |
+|---|---|
+| `isamples_202608_wide.parquet` | Full sample detail, one row per entity — everything derives from this |
+| `isamples_202608_samples_map_lite_v3.parquet` | Slim map/table columns (coords, label, place, date, h3 cells) |
+| `isamples_202608_sample_facets_v4.parquet` | Per-sample facet URIs + search-text blob (the `?fts=off` scan target) |
+| `isamples_202608_h3_summary_res{4,6,8}.parquet` | Pre-counted globe clusters at 3 zoom tiers |
+| `isamples_202608_facet_summaries.parquet` | Facet checkbox counts, no filters active |
+| `isamples_202608_facet_cross_filter.parquet` | Facet counts under single cross-filter |
+| `isamples_202608_facet_tree_summaries.parquet` | Hierarchical (tree) facet counts |
+| `isamples_202608_facet_tree_cross_filter.parquet` | Tree facet counts under cross-filter |
+| `isamples_202608_sample_facet_membership.parquet` | Sample↔facet-tree membership |
+| `isamples_202608_sample_facet_masks.parquet` | Bitmask substrate for multi-filter counts |
+| `isamples_202608_facet_node_bits.parquet` | Facet-node bit assignments for the masks |
+| `isamples_202608_sample_facet_index.parquet` | Per-sample facet index (multi-filter fast path) |
+| `isamples_202608_sample_facet_index_meta.parquet` | ~1 KB trusted manifest for the above (#313/#317 boot-race fix) |
+| `vocab_labels_202608.parquet` | URI → human label (539 entries; verified complete 2026-07-22) |
+| `isamples_202608_search_index_v1/` | Sharded FTS index — the default search since 2026-07-17. Runtime loads: `build_stats.json`, `hot_tokens.json`, `shard_sizes.json`, `hot_topk.parquet`, + 256 base token shards and ~591 hot-token sub-files (852 objects total incl. sidecars; base inventory in `shard_sizes.json`, totals in `build_stats.json`); `df.parquet` is offline-only |
+
+The ~9-file facet family looks baroque but is load-bearing: it is the price of
+fast multi-filter counts with no server. See `EXPLORER_QUERIES.md` for how each
+is queried and `DATA_PROVENANCE.md` for how each is built.
+
+## 2. Superseded versions (still served; do not use)
+
+Each suffix bump was a real fix that left its predecessor in place — the #326
+incident ("two sibling files silently diverged in vintage") is what happens when
+nothing marks the old ones dead. That marking is this table.
+
+| Superseded file | Replaced by | Why (the bump's fix) |
+|---|---|---|
+| `isamples_202608_sample_facets_v2.parquet` | `_v3` | concept labels folded into search text (#277; v2 introduced in `10fd415`, bumped in `8a6ebf5`) |
+| `isamples_202608_sample_facets_v3.parquet` | `_v4` | place_name folded into search text (#326, `7e900a0`) — **the file whose staleness caused the #326 bug** |
+| `isamples_202608_samples_map_lite.parquet` | `_v2` | h3_res4/res6 columns for filtered clusters (#300, `1715142`) |
+| `isamples_202608_samples_map_lite_v2.parquet` | `_v3` | corrected place_name + Array.isArray fix (#311, `3f2f20a`) |
+
+**Policy going forward:** a suffix bump MUST add a row here and regenerate the
+release manifest in the same change. (The manifest makes violations visible at
+boot; this table makes them legible to humans.)
+
+## 3. Snapshot generations (who serves whom)
+
+| Generation | Status |
+|---|---|
+| `isamples_202512_*` | Narrow only — **archival** (in the Zenodo deposit; there is no 202601 narrow — known label mismatch, documented in the deposit README) |
+| `isamples_202601_*` | **Archival** — the coherent snapshot in Zenodo draft 21288719 |
+| `isamples_202604_*` | **NOT orphan — backs the stable `/current/wide.parquet` alias** used by data.qmd and multiple tutorials (see SERIALIZATIONS.md). Must NOT be moved until the alias is repointed and consumers audited |
+| `isamples_202606_*` | Pipeline intermediate (the #272 OC-concept-enriched wide that fed 202608 — see DATA_PROVENANCE.md step 3b). Not loaded by anything at runtime; safe to attic post-grant, lineage documented |
+| `isamples_202608_*` | **Live** (production Explorer) |
+
+## 4. Conveniences (not load-bearing; fine to keep, labeled)
+
+| File | Note |
+|---|---|
+| `isamples_202601_wide_h3.parquet` | Analyst convenience (wide + precomputed H3 cells). In the Zenodo deposit; NOT loaded by the Explorer |
+| `*.csv` twins of lite/H3 (~640 MB) | Convenience exports; parquet is authoritative; excluded from Zenodo by design |
+
+## 5. Post-grant simplification options (deliberately NOT done in-grant)
+
+1. Move §2's superseded versions and the 202606 pipeline intermediate to an `/attic/` prefix on R2 (don't delete — old
+   notebooks/links may reference them). Needs R2 write credentials.
+2. Collapse the facet family only alongside a real pipeline rework — the current
+   9 files trade storage (cheap) for browser CPU (scarce); collapsing them is an
+   engineering project, not cleanup.
+3. If a fresher export ever lands (#320), cut ONE coherent generation and retire
+   202608 wholesale via the manifest.
+<!-- cc:2026.07.23 -->

_quarto.yml

@@ -2,6 +2,7 @@ project:
   type: website
   output-dir: docs
   resources:
+    - isamples_202608_release_manifest.json
     - assets/js/source-palette.js
     - assets/js/sql-builders.js
     - assets/js/explorer-utils.js

explorer.qmd

@@ -912,6 +912,99 @@ index_url = `${R2_BASE}/isamples_202608_sample_facet_index.parquet`
 // runs. Always deployed paired with index_url (same build_id).
 index_meta_url = `${R2_BASE}/isamples_202608_sample_facet_index_meta.parquet`
 
+// Single source of truth for the search-index location (#334 round-1 review):
+// the runtime substrate path and the manifest check MUST share this cell, so a
+// version bump can't diverge them silently.
+search_index_base = `${R2_BASE}/isamples_202608_search_index_v1`
+
+// === #334 v0-detect: release-manifest boot cross-check ===
+// The release manifest (isamples_202608_release_manifest.json, committed in
+// this repo and deployed with the site; generated by
+// tools/build_release_manifest.py; human twin: CANONICAL.md) enumerates the
+// canonical data objects for this release. This cell cross-checks every URL
+// the page pins against that list and complains LOUDLY on divergence —
+// converting the #326 failure class ("sibling data files silently diverged
+// for weeks") into "caught at next boot". v0 = detect only; deriving URLs
+// FROM the manifest is the post-grant v1 (see #334).
+// Non-blocking by design: runs after boot-critical cells (depends only on the
+// URL cells), never throws, and a missing manifest is a console warning, not
+// a user-facing failure.
+manifestCheck = {
+  const pinned = [
+    h3_res4_url, h3_res6_url, h3_res8_url, lite_url, wide_url, facets_url,
+    facet_summaries_url, cross_filter_url, vocab_labels_url, facet_tree_url,
+    membership_url, tree_cross_filter_url, masks_url, node_bits_url,
+    index_url, index_meta_url,
+    // Runtime search-index loads, via the SHARED base cell (df.parquet is
+    // offline-only per SEARCH_INDEX_V1.md and deliberately not pinned here):
+    `${search_index_base}/build_stats.json`,
+    `${search_index_base}/hot_tokens.json`,
+    `${search_index_base}/shard_sizes.json`,
+    `${search_index_base}/hot_topk.parquet`,
+  ];
+  // Hard-coded <link rel=preload> hrefs in the page header are a separate
+  // divergence surface (round-1 P2): audit whatever is actually in the DOM.
+  for (const l of document.querySelectorAll('link[rel="preload"][as="fetch"]')) {
+    if (l.href && l.href.includes('data.isamples.org')) pinned.push(l.href);
+  }
+  const prefix = `${R2_BASE}/`;
+  const pinnedNames = [...new Set(pinned.map(u => {
+    if (u.startsWith(prefix)) return u.slice(prefix.length);
+    const m = u.match(/data\.isamples\.org\/(.+)$/);
+    return m ? m[1] : u;
+  }))];
+  const result = { status: 'unchecked', missing: [], extra: [], release_id: null };
+  try {
+    const resp = await fetch(new URL('isamples_202608_release_manifest.json', document.baseURI));
+    if (!resp.ok) {
+      console.warn(`#334 manifest check: manifest fetch HTTP ${resp.status} — skipping`);
+      result.status = 'manifest-unavailable';
+    } else {
+      const manifest = await resp.json();
+      result.release_id = manifest.release_id;
+      const manifestNames = new Set(Object.keys(manifest.files || {}));
+      // The search-index PATH must also agree (a SUBSTRATE_BASE _v1 -> _v2 bump
+      // with a stale manifest must trip the check, not slide by).
+      const mPath = ((manifest.search_index || {}).path || '').replace(/\/$/, '');
+      const localPath = search_index_base.startsWith(prefix)
+        ? search_index_base.slice(prefix.length) : search_index_base;
+      if (mPath && mPath !== localPath) result.missing.push(`search_index path: page=${localPath} manifest=${mPath}`);
+      result.missing = result.missing.concat(pinnedNames.filter(n => !manifestNames.has(n)));
+      result.extra = [...manifestNames].filter(n => !pinnedNames.includes(n));
+      result.status = result.missing.length ? 'MISMATCH' : 'ok';
+      // Idempotency (round-1 P2): re-evaluations must not stack banners, and a
+      // clean re-run must clear a stale one.
+      document.getElementById('manifestMismatchBanner')?.remove();
+      if (result.missing.length) {
+        console.error('#334 RELEASE-MANIFEST MISMATCH — the page pins data files '
+          + 'absent from the release manifest (stale manifest, or an unvetted '
+          + 'suffix bump — see CANONICAL.md policy):', result.missing);
+        const banner = document.createElement('div');
+        banner.id = 'manifestMismatchBanner';
+        banner.style.cssText = 'position:sticky;top:0;z-index:9999;background:#8a6d00;'
+          + 'color:#fff;padding:6px 12px;font:13px sans-serif;cursor:pointer;';
+        banner.textContent = 'Data release could not be validated — some data files '
+          + 'are not in the release manifest. Results may be inconsistent. '
+          + '(Details in the browser console; click to dismiss.)';
+        banner.onclick = () => banner.remove();
+        document.body.prepend(banner);
+      } else if (result.extra.length) {
+        // Informational only: the manifest may legitimately list objects this
+        // page doesn't load (e.g. future additions). Not a divergence signal.
+        console.info('#334 manifest check ok; manifest lists unpinned entries:', result.extra);
+      } else {
+        console.info(`#334 manifest check ok — ${pinnedNames.length} pinned URLs `
+          + `all present in release ${manifest.release_id}`);
+      }
+    }
+  } catch (e) {
+    console.warn('#334 manifest check failed (non-fatal):', e);
+    result.status = 'check-error';
+  }
+  if (typeof window !== 'undefined') window.__manifestCheck = result;
+  return result;
+}
+
 // Canonical palette — see issue #113. Path-relative so this works under
 // both isamples.org (custom domain at root) and project-pages fork
 // previews (rdhyee.github.io/isamplesorg.github.io/...).
@@ -5915,7 +6008,7 @@ zoomWatcher = {
     // SME A/B ranking comparison (see #172); `?fts=v1` remains accepted so
     // pre-flip opt-in links behave identically. §7 end-to-end budgets stay
     // open in #172 as the pin-overlay / downstream-latency track.
-    const SUBSTRATE_BASE = `${R2_BASE}/isamples_202608_search_index_v1`;
+    const SUBSTRATE_BASE = search_index_base;   // shared cell — see #334 manifest check
     let _substrateMeta = null;
 
     function substrateEnabled() {

isamples_202608_release_manifest.json

@@ -0,0 +1,132 @@
+{
+ "schema_version": 1,
+ "release_id": "isamples_202608",
+ "generated_at_utc": "2026-07-23T17:51:11+00:00",
+ "base": "https://data.isamples.org",
+ "files": {
+  "isamples_202608_wide.parquet": {
+   "size_bytes": 300303095,
+   "etag": "1ef84eef3618a9f9d7f140adc42d543c",
+   "last_modified": null
+  },
+  "isamples_202608_samples_map_lite_v3.parquet": {
+   "size_bytes": 62924115,
+   "etag": "36849864596413e56688378dc83582dc-2",
+   "last_modified": null
+  },
+  "isamples_202608_sample_facets_v4.parquet": {
+   "size_bytes": 69384530,
+   "etag": "29b7fcff8683ac3a937db9550bdc8d37-9",
+   "last_modified": null
+  },
+  "isamples_202608_h3_summary_res4.parquet": {
+   "size_bytes": 505651,
+   "etag": "e2364e810bc5fda6b613bdc8f3fb947f",
+   "last_modified": null
+  },
+  "isamples_202608_h3_summary_res6.parquet": {
+   "size_bytes": 1322875,
+   "etag": "65200a3ce894d7e3ea16560784c5c9f6",
+   "last_modified": null
+  },
+  "isamples_202608_h3_summary_res8.parquet": {
+   "size_bytes": 2009510,
+   "etag": "f6410fa139a75a48fc6db1d6f24d76a6",
+   "last_modified": null
+  },
+  "isamples_202608_facet_summaries.parquet": {
+   "size_bytes": 1689,
+   "etag": "6498e8f60113f973a9519fe647860d54",
+   "last_modified": null
+  },
+  "isamples_202608_facet_cross_filter.parquet": {
+   "size_bytes": 5862,
+   "etag": "da81a1eaaf97c109cef279128e1bbbd5",
+   "last_modified": null
+  },
+  "isamples_202608_facet_tree_summaries.parquet": {
+   "size_bytes": 2773,
+   "etag": "8631b499654eb19b356f30bd1cbc10f7",
+   "last_modified": null
+  },
+  "isamples_202608_facet_tree_cross_filter.parquet": {
+   "size_bytes": 7102,
+   "etag": "fe9155ec83548e873c29657ae569ae7e",
+   "last_modified": null
+  },
+  "isamples_202608_sample_facet_membership.parquet": {
+   "size_bytes": 30908543,
+   "etag": "24d5021eda3c1b75060e265c18da36bd-4",
+   "last_modified": null
+  },
+  "isamples_202608_sample_facet_masks.parquet": {
+   "size_bytes": 10138648,
+   "etag": "953b19a368d21660c9bca7219811cf3d",
+   "last_modified": null
+  },
+  "isamples_202608_facet_node_bits.parquet": {
+   "size_bytes": 1839,
+   "etag": "687e2ee9e72184c1affb293aec861850",
+   "last_modified": null
+  },
+  "isamples_202608_sample_facet_index.parquet": {
+   "size_bytes": 10152968,
+   "etag": "818b43b4c5ee5bbb06ffe1050a3bbcd7",
+   "last_modified": null
+  },
+  "isamples_202608_sample_facet_index_meta.parquet": {
+   "size_bytes": 1154,
+   "etag": "2e7686fd6f008666cbde31dc047f344c",
+   "last_modified": null
+  },
+  "vocab_labels_202608.parquet": {
+   "size_bytes": 58783,
+   "etag": "4db27051fc74301140795b66ff39ab08",
+   "last_modified": null
+  },
+  "isamples_202608_search_index_v1/build_stats.json": {
+   "size_bytes": 1980,
+   "etag": "a44862c90b538fec751fd03d372e2e1d",
+   "last_modified": null
+  },
+  "isamples_202608_search_index_v1/hot_tokens.json": {
+   "size_bytes": 17747,
+   "etag": "66c8fb0ea713915085dc12036d7e6110",
+   "last_modified": null
+  },
+  "isamples_202608_search_index_v1/shard_sizes.json": {
+   "size_bytes": 7933,
+   "etag": "488bb3c2797b81d036e8d6431ee23cf0",
+   "last_modified": null
+  },
+  "isamples_202608_search_index_v1/hot_topk.parquet": {
+   "size_bytes": 690750,
+   "etag": "a88d1228baa0b4c2ce7bcb9c1dede7bb",
+   "last_modified": null
+  },
+  "isamples_202608_search_index_v1/df.parquet": {
+   "size_bytes": 7990281,
+   "etag": "a42de35cafeeb8bfae58c91ef09023e9",
+   "last_modified": null
+  }
+ },
+ "search_index": {
+  "path": "isamples_202608_search_index_v1/",
+  "base_shard_count": 256,
+  "base_shard_bytes": 551211162,
+  "total_shard_files": 847,
+  "hot_shard_files": 591,
+  "hot_token_count": 129,
+  "runtime_sidecars": [
+   "build_stats.json",
+   "hot_tokens.json",
+   "shard_sizes.json",
+   "hot_topk.parquet"
+  ],
+  "offline_only": [
+   "df.parquet"
+  ],
+  "note": "base shards from shard_sizes.json; hot sub-files enumerated from hot_tokens.json; totals cross-checked vs build_stats shard_files"
+ },
+ "docs": "CANONICAL.md (human twin); #334 v0-detect"
+}