diff --git a/.claude/agents/5plus3-council.md b/.claude/agents/5plus3-council.md index b34bba557..5bf00607e 100644 --- a/.claude/agents/5plus3-council.md +++ b/.claude/agents/5plus3-council.md @@ -73,6 +73,21 @@ design decision: answerable YES/NO/VIOLATES-with-evidence. This is what makes Sonnet sufficient for the savants: bounded input, fixed output shape. +**Premise gate (added 2026-10-03).** Run +`.claude/agents/premise-auditor.md` twice: before the spec is frozen, on +the question; and before any option set is escalated to the operator, on +the question **together with every option** in it — a sound question can +still carry an option that files the property under the wrong mechanism. The savants +verify the options; none of them checks whether the question files a +concept under the wrong mechanism. That is how a council of fourteen +answered "which kind of alpha is *known*?" when *known* was not alpha at +all (`ISS-LXA-ALPHA-FIT`). PREMISE-SPLIT or PREMISE-WRONG re-asks the +question before Phase 1. At escalation, a SPLIT or WRONG question is +re-asked; when premise-auditor tests 1 or 2 fire, the final option set must +include "the property belongs elsewhere" (`premise-auditor.md`, Step 2). +The final question and option set are audited once more before they reach +the operator. + ## Phase 1 — the 5 (research savants, parallel, single lens each) Default panel (swap lenses per domain; declare swaps in the spec header): diff --git a/.claude/agents/README.md b/.claude/agents/README.md index 0b174426b..eda8a540a 100644 --- a/.claude/agents/README.md +++ b/.claude/agents/README.md @@ -56,6 +56,17 @@ chain terrain. **Mandatory reviewer** when any proposal adds layers without numbers, γ+φ placement is discussed, HHTL cascade is touched, or any unification is proposed without a falsifying probe. +### Councils (harnesses, not single agents) + +- **`5plus3-council`** (`/5plus3`): hardens one committed spec. 5 savants + verify, consolidate, then 3 reviewers attack draft v2. Converges and + ratifies. +- **`coresearch-council`** (`/coresearch`): explores an open question + across the code and the outside world (arXiv, known systems such as + DuckDB / Odoo, ontologies). 5 scouts in two rings, a crosswalk, then 3 + co-architects (bridge, firewall/fit, falsifier) and an exploration map. + Ratifies nothing; a chosen design goes to a plan or `/5plus3`. + --- ## Codec / Compression diff --git a/.claude/agents/coresearch-council.md b/.claude/agents/coresearch-council.md new file mode 100644 index 000000000..d9ef1c12c --- /dev/null +++ b/.claude/agents/coresearch-council.md @@ -0,0 +1,195 @@ +# Co-Research Council — exploration across the code and the outside world + +**READ BY:** the orchestrating main thread before invoking `/coresearch`; +`integration-lead` and `convergence-architect` when an idea arrives from +outside the workspace. + +> **Purpose.** Explore an OPEN question together: what the workspace already +> has, what the outside world already knows (arXiv, known systems such as +> DuckDB and Odoo, ontologies and established concepts), and where the two +> meet. The product is an **exploration map**, never a ratified design. +> +> **Prior art.** This formalises the one-off 2026-06 research council +> (`.claude/knowledge/research-council-semantics-papers-2026-06.md`: five +> Opus readers, papers read in full, a firewall verdict per idea), and keeps +> its verdict vocabulary. `convergence-architect` supplies the bridge lens; +> `prior-art-savant` supplies the internal-history lens. + +## How it differs from the 5+3 council + +| | `5plus3-council` | `coresearch-council` (this card) | +|---|---|---| +| direction | **converge**: harden one committed spec | **diverge, then map**: find and grade options | +| input | a spec with a committed resolution | a question brief with no committed answer | +| roles | 5 savants verify, 3 reviewers attack | 5 scouts gather, 3 co-architects connect and filter | +| output | ratified v3, then implementation | an exploration map: ideas, crosswalk, grades, probes | +| authority | ratifies a design | **ratifies nothing**; a chosen idea goes to the operator, then to a plan or a 5+3 | + +The two compose: coresearch decides *what is worth designing*; 5+3 hardens +*the design*. Never run them in one pass. + +## When to convene (and when NOT to) + +Convene for an open question where outside knowledge plausibly changes the +answer: + +- "has someone already solved X?" (a known system, paper or standard); +- a new subsystem before its plan exists (what to borrow, what to avoid); +- aligning with an external model (an Odoo module, an ontology, DuckDB + semantics) where the mapping itself is the question; +- a measured anomaly with no internal explanation yet. + +Do NOT convene for: + +- a decision the operator has already made (it is a frozen anchor, never + re-opened); +- a defect hunt (use the reviewers or `brutally-honest-tester`); +- a single-source lookup (read the source directly); +- anything with a committed design (that is 5+3 work). + +## Phase 0 — the QUESTION BRIEF (main thread) + +Written before any agent is cast. It replaces the 5+3's spec, and it holds +**no answer**: + +1. **THE QUESTION**, in one or two sentences, plus why it is open now. +2. **ANCHORS**: frozen decisions the exploration must not re-litigate + (operator rulings, iron rules, `CLAUDE.md` P0s), each cited. An idea that + contradicts an anchor is recorded as `CONFLICTS-ANCHOR`, never adopted. +3. **INTERNAL SURFACE**: the crates, files and board ids in scope, by path. + If unknown, run a pre-brief Explore first. +4. **EXTERNAL DOMAINS**: which outside rings to scan (see Phase 1) and the + seed terms for each, plus the existing knowledge docs that already cover + them (read those first; never re-harvest what a doc already holds). +5. **WHAT WOULD COUNT AS AN ANSWER**: the shape of a useful result (a + mechanism, a mapping table, a falsifiable probe) and what is out of + scope. +6. **BUDGET**: maximum sources per scout, and the date cut for literature. + +**Premise gate.** Run `.claude/agents/premise-auditor.md` on the question +before casting the scouts. Exploring a question that files a concept under +the wrong mechanism only maps the wrong territory in more detail. Run it +again on the question together with every option in the exploration map +before it goes to the operator; when tests 1 or 2 fire, the map must offer +"the property belongs elsewhere" (`premise-auditor.md`, Step 2). + +## Phase 1 — the 5 scouts (parallel; two rings) + +| # | scout | ring | reads | returns | +|---|---|---|---|---| +| 1 | **code cartographer** | inside | the internal surface, first-hand (`FIRST-HAND-SOURCE-LAW.md`) | what exists, graded `VERIFIED-IN-CODE` / `CLAIMED` (doc says so, code not read) / `ABSENT` (named, closed search space only) | +| 2 | **internal prior art** (`prior-art-savant` lens) | inside | board entries, `EPIPHANIES.md`, plans, knowledge docs | earlier explorations, rulings and rejected ideas, with ids. A previously rejected idea is returned with its rejection | +| 3 | **literature scout** | outside | arXiv and papers (alphaXiv tools, web) | mechanisms, each with the paper id and a read grade | +| 4 | **systems scout** | outside | known systems: DuckDB, Odoo, PostgreSQL, Datalog engines, bitmap indexes, Lance / Arrow, and the like | how they solve the question: design, invariants, known failure modes, licence | +| 5 | **concepts and ontology scout** | outside | standards and concept frameworks: OWL, DOLCE, OGIT, schema.org, relational and lattice theory, formal concept analysis | the established vocabulary and how it maps onto ours | + +Swap a scout when the question needs a different ring, and declare the swap +in the brief. + +**Read grades (external).** Every external claim carries one: + +- `READ-IN-FULL`: the scout read the whole source; +- `SECTION-READ`: named sections were read; +- `ABSTRACT-ONLY`: only the abstract or summary; +- `SECONDHAND`: known from another source, which is cited. + +Only `READ-IN-FULL` or `SECTION-READ` may support a mechanism claim. An +`ABSTRACT-ONLY` idea is at most a lead. + +**Scout output contract.** At most 10 items. Each item has: + +- a name; +- one or two sentences on the mechanism; +- its source, by id or `file:line`; +- its read grade (or code grade); +- the internal surface it touches, or `none`. + +No essays, no designs. Scouts are read-only and never write board files. + +**Model.** Scouts 2–5 accumulate across many sources, so they run on Opus. +The code cartographer may run on Sonnet when the surface is small. Never +haiku. + +## Phase 2 — the CROSSWALK (main thread, before any co-architect) + +Merge the scout output into one table, one row per idea: + +| idea | mechanism and evidence (1–2 sentences, from the scout) | outside source (grade) | nearest inside surface (grade) | relation | +|---|---|---|---|---| + +`relation` is one of: + +- `ALREADY-HAVE` (we ship it, under another name); +- `PARTIAL` (we have part; the gap is named); +- `NEW` (no counterpart inside); +- `CONFLICTS-ANCHOR` (contradicts a frozen decision; kept for the record). + +Duplicates are merged. Raw scout output is banked in the scratchpad and +never forwarded, so the crosswalk must be self-contained: each row carries +the mechanism and the evidence that supports it, enough for a co-architect +to design an adoption shape or a kill probe without the raw output. + +## Phase 3 — the 3 co-architects (parallel, on the crosswalk only) + +| # | co-architect | lens | per idea returns | +|---|---|---|---| +| 1 | **bridge architect** (`convergence-architect` lens) | where an outside idea and an inside surface share a shape, and what the smallest adoption would be | `OPPORTUNITY` / `WORTH-EXPLORING` / `DROP`, plus the adoption shape | +| 2 | **firewall and fit critic** | the workspace's non-negotiables: no float or LLM on the hot path, zero-copy, the lance family stays upstream, AdaWorldAPI forks, contract boundaries, licence fit | `PASS` / `CONFLICT` (fits only with a quarantine seam, which is named) / `TRAP` (no seam exists) | +| 3 | **falsifier designer** | for each idea that is not dropped, the smallest probe that could kill it | one pre-registered probe: input, measurement, kill condition, cost | + +Co-architects see the crosswalk and the brief, never raw scout output, and +never each other. They propose and grade; they do not decide. + +## Phase 4 — the EXPLORATION MAP (main thread) + +Each idea gets one final verdict: + +- `ADOPT-NOW`: a firewall `PASS`, small, and offline-buildable; +- `PROBE`: worth a measurement first, and the probe is named; +- `PARK`: interesting but not now, with the reason; +- `SKIP`: a `TRAP`, a duplicate, or `CONFLICTS-ANCHOR`. + +The map holds the question, the crosswalk, the verdicts, the probes and the +open points. It also states what was **not** searched, so that "nothing +found" is never read as "nothing exists". + +## Phase 5 — landing (minimal residue) + +- The map lands as **one** dated file in the transient tier, + `.claude/board/entries/YYYY-MM-DD-coresearch-.md`, plus + `python3 .claude/tools/entries_index.py --write` in the same commit. +- **Nothing is adopted by the council.** The operator chooses. A chosen + `PROBE` becomes a `STATUS_BOARD` row; a chosen design becomes a plan, and + a delicate one goes through `/5plus3`. +- A knowledge doc is written only when the operator asks for one, or when + the outside ring will clearly be reused (as the 2026-06 doc was). +- `EPIPHANIES.md` only through the § Closeout admission gate, never directly + from a council run. +- Close with the one-line closeout record, and a `MIRROR` only if it + carries something. + +## Non-negotiables for every role + +- **Search is navigation, never evidence** (`FIRST-HAND-SOURCE-LAW.md`). + The same holds outside: a search-result snippet is a lead, not a read. +- **Concepts, not code.** Outside systems are described in our own words. + Never paste external source code; record each system's licence (for + example Odoo LGPL-3, DuckDB MIT) next to any idea that would borrow from + it. +- **Human authorization is provenance, not validation.** An idea is graded + by its evidence, not by who suggested it. +- **No model identifier and no secret** in any artifact. Web access goes + through the configured proxy and tools only. +- **One writer.** Only the main thread writes the map and the board. + +## Token economy + +| role | model | why | +|---|---|---| +| brief, crosswalk, map | main thread | accumulation | +| scouts 2–5 | Opus | multi-source accumulation | +| code cartographer | Sonnet or Opus | Sonnet when the surface is small | +| co-architects | Opus | synthesis across the crosswalk | + +If the question can be answered by reading two sources, it is not +council-grade: read them. diff --git a/.claude/agents/premise-auditor.md b/.claude/agents/premise-auditor.md new file mode 100644 index 000000000..631348297 --- /dev/null +++ b/.claude/agents/premise-auditor.md @@ -0,0 +1,122 @@ +--- +name: premise-auditor +description: > + Audits the QUESTION before anyone answers it. Builds a signature for every + named concept in a question or option set (what it answers, which + coordinates it covers, how long it lives, who writes it, whether it is + baked or discardable) and blocks a question whose options file a concept + under a mechanism with a different signature — the category error that + makes every option wrong at once. Mandatory gate in Phase 0 of + /5plus3 and /coresearch, and before any option set is put to the + operator. Verdicts: PREMISE-SOUND / PREMISE-SPLIT (re-ask as two + questions) / PREMISE-WRONG (the question presupposes a false + identification). +model: opus +tools: Read, Glob, Grep +--- + +# premise-auditor — check what the question takes for granted + +**READ BY:** the main thread before writing a 5+3 spec, a coresearch brief, +or an escalation with options; both council harnesses call this card as a +gate. Loads `.claude/knowledge/reference-frame-vs-motion.md` first. + +## Why this card exists + +A council can verify every option perfectly and still answer the wrong +question. The measured instance (2026-10-03, PR #1307, `ISS-LXA-ALPHA-FIT`): + +- The question was *"how should known/unknown be carried by the alpha + channel?"* The council produced four options: (a) a baked mask shaped + like `AlphaMask`, (b) codebook entries as `NodeRow`s, (c) a new alpha + meaning "measured", (d) (a) plus the overlay as an attention recorder. +- Fourteen agents checked those options, and one reviewer came close + ("option (a) is not an alpha bit"). None asked what kind of property + "known" is. +- "Known" answers *is this coordinate defined in the reference?* It + belongs to the baked reference set: immutable, digested, one value per + entry. The alpha channel answers *what is active or different at this + SAME coordinate, at this time or rung?* It is runtime state, discardable + whole (`alpha.rs:7-16`). The two have different signatures, so **known is + not alpha at all**, and every option inherited the false premise. (a) and + (d) were right in structure and wrong in name; (b) reshaped the data to + fit the mechanism; (c) gave alpha a meaning it does not have. + +The operator had to point this out from outside. This card makes the +council find it itself. + +## Step 1 — the CONCEPT SIGNATURE TABLE + +List every concept the question or option set names: the property asked +about, and every mechanism, type, carrier or channel offered to hold it. +For each, fill the signature **from the code or the ruling**, citing +`file:line`, never from the name: + +| field | asks | +|---|---| +| **answers** | which question does this concept answer, in one line? | +| **coordinates** | which address space does it range over (reference entries, rows, rungs, versions, cycles)? | +| **lifetime** | baked/immutable, versioned, per cycle, per thought? | +| **persistence** | digested and reproducible, or discardable whole? | +| **writer** | who writes it, and when (offline bake, owner mailbox, runtime)? | +| **cardinality** | one value per what? | +| **epistemic category** | **reference frame** (calibrated from measurement, immutable per reference version: coverage, frequency, evidence, PoS, LUTs) or **motion** (session-, time- or rung-local state over the frame: alpha, attention, per-rung lanes)? See `.claude/knowledge/reference-frame-vs-motion.md` | +| **representation** | its physical form (bitmap, `u8` lane, `NodeRow`, …) | + +**Representation is recorded but never used to decide identity.** Two +concepts can share a bitmap and still be different things. + +## Step 2 — the four tests + +1. **Same name, same signature.** If one name (or one type used as a + name) covers two concepts whose signatures differ on any field except + *representation* (answers, coordinates, lifetime, persistence, writer, + cardinality, epistemic category), that is a conflation. A frame/motion mismatch alone is enough: the measured + world and a thought moving over it are never one thing, however alike + their bits are. + Reusing a **representation** is allowed; reusing the **name and + semantics** is not. +2. **The carrier fits the property.** For every option of the form "carry + X in mechanism M", compare X's signature with M's on every field except + *representation*. A mismatch on any of them (answers, coordinates, + lifetime, persistence, writer, cardinality, epistemic category) makes + the option a **category error**, however feasible it is technically. +3. **No architecture tax.** Does an option reshape the data to fit a + mechanism (more bytes per entry, a new tenant, a wider type) rather than + choose the mechanism that fits the data? If so, flag it, with the cost + computed from the code (e.g. 20,845 entries × 512 B ≈ 10.2 MiB). +4. **Is "the premise is wrong" on the menu?** If tests 1 or 2 fire, the + option set must include "the property belongs elsewhere" before it goes + to the operator. An option set without that exit is malformed. + +## Verdicts + +- **PREMISE-SOUND**: all signatures agree; the question may proceed as + asked. +- **PREMISE-SPLIT**: the question bundles two concepts; re-ask it as two + questions, one per signature, each with its own home. +- **PREMISE-WRONG**: the question presupposes that two concepts with + different signatures are one. Return the corrected question and, when + the code makes it clear, the home each concept belongs in. + +## Output contract + +1. The signature table, every cell cited (`file:line`, plan section or + ruling). +2. Each test: FIRES / SILENT, with the cells that decide it. +3. The verdict, and for SPLIT or WRONG the re-asked question(s) in one or + two sentences each. +4. At most one line on any option that survives the corrected question + (in its corrected name). + +Read-only. No designs beyond the corrected question; the council or the +operator decides the rest. + +## Discipline + +- A signature cell copied from a name or a doc comment, without reading + what the code does, is a guess. Mark it `CLAIMED` and say so. +- The gate must also be able to stay silent: a question whose concepts + share a signature passes, even if they share an ugly name. +- It runs once per question, not per finding. A council does not convene + a premise audit on its own premise audit. diff --git a/.claude/board/ISSUES.md b/.claude/board/ISSUES.md index ad6687c1e..67f67186d 100644 --- a/.claude/board/ISSUES.md +++ b/.claude/board/ISSUES.md @@ -1,6 +1,6 @@ ## ISS-LXA-ALPHA-FIT — known/unknown for the COCA bake vs the alpha channel's own definition (2026-09-30) -**Status:** OPEN — operator escalation. **Basis:** VERIFIED-IN-CODE (5+3 council on +**Status:** RESOLVED 2026-10-03 — known is NOT alpha: it is the reference set's baked coverage plane (`ReferenceCoverage`, no conversion to/from `AlphaMask`); the attention-recorder half of option (d) is a separate §5 open question. Plan §3.1 DECISION; `.claude/knowledge/reference-frame-vs-motion.md`. The bullets below record the escalation as it stood on 2026-09-30 and are SUPERSEDED: their "uses the alpha split tunnel" premise and the option-(d) recommendation were rejected (known is not alpha; the attention recorder is plan §5 open), and their "What closes it" condition is SATISFIED. D-LXA-3's only remaining blocker is D-LXC-4 (the shared `academic_20k.csv` loader, STATUS_BOARD). **Basis:** VERIFIED-IN-CODE (5+3 council on `deepnsm-v2-lexical-address-v1`, §3.1). - The ruling says known/unknown uses the alpha channel split tunnel. The alpha channel is defined as not a bake (no digest, discardable whole, `alpha.rs:11-16, 857-861`); diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 772c9ab2a..7ccc54cd8 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -309,9 +309,9 @@ evaluates one; `execute` stays the consumer's call on a scratch it owns. | D-id | scope | status | gate / falsifier | |---|---|---|---| -| D-LXA-1 | `LexicalAddress(u16)` newtype (reading key, beside the surface-form `WordId`) + `ReferenceSet {id, version, sha256, key_kind}`; cross-reference read refused | Queued | G-LEX (typed values): every declared entry resolves under its reference's own key — `(word, PoS)`, `(lemma, PoS)` or `(word, Ambiguous)`; wrong reference refused; bare `u16` is `compile_fail` | +| D-LXA-1 | `LexicalAddress(u16)` newtype (reading key, beside the surface-form `WordId`) + `ReferenceSet {id, version, sha256, key_kind}` + `ReferenceCoverage` (a reference version's coverage plane; no `From`/`Into`/`AsRef` to or from `AlphaMask`, no setter after the bake; plan §3.1); cross-reference read refused | Queued | G-LEX (typed values): every declared entry resolves under its reference's own key — `(word, PoS)`, `(lemma, PoS)` or `(word, Ambiguous)`; wrong reference refused; bare `u16` is `compile_fail`; a `ReferenceCoverage` passed as or built from an `AlphaMask` is `compile_fail` (with a passing twin built from the bake's words) | | D-LXA-2 | `lexical_correspondence.tsv` generator (id4096 / id5k / id20k / Exact·Ambiguous·Missing, three digests) | Queued | G-REF: re-derivation reproduces the measured table incl. "4 of 4,264" with ordinal 0 counted; one hand-edited ordinal reddens | -| D-LXA-3 | the COCA bake: identity table (lemma, PoS, `lemma_evidence`) + surface-form table (form, reading, `f` listed-reading share), u8, offline; unknown = unset known bit, fill byte 0 | Blocked — on operator escalation `ISS-LXA-ALPHA-FIT`, and shares `academic_20k.csv` with D-LXC-4 (Blocked); must not duplicate that loader | byte-identical re-derivation; surface `the` f = 1, surface `record` → n/v both 0 < f < 1; all-unambiguous fixture stays f = 1 | +| D-LXA-3 | the COCA bake: identity table (lemma, PoS, `lemma_evidence`) + surface-form table (form, reading, `f` listed-reading share), u8, offline; unknown = unset bit in the table's `ReferenceCoverage` plane, fill byte 0 | Blocked — shares `academic_20k.csv` with D-LXC-4 (Blocked); must not duplicate that loader. (`ISS-LXA-ALPHA-FIT` resolved 2026-10-03: known = the reference set's `ReferenceCoverage` plane, not alpha) | byte-identical re-derivation; surface `the` f = 1, surface `record` → n/v both 0 < f < 1; all-unambiguous fixture stays f = 1 | | D-LXA-4 | six-slot ClassView reading of a 12-byte facet as six `LexicalAddress`es under one `ReferenceSet`, carried by a new classid / reading mode; existing `Cam96` / `SpoFacet` readings unchanged | Blocked — a contract change needing its own contract plan (not yet written); home (second facet vs `Identity` tenant) OPEN | X-written / Y-read refused; (X, v1)-written / (X, v2)-read refused; slot rotation changes the resolved words | ## cypher-mask-lowering-v2 (D-ids minted 2026-09-30, `.claude/plans/cypher-mask-lowering-v2.md`) diff --git a/.claude/board/SUPERSESSION-INDEX.md b/.claude/board/SUPERSESSION-INDEX.md index 78e5936dc..798cd198c 100644 --- a/.claude/board/SUPERSESSION-INDEX.md +++ b/.claude/board/SUPERSESSION-INDEX.md @@ -141,7 +141,7 @@ a licence to act on it. | **RESCOPE** | `entropy-closure-causal-ground-v1` | `ThinkingStyle` | PROPOSAL (measured, unbuilt) — 2026-08-26. P | 3/8 | | **RESCOPE** | `foundry-consumer-parity-v1` | `BindSpace` | Active | 0/0 | | **RESCOPE** | `foundry-roadmap-unified-smb-medcare-v1` | `BindSpace` | Active | 0/0 | -| **RESCOPE** | `lance-graph-as-the-modelgraph-v1` | `BindSpace` | PROPOSAL (operator-set endgame, 2026-09-16). | 3/8 | +| **RESCOPE** | `lance-graph-as-the-modelgraph-v1` | `BindSpace` | PROPOSAL (operator-set endgame, 2026-09-16). | 4/8 | | **RESCOPE** | `lf-integration-mapping-v1` | `BindSpace` | Active (2026-04-25) | 0/0 | | **RESCOPE** | `lite-unified-surrealql-lance-v1` | `BindSpace` | CONJECTURE / design. **Test via feature gate | 0/0 | | **RESCOPE** | `ogit-cascade-supabase-callcenter-v1` | `BindSpace` | plan, not implementation. | 0/16 | diff --git a/.claude/board/entries/2026-10-03-coresearch-chained-hop.md b/.claude/board/entries/2026-10-03-coresearch-chained-hop.md new file mode 100644 index 000000000..f2ec461e3 --- /dev/null +++ b/.claude/board/entries/2026-10-03-coresearch-chained-hop.md @@ -0,0 +1,114 @@ +# Co-research: chained hops over masks — exploration map (2026-10-03) + +**Status:** OPEN — exploration map; ratifies nothing. Question from #1308. Harness: +`.claude/agents/coresearch-council.md` (first run). + +**Question (re-asked after the premise gate, PREMISE-SPLIT).** "A mask produced by one +program feeds the next program's Gather" was three questions: + +- **Q1:** a fixed k-hop pull chain with no intermediate mask; +- **Q2:** reachability `*1..k`, where frontier and `visited` are loop-carried state; +- **Q3:** what `ForeignPlane` *is*. + +**Anchors held:** +- A1, the tiling law (`mask-risc/src/lib.rs:16-22`); +- A2, the scatter survival condition (`ir.rs:306-314`); +- A3, the Gather doc; +- A4–A6; +- A7, reference frame vs motion. + +## Council + +**Scouts:** +- code cartographer; +- internal prior art; +- literature (7 papers, SECTION-READ); +- systems (DuckDB, Kùzu, GraphBLAS/LAGraph, PostgreSQL, Soufflé); +- concepts (semijoin algebra, DBSP, Kleene and Knaster–Tarski). + +**Co-architects:** bridge, firewall, falsifier. The crosswalk has 18 rows (X1–X18); raw output is banked in the session scratchpad. + +## What the code says (VERIFIED-IN-CODE) + +- **No loop construct.** A program is straight-line, pre-filled scratch is refused (`ir.rs:509-528`, `:519-522`), and `Out` has no loop variant (`value.rs:44-55`). +- **`ForeignPlane` is read whole by key on every tile** (`exec.rs:1714-1732`). Its fields are `pub` and it wraps a plain `&[u64]` (`ir.rs:86-92`). So **a scattered `Out::Mask` can be re-fed as a `ForeignPlane` today**. The only fence is the doc wording "resident validity" (`ir.rs:82`) plus quack's doc precondition (`quack/src/lib.rs:360-370`). +- **The deepest composed read is depth 2, over values** (`EqU32Via`, `GroupKey::Via`; ndarray `*_via`). There is no depth-k bit gather, and `mask_gather_u32` (`simd_masking_ops.rs:1099`) is a single-index scalar read. +- **Every existing graph traversal materialises its frontier:** + - `hdr_bfs`; + - `multi_hop`; + - `CsrIndex::bfs`; + - the sibling `lgj_hop`, whose `Graph.hop().hop()` re-feeds a scattered mask (pre-A1; X16). + +## Convergent outside evidence + +- **BFS step.** BFS is one masked mxv step, `f ← Aᵀf .∗ ¬v`. GraphBLAS push/pull, Datalog semi-naive evaluation, DBSP, the CTE working table and Kùzu IFE all describe the same two-state loop. +- **Operand reuse** (`Aᵀv .∗ ¬v`, Yang–Buluç–Owens) needs one carried mask, not two. +- **DBSP types loop state** as a `z⁻¹` back edge bracketed by `δ0…∫`. This is distinct from constant inputs (resident) and from the output (demanded). A linear step needs exactly one accumulator. +- **Bounded `*1..k`** is a finite union of fixed chains. Loop state is semantically necessary only for unbounded `*`. + +## Co-architect disagreement (stricter verdict kept) + +The bridge architect read loop state as lying *outside* A1's conditional clause ("when the next fold can consume directly"). The firewall critic answered **TRAP**: A1's last sentence is unconditional — *"the only population-sized writes are the DEMANDED sinks"*. An iterate `v_t` with `t < k` is population-sized and not demanded. + +Reading "can consume directly" as "with today's primitives" would also invert the missing-capability STOP rule. **So loop state needs an operator amendment to A1. It is not a reading of A1.** + +## Exploration map + +| idea | verdict | why / shape | +|---|---|---| +| **Tile-local gather chain** `start[fkʲ(i)]` (X9 reframed, X10) | **PROBE → substrate-first** | Answers Q1, and bounded `*1..k` over FUNCTIONAL (in-row fk) hops, with **no amendment**. State is tile-sized (`cur: [u32; TILE_ROWS]`, `acc`) and every source is resident. Firewall PASS, STOP-gated. Needs one ndarray primitive (`mask_gather_chain_u32` / `mask_gather_via_u32`, W1a contract), then a mask-risc `MaskOp::GatherChain`. A multi-valued hop is a join and is refused here. Probe: P-COMPOSE. | +| **`LoopPlane` / LOOP-STATE class** (X3+X5+X6+X8) | **PROBE, then operator decision** | Fits multi-valued hops and unbounded `*`. A driver loops a straight-line step (`Gather{src: Loop}` → `AndNot` → `Keep`), with one owner (the driver frame) and a lifetime of one invocation. Scatter is refused as the step terminal, and it never escapes before ∫. It **requires amending A1's sentence 2**. Probes P-REUSE and P-DETERMINISM. | +| Double buffering (X8) | **PROBE** (part of the above) | In-place `v ∨= G(v)` over-reaches under bounded k: with ascending tile order a chain advances many hops per step. It is order-free only for unbounded `*` (monotone closure). Safe Rust already forbids aliasing the Gather source with its `dst`, which settles X18. Consequence: bounded k costs 2 population buffers either way. | +| **`ForeignPlane` provenance type** (X15) | **PROBE (defect pin first)** | A doc fix alone would delete the only fence: the doc should describe the mechanism ("key-addressed, read whole"), but provenance must be a TYPE in mask-risc. Make the fields private and add named constructors `resident(..)` and a loop-state variant, with no conversion from `Out`. This makes the step named, not provable. Probe: P-Q3. | +| Semi-naive Δ/I (X4) | SKIP | Superseded by operand reuse; a second undemanded population. Measured: it buys nothing span-bound (ndarray D-GTM-1m). | +| Push carried across iterations, per-step direction switch (X2) | SKIP | CONFLICTS-ANCHOR A2. Push stays a final demanded sink. Forward reachability needs a declared reverse lane (RF-TRANSPOSE). | +| Pointer doubling / composed `fkʲ` lanes (X9) | SKIP (TRAP) | Population-sized u32 intermediates; forbidden by A1 *a fortiori*. P-COMPOSE C3 measures it only as a contrast. | +| Yannakakis full reducer / predicate transfer (X12) | PARK | Exact sets for every variable (answers v2 H-2), but only as declared demanded sinks, i.e. multi-sink at the consumer. Bloom filters are refused (approximate). | +| MS-BFS 64-lane bitsets (X14) | PARK | The bit axis would be sources, not rows; it would need a separate plane class. | +| 2-phase law (X11), Kùzu S-Join (X13) | ALREADY-HAVE | External corroboration of A2 / of Gather over a resident plane. | +| Waben "elected reusable frontier" (X17) | PARK | Broader than A1 allows; would be narrowed to the LOOP-STATE bracket if that is adopted. | +| `lgj_hop` push chain (X16) | out of scope | A tension for lance-graph-java's board, not this repo. | + +## Probes (pre-registered by the falsifier designer) + +**P-REUSE** — ndarray example, about 3 h. +- **Fixture:** functional fk, n = 4,096, seed `0xC0FFEE`, with a 3-cycle through start, a self-loop, a tail-into-cycle, a 40-chain and 5 out-of-range fks. +- **Check:** operand reuse == reuse+ANDNOT == semi-naive == two independent oracles, for every t ≤ 45. +- **Kill:** + - the wrong seed `v_0 = start` must differ from the oracle at t = 2 (can-it-fire); + - anti-vacuity: `kept*3 < n`. +- **Cost half:** 65,536 rows, v4 and v3, kill if reuse is more than 1.25× semi-naive. + +**P-DETERMINISM** — about 2 h. +- **Fixture:** a 63-chain across 64 tiles; ascending, descending and 1,000 shuffled tile orders. +- **Expected:** + - in-place at t = 1, ascending: `|v| == 63` where the oracle has `1` (fires); + - double buffering is identical across all orders (silent); + - in-place unbounded equals the closure. + +**P-COMPOSE** — about 4 h plus about 1 day for the primitive. +- **Target:** `start[fkᵏ(i)]` for k ∈ {0,1,2,3,7,16,64}, n ∈ {2¹⁶, 2²⁴}, with an fk out of range at depth 2 only. +- **Arms:** chained masks (forbidden contrast), composed walk, pointer doubling, and the union of depths (must equal P-REUSE at t = k). + +**P-Q3** — about 2 h. +- **Defect pin:** `out_mask_refeeds_as_foreign_plane_today` passing is the defect, recorded OPEN. +- **Guard:** private fields; trybuild compile-fail on the struct literal plus a source fence. +- **Silent half:** all existing tests unchanged except for renamed constructors. + +## Not searched or not read + +- Beamer SC'12 and MS-BFS (SECONDHAND only); +- DuckDB mark/semi-join build pipelining; +- RedisGraph/FalkorDB internals; +- whether quack's `ForeignPlane(pub u16)` needs the same fence. + +## Open, for the operator + +The premise gate was re-run on the final option set; it includes the "belongs elsewhere" option. The options: + +- **(a) Tile-local gather chain** — substrate-first, needs no amendment. Covers Q1 and bounded functional `*1..k`. +- **(b) A1 amendment naming LOOP-STATE** as the one non-demanded population class (bracketed, single owner, never escapes before ∫). Covers multi-valued hops and unbounded `*`. +- **(c) `ForeignPlane` provenance type** — independent of (a) and (b), and closes a live gap. +- **(d) Recursion belongs elsewhere:** mask-risc stays non-recursive, and `*` stays refused (RF-*) or lives in a consumer-side driver. + +(a) and (c) do not need (b). diff --git a/.claude/board/entries/2026-10-03-known-is-not-alpha-premise-gate.md b/.claude/board/entries/2026-10-03-known-is-not-alpha-premise-gate.md new file mode 100644 index 000000000..b1e578d29 --- /dev/null +++ b/.claude/board/entries/2026-10-03-known-is-not-alpha-premise-gate.md @@ -0,0 +1,25 @@ +# Known is not alpha; the premise gate found it blind (2026-10-03) + +**Status:** MEASURED · DECISION recorded in `deepnsm-v2-lexical-address-v1.md` §3.1 and +`.claude/knowledge/reference-frame-vs-motion.md`. + +- **The finding.** A 5+3 council on #1307 asked "which kind of alpha carries + known/unknown?" and verified four options. "Known" is a property of the calibrated + reference set (baked, digested, immutable per version, written by the bake); alpha is + same-coordinate runtime motion (per cycle, discardable, written by `claim`, + `alpha.rs:7-16`). Different signatures, so the question's premise was wrong. +- **The gate.** `.claude/agents/premise-auditor.md` builds a per-concept signature + (answers, coordinates, lifetime, persistence, writer, cardinality; representation never + decides identity) and runs four tests. Measured with a copy stripped of the worked + example (no mention of alpha, #1307 or "known"): + +| case | expected | verdict | +|---|---|---| +| #1307 §3.1 as the council left it | fire | **PREMISE-SPLIT**: (a) is the answer, typed apart from `AlphaMask`; (d) bundles an unrelated attention-recorder question; (b), (c) category errors | +| #1306 OQ-CML-1, picked as a clean control | stay silent | **PREMISE-SPLIT** — the control was not clean: "linked" covered build graph / binary residue / call path / public types, and plan:122 (`Refusal` carries `GraphError`) contradicted plan:142. Verified against `error.rs:43-47`, `Cargo.toml:25-53`, `lib.rs:43`; plan corrected | +| D-LXA-2 generator, Rust example vs Python script | stay silent | **PREMISE-SOUND** | + +- **OPEN.** Three cases is a small sample; the gate's false-positive rate on real option + sets is not measured. The clean-control run also noted that `genre_shapes.rs:18-21` + marks `academic_20k.csv` "license: unverified, do not redistribute", which bears on + committing a 20k-derived TSV (D-LXA-2); not yet checked. diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index e8d88340e..b0e063503 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,11 +25,13 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -180 entries, 2026-08-06 .. 2026-10-03. +182 entries, 2026-08-06 .. 2026-10-03. | date | entry id | finding | file | |---|---|---|---| +| 2026-10-03 | `known-is-not-alpha-premise-gate` | | [2026-10-03-known-is-not-alpha-premise-gate.md](2026-10-03-known-is-not-alpha-premise-gate.md) | | 2026-10-03 | `dir-sim-soa-quack` | Directory simulation on SoA + Quack: one-edge mutation 853 B at 1k and 100k users | [2026-10-03-dir-sim-soa-quack.md](2026-10-03-dir-sim-soa-quack.md) | +| 2026-10-03 | `coresearch-chained-hop` | | [2026-10-03-coresearch-chained-hop.md](2026-10-03-coresearch-chained-hop.md) | | 2026-09-30 | `three-reference-sets-are-not-ordinal-aligned` | | [2026-09-30-three-reference-sets-are-not-ordinal-aligned.md](2026-09-30-three-reference-sets-are-not-ordinal-aligned.md) | | 2026-09-30 | `deepnsm-v2-coverage-bands` | | [2026-09-30-deepnsm-v2-coverage-bands.md](2026-09-30-deepnsm-v2-coverage-bands.md) | | 2026-09-30 | `cypher-mask-v2-is-a-replacement-not-a-phase` | | [2026-09-30-cypher-mask-v2-is-a-replacement-not-a-phase.md](2026-09-30-cypher-mask-v2-is-a-replacement-not-a-phase.md) | diff --git a/.claude/knowledge/reference-frame-vs-motion.md b/.claude/knowledge/reference-frame-vs-motion.md new file mode 100644 index 000000000..faabde55e --- /dev/null +++ b/.claude/knowledge/reference-frame-vs-motion.md @@ -0,0 +1,83 @@ +# Reference frame vs motion — calibrated data is the fixed point, not state + +**READ BY:** `premise-auditor` (every signature table), any session touching +`ReferenceSet`, codebooks, bakes, coverage planes, `contract::alpha` / +`alpha_tunnel`, the cognitive-shader-driver fold loop, or any proposal to +"unify" two bitmaps, lanes or tables that look alike. + +**Status:** DECISION (2026-10-03, `ISS-LXA-ALPHA-FIT`). +**BASIS:** `alpha.rs:7-16` (alpha is "a second table at the *same* +addresses as an already-baked SoA spine", "not a bake", "a record of where +attention went"); `deepnsm-v2-lexical-address-v1.md` §3.1; the blind +premise-gate test that reached the same split without being told. +**REVISIT WHEN** a value that this doc places on the reference side must +change within one reference version (by this decision it cannot; that is a +new version). + +## The two categories + +``` +CALIBRATED REFERENCE (frame) COGNITIVE SESSION (motion) + +corpus, up to ~10⁹ observations + │ compressed offline, once + ▼ +ReferenceSet vN same coordinates i + ├── LexicalAddress[i] │ + ├── lemma / PoS ▼ + ├── frequency thought / rung r0 + ├── lemma_evidence │ alpha Δ + └── coverage[i] (defined / measured) ▼ + thought / rung r1 … +``` + +| | reference frame | motion | +|---|---|---| +| answers | what did a large measurement of the world establish at coordinate *i*? | what is active, attended or different at coordinate *i*, now? | +| source | corpus measurement, compressed offline | one session, one thought, one rung | +| lifetime | immutable for one reference version | per cycle / rung | +| persistence | digested, reproduced byte for byte | discardable whole, or versioned as runtime state | +| writer | the bake | the runtime (`claim`) | +| examples | coverage, frequency, `lemma_evidence`, PoS, the Fisher-z LUT | alpha overlay, attended mask, per-rung lanes | + +**The lever.** The expensive part (corpus → calibration → reference set) +runs once. Everything after it is cheap: reference set → lookup → fold → +fold → … A single fold is almost nothing; a million folds are still cheaper +than re-deriving what the reference already holds, and each of them stands +on the statistics of the original observations. The reference set is the +fixed point the folds lever against. + +**What a large N buys and what it does not.** At very large N, frequency +approaches a stable empirical distribution for the measured population. It +is still not "the truth of the language": corpus choice, genre, period and +speaker population remain bias sources. That is exactly why the reference +set is **versioned and digested**: we know which world was measured. + +## The rules + +1. **Same coordinates, different category, different type.** A coverage + plane and an alpha mask may share a bitmap layout. They must not share a + type, a name, or a conversion. Representation is allowed to repeat; + meaning is not. +2. **Motion never writes the frame.** Alpha and every other runtime overlay + read the reference set and write beside it at the same coordinates. They + never redefine, refine or "correct" a reference value. A changed + reference value is a new reference version, made by a new bake. +3. **The frame is not a cache.** Calibrated data is not derived state to be + invalidated, recomputed per session, or optimized away. Treating it as a + cache is the failure this doc exists to stop. +4. **The code carries the category, not the session's memory.** Every + reference-side type states in its doc comment that it is calibrated, + immutable per version, derived from measurement, and not attention / + alpha / runtime confidence. Every overlay type states that it is + same-coordinate, session-local, and must not redefine reference data. A + type-level guard (no conversion between the two) backs the prose. + +## The tell + +A local, reasonable-sounding simplification — *"these four bitmaps are all +per-coordinate state, let's make one overlay"* — is the signature of this +failure. Before merging two things that look alike, fill their signatures +(`.claude/agents/premise-auditor.md` § Step 1). If they differ on *answers, +lifetime, persistence or writer*, they stay apart however similar their +bytes are. diff --git a/.claude/plans/cypher-mask-lowering-v1.md b/.claude/plans/cypher-mask-lowering-v1.md index 749dd539f..4566ce685 100644 --- a/.claude/plans/cypher-mask-lowering-v1.md +++ b/.claude/plans/cypher-mask-lowering-v1.md @@ -121,7 +121,7 @@ three leaves is one immediate, hence one pass. §3.3 is built on exactly that. | …**and it did not build** (as of this read — ⊘ fixed by PR2 `c095dcc`: only `ir` is declared now, the crate is a workspace member, builds, and is clippy/fmt/test-gated in CI) | `lib.rs:56-61` declared `pub mod exec; pub mod fuse; pub mod hop; pub mod ir; pub mod reference; pub mod ternlog_table;` — **only `ir.rs` and `lib.rs` existed on disk** (verified by directory listing) | five of six modules were absent; `pub use exec::…` / `fuse::…` at `:63-64` could not resolve | | …**and it was not in the workspace** (as of this read — ⊘ PR2 adds it to `members`) | `Cargo.toml:2-28` members, `:29-…` exclude — `lance-graph-mask-risc` appeared in **neither** list | an orphan directory at the time: nothing compiled it, nothing gated it | | Shipped T1 consumer inside this repo | `crates/lance-graph-planner/src/nested_bands.rs:32` imports `gt_i32_to_mask, le_i32_to_mask, mask_and, mask_ternlog, popcount_batch_u64`; `:161` `mask_ternlog::`; `:439` `::` (⊘ now `mask_and`, the exact name, after the PR2 council); `:623` is a `#[cfg(test)]` alias of `AND2`, not a third immediate | **the precedent: `lance-graph-planner` already depends on `ndarray` (`Cargo.toml:24`) and already calls T1 by name.** No new dependency edge is needed for the planner half | -| `AlphaMask` | `crates/lance-graph-contract/src/alpha.rs:224-230` — `words: Box<[u64]>`, `len: u32`, tail bits PHANTOM and every op that could raise them must clear them | the contract-side mask carrier, same LSB-first order as `ndarray::simd` (`mask-risc/src/lib.rs:4-6`) | +| `AlphaMask` | `crates/lance-graph-contract/src/alpha.rs`, `pub struct AlphaMask` — `words: Box<[u64]>`, `len: u32`, tail bits PHANTOM and every op that could raise them must clear them | the contract-side mask carrier, same LSB-first order as `ndarray::simd` (`mask-risc/src/lib.rs:4-6`) | | The hop, as an ABI | `lance-graph-java/native/lgj-abi/src/exports.rs:2053` `lgj_hop(store, edge_classid, facet_mask, decode_mode, src_mask, dst_mask)` | the `src_mask → hop → dst_mask` shape, shipped | | …its selection algebra | `exports.rs:2026-2032` + `:2104-2118`: `selected_f = ternlog(class_f, src, struct_f)`, `dst = ⋁_f scatter(selected_f)`; *"No row is examined to decide whether it participates"* | **the pattern §3.4 reuses verbatim** | | …and its scar tissue | `exports.rs:2034-2044`, `:2118-2136`: three prior shapes each traded algebra for arithmetic; a per-row gather *"measured faster on the AoS store and shipped briefly; the operator ruled it out"* | read this before proposing a walk | diff --git a/.claude/plans/cypher-mask-lowering-v2.md b/.claude/plans/cypher-mask-lowering-v2.md index 9d910e063..e69aa3cf2 100644 --- a/.claude/plans/cypher-mask-lowering-v2.md +++ b/.claude/plans/cypher-mask-lowering-v2.md @@ -119,7 +119,11 @@ contract-first (D-CML-0 checks it). **The pipeline inside `run`, in order.** Every failure along it is a `Refusal`, so no failure can produce an `Answer`: -1. **Parse.** A parser error becomes RF-UNPARSED, carrying the upstream `GraphError`. +1. **Parse.** A parser error becomes RF-UNPARSED, carrying an **owned reason** (message + and source position) that the front-end adapter extracts from the upstream + `GraphError`. The `GraphError` itself never crosses the crate's public surface: it + wraps `DataFusionError`, `lance::Error` and `ArrowError` (`error.rs:43-74`). The same + conversion applies to every upstream error in steps 3 and 4. 2. **Label check against `LabelBinding`**, by walking the parsed AST, before planning. The upstream check is **partial**: the planner rejects an unmapped label only when a node variable is returned (`logical_plan.rs:517-526`), and semantic @@ -164,25 +168,17 @@ failure can produce an `Answer`: walks is refused (§5 row RF-BAG). After a push hop there is no terminal at all, only the mask itself (§4.1). -**OQ-CML-1 — DataFusion (and more) is still linked.** `lance-graph` depends on `datafusion` -unconditionally (in `crates/lance-graph/Cargo.toml` the `datafusion` entry under -`[dependencies]` carries no optional flag), and `error.rs` (`GraphError`) puts `DataFusionError` inside `GraphError`. So depending on the -parser pulls DataFusion into the **build graph**, even though the new crate calls none -of it. This is "off the surface", not "out of the binary". Making DataFusion optional -upstream would be an upstream edit, and that is ruled out. The two options, to be -decided by **measurement in D-CML-0**: - -- **(a)** accept it as a link-time dependency. Measure whether any DataFusion symbol - survives into a release binary of the new crate. No existing CI step can measure - this, because none builds `lance-graph` without DataFusion (it is not optional). So - D-CML-0 adds one release-build step that runs `nm` on the binary. - The crate is a library, so the measured artifact is a `--release` `[[example]]` that - calls `run`. The step's disable: an example that references `datafusion_planner` - must show DataFusion symbols, and the step must go red on it. -- **If (a) measures DataFusion in the binary, that is a STOP**, decided as its own - D-id. It is not a pre-authorised fallback. A pinned copy of `parser.rs` / `ast.rs` / - `semantic.rs` / `logical_plan.rs` inside the new crate would be a second authority - for the AST, so it is not the default remedy. +**OQ-CML-1 — what "DataFusion is linked" means, split into four questions.** +⊘ The earlier wording asked one question, "is DataFusion linked?", and answered it with +an `nm` scan. The premise gate (`.claude/agents/premise-auditor.md`, 2026-10-03) showed +that "linked" covered four concepts with different answers and different deciders: + +| | question | status | how it is decided | +|---|---|---|---| +| **C2 build graph** | does compiling the new crate compile DataFusion? | **certain, yes**: `datafusion`, `lance`, `arrow`, `object_store` and `lance-graph-hydrate` are non-optional (the `[dependencies]` block of `crates/lance-graph/Cargo.toml`), and `pub mod datafusion_planner` is ungated (`crates/lance-graph/src/lib.rs`) | not measured. **Accepted as a stated build cost.** Removing it needs an upstream edit (ruled out) or a pinned copy of the front end (8 files, 6,845 lines including `error.rs`, `config.rs`, `case_insensitive.rs`, `parameter_substitution.rs`; and the copied `error.rs` still names `DataFusionError`) — a second AST authority, so a STOP, not a remedy | +| **C4 call path** | does `run` execute any DataFusion code? | must be **no** | the import fence F-CML-FENCE (§8), with its disable per path; not `nm` | +| **C5 public types** | does any public type of the new crate name an upstream type? | must be **no** | a fork-owned design rule, decided here: upstream errors become owned reasons in the adapter (§3 step 1). F-CML-SURFACE (§8) checks it | +| **C3 binary residue** | does a release artifact still contain `datafusion*` symbols? | an observation | the `nm` step over a `--release` `[[example]]`. It is **recorded, not a gate**: a hit does not by itself trigger a STOP, because C3 can be non-empty for reasons that are not C4 (e.g. a `Display` impl reached through a formatted error) | --- @@ -485,7 +481,7 @@ are reachability or support questions. | D-id | what | depends on | STOP if | |---|---|---|---| -| **D-CML-0** | Verify the §2 upstream/fork split against upstream history. Create the crate skeleton: a workspace `members` entry (not `exclude`, no own `[workspace]`), three deps with `lance-graph` at `default-features = false`, and an import-fence test. Add CI lines: a test step in `rust-test.yml`, clippy and rustfmt lines in `style.yml`, one release `nm` step over a `[[example]]` binary for OQ-CML-1(a) (with its disable), and one `cargo tree -e features -i lance-graph` check that fails if `planner` is active in the new crate's feature set (feature unification can turn it back on). Confirm that `NodeRow` has a sound zero-copy byte view. Update `lance-graph-mask-risc/src/lib.rs:38,115`, which still names v1's in-`query.rs` `mask_lower` seam. Add a `LATEST_STATE` new-member row | — | a §2 file is fork-owned (the split is redrawn, not the design); or `NodeRow` has no sound byte view (STOP → contract-first) | +| **D-CML-0** | Verify the §2 upstream/fork split against upstream history. Create the crate skeleton: a workspace `members` entry (not `exclude`, no own `[workspace]`), three deps with `lance-graph` at `default-features = false`, and an import-fence test. Add CI lines: a test step in `rust-test.yml`, clippy and rustfmt lines in `style.yml`, one release `nm` step over a `[[example]]` binary recording OQ-CML-1's C3 residue (an observation, not a gate), and one `cargo tree -e features -i lance-graph` check that fails if `planner` is active in the new crate's feature set (feature unification can turn it back on). Confirm that `NodeRow` has a sound zero-copy byte view. Update `lance-graph-mask-risc/src/lib.rs:38,115`, which still names v1's in-`query.rs` `mask_lower` seam. Add a `LATEST_STATE` new-member row | — | a §2 file is fork-owned (the split is redrawn, not the design); or `NodeRow` has no sound byte view (STOP → contract-first) | | **D-CML-1** | `Route` + `run` stub that refuses everything with `RF-NOT-LOWERED`. The switch compiles, and its refusal reason is true | 0 | — | | **D-CML-2** | **The classifier.** Walk the public `LogicalOperator` and return `Lowerable { variables whose node sets are asked for }` or a §5 `Refusal`. Two answers only. **#1305's `consumer_semantics()` is NOT ported:** its count and binding kinds describe per-path state to be carried through a hop, and v2 refuses those queries instead (§12). Ported from #1305: the three pattern-shape refusals (§12 H-4) and the fixtures (§12 H-1..H-3). Re-run the W0-b census under v2 and report per-variant counts | 0 | — | | **D-CML-3** | `LabelBinding` — label → `LabelDTO` → `u32` classid; per property a layout declaration `(field offset, width, kind)`; per relationship type its carrier/direction declaration (§4.1), checked against the class's `ClassView`. Built from `LabelDTO::from_canonical` where the codebook covers the label, and otherwise **supplied explicitly** by the consumer, never guessed. It is the first consumer of `LabelDTO`. The corpus labels are not in the codebook (modelgraph §16.3), so the explicit path is the common one | 0 | — | @@ -512,6 +508,7 @@ first step that executes anything. D-CML-5 waits on 3b and 5a. | **F-CML-FENCE** | outside `#[cfg(test)]`, the new crate references no path under `lance_graph::query`, `lance_graph::datafusion_planner`, `lance_graph::sql_*`, or `datafusion*` | add one reference per path; the gate must go red for each | | **F-CML-CARRIER** | a relationship declared on an ordinal carrier over a class whose `edge_codec_flavor` is `Pq32x4` is refused | drop the `ClassView` check; the query must now lower | | **F-CML-UNDIRECTED** | `a-[*1..2]-a` over one undirected edge is refused (RF-DEPTH) | drop the check; the answer must now wrongly contain `a` | +| **F-CML-SURFACE** | no public item of the new crate (including every `Refusal` payload) names a `lance_graph`, `datafusion*`, `lance*` or `arrow*` type; upstream errors cross only as owned reasons (OQ-CML-1 C5) | add a `GraphError` field to one `Refusal` variant; the gate must go red | | **F-CML-PLACEHOLDER** | the lowering reads none of the `GraphConfig` placeholder fields (`id_field`, `property_fields`, `source_id_field`, `target_id_field`) | make the lowering read one; the gate must go red | | **F-CML-UNDECLARED** | a relationship type with no declaration is refused (RF-UNDECLARED-REL), and the same query with a declaration lowers | drop the check; the undeclared query must now lower or panic | | **F-CML-REFUSE** (can-fire) | every §5 variant is produced by at least one committed query | delete a variant's arm; its query must now either lower (and fail the differential) or panic | diff --git a/.claude/plans/deepnsm-v2-lexical-address-v1.md b/.claude/plans/deepnsm-v2-lexical-address-v1.md index e9be9909a..f37d9a712 100644 --- a/.claude/plans/deepnsm-v2-lexical-address-v1.md +++ b/.claude/plans/deepnsm-v2-lexical-address-v1.md @@ -1,8 +1,9 @@ # deepnsm-v2-lexical-address-v1 — a word is a 16-bit address into a versioned, baked COCA codebook > **Status:** PROPOSAL (D-LXA-1..4). Plan only; no code is authorized by this file. -> **Council:** 5+3, ratified v3 (2026-09-30). The change ledger is §7. §3.1 carries one -> **operator escalation** (the alpha-channel fit). D-LXA-3 does not start before it is ruled. +> **Council:** 5+3, ratified v3 (2026-09-30). The change ledger is §7. §3.1's operator +> escalation (the alpha-channel fit) is **resolved** (2026-10-03): known is the reference +> set's `ReferenceCoverage` plane, not alpha. D-LXA-3 is blocked only on D-LXC-4. > **Written against:** `main` `0d31c54f` (2026-09-30). > **Harvested from:** PR #1303 (`deepnsm-v2-cam96-pairwise-v5`), which was **closed without > merging**. Only four things from it are kept here: @@ -16,7 +17,7 @@ > itself), the §11/§11R execution socket, and the "baton" doc-comment edits. > **Board:** `STATUS_BOARD.md` § deepnsm-v2-lexical-address · entry > `entries/2026-09-30-three-reference-sets-are-not-ordinal-aligned.md` · `ISSUES.md` -> `ISS-CE64-EMIT-INVERSE-BIT2-DISAGREE`, `ISS-LXA-ALPHA-FIT` (the §3.1 escalation). +> `ISS-CE64-EMIT-INVERSE-BIT2-DISAGREE`, `ISS-LXA-ALPHA-FIT` (the §3.1 escalation, resolved 2026-10-03). --- @@ -197,8 +198,8 @@ The bake is a **separate, derived artifact** that crosses that line on purpose, NaN-like "unknown" value. A row whose value is unknown still holds a canonical fill byte, `0`, so the byte-for-byte gate is reproducible. The fill means nothing on its own: only the known bit says whether the byte is a value. -- **Measured or not is a bit outside the bytes** (operator ruling F4; the mechanism is - subject to §3.1). The baked table is the **spine**: complete, one row per reference +- **Measured or not is a bit outside the bytes**: the reference set's **coverage + plane**, baked with it (§3.1, resolved). The baked table is the **spine**: complete, one row per reference entry, read-only, read by every reader without a lock. Unknown is an **unset bit**. It is not a byte value and not a missing row. The table keeps its shape, so addresses stay dense and no reader ever handles a hole. @@ -207,13 +208,49 @@ The bake is a **separate, derived artifact** that crosses that line on purpose, - The known bit is forced in the type, like `Exact` / `Ambiguous`. The prior accessor returns `Option` (the same rule as `lexical.rs:46-52`: unknown is not zero). A reader cannot use `f` or `lemma_evidence` without having handled the unknown case. -- **How** the bit is carried is escalated (§3.1). The council found that the shipped - alpha API does not fit the first wording of this section. +- The coverage plane is typed `ReferenceCoverage`, not `AlphaMask` (§3.1). It may use the + same bitmap layout; it is a different thing. - `ln` is `f64::ln` and `freq_max` is the maximum over the reference being baked. `K` is written into the artifact header next to the three digests, so it is part of what is reproduced. -### §3.1 — Alpha-channel fit: ESCALATED to the operator (`ISS-LXA-ALPHA-FIT`) +### §3.1 — Known is not alpha (`ISS-LXA-ALPHA-FIT`, RESOLVED 2026-10-03) + +**DECISION.** "Known / measured" is a property of the **calibrated reference set**, not +of the alpha channel. Each baked table carries a **coverage plane** (1 bit per row, +digested with the bake, written by nothing at runtime). It may reuse `AlphaMask`'s bitmap +layout, but it is a separate type, `ReferenceCoverage`, and is never called alpha. +Alpha stays exclusively a same-coordinate overlay across time and cognitive rung. + +**BASIS.** The two answer different questions over the same coordinates: + +| | coverage (reference frame) | alpha (motion) | +|---|---|---| +| answers | is this coordinate defined / measured in this reference? | what is active or different at this coordinate, now, at this rung? | +| source | corpus measurement (up to ~10⁹ tokens), compressed offline | one session / thought | +| lifetime | immutable for one `ReferenceSet` version | per cycle / rung, sparse | +| persistence | digested, reproduced byte for byte | discardable whole (`alpha.rs:11-16`) | +| writer | the bake | runtime `claim` (`alpha.rs:682-716`) | + +Physically both can be the same bits; epistemically one is the measured world and the +other is a train of thought moving over it. Filing the first under the second would +recalibrate the metre at every measurement. Frequency, `lemma_evidence` and coverage all +sit on the reference side. Doctrine: `.claude/knowledge/reference-frame-vs-motion.md`. + +**How the earlier options resolve** (kept for the record; the premise of the question, +"which kind of alpha carries known?", was wrong — found blind by the premise gate, +`.claude/agents/premise-auditor.md`): +- (a) is the answer, renamed: a baked coverage plane, typed `ReferenceCoverage`. +- (b) is a category error and an architecture tax (20,845 × 512 B ≈ 10.2 MiB to give + 1-bit coverage a `NodeRow` home). Dropped. +- (c) is a category error: "measured" is not an alpha meaning. Dropped. +- (d) was (a) plus a second, unrelated question — an attention recorder over lexical + addresses. That question is now §5 open, and does not block D-LXA-3. + +**REVISIT WHEN** a lexical coordinate's definedness must change *within* one reference +version. By this decision it cannot; that would be a new version. + +#### The council's original escalation (superseded; kept for the record) **The ruling, verbatim** (operator, 2026-09-30, on known versus unknown): *"we already have alpha channel split tunnel trick for that"*. @@ -267,9 +304,9 @@ for this question. with the meaning its code gives it. It never stores a value in the overlay. The refinement of `f` / `lemma_evidence` is never an overlay write (F3; `claim` is stamp-only). -**The question to the operator:** confirm that "known" is a baked coverage mask shaped -like `AlphaMask` (options a or d), and that the alpha overlay keeps its "attended" -meaning. +**The question to the operator** (answered above): confirm that "known" is a baked +coverage mask shaped like `AlphaMask` (options a or d), and that the alpha overlay keeps +its "attended" meaning. `range` and `disp` are left out. They are collinear with frequency, and `disp` measures evenness across genres, not evidence. @@ -296,9 +333,9 @@ Frequency rank stays a routing signal, not meaning (ρ ≈ −0.07, archive F11) | D-id | what | gate (can fire / can stay silent) | |---|---|---| -| **D-LXA-1** | `LexicalAddress` newtype over `u16` (a **new** identity key beside the surface-form `WordId`, joined by `readings(WordId) → [LexicalAddress]`, §1) plus `ReferenceSet { id ∈ {COCA4096, COCA5K_LEMMA, COCA20K_ACAD}, version, sha256 }`, in `deepnsm-v2`. Resolving through the wrong reference is a refusal. No bare `[u8; 12]` or `u16` enters the path | **G-LEX** (over typed `LexicalAddress` values; nothing binds raw facet bytes to a reference before D-LXA-4): every declared entry of a reference resolves to exactly its declared reading **under that reference's own key** (§2 table): `(word, PoS)`, `(lemma, PoS)`, or `(word, Ambiguous{PoS set})`. A cross-reference read is refused. A `compile_fail` test proves a bare `u16` is not accepted | +| **D-LXA-1** | `LexicalAddress` newtype over `u16` (a **new** identity key beside the surface-form `WordId`, joined by `readings(WordId) → [LexicalAddress]`, §1) plus `ReferenceSet { id ∈ {COCA4096, COCA5K_LEMMA, COCA20K_ACAD}, version, sha256 }`, in `deepnsm-v2`. Resolving through the wrong reference is a refusal. No bare `[u8; 12]` or `u16` enters the path. Also `ReferenceCoverage` (§3.1): a coverage plane owned by one `ReferenceSet` version, with **no** `From` / `Into` / `AsRef` to or from `AlphaMask` and no setter after the bake | **G-LEX** (over typed `LexicalAddress` values; nothing binds raw facet bytes to a reference before D-LXA-4): every declared entry of a reference resolves to exactly its declared reading **under that reference's own key** (§2 table): `(word, PoS)`, `(lemma, PoS)`, or `(word, Ambiguous{PoS set})`. A cross-reference read is refused. A `compile_fail` test proves a bare `u16` is not accepted. A second `compile_fail` test proves a `ReferenceCoverage` cannot be passed where an `AlphaMask` is expected, nor built from one (with a passing twin that builds it from the bake's words) | | **D-LXA-2** | Generator for `lexical_correspondence.tsv`: one row per (lemma, PoS) with `id4096 \| id5k \| id20k \| status ∈ {Exact, Ambiguous{n}, Missing}`, the three digests in its header. Generated, never hand-edited | **G-REF:** re-deriving the file must reproduce §2's numbers exactly, including "4 of 4,264" **with ordinal 0 counted** (a falsy-zero generator must fail it). Hand-editing one ordinal must turn the check red | -| **D-LXA-3** | The COCA bake of §3: per reference, an identity table (`lemma`, `PoS`, `lemma_evidence`) for each identity-keyed reference and a surface-form table (`form`, reading, `f`), all `u8`, with the known bit carried as ruled in §3.1. A surface-keyed reference reaches `lemma_evidence` per candidate reading through the D-LXA-2 correspondence map, never as one aggregated value (§3). `lemma_evidence` is baked, never derived from a runtime rung (a rung is not reproducible). **Blocked on §3.1 and on D-LXC-4** | Re-deriving must reproduce the bake byte for byte. Unknown rows hold fill byte `0` with the known bit unset. A form with one `form_count = None` reading must come out unknown (fires); the all-listed fixture must not (stays silent). Pinned rows: surface `the` → `the/a` with `f = 1`; surface `record` → `record/n` with `f < 1` and `record/v` with `f > 0`, the two summing to 1 within quantization. An all-unambiguous fixture must give `f = 1` everywhere (stays silent) | +| **D-LXA-3** | The COCA bake of §3: per reference, an identity table (`lemma`, `PoS`, `lemma_evidence`) for each identity-keyed reference and a surface-form table (`form`, reading, `f`), all `u8`, with the known bit in each table's `ReferenceCoverage` plane (§3.1). A surface-keyed reference reaches `lemma_evidence` per candidate reading through the D-LXA-2 correspondence map, never as one aggregated value (§3). `lemma_evidence` is baked, never derived from a runtime rung (a rung is not reproducible). **Blocked on D-LXC-4** | Re-deriving must reproduce the bake byte for byte. Unknown rows hold fill byte `0` with the known bit unset. A form with one `form_count = None` reading must come out unknown (fires); the all-listed fixture must not (stays silent). Pinned rows: surface `the` → `the/a` with `f = 1`; surface `record` → `record/n` with `f < 1` and `record/v` with `f > 0`, the two summing to 1 within quantization. An all-unambiguous fixture must give `f = 1` everywhere (stays silent) | | **D-LXA-4** | The six-slot reading: a ClassView-selected reading of a 12-byte facet as six `LexicalAddress`es under one named `ReferenceSet`, carried by the classid. A register in any other shape is refused, never reinterpreted. **This is a contract change, gated on its own contract plan (not yet written).** Today no reader can refuse: `SpoFacet::from_register` takes a bare `[u8; 12]` (`awareness_facet.rs:106`), and `Cam96 = [u8; 12]` (`space.rs:163`) appears 68 times in 12 files (grep, counting doc comments). `ReadMode` / `ValueSchema` must first gain a lexical reading. **Existing `Cam96` / `SpoFacet` classids keep their current reading unchanged. The lexical reading exists only under a newly minted classid / reading mode, and nothing re-reads existing rows** (I-LEGACY-API-FEATURE-GATED). D-LXA-1..3 ship without it | A facet written under reference X and read under Y is refused. A facet written under `(X, v1)` and read under `(X, v2)` is refused. Rotating the six slots changes the resolved words (this proves the six positions are ordered, not a bag). These three gates move verbatim into the contract plan; the STATUS_BOARD row carries them until it exists | **Collision:** D-LXA-3 reads `academic_20k.csv`, which `D-LXC-4` (the academic loader, @@ -306,15 +343,21 @@ currently Blocked on a ruling about three duplicate (word, PoS) pairs) also owns waits for that ruling, or takes it as its own first question. It must not duplicate the loader. -**Ownership:** the baked tables and coverage masks are read-only artifacts with no -mailbox. Nothing writes them at runtime. If §3.1 rules an alpha overlay over the codebook -(options b, c or d), the overlay's writer is the owning mailbox +**Ownership:** the baked tables and their coverage planes are read-only artifacts with +no mailbox. Nothing writes them at runtime. If an attention recorder over lexical +addresses is ever built (§5), its overlay's writer is the owning mailbox (`SoaEnvelope::mailbox_owner`); a consumer never writes as itself. --- ## §5 — Open +- **An attention recorder over lexical addresses** (split out of §3.1 option (d)): does the + driver need an alpha overlay recording which lexical addresses a thought visited, per + cycle and rung? If so, its address space costs either a `NodeRow` projection of the + codebook (512 B per entry) or an `AlphaAllocation` contract change. Its own plan; it + never carries coverage, frequency or evidence. + - **Where the six slots live:** the row's second facet (bytes 16..32) or a new 16-byte `Identity` value tenant. Not decided. Either choice is additive and leaves `NODE_ROW_STRIDE` unchanged. diff --git a/.claude/skills/coresearch/SKILL.md b/.claude/skills/coresearch/SKILL.md new file mode 100644 index 000000000..11a2b714f --- /dev/null +++ b/.claude/skills/coresearch/SKILL.md @@ -0,0 +1,44 @@ +--- +name: coresearch +description: > + Convene the co-research council for an OPEN question where outside + knowledge may change the answer: 5 scouts in two rings (inside: code + cartographer + internal prior art; outside: arXiv literature, known + systems such as DuckDB / Odoo, ontologies and concepts), a crosswalk of + outside ideas onto inside surfaces, then 3 co-architects (bridge, + firewall/fit, falsifier) and an exploration map with ADOPT-NOW / PROBE / + PARK / SKIP per idea. Ratifies nothing; a chosen design goes to a plan or + /5plus3. Canonical harness: .claude/agents/coresearch-council.md. +--- + +# /coresearch — explore the code and the outside world together + +Bootload: **read `.claude/agents/coresearch-council.md` in full.** It is the +canonical harness. This file is only the invocation stub. + +Checklist (each step gates the next): + +1. **Qualify.** Is the question open, and could outside knowledge change the + answer? If it is decided, run nothing. If it is a committed design, use + `/5plus3`. If two sources answer it, read them. +2. **Phase 0 — QUESTION BRIEF** (main thread): the question, anchors + (cited, never re-opened), the internal surface by path, external + domains with seed terms and the knowledge docs that already cover them, + what would count as an answer, and the budget. +3. **Phase 1 — cast the 5 scouts** in ONE parallel spawn: code + cartographer, internal prior art, literature, systems, concepts and + ontology. Each item carries a source id and a grade: outside sources + use `READ-IN-FULL` / `SECTION-READ` / `ABSTRACT-ONLY` / `SECONDHAND`; + the code cartographer uses `VERIFIED-IN-CODE` / `CLAIMED` / `ABSENT` + (closed search space only). +4. **Phase 2 — CROSSWALK** (main thread): one row per idea, with the + relation `ALREADY-HAVE` / `PARTIAL` / `NEW` / `CONFLICTS-ANCHOR`. Raw + scout output is banked, never forwarded. +5. **Phase 3 — cast the 3 co-architects** in ONE parallel spawn, on the + crosswalk only: the bridge architect, the firewall and fit critic, and + the falsifier designer. +6. **Phase 4 — EXPLORATION MAP**: ADOPT-NOW / PROBE / PARK / SKIP per idea, + the named probes, and what was NOT searched. +7. **Phase 5 — land** one `entries/YYYY-MM-DD-coresearch-.md` plus + `entries_index.py --write`. Then ask the operator which ideas to take + forward. Nothing is adopted by the council itself. diff --git a/crates/lance-graph-contract/src/alpha.rs b/crates/lance-graph-contract/src/alpha.rs index 846021cd8..fb0b67886 100644 --- a/crates/lance-graph-contract/src/alpha.rs +++ b/crates/lance-graph-contract/src/alpha.rs @@ -45,6 +45,23 @@ //! therefore a **compile-time** property of this type, not a runtime check — //! and deliberately not a test, because a test of it could not fail. //! +//! # What the alpha channel is NOT — calibrated reference data +//! +//! Alpha is **motion over a fixed frame**: same coordinate geometry as the +//! spine, session-, time- or rung-local, sparse, discardable. It records +//! where a train of thought went. It is never the home of a value that a bake +//! measured — coverage ("is this coordinate defined in the reference?"), +//! frequency, evidence, part of speech, calibration tables. Those belong to +//! the reference set: calibrated, immutable for one reference version, +//! derived from corpus measurement, digested. A changed reference value is a +//! new reference version from a new bake, never an overlay write. +//! +//! The two may share a bitmap layout; they must not share a type, a name or a +//! conversion. In particular [`AlphaMask`] is a population bitset over this +//! overlay's addresses; a reference set's coverage plane is a separate type +//! even when its bits look the same. Rationale and the decision record: +//! `.claude/knowledge/reference-frame-vs-motion.md` (`ISS-LXA-ALPHA-FIT`). +//! //! # What this PoC does NOT do //! //! The saccade *direction* is carried as the claim order ([`AlphaStamp::seq`]), @@ -215,6 +232,14 @@ pub struct AlphaClaim { /// survives untouched: the walk still never reads alpha — the diff happens /// HERE, above both planes, on two masks that each side produced blind. /// +/// # Not a reference coverage plane +/// +/// Both operands above are populations over THIS overlay's addresses, computed +/// for a question. A bake's "is this coordinate defined / measured" plane is +/// calibrated reference data, immutable per reference version — a different +/// category with its own type, even when its bits look the same. Never build +/// one from the other (see the module doc, "What the alpha channel is NOT"). +/// /// # The one named materializer /// /// Per the same law, no unnamed materializer exists: the only way ordinals