diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 9937be851..c27f12d24 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -5,7 +5,8 @@ Plan: `.claude/plans/population-law-crosscheck-v1.md`. Entry: `entries/2026-10-0 | D-id | scope | status | gate / falsifier | |---|---|---|---| | **D-PLX-0** | cross-check of #1311 against the real-data fold experiment; reclassify the five gaps | Shipped | plan §A–§M | -| **D-PLX-1** | test-only falsifier: a K-space fold sink read back per row through an fk (lane kind vs result handle) | Queued | plan §L | +| **D-PLX-1** | test-only falsifier: a K-space fold sink read back per row through an fk (lane kind vs result handle) | Withdrawn | a fold is already the projection (plan §N); no probe kept | +| **D-PLX-3** | ANOVA real-data → ARM / SPOFC convergence: two physical population-count engines, one evidence semantics | Shipped (ANALYSIS) | plan §P | ## D-FPW — Frontend parity witness (2026-10-03) diff --git a/.claude/board/entries/2026-10-03-population-law-crosscheck.md b/.claude/board/entries/2026-10-03-population-law-crosscheck.md index 4c802346f..2e68795ef 100644 --- a/.claude/board/entries/2026-10-03-population-law-crosscheck.md +++ b/.claude/board/entries/2026-10-03-population-law-crosscheck.md @@ -23,5 +23,14 @@ re-derived from `crates/deepnsm/word_frequency/academic_20k.csv` (20,845 / 20,84 - **#1311 gaps:** G1 confirmed general; G2 split (re-anchor vs phase boundary); G3 and G5 simple missing operators; G4 mis-specified witness (by source, a missing `masked_sum_i32_via`). -- **Next:** one test-only falsifier (a fold sink read back per row through an fk) to - decide lane kind vs typed result handle. +- **⊘ Corrected 2026-10-03 (#1313):** resident data + reference register → fold → + zero-copy projection. A pivot rotates the register; only a write materializes. + Pair → spelling folds directly from O (no phase boundary). Grouped output is a + projection, not a population. Equal row count does not align rows. The falsifier + above (D-PLX-1) is withdrawn. +- **Recovered from `55bf7cde` (`recovery/1313-pre-reset`):** I→S presence over a bounded + domain is a missing per-group OR fold state, not a phase dependency. +- **ARM / SPOFC convergence (D-PLX-3, plan §P):** two physical population-count engines + (Quack / mask-risc scalar Count; arm-discovery `Dataset` / `RowMasks`), one evidence + semantics: CandidateRule evidence `|X|`, `|X∧Y|`, n; support gate `|X∧Y|/n`; + `arm_to_truth_u8` f = `|X∧Y|/|X|`, c = `|X∧Y|/(|X∧Y|+k)` → {s,p,o,f,c}. diff --git a/.claude/plans/population-law-crosscheck-v1.md b/.claude/plans/population-law-crosscheck-v1.md index 2eb1d6752..a33094682 100644 --- a/.claude/plans/population-law-crosscheck-v1.md +++ b/.claude/plans/population-law-crosscheck-v1.md @@ -1,6 +1,9 @@ # population-law-crosscheck-v1 — which execution law survives two independent witnesses > **Status:** ANALYSIS + source verification. No code changed. D-PLX-0..1. +> **⊘ Revised 2026-10-03 (#1313):** two payloads: the #1312 rollover (§N, §O) and the +> ANOVA real-data → ARM / SPOFC convergence (§P, D-PLX-3). Read §N first. §D, §E, §F, §G, §H, §I, §L and §M +> carry dated ⊘ notes. D-PLX-1 is withdrawn. > **Inputs:** merged #1311 (`2f2b67c`, `crates/lance-graph-quack/tests/gremlin_parity.rs`, > `.claude/plans/frontend-parity-witness-v1.md`) and an independent population-fold > experiment over a real 20,845-row source file (reported to this session, not run here; @@ -75,6 +78,10 @@ rule. They only decide which of 2, 3 and 4 applies. a produced result becomes an operand (G2). The only exception is a resident population whose rows already are the composed paths, and that is rule 1 again. +> **⊘ Corrected 2026-10-03 (§N).** "Either way a produced result becomes an operand" is +> withdrawn. It was not checked whether a register orientation over the resident data +> reaches this case, so no write is shown to be required. The re-anchor bullets stand. + ## E. I → S root cause (the 1.19 MB key column) - The question ("how many (spelling, PoS) identities per spelling") is a **presence @@ -101,6 +108,13 @@ rule. They only decide which of 2, 3 and 4 applies. **Classification: CURRENT API ARTIFACT** for the key column; the phase dependency itself is **SEMANTICALLY NECESSARY** without an ordered projection. +> **⊘ Corrected 2026-10-03 (§N).** Pair → spelling is **not** a phase boundary. Resident: +> the O rows. Register: pointed at spelling. PoS has a small fixed domain, so presence +> per spelling fits in 16 bits of fold state. One fold over O yields the spelling +> projection, and no (spelling, PoS) population is ever written. The current IR has no +> such fold state (`GroupFold` is `Count / MinI32 / MaxI32 / SumSymI32`); that gap is +> not a write. The key-column verdict and the layout note stand. + ## F. Result-as-operand root cause | result | physical form | typed form | readable without copy | operand / group key / predicate input today | @@ -119,7 +133,7 @@ not), and a field view over record sinks. **Where the boundary actually bites.** Quack rules HAVING deliberately: *"HAVING is not a new primitive and not a population operation… the O(K) finalization over those sinks… No O(N) mask crosses a program boundary"* (`quack/src/lib.rs`, `GroupHaving` doc). So -a phase B whose output stays in the K-space (HAVING, AVG's `avg_finish`, a scalar +a phase B whose output stays in the K-sized result domain (HAVING, AVG's `avg_finish`, a scalar statistical finish over sufficient statistics) is **already handled by design, as K-sized host finalization**. (It sits in tension with the crate's "never evaluate a `Program`" rule, but the crate states it as a choice, not a gap.) @@ -130,6 +144,15 @@ an edge row reading the per-vertex count from the previous hop (§D). No host finalization can serve that without either a host loop over N rows (the duplicate evaluator) or a population-sized copy. +> **⊘ Withdrawn 2026-10-03 (§N).** The table stays as an inventory of what each terminal +> writes. The paragraphs that name a missing capability are withdrawn: a fold over +> resident data is already the projection, so another projection is another fold under a +> rotated register, not a read of the previous output. No mechanism is proposed. +> Open, not classified here: a later pass whose per-row predicate reads a completed fold +> over other rows (e.g. lines whose partner has exactly v posted lines). Neither a +> register rotation nor a write is shown for it; withdrawing D-PLX-1 withdraws the +> probe, not the question. + ## G. G1 vs G2 — PARTIALLY SHARED - **Shared:** a completed keyed result is a population over its own coordinate space @@ -142,9 +165,14 @@ evaluator) or a population-sized copy. as lanes and compare them. - G2 that crosses back into an N-row pass is independently needed by a per-row read of an I value from F (experiment) and by the A → R1 → R2 traversal (frontend side, - §D). G2 confined to the K-space (HAVING, I→S's present-cell count) is served today + §D). G2 confined to the K-sized result domain (HAVING, I→S's present-cell count) is served today by host finalization. G1 is needed only by refolds along a key component. +> **⊘ Corrected 2026-10-03 (§N).** Struck: "a completed keyed result is a population over +> its own coordinate space". Grouped output is a projection of the resident data, not a +> new population. A refold along a key component is a fold of the same resident data +> with the register pointed at the coarser key. + ## H. Multi-terminal vs multi-phase | shape | workloads | needs | @@ -155,6 +183,9 @@ evaluator) or a population-sized copy. Adding terminals to one `Program` removes none of the multi-phase dependencies: each phase-B input is a *completed* fold. +> **⊘ Corrected 2026-10-03 (§N).** I→S leaves the multi-phase row (§E note). No remaining +> row is shown to require a write; none was checked against a rotated register. + ## I. The five #1311 gaps, reclassified | # | gap | old | new | evidence | smallest missing operation | @@ -165,6 +196,9 @@ phase-B input is a *completed* fold. | 4 | sum of a foreign value | general gap | **MIS-SPECIFIED WITNESS**; by source a **SIMPLE MISSING OPERATOR**, not measured | witness used `u32` `Country`; the local `i32` sum exists (`Terminal::MaskedSumI32`) and no terminal or ndarray kernel sums `table[fk[i]]` (Terminal enum read in full; ndarray `masked_sum_*` list) | `masked_sum_i32_via`. Its composition route (Count by fk, then a dot product) would itself need G2 | | 5 | ordered compare on `u32` | general gap | **SIMPLE MISSING OPERATOR** | ndarray has `gt/lt/ge/le` for `i32`, `u8`, `u64` but not `u32`; `Pred` has none for `u32` | `{gt,lt,ge,le}_u32_to_mask` + `Pred` variants | +> **⊘ Corrected 2026-10-03 (§N).** Gap 2's fan-in half and gap 4's composition route both +> pointed at §F's withdrawn capability. Their classification is open. + ## J. Existing production precedent 1. **Quack `GroupPlan` (forest lowering):** a produced mask re-bound as a resident plane of @@ -219,6 +253,9 @@ fixture): *"lines whose partner has exactly v posted lines"*. The same shape is the experiment's F→I read and the frontend side's A → R1 → R2 weight read, which is why it is the one probe that tests both witnesses. +> **⊘ Withdrawn 2026-10-03 (§N).** The probe assumed a fold is followed by a separate +> projection step. D-PLX-1 is withdrawn; nothing from it is kept. + ## M. Downstream handover (domain-neutral) ```text @@ -249,3 +286,150 @@ EXECUTION FACTS SURVIVING BOTH INDEPENDENT WITNESSES 10. Open, pending one probe: whether feeding typed results onward needs only a wider lane kind, or a handle that also carries key metadata. ``` + +> **⊘ Corrected 2026-10-03 (§N).** Fact 5 is struck: grouped output is a projection, not a +> population. Fact 6's second sentence is struck: presence over a small domain is fold +> state, not a write. Facts 7 and 10 are withdrawn with §F and §L. + +## N. Physical model (2026-10-03, #1313) + +```text +resident data + + +reference register + | + FOLD + | +zero-copy projection + +PIVOT = rotate the reference register +WRITE = materialization boundary +``` + +- Resident data does not move. A fold reads it through the reference register and + copies nothing. +- A pivot rotates the register. No byte of the data plane moves; nothing is transposed. +- A new projection is another fold under another register orientation, not a new + population. +- Only a write materializes. Write only when a byte must move. + +Cost: about 1,000,000 folds cost less than one write. So: rotate the register, fold, +project again. Not: fold, write, re-import, fold again. + +**Corrections that follow.** + +1. **Fan-out.** If a resident relation or path population already carries the required + multiplicity, the register re-anchors onto it and folding continues. Nothing is + written (§C rules 1 and 3, §D; measured in #1311). +2. **Pair → spelling.** The spelling projection folds directly from the resident O rows. + No intermediate (spelling, PoS) population is written (§E note). +3. **Grouped output** is a projection of the resident data, not automatically a new + population. +4. **Equal row count** does not align two resident populations row for row. A per-row + read reaches the other population through a reference lane (an fk). +5. **No new semantic identity concepts.** + +## O. Recovered from the pre-reset branch (2026-10-03) + +The pre-reset head of #1313 is preserved as the branch `recovery/1313-pre-reset` on +`origin` (`dba70c16`; `git fetch origin recovery/1313-pre-reset`), which also holds +`55bf7cde` and `ef1e9397`. The +text below is restored verbatim from it, because it does not depend on the deleted Count +probe. Sentences that did depend on the probe are left out and marked `[…]`. + +**From `55bf7cde`, §E (review thread on #1312):** + +> The presence fold does NOT need a completed I. PoS has a bounded 16-value domain, so +> one pass over O grouped by spelling can OR a 16-bit presence bitmap per group (bit +> `pos`) and popcount it when finishing. A generic per-group distinct state is the same +> counterexample. The current IR lacks that fold state (`GroupFold` has `Count / MinI32 / +> MaxI32 / SumSymI32` only), so I→S is an **AGGREGATE-STATE LIMITATION, not a phase +> dependency.** The key-column verdict (API artifact) stands. […] + +**From `55bf7cde` / `ef1e9397`, §H:** + +> Remove "I→S presence fold" from the multi-phase row: it is a missing fold state (§E +> note). HAVING phase B stays multi-phase but is K-sized result domain finalization by +> design (§F), not the N-row consumer. […] + +**From `55bf7cde`, §M fact 6:** + +> Re-rolling a mergeable fold along its key needs no phase boundary (fold the source with +> the coarser key, or merge cells). Presence over a bounded domain needs no phase +> boundary either: it is a per-group OR state. + +**From `55bf7cde`, CellSpace cross-check** (`lance-graph-report/src/result.rs`): + +> - It preserves: per-dimension `CoordSpec` and domain; the physical layout +> (`Dense{strides}`, or `Sparse{coords, index}`); the fold states; one `i64` value +> column per state. +> - […] re-rolls by merging, in host code. +> - It does not offer its values as a lane to a mask-risc program. […] + +The experiment evidence in §B (O = 20,845, I = 20,842, S = 18,559, the three n = 2 keys, +the PoS-per-spelling histogram, `Pair` grouping, sink sizes, `PowerSums` / moments) was +never changed by any version of #1313. It is byte-identical on `main`, +`recovery/1313-pre-reset` and `recovery/1313-current`. + +## P. ANOVA real-data → ARM / SPOFC convergence (D-PLX-3, 2026-10-03) + +**Result: TWO physical population-count engines, ONE evidence semantics.** The two +engines stay separate implementations. They meet only at the evidence a rule needs: + +```text +CandidateRule evidence: + |X| rows matching the antecedent + |X ∧ Y| rows matching antecedent and consequent + n rows in the window + +ARM support gate: + support = |X∧Y| / n + +Truth projection: + arm_to_truth_u8: + f = |X∧Y| / |X| + c = |X∧Y| / (|X∧Y| + k) + + -> {s,p,o,f,c} +``` + +**Engine 1, resident / hot.** Resident population + reference geometry → Quack +`Agg::Count` → mask-risc `Terminal::Count` (`popcount_batch_u64`, +`mask-risc/src/exec.rs:1738`) → scalar → ARM / SPOFC truth. For one ARM candidate: +`Filter::X` → scalar Count → `|X|`; `Filter::And([X, Y])` → scalar Count → `|X∧Y|`. +One program per count, no intermediate population, no histogram, no K-slot sink. ARM +does not need a `Pair`-grouped K-slot result. + +**Engine 2, tabular / external.** `Dataset` → `RowMasks` → `support_count` / +`and_count` (`lance-graph-arm-discovery/src/bitset.rs:78-101`) → `|X|` / `|X∧Y|` → +ARM / SPOFC truth. It has no lance-graph engine dependency and stays usable without a +resident population. Its users (`tesseract-paperless` `auto-match`, `lance-graph-osint` +tests) rely on that. + +**The real-data witness (§B) on the same boundary.** O = 20,845 resident observations, +I = 20,842 (spelling, PoS) identities, S = 18,559 spellings. Every statistic/projection +used by the witness is obtained from O under a rotated register: +- marginals: `Count` with the register on one lane; +- population-wide pair/co-occurrence projection: `Pair{hi, lo}` grouping, with no + composite key column (this is the witness's shape, not the per-candidate ARM count); +- presence over the bounded PoS domain: 16-bit OR state per spelling (§E note, §O); +- moments: `PowerSums` (n, Σx, Σx²). + +Pair → spelling projects directly from O (§N). No intermediate analytics population is +needed between that evidence and ARM / SPOFC. + +**Not count engines.** +- Witness-register counts (`bound_count`, `agreement_count`, `quorum_mantissa`) are + fixed-register reductions. +- Planner `BeliefArena` counts are control and ranking counts. +- Confidence histograms are telemetry. +- `nsm_bridge::compare_models` is test-only. +- CausalEdge64 holds f/c and semantic bits, no raw counts. + +**Open semantic note (not solved here).** ARM / SPOFC → `arm_to_truth_u8` → CausalEdge64 +f/c is the evidence-derived path. `cognitive-shader-driver` (`driver.rs:479-489`) also +writes CausalEdge64 f/c, derived from resonance (similarity). + +Fine print: DeepNSM-v2's recounts (`bible_wave`, the confidence-delta oracle in +`introspect.rs`) are an independent scientific / oracle path that checks the execution +semantics. They are not a production count engine.