Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .claude/board/STATUS_BOARD.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ Plan: `.claude/plans/population-law-crosscheck-v1.md`. Entry: `entries/2026-10-0
| D-id | scope | status | gate / falsifier |
|---|---|---|---|
| **D-PLX-0** | cross-check of #1311 against the real-data fold experiment; reclassify the five gaps | Shipped | plan §A–§M |
| **D-PLX-1** | test-only falsifier: a K-space fold sink read back per row through an fk (lane kind vs result handle) | Queued | plan §L |
| **D-PLX-1** | test-only falsifier: a K-space fold sink read back per row through an fk (lane kind vs result handle) | Withdrawn | a fold is already the projection (plan §N); no probe kept |
| **D-PLX-3** | ANOVA real-data → ARM / SPOFC convergence: two physical population-count engines, one evidence semantics | Shipped (ANALYSIS) | plan §P |

## D-FPW — Frontend parity witness (2026-10-03)

Expand Down
13 changes: 11 additions & 2 deletions .claude/board/entries/2026-10-03-population-law-crosscheck.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,5 +23,14 @@ re-derived from `crates/deepnsm/word_frequency/academic_20k.csv` (20,845 / 20,84
- **#1311 gaps:** G1 confirmed general; G2 split (re-anchor vs phase boundary); G3 and
G5 simple missing operators; G4 mis-specified witness (by source, a missing
`masked_sum_i32_via`).
- **Next:** one test-only falsifier (a fold sink read back per row through an fk) to
decide lane kind vs typed result handle.
- **⊘ Corrected 2026-10-03 (#1313):** resident data + reference register → fold →
zero-copy projection. A pivot rotates the register; only a write materializes.
Pair → spelling folds directly from O (no phase boundary). Grouped output is a
projection, not a population. Equal row count does not align rows. The falsifier
above (D-PLX-1) is withdrawn.
- **Recovered from `55bf7cde` (`recovery/1313-pre-reset`):** I→S presence over a bounded
domain is a missing per-group OR fold state, not a phase dependency.
- **ARM / SPOFC convergence (D-PLX-3, plan §P):** two physical population-count engines
(Quack / mask-risc scalar Count; arm-discovery `Dataset` / `RowMasks`), one evidence
semantics: CandidateRule evidence `|X|`, `|X∧Y|`, n; support gate `|X∧Y|/n`;
`arm_to_truth_u8` f = `|X∧Y|/|X|`, c = `|X∧Y|/(|X∧Y|+k)` → {s,p,o,f,c}.
188 changes: 186 additions & 2 deletions .claude/plans/population-law-crosscheck-v1.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
# population-law-crosscheck-v1 — which execution law survives two independent witnesses

> **Status:** ANALYSIS + source verification. No code changed. D-PLX-0..1.
> **⊘ Revised 2026-10-03 (#1313):** two payloads: the #1312 rollover (§N, §O) and the
> ANOVA real-data → ARM / SPOFC convergence (§P, D-PLX-3). Read §N first. §D, §E, §F, §G, §H, §I, §L and §M
> carry dated ⊘ notes. D-PLX-1 is withdrawn.
> **Inputs:** merged #1311 (`2f2b67c`, `crates/lance-graph-quack/tests/gremlin_parity.rs`,
> `.claude/plans/frontend-parity-witness-v1.md`) and an independent population-fold
> experiment over a real 20,845-row source file (reported to this session, not run here;
Expand Down Expand Up @@ -75,6 +78,10 @@ rule. They only decide which of 2, 3 and 4 applies.
a produced result becomes an operand (G2). The only exception is a resident
population whose rows already are the composed paths, and that is rule 1 again.

> **⊘ Corrected 2026-10-03 (§N).** "Either way a produced result becomes an operand" is
> withdrawn. It was not checked whether a register orientation over the resident data
> reaches this case, so no write is shown to be required. The re-anchor bullets stand.

## E. I → S root cause (the 1.19 MB key column)

- The question ("how many (spelling, PoS) identities per spelling") is a **presence
Expand All @@ -101,6 +108,13 @@ rule. They only decide which of 2, 3 and 4 applies.
**Classification: CURRENT API ARTIFACT** for the key column; the phase dependency itself
is **SEMANTICALLY NECESSARY** without an ordered projection.

> **⊘ Corrected 2026-10-03 (§N).** Pair → spelling is **not** a phase boundary. Resident:
> the O rows. Register: pointed at spelling. PoS has a small fixed domain, so presence
> per spelling fits in 16 bits of fold state. One fold over O yields the spelling
> projection, and no (spelling, PoS) population is ever written. The current IR has no
> such fold state (`GroupFold` is `Count / MinI32 / MaxI32 / SumSymI32`); that gap is
> not a write. The key-column verdict and the layout note stand.

## F. Result-as-operand root cause

| result | physical form | typed form | readable without copy | operand / group key / predicate input today |
Expand All @@ -119,7 +133,7 @@ not), and a field view over record sinks.
**Where the boundary actually bites.** Quack rules HAVING deliberately: *"HAVING is not
a new primitive and not a population operation… the O(K) finalization over those sinks…
No O(N) mask crosses a program boundary"* (`quack/src/lib.rs`, `GroupHaving` doc). So
a phase B whose output stays in the K-space (HAVING, AVG's `avg_finish`, a scalar
a phase B whose output stays in the K-sized result domain (HAVING, AVG's `avg_finish`, a scalar
statistical finish over sufficient statistics) is **already handled by design, as
K-sized host finalization**. (It sits in tension with the crate's "never evaluate a
`Program`" rule, but the crate states it as a choice, not a gap.)
Expand All @@ -130,6 +144,15 @@ an edge row reading the per-vertex count from the previous hop (§D). No host
finalization can serve that without either a host loop over N rows (the duplicate
evaluator) or a population-sized copy.

> **⊘ Withdrawn 2026-10-03 (§N).** The table stays as an inventory of what each terminal
> writes. The paragraphs that name a missing capability are withdrawn: a fold over
> resident data is already the projection, so another projection is another fold under a
> rotated register, not a read of the previous output. No mechanism is proposed.
Comment thread
AdaWorldAPI marked this conversation as resolved.
> Open, not classified here: a later pass whose per-row predicate reads a completed fold
> over other rows (e.g. lines whose partner has exactly v posted lines). Neither a
> register rotation nor a write is shown for it; withdrawing D-PLX-1 withdraws the
> probe, not the question.

## G. G1 vs G2 — PARTIALLY SHARED

- **Shared:** a completed keyed result is a population over its own coordinate space
Expand All @@ -142,9 +165,14 @@ evaluator) or a population-sized copy.
as lanes and compare them.
- G2 that crosses back into an N-row pass is independently needed by a per-row read
of an I value from F (experiment) and by the A → R1 → R2 traversal (frontend side,
§D). G2 confined to the K-space (HAVING, I→S's present-cell count) is served today
§D). G2 confined to the K-sized result domain (HAVING, I→S's present-cell count) is served today
by host finalization. G1 is needed only by refolds along a key component.

> **⊘ Corrected 2026-10-03 (§N).** Struck: "a completed keyed result is a population over
> its own coordinate space". Grouped output is a projection of the resident data, not a
> new population. A refold along a key component is a fold of the same resident data
> with the register pointed at the coarser key.

## H. Multi-terminal vs multi-phase

| shape | workloads | needs |
Expand All @@ -155,6 +183,9 @@ evaluator) or a population-sized copy.
Adding terminals to one `Program` removes none of the multi-phase dependencies: each
phase-B input is a *completed* fold.

> **⊘ Corrected 2026-10-03 (§N).** I→S leaves the multi-phase row (§E note). No remaining
> row is shown to require a write; none was checked against a rotated register.

## I. The five #1311 gaps, reclassified

| # | gap | old | new | evidence | smallest missing operation |
Expand All @@ -165,6 +196,9 @@ phase-B input is a *completed* fold.
| 4 | sum of a foreign value | general gap | **MIS-SPECIFIED WITNESS**; by source a **SIMPLE MISSING OPERATOR**, not measured | witness used `u32` `Country`; the local `i32` sum exists (`Terminal::MaskedSumI32`) and no terminal or ndarray kernel sums `table[fk[i]]` (Terminal enum read in full; ndarray `masked_sum_*` list) | `masked_sum_i32_via`. Its composition route (Count by fk, then a dot product) would itself need G2 |
| 5 | ordered compare on `u32` | general gap | **SIMPLE MISSING OPERATOR** | ndarray has `gt/lt/ge/le` for `i32`, `u8`, `u64` but not `u32`; `Pred` has none for `u32` | `{gt,lt,ge,le}_u32_to_mask` + `Pred` variants |

> **⊘ Corrected 2026-10-03 (§N).** Gap 2's fan-in half and gap 4's composition route both
> pointed at §F's withdrawn capability. Their classification is open.

## J. Existing production precedent

1. **Quack `GroupPlan` (forest lowering):** a produced mask re-bound as a resident plane of
Expand Down Expand Up @@ -219,6 +253,9 @@ fixture): *"lines whose partner has exactly v posted lines"*.
The same shape is the experiment's F→I read and the frontend side's A → R1 → R2 weight
read, which is why it is the one probe that tests both witnesses.

> **⊘ Withdrawn 2026-10-03 (§N).** The probe assumed a fold is followed by a separate
> projection step. D-PLX-1 is withdrawn; nothing from it is kept.

## M. Downstream handover (domain-neutral)

```text
Expand Down Expand Up @@ -249,3 +286,150 @@ EXECUTION FACTS SURVIVING BOTH INDEPENDENT WITNESSES
10. Open, pending one probe: whether feeding typed results onward needs only
a wider lane kind, or a handle that also carries key metadata.
```

> **⊘ Corrected 2026-10-03 (§N).** Fact 5 is struck: grouped output is a projection, not a
> population. Fact 6's second sentence is struck: presence over a small domain is fold
> state, not a write. Facts 7 and 10 are withdrawn with §F and §L.

## N. Physical model (2026-10-03, #1313)

```text
resident data
+
reference register
|
FOLD
|
zero-copy projection

PIVOT = rotate the reference register
WRITE = materialization boundary
```

- Resident data does not move. A fold reads it through the reference register and
copies nothing.
- A pivot rotates the register. No byte of the data plane moves; nothing is transposed.
- A new projection is another fold under another register orientation, not a new
population.
- Only a write materializes. Write only when a byte must move.

Cost: about 1,000,000 folds cost less than one write. So: rotate the register, fold,
project again. Not: fold, write, re-import, fold again.

**Corrections that follow.**

1. **Fan-out.** If a resident relation or path population already carries the required
multiplicity, the register re-anchors onto it and folding continues. Nothing is
written (§C rules 1 and 3, §D; measured in #1311).
2. **Pair → spelling.** The spelling projection folds directly from the resident O rows.
No intermediate (spelling, PoS) population is written (§E note).
3. **Grouped output** is a projection of the resident data, not automatically a new
population.
4. **Equal row count** does not align two resident populations row for row. A per-row
read reaches the other population through a reference lane (an fk).
5. **No new semantic identity concepts.**

## O. Recovered from the pre-reset branch (2026-10-03)

The pre-reset head of #1313 is preserved as the branch `recovery/1313-pre-reset` on
`origin` (`dba70c16`; `git fetch origin recovery/1313-pre-reset`), which also holds
`55bf7cde` and `ef1e9397`. The
text below is restored verbatim from it, because it does not depend on the deleted Count
probe. Sentences that did depend on the probe are left out and marked `[…]`.

**From `55bf7cde`, §E (review thread on #1312):**

> The presence fold does NOT need a completed I. PoS has a bounded 16-value domain, so
> one pass over O grouped by spelling can OR a 16-bit presence bitmap per group (bit
> `pos`) and popcount it when finishing. A generic per-group distinct state is the same
> counterexample. The current IR lacks that fold state (`GroupFold` has `Count / MinI32 /
> MaxI32 / SumSymI32` only), so I→S is an **AGGREGATE-STATE LIMITATION, not a phase
> dependency.** The key-column verdict (API artifact) stands. […]

**From `55bf7cde` / `ef1e9397`, §H:**

> Remove "I→S presence fold" from the multi-phase row: it is a missing fold state (§E
> note). HAVING phase B stays multi-phase but is K-sized result domain finalization by
> design (§F), not the N-row consumer. […]

**From `55bf7cde`, §M fact 6:**

> Re-rolling a mergeable fold along its key needs no phase boundary (fold the source with
> the coarser key, or merge cells). Presence over a bounded domain needs no phase
> boundary either: it is a per-group OR state.

**From `55bf7cde`, CellSpace cross-check** (`lance-graph-report/src/result.rs`):

> - It preserves: per-dimension `CoordSpec` and domain; the physical layout
> (`Dense{strides}`, or `Sparse{coords, index}`); the fold states; one `i64` value
> column per state.
> - […] re-rolls by merging, in host code.
> - It does not offer its values as a lane to a mask-risc program. […]

The experiment evidence in §B (O = 20,845, I = 20,842, S = 18,559, the three n = 2 keys,
the PoS-per-spelling histogram, `Pair` grouping, sink sizes, `PowerSums` / moments) was
never changed by any version of #1313. It is byte-identical on `main`,
`recovery/1313-pre-reset` and `recovery/1313-current`.

## P. ANOVA real-data → ARM / SPOFC convergence (D-PLX-3, 2026-10-03)

**Result: TWO physical population-count engines, ONE evidence semantics.** The two
engines stay separate implementations. They meet only at the evidence a rule needs:

```text
CandidateRule evidence:
|X| rows matching the antecedent
|X ∧ Y| rows matching antecedent and consequent
n rows in the window

ARM support gate:
support = |X∧Y| / n

Truth projection:
arm_to_truth_u8:
f = |X∧Y| / |X|
c = |X∧Y| / (|X∧Y| + k)

-> {s,p,o,f,c}
```

**Engine 1, resident / hot.** Resident population + reference geometry → Quack
`Agg::Count` → mask-risc `Terminal::Count` (`popcount_batch_u64`,
`mask-risc/src/exec.rs:1738`) → scalar → ARM / SPOFC truth. For one ARM candidate:
`Filter::X` → scalar Count → `|X|`; `Filter::And([X, Y])` → scalar Count → `|X∧Y|`.
One program per count, no intermediate population, no histogram, no K-slot sink. ARM
does not need a `Pair`-grouped K-slot result.

**Engine 2, tabular / external.** `Dataset` → `RowMasks` → `support_count` /
`and_count` (`lance-graph-arm-discovery/src/bitset.rs:78-101`) → `|X|` / `|X∧Y|` →
ARM / SPOFC truth. It has no lance-graph engine dependency and stays usable without a
Comment thread
AdaWorldAPI marked this conversation as resolved.
resident population. Its users (`tesseract-paperless` `auto-match`, `lance-graph-osint`
tests) rely on that.

**The real-data witness (§B) on the same boundary.** O = 20,845 resident observations,
I = 20,842 (spelling, PoS) identities, S = 18,559 spellings. Every statistic/projection
used by the witness is obtained from O under a rotated register:
- marginals: `Count` with the register on one lane;
- population-wide pair/co-occurrence projection: `Pair{hi, lo}` grouping, with no
composite key column (this is the witness's shape, not the per-candidate ARM count);
- presence over the bounded PoS domain: 16-bit OR state per spelling (§E note, §O);
- moments: `PowerSums` (n, Σx, Σx²).

Pair → spelling projects directly from O (§N). No intermediate analytics population is
needed between that evidence and ARM / SPOFC.

**Not count engines.**
- Witness-register counts (`bound_count`, `agreement_count`, `quorum_mantissa`) are
fixed-register reductions.
- Planner `BeliefArena` counts are control and ranking counts.
- Confidence histograms are telemetry.
- `nsm_bridge::compare_models` is test-only.
- CausalEdge64 holds f/c and semantic bits, no raw counts.

**Open semantic note (not solved here).** ARM / SPOFC → `arm_to_truth_u8` → CausalEdge64
f/c is the evidence-derived path. `cognitive-shader-driver` (`driver.rs:479-489`) also
writes CausalEdge64 f/c, derived from resonance (similarity).

<sub>Fine print: DeepNSM-v2's recounts (`bible_wave`, the confidence-delta oracle in
`introspect.rs`) are an independent scientific / oracle path that checks the execution
semantics. They are not a production count engine.</sub>
Loading