diff --git a/.claude/board/AGENT_LOG.md b/.claude/board/AGENT_LOG.md index 7f3465523..56dd3f01a 100644 --- a/.claude/board/AGENT_LOG.md +++ b/.claude/board/AGENT_LOG.md @@ -1,3 +1,26 @@ +## 2026-09-30 — 5+3 council: cypher-mask multiplicity contract (D-CMM-0..3) + +- Plan: `.claude/plans/cypher-mask-multiplicity-contract-v1.md` (v1 → v2 → ratified v3). +- The 5: prior-art-savant, iron-rule-savant, runtime-archaeologist, cascade-impact-savant, creative-explorer-savant. + Key findings: + - `Frontier` name collision (`nars/tactics.rs:151`). + - Rule 5 needs girth / directed-only. + - The DAG fixture cannot tell walks from trails. + - A fourth consumer kind (grouped) was missing. + - The census already over-counts Full. +- The 3: overclaim-auditor, dilution-collapse-sentinel, firewall-warden. + - BLOCKs: + - `Grouped` conflated two carriers; + - `SetOnly` over-claimed for earlier variables (`count(DISTINCT b)` 3 vs 4); + - unguarded `Serialize`; + - gate gaps. + - All resolved in v3: five carrier kinds, `!Serialize` guard test, strict G1b. +- Measured: + - G1a 4/3, 4/3; G1b DataFusion = 5 (walks), as pre-registered. + - G2: 4 disables red-then-green. + - Census Full 117 → 70. +- Also: the Phase-0 commit's model-naming trailer was removed (own branch, pre-PR). + ## 2026-09-29 — D-LXC-1 plan rewritten with Read (orchestrator, no agents, no code) - Operator-directed. The council run below read source with shell diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index a104d2a66..9e0e790a4 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,17 @@ +## 2026-09-30 — cypher-mask-multiplicity-contract-v1 — a mask is the support of a frontier, never its bag → `.claude/plans/cypher-mask-multiplicity-contract-v1.md` + +**Status:** RATIFIED v3 (5+3 council) + PR A shipped (classifier + tests + +census consumer). Corrects `cypher-mask-lowering-v1` (below): `count(*)`, +`sum`, `avg`, `RETURN ` after a hop are bag-sensitive, and a forward hop +chain is the exact support of the TERMINAL variable only. Five carrier kinds +(`ConsumerSemantics`); only `TerminalSet` lowers in v1. DataFusion counts +walks, not trails (measured 5 vs 4 on a cycle) — recorded OPEN. + +**Correction to the 2026-09-14 entry below:** its tally "35 [G] · 9 [H] · +9 [GRACE]" was already superseded by the council regrade to **31 [G] · 13 [H] · +9 [GRACE]** (stated in the PR2 entry further down). Both are grades over a +single population; eleven rows now carry a post-hop multiplicity qualifier. + ## 2026-09-29 (2) — deepnsm-v2-lexical-evidence-consumer-v1 rewritten against the DeepNSM → DeepNSM-v2 migration → `.claude/plans/deepnsm-v2-lexical-evidence-consumer-v1.md` **Status:** PROPOSAL (D-LXC-1..10). No code authorized. Supersedes entry (1) diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 80bd4a365..9c839a948 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -1,3 +1,18 @@ +## D-CMM — Cypher-mask multiplicity contract (2026-09-30) + +Plan: `.claude/plans/cypher-mask-multiplicity-contract-v1.md`. Entry: `entries/2026-09-30-cypher-mask-is-support-not-bag.md`. + +| D-id | scope | status | gate / falsifier | +|---|---|---|---| +| **D-CMM-0** | correct `cypher-mask-lowering-v1` rows T-1..T-7, T-11, R-6..R-8, §4.1, §4.6, §7 oracles, tally qualifier | In PR | contract §5 G6 | +| **D-CMM-1** | `LogicalOperator::consumer_semantics` + `ConsumerSemantics` (5 kinds, not `Serialize`) | In PR | `logical_plan.rs` unit tests; 4 disables red-then-green | +| **D-CMM-2** | DataFusion bag pins: DAG 4/3, 4/3; cycle walk count 5 | In PR | `tests/test_datafusion_varlength_complex.rs` | +| **D-CMM-3** | W0-b census consumes the classifier | In PR | Full 117 → 70 on 310 | +| **D-CMM-4** | count-lane (plus-times) hop for `TerminalCount`/`EarlierCount` | Queued | gated on a consumer + contract §3.2 R4 exactness | +| **D-CMM-5** | DataFusion walk-vs-trail divergence on cycles | Reclassified: DataFusion, Ladybug and SQL joins all compute WALK, which v1 adopts; TRAIL is a separate mode (contract §7.1-§7.2) | grace ruling: not fixed | +| **D-CMM-6** | footnote `lance-graph-as-the-modelgraph-v1.md` §15 Full fraction | Queued | — | +| **D-CMM-7** | carrier-sufficiency table + exhaustive enumerator (`.claude/tools/carrier_sufficiency.py`) | In PR | contract §7.3; every "no" prints a witness | + ## D-LXC — DeepNSM-v2 lexical-evidence consumer + candidate next parts (2026-09-29) Plan: `.claude/plans/deepnsm-v2-lexical-evidence-consumer-v1.md`. Convergence brief: `.claude/prompts/deepnsm-v2-lexical-consumer-converge.md`. diff --git a/.claude/board/entries/2026-09-30-cypher-mask-is-support-not-bag.md b/.claude/board/entries/2026-09-30-cypher-mask-is-support-not-bag.md new file mode 100644 index 000000000..21d0e44de --- /dev/null +++ b/.claude/board/entries/2026-09-30-cypher-mask-is-support-not-bag.md @@ -0,0 +1,55 @@ +# A Boolean mask is the support of a frontier, never its bag (2026-09-30) + +**Status:** MEASURED + TEST-PINNED. Plan: `.claude/plans/cypher-mask-multiplicity-contract-v1.md` (ratified v3). + +**Finding.** `cypher-mask-lowering-v1` lowered `count(*)` to the popcount of the final mask. After a hop, Cypher's bag semantics count one row per binding (path), not one per node. + +| query on KNOWS = {1→2, 1→3, 2→3, 3→4, 4→5} | DataFusion | mask chain | +|---|---|---| +| `(a)->(b)->(c) RETURN count(*)` | 4 | 3 | +| `count(DISTINCT c.id)` | 3 | 3 | +| `*1..2` from id 1, `count(*)` | 4 | 3 | +| `count(DISTINCT b)` over the 2-hop | 3 | forward `dst₁` = 4 | + +The last row is the second defect: a forward hop chain is the exact support of the TERMINAL variable only. + +**Divergence recorded OPEN.** On KNOWS = {1→2, 2→1, 2→2}, DataFusion's 2-hop `count(*)` is **5**, the walk count. Cypher's relationship uniqueness gives 4, the trail count. DataFusion is in grace, so this is not fixed. + +**Code.** `LogicalOperator::consumer_semantics()` (`logical_plan.rs`) returns one of five carrier kinds: `TerminalSet`, `EarlierSet`, `TerminalCount`, `EarlierCount`, `Bindings`. It is not `Serialize`, and a guard test enforces that. + +The G2 disables each went red-then-green: +- force sensitivity to false; +- force the focus to the terminal; +- drop the cross-variable filter; +- let `Join` through. + +**Census (W0-b), debug 0.** 310 classified. + +| | before | after | +|---|---|---| +| Full | 117 | 70 | +| Split | 193 | 240 | + +New reasons, by query count: + +| reason | queries | +|---|---| +| Bindings | 90 | +| TerminalCount | 35 | +| EarlierCount | 9 | +| EarlierSet | 7 | + +The value-DISTINCT T-12 reason moved 7 → 6. + +After the chain-linearity fix (Codex P2 on #1305), the census reads 313 queries. The three new classifier-test queries join the corpus as `Bindings`. Full is unchanged at 70, so no existing query was reclassified. + +**OPEN.** +- `lance-graph-as-the-modelgraph-v1.md` §15 still quotes the pre-contract Full fraction (37.3 %). It needs a footnote. +- No count-lane operator exists. `Weighted` lowering is gated on the exactness conditions in contract §3.2 R4. + +**Amendment (same day, before merge).** Contract §7. +- v1 semantics is WALK: DataFusion, Ladybug (`PathSemantic::WALK` default, no `r1 <> r2` in `rewriteMatchPattern`) and SQL joins all compute it. D-CMM-5 is reclassified from "DataFusion divergence" to "TRAIL is a separate mode". +- Walks and trails also diverge on an acyclic graph when the pattern changes direction: `(a)->(b)<-(c)` on {1→2} gives `count(DISTINCT c)` 1 as a walk, 0 as a trail. +- MEASURED carrier sufficiency (`python3 .claude/tools/carrier_sufficiency.py`): node support answers only Exists/Support under WALK; per-node counts add Count/CountBy of the current and later nodes; the last hop's edge population adds the previous node; nothing per-node or per-edge answers a TRAIL question two hops on (two parallel self-loops plus 1→0: 3-hop trail count 0 vs 2 with equal per-edge trail counts). +- The mask-RISC survival rule stays; §7.4 states the condition under which a later PR may relax it. + diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index 393a6e819..3f9659af6 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,10 +25,11 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -176 entries, 2026-08-06 .. 2026-09-29. +177 entries, 2026-08-06 .. 2026-09-30. | date | entry id | finding | file | |---|---|---|---| +| 2026-09-30 | `cypher-mask-is-support-not-bag` | | [2026-09-30-cypher-mask-is-support-not-bag.md](2026-09-30-cypher-mask-is-support-not-bag.md) | | 2026-09-29 | `deepnsm-v2-counted-pick-tag-deltas` | | [2026-09-29-deepnsm-v2-counted-pick-tag-deltas.md](2026-09-29-deepnsm-v2-counted-pick-tag-deltas.md) | | 2026-09-26 | `deepnsm-v2-lexical-evidence-survives-routing` | | [2026-09-26-deepnsm-v2-lexical-evidence-survives-routing.md](2026-09-26-deepnsm-v2-lexical-evidence-survives-routing.md) | | 2026-09-25 | `window-scheduling-and-two-level-ternlog` | | [2026-09-25-window-scheduling-and-two-level-ternlog.md](2026-09-25-window-scheduling-and-two-level-ternlog.md) | diff --git a/.claude/plans/cypher-mask-lowering-v1.md b/.claude/plans/cypher-mask-lowering-v1.md index 946c2976a..01fe8c07d 100644 --- a/.claude/plans/cypher-mask-lowering-v1.md +++ b/.claude/plans/cypher-mask-lowering-v1.md @@ -336,9 +336,9 @@ only because the destination index is **decoded** from the selected row, making | **R-3** | `(a)-[:R]-(b)` undirected | R-1 ∪ R-2: two hops, `mask_or`. **Not** one hop over a "both" flag | `[G]` shape · `[H]` in-repo (inherits R-1/R-2) | | **R-4** | `-[:R1\|R2]->` multi-type | one hop per type, `mask_or`-accumulated into `dst`. Mirrors `lgj_hop`'s own per-facet `⋁_f` | `[G]` shape · `[H]` in-repo (inherits R-1/R-2) | | **R-5** | `-[r {k: v}]->` relationship property filter | an extra leaf in the per-facet conjunction: `ternlog::(class_f, src, prop_f)` and then AND `struct_f` — the SAME two-predicate pattern `lgj_hop` already runs (`class_f` at facet base +0, `struct_f` at base +12) with one leaf substituted | `[G]` | -| **R-6** | `(a)-[:R]->(b)-[:S]->(c)` — fixed 2-hop | chain: `dst₁` becomes `src₂`. Target-label masks AND in **between** hops, never after, so hop 2's frontier is already narrowed | `[G]` shape · `[H]` in-repo (inherits R-1/R-2) | -| **R-7** | `-[:R*1..k]->` bounded variable length | iterate R-1 `k` times over a **DELTA frontier**, not the accumulated state: `frontier ← mask_andnot(dst, state)`, `state ← mask_or(state, dst)`, stop when `!mask_any(frontier)`. `blackboard.md:33-36` measures exactly this — *"the **NNUE reading** — spread from the DELTA frontier (`scratch & !state`), never from the accumulated state — gives the identical closure (gate green) at **8.8 µs (−48 %)**"* | `[G]` mechanism · `[H]` on graph shape | -| **R-8** | `-[:R*]->` unbounded | R-7 to fixpoint. Termination is `mask_any(frontier) == false` (`mask_any:1215`) — a **bit test over a 8 KiB plane**, not a visited-set lookup. Today's DF path unrolls and unions (`builder/expand_ops.rs`, `logical_plan.rs:72-91`: *"implemented by unrolling into multiple fixed-length paths and unioning them"*); a mask fixpoint has no unroll bound at all | `[G]` mechanism · `[H]` | +| **R-6** | `(a)-[:R]->(b)-[:S]->(c)` — fixed 2-hop | chain: `dst₁` becomes `src₂`. Target-label masks AND in **between** hops, never after, so hop 2's frontier is already narrowed | `[G]` shape · `[H]` in-repo (inherits R-1/R-2) **⊘ multiplicity (2026-09-30, `cypher-mask-multiplicity-contract-v1.md`):** exact SUPPORT of the terminal variable; carries no path count and no earlier-variable support | +| **R-7** | `-[:R*1..k]->` bounded variable length | iterate R-1 `k` times over a **DELTA frontier**, not the accumulated state: `frontier ← mask_andnot(dst, state)`, `state ← mask_or(state, dst)`, stop when `!mask_any(frontier)`. `blackboard.md:33-36` measures exactly this — *"the **NNUE reading** — spread from the DELTA frontier (`scratch & !state`), never from the accumulated state — gives the identical closure (gate green) at **8.8 µs (−48 %)**"* | `[G]` mechanism · `[H]` on graph shape **⊘ multiplicity (2026-09-30, `cypher-mask-multiplicity-contract-v1.md`):** mechanism stands for terminal support / reachability; carries no multiplicity. DataFusion counts WALKS here (measured 5 vs Cypher's 4 on a cycle) | +| **R-8** | `-[:R*]->` unbounded | R-7 to fixpoint. Termination is `mask_any(frontier) == false` (`mask_any:1215`) — a **bit test over a 8 KiB plane**, not a visited-set lookup. Today's DF path unrolls and unions (`builder/expand_ops.rs`, `logical_plan.rs:72-91`: *"implemented by unrolling into multiple fixed-length paths and unioning them"*); a mask fixpoint has no unroll bound at all | `[G]` mechanism · `[H]` **⊘ multiplicity (2026-09-30, `cypher-mask-multiplicity-contract-v1.md`):** reachability only; a path-count consumer is `[GRACE]` | | **R-9** | `WHERE` applied to a hop result | the survivor mask is just another leaf: `mask_ternlog::(dst, pred_a, pred_b)`. **This is the crosswalk's own shape** — `spog-alpha-channel-v1.md:182-184`: *"`mask_ternlog::(&sweep, &tenant_mask[n], &rung_gate, &mut survivors)` — the survivors' key set is the needle set of hop n+1"* | `[G]` shape · `[H]` in-repo (inherits R-1/R-2) | **The forbidden move, quoted because it is one line away from every one of these @@ -356,17 +356,17 @@ are aggregates). `Terminal` is `mask-risc/src/ir.rs:108-128`. | # | Cypher | lowers to | status | |---|---|---|---| -| **T-1** | `RETURN count(*)` | `popcount_batch_u64` (`ndarray/src/bitwise.rs:274`) over the final mask. `Terminal::Count` (`ir.rs:110`) | `[G]` | -| **T-2** | `RETURN count(n)` | identical to T-1. **Because there is no NULL**: `mask-risc/src/lib.rs:44-47` — *"absence in the V3 substrate is a zero-fallback, never a validity bit, so DuckDB's three-valued AND/OR collapses to Boolean algebra"*. `count(n)` and `count(*)` cannot differ | `[G]` | -| **T-3** | `RETURN n` | **the mask itself** — `Terminal::Keep` (`ir.rs:127`). Not a row list. The caller reads the plane | `[G]` | -| **T-4** | `RETURN n.prop` | **a masked projection that stays `(mask, lane_ref)`** — the pair, never an index list. `Planes::lanes` (`ir.rs:50`) + `LaneRef` (`ir.rs:16-23`) is the carrier. Materialisation exists but is **named**: exactly one symbol whose name starts with `materialize`, O(n) stated in its doc (`lance-graph-java/CLAUDE.md`'s Materialisation exception) | `[G]` shape · the named materializer is **new code**, §7 W1 | -| **T-5** | `RETURN sum(n.p)` on an `i32` lane | `masked_sum_i32:692` — widened to `i64`, *"carry-safe for every `i32` input"* (`mask-risc/src/lib.rs:49-50`) | `[G]` | -| **T-6** | `RETURN min(n.p)` / `max(n.p)` on `i32` | `masked_min_i32:1502` / `masked_max_i32:1523` — `Option`, `None` on an empty mask | `[G]` | -| **T-7** | `RETURN avg(n.p)` on `i32` | `masked_sum_i32` + `popcount_batch_u64`, divide at the terminal. Two reductions, one pass each, no intermediate | `[H]` — the result type is float; whether the caller wants exact rational or `f64` is **OQ-7** | +| **T-1** | `RETURN count(*)` | `popcount_batch_u64` (`ndarray/src/bitwise.rs:274`) over the final mask. `Terminal::Count` (`ir.rs:110`) | `[G]` **⊘ multiplicity (2026-09-30, `cypher-mask-multiplicity-contract-v1.md`):** exact over a single population · after a hop `[GRACE]` in v1 (bag semantics count one row per binding; a mask counts distinct nodes) | +| **T-2** | `RETURN count(n)` | identical to T-1. **Because there is no NULL**: `mask-risc/src/lib.rs:44-47` — *"absence in the V3 substrate is a zero-fallback, never a validity bit, so DuckDB's three-valued AND/OR collapses to Boolean algebra"*. `count(n)` and `count(*)` cannot differ | `[G]` **⊘ multiplicity (2026-09-30, `cypher-mask-multiplicity-contract-v1.md`):** the no-NULL argument stands; the hop defect of T-1 is inherited · after a hop `[GRACE]` in v1 (bag semantics count one row per binding; a mask counts distinct nodes) | +| **T-3** | `RETURN n` | **the mask itself** — `Terminal::Keep` (`ir.rs:127`). Not a row list. The caller reads the plane | `[G]` **⊘ multiplicity (2026-09-30, `cypher-mask-multiplicity-contract-v1.md`):** single population only; after a hop a node repeats once per binding · after a hop `[GRACE]` in v1 (bag semantics count one row per binding; a mask counts distinct nodes). `RETURN DISTINCT n` stays T-11 | +| **T-4** | `RETURN n.prop` | **a masked projection that stays `(mask, lane_ref)`** — the pair, never an index list. `Planes::lanes` (`ir.rs:50`) + `LaneRef` (`ir.rs:16-23`) is the carrier. Materialisation exists but is **named**: exactly one symbol whose name starts with `materialize`, O(n) stated in its doc (`lance-graph-java/CLAUDE.md`'s Materialisation exception) | `[G]` shape · the named materializer is **new code**, §7 W1 **⊘ multiplicity (2026-09-30, `cypher-mask-multiplicity-contract-v1.md`):** single population only · after a hop `[GRACE]` in v1 (bag semantics count one row per binding; a mask counts distinct nodes). `RETURN DISTINCT n.p` is T-12 | +| **T-5** | `RETURN sum(n.p)` on an `i32` lane | `masked_sum_i32:692` — widened to `i64`, *"carry-safe for every `i32` input"* (`mask-risc/src/lib.rs:49-50`) | `[G]` **⊘ multiplicity (2026-09-30, `cypher-mask-multiplicity-contract-v1.md`):** single population only; after a hop the sum is path-weighted · after a hop `[GRACE]` in v1 (bag semantics count one row per binding; a mask counts distinct nodes) | +| **T-6** | `RETURN min(n.p)` / `max(n.p)` on `i32` | `masked_min_i32:1502` / `masked_max_i32:1523` — `Option`, `None` on an empty mask | `[G]` **⊘ multiplicity (2026-09-30, `cypher-mask-multiplicity-contract-v1.md`):** after a hop exact for the TERMINAL variable only; an earlier variable needs a backward (semi-join) pass | +| **T-7** | `RETURN avg(n.p)` on `i32` | `masked_sum_i32` + `popcount_batch_u64`, divide at the terminal. Two reductions, one pass each, no intermediate | `[H]` — the result type is float; whether the caller wants exact rational or `f64` is **OQ-7** **⊘ multiplicity (2026-09-30, `cypher-mask-multiplicity-contract-v1.md`):** single population only; after a hop both numerator and denominator are path-weighted · after a hop `[GRACE]` in v1 (bag semantics count one row per binding; a mask counts distinct nodes) | | **T-8** | `RETURN sum(…)` over a **grouped** lane inside the 12-byte register | `masked_strided_group_sum:779` (`group_bytes` 1..=4, asserted at `:783`) | `[G]` | | **T-9** | `EXISTS { MATCH … }` / any existence test | `mask_any:1215` — one early-exiting word scan, no count | `[G]` | | **T-10** | `CASE WHEN THEN a ELSE b` | `blend_i32:1582` — **no compaction**; `Terminal::BlendI32` (`ir.rs:124`) writes into a caller buffer | `[G]` | -| **T-11** | `RETURN DISTINCT n` (a node variable) | **free — the identity.** A mask IS a set: a row is in it or it is not, and there is no multiplicity to collapse. `DISTINCT` over a node variable lowers to nothing at all | `[G]` | +| **T-11** | `RETURN DISTINCT n` (a node variable) | **free — the identity.** A mask IS a set: a row is in it or it is not, and there is no multiplicity to collapse. `DISTINCT` over a node variable lowers to nothing at all | `[G]` **⊘ multiplicity (2026-09-30, `cypher-mask-multiplicity-contract-v1.md`):** free for the TERMINAL variable only. `count(DISTINCT b)` over `(a)->(b)->(c)` is 3 on the contract's §0 fixture while the forward `dst₁` has 4 — an earlier variable's support needs a backward pass | | **T-12** | `RETURN DISTINCT n.p` / `count(DISTINCT n.p)` / `collect(…)` | **[GRACE]** — §4.1. Distinct over VALUES needs value identity, which a population mask does not carry | `[GRACE]` | ### §3.6 — Constructs that do not lower (7 rows, pointer only — reasons in §4) @@ -386,6 +386,12 @@ each row's own cell: **31 `[G]`**, **13 `[H]`** (⊘ PR2 council: R-3/R-4/R-6/R- **9 `[GRACE]`** (the 7 rows of §3.6 plus P-9 and T-12, which reach the same fence from inside the predicate and terminal groups). +**⊘ Multiplicity qualifier (2026-09-30, `cypher-mask-multiplicity-contract-v1.md`):** the grade counts above are +unchanged — each is the grade over a single population. Eleven rows now carry a +post-hop qualifier: T-1, T-2, T-3, T-4, T-5, T-7 are `[GRACE]` after a hop in v1; +T-6 and T-11 hold for the terminal variable only; R-6, R-7, R-8 carry support, +not multiplicity. + --- ## §4 — (b) WHAT CANNOT LOWER, AND WHY @@ -410,9 +416,15 @@ implicitly. `LIMIT` **with** `ORDER BY` is `[GRACE]`, full stop: the k that surv depends on the order, so the walk would return a different answer than DataFusion and §7's differential would — correctly — go red. -`count(DISTINCT x)` is the sharp case: `count(*)` is a popcount, and the two differ -by exactly the multiplicity a mask threw away. Do not let the first tempt anyone -into the second. +`count(DISTINCT x)` is the sharp case: `count(*)` over a single population is a +popcount, and the two differ by exactly the multiplicity a mask threw away. Do not +let the first tempt anyone into the second. + +**⊘ Narrowed (2026-09-30, `cypher-mask-multiplicity-contract-v1.md`):** after a hop, `count(*)` is **not** a +popcount — it counts bindings (paths). On KNOWS = {1→2, 1→3, 2→3, 3→4, 4→5}, +`(a)->(b)->(c) RETURN count(*)` is 4 while the terminal mask holds 3 nodes. The +consumer classes are `ConsumerSemantics` (`logical_plan.rs`); only `TerminalSet` +lowers in v1. ### §4.2 — Strings and variable-width values @@ -486,12 +498,13 @@ the binding which store each side indexes, never by whether the operator is spel | the question the construct asks | mask can answer | example | |---|---|---| | which rows? | **yes** | `MATCH`, `WHERE`, hops | -| how many rows? | **yes** (popcount) | `count(*)` | +| how many distinct nodes of the TERMINAL variable? (⊘ was "how many rows?", narrowed 2026-09-30) | **yes** (popcount) | `count(*)` over a single population; `count(DISTINCT c)` | +| how many BINDINGS (paths)? | **no in v1** — needs a count lane (contract §3.2) | `count(*)`, `sum`, `RETURN c` after a hop | | is there any row? | **yes** (`mask_any`) | `EXISTS` | | what is the total / min / max of a lane over those rows? | **yes** (masked reduction) | `sum`, `min`, `max` | | in what ORDER? | no | `ORDER BY` | | which POSITION in that order? | no | `SKIP`/`LIMIT` after `ORDER BY` | -| how many TIMES (multiplicity)? | no | `collect`, `count(DISTINCT)` | +| how many TIMES (value, binding or path multiplicity)? | no | `collect`, `count(DISTINCT )`, `count(*)` after a hop — distinct counting over the TERMINAL node variable (`count(DISTINCT c)`) is supported, row above | | what VALUE, as a new relation? | no | `UNWIND`, `WITH`-aggregation | | how CLOSE / how MUCH? | no — that is a different mechanism | vector distance, NARS truth | | across which ADDRESS SPACES? | no | cross-store `Join` | @@ -758,6 +771,11 @@ Five properties, each of which a weaker harness would drop: consistent with a bijection and proves none, so the pass condition stays the set equality against the scalar reference … the count is the cheap early filter in front of it, never a substitute."* Count first (20 ns), set second (the proof). + **⊘ Multiplicity arm (2026-09-30):** set equality cannot see bag multiplicity — + the contract's §0 fixture passes it while `count(*)` returns 3 instead of 4. The + harness therefore also records DataFusion's `count(*)` against the mask popcount + on that fixture; a mismatch routes the query to GRACE. It is a record, never a + pass condition on the mask path. 2. **The DataFusion side is the REFERENCE, and it is the one that already works.** This is the only window in which that is true — after the grace period ends there is no second implementation to diff against. Wave 0 is therefore not merely first; @@ -827,7 +845,7 @@ Scope: §3.4 R-7, R-8. | id | assertion | its DISABLE | |---|---|---| | **F-F1** | the delta-frontier closure equals the accumulated-state closure, exactly | — (this is `blackboard.md:33-36`'s own gate, re-run here) | -| **F-F2** | `*1..k` equals the DataFusion unroll-and-union for every `k` in the fixture | — | +| **F-F2** | `*1..k` equals the DataFusion unroll-and-union for every `k` in the fixture — **as a SET** (⊘ 2026-09-30: DataFusion's unroll counts walks and repeats endpoints per path; its row count is a multiplicity record, not the pass condition) | — | | **F-F3** (termination) | an unbounded `*` over a **cyclic** fixture terminates, and the step count equals the graph's eccentricity from the seed | remove the `mask_any(frontier)` test; it must hang or over-count | | **F-F4** (can-stay-silent) | `*1..k` on a fixture with no R edges returns the seed set unchanged for `min=0`, empty for `min=1` | — | @@ -838,7 +856,7 @@ Scope: Phase 2.5 in `query.rs:920-952`, the `Split` outcome, and the router. | id | assertion | its DISABLE | |---|---|---| | **F-S1** | with the lowering DISABLED, every existing test is byte-identical to today | — (the additive-seam gate; this is the one that must be run before any merge) | -| **F-S2** | a `Split` query returns the same rows as the pure-DataFusion path | — | +| **F-S2** | a `Split` query returns the same rows as the pure-DataFusion path — the same BAG, not only the same set (⊘ 2026-09-30: only a `TerminalSet` consumer may take the mask path; the contract's §0 fixture is in the fixture set) | — | | **F-S3** (the grace list is real) | every §4 construct in the fixture set is classified `[GRACE]` and takes the DataFusion path — asserted by the classifier's own output, not by the result being right | force the classifier to accept `ORDER BY`; the differential must go red | | **F-S4** (N-9) | the same lowering, reached from a Gremlin or SPARQL query that produces the same `LogicalOperator`, produces the same mask | — | @@ -958,7 +976,7 @@ PRs generates none of these obligations. 1. **The lowering table (§3): 53 rows** — 7 node/label, 9 property predicate, 9 Boolean-fusion, 9 relationship/hop, 12 return/terminal, 7 explicit non-lowering. - **31 `[G]`, 13 `[H]`** (regraded by the PR2 council — compositions inherit their components' grade; nine name a measurement in §8), **9 `[GRACE]`**. + **31 `[G]`, 13 `[H]`** (regraded by the PR2 council — compositions inherit their components' grade; nine name a measurement in §8), **9 `[GRACE]`**. ⊘ 2026-09-30: grades are over a single population; eleven rows carry a post-hop multiplicity qualifier (`cypher-mask-multiplicity-contract-v1.md`). 2. **The placement ruling (§5): three parts.** Consume the already-minted, so-far unconsumed `TERNLOG = 0x86` (`ogar-loco/src/lib.rs:607`) for all Boolean combination — do not mint. Keep `Pred` / hop / terminals as a **lowering target diff --git a/.claude/plans/cypher-mask-multiplicity-contract-v1.md b/.claude/plans/cypher-mask-multiplicity-contract-v1.md new file mode 100644 index 000000000..3aeec47aa --- /dev/null +++ b/.claude/plans/cypher-mask-multiplicity-contract-v1.md @@ -0,0 +1,252 @@ +# cypher-mask-multiplicity-contract-v1 — a mask is the SUPPORT of a frontier, never its multiplicity + +> **Status:** RATIFIED v3 (5+3 council, Phase 5), amended by §7 (2026-09-30: walk semantics, carrier sufficiency). Corrects `cypher-mask-lowering-v1.md`. +> No mint. No DataFusion extension. Debug-0 builds only. +> **D-ids:** D-CMM-0 (plan corrections), D-CMM-1 (classifier), D-CMM-2 (DataFusion pins), +> D-CMM-3 (census consumer), D-CMM-4 (count lane, queued), D-CMM-5 (walk/trail divergence, open), +> D-CMM-6 (modelgraph footnote, queued), D-CMM-7 (§7 carrier-sufficiency table + enumerator) — rows in `STATUS_BOARD.md` under `D-CMM`. + +## §0 The defect, in one fixture + +KNOWS = {1→2, 1→3, 2→3, 3→4, 4→5} — the edges of `create_knows_dataset` in `crates/lance-graph/tests/test_explain_output.rs`. + +`MATCH (a:Person)-[:KNOWS]->(b:Person)-[:KNOWS]->(c:Person) RETURN count(*)` + +- Bag semantics: paths 1→2→3, 1→3→4, 2→3→4, 3→4→5 ⇒ **4**. DataFusion returns 4 (G1a, measured). +- `cypher-mask-lowering-v1` R-6 chain + T-1 popcount: hop₁ dst = {2,3,4,5}, hop₂ dst = {3,4,5} ⇒ **3**. +- `RETURN count(DISTINCT c)` ⇒ **3**. After a hop, the plan's lowering answers the DISTINCT-endpoint question and calls it `count(*)`. Over a single population, T-1 is exact. +- `RETURN count(DISTINCT b)` ⇒ **3** ({2,3,4}), but R-6's forward `dst₁` = {2,3,4,5} ⇒ 4. A forward mask is the exact support of the TERMINAL variable only. + +On this acyclic fixture, walk count = trail count. The two diverge only on cycles (G1b). + +> ⊘ **Struck by §7:** walks and trails diverge whenever two positions of one pattern can match the same edge. A cycle is one way; a direction change is another, and needs no cycle: on the single edge {1→2}, `(a)-[:KNOWS]->(b)<-[:KNOWS]-(c) RETURN count(DISTINCT c)` is 1 as a walk and 0 as a trail. + +`-[:KNOWS*1..2]->` from `a.id = 1`: DataFusion returns 4 rows, targets Bob, Charlie, Charlie, David +(`test_datafusion_pipeline.rs:2226-2258`); distinct endpoints = 3. + +The same defect reaches `RETURN c` and `RETURN c.name` after a hop (T-3/T-4): Cypher returns one +row per binding, so Charlie appears twice above; a mask holds Charlie once. + +The plan already graces `collect` and ORDER as multiplicity (`INTEGRATION_PLANS.md:212`, `w0b_corpus_census.rs:230`). It does not recognise that `count`/`sum`/`avg`/`RETURN` after a hop are multiplicity-dependent. No entry found in `.claude/plans`, `.claude/board`, or `.claude/knowledge` states that (search: multiplicity | bag semantics | walk.*trail). + +## §1 Frozen decisions + +- F1. DataFusion is in grace: maintained, not extended (`CLAUDE.md` § ⊘ OPERATOR RULING 2026-09-05). +- F2. Mask programs are a lowering target; nothing is minted (`cypher-mask-lowering-v1.md` §5.2). +- F3. One-population, tail and transpose laws hold (`cypher-mask-lowering-v1.md` §5.3). +- F4. Absence is a zero fallback, not a NULL (`mask-risc/src/lib.rs:44-47`). +- F5. Builds use `CARGO_PROFILE_DEV_DEBUG=0 CARGO_PROFILE_TEST_DEBUG=0 CARGO_INCREMENTAL=0`. + +## §2 Input inventory (measured by the savants, file:line read) + +| site | claim | status under §0 | +|---|---|---| +| plan T-1/T-2 (`:359-360`) | `count(*)`/`count(n)` = popcount | exact with no hop; wrong after a hop | +| plan T-3/T-4 (`:361-362`) | `RETURN n` / `n.prop` = the mask / mask+lane | exact with no hop; after a hop a node repeats once per binding | +| plan T-5/T-7 (`:363,365`) | `sum` / `avg` = masked sum / popcount | wrong after a hop (weighted by path count) | +| plan T-6 (`:364`) | `min`/`max` | exact after a hop for the terminal variable only | +| plan T-11/T-12 (`:369-370`) | `DISTINCT n` free; `count(DISTINCT …)` grace | T-11 right; T-12 must split: DISTINCT over a node variable = support ([G]), over a value = [GRACE] | +| plan R-6 (`:339`) | 2-hop = `dst₁` becomes `src₂` | exact support of the TERMINAL variable only | +| plan R-7/R-8 (`:340-341`) | var-length = delta-frontier fixpoint | mechanism `[G]` for terminal support/reachability; carries no multiplicity | +| plan §4.1 (`:413-414`), §4.6 (`:489`), §11 (`:961, 969-971`) | "`count(*)` is a popcount"; "how many rows? yes" | hold for one population only | +| plan §7 F-F2 (`:830`), F-S2 (`:842`), W0-c (`:756-762`) | SET equality is the oracle | cannot see multiplicity; the §0 fixture passes it while returning 3 | +| `INTEGRATION_PLANS.md:207-209` | "35 [G] · 9 [H] · 9 [GRACE]" | already stale against the plan's own 31/13/9 | +| `examples/w0b_corpus_census.rs:209-245, 323-426` | a hop + `count`/`sum`/`avg`/`RETURN c` is lowerable; `count(DISTINCT n)` is grace | wrong in both directions; the Full fraction is biased with unknown sign | +| `logical_plan.rs:94-97, 549-560, 570-575` | aggregates live in `Project.projections` as `AggregateFunction{distinct}`; `RETURN DISTINCT` wraps `Distinct` | classifier needs no AST | +| `datafusion_planner/builder/expand_ops.rs:147-204, 281` | var-length unrolls and UNIONs; fresh rel alias per hop; no edge-inequality predicate in `:140-290` (read; no whole-crate negative claimed) | DataFusion counts WALKS: G1b measures 5 against Cypher's 4 | + +## §3 Resolution + +### §3.1 PR 0 — correct the plan (doc only; narrow, never delete) + +A mask answers *which distinct nodes the TERMINAL variable can take*. An earlier variable's support needs a backward (semi-join) pass over the transpose, which R-6 does not contain. + +- **"single population"** replaces "no hop" everywhere: no `Expand`, `VariableLengthExpand`, `Join`, or `Unwind`. +- T-1/T-2/T-3/T-4/T-5/T-7 each split: + - a single-population row keeps its grade; + - a post-hop row is `[GRACE]` in v1. That is a routing decision, not a verdict: the row becomes `[H]` under §3.2 rule 5. +- The post-hop GRACE applies to the non-DISTINCT forms only: + - `RETURN DISTINCT c` over the terminal variable stays T-11 `[G]`; + - T-9 `EXISTS` is unaffected. +- T-6 post-hop stays `[G]` for the terminal variable only. +- T-12 splits three ways: + - DISTINCT over the terminal node variable is `[G]`; + - over a non-terminal variable, `[GRACE]` in v1; + - over a value, `[GRACE]`. +- R-6/R-7/R-8: the mechanism stays `[G]` for support and reachability of the terminal variable. The rows gain "carries no multiplicity". +- §4.6 is narrowed, not deleted: + - "how many rows?" becomes "how many distinct nodes of the terminal variable? — yes"; + - a new row reads "how many bindings? — no in v1". +- §4.1's sentence is narrowed to "`count(*)` over a single population is a popcount; after a hop it is not". +- §7 F-F2, F-S2 and W0-c each gain a multiplicity arm that **records** the divergence on the §0 fixture and routes the query to GRACE. It is never a pass condition on the mask path. +- §11's row count and tally are recomputed row by row. `INTEGRATION_PLANS.md` is append-only, so its stale "35/9/9" gets a dated correction line, not an edit. + +### §3.2 PR A — the contract, and its one real consumer + +The vocabulary is the house one: *factorized* (Kuzu; `graphrag-industry-comparison.md:41`, `LATEST_STATE.md:3557`) and the plus-times semiring (GraphBLAS). `Frontier` is taken (`planner/src/nars/tactics.rs:151`), so the carrier is named `BindingFrontier`. + +``` +BindingFrontier { support: Mask, // Boolean semiring + mult: Option, // plus-times, u64, checked + binding: Option } // parent identity, factorized; native-side only +``` + +`ConsumerSemantics` names the **carrier** a query's consumer needs. It does not say the query lowers: + +| kind | carrier | examples (after a hop over a→b→c) | +|---|---|---| +| `TerminalSet` | support of the terminal variable | `RETURN DISTINCT c`, `count(DISTINCT c)`, `min(c.age)`; every single-population query | +| `EarlierSet` | support of ONE earlier variable (backward semi-join) | `count(DISTINCT b)`, `min(a.age)` | +| `TerminalCount` | support + mult keyed on the terminal | `count(*)`, `sum(c.age)`, `RETURN c`, `RETURN c, count(*)` | +| `EarlierCount` | support + mult keyed on ONE earlier variable (backward count) | `RETURN a, count(*)`, `RETURN a`, `sum(a.age)` | +| `Bindings` | identity of ≥2 variables | `RETURN a, c`, `WHERE a.age = c.age`, nested `Project` (`WITH`), `Join`, `Unwind` | + +Classification, over the top `Project` (through `Sort`/`Offset`/`Limit`/`Distinct` wrappers): + +1. `Join`, `Unwind`, a nested `Project`, a non-`Project` top, or any unknown shape ⇒ `Bindings`. The classifier fails closed. +2. No hop (single population) ⇒ `TerminalSet`: one row is one node. +3. With a hop: + - a `Filter` whose predicate references ≥2 variables ⇒ `Bindings`; + - otherwise let V = the variables the projections reference (`*` excluded). |V| ≥ 2 ⇒ `Bindings`; + - V = ∅ ⇒ the focus is the terminal; otherwise the focus is the single member of V; + - *multiplicity-sensitive* ⇔ some non-DISTINCT `count`/`sum`/`avg`/`collect`, or no aggregate and no `Distinct` wrapper; + - focus × sensitivity picks one of the four remaining kinds. +4. Orthogonal to the carrier, and judged separately by the plan §4 rows: `ORDER BY`, `SKIP`/`LIMIT`, value-`DISTINCT`, `collect` as a sequence, value-keyed grouping. + +Rules for a future lowering (none is built here): + +- **R1.** In v1 only `TerminalSet` may lower fully. For a non-`TerminalSet` consumer over a mask-lowerable prefix, the outcome is `Split` or `Grace`. The kind is recorded so `Split` stays reachable. +- **R2.** `binding` reuses CSR slices (`AdjacencyBatch`, `planner/src/adjacency/batch.rs:37-57`, `!Clone`). It never becomes an owned per-binding vector, and it never crosses to Java. +- **R3.** Unfold, when coded, is `materialize_bindings`, O(output) in its doc. `mask-risc/src/exec.rs:329` "the ONE materialiser" then gets a scope note. +- **R4.** "Exact" means mult equals the Cypher trail count: + - ⊘ **Superseded by §7.2:** exactness is relative to an explicit path semantics. v1 is WALK, which is what DataFusion, Ladybug and SQL joins compute. TRAIL is a separate mode with its own carrier requirement. The sub-bullets below state conditions under which WALK = TRAIL; they remain true as such, but "fixed k-hop over one directed type" must also exclude direction changes (§0 strike note). + - fixed k-hop over one directed type is exact iff no closed walk of length ≤ k−1 exists; + - any undirected hop is `[GRACE]`; + - var-length is exact only on a proven-acyclic relation, or within girth; + - mult comes from edge multiplicity, never from a deduplicated adjacency; + - overflow refuses, never wraps. + - Matching DataFusion's count is a separate, OPEN property (§4). +- **R5.** A `WITH` that aggregates or is `DISTINCT` resets multiplicity to 1; a plain `WITH` carries it. v1 classifies any nested `Project` as `Bindings`. + +PR A ships: + +- **The enum and the method.** `enum ConsumerSemantics` derives `Debug, Clone, Copy, PartialEq, Eq`, with no serde. A guard test proves `!Serialize` via inherent-vs-trait method resolution, and has a can-fire arm on `LogicalOperator`, which is `Serialize` (`logical_plan.rs:18`). The method is `LogicalOperator::consumer_semantics(&self)`, pure. The crate is edition 2021: no let-chains. +- **Its consumer.** `w0b_corpus_census.rs`'s `classify_plan` calls it. A non-`TerminalSet` query with a hop gets a grace reason naming its kind. The census's T-12 arm splits: `DISTINCT` over a node variable is not grace; over a value it is. +- **The differential tests (G1)** in the existing `tests/test_datafusion_varlength_complex.rs`. This crate keeps top-level test files and has no `tests/integration/`, so a new binary is not warranted. +- **No `BindingFrontier` type, no operator, no `Cargo.toml` change.** + +## §4 Non-goals + +- No DataFusion extension node, no change to `expand_batch`, no trail enforcement in DataFusion. + DataFusion counts WALKS: measured 5 on G1b, where Cypher's trail count is 4. The divergence is **recorded OPEN** (F1). +- No counting op in mask-risc's IR. No `BindingFrontier` type in code; only the `ConsumerSemantics` enum ships. +- No OPTIONAL MATCH. No change to the dense-rowid address model. + +## §5 Pre-registered gates + +- **G1a (DAG, measured green):** + - 2-hop `count(*)` = 4; `count(DISTINCT c.id)` = 3; + - `*1..2` from id 1: `count(*)` = 4; `count(DISTINCT b.id)` = 3. +- **G1b (cycle, divergence pin, measured green):** KNOWS = {1→2, 2→1, 2→2}. 2-hop `count(*)` asserts `== 5`, the walk count. The Cypher trail count is 4. It is a record of DataFusion's behaviour, not a correctness claim. +- **G2 (classifier, unit tests in `logical_plan.rs`):** at least one case per kind, including: + - `count(DISTINCT b)` ⇒ `EarlierSet` (the §2 defect); + - `RETURN c` ⇒ `TerminalCount` vs `RETURN DISTINCT c` ⇒ `TerminalSet`; + - `RETURN a, count(*)` ⇒ `EarlierCount`; `RETURN c, count(*)` ⇒ `TerminalCount`; + - `WHERE a.age = c.age` ⇒ `Bindings`; + - `WITH` ⇒ `Bindings`. +- **G2 disables (red-then-green each):** + - force the multiplicity-sensitivity to `false` ⇒ the Count arms fail; + - force the focus to the terminal ⇒ the Earlier arms fail; + - drop the cross-variable Filter check ⇒ the `Bindings` arm fails. +- **G3 (can stay silent):** single-population `count(*)` ⇒ `TerminalSet`. Paired arm: two disconnected scans (`Join`) with `count(*)` ⇒ `Bindings`. +- **G4 (census moves):** debug 0, before and after. Pre-registered: + - the new multiplicity/earlier reason fires on > 0 queries; + - the T-12 node-variable rescue is reported as its own count. + - Recorded in the board entry. +- **G5:** debug 0 for `cargo test -p lance-graph --lib logical_plan`, the G1 tests, and `cargo clippy -p lance-graph --all-targets -- -D warnings`; `cargo fmt --check`. `Cargo.toml` unchanged. No model identifier in the diff or in commit messages. +- **G6:** the PR 0 edits cite §0. The three tallies agree. Board writes use `Edit` (never an open-for-write that reads the same file), are checked with `wc -l`, and `supersession_index.py` is regenerated LAST. + +## §6 Change ledger v1 → v2 + +- `Frontier` → `BindingFrontier` (name collision). #163 ListArray citation dropped (unverified). +- Three consumer kinds → four (`Grouped` added; `CountBindings` → `Weighted`, and it now covers `sum`/`avg`/`RETURN `). +- Rule 5 generalised: girth, directed-only, edge multiplicity. +- Cyclic fixture G1b added (the DAG cannot tell walks from trails). +- G2 gets paired disables. The classifier becomes a method, and is wired into the census as its consumer (answers AP6 dead surface). +- PR 0 scope widened: T-3..T-7, T-12, §4.1, §11, the wave oracles, and the `INTEGRATION_PLANS` tally. +- DataFusion's walk semantics recorded as an OPEN divergence. +- v2 → v3 (reviewers): + - The enum is re-cut by carrier into five kinds. `Grouped` is split because the terminal and earlier variables need different carriers. + - `SetOnly` is restricted to the terminal variable; the overclaim-auditor's `count(DISTINCT b)` = 3 vs 4 finding is now in §0. + - "No hop" becomes "single population" (a `Join` has no hop). + - `WITH` reset is limited to aggregating or DISTINCT `WITH`. + - `Split` stays reachable (R1). + - G1b is a strict `== 5` divergence pin; G2 has three binding disables. + - A `!Serialize` guard is added; `INTEGRATION_PLANS` gets a correction line, not an edit. + - The model identifier was removed from the Phase-0 commit trailer. +- Retained from `cypher-mask-lowering-v1` unchanged: + - the Boolean core (§3.1-§3.3); + - R-1 to R-5 and R-9; + - T-8, T-9 and T-10; + - the §5 placement ruling and §5.3 laws. + +## §7 Amendment (2026-09-30): walk semantics and carrier sufficiency + +Added after the council, from counterexamples, before merge. Nothing in §1-§6 is deleted; the struck lines carry pointers here. + +### §7.1 Which semantics the executable references compute + +- **DataFusion:** no edge-inequality predicate between hops (`datafusion_planner/builder/expand_ops.rs`, read in §2); measured 5 on G1b. +- **Ladybug** (AdaWorldAPI fork, C++): the recursive-pattern default is `PathSemantic::WALK` (`src/include/main/client_config.h`, `RECURSIVE_PATTERN_SEMANTIC`), and `rewriteMatchPattern` in `src/binder/bind_match.cpp` adds no `r1 <> r2` for fixed-length patterns. +- **SQL joins** (DuckDB, the Quack oracle) join edge rows independently, so the same edge row can appear at two positions. + +All three compute WALK. D-CMM-5 is therefore not a DataFusion defect: DataFusion agrees with v1. The trail count 4 in G1b stays as the TRAIL-mode record. + +### §7.2 The semantics boundary + +- `step(…, Walk)` is exact with the Markov carriers below. +- `step(…, Trail)` is exact only when no two positions of the pattern can match the same edge (disjoint relationship types, or same-direction hops over a relation with no closed walk shorter than the pattern). Otherwise it needs the carrier named in §7.3, or answers `Insufficient`. + +### §7.3 Carrier sufficiency (MEASURED) + +Command: `python3 .claude/tools/carrier_sufficiency.py` (stdlib, about 2 s). Every directed multigraph on 3 nodes with up to 4 edges (parallel edges and self-loops included), every FROM population. A carrier is sufficient for an observation iff no two FROM populations on the same graph give the same carrier and a different observation; every "no" prints its witness. + +Carriers after hop k, each computed from the matches up to hop k only: S = node support of the last node; K = per-node match count; E = edge population of hop k; W = per-edge walk count; B = bindings. + +| after hop k (k ≥ 2), later question | S | K | E | W | B | +|---|---|---|---|---|---| +| Walk: Exists | yes | yes | yes | yes | yes | +| Walk: Support of node k and of any later node | yes | yes | yes | yes | yes | +| Walk: Count, CountBy of node k or any later node | no | yes | no | yes | yes | +| Walk: Support of node k−1 | no | no | yes | yes | yes | +| Walk: CountBy of node k−1 | no | no | no | yes | yes | +| Walk: anything about node k−2 or earlier | no | no | no | no | yes | +| Trail at hop k (carriers built from walks) | no | no | no | no | yes | +| Trail at hop k+1 | no | no | no | no | yes | + +After hop 1 the edge population also identifies the origin, so E1 answers every question about 1- and 2-hop matches, walk or trail. + +Smallest witnesses (edges; FROM populations; the two answers): + +- S cannot count: {0→0, 1→0}; FROM {0} vs {0,1}; walk `count(*)` 1 vs 2. +- K cannot name the previous node: {0→0, 1→0, 2→1}; FROM {0} vs {2}; 2-hop Support(n1) {0} vs {1}. +- S cannot decide a trail: {0→0, 1→0}; FROM {0} vs {1}; 2-hop trail Exists false vs true (the self-loop is reused). +- Per-edge trail counts cannot extend a trail: {0→0, 0→0, 1→0} (two parallel self-loops); FROM {0} vs {1}; 3-hop trail count 0 vs 2. +- The opposite-ends case: {0→0, 1→0}; FROM {0} vs {1}; the trail Support of c in `(a)->(b)<-(c)` is {1} vs {0} while K is equal. + +Rules that follow: + +1. Under WALK, folding forward to per-node counts loses nothing about the current or any later node. It loses the previous node, which the edge population of the last hop still has, and everything further back, which only the retained input population (by a backward pass) or the bindings have. +2. Under TRAIL, each hop must know which edges the match has already used. Per-edge trail counts suffice for one more hop; beyond that no per-node or per-edge carrier suffices, and the answer is `Insufficient` unless the pattern cannot reuse an edge. +3. A node-support mask is sufficient only for Exists and Support under WALK. + +### §7.4 The mask-RISC survival rule + +`Terminal::ScatterOrU32`'s survival condition forbids feeding a scattered mask into another program, and `Filter::Semijoin` accepts only a resident plane. That prohibition implements rule 3 conservatively and stays. A future PR may relax it only when all of these hold: + +- the semantics is WALK; +- every remaining observation is Exists or the Support of the scattered node or a later node; +- the scattered mask covers the next step's whole node space: same space and epoch, and no out-of-range key was dropped. + +Anything else needs a count or edge carrier, or the retained input population. + diff --git a/.claude/tools/carrier_sufficiency.py b/.claude/tools/carrier_sufficiency.py new file mode 100644 index 000000000..a02ada27d --- /dev/null +++ b/.claude/tools/carrier_sufficiency.py @@ -0,0 +1,142 @@ +#!/usr/bin/env python3 +"""Carrier sufficiency, measured by exhaustive enumeration. + +Question: after k hops of a forward chain, which CARRIER (the state handed to +the next step) still determines a later OBSERVATION, given that the relation +(edge table) stays resident? + +Method: every directed multigraph on 3 nodes with up to 4 edges (parallel +edges and self-loops included; edge id = position), every FROM population. +A carrier C is SUFFICIENT for an observation O iff no two FROM populations on +the same graph give the same C but a different O. Every "n" is backed by a +printed witness `(edges, FROM_a, FROM_b, O_a, O_b)`. + +Walk = edges may repeat within a match (DataFusion, Ladybug default, SQL +joins). Trail = no edge twice within a match (openCypher relationship +uniqueness). + +Carriers after hop k (Markov: computed from the matches up to hop k only): + S node support of the last node (a Boolean node mask) + K per-node match count of the last node (plus-times counts) + E edge population of hop k (a Boolean edge mask) + W per-edge walk count of hop k + TW per-edge trail count of hop k + B all matches so far (bindings) -- the reference, always sufficient + +Usage: python3 .claude/tools/carrier_sufficiency.py (stdlib only, ~1 min) +Referenced by .claude/plans/cypher-mask-multiplicity-contract-v1.md §3.3. +""" +import collections +import itertools + +N = 3 +NODES = range(N) +PAIRS = [(u, v) for u in NODES for v in NODES] +MAX_EDGES = 4 + + +def graphs(): + for m in range(MAX_EDGES + 1): + for combo in itertools.combinations_with_replacement(PAIRS, m): + yield list(combo) + + +def populations(): + for k in range(N + 1): + for s in itertools.combinations(NODES, k): + yield frozenset(s) + + +def matches(g, start, pattern): + """All matches as (edge-id tuple, node tuple). 'f' follows src->dst, + 'b' follows dst->src (a direction change).""" + out = [] + + def rec(nodes, edges): + if len(edges) == len(pattern): + out.append((tuple(edges), tuple(nodes))) + return + here, d = nodes[-1], pattern[len(edges)] + for i, (u, v) in enumerate(g): + if d == "f" and u == here: + rec(nodes + [v], edges + [i]) + elif d == "b" and v == here: + rec(nodes + [u], edges + [i]) + + for a in start: + rec([a], []) + return out + + +def trails(ms): + return [m for m in ms if len(set(m[0])) == len(m[0])] + + +def observations(ms, hops): + o = {"Exists": bool(ms), "Count": len(ms)} + for i in range(hops + 1): + name = "n%d" % i + o["Support(%s)" % name] = frozenset(m[1][i] for m in ms) + o["CountBy(%s)" % name] = tuple(sorted(collections.Counter(m[1][i] for m in ms).items())) + return o + + +def carriers(g, start, k): + walks = matches(g, start, "f" * k) + tr = trails(walks) + last = [m[0][-1] for m in walks] + return { + "S": frozenset(m[1][-1] for m in walks), + "K": tuple(sum(1 for m in walks if m[1][-1] == x) for x in NODES), + "E": frozenset(last), + "W": tuple(last.count(i) for i in range(len(g))), + "TW": tuple(sum(1 for m in tr if m[0][-1] == i) for i in range(len(g))), + "B": tuple(sorted(walks)), + } + + +def study(k, patterns): + ok, witness = {}, {} + for g in graphs(): + rows = [] + for start in populations(): + obs = {} + for pname, pat in patterns.items(): + ms = matches(g, start, pat) + for sem, sel in (("walk", ms), ("trail", trails(ms))): + for oname, val in observations(sel, len(pat)).items(): + obs["%s|%s|%s" % (pname, sem, oname)] = val + rows.append((carriers(g, start, k), obs, sorted(start))) + for cname in rows[0][0]: + groups = collections.defaultdict(list) + for c, obs, start in rows: + groups[c[cname]].append((obs, start)) + for grp in groups.values(): + o0, s0 = grp[0] + for key in o0: + ok.setdefault((cname, key), True) + for o1, s1 in grp[1:]: + if o1[key] != o0[key] and ok[(cname, key)]: + ok[(cname, key)] = False + witness[(cname, key)] = (g, s0, s1, o0[key], o1[key]) + return ok, witness + + +def report(title, k, patterns): + ok, witness = study(k, patterns) + cols = ["S", "K", "E", "W", "TW", "B"] + print("== %s (carrier after hop %d) ==" % (title, k)) + print("%-30s %s" % ("observation", " ".join("%-2s" % c for c in cols))) + for key in sorted({key for (_, key) in ok}): + print("%-30s %s" % (key, " ".join("%-2s" % ("Y" if ok[(c, key)] else "n") for c in cols))) + print("-- witnesses (edges, FROM_a, FROM_b, obs_a, obs_b) --") + for (c, key), w in sorted(witness.items()): + print("%-3s %-30s %s" % (c, key, w)) + print() + + +if __name__ == "__main__": + report("one hop, then questions about 1 and 2 hops", 1, + {"1hop": "f", "2hop": "ff", "vee": "fb"}) + report("two hops, then questions about 2 and 3 hops", 2, + {"2hop": "ff", "3hop": "fff"}) diff --git a/crates/lance-graph/examples/w0b_corpus_census.rs b/crates/lance-graph/examples/w0b_corpus_census.rs index f28d8ac4c..844b3ca53 100644 --- a/crates/lance-graph/examples/w0b_corpus_census.rs +++ b/crates/lance-graph/examples/w0b_corpus_census.rs @@ -39,7 +39,7 @@ use std::io; use std::path::{Path, PathBuf}; use lance_graph::ast::{BooleanExpression, PropertyValue, ValueExpression}; -use lance_graph::logical_plan::{LogicalOperator, LogicalPlanner}; +use lance_graph::logical_plan::{ConsumerSemantics, LogicalOperator, LogicalPlanner}; use lance_graph::parser::parse_cypher_query; use lance_graph::GraphConfig; @@ -220,9 +220,13 @@ fn classify_value(v: &ValueExpression, grace: &mut Vec) { args, distinct, } => { - // T-12: DISTINCT over VALUES needs value identity a mask has not. - if *distinct { - grace.push("§4.1 T-12 count(DISTINCT …)"); + // T-12 splits (multiplicity contract §3.1): DISTINCT over a NODE + // variable is support — whether it is the terminal variable's + // support is `consumer_semantics`'s call, in `main`. DISTINCT over + // a VALUE needs value identity a mask has not. + let over_node = matches!(args.as_slice(), [ValueExpression::Variable(_)]); + if *distinct && !over_node { + grace.push("§4.1 T-12 count(DISTINCT )"); } match name.to_lowercase().as_str() { // T-1/T-2 count · T-5 sum · T-6 min/max · T-7 avg (`[H]`, OQ-7). @@ -494,6 +498,24 @@ fn main() -> io::Result<()> { let mut grace = Vec::new(); let mut lowered = 0usize; classify_plan(&plan, &mut grace, &mut lowered); + // The multiplicity contract (cypher-mask-multiplicity-contract-v1 + // §3.2): a mask is the support of the TERMINAL variable. Any other + // consumer needs a carrier the lowering plan's v1 does not build. + match plan.consumer_semantics() { + ConsumerSemantics::TerminalSet => {} + ConsumerSemantics::EarlierSet => { + grace.push("contract EarlierSet — earlier-variable support (backward pass)") + } + ConsumerSemantics::TerminalCount => { + grace.push("contract TerminalCount — path count after a hop") + } + ConsumerSemantics::EarlierCount => { + grace.push("contract EarlierCount — path count keyed on an earlier variable") + } + ConsumerSemantics::Bindings => { + grace.push("contract Bindings — identity of two or more variables") + } + } for r in &grace { *reason_hist.entry(r).or_insert(0) += 1; } diff --git a/crates/lance-graph/src/logical_plan.rs b/crates/lance-graph/src/logical_plan.rs index 9d6c32f8a..e0aa5058f 100644 --- a/crates/lance-graph/src/logical_plan.rs +++ b/crates/lance-graph/src/logical_plan.rs @@ -625,6 +625,275 @@ impl<'a> LogicalPlanner<'a> { } } +/// Which carrier a query's consumer needs from the frontier the pattern +/// produces — the multiplicity contract of +/// `.claude/plans/cypher-mask-multiplicity-contract-v1.md` §3.2. +/// +/// A Boolean mask is the SUPPORT of a frontier: which distinct nodes a +/// variable can take. After a hop, Cypher's bag semantics count one row per +/// BINDING (path), so `count(*)` over `(a)->(b)->(c)` is the path count, not +/// the popcount of `c`'s mask, and a forward hop chain gives the exact support +/// of the TERMINAL variable only — an earlier variable needs a backward pass. +/// +/// This says what the consumer NEEDS. It does not say the query lowers: +/// ordering, `SKIP`/`LIMIT`, value-`DISTINCT` and strings are judged +/// separately (lowering plan §4). +/// +/// Deliberately not `Serialize`: it is consumed once, in process. A byte of it +/// surviving the query would trip the lowering plan's §5.2 mint condition. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ConsumerSemantics { + /// The support of the terminal variable suffices. Every single-population + /// query is this: one row is one node. + TerminalSet, + /// The support of ONE earlier variable — Boolean, but it needs a backward + /// (semi-join) pass; the forward chain over-approximates it. + EarlierSet, + /// Support plus a per-node path count keyed on the terminal variable. + TerminalCount, + /// Support plus a path count keyed on ONE earlier variable. + EarlierCount, + /// The identity of two or more variables: rows must be enumerated. + Bindings, +} + +impl LogicalOperator { + /// Classify the carrier this plan's consumer needs. Pure; fails closed to + /// [`ConsumerSemantics::Bindings`] on any shape it does not recognise. + pub fn consumer_semantics(&self) -> ConsumerSemantics { + // Peel the wrappers that sit above the RETURN projection. + let mut op = self; + let mut distinct = false; + loop { + match op { + LogicalOperator::Sort { input, .. } + | LogicalOperator::Offset { input, .. } + | LogicalOperator::Limit { input, .. } => op = input, + LogicalOperator::Distinct { input } => { + distinct = true; + op = input; + } + _ => break, + } + } + let (input, projections) = match op { + LogicalOperator::Project { input, projections } => (input, projections), + _ => return ConsumerSemantics::Bindings, + }; + + let pattern = match PatternShape::of(input) { + Some(p) => p, + None => return ConsumerSemantics::Bindings, + }; + if pattern.hops == 0 { + return ConsumerSemantics::TerminalSet; + } + if pattern.cross_variable_filter { + return ConsumerSemantics::Bindings; + } + + let mut vars: Vec<&str> = Vec::new(); + let mut has_aggregate = false; + let mut sensitive_aggregate = false; + for p in projections { + collect_value_vars(&p.expression, &mut vars); + aggregate_flags(&p.expression, &mut has_aggregate, &mut sensitive_aggregate); + } + vars.sort_unstable(); + vars.dedup(); + + let focus = match vars.as_slice() { + [] => pattern.terminal.as_str(), + [one] => one, + _ => return ConsumerSemantics::Bindings, + }; + if pattern.relationship_vars.iter().any(|r| r == focus) { + return ConsumerSemantics::Bindings; + } + let terminal = focus == pattern.terminal; + let sensitive = sensitive_aggregate || (!has_aggregate && !distinct); + match (terminal, sensitive) { + (true, false) => ConsumerSemantics::TerminalSet, + (false, false) => ConsumerSemantics::EarlierSet, + (true, true) => ConsumerSemantics::TerminalCount, + (false, true) => ConsumerSemantics::EarlierCount, + } + } +} + +/// The pattern under a RETURN projection, as the classifier needs it. +struct PatternShape { + hops: usize, + terminal: String, + relationship_vars: Vec, + cross_variable_filter: bool, +} + +impl PatternShape { + /// `None` for anything that is not one linear pattern: a `Join`, an + /// `Unwind`, a nested projection (`WITH`), an unknown operator, hops that + /// do not chain (each hop's source must be the next inner hop's target, + /// or the scanned variable), or a variable bound twice. The last two are + /// what the planner emits for `MATCH (a)->(b), (a)->(c)` and + /// `(a)->(b)->(a)`: both need binding identity a forward mask chain lacks. + fn of(op: &LogicalOperator) -> Option { + let mut shape = PatternShape { + hops: 0, + terminal: String::new(), + relationship_vars: Vec::new(), + cross_variable_filter: false, + }; + // The source the next inner hop (or the scan) must produce. + let mut expect: Option<&str> = None; + let mut bound: Vec<&str> = Vec::new(); + let mut op = op; + loop { + match op { + LogicalOperator::Filter { input, predicate } => { + let mut vars = Vec::new(); + collect_bool_vars(predicate, &mut vars); + vars.sort_unstable(); + vars.dedup(); + if vars.len() >= 2 { + shape.cross_variable_filter = true; + } + op = input; + } + LogicalOperator::Expand { + input, + source_variable, + target_variable, + relationship_variable, + .. + } + | LogicalOperator::VariableLengthExpand { + input, + source_variable, + target_variable, + relationship_variable, + .. + } => { + if expect.is_some_and(|e| e != target_variable) + || bound.contains(&target_variable.as_str()) + { + return None; + } + bound.push(target_variable); + expect = Some(source_variable); + // The outermost hop is the last one: its target is terminal. + if shape.hops == 0 { + shape.terminal = target_variable.clone(); + } + shape.hops += 1; + if let Some(r) = relationship_variable { + shape.relationship_vars.push(r.clone()); + } + op = input; + } + LogicalOperator::ScanByLabel { variable, .. } => { + if expect.is_some_and(|e| e != variable) || bound.contains(&variable.as_str()) { + return None; + } + if shape.hops == 0 { + shape.terminal = variable.clone(); + } + return Some(shape); + } + _ => return None, + } + } + } +} + +fn collect_value_vars<'a>(v: &'a ValueExpression, out: &mut Vec<&'a str>) { + match v { + ValueExpression::Variable(name) => { + if name != "*" { + out.push(name); + } + } + ValueExpression::Property(p) => out.push(&p.variable), + ValueExpression::Literal(_) + | ValueExpression::Parameter(_) + | ValueExpression::VectorLiteral(_) => {} + ValueExpression::ScalarFunction { args, .. } + | ValueExpression::AggregateFunction { args, .. } => { + for a in args { + collect_value_vars(a, out); + } + } + ValueExpression::Arithmetic { left, right, .. } + | ValueExpression::VectorDistance { left, right, .. } + | ValueExpression::VectorSimilarity { left, right, .. } => { + collect_value_vars(left, out); + collect_value_vars(right, out); + } + } +} + +fn collect_bool_vars<'a>(e: &'a BooleanExpression, out: &mut Vec<&'a str>) { + match e { + BooleanExpression::Comparison { left, right, .. } => { + collect_value_vars(left, out); + collect_value_vars(right, out); + } + BooleanExpression::And(l, r) | BooleanExpression::Or(l, r) => { + collect_bool_vars(l, out); + collect_bool_vars(r, out); + } + BooleanExpression::Not(inner) => collect_bool_vars(inner, out), + BooleanExpression::Exists(p) => out.push(&p.variable), + BooleanExpression::In { expression, list } => { + collect_value_vars(expression, out); + for v in list { + collect_value_vars(v, out); + } + } + BooleanExpression::Like { expression, .. } + | BooleanExpression::ILike { expression, .. } + | BooleanExpression::Contains { expression, .. } + | BooleanExpression::StartsWith { expression, .. } + | BooleanExpression::EndsWith { expression, .. } + | BooleanExpression::IsNull(expression) + | BooleanExpression::IsNotNull(expression) => collect_value_vars(expression, out), + } +} + +/// `has` — any aggregate at all. `sensitive` — an aggregate whose answer +/// moves with the path count: a non-DISTINCT `count`/`sum`/`avg`/`collect`. +/// `min`/`max` and every DISTINCT aggregate see only the support. +fn aggregate_flags(v: &ValueExpression, has: &mut bool, sensitive: &mut bool) { + match v { + ValueExpression::AggregateFunction { + name, + args, + distinct, + } => { + *has = true; + let lower = name.to_lowercase(); + let bag = matches!(lower.as_str(), "count" | "sum" | "avg" | "collect"); + if bag && !*distinct { + *sensitive = true; + } + for a in args { + aggregate_flags(a, has, sensitive); + } + } + ValueExpression::ScalarFunction { args, .. } => { + for a in args { + aggregate_flags(a, has, sensitive); + } + } + ValueExpression::Arithmetic { left, right, .. } + | ValueExpression::VectorDistance { left, right, .. } + | ValueExpression::VectorSimilarity { left, right, .. } => { + aggregate_flags(left, has, sensitive); + aggregate_flags(right, has, sensitive); + } + _ => {} + } +} + #[cfg(test)] mod tests { use super::*; @@ -1414,4 +1683,178 @@ mod tests { err_msg ); } + + // --- multiplicity contract (cypher-mask-multiplicity-contract-v1 §5 G2/G3) --- + + fn semantics(query: &str) -> ConsumerSemantics { + let ast = parse_cypher_query(query).unwrap(); + let config = GraphConfig::default(); + let mut planner = LogicalPlanner::new(&config); + planner.plan(&ast).unwrap().consumer_semantics() + } + + const TWO_HOP: &str = "MATCH (a:Person)-[:KNOWS]->(b:Person)-[:KNOWS]->(c:Person)"; + + fn two_hop(tail: &str) -> ConsumerSemantics { + semantics(&format!("{TWO_HOP} {tail}")) + } + + #[test] + fn terminal_set_consumers_need_only_the_terminal_support() { + assert_eq!( + two_hop("RETURN DISTINCT c.name"), + ConsumerSemantics::TerminalSet + ); + assert_eq!( + two_hop("RETURN count(DISTINCT c) AS n"), + ConsumerSemantics::TerminalSet + ); + assert_eq!( + two_hop("RETURN min(c.age) AS m"), + ConsumerSemantics::TerminalSet + ); + } + + #[test] + fn an_earlier_variable_is_not_the_forward_frontier() { + // count(DISTINCT b) = 3 on the §0 fixture while the forward dst₁ is 4. + assert_eq!( + two_hop("RETURN count(DISTINCT b) AS n"), + ConsumerSemantics::EarlierSet + ); + assert_eq!( + two_hop("RETURN min(a.age) AS m"), + ConsumerSemantics::EarlierSet + ); + } + + #[test] + fn path_counting_consumers_need_a_count_lane() { + assert_eq!( + two_hop("RETURN count(*) AS n"), + ConsumerSemantics::TerminalCount + ); + assert_eq!( + two_hop("RETURN sum(c.age) AS s"), + ConsumerSemantics::TerminalCount + ); + // The pair that keeps T-11 honest: the same variable, with and without DISTINCT. + assert_eq!(two_hop("RETURN c.name"), ConsumerSemantics::TerminalCount); + assert_eq!( + two_hop("RETURN DISTINCT c.name"), + ConsumerSemantics::TerminalSet + ); + assert_eq!( + two_hop("RETURN c.name, count(*) AS n"), + ConsumerSemantics::TerminalCount + ); + assert_eq!( + two_hop("RETURN a.name, count(*) AS n"), + ConsumerSemantics::EarlierCount + ); + assert_eq!( + two_hop("RETURN sum(a.age) AS s"), + ConsumerSemantics::EarlierCount + ); + } + + #[test] + fn two_variables_need_the_bindings() { + assert_eq!( + two_hop("RETURN a.name, c.name"), + ConsumerSemantics::Bindings + ); + assert_eq!( + two_hop("WHERE a.age = c.age RETURN count(DISTINCT c) AS n"), + ConsumerSemantics::Bindings + ); + assert_eq!( + semantics("MATCH (a:Person)-[:KNOWS]->(b:Person) WITH b RETURN count(*) AS n"), + ConsumerSemantics::Bindings + ); + } + + #[test] + fn a_single_population_is_one_row_per_node() { + // G3 — the can-stay-silent half: no hop, so a popcount is exact. + assert_eq!( + semantics("MATCH (n:Person) WHERE n.age > 30 RETURN count(*) AS n"), + ConsumerSemantics::TerminalSet + ); + assert_eq!( + semantics("MATCH (n:Person) RETURN n.name"), + ConsumerSemantics::TerminalSet + ); + // A single-variable WHERE after a hop is not a cross-variable filter. + assert_eq!( + two_hop("WHERE c.age > 30 RETURN count(DISTINCT c) AS n"), + ConsumerSemantics::TerminalSet + ); + } + + #[test] + fn hops_that_do_not_chain_are_not_a_forward_mask_chain() { + // Two arms sharing `a`: the planner nests the Expands, but the outer + // hop's source is `a`, not the inner hop's target `b`. + assert_eq!( + semantics( + "MATCH (a:Person)-[:KNOWS]->(b:Person), (a)-[:KNOWS]->(c:Person) \ + RETURN count(DISTINCT c) AS n" + ), + ConsumerSemantics::Bindings + ); + // A variable bound twice closes a cycle: binding identity again. + assert_eq!( + semantics( + "MATCH (a:Person)-[:KNOWS]->(b:Person)-[:KNOWS]->(a) \ + RETURN count(DISTINCT b) AS n" + ), + ConsumerSemantics::Bindings + ); + // Rebinding mid-chain: only the per-hop check sees it (the scan's + // variable `a` is fresh here). + assert_eq!( + semantics( + "MATCH (a:Person)-[:KNOWS]->(b:Person)-[:KNOWS]->(c:Person)-[:KNOWS]->(b) \ + RETURN count(DISTINCT c) AS n" + ), + ConsumerSemantics::Bindings + ); + // Silence twin: a genuine chain still classifies. + assert_eq!( + two_hop("RETURN count(DISTINCT c) AS n"), + ConsumerSemantics::TerminalSet + ); + } + + #[test] + fn two_disconnected_patterns_are_not_a_single_population() { + // G3's paired arm: no hop, but a Join — a cross product, not a popcount. + assert_eq!( + semantics("MATCH (a:Person), (b:Person) RETURN count(*) AS n"), + ConsumerSemantics::Bindings + ); + } + + /// `ConsumerSemantics` must never become persistable (§5.2 mint trip-wire). + /// Inherent methods whose impl bound fails are skipped, so the call falls + /// back to the trait method only when the type is NOT `Serialize`. + #[test] + fn consumer_semantics_is_not_serialize() { + struct Probe(std::marker::PhantomData); + trait NotSerialize { + fn is_serialize(&self) -> bool { + false + } + } + impl NotSerialize for Probe {} + impl Probe { + fn is_serialize(&self) -> bool { + true + } + } + assert!(!Probe::(std::marker::PhantomData).is_serialize()); + // Can-fire: the probe does detect a Serialize type. + assert!(Probe::(std::marker::PhantomData).is_serialize()); + } } diff --git a/crates/lance-graph/tests/test_datafusion_varlength_complex.rs b/crates/lance-graph/tests/test_datafusion_varlength_complex.rs index 46c220a9f..1a5a5a807 100644 --- a/crates/lance-graph/tests/test_datafusion_varlength_complex.rs +++ b/crates/lance-graph/tests/test_datafusion_varlength_complex.rs @@ -834,3 +834,119 @@ async fn test_varlength_all_pairs_reachability() { "Should find at least 15 connected pairs" ); } + +// --------------------------------------------------------------------------- +// Multiplicity: a mask is the SUPPORT of a frontier, never its path count. +// +// These pin DataFusion's bag answers — the reference any mask lowering is +// diffed against (`.claude/plans/cypher-mask-multiplicity-contract-v1.md` +// §5 G1). Each query is paired with its DISTINCT twin, so a lowering that +// returns the distinct count for `count(*)` fails here, which a set-equality +// oracle cannot see. +// --------------------------------------------------------------------------- + +fn multiplicity_people(ids: &[i64]) -> RecordBatch { + let schema = Arc::new(Schema::new(vec![Field::new("id", DataType::Int64, false)])); + RecordBatch::try_new(schema, vec![Arc::new(Int64Array::from(ids.to_vec()))]).unwrap() +} + +fn multiplicity_knows(edges: &[(i64, i64)]) -> RecordBatch { + let schema = Arc::new(Schema::new(vec![ + Field::new("src_person_id", DataType::Int64, false), + Field::new("dst_person_id", DataType::Int64, false), + ])); + let src: Vec = edges.iter().map(|e| e.0).collect(); + let dst: Vec = edges.iter().map(|e| e.1).collect(); + RecordBatch::try_new( + schema, + vec![ + Arc::new(Int64Array::from(src)), + Arc::new(Int64Array::from(dst)), + ], + ) + .unwrap() +} + +/// Run `query` (which must alias its single result `n`) and return it. +async fn multiplicity_count(query: &str, ids: &[i64], edges: &[(i64, i64)]) -> i64 { + let mut datasets = HashMap::new(); + datasets.insert("Person".to_string(), multiplicity_people(ids)); + datasets.insert("KNOWS".to_string(), multiplicity_knows(edges)); + let out = CypherQuery::new(query) + .unwrap() + .with_config(create_complex_graph_config()) + .execute(datasets, Some(ExecutionStrategy::DataFusion)) + .await + .unwrap(); + assert_eq!(out.num_rows(), 1, "{query}"); + out.column_by_name("n") + .unwrap() + .as_any() + .downcast_ref::() + .unwrap() + .value(0) +} + +/// A DAG: 1→2, 1→3, 2→3, 3→4, 4→5. +const DAG: &[(i64, i64)] = &[(1, 2), (1, 3), (2, 3), (3, 4), (4, 5)]; + +#[tokio::test] +async fn two_hop_count_star_counts_paths_not_endpoints() { + let ids = [1, 2, 3, 4, 5]; + let paths = multiplicity_count( + "MATCH (a:Person)-[:KNOWS]->(b:Person)-[:KNOWS]->(c:Person) RETURN count(*) AS n", + &ids, + DAG, + ) + .await; + let endpoints = multiplicity_count( + "MATCH (a:Person)-[:KNOWS]->(b:Person)-[:KNOWS]->(c:Person) \ + RETURN count(DISTINCT c.id) AS n", + &ids, + DAG, + ) + .await; + // 1→2→3, 1→3→4, 2→3→4, 3→4→5 — but only {3, 4, 5} as endpoints. + assert_eq!(paths, 4); + assert_eq!(endpoints, 3); +} + +#[tokio::test] +async fn var_length_count_star_counts_paths_not_endpoints() { + let ids = [1, 2, 3, 4, 5]; + let paths = multiplicity_count( + "MATCH (a:Person)-[:KNOWS*1..2]->(b:Person) WHERE a.id = 1 RETURN count(*) AS n", + &ids, + DAG, + ) + .await; + let endpoints = multiplicity_count( + "MATCH (a:Person)-[:KNOWS*1..2]->(b:Person) WHERE a.id = 1 \ + RETURN count(DISTINCT b.id) AS n", + &ids, + DAG, + ) + .await; + // 1→2, 1→3, 1→2→3, 1→3→4 — endpoints {2, 3, 4}. + assert_eq!(paths, 4); + assert_eq!(endpoints, 3); +} + +/// On a cycle, walks and trails differ. Cypher's relationship uniqueness +/// forbids 2→2→2 (it reuses the self-loop), so Cypher answers 4. This pins +/// what DataFusion actually returns — an OPEN divergence, not a fix. +#[tokio::test] +async fn two_hop_on_a_cycle_counts_walks() { + let cyclic: &[(i64, i64)] = &[(1, 2), (2, 1), (2, 2)]; + let paths = multiplicity_count( + "MATCH (a:Person)-[:KNOWS]->(b:Person)-[:KNOWS]->(c:Person) RETURN count(*) AS n", + &[1, 2], + cyclic, + ) + .await; + // Walks: 1→2→1, 1→2→2, 2→1→2, 2→2→1, 2→2→2 = 5. Trails drop 2→2→2 = 4. + assert_eq!( + paths, 5, + "DataFusion counts walks; Cypher's trail count is 4" + ); +}