diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 9f1daee4f..8bf267f0d 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -31,7 +31,7 @@ primitive" to "add a fusion rule". | D-WFL-EXPR | **Mask algebra is globally NON-MATERIALIZING by default.** A mask EXPRESSION denotes membership; it does not imply a bitmap exists. Materialization happens only at an explicit TERMINAL, when the membership set is requested as a carrier. Stronger than "folds are zero-copy" because folding, masking, ternlog, gating, projection and reduction all join ONE algebra — the expression stays unevaluated as population state all the way to a low-entropy terminal | Queued | three concepts kept distinct in every plan: MASKING (operation) · MASK EXPRESSION (composition) · MATERIALIZED MASK (bitmap). Falsified if a plan cannot express a multi-operand masking chain that emits no membership bits | | D-WFL-MASKOP | ⊘ **`MaskOp` must not semantically mean "produce a Scratch mask" — it must mean CONTRIBUTE TO A MASK EXPRESSION.** Scratch is one physical LOWERING, never the semantics. Read-verified: `Terminal::Keep{mask}` (`ir.rs:184`, *"the final mask itself stays in `mask`… nothing is reduced"*) IS the materialization election, but `MaskOp::And{a,b,dst}` is `dst = a & b` — every op is an assignment, so the ops destroy at level N−1 the choice the terminals encode at level N, and `exec.rs:566` then forces every slot to `words_for(n_rows)` | Queued | **This is the deepest correction in the arc and it precedes W0/W1.** Falsified if changing `MaskOp` semantics does not remove the need for per-op fixes | | D-WFL-SEAMB″ | ⊘ `Pred::Range → Scratch` is a **SYMPTOM, not the disease** — every earlier framing (performance complaint · T1 conformance failure · fold-law violation · absent decision) was chasing one op. Fixing `Range` alone leaves `And`/`Or`/`Xor`/`AndNot`/`Ternlog` all writing full planes | Queued | the fix is judged at the execution MODEL, not at one variant | -| D-WFL-FUSE | **Half the "missing primitives" dissolve into lowering rules.** The fused `popcount(a & b)` of `D-WFL-T1-FUSED` is not a bespoke instruction — it is what a fuser emits for `MaskExpr → Terminal::Count`. `fuse.rs` already collapses a Boolean tree into one ternlog; what it does NOT do is fuse across the **op → terminal** boundary, which is exactly the boundary D-WFL-MASKOP moves | Queued | re-audit every "missing op" against this before minting any. A primitive that a fusion rule could emit is not a primitive | +| D-WFL-FUSE | **Half the "missing primitives" dissolve into lowering rules.** The fused `popcount(a & b)` of `D-WFL-T1-FUSED` is not a bespoke instruction — it is what a fuser emits for `MaskExpr → Terminal::Count`. `fuse.rs` already collapses a Boolean tree into one ternlog; what it does NOT do is fuse across the **op → terminal** boundary, which is exactly the boundary D-WFL-MASKOP moves | In PR for the single-op case — `Program::fused_ternlog` folds one resident 2/3-input op → Count/Any; multi-op trees (fuse.rs → one ternlog → this fold) still OPEN | re-audit every "missing op" against this before minting any. A primitive that a fusion rule could emit is not a primitive | **Why it took a day, so it is not repeated:** the design already encoded the distinction (`Keep` vs `Count`; `WideFieldMask` as a field-PARTICIPATION @@ -70,7 +70,7 @@ defines what a fold IS — never what the machine may do. | D-WFL-SIBLING | **The rule governs the TRANSITION, not the bytes:** crossing from fold-native to mask-native execution must be deliberate and visible at the T2 planning membrane. Once MASK is elected, behaving like a mask engine (AND → TERNLOG → shift → cache) is legitimate. Forbidden only: the planner believes it is folding, a helper silently allocates `words_for(N)`, and nobody made the decision | Queued | the BBB question must be answerable for every plan: *who elected the mask, on what basis?* Falsified if a plan can become mask-native without an election appearing in it | | D-WFL-SEAMB′ | ⊘ **restates Seam B more precisely than every earlier framing** (performance complaint · T1 conformance failure · fold-law violation — all circling this). The defect in `Pred::Range` is NOT that it writes a mask. It is that the planner can neither elect nor decline: there is exactly ONE path, so **the choice does not exist**. Seam B is an ABSENT DECISION, not a present mask | Queued | fixed when both paths exist and the plan records which was taken — not when the mask disappears | | D-WFL-W2b″ | ⊘ **supersedes D-WFL-W2b′'s "must not write".** W2b demonstrates BOTH legal paths over identical semantics: FOLD-NATIVE (`Range ∩ resident → Count/Any`, no second mask) and MASK-NATIVE (`→ a bounded/cached mask` because a consumer reuses it). Pipeline vs materialize | Queued | the two arms differentially checked against each other AND the oracle — identical row sets, identical Count/Any. The earlier "zero derived buffers, asserted by counter" gate now scopes to the FOLD arm only | -| D-WFL-T1-FUSED′ | ⊘ upgrade from optimization to **enabler**: without a fused `popcount(a & b)` over a span there is no intermediate-buffer-free path, so **the fold-native arm does not exist at all**. The primitive CREATES the choice — which is exactly why Seam B had no decision in it | Refuted for `Range ∩ plane` (existing `popcount_batch_u64`/`mask_any` over the borrowed span suffice); plane∩plane `and_popcount` still OPEN | unchanged differential gate vs `mask_and` + `popcount_batch_u64`; the framing change raises its priority from nice-to-have to W2b-blocking | +| D-WFL-T1-FUSED′ | ⊘ upgrade from optimization to **enabler**: without a fused `popcount(a & b)` over a span there is no intermediate-buffer-free path, so **the fold-native arm does not exist at all**. The primitive CREATES the choice — which is exactly why Seam B had no decision in it | In PR — plane∩plane closed as the `AND2` table of the generalized `mask_ternlog_popcount`/`_any` (ndarray #322; no new ISA primitive — `U64x8` composition suffices); `entries/2026-09-23-ternlog-count-any-fold.md` | unchanged differential gate vs `mask_and` + `popcount_batch_u64`; the framing change raises its priority from nice-to-have to W2b-blocking | | D-WFL-ELECT | the election rule, static first: `terminal Count → FOLD`; `one AND then Count → probably FOLD`; `reuse_count > 1 → consider MASK`; `shared cached result → MASK`; `Wabe frontier reused → maybe MASK`; `~11 ns cached mask → almost certainly MASK`. DuckDB-style dynamic costing later | Queued | static rules must be inspectable in the plan. Falsified if the rule set fires the same way on every program (it would carry no information — cf. the can-it-stay-silent twin) | ## D-WFL — the cache scoping (2026-09-19): frozen is fine, marching is the disaster @@ -106,7 +106,7 @@ newly written derived buffer. | D-id | scope | status | gate | |---|---|---|---| | D-WFL-W2b′ | **respec: bounded composition must not WRITE the intersection.** `WRONG: Range × resident mask → write a bounded mask → Count/Any`. `RIGHT: peek only the intersecting resident words → AND in registers → Count/Any`. The moment the bounded mask is written the program has crossed into RECONSTRUCT — legitimately perhaps, but it is no longer a fold and must be named | Queued | zero derived buffers allocated or written between the bound and the terminal, asserted by counter. A bounded-mask write fails the wave even at 12 words | -| D-WFL-T1-FUSED | the clean case for the anti-zoo rule licensing a NEW T1 primitive: a fused `popcount(a[i] & b[i])` accumulated over a word span. It cannot be expressed by the existing algebra without an intermediate buffer, so it exposes a genuinely new zero-copy operation rather than a convenience | Queued | differential vs `mask_and` + `popcount_batch_u64` over the same span; identical answer, zero intermediate bytes. Falsified if composition already achieves it without a buffer | +| D-WFL-T1-FUSED | the clean case for the anti-zoo rule licensing a NEW T1 primitive: a fused `popcount(a[i] & b[i])` accumulated over a word span. It cannot be expressed by the existing algebra without an intermediate buffer, so it exposes a genuinely new zero-copy operation rather than a convenience | In PR — composition falsifier HELD at the ISA level (`U64x8` ternlog/popcnt everywhere); the slice loop landed generalized over all 256 tables (ndarray #322); differential gate met | differential vs `mask_and` + `popcount_batch_u64` over the same span; identical answer, zero intermediate bytes. Falsified if composition already achieves it without a buffer | ## D-WFL-L0 — the foundational ruling, to land BEFORE any W0/W1 code (2026-09-19) diff --git a/.claude/board/entries/2026-09-23-ternlog-count-any-fold.md b/.claude/board/entries/2026-09-23-ternlog-count-any-fold.md new file mode 100644 index 000000000..763948d31 --- /dev/null +++ b/.claude/board/entries/2026-09-23-ternlog-count-any-fold.md @@ -0,0 +1,58 @@ +# 2026-09-23 — Boolean membership → Count/Any without a mask (ternlog fold) + +**Status:** MEASURED (fold shipped in PR, ndarray #322 + this lance-graph PR) · OPEN (multi-op trees, compare-count) +**D-ids:** D-WFL-T1-FUSED, D-WFL-T1-FUSED′, D-WFL-FUSE (single-op case). OQ-5 is untouched and still open. + +## The question, and its answer +The question was the smallest T1 operation that lets a 2/3-input Boolean membership end in Count/Any without writing a mask, and whether the existing `U64x8` composition already does it without writing to memory. + +**Answer (MEASURED):** +- **No new ISA primitive is needed.** `U64x8::{ternlog::, popcnt, +, |, reduce_sum}` exists on every realization: avx512, avx2-polyfill, scalar, neon and wasm. +- **Only the slice loop was missing.** It belongs in T1, because mask-risc must not host a SIMD loop. +- **What landed in ndarray #322** is one family, not one function per op: `mask_ternlog_popcount::` and `mask_ternlog_any::`. A 2-input op is a table that ignores `c`. +- **Effect on the D-WFL-T1-FUSED rows:** + - D-WFL-T1-FUSED's own falsifier held ("falsified if composition already achieves it without a buffer") at the ISA level. + - Its differential gate (`mask_and` + `popcount_batch_u64`, identical answer, zero intermediate bytes) is met by the generalized function for all 256 tables. + - D-WFL-T1-FUSED′'s open "plane∩plane `and_popcount`" is closed as the `AND2` table of this family. + +Probe: ndarray `examples/ternlog_fold_probe.rs` compares materialize-then-reduce against the register fold. + +| backend | words | Count gain | Any gain (worst case, no hit anywhere) | +|---|---|---|---| +| avx2 | 16 384 | 1.27× | 3.24× | +| avx2 | 262 144 | 1.50× | 6.18× | +| avx512 | 16 384 | 1.90× | 2.59× | +| avx512 | 262 144 | 1.94× | 4.39× | + +- The first version of Any tested the accumulator after every chunk and lost to materializing on avx2 at 16 384 words (0.91×). The shipped form tests once per block of 8 chunks. +- A plain scalar fused loop about ties the register fold on avx2. **The win comes from not writing the mask, not from SIMD.** + +## The mask-risc fold (CURRENT-CONTRACT, TEST-PINNED) +- **Shape recognised:** `Program::fused_ternlog()` matches a single `And`, `Or`, `Xor`, `AndNot` or `Ternlog` whose operands are all `Operand::Plane`, folded by `Count` or `Any` of its own `dst`. Like `fused_terminal`, this is a derived predicate. `requires_scratch()` now accounts for both. +- **Execution:** + - Whole words go through `ternlog_{popcount,any}_dispatch`, a generated 256-arm table checked by `--check` and by the arm-count test. + - Each word the extent cuts (at most two) is combined in a one-word register and ANDed with `edge_mask`. +- **Odd tables:** + - The population's last word counts as a cut word, so an odd table (true on all-zero inputs) never counts dead tail bits. + - The ndarray function counts at word level; the tail is the caller's to mask, and it is masked here. +- **Not fused** (these keep the tiled path): `Keep`, `All`, `Not`, scratch operands, and multi-op programs. + +## Evidence +- **`tests/fused_ternlog.rs`:** the fold equals the Keep arm and a bit-serial oracle: + - the four 2-input ops across 7 populations (1 … 4133) × 4 plane shapes × absolute extents cut at 63/64/65/127/128/129; + - all 256 tables at 65 and 133 rows; + - the odd-table tail; + - zero allocation and an untouched poisoned arena, plus the paired half proving the probe can see a `Keep` carve; + - the recogniser's admit and refuse halves. +- **Disable runs,** made after committing and each failing as intended: + - recogniser off: 5 of 5 red; + - tail cut ignored: 3 red; + - head cut ignored: 2 red; + - wrong OR table: 1 red; + - edge word not masked: 3 red. +- **Probe:** `examples/ternlog_fused_probe.rs` at N = 1M rows, fold vs `Keep` + `popcount_batch_u64`. Count: 1.25–1.44× at 1 row, 3.4–7.4× at 1 %, 5.3–20× whole. The whole-population Any ratios (600–960×) are almost all early exit on dense planes, so they are **not** a like-for-like cost comparison; the worst-case figure for Any is the ndarray row above. + +## What remains +- **Multi-op Boolean trees** (e.g. `(a & b) | !c` spread across ops) still materialize scratch. The next step is `fuse.rs` collapsing such a tree into ONE ternlog and handing that to this fold. Not started. +- **Lane predicate → Count/Any** (compare-count / compare-fold) is the next PR. Not started. +- **OQ-5** (thread/rayon vendor) is untouched. No scheduler work has started. diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index d41cf0bf1..98054fc98 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,10 +25,11 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -151 entries, 2026-08-06 .. 2026-09-23. +152 entries, 2026-08-06 .. 2026-09-23. | date | entry id | finding | file | |---|---|---|---| +| 2026-09-23 | `ternlog-count-any-fold` | | [2026-09-23-ternlog-count-any-fold.md](2026-09-23-ternlog-count-any-fold.md) | | 2026-09-23 | `terminal-elects-materialization-range-plane` | | [2026-09-23-terminal-elects-materialization-range-plane.md](2026-09-23-terminal-elects-materialization-range-plane.md) | | 2026-09-23 | `quack-having-sym-sum-presence-mask` | | [2026-09-23-quack-having-sym-sum-presence-mask.md](2026-09-23-quack-having-sym-sum-presence-mask.md) | | 2026-09-23 | `cubecl-llvm-boundary-and-audit-regrade` | | [2026-09-23-cubecl-llvm-boundary-and-audit-regrade.md](2026-09-23-cubecl-llvm-boundary-and-audit-regrade.md) | diff --git a/crates/lance-graph-mask-risc/examples/ternlog_fused_probe.rs b/crates/lance-graph-mask-risc/examples/ternlog_fused_probe.rs new file mode 100644 index 000000000..ff14710ce --- /dev/null +++ b/crates/lance-graph-mask-risc/examples/ternlog_fused_probe.rs @@ -0,0 +1,171 @@ +//! Boolean membership → Count/Any at N = 1M rows: the fold vs the kept mask. +//! +//! One op over three resident planes, two physical endings: +//! +//! - FOLD: `Count` / `Any` of the op's own slot — `Program::fused_ternlog` +//! lowers it onto `ndarray::simd::mask_ternlog_{popcount,any}`; no slot is +//! carved and no membership word is written. +//! - MATERIALIZE: the same op with a `Keep` terminal into an `Out::Mask`, then +//! `popcount_batch_u64` / `mask_any` over the kept words — the +//! bitmap-then-reduce shape the fold replaces. +//! +//! Reported per shape × extent: median ns for each arm and the membership +//! words the materializing arm writes (the fold writes none). Every result is +//! checked against a bit-serial scalar oracle over the same absolute rows. +//! +//! `cargo run --release -p lance-graph-mask-risc --example ternlog_fused_probe` + +use std::time::Instant; + +use lance_graph_mask_risc::exec::{execute_extent, Scratch}; +use lance_graph_mask_risc::{ + touched_words, Foreign, MaskOp, Operand, Out, Planes, Program, Terminal, Value, +}; +use ndarray::simd::{mask_any, popcount_batch_u64}; + +fn lcg(seed: &mut u64) -> u64 { + *seed = seed + .wrapping_mul(6364136223846793005) + .wrapping_add(1442695040888963407); + *seed >> 11 +} + +fn median(reps: usize, mut f: impl FnMut() -> T) -> (f64, T) { + let mut ts = Vec::with_capacity(reps); + let mut last = None; + for _ in 0..reps { + let t = Instant::now(); + last = Some(std::hint::black_box(f())); + ts.push(t.elapsed().as_nanos() as f64); + } + ts.sort_by(|a, b| a.total_cmp(b)); + (ts[reps / 2], last.expect("reps > 0")) +} + +fn main() { + let n = 1usize << 20; + let words = n.div_ceil(64); + let mut seed = 0x7E4F_u64; + let mut mk = |dense: bool| { + let mut w = vec![0u64; words]; + for r in 0..n { + let x = lcg(&mut seed); + if (dense && !x.is_multiple_of(3)) || (!dense && x.is_multiple_of(29)) { + w[r / 64] |= 1 << (r % 64); + } + } + w + }; + let (pa, pb, pc) = (mk(true), mk(true), mk(false)); + let masks: [&[u64]; 3] = [&pa, &pb, &pc]; + let planes = Planes { + n_rows: n, + masks: &masks, + lanes: &[], + }; + let (a, b, c) = (Operand::Plane(0), Operand::Plane(1), Operand::Plane(2)); + let shapes: [(&str, MaskOp, u8); 4] = [ + ("And", MaskOp::And { a, b, dst: 0 }, 0xC0), + ("Xor", MaskOp::Xor { a, b, dst: 0 }, 0x3C), + ( + "MAJ3", + MaskOp::Ternlog { + imm: 0xE8, + a, + b, + c, + dst: 0, + }, + 0xE8, + ), + ( + "NOR3", + MaskOp::Ternlog { + imm: 0x01, + a, + b, + c, + dst: 0, + }, + 0x01, + ), + ]; + let mid = n / 2 + 17; + let extents = [ + ("1 row", mid, mid + 1), + ("1%", mid, mid + n / 100), + ("whole", 0, n), + ]; + let bit = |p: &[u64], r: usize| (p[r / 64] >> (r % 64) & 1) as u8; + println!( + "{:>5} {:>6} {:>5} {:>11} {:>11} {:>7} {:>10}", + "op", "extent", "term", "fold_ns", "keep_ns", "k/f", "keep_wr" + ); + for (name, op, table) in shapes { + let count = Program::new( + vec![op], + Terminal::Count { + mask: Operand::Scratch(0), + }, + ); + let any = Program::new( + vec![op], + Terminal::Any { + mask: Operand::Scratch(0), + }, + ); + let keep = Program::new( + vec![op], + Terminal::Keep { + mask: Operand::Scratch(0), + }, + ); + assert!(count.fused_ternlog().is_some() && any.fused_ternlog().is_some()); + let mut ks = Scratch::for_program(&keep, n).expect("scratch"); + let mut out = vec![0u64; words]; + for (ename, lo, hi) in extents { + let want = (lo..hi) + .filter(|&r| table >> (bit(&pa, r) << 2 | bit(&pb, r) << 1 | bit(&pc, r)) & 1 == 1) + .count(); + let reps = if hi - lo > 100_000 { 41 } else { 2001 }; + let span = touched_words(lo as u32, hi as u32); + for (term, p) in [("Count", &count), ("Any", &any)] { + let (fns, fv) = median(reps, || { + let mut s = Scratch::new(0, 0); + execute_extent(p, &planes, &Foreign::NONE, &mut s, Out::None, lo..hi) + .expect("fold") + }); + let (kns, kv) = median(reps, || { + out[span.clone()].fill(0); + execute_extent( + &keep, + &planes, + &Foreign::NONE, + &mut ks, + Out::Mask(&mut out), + lo..hi, + ) + .expect("keep"); + if term == "Count" { + Value::Count(popcount_batch_u64(&out[span.clone()]) as usize) + } else { + Value::Bool(mask_any(&out[span.clone()])) + } + }); + let expect = if term == "Count" { + Value::Count(want) + } else { + Value::Bool(want > 0) + }; + assert_eq!(fv, expect, "fold {name} {ename} {term}"); + assert_eq!(kv, expect, "keep {name} {ename} {term}"); + println!( + "{name:>5} {ename:>6} {term:>5} {fns:>11.0} {kns:>11.0} {:>7.2} {:>10}", + kns / fns, + span.len() + ); + } + } + } + println!("(n = {n}, {words} population words; the fold writes 0 membership words)"); +} diff --git a/crates/lance-graph-mask-risc/src/exec.rs b/crates/lance-graph-mask-risc/src/exec.rs index 6ade6d0f7..79281220a 100644 --- a/crates/lance-graph-mask-risc/src/exec.rs +++ b/crates/lance-graph-mask-risc/src/exec.rs @@ -38,11 +38,14 @@ use ndarray::simd::{ }; use crate::ir::{ - span_words, touched_words, Foreign, FusedFold, FusedTerminal, GroupFold, GroupKey, LaneRef, - MaskOp, Operand, Planes, Pred, Program, Terminal, MAX_SCRATCH_SLOTS, + span_words, touched_words, Foreign, FusedFold, FusedTerminal, FusedTernlog, GroupFold, + GroupKey, LaneRef, MaskOp, Operand, Planes, Pred, Program, Terminal, FUSED_SLOT_CAP, + MAX_SCRATCH_SLOTS, }; use crate::reference::{out_shape, validate}; -use crate::ternlog_dispatch::{ternlog_dispatch, ternlog_dispatch_assign}; +use crate::ternlog_dispatch::{ + ternlog_any_dispatch, ternlog_dispatch, ternlog_dispatch_assign, ternlog_popcount_dispatch, +}; use crate::value::{ExecError, Out, Value}; use crate::words_for; use core::ops::Range; @@ -712,6 +715,77 @@ fn run_fused(f: FusedTerminal, planes: &Planes<'_>) -> Value { } } +/// Evaluate a [`FusedTernlog`] over the absolute extent `[lo, hi)`. +/// +/// Nothing is written. Whole words inside the extent go straight to +/// `ndarray::simd::mask_ternlog_{popcount,any}` over the borrowed resident +/// planes. A word the extent cuts — an unaligned `lo`, or an unaligned `hi`, +/// which includes the population's own last word when `n_rows % 64 != 0` — is +/// combined one word at a time into a one-word register and restricted by +/// [`edge_mask`] before it is counted. That last case is load-bearing for an +/// ODD table: `f(0,0,0) = 1` sets the dead tail bits of the last word, and +/// only the edge mask keeps them out of the count (the tiled path clears the +/// same tail against `n_rows`). +fn run_fused_ternlog(f: FusedTernlog, planes: &Planes<'_>, lo: usize, hi: usize) -> Value { + let span = span_words(lo, hi); + let empty = match f.fold { + FusedFold::Count => Value::Count(0), + FusedFold::Any => Value::Bool(false), + }; + if span.is_empty() { + return empty; + } + let (a, b, c) = ( + planes.masks[usize::from(f.a)], + planes.masks[usize::from(f.b)], + planes.masks[usize::from(f.c)], + ); + let (first, last) = (span.start, span.end - 1); + let head_cut = !lo.is_multiple_of(64); + let tail_cut = !hi.is_multiple_of(64); + // One cut word, combined in a register and restricted to the extent. + let edge = |w: usize| -> u64 { + let mut reg = [0u64; 1]; + ternlog_dispatch(f.imm, &a[w..=w], &b[w..=w], &c[w..=w], &mut reg); + reg[0] & edge_mask(w, lo, hi) + }; + let mut edges = [None, None]; + if head_cut { + edges[0] = Some(edge(first)); + } + if tail_cut && (last != first || !head_cut) { + edges[1] = Some(edge(last)); + } + let from = first + usize::from(head_cut); + let to = (last + 1).saturating_sub(usize::from(tail_cut)).max(from); + let interior = from..to; + match f.fold { + FusedFold::Count => { + let e: u64 = edges + .iter() + .flatten() + .map(|w| u64::from(w.count_ones())) + .sum(); + let n = ternlog_popcount_dispatch( + f.imm, + &a[interior.clone()], + &b[interior.clone()], + &c[interior], + ); + Value::Count((e + n) as usize) + } + FusedFold::Any => Value::Bool( + edges.iter().flatten().any(|&w| w != 0) + || ternlog_any_dispatch( + f.imm, + &a[interior.clone()], + &b[interior.clone()], + &c[interior], + ), + ), + } +} + /// The tiles an execution over the ABSOLUTE row extent `[lo, hi)` visits, /// each as `(word range, edge)`. /// @@ -925,7 +999,7 @@ pub fn execute_extent( // Validation stays total — the one declared slot is tracked in a local // word of read-before-write bookkeeping, never in the caller's arena. if let Some(f) = program.fused_terminal() { - let mut written = [0u64; 1]; + let mut written = [0u64; FUSED_SLOT_CAP.div_ceil(64)]; validate(program, planes, foreign, out_shape(&out), &mut written)?; // The extent composes with the program's own range by intersection, // in absolute rows — the #1268 fold, over a narrower span. @@ -941,6 +1015,13 @@ pub fn execute_extent( }; return Ok(run_fused(f, planes)); } + // The Boolean-membership fold: a single 2/3-input op over resident planes, + // folded by Count/Any — also no slot, no membership bit written. + if let Some(f) = program.fused_ternlog() { + let mut written = [0u64; FUSED_SLOT_CAP.div_ceil(64)]; + validate(program, planes, foreign, out_shape(&out), &mut written)?; + return Ok(run_fused_ternlog(f, planes, elo, ehi)); + } if scratch.slots() < program.scratch_slots as usize { return Err(ExecError::ScratchTooSmall { need: program.scratch_slots, diff --git a/crates/lance-graph-mask-risc/src/ir.rs b/crates/lance-graph-mask-risc/src/ir.rs index 31d84e062..4251a7149 100644 --- a/crates/lance-graph-mask-risc/src/ir.rs +++ b/crates/lance-graph-mask-risc/src/ir.rs @@ -597,6 +597,9 @@ impl Program { // it, so the shape is not fusable. Some(Operand::Scratch(_)) => return None, }; + if usize::from(*dst) >= FUSED_SLOT_CAP { + return None; + } let fold = match self.terminal { Terminal::Count { mask } if mask == Operand::Scratch(*dst) => FusedFold::Count, Terminal::Any { mask } if mask == Operand::Scratch(*dst) => FusedFold::Any, @@ -610,14 +613,54 @@ impl Program { }) } + /// The no-mask lowering of a Boolean membership over RESIDENT planes, if + /// this program is one: a single `And` / `Or` / `Xor` / `AndNot` / + /// `Ternlog` whose operands are all [`Operand::Plane`], folded by `Count` + /// or `Any` of its own `dst`. + /// + /// Every such op is one 3-input truth table (a 2-input op is a table that + /// ignores `c`), and `ndarray::simd::mask_ternlog_popcount` / + /// `mask_ternlog_any` fold a table straight from its operands' words to a + /// scalar. So the membership never becomes a bitmap: nothing is written. + /// + /// A scratch operand, `Not` (which clears the tail against `n_rows` — a + /// different shape), a predicate, a gather, or any other terminal returns + /// `None` and runs the ordinary path. `Keep` is NEVER fused. + pub fn fused_ternlog(&self) -> Option { + let plane = |o: &Operand| match *o { + Operand::Plane(p) => Some(p), + Operand::Scratch(_) => None, + }; + let (imm, a, b, c, dst) = match self.ops.as_slice() { + [MaskOp::And { a, b, dst }] => (TABLE_AND, plane(a)?, plane(b)?, plane(b)?, *dst), + [MaskOp::Or { a, b, dst }] => (TABLE_OR, plane(a)?, plane(b)?, plane(b)?, *dst), + [MaskOp::Xor { a, b, dst }] => (TABLE_XOR, plane(a)?, plane(b)?, plane(b)?, *dst), + [MaskOp::AndNot { a, b, dst }] => (TABLE_ANDNOT, plane(a)?, plane(b)?, plane(b)?, *dst), + [MaskOp::Ternlog { imm, a, b, c, dst }] => { + (*imm, plane(a)?, plane(b)?, plane(c)?, *dst) + } + _ => return None, + }; + if usize::from(dst) >= FUSED_SLOT_CAP { + return None; + } + let fold = match self.terminal { + Terminal::Count { mask } if mask == Operand::Scratch(dst) => FusedFold::Count, + Terminal::Any { mask } if mask == Operand::Scratch(dst) => FusedFold::Any, + _ => return None, + }; + Some(FusedTernlog { imm, a, b, c, fold }) + } + /// Whether executing this program needs any scratch slot at all. /// /// DERIVED, not declared: a flag is a claim, a derived predicate is a - /// proof. `false` exactly when [`Program::fused_terminal`] lowers the - /// program (or it names no slot), and [`crate::Scratch::for_program`] - /// carves zero slots for such a program. + /// proof. `false` exactly when [`Program::fused_terminal`] or + /// [`Program::fused_ternlog`] lowers the program (or it names no slot), + /// and [`crate::Scratch::for_program`] carves zero slots for such a + /// program. pub fn requires_scratch(&self) -> bool { - self.scratch_slots > 0 && self.fused_terminal().is_none() + self.scratch_slots > 0 && self.fused_terminal().is_none() && self.fused_ternlog().is_none() } /// Count of ops of each physical kind — the "logical ops vs physical @@ -678,6 +721,44 @@ pub struct FusedTerminal { pub fold: FusedFold, } +/// A Boolean membership over resident planes that the executor folds +/// without writing membership bits: see [`Program::fused_ternlog`]. +/// +/// The table is in the VPTERNLOG index convention `(a << 2) | (b << 1) | c` +/// that [`MaskOp::Ternlog`] already uses; a 2-input op reaches here as a +/// table that ignores `c` (and `c` repeats `b`). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct FusedTernlog { + /// The 8-bit truth table. + pub imm: u8, + /// Resident plane read as the table's `a`. + pub a: u16, + /// Resident plane read as the table's `b`. + pub b: u16, + /// Resident plane read as the table's `c`. + pub c: u16, + /// The scalar the terminal demands. + pub fold: FusedFold, +} + +/// `a & b` — true at indices 6, 7 (`a = b = 1`, either `c`). +pub(crate) const TABLE_AND: u8 = 0xC0; +/// `a | b` — true wherever `a` (4..7) or `b` (2, 3, 6, 7) is set. +pub(crate) const TABLE_OR: u8 = 0xFC; +/// `a ^ b` — true at 2, 3 (`b` only) and 4, 5 (`a` only). +pub(crate) const TABLE_XOR: u8 = 0x3C; +/// `a & !b` — true at 4, 5 (`a = 1`, `b = 0`). +pub(crate) const TABLE_ANDNOT: u8 = 0x30; + +/// The highest scratch slot (exclusive) a fused shape may name. +/// +/// The fused paths validate a program with an on-stack slot bitmap of +/// `FUSED_SLOT_CAP.div_ceil(64)` words and never touch real scratch, so a +/// slot the bitmap cannot mark would be rejected as a read before a write. +/// A shape naming a higher slot is simply not fused and runs on the tiled +/// path. A bound of the recogniser, never of the semantics. +pub const FUSED_SLOT_CAP: usize = 32; + /// The scalar folds that may consume membership without materializing it. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum FusedFold { diff --git a/crates/lance-graph-mask-risc/src/lib.rs b/crates/lance-graph-mask-risc/src/lib.rs index 280d24652..5ffa38abc 100644 --- a/crates/lance-graph-mask-risc/src/lib.rs +++ b/crates/lance-graph-mask-risc/src/lib.rs @@ -124,14 +124,16 @@ pub use exec::{ }; pub use fuse::{fuse, fuse_program, ternlog_imm, BoolExpr, FuseError, Fused}; pub use ir::{ - touched_words, Foreign, ForeignPlane, FusedFold, FusedTerminal, GroupFold, GroupKey, LaneRef, - MaskOp, Operand, Planes, Pred, Program, Terminal, GROUP_SUM_SYM_MAX_ROWS, - MASKED_SUM_I32_MAX_ROWS, MAX_SCRATCH_SLOTS, + touched_words, Foreign, ForeignPlane, FusedFold, FusedTerminal, FusedTernlog, GroupFold, + GroupKey, LaneRef, MaskOp, Operand, Planes, Pred, Program, Terminal, FUSED_SLOT_CAP, + GROUP_SUM_SYM_MAX_ROWS, MASKED_SUM_I32_MAX_ROWS, MAX_SCRATCH_SLOTS, }; pub use reference::{ reference_execute, reference_execute_into, reference_scratch, reference_scratch_with_foreign, }; -pub use ternlog_dispatch::{ternlog_dispatch, ternlog_dispatch_assign}; +pub use ternlog_dispatch::{ + ternlog_any_dispatch, ternlog_dispatch, ternlog_dispatch_assign, ternlog_popcount_dispatch, +}; pub use value::{ExecError, LaneKind, Out, Value}; /// Number of `u64` words a mask over `n_rows` occupies. diff --git a/crates/lance-graph-mask-risc/src/ternlog_dispatch.rs b/crates/lance-graph-mask-risc/src/ternlog_dispatch.rs index d6cb8c8fa..3466c50ec 100644 --- a/crates/lance-graph-mask-risc/src/ternlog_dispatch.rs +++ b/crates/lance-graph-mask-risc/src/ternlog_dispatch.rs @@ -52,7 +52,7 @@ //! when the committed file already matches. CI runs this on every PR that //! touches either the generator or this file, so a stale table fails the Rust Tests workflow. -use ndarray::simd::{mask_ternlog, mask_ternlog_assign}; +use ndarray::simd::{mask_ternlog, mask_ternlog_any, mask_ternlog_assign, mask_ternlog_popcount}; // GEN-TERNLOG-DISPATCH-BEGIN /// `dst = table[imm](a, b, c)` — the runtime immediate routed to the const-generic facade word. pub fn ternlog_dispatch(imm: u8, a: &[u64], b: &[u64], c: &[u64], dst: &mut [u64]) { @@ -577,6 +577,530 @@ pub fn ternlog_dispatch_assign(imm: u8, a: &mut [u64], b: &[u64], c: &[u64]) { 255 => mask_ternlog_assign::<255>(a, b, c), } } + +/// `Σ popcount(table[imm](a, b, c))` — the no-mask Count fold, routed like the others. +pub fn ternlog_popcount_dispatch(imm: u8, a: &[u64], b: &[u64], c: &[u64]) -> u64 { + match imm { + 0 => mask_ternlog_popcount::<0>(a, b, c), + 1 => mask_ternlog_popcount::<1>(a, b, c), + 2 => mask_ternlog_popcount::<2>(a, b, c), + 3 => mask_ternlog_popcount::<3>(a, b, c), + 4 => mask_ternlog_popcount::<4>(a, b, c), + 5 => mask_ternlog_popcount::<5>(a, b, c), + 6 => mask_ternlog_popcount::<6>(a, b, c), + 7 => mask_ternlog_popcount::<7>(a, b, c), + 8 => mask_ternlog_popcount::<8>(a, b, c), + 9 => mask_ternlog_popcount::<9>(a, b, c), + 10 => mask_ternlog_popcount::<10>(a, b, c), + 11 => mask_ternlog_popcount::<11>(a, b, c), + 12 => mask_ternlog_popcount::<12>(a, b, c), + 13 => mask_ternlog_popcount::<13>(a, b, c), + 14 => mask_ternlog_popcount::<14>(a, b, c), + 15 => mask_ternlog_popcount::<15>(a, b, c), + 16 => mask_ternlog_popcount::<16>(a, b, c), + 17 => mask_ternlog_popcount::<17>(a, b, c), + 18 => mask_ternlog_popcount::<18>(a, b, c), + 19 => mask_ternlog_popcount::<19>(a, b, c), + 20 => mask_ternlog_popcount::<20>(a, b, c), + 21 => mask_ternlog_popcount::<21>(a, b, c), + 22 => mask_ternlog_popcount::<22>(a, b, c), + 23 => mask_ternlog_popcount::<23>(a, b, c), + 24 => mask_ternlog_popcount::<24>(a, b, c), + 25 => mask_ternlog_popcount::<25>(a, b, c), + 26 => mask_ternlog_popcount::<26>(a, b, c), + 27 => mask_ternlog_popcount::<27>(a, b, c), + 28 => mask_ternlog_popcount::<28>(a, b, c), + 29 => mask_ternlog_popcount::<29>(a, b, c), + 30 => mask_ternlog_popcount::<30>(a, b, c), + 31 => mask_ternlog_popcount::<31>(a, b, c), + 32 => mask_ternlog_popcount::<32>(a, b, c), + 33 => mask_ternlog_popcount::<33>(a, b, c), + 34 => mask_ternlog_popcount::<34>(a, b, c), + 35 => mask_ternlog_popcount::<35>(a, b, c), + 36 => mask_ternlog_popcount::<36>(a, b, c), + 37 => mask_ternlog_popcount::<37>(a, b, c), + 38 => mask_ternlog_popcount::<38>(a, b, c), + 39 => mask_ternlog_popcount::<39>(a, b, c), + 40 => mask_ternlog_popcount::<40>(a, b, c), + 41 => mask_ternlog_popcount::<41>(a, b, c), + 42 => mask_ternlog_popcount::<42>(a, b, c), + 43 => mask_ternlog_popcount::<43>(a, b, c), + 44 => mask_ternlog_popcount::<44>(a, b, c), + 45 => mask_ternlog_popcount::<45>(a, b, c), + 46 => mask_ternlog_popcount::<46>(a, b, c), + 47 => mask_ternlog_popcount::<47>(a, b, c), + 48 => mask_ternlog_popcount::<48>(a, b, c), + 49 => mask_ternlog_popcount::<49>(a, b, c), + 50 => mask_ternlog_popcount::<50>(a, b, c), + 51 => mask_ternlog_popcount::<51>(a, b, c), + 52 => mask_ternlog_popcount::<52>(a, b, c), + 53 => mask_ternlog_popcount::<53>(a, b, c), + 54 => mask_ternlog_popcount::<54>(a, b, c), + 55 => mask_ternlog_popcount::<55>(a, b, c), + 56 => mask_ternlog_popcount::<56>(a, b, c), + 57 => mask_ternlog_popcount::<57>(a, b, c), + 58 => mask_ternlog_popcount::<58>(a, b, c), + 59 => mask_ternlog_popcount::<59>(a, b, c), + 60 => mask_ternlog_popcount::<60>(a, b, c), + 61 => mask_ternlog_popcount::<61>(a, b, c), + 62 => mask_ternlog_popcount::<62>(a, b, c), + 63 => mask_ternlog_popcount::<63>(a, b, c), + 64 => mask_ternlog_popcount::<64>(a, b, c), + 65 => mask_ternlog_popcount::<65>(a, b, c), + 66 => mask_ternlog_popcount::<66>(a, b, c), + 67 => mask_ternlog_popcount::<67>(a, b, c), + 68 => mask_ternlog_popcount::<68>(a, b, c), + 69 => mask_ternlog_popcount::<69>(a, b, c), + 70 => mask_ternlog_popcount::<70>(a, b, c), + 71 => mask_ternlog_popcount::<71>(a, b, c), + 72 => mask_ternlog_popcount::<72>(a, b, c), + 73 => mask_ternlog_popcount::<73>(a, b, c), + 74 => mask_ternlog_popcount::<74>(a, b, c), + 75 => mask_ternlog_popcount::<75>(a, b, c), + 76 => mask_ternlog_popcount::<76>(a, b, c), + 77 => mask_ternlog_popcount::<77>(a, b, c), + 78 => mask_ternlog_popcount::<78>(a, b, c), + 79 => mask_ternlog_popcount::<79>(a, b, c), + 80 => mask_ternlog_popcount::<80>(a, b, c), + 81 => mask_ternlog_popcount::<81>(a, b, c), + 82 => mask_ternlog_popcount::<82>(a, b, c), + 83 => mask_ternlog_popcount::<83>(a, b, c), + 84 => mask_ternlog_popcount::<84>(a, b, c), + 85 => mask_ternlog_popcount::<85>(a, b, c), + 86 => mask_ternlog_popcount::<86>(a, b, c), + 87 => mask_ternlog_popcount::<87>(a, b, c), + 88 => mask_ternlog_popcount::<88>(a, b, c), + 89 => mask_ternlog_popcount::<89>(a, b, c), + 90 => mask_ternlog_popcount::<90>(a, b, c), + 91 => mask_ternlog_popcount::<91>(a, b, c), + 92 => mask_ternlog_popcount::<92>(a, b, c), + 93 => mask_ternlog_popcount::<93>(a, b, c), + 94 => mask_ternlog_popcount::<94>(a, b, c), + 95 => mask_ternlog_popcount::<95>(a, b, c), + 96 => mask_ternlog_popcount::<96>(a, b, c), + 97 => mask_ternlog_popcount::<97>(a, b, c), + 98 => mask_ternlog_popcount::<98>(a, b, c), + 99 => mask_ternlog_popcount::<99>(a, b, c), + 100 => mask_ternlog_popcount::<100>(a, b, c), + 101 => mask_ternlog_popcount::<101>(a, b, c), + 102 => mask_ternlog_popcount::<102>(a, b, c), + 103 => mask_ternlog_popcount::<103>(a, b, c), + 104 => mask_ternlog_popcount::<104>(a, b, c), + 105 => mask_ternlog_popcount::<105>(a, b, c), + 106 => mask_ternlog_popcount::<106>(a, b, c), + 107 => mask_ternlog_popcount::<107>(a, b, c), + 108 => mask_ternlog_popcount::<108>(a, b, c), + 109 => mask_ternlog_popcount::<109>(a, b, c), + 110 => mask_ternlog_popcount::<110>(a, b, c), + 111 => mask_ternlog_popcount::<111>(a, b, c), + 112 => mask_ternlog_popcount::<112>(a, b, c), + 113 => mask_ternlog_popcount::<113>(a, b, c), + 114 => mask_ternlog_popcount::<114>(a, b, c), + 115 => mask_ternlog_popcount::<115>(a, b, c), + 116 => mask_ternlog_popcount::<116>(a, b, c), + 117 => mask_ternlog_popcount::<117>(a, b, c), + 118 => mask_ternlog_popcount::<118>(a, b, c), + 119 => mask_ternlog_popcount::<119>(a, b, c), + 120 => mask_ternlog_popcount::<120>(a, b, c), + 121 => mask_ternlog_popcount::<121>(a, b, c), + 122 => mask_ternlog_popcount::<122>(a, b, c), + 123 => mask_ternlog_popcount::<123>(a, b, c), + 124 => mask_ternlog_popcount::<124>(a, b, c), + 125 => mask_ternlog_popcount::<125>(a, b, c), + 126 => mask_ternlog_popcount::<126>(a, b, c), + 127 => mask_ternlog_popcount::<127>(a, b, c), + 128 => mask_ternlog_popcount::<128>(a, b, c), + 129 => mask_ternlog_popcount::<129>(a, b, c), + 130 => mask_ternlog_popcount::<130>(a, b, c), + 131 => mask_ternlog_popcount::<131>(a, b, c), + 132 => mask_ternlog_popcount::<132>(a, b, c), + 133 => mask_ternlog_popcount::<133>(a, b, c), + 134 => mask_ternlog_popcount::<134>(a, b, c), + 135 => mask_ternlog_popcount::<135>(a, b, c), + 136 => mask_ternlog_popcount::<136>(a, b, c), + 137 => mask_ternlog_popcount::<137>(a, b, c), + 138 => mask_ternlog_popcount::<138>(a, b, c), + 139 => mask_ternlog_popcount::<139>(a, b, c), + 140 => mask_ternlog_popcount::<140>(a, b, c), + 141 => mask_ternlog_popcount::<141>(a, b, c), + 142 => mask_ternlog_popcount::<142>(a, b, c), + 143 => mask_ternlog_popcount::<143>(a, b, c), + 144 => mask_ternlog_popcount::<144>(a, b, c), + 145 => mask_ternlog_popcount::<145>(a, b, c), + 146 => mask_ternlog_popcount::<146>(a, b, c), + 147 => mask_ternlog_popcount::<147>(a, b, c), + 148 => mask_ternlog_popcount::<148>(a, b, c), + 149 => mask_ternlog_popcount::<149>(a, b, c), + 150 => mask_ternlog_popcount::<150>(a, b, c), + 151 => mask_ternlog_popcount::<151>(a, b, c), + 152 => mask_ternlog_popcount::<152>(a, b, c), + 153 => mask_ternlog_popcount::<153>(a, b, c), + 154 => mask_ternlog_popcount::<154>(a, b, c), + 155 => mask_ternlog_popcount::<155>(a, b, c), + 156 => mask_ternlog_popcount::<156>(a, b, c), + 157 => mask_ternlog_popcount::<157>(a, b, c), + 158 => mask_ternlog_popcount::<158>(a, b, c), + 159 => mask_ternlog_popcount::<159>(a, b, c), + 160 => mask_ternlog_popcount::<160>(a, b, c), + 161 => mask_ternlog_popcount::<161>(a, b, c), + 162 => mask_ternlog_popcount::<162>(a, b, c), + 163 => mask_ternlog_popcount::<163>(a, b, c), + 164 => mask_ternlog_popcount::<164>(a, b, c), + 165 => mask_ternlog_popcount::<165>(a, b, c), + 166 => mask_ternlog_popcount::<166>(a, b, c), + 167 => mask_ternlog_popcount::<167>(a, b, c), + 168 => mask_ternlog_popcount::<168>(a, b, c), + 169 => mask_ternlog_popcount::<169>(a, b, c), + 170 => mask_ternlog_popcount::<170>(a, b, c), + 171 => mask_ternlog_popcount::<171>(a, b, c), + 172 => mask_ternlog_popcount::<172>(a, b, c), + 173 => mask_ternlog_popcount::<173>(a, b, c), + 174 => mask_ternlog_popcount::<174>(a, b, c), + 175 => mask_ternlog_popcount::<175>(a, b, c), + 176 => mask_ternlog_popcount::<176>(a, b, c), + 177 => mask_ternlog_popcount::<177>(a, b, c), + 178 => mask_ternlog_popcount::<178>(a, b, c), + 179 => mask_ternlog_popcount::<179>(a, b, c), + 180 => mask_ternlog_popcount::<180>(a, b, c), + 181 => mask_ternlog_popcount::<181>(a, b, c), + 182 => mask_ternlog_popcount::<182>(a, b, c), + 183 => mask_ternlog_popcount::<183>(a, b, c), + 184 => mask_ternlog_popcount::<184>(a, b, c), + 185 => mask_ternlog_popcount::<185>(a, b, c), + 186 => mask_ternlog_popcount::<186>(a, b, c), + 187 => mask_ternlog_popcount::<187>(a, b, c), + 188 => mask_ternlog_popcount::<188>(a, b, c), + 189 => mask_ternlog_popcount::<189>(a, b, c), + 190 => mask_ternlog_popcount::<190>(a, b, c), + 191 => mask_ternlog_popcount::<191>(a, b, c), + 192 => mask_ternlog_popcount::<192>(a, b, c), + 193 => mask_ternlog_popcount::<193>(a, b, c), + 194 => mask_ternlog_popcount::<194>(a, b, c), + 195 => mask_ternlog_popcount::<195>(a, b, c), + 196 => mask_ternlog_popcount::<196>(a, b, c), + 197 => mask_ternlog_popcount::<197>(a, b, c), + 198 => mask_ternlog_popcount::<198>(a, b, c), + 199 => mask_ternlog_popcount::<199>(a, b, c), + 200 => mask_ternlog_popcount::<200>(a, b, c), + 201 => mask_ternlog_popcount::<201>(a, b, c), + 202 => mask_ternlog_popcount::<202>(a, b, c), + 203 => mask_ternlog_popcount::<203>(a, b, c), + 204 => mask_ternlog_popcount::<204>(a, b, c), + 205 => mask_ternlog_popcount::<205>(a, b, c), + 206 => mask_ternlog_popcount::<206>(a, b, c), + 207 => mask_ternlog_popcount::<207>(a, b, c), + 208 => mask_ternlog_popcount::<208>(a, b, c), + 209 => mask_ternlog_popcount::<209>(a, b, c), + 210 => mask_ternlog_popcount::<210>(a, b, c), + 211 => mask_ternlog_popcount::<211>(a, b, c), + 212 => mask_ternlog_popcount::<212>(a, b, c), + 213 => mask_ternlog_popcount::<213>(a, b, c), + 214 => mask_ternlog_popcount::<214>(a, b, c), + 215 => mask_ternlog_popcount::<215>(a, b, c), + 216 => mask_ternlog_popcount::<216>(a, b, c), + 217 => mask_ternlog_popcount::<217>(a, b, c), + 218 => mask_ternlog_popcount::<218>(a, b, c), + 219 => mask_ternlog_popcount::<219>(a, b, c), + 220 => mask_ternlog_popcount::<220>(a, b, c), + 221 => mask_ternlog_popcount::<221>(a, b, c), + 222 => mask_ternlog_popcount::<222>(a, b, c), + 223 => mask_ternlog_popcount::<223>(a, b, c), + 224 => mask_ternlog_popcount::<224>(a, b, c), + 225 => mask_ternlog_popcount::<225>(a, b, c), + 226 => mask_ternlog_popcount::<226>(a, b, c), + 227 => mask_ternlog_popcount::<227>(a, b, c), + 228 => mask_ternlog_popcount::<228>(a, b, c), + 229 => mask_ternlog_popcount::<229>(a, b, c), + 230 => mask_ternlog_popcount::<230>(a, b, c), + 231 => mask_ternlog_popcount::<231>(a, b, c), + 232 => mask_ternlog_popcount::<232>(a, b, c), + 233 => mask_ternlog_popcount::<233>(a, b, c), + 234 => mask_ternlog_popcount::<234>(a, b, c), + 235 => mask_ternlog_popcount::<235>(a, b, c), + 236 => mask_ternlog_popcount::<236>(a, b, c), + 237 => mask_ternlog_popcount::<237>(a, b, c), + 238 => mask_ternlog_popcount::<238>(a, b, c), + 239 => mask_ternlog_popcount::<239>(a, b, c), + 240 => mask_ternlog_popcount::<240>(a, b, c), + 241 => mask_ternlog_popcount::<241>(a, b, c), + 242 => mask_ternlog_popcount::<242>(a, b, c), + 243 => mask_ternlog_popcount::<243>(a, b, c), + 244 => mask_ternlog_popcount::<244>(a, b, c), + 245 => mask_ternlog_popcount::<245>(a, b, c), + 246 => mask_ternlog_popcount::<246>(a, b, c), + 247 => mask_ternlog_popcount::<247>(a, b, c), + 248 => mask_ternlog_popcount::<248>(a, b, c), + 249 => mask_ternlog_popcount::<249>(a, b, c), + 250 => mask_ternlog_popcount::<250>(a, b, c), + 251 => mask_ternlog_popcount::<251>(a, b, c), + 252 => mask_ternlog_popcount::<252>(a, b, c), + 253 => mask_ternlog_popcount::<253>(a, b, c), + 254 => mask_ternlog_popcount::<254>(a, b, c), + 255 => mask_ternlog_popcount::<255>(a, b, c), + } +} + +/// `table[imm](a, b, c) != 0` anywhere — the no-mask Any fold, routed like the others. +pub fn ternlog_any_dispatch(imm: u8, a: &[u64], b: &[u64], c: &[u64]) -> bool { + match imm { + 0 => mask_ternlog_any::<0>(a, b, c), + 1 => mask_ternlog_any::<1>(a, b, c), + 2 => mask_ternlog_any::<2>(a, b, c), + 3 => mask_ternlog_any::<3>(a, b, c), + 4 => mask_ternlog_any::<4>(a, b, c), + 5 => mask_ternlog_any::<5>(a, b, c), + 6 => mask_ternlog_any::<6>(a, b, c), + 7 => mask_ternlog_any::<7>(a, b, c), + 8 => mask_ternlog_any::<8>(a, b, c), + 9 => mask_ternlog_any::<9>(a, b, c), + 10 => mask_ternlog_any::<10>(a, b, c), + 11 => mask_ternlog_any::<11>(a, b, c), + 12 => mask_ternlog_any::<12>(a, b, c), + 13 => mask_ternlog_any::<13>(a, b, c), + 14 => mask_ternlog_any::<14>(a, b, c), + 15 => mask_ternlog_any::<15>(a, b, c), + 16 => mask_ternlog_any::<16>(a, b, c), + 17 => mask_ternlog_any::<17>(a, b, c), + 18 => mask_ternlog_any::<18>(a, b, c), + 19 => mask_ternlog_any::<19>(a, b, c), + 20 => mask_ternlog_any::<20>(a, b, c), + 21 => mask_ternlog_any::<21>(a, b, c), + 22 => mask_ternlog_any::<22>(a, b, c), + 23 => mask_ternlog_any::<23>(a, b, c), + 24 => mask_ternlog_any::<24>(a, b, c), + 25 => mask_ternlog_any::<25>(a, b, c), + 26 => mask_ternlog_any::<26>(a, b, c), + 27 => mask_ternlog_any::<27>(a, b, c), + 28 => mask_ternlog_any::<28>(a, b, c), + 29 => mask_ternlog_any::<29>(a, b, c), + 30 => mask_ternlog_any::<30>(a, b, c), + 31 => mask_ternlog_any::<31>(a, b, c), + 32 => mask_ternlog_any::<32>(a, b, c), + 33 => mask_ternlog_any::<33>(a, b, c), + 34 => mask_ternlog_any::<34>(a, b, c), + 35 => mask_ternlog_any::<35>(a, b, c), + 36 => mask_ternlog_any::<36>(a, b, c), + 37 => mask_ternlog_any::<37>(a, b, c), + 38 => mask_ternlog_any::<38>(a, b, c), + 39 => mask_ternlog_any::<39>(a, b, c), + 40 => mask_ternlog_any::<40>(a, b, c), + 41 => mask_ternlog_any::<41>(a, b, c), + 42 => mask_ternlog_any::<42>(a, b, c), + 43 => mask_ternlog_any::<43>(a, b, c), + 44 => mask_ternlog_any::<44>(a, b, c), + 45 => mask_ternlog_any::<45>(a, b, c), + 46 => mask_ternlog_any::<46>(a, b, c), + 47 => mask_ternlog_any::<47>(a, b, c), + 48 => mask_ternlog_any::<48>(a, b, c), + 49 => mask_ternlog_any::<49>(a, b, c), + 50 => mask_ternlog_any::<50>(a, b, c), + 51 => mask_ternlog_any::<51>(a, b, c), + 52 => mask_ternlog_any::<52>(a, b, c), + 53 => mask_ternlog_any::<53>(a, b, c), + 54 => mask_ternlog_any::<54>(a, b, c), + 55 => mask_ternlog_any::<55>(a, b, c), + 56 => mask_ternlog_any::<56>(a, b, c), + 57 => mask_ternlog_any::<57>(a, b, c), + 58 => mask_ternlog_any::<58>(a, b, c), + 59 => mask_ternlog_any::<59>(a, b, c), + 60 => mask_ternlog_any::<60>(a, b, c), + 61 => mask_ternlog_any::<61>(a, b, c), + 62 => mask_ternlog_any::<62>(a, b, c), + 63 => mask_ternlog_any::<63>(a, b, c), + 64 => mask_ternlog_any::<64>(a, b, c), + 65 => mask_ternlog_any::<65>(a, b, c), + 66 => mask_ternlog_any::<66>(a, b, c), + 67 => mask_ternlog_any::<67>(a, b, c), + 68 => mask_ternlog_any::<68>(a, b, c), + 69 => mask_ternlog_any::<69>(a, b, c), + 70 => mask_ternlog_any::<70>(a, b, c), + 71 => mask_ternlog_any::<71>(a, b, c), + 72 => mask_ternlog_any::<72>(a, b, c), + 73 => mask_ternlog_any::<73>(a, b, c), + 74 => mask_ternlog_any::<74>(a, b, c), + 75 => mask_ternlog_any::<75>(a, b, c), + 76 => mask_ternlog_any::<76>(a, b, c), + 77 => mask_ternlog_any::<77>(a, b, c), + 78 => mask_ternlog_any::<78>(a, b, c), + 79 => mask_ternlog_any::<79>(a, b, c), + 80 => mask_ternlog_any::<80>(a, b, c), + 81 => mask_ternlog_any::<81>(a, b, c), + 82 => mask_ternlog_any::<82>(a, b, c), + 83 => mask_ternlog_any::<83>(a, b, c), + 84 => mask_ternlog_any::<84>(a, b, c), + 85 => mask_ternlog_any::<85>(a, b, c), + 86 => mask_ternlog_any::<86>(a, b, c), + 87 => mask_ternlog_any::<87>(a, b, c), + 88 => mask_ternlog_any::<88>(a, b, c), + 89 => mask_ternlog_any::<89>(a, b, c), + 90 => mask_ternlog_any::<90>(a, b, c), + 91 => mask_ternlog_any::<91>(a, b, c), + 92 => mask_ternlog_any::<92>(a, b, c), + 93 => mask_ternlog_any::<93>(a, b, c), + 94 => mask_ternlog_any::<94>(a, b, c), + 95 => mask_ternlog_any::<95>(a, b, c), + 96 => mask_ternlog_any::<96>(a, b, c), + 97 => mask_ternlog_any::<97>(a, b, c), + 98 => mask_ternlog_any::<98>(a, b, c), + 99 => mask_ternlog_any::<99>(a, b, c), + 100 => mask_ternlog_any::<100>(a, b, c), + 101 => mask_ternlog_any::<101>(a, b, c), + 102 => mask_ternlog_any::<102>(a, b, c), + 103 => mask_ternlog_any::<103>(a, b, c), + 104 => mask_ternlog_any::<104>(a, b, c), + 105 => mask_ternlog_any::<105>(a, b, c), + 106 => mask_ternlog_any::<106>(a, b, c), + 107 => mask_ternlog_any::<107>(a, b, c), + 108 => mask_ternlog_any::<108>(a, b, c), + 109 => mask_ternlog_any::<109>(a, b, c), + 110 => mask_ternlog_any::<110>(a, b, c), + 111 => mask_ternlog_any::<111>(a, b, c), + 112 => mask_ternlog_any::<112>(a, b, c), + 113 => mask_ternlog_any::<113>(a, b, c), + 114 => mask_ternlog_any::<114>(a, b, c), + 115 => mask_ternlog_any::<115>(a, b, c), + 116 => mask_ternlog_any::<116>(a, b, c), + 117 => mask_ternlog_any::<117>(a, b, c), + 118 => mask_ternlog_any::<118>(a, b, c), + 119 => mask_ternlog_any::<119>(a, b, c), + 120 => mask_ternlog_any::<120>(a, b, c), + 121 => mask_ternlog_any::<121>(a, b, c), + 122 => mask_ternlog_any::<122>(a, b, c), + 123 => mask_ternlog_any::<123>(a, b, c), + 124 => mask_ternlog_any::<124>(a, b, c), + 125 => mask_ternlog_any::<125>(a, b, c), + 126 => mask_ternlog_any::<126>(a, b, c), + 127 => mask_ternlog_any::<127>(a, b, c), + 128 => mask_ternlog_any::<128>(a, b, c), + 129 => mask_ternlog_any::<129>(a, b, c), + 130 => mask_ternlog_any::<130>(a, b, c), + 131 => mask_ternlog_any::<131>(a, b, c), + 132 => mask_ternlog_any::<132>(a, b, c), + 133 => mask_ternlog_any::<133>(a, b, c), + 134 => mask_ternlog_any::<134>(a, b, c), + 135 => mask_ternlog_any::<135>(a, b, c), + 136 => mask_ternlog_any::<136>(a, b, c), + 137 => mask_ternlog_any::<137>(a, b, c), + 138 => mask_ternlog_any::<138>(a, b, c), + 139 => mask_ternlog_any::<139>(a, b, c), + 140 => mask_ternlog_any::<140>(a, b, c), + 141 => mask_ternlog_any::<141>(a, b, c), + 142 => mask_ternlog_any::<142>(a, b, c), + 143 => mask_ternlog_any::<143>(a, b, c), + 144 => mask_ternlog_any::<144>(a, b, c), + 145 => mask_ternlog_any::<145>(a, b, c), + 146 => mask_ternlog_any::<146>(a, b, c), + 147 => mask_ternlog_any::<147>(a, b, c), + 148 => mask_ternlog_any::<148>(a, b, c), + 149 => mask_ternlog_any::<149>(a, b, c), + 150 => mask_ternlog_any::<150>(a, b, c), + 151 => mask_ternlog_any::<151>(a, b, c), + 152 => mask_ternlog_any::<152>(a, b, c), + 153 => mask_ternlog_any::<153>(a, b, c), + 154 => mask_ternlog_any::<154>(a, b, c), + 155 => mask_ternlog_any::<155>(a, b, c), + 156 => mask_ternlog_any::<156>(a, b, c), + 157 => mask_ternlog_any::<157>(a, b, c), + 158 => mask_ternlog_any::<158>(a, b, c), + 159 => mask_ternlog_any::<159>(a, b, c), + 160 => mask_ternlog_any::<160>(a, b, c), + 161 => mask_ternlog_any::<161>(a, b, c), + 162 => mask_ternlog_any::<162>(a, b, c), + 163 => mask_ternlog_any::<163>(a, b, c), + 164 => mask_ternlog_any::<164>(a, b, c), + 165 => mask_ternlog_any::<165>(a, b, c), + 166 => mask_ternlog_any::<166>(a, b, c), + 167 => mask_ternlog_any::<167>(a, b, c), + 168 => mask_ternlog_any::<168>(a, b, c), + 169 => mask_ternlog_any::<169>(a, b, c), + 170 => mask_ternlog_any::<170>(a, b, c), + 171 => mask_ternlog_any::<171>(a, b, c), + 172 => mask_ternlog_any::<172>(a, b, c), + 173 => mask_ternlog_any::<173>(a, b, c), + 174 => mask_ternlog_any::<174>(a, b, c), + 175 => mask_ternlog_any::<175>(a, b, c), + 176 => mask_ternlog_any::<176>(a, b, c), + 177 => mask_ternlog_any::<177>(a, b, c), + 178 => mask_ternlog_any::<178>(a, b, c), + 179 => mask_ternlog_any::<179>(a, b, c), + 180 => mask_ternlog_any::<180>(a, b, c), + 181 => mask_ternlog_any::<181>(a, b, c), + 182 => mask_ternlog_any::<182>(a, b, c), + 183 => mask_ternlog_any::<183>(a, b, c), + 184 => mask_ternlog_any::<184>(a, b, c), + 185 => mask_ternlog_any::<185>(a, b, c), + 186 => mask_ternlog_any::<186>(a, b, c), + 187 => mask_ternlog_any::<187>(a, b, c), + 188 => mask_ternlog_any::<188>(a, b, c), + 189 => mask_ternlog_any::<189>(a, b, c), + 190 => mask_ternlog_any::<190>(a, b, c), + 191 => mask_ternlog_any::<191>(a, b, c), + 192 => mask_ternlog_any::<192>(a, b, c), + 193 => mask_ternlog_any::<193>(a, b, c), + 194 => mask_ternlog_any::<194>(a, b, c), + 195 => mask_ternlog_any::<195>(a, b, c), + 196 => mask_ternlog_any::<196>(a, b, c), + 197 => mask_ternlog_any::<197>(a, b, c), + 198 => mask_ternlog_any::<198>(a, b, c), + 199 => mask_ternlog_any::<199>(a, b, c), + 200 => mask_ternlog_any::<200>(a, b, c), + 201 => mask_ternlog_any::<201>(a, b, c), + 202 => mask_ternlog_any::<202>(a, b, c), + 203 => mask_ternlog_any::<203>(a, b, c), + 204 => mask_ternlog_any::<204>(a, b, c), + 205 => mask_ternlog_any::<205>(a, b, c), + 206 => mask_ternlog_any::<206>(a, b, c), + 207 => mask_ternlog_any::<207>(a, b, c), + 208 => mask_ternlog_any::<208>(a, b, c), + 209 => mask_ternlog_any::<209>(a, b, c), + 210 => mask_ternlog_any::<210>(a, b, c), + 211 => mask_ternlog_any::<211>(a, b, c), + 212 => mask_ternlog_any::<212>(a, b, c), + 213 => mask_ternlog_any::<213>(a, b, c), + 214 => mask_ternlog_any::<214>(a, b, c), + 215 => mask_ternlog_any::<215>(a, b, c), + 216 => mask_ternlog_any::<216>(a, b, c), + 217 => mask_ternlog_any::<217>(a, b, c), + 218 => mask_ternlog_any::<218>(a, b, c), + 219 => mask_ternlog_any::<219>(a, b, c), + 220 => mask_ternlog_any::<220>(a, b, c), + 221 => mask_ternlog_any::<221>(a, b, c), + 222 => mask_ternlog_any::<222>(a, b, c), + 223 => mask_ternlog_any::<223>(a, b, c), + 224 => mask_ternlog_any::<224>(a, b, c), + 225 => mask_ternlog_any::<225>(a, b, c), + 226 => mask_ternlog_any::<226>(a, b, c), + 227 => mask_ternlog_any::<227>(a, b, c), + 228 => mask_ternlog_any::<228>(a, b, c), + 229 => mask_ternlog_any::<229>(a, b, c), + 230 => mask_ternlog_any::<230>(a, b, c), + 231 => mask_ternlog_any::<231>(a, b, c), + 232 => mask_ternlog_any::<232>(a, b, c), + 233 => mask_ternlog_any::<233>(a, b, c), + 234 => mask_ternlog_any::<234>(a, b, c), + 235 => mask_ternlog_any::<235>(a, b, c), + 236 => mask_ternlog_any::<236>(a, b, c), + 237 => mask_ternlog_any::<237>(a, b, c), + 238 => mask_ternlog_any::<238>(a, b, c), + 239 => mask_ternlog_any::<239>(a, b, c), + 240 => mask_ternlog_any::<240>(a, b, c), + 241 => mask_ternlog_any::<241>(a, b, c), + 242 => mask_ternlog_any::<242>(a, b, c), + 243 => mask_ternlog_any::<243>(a, b, c), + 244 => mask_ternlog_any::<244>(a, b, c), + 245 => mask_ternlog_any::<245>(a, b, c), + 246 => mask_ternlog_any::<246>(a, b, c), + 247 => mask_ternlog_any::<247>(a, b, c), + 248 => mask_ternlog_any::<248>(a, b, c), + 249 => mask_ternlog_any::<249>(a, b, c), + 250 => mask_ternlog_any::<250>(a, b, c), + 251 => mask_ternlog_any::<251>(a, b, c), + 252 => mask_ternlog_any::<252>(a, b, c), + 253 => mask_ternlog_any::<253>(a, b, c), + 254 => mask_ternlog_any::<254>(a, b, c), + 255 => mask_ternlog_any::<255>(a, b, c), + } +} // GEN-TERNLOG-DISPATCH-END #[cfg(test)] @@ -754,5 +1278,52 @@ mod tests { assign_ns, expected, "ternlog_dispatch_assign arms missing, duplicated, or reordered" ); + for (prefix, name) in [ + ("=> mask_ternlog_popcount::<", "ternlog_popcount_dispatch"), + ("=> mask_ternlog_any::<", "ternlog_any_dispatch"), + ] { + let ns: Vec = region_lines + .iter() + .filter_map(|line| extract_imm(line, prefix)) + .collect(); + assert_eq!( + ns, expected, + "{name} arms missing, duplicated, or reordered" + ); + } + } + + /// FAILS IF: either no-mask fold disagrees, for ANY immediate, with the + /// materializing pair it replaces — `ternlog_dispatch` into a buffer, then + /// the popcount / any of that buffer. A fold arm routed to the wrong table + /// would count a different function; the distinct-count guard below keeps + /// the fixture from being too uniform to tell tables apart. + #[test] + fn the_no_mask_folds_match_dispatch_then_reduce_for_every_immediate() { + let (a, b, c) = seeded_triple(0xF01D_0D15_BA7C_0001, 11); + let mut distinct = HashSet::new(); + for imm in 0u8..=255u8 { + let mut dst = vec![0u64; a.len()]; + ternlog_dispatch(imm, &a, &b, &c, &mut dst); + let count: u64 = dst.iter().map(|w| u64::from(w.count_ones())).sum(); + assert_eq!( + ternlog_popcount_dispatch(imm, &a, &b, &c), + count, + "ternlog_popcount_dispatch mismatch at imm={imm:#04x}" + ); + assert_eq!( + ternlog_any_dispatch(imm, &a, &b, &c), + dst.iter().any(|&w| w != 0), + "ternlog_any_dispatch mismatch at imm={imm:#04x}" + ); + distinct.insert(count); + } + assert!( + distinct.len() >= 100, + "only {} distinct counts", + distinct.len() + ); + // Any's `false` arm is reachable: the constant-zero table on any input. + assert!(!ternlog_any_dispatch(0x00, &a, &b, &c)); } } diff --git a/crates/lance-graph-mask-risc/tests/fused_ternlog.rs b/crates/lance-graph-mask-risc/tests/fused_ternlog.rs new file mode 100644 index 000000000..6f97e629f --- /dev/null +++ b/crates/lance-graph-mask-risc/tests/fused_ternlog.rs @@ -0,0 +1,482 @@ +//! A Boolean membership over resident planes ends in Count/Any with no mask. +//! +//! `MaskOp::{And, Or, Xor, AndNot, Ternlog}` over `Operand::Plane`s, folded by +//! `Count` or `Any` of its own `dst`, is one truth table. The executor folds it +//! with `ndarray::simd::mask_ternlog_{popcount,any}` over the borrowed planes' +//! words, and combines the (at most two) words an extent cuts in a one-word +//! register. No scratch slot is carved and no membership word is written. +//! +//! Every case checks the FOLD against the MATERIALIZE arm (the same op with a +//! `Keep` terminal, which is never fused) and against a bit-serial scalar +//! oracle over the same absolute rows. + +use std::alloc::{GlobalAlloc, Layout, System}; +use std::cell::Cell; + +use lance_graph_mask_risc::exec::{execute_extent, Scratch}; +use lance_graph_mask_risc::{ + Foreign, FusedFold, MaskOp, Operand, Out, Planes, Program, Terminal, Value, +}; + +struct Counting; + +thread_local! { + static BYTES: Cell = const { Cell::new(0) }; +} + +fn bytes() -> usize { + BYTES.with(Cell::get) +} + +// SAFETY: a pure pass-through to `System`; the counter is the only addition. +unsafe impl GlobalAlloc for Counting { + unsafe fn alloc(&self, layout: Layout) -> *mut u8 { + let _ = BYTES.try_with(|b| b.set(b.get() + layout.size())); + // SAFETY: same layout, same contract as the caller's. + unsafe { System.alloc(layout) } + } + unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) { + // SAFETY: `ptr` came from `alloc` above with this `layout`. + unsafe { System.dealloc(ptr, layout) } + } +} + +#[global_allocator] +static A: Counting = Counting; + +fn lcg(seed: &mut u64) -> u64 { + *seed = seed + .wrapping_mul(6364136223846793005) + .wrapping_add(1442695040888963407); + *seed >> 11 +} + +fn words_for(n: usize) -> usize { + n.div_ceil(64) +} + +/// A conforming plane of `n` rows (tail bits clear). +fn plane(n: usize, set: impl Fn(usize) -> bool) -> Vec { + let mut w = vec![0u64; words_for(n)]; + for r in 0..n { + if set(r) { + w[r / 64] |= 1u64 << (r % 64); + } + } + w +} + +fn bit(p: &[u64], r: usize) -> u8 { + (p[r / 64] >> (r % 64) & 1) as u8 +} + +/// The shapes under test: each op kind with its truth table (the ORACLE's +/// spelling — independent of the crate's own constants). +#[derive(Clone, Copy, Debug)] +enum Shape { + And, + Or, + Xor, + AndNot, + Ternlog(u8), +} + +impl Shape { + fn table(self) -> u8 { + match self { + Shape::Ternlog(t) => t, + // Evaluate the two-input function on each (a, b, c) index. + s => (0u8..8).fold(0u8, |acc, i| { + let (a, b) = (i >> 2 & 1, i >> 1 & 1); + let v = match s { + Shape::And => a & b, + Shape::Or => a | b, + Shape::Xor => a ^ b, + Shape::AndNot => a & (1 - b), + Shape::Ternlog(_) => unreachable!(), + }; + acc | (v << i) + }), + } + } + + fn op(self, dst: u16) -> MaskOp { + let (a, b, c) = (Operand::Plane(0), Operand::Plane(1), Operand::Plane(2)); + match self { + Shape::And => MaskOp::And { a, b, dst }, + Shape::Or => MaskOp::Or { a, b, dst }, + Shape::Xor => MaskOp::Xor { a, b, dst }, + Shape::AndNot => MaskOp::AndNot { a, b, dst }, + Shape::Ternlog(imm) => MaskOp::Ternlog { imm, a, b, c, dst }, + } + } +} + +fn program(s: Shape, terminal: Terminal) -> Program { + Program::new(vec![s.op(0)], terminal) +} + +fn count_p(s: Shape) -> Program { + program( + s, + Terminal::Count { + mask: Operand::Scratch(0), + }, + ) +} + +fn any_p(s: Shape) -> Program { + program( + s, + Terminal::Any { + mask: Operand::Scratch(0), + }, + ) +} + +/// Bit-serial oracle over the ABSOLUTE rows `[lo, hi)`. +fn oracle(s: Shape, m: &[Vec; 3], lo: usize, hi: usize) -> usize { + let t = s.table(); + // Two-input ops read `b` as their third input, matching how the op is + // defined (the table ignores `c`, so either choice is exact). + (lo..hi) + .filter(|&r| { + let idx = bit(&m[0], r) << 2 | bit(&m[1], r) << 1 | bit(&m[2], r); + t >> idx & 1 == 1 + }) + .count() +} + +fn fold(p: &Program, planes: &Planes<'_>, lo: usize, hi: usize) -> Value { + let mut s = Scratch::new(0, 0); + execute_extent(p, planes, &Foreign::NONE, &mut s, Out::None, lo..hi).expect("fold arm") +} + +/// The MATERIALIZE arm: the same op kept as a mask over the extent, counted. +fn keep_count(s: Shape, planes: &Planes<'_>, lo: usize, hi: usize) -> usize { + let p = program( + s, + Terminal::Keep { + mask: Operand::Scratch(0), + }, + ); + assert!(p.fused_ternlog().is_none(), "Keep is never fused"); + let mut sc = Scratch::for_program(&p, planes.n_rows).expect("scratch"); + let mut out = vec![0u64; words_for(planes.n_rows)]; + execute_extent( + &p, + planes, + &Foreign::NONE, + &mut sc, + Out::Mask(&mut out), + lo..hi, + ) + .expect("keep arm"); + out.iter().map(|w| w.count_ones() as usize).sum() +} + +fn extents(n: usize) -> Vec<(usize, usize)> { + let mut v = vec![(0, n), (0, 0), (n, n)]; + for k in [1usize, 63, 64, 65, 127, 128, 129] { + if k <= n { + v.push((k, n)); + v.push((0, k)); + } + } + if n >= 3 { + v.push((n / 3, 2 * n / 3 + 1)); + v.push((n / 2, n / 2 + 1)); + } + v.retain(|&(lo, hi)| lo <= hi && hi <= n); + v +} + +fn planes_for(n: usize, seed: u64) -> Vec<[Vec; 3]> { + let mut s = seed; + let rnd = |s: &mut u64, dense: bool| { + let bits: Vec = (0..n) + .map(|_| { + let x = lcg(s); + if dense { + x & 1 == 1 + } else { + x.is_multiple_of(13) + } + }) + .collect(); + plane(n, |r| bits[r]) + }; + vec![ + [rnd(&mut s, true), rnd(&mut s, true), rnd(&mut s, true)], + [rnd(&mut s, false), rnd(&mut s, false), rnd(&mut s, true)], + [ + plane(n, |_| false), + plane(n, |_| false), + plane(n, |_| false), + ], + [plane(n, |_| true), plane(n, |_| true), plane(n, |_| false)], + ] +} + +fn check(s: Shape, n: usize, m: &[Vec; 3]) { + let masks: [&[u64]; 3] = [&m[0], &m[1], &m[2]]; + let planes = Planes { + n_rows: n, + masks: &masks, + lanes: &[], + }; + let (cp, ap) = (count_p(s), any_p(s)); + assert_eq!( + cp.fused_ternlog().map(|f| f.fold), + Some(FusedFold::Count), + "{s:?}" + ); + assert_eq!( + ap.fused_ternlog().map(|f| f.fold), + Some(FusedFold::Any), + "{s:?}" + ); + for (lo, hi) in extents(n) { + let want = oracle(s, m, lo, hi); + let kept = keep_count(s, &planes, lo, hi); + assert_eq!(kept, want, "keep arm vs oracle {s:?} n={n} [{lo},{hi})"); + assert_eq!( + fold(&cp, &planes, lo, hi), + Value::Count(want), + "count {s:?} n={n} [{lo},{hi})" + ); + assert_eq!( + fold(&ap, &planes, lo, hi), + Value::Bool(want > 0), + "any {s:?} n={n} [{lo},{hi})" + ); + } +} + +/// FAILS IF: the fold disagrees with the kept mask or the scalar oracle for any +/// op kind × population × plane shape × absolute extent — including an empty +/// extent, one row, cuts at 63/64/65/127/128/129, a sub-64-row population tail +/// and the whole population. +#[test] +fn the_fold_agrees_with_keep_and_the_oracle_for_every_two_input_op() { + for n in [1usize, 63, 64, 65, 133, 1317, 4133] { + for m in planes_for(n, 0xF0_1D ^ n as u64) { + for s in [Shape::And, Shape::Or, Shape::Xor, Shape::AndNot] { + check(s, n, &m); + } + } + } +} + +/// FAILS IF: any of the 256 truth tables folds wrongly — an odd table (true of +/// all-zero inputs) in particular, whose dead population-tail bits must never +/// be counted. +#[test] +fn the_fold_agrees_for_all_256_tables() { + for n in [65usize, 133] { + for m in planes_for(n, 0x7AB1E ^ n as u64) { + for imm in 0u8..=255 { + check(Shape::Ternlog(imm), n, &m); + } + } + } +} + +/// FAILS IF: an odd table's dead tail bits leak into the count. NOR over +/// all-zero planes is true on every LIVE row: 70 rows, never 128. +#[test] +fn an_odd_table_never_counts_the_population_tail() { + for n in [1usize, 63, 64, 65, 70, 1000] { + let z = plane(n, |_| false); + let masks: [&[u64]; 3] = [&z, &z, &z]; + let planes = Planes { + n_rows: n, + masks: &masks, + lanes: &[], + }; + let nor = Shape::Ternlog(0x01); + assert_eq!(fold(&count_p(nor), &planes, 0, n), Value::Count(n), "n={n}"); + assert_eq!(fold(&any_p(nor), &planes, 0, n), Value::Bool(true), "n={n}"); + // A partial extent ending short of the tail counts its own rows only. + assert_eq!( + fold(&count_p(nor), &planes, 0, n / 2), + Value::Count(n / 2), + "n={n}" + ); + } +} + +/// FAILS IF: the fold arm materializes anything — carves a slot, writes a +/// scratch word, or allocates — or needs a scratch at all. +#[test] +fn the_fold_writes_no_membership_and_allocates_nothing() { + let n = 4133; + let m = &planes_for(n, 0xA11C)[0]; + let masks: [&[u64]; 3] = [&m[0], &m[1], &m[2]]; + let planes = Planes { + n_rows: n, + masks: &masks, + lanes: &[], + }; + for s in [ + Shape::And, + Shape::Xor, + Shape::Ternlog(0xEA), + Shape::Ternlog(0x01), + ] { + for p in [count_p(s), any_p(s)] { + assert!(!p.requires_scratch(), "{s:?} must need no scratch"); + // A poisoned arena. `over_for_program` zero-fills whatever it + // carves, so any carved slot — and any derived word written into + // one — shows up as a changed word. + let mut poison = vec![u64::MAX; 4 * words_for(n)]; + { + let mut sc = Scratch::over_for_program(&mut poison, &p, n).expect("arena"); + let before = bytes(); + let v = execute_extent(&p, &planes, &Foreign::NONE, &mut sc, Out::None, 7..n - 3) + .expect("fold"); + assert_eq!(bytes(), before, "{s:?} allocated during the fold"); + assert!(matches!(v, Value::Count(_) | Value::Bool(_))); + } + assert!( + poison.iter().all(|&w| w == u64::MAX), + "{s:?} wrote into scratch" + ); + } + } + // Twin half: the probe CAN see a carve — the same op kept as a mask + // carves a slot out of the same kind of arena. + let keep = program( + Shape::And, + Terminal::Keep { + mask: Operand::Scratch(0), + }, + ); + let mut poison = vec![u64::MAX; 4 * words_for(n)]; + drop(Scratch::over_for_program(&mut poison, &keep, n).expect("arena")); + assert!( + poison.iter().any(|&w| w != u64::MAX), + "the poison probe must be able to see a carve" + ); +} + +/// FAILS IF: the recogniser admits a shape it cannot fold — a derived +/// (scratch) operand, a complement, a non-scalar terminal — or refuses the +/// shapes it can. The admit half keeps this from passing vacuously. +#[test] +fn the_recogniser_admits_exactly_resident_single_op_count_and_any() { + assert!(count_p(Shape::And).fused_ternlog().is_some()); + assert!(any_p(Shape::Ternlog(0x96)).fused_ternlog().is_some()); + + let scratch_operand = Program::new( + vec![ + MaskOp::And { + a: Operand::Plane(0), + b: Operand::Plane(1), + dst: 0, + }, + MaskOp::Or { + a: Operand::Scratch(0), + b: Operand::Plane(2), + dst: 1, + }, + ], + Terminal::Count { + mask: Operand::Scratch(1), + }, + ); + assert!(scratch_operand.fused_ternlog().is_none(), "derived operand"); + assert!(scratch_operand.requires_scratch()); + + let not = Program::new( + vec![MaskOp::Not { + a: Operand::Plane(0), + dst: 0, + }], + Terminal::Count { + mask: Operand::Scratch(0), + }, + ); + assert!( + not.fused_ternlog().is_none(), + "Not clears the tail: a different shape" + ); + + for t in [ + Terminal::Keep { + mask: Operand::Scratch(0), + }, + Terminal::All { + mask: Operand::Scratch(0), + }, + ] { + let p = program(Shape::Or, t); + assert!(p.fused_ternlog().is_none(), "only Count/Any fold"); + assert!(p.requires_scratch()); + } +} + +/// A fusable shape whose slot sits past the fused path's validation bitmap +/// must still RUN — on the tiled path — never be rejected as a read before a +/// write. Both folds are covered: the ternlog fold and the #1268 range fold +/// share the same no-scratch validation, and a slot the bitmap cannot mark +/// once made `Count(Scratch(64))` fail with `ScratchReadBeforeWrite`. +#[test] +fn a_fusable_shape_on_a_high_slot_still_executes() { + use lance_graph_mask_risc::reference::reference_execute; + use lance_graph_mask_risc::Pred; + let n = 1000usize; + let words = n.div_ceil(64); + let mut seed = 0xB16_u64; + let mut mk = || -> Vec { + (0..words) + .map(|_| { + seed = seed + .wrapping_mul(6364136223846793005) + .wrapping_add(1442695040888963407); + seed + }) + .collect() + }; + let (mut pa, mut pb) = (mk(), mk()); + let tail = !0u64 >> (64 * words - n); + pa[words - 1] &= tail; + pb[words - 1] &= tail; + let masks: [&[u64]; 2] = [&pa, &pb]; + let planes = Planes { + n_rows: n, + masks: &masks, + lanes: &[], + }; + for dst in [0u16, 31, 32, 63, 64, 65, 1000] { + let shapes = [ + MaskOp::And { + a: Operand::Plane(0), + b: Operand::Plane(1), + dst, + }, + MaskOp::Pred { + pred: Pred::Range { lo: 3, hi: 777 }, + under: Some(Operand::Plane(0)), + dst, + }, + ]; + for op in shapes { + for terminal in [ + Terminal::Count { + mask: Operand::Scratch(dst), + }, + Terminal::Any { + mask: Operand::Scratch(dst), + }, + ] { + let p = Program::new(vec![op], terminal); + let want = reference_execute(&p, &planes, None).expect("oracle"); + let mut s = Scratch::for_program(&p, n).expect("scratch"); + let got = execute_extent(&p, &planes, &Foreign::NONE, &mut s, Out::None, 0..n) + .unwrap_or_else(|e| panic!("dst {dst} {op:?}: {e:?}")); + assert_eq!(got, want, "dst {dst} {op:?}"); + } + } + } +} diff --git a/crates/lance-graph-mask-risc/tools/gen_ternlog_dispatch.py b/crates/lance-graph-mask-risc/tools/gen_ternlog_dispatch.py index bd9cbc48f..9035ca3b0 100755 --- a/crates/lance-graph-mask-risc/tools/gen_ternlog_dispatch.py +++ b/crates/lance-graph-mask-risc/tools/gen_ternlog_dispatch.py @@ -2,7 +2,8 @@ """Generate the 256-arm dispatch tables in `src/ternlog_dispatch.rs`. A runtime `imm: u8` must reach a const-generic `ndarray::simd::mask_ternlog::` -(and its in-place sibling `mask_ternlog_assign::`) — the const generic can only +(and its in-place sibling `mask_ternlog_assign::`, and the two no-mask folds +`mask_ternlog_popcount::` / `mask_ternlog_any::`) — the const generic can only ever be instantiated with a literal, so there is no way to route a runtime byte into it except a 256-arm `match`, one arm per immediate. That match is pure boilerplate — 256 near-identical lines twice over — so it is generated here instead of hand-typed, @@ -90,6 +91,36 @@ def generate_region_lines() -> list[str]: lines.append(" }") lines.append("}") + lines.append("") + + lines.append( + "/// `Σ popcount(table[imm](a, b, c))` — the no-mask Count fold, routed " + "like the others." + ) + lines.append( + "pub fn ternlog_popcount_dispatch(imm: u8, a: &[u64], b: &[u64], c: &[u64]) -> u64 {" + ) + lines.append(" match imm {") + for imm in range(256): + lines.append(_arm("mask_ternlog_popcount", imm, "a, b, c")) + lines.append(" }") + lines.append("}") + + lines.append("") + + lines.append( + "/// `table[imm](a, b, c) != 0` anywhere — the no-mask Any fold, routed " + "like the others." + ) + lines.append( + "pub fn ternlog_any_dispatch(imm: u8, a: &[u64], b: &[u64], c: &[u64]) -> bool {" + ) + lines.append(" match imm {") + for imm in range(256): + lines.append(_arm("mask_ternlog_any", imm, "a, b, c")) + lines.append(" }") + lines.append("}") + lines.append(END_MARKER) return lines