diff --git a/.claude/board/entries/2026-09-23-program-collapse-boolean-chains.md b/.claude/board/entries/2026-09-23-program-collapse-boolean-chains.md new file mode 100644 index 000000000..e0c515d74 --- /dev/null +++ b/.claude/board/entries/2026-09-23-program-collapse-boolean-chains.md @@ -0,0 +1,60 @@ +# 2026-09-23 — Multi-op Boolean chains collapse onto the ternlog Count/Any fold + +**Status:** MEASURED (local branch, not yet a PR — waits on #1270's manual review and merge) · OPEN (lane predicates, >3 leaves, the tiled path's own per-tile cost) +**D-ids:** D-WFL-FUSE (multi-op case). Builds on D-WFL-T1-FUSED / D-WFL-T1-FUSED′ (#1270, ndarray #322). OQ-5, Rayon and any scheduler: untouched. + +## What changed +`Program::fused_ternlog()` no longer matches one op. It interprets the whole op sequence **symbolically**: +- every resident plane is a leaf table, in first-read order (`0xF0`, `0xCC`, `0xAA`); +- every op combines tables with the op's own Boolean law (`Ternlog` applies its immediate bitwise to the three input tables); +- each op writes the table to its destination slot, so slot reuse is modelled exactly. + +The terminal slot's table becomes the fold immediate. Recognition is a fixed on-stack array (`FUSED_SLOT_CAP = 32`) and never allocates. + +**Refused, and so left on the tiled path:** +- a fourth distinct plane; +- a `Pred` (comparison leaf) or `Gather` anywhere in the chain; +- a scratch slot read before it is written; +- a slot at or above `FUSED_SLOT_CAP`; +- any terminal other than Count/Any. `Keep` stays the explicit election of a bitmap. + +## Gates +- **End to end:** `fuse_program` output reaches the fold. +- **Differential:** raw chains are checked against the tiled Keep path across extents and against `reference_execute` over the whole population. The raw chains are a catalogue of 7 plus 400 random ones, with an anti-vacuity check of at least 20 distinct counts. +- **Zero materialization:** a poisoned-scratch gate plus a counting allocator, with its Keep twin. +- **Disable runs** (committed first, red, then restored to green): + - the interpreter returning only single-op programs; + - a transposed table-apply; + - slot reuse ignored (first write wins); + - a fourth leaf silently aliased to leaf 0. + + All four fail inside `program_collapse.rs` itself. + +## Measurement +Command: `cargo run --release -p lance-graph-mask-risc --example program_collapse_probe`. Setup: +- **Tier:** `x86-64-v3`, i.e. the AVX2 polyfill, which is the lance-graph default config. The host has avx512f, but **v4 was not measured**. +- **Size:** N = 1 048 576 rows, which is 16 384 words. +- **Values:** median ns, oracle-checked. + +| chain | ops | extent | fold Count | tiled Count | fold Any | tiled Any | derived words the tiled path writes | +|---|---|---|---|---|---|---|---| +| `(a&b)\|!c` | 3 | whole | 8.6 µs | 224 µs | 0.12 µs* | 218 µs | 49 152 | +| `((a^b)&!c)\|(a&c)` | 4 | whole | 16.2 µs | 352 µs | 0.14 µs* | 351 µs | 65 536 | +| `maj(a,b,c)^a` | 2 | whole | 17.4 µs | 166 µs | 0.14 µs* | 167 µs | 32 768 | +| `a&d` (d = !a; empty by data) | 1 | whole | 6.5 µs | 120 µs | 4.4 µs | 111 µs | 16 384 | +| `(a&b)\|(c&d)`, **4 planes** | 3 | whole | — (refused) | 283 µs | — | 278 µs | 49 152 | +| `(a&b)\|!c` | 3 | 1% | 0.27 µs | 2.7 µs | 0.17 µs | 2.6 µs | 495 | + +`*` = the fold exits at its first non-empty block; the tiled Any does not short-circuit. The data-empty row isolates the scan itself: **25×** for Any and **18×** for Count, with no early exit on either path. + +**Readings:** +- The collapsed fold writes **0** derived words. The tiled path writes `ops × touched words`. +- **Whole-population Count gain: 10–26×.** This exceeds ndarray #322's single-op 1.3–1.9×. **Not every part of that gap is write elimination**: the tiled path also pays per-tile interpreter dispatch (TILE_WORDS = 8) for every op. This probe does not separate the two costs. +- **The boundary is visible.** The 4-plane chain has the same op count as `(a&b)|!c`, and at 283 µs it pays the whole tiled price. One ternlog addresses three inputs, and the collapse stops exactly there. +- **An algebraically constant chain** (e.g. `a & b & !a` → table `0x00`) monomorphizes to a fold that never reads the planes. That is correct, but it is not a scan, which is why the empty row is empty by data rather than by algebra. + +## Open +- Lane predicates (comparison leaves) are deliberately not attempted. They deserve their own pause. +- For more than three leaves: tree splitting / multi-pass is not attempted. Only its boundary is measured. +- The split between per-tile dispatch cost and write cost on the tiled path is unmeasured. +- The avx512 (v4) tier is unmeasured. diff --git a/.claude/board/entries/README.md b/.claude/board/entries/README.md index 669130209..27d6f62d0 100644 --- a/.claude/board/entries/README.md +++ b/.claude/board/entries/README.md @@ -25,7 +25,7 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately opposite directions; the stranding this convention prevents shows up in exactly one of them, never both. -152 entries, 2026-08-06 .. 2026-09-23. +154 entries, 2026-08-06 .. 2026-09-23. | date | entry id | finding | file | |---|---|---|---| @@ -33,6 +33,7 @@ exactly one of them, never both. | 2026-09-23 | `terminal-elects-materialization-range-plane` | | [2026-09-23-terminal-elects-materialization-range-plane.md](2026-09-23-terminal-elects-materialization-range-plane.md) | | 2026-09-23 | `report-plan-zero-copy-pivot-docir-convergence` | ReportPlan lowers into Quack; pivot = view over shared Arc; reports become OGAR ObjectSlot sources via grid_of (OGAR #307) | [2026-09-23-report-plan-zero-copy-pivot-docir-convergence.md](2026-09-23-report-plan-zero-copy-pivot-docir-convergence.md) | | 2026-09-23 | `quack-having-sym-sum-presence-mask` | | [2026-09-23-quack-having-sym-sum-presence-mask.md](2026-09-23-quack-having-sym-sum-presence-mask.md) | +| 2026-09-23 | `program-collapse-boolean-chains` | | [2026-09-23-program-collapse-boolean-chains.md](2026-09-23-program-collapse-boolean-chains.md) | | 2026-09-23 | `cubecl-llvm-boundary-and-audit-regrade` | | [2026-09-23-cubecl-llvm-boundary-and-audit-regrade.md](2026-09-23-cubecl-llvm-boundary-and-audit-regrade.md) | | 2026-09-23 | `absolute-execution-extent` | | [2026-09-23-absolute-execution-extent.md](2026-09-23-absolute-execution-extent.md) | | 2026-09-22 | `quack-duckdb-parity-t0-keyed-reduction` | | [2026-09-22-quack-duckdb-parity-t0-keyed-reduction.md](2026-09-22-quack-duckdb-parity-t0-keyed-reduction.md) | diff --git a/crates/lance-graph-mask-risc/examples/program_collapse_probe.rs b/crates/lance-graph-mask-risc/examples/program_collapse_probe.rs new file mode 100644 index 000000000..79cf92ef6 --- /dev/null +++ b/crates/lance-graph-mask-risc/examples/program_collapse_probe.rs @@ -0,0 +1,308 @@ +//! Multi-op Boolean chain → Count/Any at N = 1M rows: collapsed fold vs tiled. +//! +//! The same raw op sequence, three physical endings: +//! +//! - FOLD: `Program::fused_ternlog` interprets the chain symbolically and +//! lowers it onto ONE `mask_ternlog_{popcount,any}` pass; no slot is carved +//! and no derived word is written. +//! - TILED: the identical chain with its terminal routed through slot +//! [`FUSED_SLOT_CAP`] — the recogniser refuses (a bound of the recogniser, +//! never of the semantics), so every op writes its tile slot and the +//! terminal reads the last one back. Same answer, today's scratch path. +//! - KEEP: the chain with a `Keep` terminal into an `Out::Mask`, then +//! `popcount_batch_u64` / `mask_any` over the kept words. +//! +//! Reported per chain × extent: median ns per arm and the derived words the +//! tiled arm writes (`ops × touched words`; the fold writes 0). A four-plane +//! chain is measured on the tiled path only: it is outside the algebra (one +//! ternlog addresses three inputs), and its row records where the collapse +//! stops. Every result is checked against a bit-serial scalar oracle. +//! +//! `cargo run --release -p lance-graph-mask-risc --example program_collapse_probe` + +use std::time::Instant; + +use lance_graph_mask_risc::exec::{execute_extent, Scratch}; +use lance_graph_mask_risc::{ + touched_words, Foreign, MaskOp, Operand, Out, Planes, Program, Terminal, Value, FUSED_SLOT_CAP, +}; +use ndarray::simd::{mask_any, popcount_batch_u64}; + +fn lcg(seed: &mut u64) -> u64 { + *seed = seed + .wrapping_mul(6364136223846793005) + .wrapping_add(1442695040888963407); + *seed >> 11 +} + +fn median(reps: usize, mut f: impl FnMut() -> T) -> (f64, T) { + let mut ts = Vec::with_capacity(reps); + let mut last = None; + for _ in 0..reps { + let t = Instant::now(); + last = Some(std::hint::black_box(f())); + ts.push(t.elapsed().as_nanos() as f64); + } + ts.sort_by(|a, b| a.total_cmp(b)); + (ts[reps / 2], last.expect("reps > 0")) +} + +/// Re-address the op writing `from` (and every later read of it) to `to`. +fn retarget(ops: &[MaskOp], from: u16, to: u16) -> Vec { + let r = |o: Operand| match o { + Operand::Scratch(s) if s == from => Operand::Scratch(to), + o => o, + }; + let d = |x: u16| if x == from { to } else { x }; + ops.iter() + .map(|op| match *op { + MaskOp::And { a, b, dst } => MaskOp::And { + a: r(a), + b: r(b), + dst: d(dst), + }, + MaskOp::Or { a, b, dst } => MaskOp::Or { + a: r(a), + b: r(b), + dst: d(dst), + }, + MaskOp::Xor { a, b, dst } => MaskOp::Xor { + a: r(a), + b: r(b), + dst: d(dst), + }, + MaskOp::AndNot { a, b, dst } => MaskOp::AndNot { + a: r(a), + b: r(b), + dst: d(dst), + }, + MaskOp::Not { a, dst } => MaskOp::Not { + a: r(a), + dst: d(dst), + }, + MaskOp::Ternlog { imm, a, b, c, dst } => MaskOp::Ternlog { + imm, + a: r(a), + b: r(b), + c: r(c), + dst: d(dst), + }, + other => other, + }) + .collect() +} + +fn main() { + let n = 1usize << 20; + let words = n.div_ceil(64); + let mut seed = 0xC0_11A5_u64; + let mut mk = |modulus: u64| { + let mut w = vec![0u64; words]; + for r in 0..n { + if lcg(&mut seed).is_multiple_of(modulus) { + w[r / 64] |= 1 << (r % 64); + } + } + w + }; + let (pa, pb, pc) = (mk(2), mk(3), mk(5)); + // d = !a (n is a multiple of 64, so no tail bits): `a & d` is empty BY + // DATA over a non-constant table, so Any has no first-block exit and the + // fold must genuinely scan. (A chain that is empty by ALGEBRA, e.g. + // `a & b & !a`, lowers to the constant table 0x00, and the monomorphized + // fold never reads the planes at all.) + let pd: Vec = pa.iter().map(|w| !w).collect(); + let masks: [&[u64]; 4] = [&pa, &pb, &pc, &pd]; + let planes = Planes { + n_rows: n, + masks: &masks, + lanes: &[], + }; + let bit = |p: &[u64], r: usize| p[r / 64] >> (r % 64) & 1 == 1; + let (a, b, c, d) = ( + Operand::Plane(0), + Operand::Plane(1), + Operand::Plane(2), + Operand::Plane(3), + ); + let s = Operand::Scratch; + type Oracle = fn(bool, bool, bool, bool) -> bool; + let chains: [(&str, Vec, u16, Oracle); 5] = [ + ( + "(a&b)|!c", + vec![ + MaskOp::And { a, b, dst: 0 }, + MaskOp::Not { a: c, dst: 1 }, + MaskOp::Or { + a: s(0), + b: s(1), + dst: 2, + }, + ], + 2, + |a, b, c, _| (a && b) || !c, + ), + ( + "((a^b)&!c)|(a&c)", + vec![ + MaskOp::Xor { a, b, dst: 0 }, + MaskOp::AndNot { + a: s(0), + b: c, + dst: 1, + }, + MaskOp::And { a, b: c, dst: 2 }, + MaskOp::Or { + a: s(1), + b: s(2), + dst: 0, + }, + ], + 0, + |a, b, c, _| ((a ^ b) && !c) || (a && c), + ), + ( + "maj(a,b,c)^a", + vec![ + MaskOp::Ternlog { + imm: 0xE8, + a, + b, + c, + dst: 0, + }, + MaskOp::Xor { + a: s(0), + b: a, + dst: 1, + }, + ], + 1, + |a, b, c, _| ((a as u8 + b as u8 + c as u8) >= 2) ^ a, + ), + ( + // Empty by data: Any cannot exit early on either path, so this + // row measures the scan itself rather than a first-block exit. + "empty:a&d", + vec![MaskOp::And { a, b: d, dst: 0 }], + 0, + |a, _, _, d| a && d, + ), + ( + "4p:(a&b)|(c&d)", + vec![ + MaskOp::And { a, b, dst: 0 }, + MaskOp::And { a: c, b: d, dst: 1 }, + MaskOp::Or { + a: s(0), + b: s(1), + dst: 2, + }, + ], + 2, + |a, b, c, d| (a && b) || (c && d), + ), + ]; + let mid = n / 2 + 17; + let extents = [("1%", mid, mid + n / 100), ("whole", 0, n)]; + let cap = FUSED_SLOT_CAP as u16; + println!( + "{:>17} {:>5} {:>5} {:>3} {:>10} {:>10} {:>10} {:>6} {:>6} {:>9}", + "chain", "ext", "term", "ops", "fold_ns", "tiled_ns", "keep_ns", "t/f", "k/f", "tiled_wr" + ); + for (name, ops, last, oracle) in chains { + let term = |mask| (Terminal::Count { mask }, Terminal::Any { mask }); + let (tc, ta) = term(s(last)); + let fold = [Program::new(ops.clone(), tc), Program::new(ops.clone(), ta)]; + let tiled_ops = retarget(&ops, last, cap); + let (ttc, tta) = term(s(cap)); + let tiled = [ + Program::new(tiled_ops.clone(), ttc), + Program::new(tiled_ops, tta), + ]; + let keep = Program::new(ops.clone(), Terminal::Keep { mask: s(last) }); + let collapses = fold[0].fused_ternlog().is_some(); + assert_eq!(collapses, fold[1].fused_ternlog().is_some()); + assert_eq!( + collapses, + !name.starts_with("4p"), + "{name}: collapse boundary" + ); + assert!(tiled.iter().all(|p| p.fused_ternlog().is_none())); + let mut ts = Scratch::for_program(&tiled[0], n).expect("scratch"); + let mut ks = Scratch::for_program(&keep, n).expect("scratch"); + let mut out = vec![0u64; words]; + for (ename, lo, hi) in extents { + let want = (lo..hi) + .filter(|&r| oracle(bit(&pa, r), bit(&pb, r), bit(&pc, r), bit(&pd, r))) + .count(); + let reps = if hi - lo > 100_000 { 41 } else { 1001 }; + let span = touched_words(lo as u32, hi as u32); + for (k, tname) in ["Count", "Any"].into_iter().enumerate() { + let expect = if k == 0 { + Value::Count(want) + } else { + Value::Bool(want > 0) + }; + let (fns, fv) = if collapses { + median(reps, || { + let mut s0 = Scratch::new(0, 0); + execute_extent( + &fold[k], + &planes, + &Foreign::NONE, + &mut s0, + Out::None, + lo..hi, + ) + .expect("fold") + }) + } else { + (f64::NAN, expect) + }; + let (tns, tv) = median(reps, || { + execute_extent( + &tiled[k], + &planes, + &Foreign::NONE, + &mut ts, + Out::None, + lo..hi, + ) + .expect("tiled") + }); + let (kns, kv) = median(reps, || { + out[span.clone()].fill(0); + execute_extent( + &keep, + &planes, + &Foreign::NONE, + &mut ks, + Out::Mask(&mut out), + lo..hi, + ) + .expect("keep"); + if k == 0 { + Value::Count(popcount_batch_u64(&out[span.clone()]) as usize) + } else { + Value::Bool(mask_any(&out[span.clone()])) + } + }); + assert_eq!(fv, expect, "fold {name} {ename} {tname}"); + assert_eq!(tv, expect, "tiled {name} {ename} {tname}"); + assert_eq!(kv, expect, "keep {name} {ename} {tname}"); + println!( + "{name:>17} {ename:>5} {tname:>5} {:>3} {fns:>10.0} {tns:>10.0} {kns:>10.0} {:>6.2} {:>6.2} {:>9}", + ops.len(), + tns / fns, + kns / fns, + ops.len() * span.len() + ); + } + } + } + println!( + "(n = {n}, {words} population words; tiled_wr = derived words the tiled path \ + writes, ops × touched words; the fold writes 0; fold_ns NaN = not collapsible)" + ); +} diff --git a/crates/lance-graph-mask-risc/src/exec.rs b/crates/lance-graph-mask-risc/src/exec.rs index 79281220a..4d2fda6a4 100644 --- a/crates/lance-graph-mask-risc/src/exec.rs +++ b/crates/lance-graph-mask-risc/src/exec.rs @@ -1885,12 +1885,13 @@ mod tests { ); // CAN-STAY-SILENT: an ordinary program still gets its arena, sized // exactly. A guard that refused everything would pass the half above. + // `All` is never fused, so this program genuinely needs its slots. let ok = Program::new( vec![MaskOp::Not { a: Operand::Plane(0), dst: 2, }], - Terminal::Count { + Terminal::All { mask: Operand::Scratch(2), }, ); @@ -2016,12 +2017,14 @@ mod tests { masks: &masks, lanes: &[], }; + // `All` is never fused (a `Count`/`Any` over this op would fold and + // need no scratch at all), so the scratch checks below are reached. let p = Program::new( vec![MaskOp::Not { a: Operand::Plane(0), dst: 3, }], - Terminal::Any { + Terminal::All { mask: Operand::Scratch(3), }, ); diff --git a/crates/lance-graph-mask-risc/src/ir.rs b/crates/lance-graph-mask-risc/src/ir.rs index 4251a7149..a892bafdf 100644 --- a/crates/lance-graph-mask-risc/src/ir.rs +++ b/crates/lance-graph-mask-risc/src/ir.rs @@ -614,42 +614,127 @@ impl Program { } /// The no-mask lowering of a Boolean membership over RESIDENT planes, if - /// this program is one: a single `And` / `Or` / `Xor` / `AndNot` / - /// `Ternlog` whose operands are all [`Operand::Plane`], folded by `Count` - /// or `Any` of its own `dst`. + /// this program is one: a sequence of `And` / `Or` / `Xor` / `AndNot` / + /// `Not` / `Ternlog` ops that, taken together, read at most THREE distinct + /// [`Operand::Plane`]s, folded by `Count` or `Any` of a slot the sequence + /// wrote. /// - /// Every such op is one 3-input truth table (a 2-input op is a table that - /// ignores `c`), and `ndarray::simd::mask_ternlog_popcount` / - /// `mask_ternlog_any` fold a table straight from its operands' words to a - /// scalar. So the membership never becomes a bitmap: nothing is written. + /// The program representation collapses before execution: each op is + /// interpreted SYMBOLICALLY as an 8-bit truth table over the (at most + /// three) plane leaves, in the VPTERNLOG input convention (leaf 0 reads + /// `0xF0`, leaf 1 `0xCC`, leaf 2 `0xAA`). `And`/`Or`/`Xor`/`AndNot`/`Not` + /// combine their inputs' tables bitwise, and `Ternlog` applies its own + /// immediate to its three inputs' tables bit by bit. The table left in the + /// terminal's slot IS the immediate of one ternlog over the leaves, so + /// `ndarray::simd::mask_ternlog_popcount` / `mask_ternlog_any` fold the + /// whole sequence straight from the planes' words to a scalar. No + /// intermediate slot is ever written: the scratch ops exist only in the + /// program text, never in memory. /// - /// A scratch operand, `Not` (which clears the tail against `n_rows` — a - /// different shape), a predicate, a gather, or any other terminal returns - /// `None` and runs the ordinary path. `Keep` is NEVER fused. + /// Slots are tracked in program order, so a slot overwritten mid-sequence + /// reads its latest value, exactly as the tiled path executes it. `Not`'s + /// tail clearing needs no special case: the fold restricts the + /// population's own last word to its live rows, which is the same result + /// on every live row. + /// + /// Declines (returns `None`, ordinary path) on: a fourth distinct plane + /// anywhere in the sequence, a `Pred` or `Gather`, a read of a slot the + /// sequence has not yet written, a slot at or above + /// [`FUSED_SLOT_CAP`] (the interpreter keeps its tables in a fixed + /// on-stack array so that recognition never allocates), or any terminal + /// other than `Count`/`Any`. `Keep` is NEVER fused: it is the explicit + /// election of a bitmap. pub fn fused_ternlog(&self) -> Option { - let plane = |o: &Operand| match *o { - Operand::Plane(p) => Some(p), - Operand::Scratch(_) => None, - }; - let (imm, a, b, c, dst) = match self.ops.as_slice() { - [MaskOp::And { a, b, dst }] => (TABLE_AND, plane(a)?, plane(b)?, plane(b)?, *dst), - [MaskOp::Or { a, b, dst }] => (TABLE_OR, plane(a)?, plane(b)?, plane(b)?, *dst), - [MaskOp::Xor { a, b, dst }] => (TABLE_XOR, plane(a)?, plane(b)?, plane(b)?, *dst), - [MaskOp::AndNot { a, b, dst }] => (TABLE_ANDNOT, plane(a)?, plane(b)?, plane(b)?, *dst), - [MaskOp::Ternlog { imm, a, b, c, dst }] => { - (*imm, plane(a)?, plane(b)?, plane(c)?, *dst) - } + let fold = match self.terminal { + Terminal::Count { + mask: Operand::Scratch(s), + } => (FusedFold::Count, s), + Terminal::Any { + mask: Operand::Scratch(s), + } => (FusedFold::Any, s), _ => return None, }; - if usize::from(dst) >= FUSED_SLOT_CAP { + if self.ops.is_empty() { return None; } - let fold = match self.terminal { - Terminal::Count { mask } if mask == Operand::Scratch(dst) => FusedFold::Count, - Terminal::Any { mask } if mask == Operand::Scratch(dst) => FusedFold::Any, - _ => return None, + // Distinct plane leaves in first-read order, and the table each one + // contributes in the VPTERNLOG input convention. + const LEAF_TABLES: [u8; 3] = [0xF0, 0xCC, 0xAA]; + let mut leaves: [u16; 3] = [0; 3]; + let mut n_leaves = 0usize; + let mut slots: [Option; FUSED_SLOT_CAP] = [None; FUSED_SLOT_CAP]; + let read = |o: &Operand, + leaves: &mut [u16; 3], + n_leaves: &mut usize, + slots: &[Option; FUSED_SLOT_CAP]| + -> Option { + match *o { + Operand::Plane(p) => { + let i = match leaves[..*n_leaves].iter().position(|&q| q == p) { + Some(i) => i, + None if *n_leaves < 3 => { + leaves[*n_leaves] = p; + *n_leaves += 1; + *n_leaves - 1 + } + None => return None, + }; + Some(LEAF_TABLES[i]) + } + Operand::Scratch(s) => *slots.get(usize::from(s))?, + } }; - Some(FusedTernlog { imm, a, b, c, fold }) + for op in &self.ops { + let (t, dst) = match op { + MaskOp::And { a, b, dst } => ( + read(a, &mut leaves, &mut n_leaves, &slots)? + & read(b, &mut leaves, &mut n_leaves, &slots)?, + *dst, + ), + MaskOp::Or { a, b, dst } => ( + read(a, &mut leaves, &mut n_leaves, &slots)? + | read(b, &mut leaves, &mut n_leaves, &slots)?, + *dst, + ), + MaskOp::Xor { a, b, dst } => ( + read(a, &mut leaves, &mut n_leaves, &slots)? + ^ read(b, &mut leaves, &mut n_leaves, &slots)?, + *dst, + ), + MaskOp::AndNot { a, b, dst } => ( + read(a, &mut leaves, &mut n_leaves, &slots)? + & !read(b, &mut leaves, &mut n_leaves, &slots)?, + *dst, + ), + MaskOp::Not { a, dst } => (!read(a, &mut leaves, &mut n_leaves, &slots)?, *dst), + MaskOp::Ternlog { imm, a, b, c, dst } => { + let (ta, tb, tc) = ( + read(a, &mut leaves, &mut n_leaves, &slots)?, + read(b, &mut leaves, &mut n_leaves, &slots)?, + read(c, &mut leaves, &mut n_leaves, &slots)?, + ); + (apply_table(*imm, ta, tb, tc), *dst) + } + MaskOp::Pred { .. } | MaskOp::Gather { .. } => return None, + }; + *slots.get_mut(usize::from(dst))? = Some(t); + } + let imm = (*slots.get(usize::from(fold.1))?)?; + if n_leaves == 0 { + return None; + } + // Unused leaf positions are don't-cares of `imm` (it was computed + // without them); bind them to leaf 0 so every operand is a real plane. + let a = leaves[0]; + let b = if n_leaves > 1 { leaves[1] } else { a }; + let c = if n_leaves > 2 { leaves[2] } else { a }; + Some(FusedTernlog { + imm, + a, + b, + c, + fold: fold.0, + }) } /// Whether executing this program needs any scratch slot at all. @@ -725,8 +810,9 @@ pub struct FusedTerminal { /// without writing membership bits: see [`Program::fused_ternlog`]. /// /// The table is in the VPTERNLOG index convention `(a << 2) | (b << 1) | c` -/// that [`MaskOp::Ternlog`] already uses; a 2-input op reaches here as a -/// table that ignores `c` (and `c` repeats `b`). +/// that [`MaskOp::Ternlog`] already uses. It is the WHOLE op sequence's +/// collapsed function; a sequence over fewer than three distinct planes gets a +/// table that ignores the unused positions, which are bound to `a`. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub struct FusedTernlog { /// The 8-bit truth table. @@ -741,22 +827,27 @@ pub struct FusedTernlog { pub fold: FusedFold, } -/// `a & b` — true at indices 6, 7 (`a = b = 1`, either `c`). -pub(crate) const TABLE_AND: u8 = 0xC0; -/// `a | b` — true wherever `a` (4..7) or `b` (2, 3, 6, 7) is set. -pub(crate) const TABLE_OR: u8 = 0xFC; -/// `a ^ b` — true at 2, 3 (`b` only) and 4, 5 (`a` only). -pub(crate) const TABLE_XOR: u8 = 0x3C; -/// `a & !b` — true at 4, 5 (`a = 1`, `b = 0`). -pub(crate) const TABLE_ANDNOT: u8 = 0x30; +/// Apply the VPTERNLOG table `imm` bitwise to three input tables: output bit +/// `i` is `imm[(ta_i << 2) | (tb_i << 1) | tc_i]`. +fn apply_table(imm: u8, ta: u8, tb: u8, tc: u8) -> u8 { + let mut out = 0u8; + for i in 0..8 { + let idx = ((ta >> i) & 1) << 2 | ((tb >> i) & 1) << 1 | ((tc >> i) & 1); + out |= ((imm >> idx) & 1) << i; + } + out +} /// The highest scratch slot (exclusive) a fused shape may name. /// /// The fused paths validate a program with an on-stack slot bitmap of /// `FUSED_SLOT_CAP.div_ceil(64)` words and never touch real scratch, so a /// slot the bitmap cannot mark would be rejected as a read before a write. -/// A shape naming a higher slot is simply not fused and runs on the tiled -/// path. A bound of the recogniser, never of the semantics. +/// [`Program::fused_ternlog`]'s symbolic interpreter keeps its per-slot +/// tables in a fixed on-stack array of the same length, because recognition +/// runs on the execute path and must not allocate. A shape naming a higher +/// slot is simply not fused and runs on the tiled path. A bound of the +/// recogniser, never of the semantics. pub const FUSED_SLOT_CAP: usize = 32; /// The scalar folds that may consume membership without materializing it. diff --git a/crates/lance-graph-mask-risc/tests/fused_ternlog.rs b/crates/lance-graph-mask-risc/tests/fused_ternlog.rs index 6f97e629f..a370b2e07 100644 --- a/crates/lance-graph-mask-risc/tests/fused_ternlog.rs +++ b/crates/lance-graph-mask-risc/tests/fused_ternlog.rs @@ -360,48 +360,102 @@ fn the_fold_writes_no_membership_and_allocates_nothing() { ); } -/// FAILS IF: the recogniser admits a shape it cannot fold — a derived -/// (scratch) operand, a complement, a non-scalar terminal — or refuses the -/// shapes it can. The admit half keeps this from passing vacuously. +/// FAILS IF: the recogniser admits a shape it cannot fold — a fourth distinct +/// plane, a predicate or gather in the chain, a slot read before any op wrote +/// it, a non-scalar terminal — or refuses a shape it can (a multi-op chain +/// over at most three planes, a bare complement). The admit half keeps the +/// refuse half from passing vacuously. #[test] -fn the_recogniser_admits_exactly_resident_single_op_count_and_any() { +fn the_recogniser_admits_exactly_collapsible_chains_to_count_and_any() { + let (p0, p1, p2, p3) = ( + Operand::Plane(0), + Operand::Plane(1), + Operand::Plane(2), + Operand::Plane(3), + ); + let count = |s: u16| Terminal::Count { + mask: Operand::Scratch(s), + }; + // Admit: single ops, a derived operand, and a bare complement. assert!(count_p(Shape::And).fused_ternlog().is_some()); assert!(any_p(Shape::Ternlog(0x96)).fused_ternlog().is_some()); + let chain = Program::new( + vec![ + MaskOp::And { + a: p0, + b: p1, + dst: 0, + }, + MaskOp::Or { + a: Operand::Scratch(0), + b: p2, + dst: 1, + }, + ], + count(1), + ); + assert!( + chain.fused_ternlog().is_some(), + "a derived operand collapses" + ); + assert!(!chain.requires_scratch()); + let not = Program::new(vec![MaskOp::Not { a: p0, dst: 0 }], count(0)); + assert!(not.fused_ternlog().is_some(), "a complement collapses"); - let scratch_operand = Program::new( + // Refuse: a fourth distinct plane anywhere in the chain. + let four = Program::new( vec![ MaskOp::And { - a: Operand::Plane(0), - b: Operand::Plane(1), + a: p0, + b: p1, dst: 0, }, MaskOp::Or { + a: p2, + b: p3, + dst: 1, + }, + MaskOp::Xor { a: Operand::Scratch(0), - b: Operand::Plane(2), + b: Operand::Scratch(1), + dst: 2, + }, + ], + count(2), + ); + assert!(four.fused_ternlog().is_none(), "four distinct planes"); + assert!(four.requires_scratch()); + + // Refuse: a predicate in the chain. + let pred = Program::new( + vec![ + MaskOp::Pred { + pred: lance_graph_mask_risc::Pred::Range { lo: 0, hi: 5 }, + under: None, + dst: 0, + }, + MaskOp::And { + a: Operand::Scratch(0), + b: p1, dst: 1, }, ], - Terminal::Count { - mask: Operand::Scratch(1), - }, + count(1), ); - assert!(scratch_operand.fused_ternlog().is_none(), "derived operand"); - assert!(scratch_operand.requires_scratch()); + assert!(pred.fused_ternlog().is_none(), "a predicate is not a plane"); - let not = Program::new( - vec![MaskOp::Not { - a: Operand::Plane(0), + // Refuse: a slot read before any op wrote it. + let unwritten = Program::new( + vec![MaskOp::And { + a: Operand::Scratch(5), + b: p1, dst: 0, }], - Terminal::Count { - mask: Operand::Scratch(0), - }, - ); - assert!( - not.fused_ternlog().is_none(), - "Not clears the tail: a different shape" + count(0), ); + assert!(unwritten.fused_ternlog().is_none(), "read before write"); + // Refuse: only Count/Any fold. for t in [ Terminal::Keep { mask: Operand::Scratch(0), diff --git a/crates/lance-graph-mask-risc/tests/program_collapse.rs b/crates/lance-graph-mask-risc/tests/program_collapse.rs new file mode 100644 index 000000000..60382defc --- /dev/null +++ b/crates/lance-graph-mask-risc/tests/program_collapse.rs @@ -0,0 +1,503 @@ +//! A composed Boolean PROGRAM collapses before execution and terminates in +//! Count/Any without its membership ever becoming bits. +//! +//! Two routes reach the same fold: +//! +//! - the existing fuser: a `BoolExpr` over at most three distinct planes is +//! lowered by `fuse_program` to ONE `Ternlog`, which the fold consumes; +//! - a raw op sequence (`And` → `Or` → `Not` → …) is interpreted symbolically +//! by `Program::fused_ternlog` as an 8-bit truth table over its (at most +//! three) plane leaves, and that table is the fold's immediate. +//! +//! Every case is checked three ways over the same absolute rows: the fold +//! (run with a zero-slot scratch, so it CANNOT write a derived word), the +//! MATERIALIZE arm (the same ops with a `Keep` terminal, which is never fused +//! and runs the ordinary tiled path), and the crate's row-by-row reference +//! oracle. + +use std::alloc::{GlobalAlloc, Layout, System}; +use std::cell::Cell; + +use lance_graph_mask_risc::exec::{execute_extent, Scratch}; +use lance_graph_mask_risc::fuse::{fuse_program, BoolExpr}; +use lance_graph_mask_risc::{ + reference_execute, Foreign, MaskOp, Operand, Out, Planes, Program, Terminal, Value, +}; + +struct Counting; + +thread_local! { + static BYTES: Cell = const { Cell::new(0) }; +} + +fn bytes() -> usize { + BYTES.with(Cell::get) +} + +// SAFETY: a pure pass-through to `System`; the counter is the only addition. +unsafe impl GlobalAlloc for Counting { + unsafe fn alloc(&self, layout: Layout) -> *mut u8 { + let _ = BYTES.try_with(|b| b.set(b.get() + layout.size())); + // SAFETY: same layout, same contract as the caller's. + unsafe { System.alloc(layout) } + } + unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) { + // SAFETY: `ptr` came from `alloc` above with this `layout`. + unsafe { System.dealloc(ptr, layout) } + } +} + +#[global_allocator] +static A: Counting = Counting; + +fn lcg(seed: &mut u64) -> u64 { + *seed = seed + .wrapping_mul(6364136223846793005) + .wrapping_add(1442695040888963407); + *seed >> 11 +} + +fn words_for(n: usize) -> usize { + n.div_ceil(64) +} + +fn random_plane(n: usize, seed: &mut u64, sparse: bool) -> Vec { + let mut w = vec![0u64; words_for(n)]; + for r in 0..n { + let x = lcg(seed); + if (sparse && x.is_multiple_of(11)) || (!sparse && x & 1 == 1) { + w[r / 64] |= 1u64 << (r % 64); + } + } + w +} + +fn extents(n: usize) -> Vec<(usize, usize)> { + let mut v = vec![(0, n), (0, 0)]; + for k in [1usize, 63, 64, 65, 129] { + if k <= n { + v.push((k, n)); + v.push((0, k)); + } + } + if n >= 3 { + v.push((n / 3, 2 * n / 3 + 1)); + } + v.retain(|&(lo, hi)| lo <= hi && hi <= n); + v +} + +/// The same ops with the terminal swapped for `Keep` (never fused), counted +/// over the extent — the tiled, scratch-writing execution of the program. +fn keep_count(p: &Program, planes: &Planes<'_>, lo: usize, hi: usize) -> usize { + let mask = match p.terminal { + Terminal::Count { mask } | Terminal::Any { mask } => mask, + _ => unreachable!("fixtures only use Count/Any"), + }; + let keep = Program::new(p.ops.clone(), Terminal::Keep { mask }); + assert!(keep.fused_ternlog().is_none(), "Keep is never fused"); + let mut sc = Scratch::for_program(&keep, planes.n_rows).expect("scratch"); + let mut out = vec![0u64; words_for(planes.n_rows)]; + execute_extent( + &keep, + planes, + &Foreign::NONE, + &mut sc, + Out::Mask(&mut out), + lo..hi, + ) + .expect("keep arm"); + out.iter().map(|w| w.count_ones() as usize).sum() +} + +/// Check one Count program (and its Any twin) over every extent, three ways. +fn check(p_count: &Program, planes: &Planes<'_>, what: &str) { + let mask = match p_count.terminal { + Terminal::Count { mask } => mask, + _ => unreachable!(), + }; + let p_any = Program::new(p_count.ops.clone(), Terminal::Any { mask }); + assert!(p_count.fused_ternlog().is_some(), "{what}: must collapse"); + assert!(p_any.fused_ternlog().is_some(), "{what}: Any must collapse"); + for (lo, hi) in extents(planes.n_rows) { + let kept = keep_count(p_count, planes, lo, hi); + let mut s = Scratch::new(0, 0); + let folded = execute_extent(p_count, planes, &Foreign::NONE, &mut s, Out::None, lo..hi) + .expect("fold count"); + assert_eq!(folded, Value::Count(kept), "{what} count [{lo},{hi})"); + let any = execute_extent(&p_any, planes, &Foreign::NONE, &mut s, Out::None, lo..hi) + .expect("fold any"); + assert_eq!(any, Value::Bool(kept > 0), "{what} any [{lo},{hi})"); + } + // The whole-population answer agrees with the independent row oracle too. + let whole = reference_execute(p_count, planes, None).expect("oracle"); + let mut s = Scratch::new(0, 0); + let folded = execute_extent( + p_count, + planes, + &Foreign::NONE, + &mut s, + Out::None, + 0..planes.n_rows, + ) + .expect("fold"); + assert_eq!(folded, whole, "{what} vs reference oracle"); +} + +fn leaf(i: u16) -> Box { + Box::new(BoolExpr::Leaf(Operand::Plane(i))) +} + +/// FAILS IF: the existing fuser's output for a ≤3-leaf expression does not +/// reach the fold — i.e. `fuse_program` → one `Ternlog` → `Count`/`Any` still +/// writes a mask — or folds to a wrong answer. +#[test] +fn a_fused_boolexpr_reaches_the_fold_end_to_end() { + let exprs: [(&str, BoolExpr); 4] = [ + ( + "(a & b) | !c", + BoolExpr::Or( + Box::new(BoolExpr::And(leaf(0), leaf(1))), + Box::new(BoolExpr::Not(leaf(2))), + ), + ), + ( + "(a & b) | (a & !b)", + BoolExpr::Or( + Box::new(BoolExpr::And(leaf(0), leaf(1))), + Box::new(BoolExpr::And(leaf(0), Box::new(BoolExpr::Not(leaf(1))))), + ), + ), + ( + "!(a ^ b)", + BoolExpr::Not(Box::new(BoolExpr::Xor(leaf(0), leaf(1)))), + ), + ( + "((a | b) & c) ^ (a & c)", + BoolExpr::Xor( + Box::new(BoolExpr::And( + Box::new(BoolExpr::Or(leaf(0), leaf(1))), + leaf(2), + )), + Box::new(BoolExpr::And(leaf(0), leaf(2))), + ), + ), + ]; + for n in [1usize, 65, 1317] { + let mut seed = 0xE2E_u64 ^ n as u64; + let m = [ + random_plane(n, &mut seed, false), + random_plane(n, &mut seed, true), + random_plane(n, &mut seed, false), + ]; + let masks: [&[u64]; 3] = [&m[0], &m[1], &m[2]]; + let planes = Planes { + n_rows: n, + masks: &masks, + lanes: &[], + }; + for (what, e) in &exprs { + let p = fuse_program(e, 0, |mask| Terminal::Count { mask }).expect("fuse"); + assert_eq!(p.ops.len(), 1, "{what}: the fuser emits ONE ternlog"); + assert!(matches!(p.ops[0], MaskOp::Ternlog { .. }), "{what}"); + check(&p, &planes, what); + } + } +} + +/// A hand-written catalogue of raw sequences the recogniser must collapse, +/// each shaped the way a lowering pass emits ops (not through `BoolExpr`). +fn catalogue() -> Vec<(&'static str, Program)> { + let (a, b, c) = (Operand::Plane(0), Operand::Plane(1), Operand::Plane(2)); + let s = Operand::Scratch; + let count = |x: u16| Terminal::Count { + mask: Operand::Scratch(x), + }; + vec![ + ( + "and→not→or", + Program::new( + vec![ + MaskOp::And { a, b, dst: 0 }, + MaskOp::Not { a: c, dst: 1 }, + MaskOp::Or { + a: s(0), + b: s(1), + dst: 2, + }, + ], + count(2), + ), + ), + ( + "slot reuse in place", + Program::new( + vec![ + MaskOp::And { a, b, dst: 0 }, + MaskOp::Or { + a: s(0), + b: c, + dst: 0, + }, + MaskOp::Not { a: s(0), dst: 0 }, + ], + count(0), + ), + ), + ( + "andnot chain with a repeated leaf", + Program::new( + vec![ + MaskOp::AndNot { a, b, dst: 0 }, + MaskOp::AndNot { + a: s(0), + b: c, + dst: 1, + }, + MaskOp::Or { + a: s(1), + b: a, + dst: 2, + }, + ], + count(2), + ), + ), + ( + "ternlog over a derived input", + Program::new( + vec![ + MaskOp::Xor { a, b, dst: 0 }, + MaskOp::Ternlog { + imm: 0xE8, + a: s(0), + b, + c, + dst: 1, + }, + ], + count(1), + ), + ), + ( + "dead op beside the result", + Program::new( + vec![ + MaskOp::And { a, b, dst: 0 }, + MaskOp::Or { a: b, b: c, dst: 1 }, + ], + count(0), + ), + ), + ( + "double complement", + Program::new( + vec![MaskOp::Not { a, dst: 0 }, MaskOp::Not { a: s(0), dst: 1 }], + count(1), + ), + ), + ( + "contradiction a & !a", + Program::new( + vec![ + MaskOp::Not { a, dst: 0 }, + MaskOp::And { a, b: s(0), dst: 1 }, + ], + count(1), + ), + ), + ] +} + +/// FAILS IF: a raw op sequence over at most three planes fails to collapse, or +/// the collapsed fold disagrees with the tiled path or the oracle. +#[test] +fn raw_op_sequences_collapse_and_agree_with_the_tiled_path() { + for n in [1usize, 63, 64, 65, 133, 1317] { + for sparse in [false, true] { + let mut seed = 0xC011_A95E ^ n as u64 ^ u64::from(sparse); + let m = [ + random_plane(n, &mut seed, sparse), + random_plane(n, &mut seed, false), + random_plane(n, &mut seed, sparse), + ]; + let masks: [&[u64]; 3] = [&m[0], &m[1], &m[2]]; + let planes = Planes { + n_rows: n, + masks: &masks, + lanes: &[], + }; + for (what, p) in catalogue() { + check(&p, &planes, what); + } + } + } +} + +/// FAILS IF: ANY randomly generated chain of the six Boolean ops over three +/// planes, with slot reuse and derived operands, collapses to a wrong answer — +/// or fails to collapse. 400 chains, length 1..=8, slots 0..6. +#[test] +fn random_chains_collapse_and_agree_with_the_tiled_path() { + let n = 333; + let mut seed = 0xBAD_C0DE_u64; + let m = [ + random_plane(n, &mut seed, false), + random_plane(n, &mut seed, true), + random_plane(n, &mut seed, false), + ]; + let masks: [&[u64]; 3] = [&m[0], &m[1], &m[2]]; + let planes = Planes { + n_rows: n, + masks: &masks, + lanes: &[], + }; + let mut distinct_results = std::collections::HashSet::new(); + for _ in 0..400 { + let len = 1 + (lcg(&mut seed) % 8) as usize; + let mut written: Vec = Vec::new(); + let mut ops = Vec::new(); + let pick = |seed: &mut u64, written: &Vec| { + if !written.is_empty() && lcg(seed).is_multiple_of(2) { + Operand::Scratch(written[(lcg(seed) as usize) % written.len()]) + } else { + Operand::Plane((lcg(seed) % 3) as u16) + } + }; + let mut last = 0u16; + for _ in 0..len { + let (x, y, z) = ( + pick(&mut seed, &written), + pick(&mut seed, &written), + pick(&mut seed, &written), + ); + let dst = (lcg(&mut seed) % 6) as u16; + ops.push(match lcg(&mut seed) % 6 { + 0 => MaskOp::And { a: x, b: y, dst }, + 1 => MaskOp::Or { a: x, b: y, dst }, + 2 => MaskOp::Xor { a: x, b: y, dst }, + 3 => MaskOp::AndNot { a: x, b: y, dst }, + 4 => MaskOp::Not { a: x, dst }, + _ => MaskOp::Ternlog { + imm: (lcg(&mut seed) & 0xFF) as u8, + a: x, + b: y, + c: z, + dst, + }, + }); + if !written.contains(&dst) { + written.push(dst); + } + last = dst; + } + let p = Program::new( + ops, + Terminal::Count { + mask: Operand::Scratch(last), + }, + ); + check(&p, &planes, "random chain"); + let mut s = Scratch::new(0, 0); + if let Ok(Value::Count(c)) = + execute_extent(&p, &planes, &Foreign::NONE, &mut s, Out::None, 0..n) + { + distinct_results.insert(c); + } + } + // Anti-vacuity: the generator must produce genuinely different functions, + // not 400 copies of a constant. + assert!( + distinct_results.len() >= 20, + "only {} distinct counts", + distinct_results.len() + ); +} + +/// FAILS IF: a chain over FOUR planes is collapsed (it cannot fit one 3-input +/// table) or runs to a wrong answer on the path it falls back to. This is the +/// boundary where the compact algebra stops, pinned rather than papered over. +#[test] +fn a_four_plane_chain_stays_on_the_tiled_path_and_is_correct() { + let n = 1000; + let mut seed = 0x4_u64; + let m: Vec> = (0..4).map(|_| random_plane(n, &mut seed, false)).collect(); + let masks: [&[u64]; 4] = [&m[0], &m[1], &m[2], &m[3]]; + let planes = Planes { + n_rows: n, + masks: &masks, + lanes: &[], + }; + let (a, b, c, d) = ( + Operand::Plane(0), + Operand::Plane(1), + Operand::Plane(2), + Operand::Plane(3), + ); + let p = Program::new( + vec![ + MaskOp::And { a, b, dst: 0 }, + MaskOp::Or { a: c, b: d, dst: 1 }, + MaskOp::Xor { + a: Operand::Scratch(0), + b: Operand::Scratch(1), + dst: 2, + }, + ], + Terminal::Count { + mask: Operand::Scratch(2), + }, + ); + assert!(p.fused_ternlog().is_none()); + assert!(p.requires_scratch()); + let mut sc = Scratch::for_program(&p, n).expect("scratch"); + let got = execute_extent(&p, &planes, &Foreign::NONE, &mut sc, Out::None, 0..n).expect("tiled"); + assert_eq!(got, reference_execute(&p, &planes, None).expect("oracle")); +} + +/// FAILS IF: a collapsed chain carves a slot, writes a scratch word, or +/// allocates. The twin half proves the poison probe can see a carve. +#[test] +fn a_collapsed_chain_writes_nothing_and_allocates_nothing() { + let n = 4133; + let mut seed = 0xA11_u64; + let m = [ + random_plane(n, &mut seed, false), + random_plane(n, &mut seed, false), + random_plane(n, &mut seed, true), + ]; + let masks: [&[u64]; 3] = [&m[0], &m[1], &m[2]]; + let planes = Planes { + n_rows: n, + masks: &masks, + lanes: &[], + }; + for (what, p) in catalogue() { + assert!(!p.requires_scratch(), "{what}"); + let mut poison = vec![u64::MAX; 8 * words_for(n)]; + { + let mut sc = Scratch::over_for_program(&mut poison, &p, n).expect("arena"); + let before = bytes(); + let v = execute_extent(&p, &planes, &Foreign::NONE, &mut sc, Out::None, 7..n - 3) + .expect("fold"); + assert_eq!(bytes(), before, "{what} allocated"); + assert!(matches!(v, Value::Count(_))); + } + assert!( + poison.iter().all(|&w| w == u64::MAX), + "{what} wrote scratch" + ); + } + let keep = Program::new( + catalogue()[0].1.ops.clone(), + Terminal::Keep { + mask: Operand::Scratch(2), + }, + ); + let mut poison = vec![u64::MAX; 8 * words_for(n)]; + drop(Scratch::over_for_program(&mut poison, &keep, n).expect("arena")); + assert!( + poison.iter().any(|&w| w != u64::MAX), + "probe must see a carve" + ); +}