Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .claude/board/STATUS_BOARD.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ materialization, and that is the ideal Layer-0 operation, not a compromise.
| D-WFL-AXIS | **Three independent axes, not one binary:** OPERATORS (fold · mask/ternlog · project · rotate · neighbour · reduce) × CARRIERS (canonical lane · range · descriptor · resident mask · cached mask) × MATERIALIZATION CHOICE (fused vs materialized bitmap). The BBB question is not *fold or mask?* but **is this membership relation transient algebra, or has it been PROMOTED to a mask carrier?** | Queued | every plan must record the promotion, not the operator choice. Falsified if a plan can promote a membership relation to a carrier without that appearing in it |
| D-WFL-ENTROPY | **Representation entropy should follow ANSWER entropy.** *"Do these two million-row regions intersect?"* ≈ 1 bit; building 125 KB of mask to find it is the obscenity — and 125 KB is Seam B's MEASURED number at N=1M, not rhetoric. *"How many overlap?"* = 32–64 bits; also fused. *"Give me the overlap, six thoughts will manipulate it"* justifies the bitmap, which is then low-entropy **relative to its future workload** | Queued | the ratio `materialized bytes : answer bytes` reported per operation (§6's `R_info`), with the downstream workload named whenever it exceeds 1 |
| D-WFL-W2b‴ | ⊘ supersedes W2b″'s "fold arm vs mask arm" — **both arms mask.** W2b-A: `Range × resident → FUSED masking → Count/Any`, no result mask. W2b-B: `→ masking → a MATERIALIZED bounded mask`. Same masking semantics, different result carrier | In PR — W2b-A shipped (fused `Range ∩ plane → Count/Any`, 0 derived words; `entries/2026-09-23-terminal-elects-materialization-range-plane.md`); W2b-B reuse burden OPEN | differential across arms and against the oracle. **W2b-B carries a burden W2b-A does not: it must NAME and MEASURE the downstream reuse justifying the carrier — a materialization with no demonstrated consumer FAILS the arm.** That is what deliberate promotion costs |
| D-WFL-EXTENT | Absolute execution extent: `execute_extent(program, planes, foreign, scratch, out, lo..hi)`; the extent is an OUTER restriction in `Planes`' absolute row coordinates (never a rebasing); `execute_into` = extent `0..n_rows` | In PR — shipped with split/merge, non-rebasing and structural gates; `entries/2026-09-23-absolute-execution-extent.md` | whole == merge of any partition in any order; extent starting at 65/129 reads absolute rows (worker-local reading shown to differ); a 1-row extent over 1M rows visits one word. OQ-5 untouched |

**Shortest form:** fold the datasets, mask the folds, materialize only when the
mask itself is worth keeping.
Expand Down
122 changes: 122 additions & 0 deletions .claude/board/entries/2026-09-23-absolute-execution-extent.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
# 2026-09-23 — Absolute execution extent for mask-risc

**Status:** MEASURED (extent seam shipped in PR) · OPEN (scheduler, OQ-5, the `_sym` / group merge laws)
**D-ids:** D-WFL-EXTENT (new, In PR). Corrects the wording of
`entries/2026-09-23-cubecl-llvm-boundary-and-audit-regrade.md` lesson 1 (below).
OQ-5 is untouched and still open.

## API shape
`execute_extent(program, planes, foreign, scratch, out, lo..hi)` in
`crates/lance-graph-mask-risc/src/exec.rs`. `execute_into` is now that call with
`0..n_rows`, so existing callers are unchanged and the whole extent accepts every terminal.
`extent_tiles` (the tile plan the executor iterates) is **crate-private**: tile width and
edge representation are executor detail, not API. The structural claim is pinned by the
in-crate test `exec::extent_tile_tests`; the benchmark reports touched words from the
semantic span (`touched_words`), which that test proves the plan covers exactly.

## The absolute-coordinate law (CURRENT-CONTRACT, TEST-PINNED)
The extent is an OUTER restriction in the same row coordinates as `Planes`. A program
`Range [1000, 2000)` executed over the extent `[1500, 1700)` means
`[1000, 2000) ∩ [1500, 1700)`. Lane element `r` is row `r` whatever the extent, and tile word
`w` is word `w` of every resident mask. Nothing is rebased, copied or renumbered.

**Implementation:**
- **Word bounds:** only `span_words(lo, hi)` is walked, the same law `touched_words`
now delegates to.
- **Tile start/stop:** a word the extent cuts (an unaligned `lo`, or an unaligned `hi` short
of `n_rows`) is its own one-word tile.
- **Edge masks:** on that tile the terminal's mask is ANDed with the in-extent bits in a
one-word register temporary. For `All`, the out-of-extent bits are ORed in instead, since
set bits are its identity.
- **No lane offsets:** lanes are read at `w * 64` exactly as before.
- **No copies:** no lane or mask copy was required.

The #1268 fused fold composes by intersection
(`program_range ∩ extent ∩ resident_plane`) inside the same seam. There is no second
evaluator.

## What execution now skips / what still materializes
- **Skipped:** every tile outside the extent. A 1-row extent over 1M rows visits one word.
- **Still materializes, deliberately:**
- `Keep` writes its population-addressed `Out::Mask`, but only the in-extent bits: edge
words are merged bit-exactly, and every other bit stays as the caller holds it. Disjoint
extents therefore compose into one absolute buffer in any **sequential** order.
- **Not a concurrency licence (CURRENT-CONTRACT).** An unaligned split such as
`[0, 65)` + `[65, N)` puts both extents in the same physical `u64`, and the edge merge
is a read-modify-write. Partial `Keep` sinks compose in any sequential order; concurrent
execution requires word-disjoint sink ownership (boundaries on multiples of 64),
separate partial sinks plus a merge, or another explicitly synchronized strategy. A
future scheduler must not infer two simultaneous writers on one `Out::Mask` from this.
- The tiled (non-fused) path still writes tile-local scratch, but only for touched tiles.
- **Refused on a partial extent**, as `ExtentUnsupported`, because there is no shipped merge
law or disjoint sink here: `BlendI32`, `ScatterOrU32`, `ScatterCountU32`,
`CountKeyRunsU32`, `GroupSumI32`, `GroupSumViaI32`, `GroupReduce`. `_sym` stays a recorded
law only (`TD-SYM-SUM-MERGE-IS-NOT-ADDITION-1`).
- **Foreign planes and lanes** (`Gather`, `GroupKey::Via`) are addressed by key and are never
sliced. That is pinned by a test whose extent rows all name foreign keys below the extent's
own row numbers.

## Split/merge (TEST-PINNED, `tests/extent.rs`)
- **Terminals:** Count (+), Any (∨), All (∧), MaskedSumI32 (+), MaskedMin/Max (min/max),
each over the fused and tiled shapes.
- **Partitions:**
- two-way at k ∈ {0, 1, 63, 64, 65, 127, 128, 129, N/2, N−1, N};
- 40 random three-way partitions;
- N ∈ {1317, 4133}, neither a multiple of 64.
- **Orders:** every partition is merged in forward, reverse and rotated order, and each equals
whole-population execution. `Keep` partials written in forward and reverse order into one
buffer equal the whole `Keep`.

## Non-rebasing falsifier
Distinct lane values sit at rows 63, 64, 65, 127, 128 and 129.
- `MaskedSumI32` over extents starting at 65, 129, 63 and 127 must equal the absolute-row sum.
- The test also asserts that the worker-local sum differs, so a rebasing executor cannot pass.
- A second test pins program `Range [1000,2000)` over extent `[1500,1700)` as exactly 200
rows, and `Keep` writes exactly bits 1500..1700.

## Disable runs (committed first, then each restored)
| disable | tests red |
|---|---|
| tiles walk from row 0 instead of the extent | 7 of 8, incl. the structural tile gate (since moved in-crate as `exec::extent_tile_tests`) |
| edge word not restricted | 6 |
| lanes read worker-local (`r0` relative to the extent) | 5, incl. the absolute-rows falsifier |
| fused fold ignores the extent | 4 |
| `Keep` overwrites the whole edge word | 3 |

## Benchmarks
Run with `cargo run --release -p lance-graph-mask-risc --example extent_probe`: N = 1,048,576
rows, a 2/3-dense plane and unaligned extents. Every result is checked against a scalar
oracle over the same absolute rows.

| shape | extent | median ns | population words read | lane elements | derived words written |
|---|---|---|---|---|---|
| fused Range∩plane→Count | 1 row | 62 | 1 | 0 | 0 |
| fused | 512 rows | 70 | 9 | 0 | 0 |
| fused | 25% | 1,172 | 4,097 | 0 | 0 |
| fused | whole | 4,102 | 16,354 | 0 | 0 |
| tiled Range→And→Count | 1 row | 161 | 1 | 0 | 2 |
| tiled | 1% | 2,820 | 165 | 0 | 330 |
| tiled | whole | 158,051 | 16,384 | 0 | 32,768 |
| lane EqU32 under plane→MaskedSumI32 | 1 row | 172 | 1 | 128 | 1 |
| lane | 25% | 379,509 | 4,097 | 524,416 | 4,097 |
| lane | whole | 1,828,878 | 16,384 | 2,097,152 | 16,384 |

- Cost follows extent width, not `n_rows`: a 1-row extent is 62–172 ns on every shape.
- The word columns come from the extent's semantic span (`touched_words`), which the in-crate
tile-plan test pins the executor to cover exactly; not from instrumentation.

## Correction of #1267 (append-only)
`2026-09-23-cubecl-llvm-boundary-and-audit-regrade.md` lesson 1 says the ranged entry point
needs "rebasing every mask and lane for ranges that do not start on a word boundary".
**The required property is correct absolute execution over a non-zero, non-word-aligned
extent, not physical rebasing.** It is met with word bounds and register edge masks, and
rebasing would have broken the absolute-coordinate law above.

## What remains before scheduling
- OQ-5 (thread/rayon vendor) is untouched and not resolved.
- No scheduler, queue, thread or affinity exists, and no scheduler work has started.
- Merge laws are still missing for the group terminals and the `_sym` SUM, which are refused on
partial extents.
- The next expression→terminal folds (`plane ∩ plane → Count`, lane predicate → Count,
ternlog → Count) would make an extent-dispatched unit a small algebraic program rather than a
tile-writing bitmap machine. They are not started.
3 changes: 2 additions & 1 deletion .claude/board/entries/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,14 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately
opposite directions; the stranding this convention prevents shows up in
exactly one of them, never both.

150 entries, 2026-08-06 .. 2026-09-23.
151 entries, 2026-08-06 .. 2026-09-23.

| date | entry id | finding | file |
|---|---|---|---|
| 2026-09-23 | `terminal-elects-materialization-range-plane` | | [2026-09-23-terminal-elects-materialization-range-plane.md](2026-09-23-terminal-elects-materialization-range-plane.md) |
| 2026-09-23 | `quack-having-sym-sum-presence-mask` | | [2026-09-23-quack-having-sym-sum-presence-mask.md](2026-09-23-quack-having-sym-sum-presence-mask.md) |
| 2026-09-23 | `cubecl-llvm-boundary-and-audit-regrade` | | [2026-09-23-cubecl-llvm-boundary-and-audit-regrade.md](2026-09-23-cubecl-llvm-boundary-and-audit-regrade.md) |
| 2026-09-23 | `absolute-execution-extent` | | [2026-09-23-absolute-execution-extent.md](2026-09-23-absolute-execution-extent.md) |
| 2026-09-22 | `quack-duckdb-parity-t0-keyed-reduction` | | [2026-09-22-quack-duckdb-parity-t0-keyed-reduction.md](2026-09-22-quack-duckdb-parity-t0-keyed-reduction.md) |
| 2026-09-22 | `E-W0C-THE-ROW-BRIDGE-IS-A-DIALECT-NOT-AN-INTERPRETER-1` | a merged relational op carried as loco program data reaches the fused executor with no population crossing; the enum explosion is upstream of mask-risc | [2026-09-22-e-w0c-the-row-bridge-is-a-dialect-not-an-interpreter-1.md](2026-09-22-e-w0c-the-row-bridge-is-a-dialect-not-an-interpreter-1.md) |
| 2026-09-22 | `E-CATS-FOLD-DOES-NOT-RETAIN-A-POPULATION-BITMAP-1` | CATS aggregate lowers to one tiled grouped terminal; bitmap realization is a requested boundary sink | [2026-09-22-e-cats-fold-does-not-retain-a-population-bitmap-1.md](2026-09-22-e-cats-fold-does-not-retain-a-population-bitmap-1.md) |
Expand Down
173 changes: 173 additions & 0 deletions crates/lance-graph-mask-risc/examples/extent_probe.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
//! Dispatch locality of an absolute execution extent at N = 1M rows.
//!
//! One program family, three physical shapes, each run over extents of
//! 1 row, 64 rows, 512 rows, 1 %, 25 % and the whole population:
//!
//! - FUSED: `Range ∩ plane → Count` (the #1268 fold, intersected with the
//! extent — writes nothing);
//! - TILED: `Range → And(plane) → Count` (writes two scratch slots per tile);
//! - LANE: `EqU32(lane) under plane → MaskedSumI32` (reads two value lanes).
//!
//! Reported separately: median latency, population words visited, lane
//! elements visited, and derived words written. The word counts are DERIVED
//! from the extent's semantic span (`touched_words`), which the executor's
//! tile plan is test-pinned to cover exactly — not guessed; every extent result is cross-checked
//! against a scalar oracle over the same absolute rows.
//!
//! `cargo run --release -p lance-graph-mask-risc --example extent_probe`

use std::time::Instant;

use lance_graph_mask_risc::exec::{execute_extent, Scratch};
use lance_graph_mask_risc::{
touched_words, Foreign, LaneRef, MaskOp, Operand, Out, Planes, Pred, Program, Terminal, Value,
TILE_WORDS,
};

fn lcg(seed: &mut u64) -> u64 {
*seed = seed
.wrapping_mul(6364136223846793005)
.wrapping_add(1442695040888963407);
*seed >> 11
}

fn median_ns(mut f: impl FnMut() -> Value, reps: usize) -> (f64, Value) {
let mut times = Vec::with_capacity(reps);
let mut last = Value::Count(0);
for _ in 0..reps {
let t = Instant::now();
last = std::hint::black_box(f());
times.push(t.elapsed().as_nanos() as f64);
}
times.sort_by(|a, b| a.total_cmp(b));
(times[reps / 2], last)
}

fn main() {
let n = 1usize << 20;
let mut seed = 0xE7E_u64;
let bits: Vec<bool> = (0..n).map(|_| !lcg(&mut seed).is_multiple_of(3)).collect();
let mut pl = vec![0u64; n.div_ceil(64)];
for (r, &b) in bits.iter().enumerate() {
if b {
pl[r / 64] |= 1 << (r % 64);
}
}
let keys: Vec<u32> = (0..n).map(|_| (lcg(&mut seed) % 8) as u32).collect();
let vals: Vec<i32> = (0..n).map(|_| (lcg(&mut seed) % 1000) as i32).collect();
let masks: [&[u64]; 1] = [&pl];
let lanes = [LaneRef::U32(&keys), LaneRef::I32(&vals)];
let planes = Planes {
n_rows: n,
masks: &masks,
lanes: &lanes,
};
let (plo, phi) = (1000u32, n as u32 - 1000);
let fused = Program::new(
vec![MaskOp::Pred {
pred: Pred::Range { lo: plo, hi: phi },
under: Some(Operand::Plane(0)),
dst: 0,
}],
Terminal::Count {
mask: Operand::Scratch(0),
},
);
let tiled = Program::new(
vec![
MaskOp::Pred {
pred: Pred::Range { lo: plo, hi: phi },
under: None,
dst: 0,
},
MaskOp::And {
a: Operand::Scratch(0),
b: Operand::Plane(0),
dst: 1,
},
],
Terminal::Count {
mask: Operand::Scratch(1),
},
);
let lane = Program::new(
vec![MaskOp::Pred {
pred: Pred::EqU32 { lane: 0, v: 3 },
under: Some(Operand::Plane(0)),
dst: 0,
}],
Terminal::MaskedSumI32 {
mask: Operand::Scratch(0),
lane: 1,
},
);
let mid = n / 2 + 17; // deliberately unaligned
let extents: [(&str, usize, usize); 6] = [
("1 row", mid, mid + 1),
("64 rows", mid, mid + 64),
("512 rows", mid, mid + 512),
("1%", mid, mid + n / 100),
("25%", n / 5 + 3, n / 5 + 3 + n / 4),
("whole", 0, n),
];
println!(
"{:>6} {:>9} {:>10} {:>12} {:>12} {:>14}",
"shape", "extent", "median_ns", "words_read", "lane_elems", "derived_wr"
);
for (shape, p, slots) in [
("fused", &fused, 0usize),
("tiled", &tiled, 2),
("lane", &lane, 1),
] {
let mut s = Scratch::for_program(p, n).expect("scratch");
for (name, lo, hi) in extents {
// Scalar oracle over the same ABSOLUTE rows.
let want = match shape {
"lane" => Value::SumI64(
(lo..hi)
.filter(|&r| bits[r] && keys[r] == 3)
.map(|r| i64::from(vals[r]))
.sum(),
),
_ => Value::Count(
(lo..hi)
.filter(|&r| bits[r] && (plo as usize..phi as usize).contains(&r))
.count(),
),
};
let reps = if hi - lo > 100_000 { 31 } else { 2001 };
let (ns, got) = median_ns(
|| {
execute_extent(p, &planes, &Foreign::NONE, &mut s, Out::None, lo..hi)
.expect("extent")
},
reps,
);
assert_eq!(got, want, "{shape} {name}");
// Words the extent touches: the semantic span, which the executor's
// tile plan covers exactly (pinned in-crate by `extent_tile_tests`).
let tiles_words = touched_words(lo as u32, hi as u32).len();
let (words_read, lane_elems, derived) = match shape {
"fused" => {
let (a, b) = ((plo as usize).max(lo), (phi as usize).min(hi));
let span = if a < b {
touched_words(a as u32, b as u32).len()
} else {
0
};
(span, 0, 0)
}
"tiled" => (tiles_words, 0, slots * tiles_words),
_ => (tiles_words, 2 * tiles_words * 64, slots * tiles_words),
};
println!(
"{shape:>6} {name:>9} {ns:>10.0} {words_read:>12} {lane_elems:>12} {derived:>14}"
);
}
}
println!(
"(n = {n}, {} population words, scratch tile = {} words)",
n / 64,
TILE_WORDS
);
}
Loading
Loading