lgj-abi minor 12: the grouped sum that never builds a selection (fold-distillation wave 4)

.claude/board/LATEST_STATE.md

@@ -1,3 +1,35 @@
+## 2026-09-22 — minor 12: `lgj_plan_group_sum_i32`, the grouped sum that never builds a selection (fold-distillation wave 4)
+
+Upstream first (lance-graph #1256 merged `99cdca38`: the tiled executor,
+`Terminal::GroupSumI32`/`GroupSumViaI32`, `Out::I64`; ndarray #318 merged:
+the T1 kernels; lance-graph-java #83 merged: the status arms). This is the
+Java→Panama→mask-risc proof that wave: a `GROUP BY` crosses once and no
+selection exists at any point.
+
+- **ABI:** ONE new symbol, no new status, no manifest growth — the plan
+  surface is reused; only the terminal changed (`Keep` → `GroupSumI32`, or
+  `GroupSumViaI32` when `via_res != 0`). `n_ops == 0` is legal here (the
+  whole-lane `Range`, the one `Range` this crate emits, pinned to be exactly
+  `0..n_rows`). `docs/abi.md` §20.
+- **Measured through the membrane:** `View.sumByGroup` = **1 crossing** at
+  1,024 and 65,536 rows for 16 groups; the two-crossing-per-group path it
+  replaces measured **32** beside it in the same run (the `bricks` number,
+  reproduced). `sumByGroupVia` (the fk-keyed form, `SUM(line) GROUP BY
+  partner.key`) also 1.
+- **Gates:** native **191** tests (+8), clippy `-D warnings` + fmt clean;
+  Java **612** checks (409 + `GroupSumTest` 203) under JDK 28
+  `--enable-preview` against `abi 0.12`; six disable arms red-then-green
+  (§20.6). `OldAbiCompatTest` gains a minor-12 leg, run BOTH ways: 13/13
+  against this library, and against a minor-11 library built from `07e044f`
+  `View.sumByGroup` throws `AbiMismatchException` naming minor 12 — never a
+  missing-symbol failure.
+- **The eighth named materialisation site:** `Engine.groupSumI32`'s
+  `toArray`, sized by `groups` (the question), pinned in `DoctrineFenceTest`
+  and listed in root `CLAUDE.md`. `GroupTotals` exposes no array.
+- **Still owed:** the `bricks` consumer's `sumBy()` still runs the 32-crossing
+  path; migrating it to `sumByGroup` is a consumer-wave change, not this
+  one. `lgj_hop` still holds mask-sized Vecs (pre-existing, unrelated).
+
 ## 2026-09-19 — PR #81 merged (`07aa441`): production IS the Valhalla arm; Panama × Valhalla is one membrane
 
 **The frame, because it is easy to file this wrong:** this was not a JDK

.claude/board/STATUS_BOARD.md

@@ -118,3 +118,18 @@ layout wired end to end. Doctrine: `E-LGJ-THE-MIDDLE-TIER-IS-DELETED-NOT-WRAPPED
 | D-LGJ-W5 | Three consumer examples (trades / bricks / graph) — one plan file each | **trades DONE 2026-08-17** — `consumers/trades/` (own compile unit, core consumed as a third-party would): `Trade` (schema-not-entity: zero public ctors, zero instance fields, reflection-forced construction still throws), `World.open` → the existing lazy `View` under domain names, zero new membrane surface. TradesParityTest 12/12 (chain vs transcribed-generator recomputation at 1K+64K rows; 0 crossings composing / 1 at terminal THROUGH the domain vocabulary; reflection guard). TradesAllocationTest 3/3 — **the poster's number, measured: 240 bytes/query, IDENTICAL at 64K and 1M rows** (row-count independence is the thesis assertion; 64 KiB absolute backstop). Disable-run: VENUE pointed at the wrong lane → the membrane's own LANE_KIND_MISMATCH rejected it (the binding is checked, not trusted); restored green. **bricks DONE 2026-08-17** — `consumers/bricks/` (2 Sonnet workers K1/K2 per `.claude/waves/wave-consumer-bricks.md`): mask-first RBAC where `authorize(Role)` is a real natively-evaluated predicate in the SAME lazy chain as `where(...)` (`Role.EU_ONLY` = `REGION.eq(EU)`, `DENY_ALL` = `REGION.eq(0xFFFF)` — a genuine impossible predicate, not a Java branch), fail-closed (`UnauthorizedQueryException` BEFORE any crossing; no default-allow path exists), aggregate-only egress (every public method returns `BricksQuery`/`long`/`Map` — structurally no row-shaped type). BricksAuthTest **62/62**: parity vs transcribed generator at 1K+64K; RBAC-as-predicate equivalence (EU_ONLY result == GLOBAL+explicit-where); DENY_ALL counts 0 while paying a real crossing; crossing arithmetic — count()=1, sumBy()=**32 crossings (16 groups × 2: plan_eval + lgj_reduce_sum_i32), IDENTICAL at both row counts** (the thesis: crossings ∝ groups, never rows — the measured 32 corrected K1's "1 per group" Javadoc claim, a real finding about sum-terminal cost); reflection guards. Disable-run: `requireAuthorized` short-circuited → **exactly the 3 can-fire fail-closed checks red, 59 green**; restored, 62/62. Core suite unaffected (188/188). **graph DONE 2026-08-18** — `consumers/graph/` (2 Sonnet workers G1/G2 per `.claude/waves/wave-consumer-graph.md`, dispatched only after the substrate was proven complete at all three levels: generator, ABI, public facade — D-LGJ-W6/W7): `Graph`/`Edge` (schema-not-entity `Edge`, immutable-chaining `Graph` — `from`/`hop`/`minus` each return a NEW `Graph`, mirroring `BricksQuery`'s shape rather than `View`'s laziness since every step but `hop` is already zero-cost). The row-set currency is a Java-side `long[]`, not a native `Mask` — checked and confirmed no public mask-from-row-indices constructor exists; ruled as a deliberate, documented simplification (D1 in the wave file) rather than building a FOURTH core-facade capability under time pressure. `GraphHopTest` **43/43**: hop correctness against the pinned regression (19 @ 1 hop, 29 @ 2 hops) via TWO independently-written pure-Java BFS transcriptions that never call into `Graph`; anti-vacuity; zero-serialization (structural + a reflective public-surface type check); `Edge`'s reflection guard. Disable-run: the target-decode offset corrupted by +4 → hop correctness went red exactly as required (a set-equality check caught it even though the coincidental row COUNT still matched — vindicating G2's choice to assert set equality, not just size). Core suite (204/204) and both prior consumers (trades 12+3/12+3, bricks 62/62) unaffected. **A real measured finding caught and fixed before landing, not shipped wrong:** G2's first draft asserted every hop costs an identical number of crossings; measured, hop 1 on a fresh store costs 2 (the `facetMatches` crossing plus a one-time `RowStore.rawLane()` resolution its first payload read triggers) while hop 2 onward costs exactly 1, steady-state — confirmed directly across 4 consecutive hops before touching the shipped test. Both `Graph.hop()`'s javadoc and `GraphHopTest`'s crossing assertions were corrected to state the true, now-precisely-measured relationship instead of the wrong "identical every hop" assumption |
 | D-LGJ-W6 | Edge-bearing row store ABI addition (`lgj_rowstore_open_with_edges`, minor 2→3, docs/abi.md §12) — the D1b-shaped "must land as its own W-tier PR before the consumer wave" the graph wave itself named | **DONE 2026-08-18** — orchestrator-authored (genuinely new ABI surface, not consumer-scope work): `registry::open_rowstore_with_edges` + `lgj_rowstore_open_with_edges` (mirrors `lgj_rowstore_open` exactly: same resource kind, same lane shape, no new mask op — purely an alternative constructor), `Engine.openRowStoreWithEdges`/`Abi.requireMinor(3)`, `RowStore.openWithEdges`. `cargo test` **93/93** (+3: registry-level open/describe, out-of-range-classid-matches-plain, radius-overflow-rejected), clippy/fmt clean, release build exports the new symbol (`nm -D`). Java: `AllTests` **194/194** (+6, all in `RowStoreParityTest`) — the strongest new result is a cross-language reproduction of the D1a hop mechanism itself: Java facet-matches + raw-lane-0 payload decode (zero new ABI op) reaches the EXACT same measured hop counts already pinned as a Rust regression (10-row seed → 19 at 1 hop → 29 at 2 hops, `n=2000, seed=0xF00D_CAFE, edge_classid=0, gate_mask=0x0, radius=25`) — proving the two sides of the membrane see identical edge structure, not merely identical classids. Two disable-runs, both red-then-green: (1) registry-level, a classid-not-threaded bug (`open_rowstore_with_edges` hardcoded classid `0`) caught by the out-of-range-parity test; (2) Java-level, the hop's classid-match condition forced to always skip → 1-hop/2-hop both went to 0 and the anti-vacuity assertion failed, exactly as expected. Caught mid-dispatch: the ABI-facing symbol did not exist before this pass (only the bare `RowStore::generate_with_edges` Rust function did, from the prior session) — the graph wave's own STOP-condition-RESOLVED note undersold what was still missing; closed here rather than discovered by G1/G2 mid-flight |
 | D-LGJ-W7 | Core public facade: `RowStore.classidAt`/`payloadLow64At`/`payloadHi32At` — the per-row zero-copy escape hatch a real `Graph.hop()` needs, since neither `maskOfFacetClass` nor `facetMatches` exposes payload bytes and `ApiSurfaceTest` forbids `MemorySegment`/`internal.*` in any consumer-package signature | **DONE 2026-08-18** — found while checking, concretely, how `consumers/graph` (a genuinely external compile unit per every prior consumer wave's own convention) could ever decode a matched facet's target row: it couldn't — D1a's design assumed a zero-copy raw-lane read Java-side, but nothing on the PUBLIC `RowStore` facade exposed one; only `internal.ffm.Engine.describeLane` did, off-limits to a consumer package by construction. Fixed with THREE new primitive-returning `RowStore` methods, zero new ABI surface at all (reuses `lgj_lane_describe`, already ABI minor 1 — a "lifecycle" crossing per abi.md §6, resolved once and cached; every read after is in-process, matching exports.rs's own stated doctrine "if Java wants one row it reads the MemorySegment in-process, with no crossing at all"). `AllTests` **204/204** (+10 over D-LGJ-W6: 6 in `RowStoreParityTest` — the SAME pinned hop numbers (19/29) reproduced a SECOND time, this time through the genuinely public path a real consumer has to use, plus bounds checks; 6 in `RowStoreLifetimeTest` — closed-before-first-read and closed-after-caching, both guarded). **A real self-caught redundancy, not shipped silently**: the first draft carried the closed-store guard in TWO places (`rawLane()` and `rowOffset()`); disabling one was masked by the other and produced a false-negative disable-run (30/30 green under a broken guard) — caught by re-reading WHY the disable didn't fire rather than accepting the green result, traced to Java method-call evaluation order (`rawLane()`, the receiver, evaluates before `rowOffset()`, its argument), de-duplicated to the ONE correct location, disable-run re-run and confirmed genuinely red-then-green. `ApiSurfaceTest` still 3/3 — zero FFM-typed public signature introduced. **The graph-consumer wave is now dispatchable for real**, proven at three independent levels: Rust generator (D-LGJ-W5's own entry below), ABI membrane (D-LGJ-W6), and the public core facade a consumer package can actually compile against (this row) |
+
+---
+
+## fold-distillation wave — the Java → Panama → mask-risc grouped sum (2026-09-22)
+
+Upstream substrate: lance-graph #1256 (tiled executor, `GroupSumI32` /
+`GroupSumViaI32` terminals, `Out::I64`), ndarray #318 (the T1 kernels), both
+merged; lance-graph-java #83 (status arms, merged). Consumer witness on the
+same lowering: lance-graph #1257 (SAP CATS, merged).
+
+| D-id | Deliverable | Status |
+|---|---|---|
+| D-LGJ-FOLD-4 | ABI minor 12 `lgj_plan_group_sum_i32` + `View.sumByGroup` / `sumByGroupVia` + `GroupTotals`: a `GROUP BY … SUM` as ONE program, one crossing, no selection (`docs/abi.md` §20) | **DONE 2026-09-22** — native 191 tests, Java 612 checks (JDK 28), 1 crossing measured beside the 32-crossing path, six disable arms red-then-green |
+| D-LGJ-FOLD-5 | Migrate `consumers/bricks` `sumBy()` from the 32-crossing per-group path onto `sumByGroup` (re-pin BricksAuthTest's crossing arithmetic 32 → 1) | Queued |
+

CLAUDE.md

@@ -283,10 +283,12 @@ build):
   not row count — verified fixed-size on both the Rust and Java sides);
   `Abi.java`'s `readCarvings` (bounded by `CARVING_SLOTS`, a manifest
   constant, not n_rows); `Engine.facetSumResolved`'s fixed `long[2]`
-  result pair; `View.where()`'s `List.copyOf` of the PREDICATE chain; and
-  `NativePattern.plan()`'s `predicates.stream()…toList()`. None of the
-  seven is a hidden proportional-to-n_rows population copy — keep this list
-  exhaustive when an eighth site is added, rather than letting the
+  result pair; `View.where()`'s `List.copyOf` of the PREDICATE chain;
+  `NativePattern.plan()`'s `predicates.stream()…toList()`; and
+  `Engine.groupSumI32`'s `toArray` of the group totals (minor 12 — sized by
+  `groups`, the key domain the caller asked for, never by rows). None of the
+  eight is a hidden proportional-to-n_rows population copy — keep this list
+  exhaustive when a ninth site is added, rather than letting the
   enumeration silently go stale again.
 
   > **⊘ RE-AUDITED 2026-09-16 and the list WAS stale — it claimed five and