diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 3c59d8f..98d593d 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -1,3 +1,35 @@ +## 2026-09-22 — minor 12: `lgj_plan_group_sum_i32`, the grouped sum that never builds a selection (fold-distillation wave 4) + +Upstream first (lance-graph #1256 merged `99cdca38`: the tiled executor, +`Terminal::GroupSumI32`/`GroupSumViaI32`, `Out::I64`; ndarray #318 merged: +the T1 kernels; lance-graph-java #83 merged: the status arms). This is the +Java→Panama→mask-risc proof that wave: a `GROUP BY` crosses once and no +selection exists at any point. + +- **ABI:** ONE new symbol, no new status, no manifest growth — the plan + surface is reused; only the terminal changed (`Keep` → `GroupSumI32`, or + `GroupSumViaI32` when `via_res != 0`). `n_ops == 0` is legal here (the + whole-lane `Range`, the one `Range` this crate emits, pinned to be exactly + `0..n_rows`). `docs/abi.md` §20. +- **Measured through the membrane:** `View.sumByGroup` = **1 crossing** at + 1,024 and 65,536 rows for 16 groups; the two-crossing-per-group path it + replaces measured **32** beside it in the same run (the `bricks` number, + reproduced). `sumByGroupVia` (the fk-keyed form, `SUM(line) GROUP BY + partner.key`) also 1. +- **Gates:** native **191** tests (+8), clippy `-D warnings` + fmt clean; + Java **612** checks (409 + `GroupSumTest` 203) under JDK 28 + `--enable-preview` against `abi 0.12`; six disable arms red-then-green + (§20.6). `OldAbiCompatTest` gains a minor-12 leg, run BOTH ways: 13/13 + against this library, and against a minor-11 library built from `07e044f` + `View.sumByGroup` throws `AbiMismatchException` naming minor 12 — never a + missing-symbol failure. +- **The eighth named materialisation site:** `Engine.groupSumI32`'s + `toArray`, sized by `groups` (the question), pinned in `DoctrineFenceTest` + and listed in root `CLAUDE.md`. `GroupTotals` exposes no array. +- **Still owed:** the `bricks` consumer's `sumBy()` still runs the 32-crossing + path; migrating it to `sumByGroup` is a consumer-wave change, not this + one. `lgj_hop` still holds mask-sized Vecs (pre-existing, unrelated). + ## 2026-09-19 — PR #81 merged (`07aa441`): production IS the Valhalla arm; Panama × Valhalla is one membrane **The frame, because it is easy to file this wrong:** this was not a JDK diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index ad272d7..1da2a74 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -118,3 +118,18 @@ layout wired end to end. Doctrine: `E-LGJ-THE-MIDDLE-TIER-IS-DELETED-NOT-WRAPPED | D-LGJ-W5 | Three consumer examples (trades / bricks / graph) — one plan file each | **trades DONE 2026-08-17** — `consumers/trades/` (own compile unit, core consumed as a third-party would): `Trade` (schema-not-entity: zero public ctors, zero instance fields, reflection-forced construction still throws), `World.open` → the existing lazy `View` under domain names, zero new membrane surface. TradesParityTest 12/12 (chain vs transcribed-generator recomputation at 1K+64K rows; 0 crossings composing / 1 at terminal THROUGH the domain vocabulary; reflection guard). TradesAllocationTest 3/3 — **the poster's number, measured: 240 bytes/query, IDENTICAL at 64K and 1M rows** (row-count independence is the thesis assertion; 64 KiB absolute backstop). Disable-run: VENUE pointed at the wrong lane → the membrane's own LANE_KIND_MISMATCH rejected it (the binding is checked, not trusted); restored green. **bricks DONE 2026-08-17** — `consumers/bricks/` (2 Sonnet workers K1/K2 per `.claude/waves/wave-consumer-bricks.md`): mask-first RBAC where `authorize(Role)` is a real natively-evaluated predicate in the SAME lazy chain as `where(...)` (`Role.EU_ONLY` = `REGION.eq(EU)`, `DENY_ALL` = `REGION.eq(0xFFFF)` — a genuine impossible predicate, not a Java branch), fail-closed (`UnauthorizedQueryException` BEFORE any crossing; no default-allow path exists), aggregate-only egress (every public method returns `BricksQuery`/`long`/`Map` — structurally no row-shaped type). BricksAuthTest **62/62**: parity vs transcribed generator at 1K+64K; RBAC-as-predicate equivalence (EU_ONLY result == GLOBAL+explicit-where); DENY_ALL counts 0 while paying a real crossing; crossing arithmetic — count()=1, sumBy()=**32 crossings (16 groups × 2: plan_eval + lgj_reduce_sum_i32), IDENTICAL at both row counts** (the thesis: crossings ∝ groups, never rows — the measured 32 corrected K1's "1 per group" Javadoc claim, a real finding about sum-terminal cost); reflection guards. Disable-run: `requireAuthorized` short-circuited → **exactly the 3 can-fire fail-closed checks red, 59 green**; restored, 62/62. Core suite unaffected (188/188). **graph DONE 2026-08-18** — `consumers/graph/` (2 Sonnet workers G1/G2 per `.claude/waves/wave-consumer-graph.md`, dispatched only after the substrate was proven complete at all three levels: generator, ABI, public facade — D-LGJ-W6/W7): `Graph`/`Edge` (schema-not-entity `Edge`, immutable-chaining `Graph` — `from`/`hop`/`minus` each return a NEW `Graph`, mirroring `BricksQuery`'s shape rather than `View`'s laziness since every step but `hop` is already zero-cost). The row-set currency is a Java-side `long[]`, not a native `Mask` — checked and confirmed no public mask-from-row-indices constructor exists; ruled as a deliberate, documented simplification (D1 in the wave file) rather than building a FOURTH core-facade capability under time pressure. `GraphHopTest` **43/43**: hop correctness against the pinned regression (19 @ 1 hop, 29 @ 2 hops) via TWO independently-written pure-Java BFS transcriptions that never call into `Graph`; anti-vacuity; zero-serialization (structural + a reflective public-surface type check); `Edge`'s reflection guard. Disable-run: the target-decode offset corrupted by +4 → hop correctness went red exactly as required (a set-equality check caught it even though the coincidental row COUNT still matched — vindicating G2's choice to assert set equality, not just size). Core suite (204/204) and both prior consumers (trades 12+3/12+3, bricks 62/62) unaffected. **A real measured finding caught and fixed before landing, not shipped wrong:** G2's first draft asserted every hop costs an identical number of crossings; measured, hop 1 on a fresh store costs 2 (the `facetMatches` crossing plus a one-time `RowStore.rawLane()` resolution its first payload read triggers) while hop 2 onward costs exactly 1, steady-state — confirmed directly across 4 consecutive hops before touching the shipped test. Both `Graph.hop()`'s javadoc and `GraphHopTest`'s crossing assertions were corrected to state the true, now-precisely-measured relationship instead of the wrong "identical every hop" assumption | | D-LGJ-W6 | Edge-bearing row store ABI addition (`lgj_rowstore_open_with_edges`, minor 2→3, docs/abi.md §12) — the D1b-shaped "must land as its own W-tier PR before the consumer wave" the graph wave itself named | **DONE 2026-08-18** — orchestrator-authored (genuinely new ABI surface, not consumer-scope work): `registry::open_rowstore_with_edges` + `lgj_rowstore_open_with_edges` (mirrors `lgj_rowstore_open` exactly: same resource kind, same lane shape, no new mask op — purely an alternative constructor), `Engine.openRowStoreWithEdges`/`Abi.requireMinor(3)`, `RowStore.openWithEdges`. `cargo test` **93/93** (+3: registry-level open/describe, out-of-range-classid-matches-plain, radius-overflow-rejected), clippy/fmt clean, release build exports the new symbol (`nm -D`). Java: `AllTests` **194/194** (+6, all in `RowStoreParityTest`) — the strongest new result is a cross-language reproduction of the D1a hop mechanism itself: Java facet-matches + raw-lane-0 payload decode (zero new ABI op) reaches the EXACT same measured hop counts already pinned as a Rust regression (10-row seed → 19 at 1 hop → 29 at 2 hops, `n=2000, seed=0xF00D_CAFE, edge_classid=0, gate_mask=0x0, radius=25`) — proving the two sides of the membrane see identical edge structure, not merely identical classids. Two disable-runs, both red-then-green: (1) registry-level, a classid-not-threaded bug (`open_rowstore_with_edges` hardcoded classid `0`) caught by the out-of-range-parity test; (2) Java-level, the hop's classid-match condition forced to always skip → 1-hop/2-hop both went to 0 and the anti-vacuity assertion failed, exactly as expected. Caught mid-dispatch: the ABI-facing symbol did not exist before this pass (only the bare `RowStore::generate_with_edges` Rust function did, from the prior session) — the graph wave's own STOP-condition-RESOLVED note undersold what was still missing; closed here rather than discovered by G1/G2 mid-flight | | D-LGJ-W7 | Core public facade: `RowStore.classidAt`/`payloadLow64At`/`payloadHi32At` — the per-row zero-copy escape hatch a real `Graph.hop()` needs, since neither `maskOfFacetClass` nor `facetMatches` exposes payload bytes and `ApiSurfaceTest` forbids `MemorySegment`/`internal.*` in any consumer-package signature | **DONE 2026-08-18** — found while checking, concretely, how `consumers/graph` (a genuinely external compile unit per every prior consumer wave's own convention) could ever decode a matched facet's target row: it couldn't — D1a's design assumed a zero-copy raw-lane read Java-side, but nothing on the PUBLIC `RowStore` facade exposed one; only `internal.ffm.Engine.describeLane` did, off-limits to a consumer package by construction. Fixed with THREE new primitive-returning `RowStore` methods, zero new ABI surface at all (reuses `lgj_lane_describe`, already ABI minor 1 — a "lifecycle" crossing per abi.md §6, resolved once and cached; every read after is in-process, matching exports.rs's own stated doctrine "if Java wants one row it reads the MemorySegment in-process, with no crossing at all"). `AllTests` **204/204** (+10 over D-LGJ-W6: 6 in `RowStoreParityTest` — the SAME pinned hop numbers (19/29) reproduced a SECOND time, this time through the genuinely public path a real consumer has to use, plus bounds checks; 6 in `RowStoreLifetimeTest` — closed-before-first-read and closed-after-caching, both guarded). **A real self-caught redundancy, not shipped silently**: the first draft carried the closed-store guard in TWO places (`rawLane()` and `rowOffset()`); disabling one was masked by the other and produced a false-negative disable-run (30/30 green under a broken guard) — caught by re-reading WHY the disable didn't fire rather than accepting the green result, traced to Java method-call evaluation order (`rawLane()`, the receiver, evaluates before `rowOffset()`, its argument), de-duplicated to the ONE correct location, disable-run re-run and confirmed genuinely red-then-green. `ApiSurfaceTest` still 3/3 — zero FFM-typed public signature introduced. **The graph-consumer wave is now dispatchable for real**, proven at three independent levels: Rust generator (D-LGJ-W5's own entry below), ABI membrane (D-LGJ-W6), and the public core facade a consumer package can actually compile against (this row) | + +--- + +## fold-distillation wave — the Java → Panama → mask-risc grouped sum (2026-09-22) + +Upstream substrate: lance-graph #1256 (tiled executor, `GroupSumI32` / +`GroupSumViaI32` terminals, `Out::I64`), ndarray #318 (the T1 kernels), both +merged; lance-graph-java #83 (status arms, merged). Consumer witness on the +same lowering: lance-graph #1257 (SAP CATS, merged). + +| D-id | Deliverable | Status | +|---|---|---| +| D-LGJ-FOLD-4 | ABI minor 12 `lgj_plan_group_sum_i32` + `View.sumByGroup` / `sumByGroupVia` + `GroupTotals`: a `GROUP BY … SUM` as ONE program, one crossing, no selection (`docs/abi.md` §20) | **DONE 2026-09-22** — native 191 tests, Java 612 checks (JDK 28), 1 crossing measured beside the 32-crossing path, six disable arms red-then-green | +| D-LGJ-FOLD-5 | Migrate `consumers/bricks` `sumBy()` from the 32-crossing per-group path onto `sumByGroup` (re-pin BricksAuthTest's crossing arithmetic 32 → 1) | Queued | + diff --git a/CLAUDE.md b/CLAUDE.md index 708afa1..484446e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -283,10 +283,12 @@ build): not row count — verified fixed-size on both the Rust and Java sides); `Abi.java`'s `readCarvings` (bounded by `CARVING_SLOTS`, a manifest constant, not n_rows); `Engine.facetSumResolved`'s fixed `long[2]` - result pair; `View.where()`'s `List.copyOf` of the PREDICATE chain; and - `NativePattern.plan()`'s `predicates.stream()…toList()`. None of the - seven is a hidden proportional-to-n_rows population copy — keep this list - exhaustive when an eighth site is added, rather than letting the + result pair; `View.where()`'s `List.copyOf` of the PREDICATE chain; + `NativePattern.plan()`'s `predicates.stream()…toList()`; and + `Engine.groupSumI32`'s `toArray` of the group totals (minor 12 — sized by + `groups`, the key domain the caller asked for, never by rows). None of the + eight is a hidden proportional-to-n_rows population copy — keep this list + exhaustive when a ninth site is added, rather than letting the enumeration silently go stale again. > **⊘ RE-AUDITED 2026-09-16 and the list WAS stale — it claimed five and diff --git a/docs/abi.md b/docs/abi.md index 613f0e4..a58ce1e 100644 --- a/docs/abi.md +++ b/docs/abi.md @@ -62,8 +62,9 @@ cannot disagree with itself. The ABI is a **machine membrane**. It is not the product. The product is the Java semantic API (see `architecture.md`). Therefore: -- It is **small** — currently 29 symbols (minor 11's three additions are argued - in §19, and the same section argues the SEVEN capabilities it deliberately did +- It is **small** — currently 30 symbols (minor 12's one addition — the grouped + sum that never builds a selection — is argued in §20; 29 at minor 11, whose three + additions are argued in §19, and the same section argues the SEVEN capabilities it deliberately did NOT spend a symbol on; 26 at minor 10, whose one addition — the columnar constructor — is argued in §18; 25 at minor 9, whose one addition is argued in §11: a reduction Java was performing on the wrong side of the membrane, @@ -85,7 +86,7 @@ semantic API (see `architecture.md`). Therefore: ``` LGJ_ABI_MAJOR = 0 // incompatible change ⇒ bump; Java refuses to load -LGJ_ABI_MINOR = 11 // additive change ⇒ bump; older Java may still load +LGJ_ABI_MINOR = 12 // additive change ⇒ bump; older Java may still load LGJ_MAGIC = 0x4C_47_4A_5F_41_42_49_00 // "LGJ_ABI\0" big-endian-read ``` @@ -145,6 +146,17 @@ required — a gate that rejected everything would satisfy a rejection-only test ### Minor version history +- **Minor 12** (2026-09-22) — `lgj_plan_group_sum_i32` (§20): the fused plan + run STRAIGHT INTO a grouped-sum terminal. Every reduction before this minor + paid for a selection first — `lgj_plan_eval` into a mask, then a + `lgj_reduce_*` over it — and a `GROUP BY` paid it once per group (the + `bricks` consumer measured 32 crossings for 16). One symbol lowers the plan + and `GROUP BY key SUM(val)` as ONE `mask_risc::Program` whose accumulator is + tile-local scratch and whose only answer-sized state is the caller's `i64` + per group; a second table's key may be read THROUGH this table's key lane + (`via_res`/`via_lane`) with no partner-side mask. **One crossing for any + number of groups and any number of rows.** No new status; no manifest + growth; a minor-11 Java loads and sees none of it. - **Minor 11** (2026-09-14) — the masking-op completion (§19): the ndarray masking facade finished growing, and this minor consumes what it grew. **Three symbols and seven op-codes for fifteen capabilities**, which is the @@ -427,7 +439,7 @@ predicates or rows are involved. The unfused per-predicate ops are retained only so the fused path can be benchmarked *against* something and so parity can be checked predicate-by-predicate. -## 7. The function surface (29 symbols) +## 7. The function surface (30 symbols) All symbols are prefixed `lgj_`. All return `i32` status except the manifest getter. `out_*` parameters are written only on `OK`. @@ -529,6 +541,21 @@ i32 lgj_reduce_i32(u64 res, u32 lane_id, u32 reduce_op, u64 mask, Sums the `I32` lane over set mask bits into a widened `i64` (no overflow for `n_rows ≤ 2^32` on `i32` inputs). +### Grouped reduction (ABI minor ≥ 12) + +``` +i32 lgj_plan_group_sum_i32(u64 res, const LgjOpDesc* ops, u32 n_ops, + u32 group_lane, u32 val_lane, + u64 via_res, u32 via_lane, + i64* out_sums, u64 n_groups) // minor >= 12, §20 +``` + +`GROUP BY group_lane SUM(val_lane)` over the rows the plan selects, in ONE +crossing and ONE program — no selection is evaluated first and no mask handle +is involved. `out_sums[g]` for `g < n_groups`; a key past `n_groups` is +dropped. `n_ops == 0` is legal and means every row. `via_res != 0` reads the +group key THROUGH `group_lane` into `via_res`'s `via_lane` (the fk-keyed form). + ### Parity escape hatch ``` @@ -1667,3 +1694,112 @@ is textually ABSENT from the function body afterwards, not merely that a replacement occurred), and both then went red. The lesson is the one already on record and worth one more instance: assert what the disable REMOVED, never only that an edit landed. + +## 20. The grouped sum that never builds a selection (ABI minor ≥ 12) + +``` +i32 lgj_plan_group_sum_i32(u64 res, const LgjOpDesc* ops, u32 n_ops, + u32 group_lane, u32 val_lane, + u64 via_res, u32 via_lane, + i64* out_sums, u64 n_groups) +``` + +### 20.1 What was wrong before it + +Every reduction this ABI carried until minor 11 consumed a **mask**: `lgj_plan_eval` +landed the plan's answer in a mask handle, and `lgj_reduce_sum_i32` / +`lgj_reduce_i32` / the register sweeps read that handle. Java never HELD the +selection — it lived natively — but it was still a population-sized thing that +existed only to be consumed by the very next fold, which is the intermediate +materialisation the fold algebra exists to remove (lance-graph #1256's ruling: +*any intermediate population is prohibited if an addressable projection can be +consumed directly by the next fold*). A `GROUP BY` multiplied it by the group +count: `where(key.eq(g)).sumOf(value)` per `g`, measured in the `bricks` +consumer at **32 crossings for 16 groups**. + +### 20.2 What it does + +The plan is lowered exactly as `lgj_plan_eval` lowers it — the prefix rewrite +and the survivor skip of `plan_lower` — but the program ends in +`Terminal::GroupSumI32 { mask: acc, key, val }` instead of `Terminal::Keep`. The +executor runs it tile by tile over `SLOTS × tile_words_for(rows)` words of +scratch; for each tile the terminal folds `val[i]` into `out_sums[key[i]]` for +every selected row and moves on. Nothing population-sized exists at any point: +the accumulator is a tile, the sink is `n_groups` integers, and the crossing +returns when the last tile has folded. + +`via_res != 0` is the fk-keyed form. `group_lane` is then a foreign KEY into +`via_res` (another pattern resource) and the group of row `i` is +`via_lane[group_lane[i]]` — `SUM(line.amount) GROUP BY partner.country` in one +program, `Terminal::GroupSumViaI32`, with the indirection fused inside the +terminal: no partner-side mask, no remapped key lane, no second program. A key +that names no row of `via_res` drops the row (zero fallback at both hops, +`mask_risc`'s contract). `via_res` may equal `res`. + +**`n_ops == 0` is legal here** and means every row. `lgj_plan_eval` refuses an +empty plan because its caller can fill the destination mask itself; a grouped +sum has no destination mask, so the whole-lane case is a program — the one +`Pred::Range` this crate emits, always `0..n_rows`, pinned by +`the_group_sum_range_is_the_whole_lane_and_nothing_else` so that +`exec_error_to_status`'s `RangeOutOfBounds` arm stays an internal-bug mapping. + +### 20.3 Contract + +- `out_sums[g] = Σ val_lane[i]` over selected rows `i` with key `g`, for + `g < n_groups`; a selected row whose key is `>= n_groups` names no group and + is dropped, never an error. Every element is written on `OK` (zero for a + group no row names); none on any failure — `execute_into` validates the whole + program before its first write. +- `group_lane` must be a `U32` lane of `res`, `val_lane` an `I32` lane of `res`, + `via_lane` a `U32` lane of `via_res` when given. +- Statuses: `NULL_ARGUMENT` for a null `out_sums`, a null `ops` with + `n_ops > 0`, or `n_groups == 0` (a zero-length sink is no sink); + `INVALID_HANDLE` / `WRONG_RESOURCE_KIND` for `res` or a non-zero `via_res` that + is not a live pattern; every plan defect as `lgj_plan_eval` reports it; + `INVALID_LANE` / `LANE_KIND_MISMATCH` for the three lanes; `LENGTH_OVERFLOW` + past `2^32 - 1` rows (`Pred::Range` and the sum carry are both `u32`-bounded). + **No new status.** +- Bulk (§6): `O(rows)` in one pass over the predicate lanes plus one read of + the key and value lanes for the selected rows. No per-row and no per-group + crossing. + +### 20.4 Why one symbol, and why no scalar twin + +The plan surface (`LgjOpDesc`) is reused unchanged, so the whole cost of the +capability is the terminal, and the terminal's two shapes (local key / key +through a second table) are one parameter (`via_res`) rather than two symbols +— the shape §15 mandated for the reductions. There is no +`lgj_plan_group_sum_i32_scalar`: parity is falsified in the crate against +`mask_risc`'s row-at-a-time reference executor on the identical lowered +program (both key shapes, multi-tile), and through the membrane against the +two-crossing path it replaces — which runs a different kernel behind a +different terminal, so agreement is evidence rather than a tautology. + +### 20.5 The Java spelling + +`View.sumByGroup(key, value, groups)` → `GroupTotals`, and +`View.sumByGroupVia(key, via, viaKey, value, groups)`. `GroupTotals` is +addressed by key (`total(g)`, `groups()`) and exposes no array — it is sized +by the question the caller typed, never by the data, which is the eighth named +materialisation site (`Engine.groupSumI32`'s `toArray`, pinned in +`DoctrineFenceTest`). Measured: **one crossing** at 1,024 and at 65,536 rows +for 16 groups, beside the per-group path measured at **32** in the same run. + +### 20.6 The disable table (red-then-green, or it is not evidence) + +| what was disabled | test that went red | +|---|---| +| the `via` key silently downgraded to the local key | `via_reads_the_second_table_through_the_key_lane` | +| the all-rows range emptied to `0..0` | `an_empty_plan_and_an_all_or_plan_both_group_every_row`, `the_group_sum_range_is_the_whole_lane_and_nothing_else`, `grouped_sums_agree_with_quack_over_the_combine_sweep` | +| the `n_groups == 0` guard removed | `every_refusal_is_a_status_and_leaves_out_sums_untouched` (mask-risc's own refusal surfaced as `ALLOCATION_FAILED` instead) | +| Java: the `toArray` pin reverted to 1 | `DoctrineFenceTest` — *"UNFENCED MATERIALIZATION: Engine.java\|.toArray( 2 (pinned 1)"* | +| Java: `requirePositiveGroups` made a no-op | `GroupSumTest` — `groups = 0` reached the membrane and came back as the ABI's `NULL_ARGUMENT`, not the Java exception | +| Java: `Engine` passes `viaResource = 0` always | `GroupSumTest` — every `via group g` | + +Each restored from the commit (the disable cycle runs only against committed +work) and re-run green. The compatibility direction was run for real, not +assumed: `OldAbiCompatTest` against a minor-11 library built from `07e044f` +reports *"View.sumByGroup (minor 12) reports an ABI mismatch, not a missing +symbol (threw AbiMismatchException)"*, with every minor-11 feature still +working beside it. + diff --git a/java/src/main/java/com/adaworldapi/lancegraph/GroupTotals.java b/java/src/main/java/com/adaworldapi/lancegraph/GroupTotals.java new file mode 100644 index 0000000..b4f50b6 --- /dev/null +++ b/java/src/main/java/com/adaworldapi/lancegraph/GroupTotals.java @@ -0,0 +1,53 @@ +package com.adaworldapi.lancegraph; + +/** + * The answer to a grouped sum: one widened total per group, addressed by the group's key value. + * + *

This is what {@code SELECT key, SUM(value) … GROUP BY key} hands back — the same thing a + * {@code ResultSet} would, without the cursor. {@link #total(int)} is the row for key {@code g}; + * {@link #groups()} is how many keys were asked for. A key no selected row carries has a total of + * {@code 0}, exactly as SQL would report it with an outer join onto the key domain. + * + *

Sized by the question, never by the data: a {@code GroupTotals} over 16 keys is 16 numbers + * whether the view spans a thousand rows or a billion. No row, no selection and no index list is + * behind it — the totals were folded natively in one pass and only the totals crossed. + * + *

Immutable. There is no accessor for the underlying storage; the totals are read one at a + * time by key, which is also the only way a caller ever needs them. + */ +public final class GroupTotals { + + private final long[] totals; + + GroupTotals(long[] totals) { + this.totals = totals; + } + + /** How many groups this answer covers — the {@code groups} the caller asked for. */ + public int groups() { + return totals.length; + } + + /** + * The sum for key {@code group}, widened to 64 bits; {@code 0} when no selected row carried + * that key. + * + * @throws IndexOutOfBoundsException if {@code group} is not in {@code [0, groups())} + */ + public long total(int group) { + java.util.Objects.checkIndex(group, totals.length); + return totals[group]; + } + + @Override + public String toString() { + StringBuilder sb = new StringBuilder("GroupTotals["); + for (int g = 0; g < totals.length; g++) { + if (g > 0) { + sb.append(", "); + } + sb.append(g).append('=').append(totals[g]); + } + return sb.append(']').toString(); + } +} diff --git a/java/src/main/java/com/adaworldapi/lancegraph/NativePattern.java b/java/src/main/java/com/adaworldapi/lancegraph/NativePattern.java index f1bd991..bbaf50c 100644 --- a/java/src/main/java/com/adaworldapi/lancegraph/NativePattern.java +++ b/java/src/main/java/com/adaworldapi/lancegraph/NativePattern.java @@ -226,6 +226,41 @@ OptionalLong maxOf(List predicates, I32Field field) { return reduceI32(predicates, field, Layouts.REDUCE_OP_MAX); } + /** + * {@code GROUP BY key SUM(value)} over the rows the plan selects, as ONE native program + * (docs/abi.md §20). No selection is built: the plan is not evaluated into a mask and then + * reduced per group, it is folded straight into the group totals tile by tile. An empty + * plan crosses as such — the ABI treats it as every row — rather than being routed through + * the cached all-rows selection {@link #sumOf} uses, because here there is nothing for a + * selection to be consumed by. + */ + GroupTotals sumByGroup(List predicates, U32Field key, I32Field value, int groups) { + requireOpen("sumByGroup()"); + synchronized (lock) { + requireOpen("sumByGroup()"); + return new GroupTotals(Engine.groupSumI32(handle, plan(predicates), + key.lane().index(), value.lane().index(), 0L, 0, groups)); + } + } + + /** + * The fk-keyed form of {@link #sumByGroup}: the group of a row is {@code viaKey} read on + * {@code via} at the row index this resource's {@code key} names — {@code SUM(line.amount) + * GROUP BY partner.country}, still one program and one crossing, with no selection on either + * resource. {@code via} may be this resource. + */ + GroupTotals sumByGroupVia(List predicates, U32Field key, NativePattern via, + U32Field viaKey, I32Field value, int groups) { + requireOpen("sumByGroupVia()"); + via.requireOpen("sumByGroupVia() via"); + synchronized (lock) { + requireOpen("sumByGroupVia()"); + return new GroupTotals(Engine.groupSumI32(handle, plan(predicates), + key.lane().index(), value.lane().index(), via.handle(), + viaKey.lane().index(), groups)); + } + } + /** Materialise a selection the caller owns and closes. */ Mask selectInto(List predicates) { requireOpen("select()"); diff --git a/java/src/main/java/com/adaworldapi/lancegraph/View.java b/java/src/main/java/com/adaworldapi/lancegraph/View.java index bdd2d04..7c21e89 100644 --- a/java/src/main/java/com/adaworldapi/lancegraph/View.java +++ b/java/src/main/java/com/adaworldapi/lancegraph/View.java @@ -114,6 +114,64 @@ public OptionalLong maxOf(I32Field field) { return owner.maxOf(predicates, field); } + /** + * {@code SELECT key, SUM(value) FROM … WHERE GROUP BY key}, for keys + * {@code 0..groups}. + * + *

One crossing, whatever the number of conditions, groups or rows. The + * chain and the grouped sum are one fused native program; nothing is selected first and + * nothing per group is asked separately. Before this existed the same question cost two + * crossings per group — a {@code sumOf} over {@code where(key.eq(g))} for each {@code g} — + * which the {@code bricks} consumer measured at 32 for 16 groups; it is now 1 for any number. + * + *

A key no selected row carries totals {@code 0}. A selected row whose key is + * {@code >= groups} belongs to no requested group and is dropped, as a {@code WHERE key < n} + * would drop it; ask for more groups to see it. + * + * @param key an unsigned 32-bit column whose values are the group keys + * @param value the signed 32-bit column to sum, widened to 64 bits + * @param groups how many keys to answer for, {@code > 0} + * @throws AbiMismatchException if the loaded library reports ABI minor < 12 + */ + public GroupTotals sumByGroup(U32Field key, I32Field value, int groups) { + java.util.Objects.requireNonNull(key, "key"); + java.util.Objects.requireNonNull(value, "value"); + requirePositiveGroups(groups); + return owner.sumByGroup(predicates, key, value, groups); + } + + /** + * {@code SELECT via.viaKey, SUM(value) FROM this JOIN via ON via.row = this.key … GROUP BY + * via.viaKey} — the grouped sum keyed through a second resource, still one + * crossing. + * + *

{@code key} holds, for each row here, the row index in {@code via} it refers to; the + * group of that row is the {@code viaKey} value found there. The lookup is fused into the + * native fold — no selection on either resource, no remapped key column. A {@code key} that + * names no row of {@code via} drops the row, as an inner join would. + * + * @param via the resource whose {@code viaKey} column supplies the group; may be this view's + * own resource + * @throws AbiMismatchException if the loaded library reports ABI minor < 12 + */ + public GroupTotals sumByGroupVia(U32Field key, NativePattern via, U32Field viaKey, + I32Field value, int groups) { + java.util.Objects.requireNonNull(key, "key"); + java.util.Objects.requireNonNull(via, "via"); + java.util.Objects.requireNonNull(viaKey, "viaKey"); + java.util.Objects.requireNonNull(value, "value"); + requirePositiveGroups(groups); + return owner.sumByGroupVia(predicates, key, via, viaKey, value, groups); + } + + private static void requirePositiveGroups(int groups) { + if (groups <= 0) { + throw new IllegalArgumentException( + "groups must be positive (the number of key values to answer for), was " + + groups); + } + } + /** * A projection of one column through this view. * diff --git a/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Downcalls.java b/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Downcalls.java index 8e8e38e..7b1cc60 100644 --- a/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Downcalls.java +++ b/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Downcalls.java @@ -491,6 +491,21 @@ private static final class Minor11 { private Minor11() {} } + /** + * ABI minor 12 symbol (docs/abi.md §20): the fused plan run straight into a grouped-sum + * terminal — one crossing returns one {@code i64} per group and no selection ever exists. + * Lazy per the minor-2..11 rule. + */ + private static final class Minor12 { + static final MethodHandle PLAN_GROUP_SUM_I32 = mh("lgj_plan_group_sum_i32", + FunctionDescriptor.of(ValueLayout.JAVA_INT, ValueLayout.JAVA_LONG, + ValueLayout.ADDRESS, ValueLayout.JAVA_INT, ValueLayout.JAVA_INT, + ValueLayout.JAVA_INT, ValueLayout.JAVA_LONG, ValueLayout.JAVA_INT, + ValueLayout.ADDRESS, ValueLayout.JAVA_LONG)); + + private Minor12() {} + } + /** * Sum one facet's 12-byte register, under {@code carving}, over the rows a mask selects. * @@ -639,6 +654,29 @@ public static long reduceI32(long res, int laneId, int reduceOp, long mask, return outValue.get(ValueLayout.JAVA_LONG, 0); } + /** + * {@code GROUP BY groupLane SUM(valLane)} over the rows the plan selects, in ONE crossing + * (docs/abi.md §20, minor 12). {@code outSums} receives {@code nGroups} widened totals; a + * selected row whose key is past {@code nGroups} is dropped, never an error. With + * {@code viaRes != 0} the key is read THROUGH {@code groupLane} into {@code viaRes}'s + * {@code viaLane} (the fk-keyed form). {@code nOps == 0} is legal and means every row. + * + *

Bulk in the §6 sense: one pass over the predicate lanes, one read of the key and value + * lanes for the selected rows, no per-row and no per-group crossing. + */ + public static void planGroupSumI32(long res, MemorySegment ops, int nOps, int groupLane, + int valLane, long viaRes, int viaLane, MemorySegment outSums, long nGroups) { + crossed(); + int st; + try { + st = (int) Minor12.PLAN_GROUP_SUM_I32.invokeExact(res, ops, nOps, groupLane, valLane, + viaRes, viaLane, outSums, nGroups); + } catch (Throwable t) { + throw wrap("lgj_plan_group_sum_i32", t); + } + Status.check("lgj_plan_group_sum_i32", st); + } + // ── row store (docs/abi.md §11, ABI minor 2) ───────────────────────────────────────────── // // Callers above this class are expected to have already checked Abi.requireMinor(2) — these diff --git a/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Engine.java b/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Engine.java index f7d59a1..4a74f37 100644 --- a/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Engine.java +++ b/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Engine.java @@ -379,6 +379,37 @@ public static ReduceOutcome reduceI32(long resource, int laneId, int reduceOp, l return new ReduceOutcome(value, present); } + /** + * The grouped sum that never builds a selection (docs/abi.md §20, ABI minor 12): + * {@code GROUP BY groupLane SUM(valLane)} over the rows {@code plan} selects, evaluated as + * ONE native program and returned as one widened total per group. Unlike every reduce + * above, no mask handle is involved at any point — the plan is not evaluated INTO anything; + * the fold reads it tile by tile and lands only the group totals. + * + *

{@code viaResource != 0} selects the fk-keyed form: the group of a row is + * {@code viaLane[groupLane[row]]}, read on {@code viaResource}, with no partner-side + * selection and no remapped key lane (the indirection is fused inside the native terminal). + * + *

An empty {@code plan} is legal here and means every row — the ABI has no destination + * mask for Java to fill on its own, so the whole-lane case crosses as a plan of zero ops + * rather than being answered locally. Requires ABI minor >= 12. + * + *

The returned array is sized by {@code groups} — the shape of the QUESTION the caller + * typed, never by the row count — which is what keeps it on the right side of the + * materialization rule (the eighth named site; see the repo's zero-copy section). + */ + public static long[] groupSumI32(long resource, List plan, int groupLane, int valLane, + long viaResource, int viaLane, int groups) { + Abi.requireMinor(12); + MemorySegment ops = plan.isEmpty() ? MemorySegment.NULL : marshal(plan); + try (Arena a = Arena.ofConfined()) { + MemorySegment out = a.allocate(ValueLayout.JAVA_LONG, groups); + Downcalls.planGroupSumI32(resource, ops, plan.size(), groupLane, valLane, viaResource, + viaLane, out, groups); + return out.toArray(ValueLayout.JAVA_LONG); + } + } + // ── mask complement + hop (docs/abi.md §13, ABI minor ≥ 4) ───────────────────────────── // // Same requireMinor-before-any-downcall discipline as the row store section above: a Java diff --git a/java/src/test/java/com/adaworldapi/lancegraph/AllTests.java b/java/src/test/java/com/adaworldapi/lancegraph/AllTests.java index 054c4a9..917e21d 100644 --- a/java/src/test/java/com/adaworldapi/lancegraph/AllTests.java +++ b/java/src/test/java/com/adaworldapi/lancegraph/AllTests.java @@ -34,6 +34,7 @@ public static void main(String[] args) { suites.put("CarvingTableTest", CarvingTableTest::run); suites.put("ColumnarStoreTest", ColumnarStoreTest::run); suites.put("MaskingOpCompletionTest", MaskingOpCompletionTest::run); + suites.put("GroupSumTest", GroupSumTest::run); if (!NativeRuntime.isAvailable()) { // ApiSurfaceTest and DoctrineFenceTest need no native library — the API's shape is a diff --git a/java/src/test/java/com/adaworldapi/lancegraph/DoctrineFenceTest.java b/java/src/test/java/com/adaworldapi/lancegraph/DoctrineFenceTest.java index 6ee74f5..3650b1f 100644 --- a/java/src/test/java/com/adaworldapi/lancegraph/DoctrineFenceTest.java +++ b/java/src/test/java/com/adaworldapi/lancegraph/DoctrineFenceTest.java @@ -100,6 +100,9 @@ private DoctrineFenceTest() {} // five named sites, mechanically: // Mask.materializeRows() → Arrays.copyOf ×2 + new long[ ×1 (the named terminal) // Engine.rowLayoutProbe → .toArray( ×1 (≤32 B diagnostic) + // Engine.groupSumI32 → .toArray( ×1 (one i64 per GROUP — + // sized by the question the caller typed, never by rows; + // the eighth named site, minor 12) // Engine.facetSumResolved → new long[ ×1 (fixed [2] pair) // Abi manifest-name read (cString) → new byte[ ×1 + MemorySegment.copy ×1 // Abi.readCarvings → new int[ ×2 (incl. empty arm) @@ -108,7 +111,7 @@ private DoctrineFenceTest() {} static { MATERIALIZATION_PINS.put("Mask.java|Arrays.copyOf", 2); MATERIALIZATION_PINS.put("Mask.java|new long[", 1); - MATERIALIZATION_PINS.put("Engine.java|.toArray(", 1); + MATERIALIZATION_PINS.put("Engine.java|.toArray(", 2); MATERIALIZATION_PINS.put("Engine.java|new long[", 1); MATERIALIZATION_PINS.put("Abi.java|new int[", 2); MATERIALIZATION_PINS.put("Abi.java|new byte[", 1); diff --git a/java/src/test/java/com/adaworldapi/lancegraph/GroupSumTest.java b/java/src/test/java/com/adaworldapi/lancegraph/GroupSumTest.java new file mode 100644 index 0000000..2d951cb --- /dev/null +++ b/java/src/test/java/com/adaworldapi/lancegraph/GroupSumTest.java @@ -0,0 +1,191 @@ +package com.adaworldapi.lancegraph; + +/** + * Minor 12 through the public facade: a {@code GROUP BY} that costs one crossing and builds no + * selection. + * + *

Three claims, each measured rather than asserted: + * + *

    + *
  1. Parity. {@link View#sumByGroup} answers exactly what the two-crossing + * path it replaces answered — {@code where(key.eq(g)).sumOf(value)} for every {@code g} — + * on a single-tile and a multi-tile resource, with and without a chain. The two paths run + * different native kernels behind different terminals; the numbers must still agree. + *
  2. Cost. One crossing for 16 groups, at 1,024 rows and at 65,536 rows, + * where the per-group path pays 32. The counter is the membrane's own. + *
  3. The keyed form is a real join. {@link View#sumByGroupVia} regroups + * through a second resource; the oracle recomputes the regrouping from that resource's + * own key column (read through the ONE named materialiser, in test code, where it belongs) + * and the local totals, and the two must differ from each other. + *
+ */ +public final class GroupSumTest { + + private GroupSumTest() {} + + private static final int GROUPS = 16; // Pattern.CLASS spans 0..15 + + public static void main(String[] args) { + System.out.println("GroupSumTest"); + if (!NativeRuntime.isAvailable()) { + System.exit(Checks.reportUnavailable("GroupSumTest")); + } + Checks c = new Checks("GroupSumTest"); + run(c); + System.exit(c.report()); + } + + public static void run(Checks c) { + if (NativeRuntime.abiMinor() < 12) { + c.that("SKIPPED: library minor " + NativeRuntime.abiMinor() + + " predates the grouped sum (minor 12); OldAbiCompatTest covers the gate", true); + return; + } + + c.section("parity: one crossing answers what sixteen pairs of crossings answered"); + for (long rows : new long[] {1_024L, 65_536L}) { + try (NativePattern data = NativePattern.open(rows, 0x51DEL)) { + View[] views = { + data.view(), + data.view().where(Pattern.VALUE.gt(100)), + data.view().where(Pattern.VALUE.gt(100)).where(Pattern.VALUE.gt(-50)), + data.view().where(Pattern.CLASS.eq(7)).where(Pattern.VALUE.gt(0)), + }; + for (View v : views) { + GroupTotals t = v.sumByGroup(Pattern.CLASS, Pattern.VALUE, GROUPS); + c.eq(rows + " rows, " + v + ": groups()", GROUPS, t.groups()); + long live = 0; + for (int g = 0; g < GROUPS; g++) { + long want = v.where(Pattern.CLASS.eq(g)).sumOf(Pattern.VALUE); + c.eq(rows + " rows, " + v + ", group " + g, want, t.total(g)); + if (want != 0) { + live++; + } + } + // Anti-vacuity: the totals are not sixteen zeros agreeing with sixteen zeros. + c.that(rows + " rows, " + v + ": at least one group is non-zero", live >= 1); + } + GroupTotals all = views[0].sumByGroup(Pattern.CLASS, Pattern.VALUE, GROUPS); + long everything = 0; + for (int g = 0; g < GROUPS; g++) { + everything += all.total(g); + } + c.eq(rows + " rows: the empty chain's groups partition the whole sum", + views[0].sumOf(Pattern.VALUE), everything); + } + } + + c.section("a narrower key domain drops, a wider one pads with zero"); + try (NativePattern data = NativePattern.open(4_096L, 0x51DEL)) { + View v = data.view().where(Pattern.VALUE.gt(0)); + GroupTotals full = v.sumByGroup(Pattern.CLASS, Pattern.VALUE, GROUPS); + GroupTotals eight = v.sumByGroup(Pattern.CLASS, Pattern.VALUE, 8); + GroupTotals twenty = v.sumByGroup(Pattern.CLASS, Pattern.VALUE, 20); + for (int g = 0; g < 8; g++) { + c.eq("groups=8 keeps group " + g, full.total(g), eight.total(g)); + } + for (int g = 0; g < GROUPS; g++) { + c.eq("groups=20 keeps group " + g, full.total(g), twenty.total(g)); + } + c.eq("groups=20: key 16 is unnamed and zero", 0, twenty.total(16)); + c.eq("groups=20: key 19 is unnamed and zero", 0, twenty.total(19)); + c.throwsUp("total(groups()) is out of range", IndexOutOfBoundsException.class, + () -> eight.total(8)); + } + + c.section("cost: one crossing for sixteen groups, at any row count"); + try (NativePattern small = NativePattern.open(1_024L, 0xC0DEL); + NativePattern large = NativePattern.open(65_536L, 0xC0DEL)) { + View sv = small.view().where(Pattern.VALUE.gt(100)); + View lv = large.view().where(Pattern.VALUE.gt(100)); + // Warm: the first terminal op on a resource may resolve lazily-held state. + sv.sumByGroup(Pattern.CLASS, Pattern.VALUE, GROUPS); + lv.sumByGroup(Pattern.CLASS, Pattern.VALUE, GROUPS); + + long a = Diagnostics.crossings(); + sv.sumByGroup(Pattern.CLASS, Pattern.VALUE, GROUPS); + long smallCost = Diagnostics.crossings() - a; + long b = Diagnostics.crossings(); + lv.sumByGroup(Pattern.CLASS, Pattern.VALUE, GROUPS); + long largeCost = Diagnostics.crossings() - b; + c.eq("1,024 rows, 16 groups: sumByGroup crosses once", 1, smallCost); + c.eq("65,536 rows, 16 groups: sumByGroup crosses once", 1, largeCost); + + // The path it replaces, measured beside it rather than remembered from bricks. + lv.where(Pattern.CLASS.eq(0)).sumOf(Pattern.VALUE); + long d = Diagnostics.crossings(); + for (int g = 0; g < GROUPS; g++) { + lv.where(Pattern.CLASS.eq(g)).sumOf(Pattern.VALUE); + } + long perGroupCost = Diagnostics.crossings() - d; + c.eq("the per-group path pays two crossings per group", 2L * GROUPS, perGroupCost); + + long e = Diagnostics.crossings(); + large.view().sumByGroup(Pattern.CLASS, Pattern.VALUE, GROUPS); + c.eq("an empty chain still crosses exactly once (the plan of zero ops is the ABI's)", + 1, Diagnostics.crossings() - e); + } + + c.section("the keyed form regroups through a second resource, in one crossing"); + try (NativePattern lines = NativePattern.open(4_096L, 0xA11CEL); + NativePattern partners = NativePattern.open(64L, 0xB0BL)) { + View v = lines.view().where(Pattern.VALUE.gt(100)); + GroupTotals local = v.sumByGroup(Pattern.CLASS, Pattern.VALUE, GROUPS); + GroupTotals via = v.sumByGroupVia(Pattern.CLASS, partners, Pattern.CLASS, + Pattern.VALUE, GROUPS); + + // Oracle: partner row k (k < 16, the only rows a line's CLASS can name) carries + // CLASS = c(k); the via total for g is the sum of the local totals over every k + // with c(k) == g. c(k) is read from `partners` through the one named materialiser. + long[] want = new long[GROUPS]; + for (int g = 0; g < GROUPS; g++) { + try (Mask m = partners.view().where(Pattern.CLASS.eq(g)).select()) { + for (long k : m.materializeRows()) { + if (k < GROUPS) { + want[g] += local.total((int) k); + } + } + } + } + boolean differs = false; + for (int g = 0; g < GROUPS; g++) { + c.eq("via group " + g, want[g], via.total(g)); + differs |= via.total(g) != local.total(g); + } + c.that("the join genuinely regrouped (via totals differ from local totals)", differs); + + long x = Diagnostics.crossings(); + v.sumByGroupVia(Pattern.CLASS, partners, Pattern.CLASS, Pattern.VALUE, GROUPS); + c.eq("sumByGroupVia crosses once", 1, Diagnostics.crossings() - x); + + GroupTotals self = v.sumByGroupVia(Pattern.CLASS, lines, Pattern.CLASS, + Pattern.VALUE, GROUPS); + c.eq("a self-join answers 16 groups", GROUPS, self.groups()); + } + + c.section("arguments are checked in Java, before any crossing"); + try (NativePattern data = NativePattern.open(256L, 1L)) { + long before = Diagnostics.crossings(); + c.throwsUp("groups = 0", IllegalArgumentException.class, + () -> data.view().sumByGroup(Pattern.CLASS, Pattern.VALUE, 0)); + c.throwsUp("groups < 0", IllegalArgumentException.class, + () -> data.view().sumByGroup(Pattern.CLASS, Pattern.VALUE, -3)); + c.throwsUp("null key", NullPointerException.class, + () -> data.view().sumByGroup(null, Pattern.VALUE, 4)); + c.eq("none of those crossed", 0, Diagnostics.crossings() - before); + + NativePattern gone = NativePattern.open(64L, 2L); + gone.close(); + c.throwsUp("a closed via resource is rejected", ClosedResourceException.class, + () -> data.view().sumByGroupVia(Pattern.CLASS, gone, Pattern.CLASS, + Pattern.VALUE, 4)); + data.close(); + c.throwsUp("a closed resource is rejected", ClosedResourceException.class, + () -> data.view().sumByGroup(Pattern.CLASS, Pattern.VALUE, 4)); + } catch (ClosedResourceException expected) { + // try-with-resources closes `data` a second time after the test closed it: that + // double close is itself an error by contract, and the one this block expects. + c.that("double close is an error, not a no-op", true); + } + } +} diff --git a/java/src/test/java/com/adaworldapi/lancegraph/OldAbiCompatTest.java b/java/src/test/java/com/adaworldapi/lancegraph/OldAbiCompatTest.java index 00a08af..5408e76 100644 --- a/java/src/test/java/com/adaworldapi/lancegraph/OldAbiCompatTest.java +++ b/java/src/test/java/com/adaworldapi/lancegraph/OldAbiCompatTest.java @@ -108,6 +108,19 @@ public static void run(Checks c) { } }); + // Minor 12 — the grouped sum that never builds a selection. Like the minor-11 reduce + // above it is built entirely on NativePattern, so it is gated unconditionally here rather + // than inside the minor-2 block: against a genuinely minor-1 library it must name minor + // 12, never die on a missing symbol. + gate(c, loaded, 12, "View.sumByGroup", () -> { + try (NativePattern p = NativePattern.open(64, 0x1234L)) { + GroupTotals t = p.view().sumByGroup(Pattern.CLASS, Pattern.VALUE, 16); + if (t.groups() != 16) { + throw new IllegalStateException("asked for 16 groups, got " + t.groups()); + } + } + }); + // Minor 4 — mask complement. Needs a minor-2 store to build masks on, so it is only // meaningful once the library has minor 2 as well. if (loaded >= 2) { diff --git a/native/lgj-abi/src/abi.rs b/native/lgj-abi/src/abi.rs index 12e6de5..2dc629d 100644 --- a/native/lgj-abi/src/abi.rs +++ b/native/lgj-abi/src/abi.rs @@ -82,7 +82,18 @@ pub const LGJ_ABI_MAJOR: u32 = 0; /// [`LgjOpDesc`] op-codes that cost no symbol at all /// ([`LGJ_OP_NE_U32`] … [`LGJ_OP_TERNARY_MATCH_U32`]). No new status and no /// manifest growth, so a minor-10 Java loads and sees none of it. -pub const LGJ_ABI_MINOR: u32 = 11; +/// +/// **Minor 12** (docs/abi.md §20): [`crate::exports::lgj_plan_group_sum_i32`] +/// — the fused plan run STRAIGHT INTO a grouped-sum terminal. Until now every +/// Java reduction paid for a selection first (`lgj_plan_eval` into a mask, +/// then `lgj_reduce_*` over it); this symbol lowers the plan and the +/// `GROUP BY key SUM(val)` as ONE `mask_risc::Program`, so the only +/// population-sized state is a tile-local scratch word and what crosses back +/// is one `i64` per group. A second table's `u32` lane may be read THROUGH a +/// key lane of this one (`via_res`/`via_lane`, the fk-keyed group sum) with +/// no partner-side mask and no second program. No new status and no manifest +/// growth, so a minor-11 Java loads and sees none of it. +pub const LGJ_ABI_MINOR: u32 = 12; /// `"LGJ_ABI\0"` read big-endian. /// diff --git a/native/lgj-abi/src/exports.rs b/native/lgj-abi/src/exports.rs index b2fd0a0..15558d5 100644 --- a/native/lgj-abi/src/exports.rs +++ b/native/lgj-abi/src/exports.rs @@ -1719,11 +1719,15 @@ fn validate_plan(pattern: &ResourceEntry, ops: &[LgjOpDesc]) -> Result<(), i32> /// **Every arm here is unreachable through the ABI**, and saying so is worth /// more than implying otherwise. `validate_plan` runs first and rejects an /// unknown opcode, a bad combine, an out-of-range lane and a kind mismatch -/// before the lowering is even built; the lowering names no input plane -/// (`Planes::masks` is `&[]`), no sum terminal, no blend and no `Pred::Range`. -/// What is left — the scratch-sizing family, `ScratchReadBeforeWrite`, -/// `GateAliasesDst`, `RangeOutOfBounds` — would be a bug in THIS file, not in -/// a caller's plan. +/// before the lowering is even built; the `Keep` lowering names no input +/// plane (`Planes::masks` is `&[]`), no sum terminal, no blend and no +/// `Pred::Range`. The grouped-sum lowering (minor 12) names a sum terminal +/// and ONE `Range` — always `0..n_rows`, so `RangeOutOfBounds` stays a bug in +/// this file; its `SumRowBound` needs more than `2^32` rows, which no pattern +/// this crate can open reaches; and its foreign-lane checks are done here, +/// before lowering, against the resolved `via` resource. What is left — the +/// scratch-sizing family, `ScratchReadBeforeWrite`, `GateAliasesDst`, +/// `RangeOutOfBounds` — would be a bug in THIS file, not in a caller's plan. /// /// So the map exists to turn such a bug into a status a caller can see /// instead of a panic, and its arms are deliberately NOT claimed to be @@ -2137,6 +2141,295 @@ pub unsafe extern "C" fn lgj_reduce_i32( }) } +// ─────────────────────────────────────────────────────────────────────────── +// Grouped reduction (ABI minor ≥ 12) — the fused plan run STRAIGHT INTO a +// grouped-sum terminal (docs/abi.md §20). +// ─────────────────────────────────────────────────────────────────────────── + +// Nine arguments because the ABI symbol has nine; a struct would be a +// second spelling of the same signature with nothing to check it against. +#[allow(clippy::too_many_arguments)] +fn plan_group_sum_impl( + res: u64, + ops: *const LgjOpDesc, + n_ops: u32, + group_lane: u32, + val_lane: u32, + via_res: u64, + via_lane: u32, + out_sums: *mut i64, + n_groups: u64, +) -> i32 { + // An empty plan is LEGAL here, unlike `lgj_plan_eval`: there is no + // destination mask a caller could fill on its own, so "every row" has to + // be a program (the whole-lane `Range`, `plan_lower::lower_group_sum`). + // `ops` may therefore be null exactly when `n_ops == 0`. + if out_sums.is_null() || (n_ops > 0 && ops.is_null()) { + return LGJ_ERR_NULL_ARGUMENT; + } + // A zero-length `out_sums` is no output buffer: no group can land + // anywhere, and `mask_risc` would refuse it as a missing sink. Reported + // as the argument defect it is rather than as the executor's refusal. + if n_groups == 0 { + return LGJ_ERR_NULL_ARGUMENT; + } + let groups = match usize::try_from(n_groups) { + Ok(g) if g <= isize::MAX as usize / std::mem::size_of::() => g, + _ => return LGJ_ERR_LENGTH_OVERFLOW, + }; + let pattern = match registry::resolve_kind(res, LGJ_RESOURCE_PATTERN) { + Ok(p) => p, + Err(e) => return e, + }; + // SAFETY: when `n_ops > 0`, `ops` is non-null (checked) and the caller + // states it points at `n_ops` contiguous `LgjOpDesc` — the same contract + // as `lgj_plan_eval`. When `n_ops == 0` no pointer is read at all. + let ops: &[LgjOpDesc] = if n_ops == 0 { + &[] + } else { + unsafe { std::slice::from_raw_parts(ops, n_ops as usize) } + }; + if !ops.is_empty() { + if let Err(e) = validate_plan(&pattern, ops) { + return e; + } + } + + // The group lane is a `U32` lane of THIS table (the key, or the foreign + // key when `via_res` is given); the value lane is an `I32` lane of this + // table. Both are checked before anything is lowered or written. + let group_lane16 = match u16::try_from(group_lane) { + Ok(l) => l, + Err(_) => return LGJ_ERR_INVALID_LANE, + }; + let val_lane16 = match u16::try_from(val_lane) { + Ok(l) => l, + Err(_) => return LGJ_ERR_INVALID_LANE, + }; + match lane_view(&pattern, group_lane) { + Ok(LaneView::U32(_)) => {} + Ok(_) => return LGJ_ERR_LANE_KIND_MISMATCH, + Err(e) => return e, + } + match lane_view(&pattern, val_lane) { + Ok(LaneView::I32(_)) => {} + Ok(_) => return LGJ_ERR_LANE_KIND_MISMATCH, + Err(e) => return e, + } + // `via_res == 0` is "no second table" — 0 is never a live handle (the + // registry's generation is never zero), so it cannot alias a resource. + let via = if via_res == 0 { + None + } else { + let entry = match registry::resolve_kind(via_res, LGJ_RESOURCE_PATTERN) { + Ok(p) => p, + Err(e) => return e, + }; + match lane_view(&entry, via_lane) { + Ok(LaneView::U32(_)) => {} + Ok(_) => return LGJ_ERR_LANE_KIND_MISMATCH, + Err(e) => return e, + } + Some(entry) + }; + + let n_rows = pattern.n_rows; + let rows = match usize::try_from(n_rows) { + Ok(r) => r, + Err(_) => return LGJ_ERR_LENGTH_OVERFLOW, + }; + // `Pred::Range` is `u32`-bounded, and so (via `MASKED_SUM_I32_MAX_ROWS`) + // is the sum terminal's carry: a pattern past `2^32 - 1` rows cannot be + // grouped by this symbol at all, and says so before lowering. + let rows32 = match u32::try_from(n_rows) { + Ok(r) => r, + Err(_) => return LGJ_ERR_LENGTH_OVERFLOW, + }; + let group = if via.is_some() { + // The one foreign lane this call supplies sits at index 0 of + // `Foreign::lanes`; the program's own lane ids are untouched. + plan_lower::GroupKey::Via { + fk: group_lane16, + key: 0, + } + } else { + plan_lower::GroupKey::Local(group_lane16) + }; + let program = match plan_lower::lower_group_sum(ops, rows32, group, val_lane16) { + Some(p) => p, + // Unreachable: `validate_plan` already rejected every unknown opcode. + None => return LGJ_ERR_UNKNOWN_OPCODE, + }; + + let fixture = match pattern.fixture() { + Some(f) => f, + None => return LGJ_ERR_WRONG_RESOURCE_KIND, + }; + // Same index order as `plan_eval_impl`, pinned by the same `const _` + // assertions: LANE_IDS, LANE_CLASSES, LANE_VALUES. + let lanes = [ + LaneRef::U64(fixture.ids()), + LaneRef::U32(fixture.classes()), + LaneRef::I32(fixture.values()), + ]; + let planes = Planes { + n_rows: rows, + masks: &[], + lanes: &lanes, + }; + // The foreign lane, borrowed from the `via` pattern for the length of + // this call. Its length is the OTHER table's row count and is never + // checked against `n_rows` — a key past it drops the row (zero fallback, + // `mask_risc`'s contract), it does not error. + let via_lanes: [LaneRef<'_>; 1]; + let foreign = match &via { + Some(entry) => { + let classes = match lane_view(entry, via_lane) { + Ok(LaneView::U32(v)) => v, + // Checked above; kept as a status rather than an unwrap. + _ => return LGJ_ERR_LANE_KIND_MISMATCH, + }; + via_lanes = [LaneRef::U32(classes)]; + Foreign { + planes: &[], + lanes: &via_lanes, + } + } + None => Foreign::NONE, + }; + + // TILED, exactly as `plan_eval_impl`: the scratch is `SLOTS × + // tile_words_for(rows)` words however many rows the resource holds, and + // the terminal adds each tile's contribution into the caller's `i64` + // buffer — the demanded sink, and the ONLY thing sized by the answer + // rather than by a tile. No `Keep`, no mask handle, no population. + let tile = tile_words_for(rows); + let need = match scratch_words_for(tile, plan_lower::SLOTS as usize) { + Some(n) => n, + None => return LGJ_ERR_LENGTH_OVERFLOW, + }; + // SAFETY: non-null (checked) and the caller states it points at + // `n_groups` contiguous, writable `i64`. `execute_into` validates the + // whole program before its first write to this slice, so a refused call + // leaves it byte-for-byte as it was; on `LGJ_OK` every element holds its + // group's sum (zero for a group no selected row names). + let out: &mut [i64] = unsafe { std::slice::from_raw_parts_mut(out_sums, groups) }; + PLAN_SCRATCH.with(|cell| { + let mut buf = cell.borrow_mut(); + if buf.len() < need { + buf.resize(need, 0); + } + let mut scratch = match Scratch::over(&mut buf[..need], tile, plan_lower::SLOTS as usize) { + Ok(s) => s, + Err(e) => return exec_error_to_status(e), + }; + match execute_into(&program, &planes, &foreign, &mut scratch, Out::I64(out)) { + Ok(Value::GroupSummed) => LGJ_OK, + // The lowering emits exactly one terminal and it is a grouped + // sum; any other value means this file built a program it did + // not intend to. + Ok(_) => LGJ_ERR_ALLOCATION_FAILED, + Err(e) => exec_error_to_status(e), + } + }) +} + +/// `GROUP BY group_lane SUM(val_lane)` over the rows the plan selects — **one +/// crossing, one program, and no selection anywhere** (ABI minor ≥ 12, +/// `docs/abi.md` §20). +/// +/// # Why this symbol exists +/// +/// Every reduction Java could ask for before minor 12 paid for a selection +/// first: `lgj_plan_eval` landed the plan's answer in a mask handle, and a +/// `lgj_reduce_*` then read that mask. The selection was never HELD by Java +/// — it lived in a native mask — but it was still a population-sized thing +/// that existed only to be consumed by the very next fold, which is the +/// intermediate materialisation the fold algebra exists to remove. A +/// `GROUP BY` made it worse by a factor of the group count: the `bricks` +/// consumer's `sumBy()` measured **32 crossings for 16 groups** (one plan +/// evaluation and one reduce per group). +/// +/// This symbol lowers the plan and the grouped sum as ONE +/// `mask_risc::Program` — the same prefix rewrite and survivor skip as +/// `lgj_plan_eval` (`plan_lower::lower_group_sum`), ending in +/// `Terminal::GroupSumI32` over the accumulator instead of `Keep`. The +/// executor runs it tile by tile; the accumulator is `TILE_WORDS` of scratch; +/// the terminal folds `val_lane[i]` into `out_sums[group_lane[i]]` for every +/// selected row of the tile and moves on. The only state sized by the answer +/// is `out_sums` itself, and the only thing that crosses back is one `i64` +/// per group. 16 groups cost one crossing; so do 10⁹ rows. +/// +/// # Semantics +/// +/// - `ops`/`n_ops` are the fused plan, exactly as `lgj_plan_eval` reads them +/// (accumulator starts as all rows; each op combines per its `combine`). +/// **`n_ops == 0` is legal here** and means every row — there is no +/// destination mask for a caller to fill itself, so the empty plan lowers +/// to a whole-lane range rather than being refused (`EMPTY_PLAN` is +/// `lgj_plan_eval`'s answer, not this symbol's). +/// - `group_lane` is a `U32` lane of `res`; `val_lane` an `I32` lane of `res`. +/// - `out_sums[g]` receives `Σ val_lane[i]` over selected rows `i` with +/// `group_lane[i] == g`, for `g < n_groups`; a selected row whose key is +/// `>= n_groups` names no group and is dropped, never an error. Every +/// element is written on `LGJ_OK` (zero for a group no row names); none is +/// written on any failure. +/// - **`via_res != 0` is the fk-keyed form**: `group_lane` is then read as a +/// foreign KEY into `via_res` (another pattern resource), and the group of +/// row `i` is `via_lane[group_lane[i]]` — `via_lane` a `U32` lane of +/// `via_res`. `SUM(line.amount) GROUP BY partner.country` in one program, +/// with no partner-side mask and no remapped key lane: the indirection is +/// fused inside `Terminal::GroupSumViaI32`. A key that names no row of +/// `via_res` drops the row (zero fallback at both hops). `via_res` may +/// equal `res`. `via_lane` is ignored when `via_res == 0`. +/// +/// # Statuses +/// +/// `NULL_ARGUMENT` for a null `out_sums`, a null `ops` with `n_ops > 0`, or +/// `n_groups == 0`; `INVALID_HANDLE` / `WRONG_RESOURCE_KIND` for `res` or a +/// non-zero `via_res` that is not a live pattern; every plan defect exactly +/// as `lgj_plan_eval` reports it; `INVALID_LANE` / `LANE_KIND_MISMATCH` for a +/// `group_lane` that is not `U32`, a `val_lane` that is not `I32`, or a +/// `via_lane` that is not `U32` on `via_res`; `LENGTH_OVERFLOW` for a pattern +/// past `2^32 - 1` rows. All of it before any write. +/// +/// # Bulk (§6) +/// +/// `O(rows)` in ONE pass over the predicate lanes plus, for the selected +/// rows only, one read of the key and value lanes. No per-row crossing, no +/// per-group crossing. +/// +/// # Safety +/// +/// A null pointer is *handled*, not UB: `NULL_ARGUMENT`. Beyond that, `ops` +/// must point to `n_ops` contiguous, initialized `LgjOpDesc` (24 bytes, align +/// 8) when `n_ops > 0`, and `out_sums` to `n_groups` writable, aligned `i64`; +/// both stay valid for the call and neither is touched after the null checks +/// except as stated above. +/// +/// `unsafe` here is a note to Rust callers linking the `rlib`. The JVM, +/// which is the real caller, has no such concept — it upholds the same +/// contract by construction, because every pointer it passes comes from a +/// `MemorySegment` whose size and alignment it derived from the manifest. +#[no_mangle] +pub unsafe extern "C" fn lgj_plan_group_sum_i32( + res: u64, + ops: *const LgjOpDesc, + n_ops: u32, + group_lane: u32, + val_lane: u32, + via_res: u64, + via_lane: u32, + out_sums: *mut i64, + n_groups: u64, +) -> i32 { + guard(|| { + plan_group_sum_impl( + res, ops, n_ops, group_lane, val_lane, via_res, via_lane, out_sums, n_groups, + ) + }) +} + // ─────────────────────────────────────────────────────────────────────────── // Graph traversal (ABI minor ≥ 4) — the first symbol gated by the // lance-graph-contract ClassView/FieldMask LAW (docs/abi.md §13). @@ -2393,6 +2686,10 @@ mod tests { use super::*; use crate::fixture::{Fixture, LANE_CLASSES, LANE_IDS, LANE_VALUES}; + /// Minor 12: the grouped sum that never builds a selection — parity + /// against the two-crossing path it replaces, the scalar oracle, the + /// fk-keyed form against a hand-walked join, and every refusal. + mod group_sum; /// The lowering differential: this crate's `plan_lower` (a flat op list) /// against `lance-graph-quack`'s `lower` (a Boolean tree), both producing /// a `mask_risc::Program`. Two implementations of one law; until this @@ -3991,6 +4288,29 @@ mod tests { } } + mod call12 { + use super::*; + + #[allow(clippy::too_many_arguments)] + pub fn plan_group_sum_i32( + res: u64, + ops: *const LgjOpDesc, + n_ops: u32, + group_lane: u32, + val_lane: u32, + via_res: u64, + via_lane: u32, + out_sums: *mut i64, + n_groups: u64, + ) -> i32 { + unsafe { + lgj_plan_group_sum_i32( + res, ops, n_ops, group_lane, val_lane, via_res, via_lane, out_sums, n_groups, + ) + } + } + } + /// Pack a `LGJ_OP_TERNARY_MATCH_U32` operand the way `abi.rs` documents: /// pattern in the low 32 bits, care in the high 32. ONE spelling in the /// tests, so a hand-packing typo cannot make a passing arm mean something diff --git a/native/lgj-abi/src/exports/tests/group_sum.rs b/native/lgj-abi/src/exports/tests/group_sum.rs new file mode 100644 index 0000000..c322b37 --- /dev/null +++ b/native/lgj-abi/src/exports/tests/group_sum.rs @@ -0,0 +1,536 @@ +//! Minor 12 — `lgj_plan_group_sum_i32`, the grouped sum that never builds a +//! selection. +//! +//! Three oracles, none of them the code under test: +//! +//! 1. **The two-crossing path it replaces.** `lgj_plan_eval` into a mask, +//! then `lgj_reduce_sum_i32` over it, once per group with `class == g` +//! appended as an AND. That is exactly what `bricks`' `sumBy()` measured +//! at 32 crossings for 16 groups; the new symbol must answer identically +//! in one. Different kernels (`masked_sum_i32` vs `masked_group_sum_i32`), +//! different lowering (`Keep` vs a sum terminal), same numbers. +//! 2. **`mask_risc`'s row-at-a-time reference executor**, run on the SAME +//! lowered program — whole-evaluator-against-whole-evaluator, the +//! discipline `lgj_plan_eval_scalar` gives the `Keep` path. +//! 3. **A hand walk over the fixture arrays** with a hand-evaluated +//! predicate, for the fk-keyed form: no kernel, no lowering, no mask — +//! `for i: if value[i] > 100 { out[b.class[a.class[i]]] += value[i] }`. +//! +//! And every refusal, each pinned to leave `out_sums` byte-for-byte alone. + +use super::*; +use lance_graph_mask_risc::reference_execute_into; + +fn open(n: u64, seed: u64) -> u64 { + let mut h = 0u64; + assert_eq!(call::pattern_open(n, seed, &mut h), LGJ_OK); + h +} + +/// Every element pre-filled with a sentinel no real sum produces, so a +/// success can be checked for having written EVERY group (zero included) and +/// a refusal for having written none. +const SENTINEL: i64 = i64::MIN + 0x5EED; + +/// Run the symbol; `ops` empty passes a NULL pointer, the way a caller with +/// no predicate would. +fn group_sum( + res: u64, + ops: &[LgjOpDesc], + group_lane: u32, + val_lane: u32, + via: (u64, u32), + groups: usize, +) -> (i32, Vec) { + let mut out = vec![SENTINEL; groups]; + let ptr = if ops.is_empty() { + std::ptr::null() + } else { + ops.as_ptr() + }; + let st = call12::plan_group_sum_i32( + res, + ptr, + ops.len() as u32, + group_lane, + val_lane, + via.0, + via.1, + out.as_mut_ptr(), + groups as u64, + ); + (st, out) +} + +/// Oracle 1: the two-crossing path, per group. +fn two_crossing_oracle(res: u64, ops: &[LgjOpDesc], groups: usize) -> Vec { + (0..groups as u32) + .map(|g| { + let mut plan = ops.to_vec(); + plan.push(op( + LGJ_OP_EQ_U32, + LANE_CLASSES, + i64::from(g), + LGJ_COMBINE_AND, + )); + let mut mask = 0u64; + assert_eq!( + call::mask_create(res, LGJ_MASK_INIT_EMPTY, &mut mask), + LGJ_OK + ); + let mut count = 0u64; + assert_eq!( + call::plan_eval(res, plan.as_ptr(), plan.len() as u32, mask, &mut count), + LGJ_OK + ); + let mut sum = 0i64; + assert_eq!( + call::reduce_sum_i32(res, LANE_VALUES, mask, &mut sum), + LGJ_OK + ); + assert_eq!(lgj_close(mask), LGJ_OK); + sum + }) + .collect() +} + +const TWO_OP: [(u32, u32, i64); 2] = [ + (LGJ_OP_EQ_U32, LANE_CLASSES, 7), + (LGJ_OP_GT_I32, LANE_VALUES, 100), +]; +const THREE_OP: [(u32, u32, i64); 3] = [ + (LGJ_OP_GT_I32, LANE_VALUES, 100), + (LGJ_OP_EQ_U32, LANE_CLASSES, 3), + (LGJ_OP_LT_I32, LANE_VALUES, 300), +]; + +/// `{AND, OR}^n` over both op sets — 4 + 8 = 12 vectors, plus the empty plan. +fn for_each_plan(mut visit: impl FnMut(String, Vec)) { + visit("n=0 []".into(), Vec::new()); + for (base, arity) in [(&TWO_OP[..], 2u32), (&THREE_OP[..], 3u32)] { + for bits in 0u32..(1 << arity) { + let ops: Vec = base + .iter() + .enumerate() + .map(|(i, &(opcode, lane, operand))| { + let combine = if bits & (1 << i) == 0 { + LGJ_COMBINE_AND + } else { + LGJ_COMBINE_OR + }; + op(opcode, lane, operand, combine) + }) + .collect(); + let names: Vec<&str> = (0..arity) + .map(|i| if bits & (1 << i) == 0 { "AND" } else { "OR" }) + .collect(); + visit(format!("n={arity} [{}]", names.join(", ")), ops); + } + } +} + +/// CAN-FIRE + parity: one crossing answers what sixteen pairs of crossings +/// answered, on a single-tile and a multi-tile population. +#[test] +fn grouped_sum_matches_the_two_crossing_path_over_the_combine_sweep() { + for (n, seed) in [(1000u64, 33u64), (65_536, 7)] { + let res = open(n, seed); + let mut distinct_answers = std::collections::HashSet::new(); + for_each_plan(|label, ops| { + let (st, got) = group_sum(res, &ops, LANE_CLASSES, LANE_VALUES, (0, 0), 16); + assert_eq!(st, LGJ_OK, "n={n} {label}"); + let want = two_crossing_oracle(res, &ops, 16); + assert_eq!( + got, want, + "n={n} {label}: one crossing vs the two-crossing path" + ); + assert!( + got.iter().all(|&v| v != SENTINEL), + "{label}: every group is written on success, zero included" + ); + distinct_answers.insert(got); + }); + // Anti-vacuity: the sweep is not one answer repeated. AND-heavy and + // OR-heavy vectors select different populations and must sum + // differently. + assert!( + distinct_answers.len() >= 4, + "n={n}: only {} distinct grouped answers across 13 plans", + distinct_answers.len() + ); + assert_eq!(lgj_close(res), LGJ_OK); + } +} + +/// The empty plan is legal here (unlike `lgj_plan_eval`) and means every +/// row; so does a plan whose every combine is OR. Both equal the per-class +/// sum over the whole fixture, hand-walked. +#[test] +fn an_empty_plan_and_an_all_or_plan_both_group_every_row() { + let (n, seed) = (4096u64, 0xC0FFEE_u64); + let f = Fixture::generate(n, seed).unwrap(); + let mut want = vec![0i64; 16]; + for (c, v) in f.classes().iter().zip(f.values()) { + want[*c as usize] += i64::from(*v); + } + let res = open(n, seed); + let (st, empty) = group_sum(res, &[], LANE_CLASSES, LANE_VALUES, (0, 0), 16); + assert_eq!(st, LGJ_OK); + assert_eq!(empty, want, "n_ops == 0 groups every row"); + let all_or = [ + op(LGJ_OP_EQ_U32, LANE_CLASSES, 7, LGJ_COMBINE_OR), + op(LGJ_OP_GT_I32, LANE_VALUES, 100, LGJ_COMBINE_OR), + ]; + let (st, ored) = group_sum(res, &all_or, LANE_CLASSES, LANE_VALUES, (0, 0), 16); + assert_eq!(st, LGJ_OK); + assert_eq!(ored, want, "an all-OR plan is dead and groups every row"); + assert!( + want.iter().filter(|&&v| v != 0).count() == 16, + "anti-vacuity: 16 live groups" + ); + assert_eq!(lgj_close(res), LGJ_OK); +} + +/// `n_groups` bounds the SINK, not the keys: a selected row whose key is past +/// it is dropped, silently, and the first `n_groups` totals are unchanged. +#[test] +fn keys_past_n_groups_are_dropped_not_errors() { + let res = open(2048, 5); + let ops = [op(LGJ_OP_GT_I32, LANE_VALUES, 0, LGJ_COMBINE_AND)]; + let (st16, full) = group_sum(res, &ops, LANE_CLASSES, LANE_VALUES, (0, 0), 16); + let (st8, eight) = group_sum(res, &ops, LANE_CLASSES, LANE_VALUES, (0, 0), 8); + let (st20, twenty) = group_sum(res, &ops, LANE_CLASSES, LANE_VALUES, (0, 0), 20); + assert_eq!((st16, st8, st20), (LGJ_OK, LGJ_OK, LGJ_OK)); + assert_eq!( + eight, + full[..8], + "a narrower sink keeps the groups it can hold" + ); + assert_eq!( + twenty[..16], + full[..], + "a wider sink changes nothing below 16" + ); + assert!( + twenty[16..].iter().all(|&v| v == 0), + "groups no key names are zero, not sentinel" + ); + assert!( + full[8..].iter().any(|&v| v != 0), + "anti-vacuity: rows WERE dropped at n_groups=8" + ); + assert_eq!(lgj_close(res), LGJ_OK); +} + +/// Oracle 2: the row-at-a-time reference executor on the same lowered +/// program, multi-tile, both key shapes. +#[test] +fn the_tiled_executor_and_the_scalar_reference_agree_on_the_lowered_program() { + let (n, seed) = (5000u64, 0xBEEF_u64); + let a = Fixture::generate(n, seed).unwrap(); + let b = Fixture::generate(64, 0xB0B).unwrap(); + let lanes = [ + LaneRef::U64(a.ids()), + LaneRef::U32(a.classes()), + LaneRef::I32(a.values()), + ]; + let planes = Planes { + n_rows: n as usize, + masks: &[], + lanes: &lanes, + }; + let via_lanes = [LaneRef::U32(b.classes())]; + let foreign = Foreign { + planes: &[], + lanes: &via_lanes, + }; + let keys = [ + plan_lower::GroupKey::Local(LANE_CLASSES as u16), + plan_lower::GroupKey::Via { + fk: LANE_CLASSES as u16, + key: 0, + }, + ]; + let mut checked = 0usize; + for_each_plan(|label, ops| { + for key in keys { + let program = + plan_lower::lower_group_sum(&ops, n as u32, key, LANE_VALUES as u16).unwrap(); + let tile = tile_words_for(n as usize); + let need = scratch_words_for(tile, plan_lower::SLOTS as usize).unwrap(); + let mut buf = vec![0u64; need]; + let mut scratch = Scratch::over(&mut buf, tile, plan_lower::SLOTS as usize).unwrap(); + assert!( + tile < n.div_ceil(64) as usize, + "the population spans several tiles" + ); + let mut simd = vec![0i64; 16]; + let mut scalar = vec![0i64; 16]; + let v = execute_into( + &program, + &planes, + &foreign, + &mut scratch, + Out::I64(&mut simd), + ) + .unwrap(); + assert!(matches!(v, Value::GroupSummed), "{label}"); + let r = + reference_execute_into(&program, &planes, &foreign, Out::I64(&mut scalar)).unwrap(); + assert!(matches!(r, Value::GroupSummed), "{label}"); + assert_eq!( + simd, scalar, + "{label} {key:?}: tiled executor vs scalar reference" + ); + checked += 1; + } + }); + assert_eq!(checked, 26); +} + +/// Oracle 3: the fk-keyed form against a hand walk — `group of row i` is +/// `b.class[a.class[i]]`, with the fixture's own arrays, a hand-evaluated +/// predicate, and no kernel anywhere. Also pins the zero fallback: a `via` +/// table shorter than the key range drops exactly the rows whose key names +/// no row of it. +#[test] +fn via_reads_the_second_table_through_the_key_lane() { + let (n, seed_a) = (4096u64, 0xA11CE_u64); + let a = Fixture::generate(n, seed_a).unwrap(); + let res = open(n, seed_a); + let ops = [op(LGJ_OP_GT_I32, LANE_VALUES, 100, LGJ_COMBINE_AND)]; + for (b_rows, seed_b) in [(64u64, 0xB0B_u64), (8, 0xB0B)] { + let b = Fixture::generate(b_rows, seed_b).unwrap(); + let via = open(b_rows, seed_b); + let mut want = vec![0i64; 16]; + let mut dropped = 0usize; + for i in 0..n as usize { + if a.values()[i] <= 100 { + continue; + } + let k = a.classes()[i] as usize; + if k >= b.classes().len() { + dropped += 1; + continue; + } + want[b.classes()[k] as usize] += i64::from(a.values()[i]); + } + let (st, got) = group_sum( + res, + &ops, + LANE_CLASSES, + LANE_VALUES, + (via, LANE_CLASSES), + 16, + ); + assert_eq!(st, LGJ_OK, "via rows={b_rows}"); + assert_eq!( + got, want, + "via rows={b_rows}: fk-keyed grouped sum vs the hand walk" + ); + // The join genuinely regrouped: the local answer must differ. + let (_, local) = group_sum(res, &ops, LANE_CLASSES, LANE_VALUES, (0, 0), 16); + assert_ne!( + got, local, + "via rows={b_rows}: the fk-keyed form is not the local form" + ); + if b_rows == 8 { + assert!( + dropped > 0, + "anti-vacuity: keys past the via table were dropped" + ); + } else { + assert_eq!( + dropped, 0, + "16 classes all name a row of a 64-row via table" + ); + // With no drops, the totals are a regrouping of the local totals. + assert_eq!(got.iter().sum::(), local.iter().sum::()); + } + assert_eq!(lgj_close(via), LGJ_OK); + } + // `via_res == res` is a legal self-join. + let (st, selfjoin) = group_sum( + res, + &ops, + LANE_CLASSES, + LANE_VALUES, + (res, LANE_CLASSES), + 16, + ); + assert_eq!(st, LGJ_OK); + let mut want = vec![0i64; 16]; + for i in 0..n as usize { + if a.values()[i] > 100 { + want[a.classes()[a.classes()[i] as usize] as usize] += i64::from(a.values()[i]); + } + } + assert_eq!(selfjoin, want); + assert_eq!(lgj_close(res), LGJ_OK); +} + +/// Every refusal is a status, reached before any write: the sentinel survives +/// in every element. +#[test] +fn every_refusal_is_a_status_and_leaves_out_sums_untouched() { + let res = open(512, 1); + let ok_ops = [op(LGJ_OP_GT_I32, LANE_VALUES, 0, LGJ_COMBINE_AND)]; + type Refusal<'a> = (&'a str, i32, (i32, Vec)); + let refusals: Vec> = vec![ + ( + "n_groups == 0", + LGJ_ERR_NULL_ARGUMENT, + group_sum(res, &ok_ops, LANE_CLASSES, LANE_VALUES, (0, 0), 0), + ), + ( + "group lane is I32", + LGJ_ERR_LANE_KIND_MISMATCH, + group_sum(res, &ok_ops, LANE_VALUES, LANE_VALUES, (0, 0), 16), + ), + ( + "group lane is U64", + LGJ_ERR_LANE_KIND_MISMATCH, + group_sum(res, &ok_ops, LANE_IDS, LANE_VALUES, (0, 0), 16), + ), + ( + "value lane is U32", + LGJ_ERR_LANE_KIND_MISMATCH, + group_sum(res, &ok_ops, LANE_CLASSES, LANE_CLASSES, (0, 0), 16), + ), + ( + "group lane out of range", + LGJ_ERR_INVALID_LANE, + group_sum(res, &ok_ops, 9, LANE_VALUES, (0, 0), 16), + ), + ( + "value lane out of range", + LGJ_ERR_INVALID_LANE, + group_sum(res, &ok_ops, LANE_CLASSES, 9, (0, 0), 16), + ), + ( + "unknown opcode in the plan", + LGJ_ERR_UNKNOWN_OPCODE, + group_sum( + res, + &[op(77, LANE_VALUES, 0, LGJ_COMBINE_AND)], + LANE_CLASSES, + LANE_VALUES, + (0, 0), + 16, + ), + ), + ( + "plan lane kind mismatch", + LGJ_ERR_LANE_KIND_MISMATCH, + group_sum( + res, + &[op(LGJ_OP_GT_I32, LANE_CLASSES, 0, LGJ_COMBINE_AND)], + LANE_CLASSES, + LANE_VALUES, + (0, 0), + 16, + ), + ), + ( + "via handle is fabricated", + LGJ_ERR_INVALID_HANDLE, + group_sum( + res, + &ok_ops, + LANE_CLASSES, + LANE_VALUES, + (0xDEAD_BEEF, LANE_CLASSES), + 16, + ), + ), + ( + "via lane is I32", + LGJ_ERR_LANE_KIND_MISMATCH, + group_sum( + res, + &ok_ops, + LANE_CLASSES, + LANE_VALUES, + (res, LANE_VALUES), + 16, + ), + ), + ( + "via lane out of range", + LGJ_ERR_INVALID_LANE, + group_sum(res, &ok_ops, LANE_CLASSES, LANE_VALUES, (res, 9), 16), + ), + ( + "res is fabricated", + LGJ_ERR_INVALID_HANDLE, + group_sum(0xDEAD_BEEF, &ok_ops, LANE_CLASSES, LANE_VALUES, (0, 0), 16), + ), + ]; + for (what, want, (st, out)) in refusals { + assert_eq!(st, want, "{what}"); + assert!( + out.iter().all(|&v| v == SENTINEL), + "{what}: out_sums was written on a refusal" + ); + } + + // A mask handle where a pattern is expected. + let mut mask = 0u64; + assert_eq!(call::mask_create(res, LGJ_MASK_INIT_ALL, &mut mask), LGJ_OK); + let (st, out) = group_sum(mask, &ok_ops, LANE_CLASSES, LANE_VALUES, (0, 0), 16); + assert_eq!(st, LGJ_ERR_WRONG_RESOURCE_KIND); + assert!(out.iter().all(|&v| v == SENTINEL)); + let (st, out) = group_sum( + res, + &ok_ops, + LANE_CLASSES, + LANE_VALUES, + (mask, LANE_CLASSES), + 16, + ); + assert_eq!( + st, LGJ_ERR_WRONG_RESOURCE_KIND, + "a mask is not a via table either" + ); + assert!(out.iter().all(|&v| v == SENTINEL)); + assert_eq!(lgj_close(mask), LGJ_OK); + + // Null pointers are statuses, not UB. + let mut sink = [SENTINEL; 4]; + assert_eq!( + call12::plan_group_sum_i32( + res, + ok_ops.as_ptr(), + 1, + LANE_CLASSES, + LANE_VALUES, + 0, + 0, + std::ptr::null_mut(), + 4 + ), + LGJ_ERR_NULL_ARGUMENT + ); + assert_eq!( + call12::plan_group_sum_i32( + res, + std::ptr::null(), + 1, + LANE_CLASSES, + LANE_VALUES, + 0, + 0, + sink.as_mut_ptr(), + 4 + ), + LGJ_ERR_NULL_ARGUMENT, + "a null plan with n_ops > 0" + ); + assert!(sink.iter().all(|&v| v == SENTINEL)); + + // And after close, the generation-checked handle refuses. + assert_eq!(lgj_close(res), LGJ_OK); + let (st, out) = group_sum(res, &ok_ops, LANE_CLASSES, LANE_VALUES, (0, 0), 16); + assert_eq!(st, LGJ_ERR_INVALID_HANDLE); + assert!(out.iter().all(|&v| v == SENTINEL)); +} diff --git a/native/lgj-abi/src/exports/tests/lowering_convergence.rs b/native/lgj-abi/src/exports/tests/lowering_convergence.rs index b9bda78..1c8ffaf 100644 --- a/native/lgj-abi/src/exports/tests/lowering_convergence.rs +++ b/native/lgj-abi/src/exports/tests/lowering_convergence.rs @@ -68,7 +68,7 @@ use super::*; use lance_graph_mask_risc::Program; -use lance_graph_quack::{Agg, Cmp, Col, Filter, Query}; +use lance_graph_quack::{Agg, Cmp, Col, Filter, Mask as QuackMask, Query}; /// `n = 1000, seed = 33` — the identical fixture /// `pr4_matrix.rs::combine_mode_products_agree_with_the_oracle` sweeps, so @@ -654,3 +654,69 @@ fn every_opcode_maps_to_the_same_predicate_in_both_lowerings() { "TCAM swapped halves at position 1 of 3", ); } + +/// The minor-12 arm of the same differential: `plan_lower::lower_group_sum` +/// against `lance_graph_quack::lower` with `Agg::GroupSumI32`, both run +/// through `execute_into` into an `Out::I64` — one law, two lowerings, now +/// for the grouped sum as well as the count. The all-rows plan is included: +/// quack spells it `Filter::Plane(alpha)` over a resident all-ones plane, +/// `plan_lower` spells it the whole-lane `Range`; they must total the same. +#[test] +fn grouped_sums_agree_with_quack_over_the_combine_sweep() { + let fixture = Fixture::generate(SWEEP_N, SWEEP_SEED).expect("fixture"); + let lanes = [ + LaneRef::U64(fixture.ids()), + LaneRef::U32(fixture.classes()), + LaneRef::I32(fixture.values()), + ]; + let words = (SWEEP_N as usize).div_ceil(64); + let mut alpha = vec![u64::MAX; words]; + let tail = (SWEEP_N as usize) % 64; + if tail != 0 { + alpha[words - 1] = (1u64 << tail) - 1; + } + let alpha_ref: &[u64] = α + let planes = Planes { + n_rows: SWEEP_N as usize, + masks: &[alpha_ref], + lanes: &lanes, + }; + let run_groups = |p: &Program| -> Vec { + let mut scratch = Scratch::for_program(p, planes.n_rows).expect("addressable"); + let mut out = vec![0i64; 16]; + let v = execute_into(p, &planes, &Foreign::NONE, &mut scratch, Out::I64(&mut out)) + .expect("runs"); + assert!(matches!(v, Value::GroupSummed), "not a grouped sum: {v:?}"); + out + }; + let mut distinct = std::collections::HashSet::new(); + let mut check = |label: String, ops: Vec| { + let ours = plan_lower::lower_group_sum( + &ops, + SWEEP_N as u32, + plan_lower::GroupKey::Local(LANE_CLASSES as u16), + LANE_VALUES as u16, + ) + .expect("lowers"); + let filter = fold_to_tree(&ops).unwrap_or(Filter::Plane(QuackMask(0))); + let q = Query { + filter, + agg: Agg::GroupSumI32 { + key: Col(LANE_CLASSES as u16), + val: Col(LANE_VALUES as u16), + }, + }; + let theirs = lance_graph_quack::lower(&q).expect("quack lowers"); + let a = run_groups(&ours); + let b = run_groups(&theirs); + assert_eq!(a, b, "{label}: plan_lower's grouped sum vs quack's"); + distinct.insert(a); + }; + check("n=0 []".into(), Vec::new()); + for_each_combine_vector(&mut check); + assert!( + distinct.len() >= 4, + "anti-vacuity: {} distinct answers", + distinct.len() + ); +} diff --git a/native/lgj-abi/src/plan_lower.rs b/native/lgj-abi/src/plan_lower.rs index d1e3743..7675001 100644 --- a/native/lgj-abi/src/plan_lower.rs +++ b/native/lgj-abi/src/plan_lower.rs @@ -42,6 +42,19 @@ use crate::abi::{ }; use lance_graph_mask_risc::{MaskOp, Operand, Pred, Program, Terminal}; +/// Which lane a grouped sum groups by — this table's own `u32` lane, or a +/// `u32` lane of ANOTHER table read through a `u32` key lane of this one. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum GroupKey { + /// `GROUP BY key` where `key` is a `U32` lane of the executing table. + Local(u16), + /// `GROUP BY other.key` read as `foreign.lanes[key][fk[i]]` — `fk` is a + /// `U32` lane of the executing table, `key` indexes the `Foreign::lanes` + /// the caller supplies. The indirection is fused inside the terminal; no + /// remapped key lane and no partner-side mask ever exist. + Via { fk: u16, key: u16 }, +} + /// Scratch slots the lowering ever names: slot 0 is the accumulator (the old /// `acc`, now living in the caller's arena rather than in a per-call `vec!`), /// slot 1 the per-op predicate destination (the old `scratch`). @@ -108,6 +121,53 @@ pub(crate) fn lower_plan(ops: &[LgjOpDesc]) -> Option { ))) } +/// Lower a validated plan STRAIGHT INTO a grouped sum — the same prefix +/// rewrite and survivor skip as [`lower_plan`], ending in +/// [`Terminal::GroupSumI32`] / [`Terminal::GroupSumViaI32`] over the +/// accumulator instead of [`Terminal::Keep`]. +/// +/// This is the shape that lets a `GROUP BY … SUM` cross the membrane once +/// and return only group totals: no `Keep` lands a population anywhere, the +/// accumulator lives in tile-local scratch, and the terminal folds `val` into +/// the caller's `i64` buffer tile by tile. +/// +/// # The all-rows arm +/// +/// [`lower_plan`] answers `AllRows` without a program because its caller can +/// fill the destination mask itself. A grouped sum has no destination mask +/// to fill, so the all-rows case — an empty plan, or one whose every combine +/// is OR — needs an accumulator that IS every row. That is +/// [`Pred::Range`]`{ lo: 0, hi: n_rows }`: a three-pass fill over the tile's +/// words (`mask_set_range`), reading no lane. It is the ONLY `Range` this +/// crate ever emits, and with `hi == n_rows` exactly it can never leave the +/// lane, which is what keeps `exec_error_to_status`'s `RangeOutOfBounds` arm +/// an internal-bug mapping — pinned by +/// `range_falsifier::the_group_sum_range_is_the_whole_lane_and_nothing_else`. +/// +/// `n_rows` is `u32` because [`Pred::Range`] is; a caller with more rows than +/// that cannot take the all-rows arm and must say so before lowering. +pub(crate) fn lower_group_sum( + ops: &[LgjOpDesc], + n_rows: u32, + group: GroupKey, + val: u16, +) -> Option { + let mask = Operand::Scratch(ACC_SLOT); + let terminal = match group { + GroupKey::Local(key) => Terminal::GroupSumI32 { mask, key, val }, + GroupKey::Via { fk, key } => Terminal::GroupSumViaI32 { mask, fk, key, val }, + }; + let program_ops = match lower_plan(ops)? { + Lowered::AllRows => vec![MaskOp::Pred { + pred: Pred::Range { lo: 0, hi: n_rows }, + under: None, + dst: ACC_SLOT, + }], + Lowered::Program(p) => p.ops, + }; + Some(Program::new(program_ops, terminal)) +} + /// One `LgjOpDesc` opcode + operand → one `Pred`. /// /// Every operand conversion is the one `kernels::eval_predicate` performs, and @@ -182,6 +242,12 @@ mod range_falsifier { /// whole purpose: it forces the author to decide, deliberately and /// visibly, what a caller should see when a range leaves the lane — rather /// than inheriting `LGJ_ERR_ALLOCATION_FAILED`, which would then be a lie. + /// + /// Minor 12 did gain ONE `Range` — [`lower_group_sum`]'s all-rows arm — and + /// decided it deliberately: that range is always `0..n_rows`, so it cannot + /// leave the lane and the internal-bug mapping stays true. It is pinned by + /// the sibling test below; this sweep still holds for every opcode through + /// `lower_plan`, which is the path a caller's operand reaches. #[test] fn no_opcode_lowers_to_pred_range() { let mut lowered_count = 0usize; @@ -220,4 +286,68 @@ mod range_falsifier { "the sweep lowered only {lowered_count} programs — it is not exercising the lowering" ); } + + /// The one `Range` this crate emits is the whole lane, in the all-rows arm + /// only. An empty plan and an all-OR plan both take it; a plan with an AND + /// emits none. Any other `Range` would make `exec_error_to_status`'s + /// `RangeOutOfBounds` arm reachable through the ABI, and this is the test + /// that would say so. + #[test] + fn the_group_sum_range_is_the_whole_lane_and_nothing_else() { + let or_op = LgjOpDesc { + op: LGJ_OP_EQ_U32, + lane_id: 1, + operand: 7, + combine: LGJ_COMBINE_OR, + _reserved: 0, + }; + let and_op = LgjOpDesc { + combine: LGJ_COMBINE_AND, + ..or_op + }; + let ranges = |ops: &[LgjOpDesc]| -> Vec<(u32, u32)> { + let p = lower_group_sum(ops, 1000, GroupKey::Local(1), 2).expect("lowers"); + p.ops + .iter() + .filter_map(|op| match op { + MaskOp::Pred { + pred: Pred::Range { lo, hi }, + .. + } => Some((*lo, *hi)), + _ => None, + }) + .collect() + }; + assert_eq!(ranges(&[]), vec![(0, 1000)], "an empty plan is every row"); + assert_eq!( + ranges(&[or_op, or_op]), + vec![(0, 1000)], + "an all-OR plan is every row" + ); + assert!( + ranges(&[and_op]).is_empty(), + "a plan with an AND needs no range" + ); + assert!(ranges(&[or_op, and_op, or_op]).is_empty()); + // And the terminal is the grouped sum over the accumulator, both shapes. + let local = lower_group_sum(&[and_op], 1000, GroupKey::Local(1), 2).unwrap(); + assert_eq!( + local.terminal, + Terminal::GroupSumI32 { + mask: Operand::Scratch(ACC_SLOT), + key: 1, + val: 2 + } + ); + let via = lower_group_sum(&[], 1000, GroupKey::Via { fk: 1, key: 0 }, 2).unwrap(); + assert_eq!( + via.terminal, + Terminal::GroupSumViaI32 { + mask: Operand::Scratch(ACC_SLOT), + fk: 1, + key: 0, + val: 2 + } + ); + } }