From c5914d77cd2f690de60b37fdc1a3e390badee947 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 04:55:37 +0000 Subject: [PATCH 1/2] lgj-abi minor 12: the grouped sum that never builds a selection MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `lgj_plan_group_sum_i32` lowers the fused plan and `GROUP BY key SUM(val)` as ONE mask-risc program (`plan_lower::lower_group_sum`, the same prefix rewrite and survivor skip as `plan_eval`, ending in `Terminal::GroupSumI32` over the accumulator instead of `Keep`). The executor runs it tile by tile; the only state sized by the answer is the caller's `i64` buffer, one per group, and that is all that crosses back. Java's `View.sumByGroup` is therefore one crossing for 16 groups at any row count, where the path it replaces (plan_eval into a mask, then reduce per group) measured 32. `via_res`/`via_lane` give the fk-keyed form: the group of a row is the second table's `u32` lane read through this table's key lane, fused inside `Terminal::GroupSumViaI32` — no partner-side mask, no remapped key lane. `View.sumByGroupVia` is the Java spelling, still one crossing. An empty plan is legal here (there is no destination mask a caller could fill itself) and lowers to the whole-lane `Pred::Range { 0, n_rows }` — the one Range this crate emits, pinned to be that range and nothing else so `exec_error_to_status`'s RangeOutOfBounds arm stays an internal-bug mapping. Native: 191 tests (8 new: two-crossing-path parity over the combine sweep on single- and multi-tile populations, empty/all-OR plans against a hand walk, n_groups as a sink bound, tiled executor vs scalar reference on both key shapes, the fk-keyed form against a hand-walked join with zero-fallback drops, every refusal leaving out_sums untouched, and a quack-lowering convergence arm with Agg::GroupSumI32). Clippy -D warnings and fmt clean. Java (JDK 28, --enable-preview): 612 checks (409 + GroupSumTest 203): parity against per-group sumOf, one-crossing cost measured beside the 32-crossing path, the keyed form against an oracle that reads the second resource's keys through the one named materialiser, argument checks before any crossing. Minor12 lazy holder, requireMinor(12), an OldAbiCompatTest gate, and the DoctrineFenceTest pin for the eighth named materialisation site (Engine.groupSumI32's toArray, sized by groups, never rows). Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GXUahz73MZxtxWcfpHp9dG --- .../adaworldapi/lancegraph/GroupTotals.java | 53 ++ .../adaworldapi/lancegraph/NativePattern.java | 35 ++ .../java/com/adaworldapi/lancegraph/View.java | 58 ++ .../lancegraph/internal/ffm/Downcalls.java | 38 ++ .../lancegraph/internal/ffm/Engine.java | 31 + .../com/adaworldapi/lancegraph/AllTests.java | 1 + .../lancegraph/DoctrineFenceTest.java | 5 +- .../adaworldapi/lancegraph/GroupSumTest.java | 191 +++++++ .../lancegraph/OldAbiCompatTest.java | 13 + native/lgj-abi/src/abi.rs | 13 +- native/lgj-abi/src/exports.rs | 330 ++++++++++- native/lgj-abi/src/exports/tests/group_sum.rs | 536 ++++++++++++++++++ .../src/exports/tests/lowering_convergence.rs | 68 ++- native/lgj-abi/src/plan_lower.rs | 130 +++++ 14 files changed, 1494 insertions(+), 8 deletions(-) create mode 100644 java/src/main/java/com/adaworldapi/lancegraph/GroupTotals.java create mode 100644 java/src/test/java/com/adaworldapi/lancegraph/GroupSumTest.java create mode 100644 native/lgj-abi/src/exports/tests/group_sum.rs diff --git a/java/src/main/java/com/adaworldapi/lancegraph/GroupTotals.java b/java/src/main/java/com/adaworldapi/lancegraph/GroupTotals.java new file mode 100644 index 0000000..b4f50b6 --- /dev/null +++ b/java/src/main/java/com/adaworldapi/lancegraph/GroupTotals.java @@ -0,0 +1,53 @@ +package com.adaworldapi.lancegraph; + +/** + * The answer to a grouped sum: one widened total per group, addressed by the group's key value. + * + *

This is what {@code SELECT key, SUM(value) … GROUP BY key} hands back — the same thing a + * {@code ResultSet} would, without the cursor. {@link #total(int)} is the row for key {@code g}; + * {@link #groups()} is how many keys were asked for. A key no selected row carries has a total of + * {@code 0}, exactly as SQL would report it with an outer join onto the key domain. + * + *

Sized by the question, never by the data: a {@code GroupTotals} over 16 keys is 16 numbers + * whether the view spans a thousand rows or a billion. No row, no selection and no index list is + * behind it — the totals were folded natively in one pass and only the totals crossed. + * + *

Immutable. There is no accessor for the underlying storage; the totals are read one at a + * time by key, which is also the only way a caller ever needs them. + */ +public final class GroupTotals { + + private final long[] totals; + + GroupTotals(long[] totals) { + this.totals = totals; + } + + /** How many groups this answer covers — the {@code groups} the caller asked for. */ + public int groups() { + return totals.length; + } + + /** + * The sum for key {@code group}, widened to 64 bits; {@code 0} when no selected row carried + * that key. + * + * @throws IndexOutOfBoundsException if {@code group} is not in {@code [0, groups())} + */ + public long total(int group) { + java.util.Objects.checkIndex(group, totals.length); + return totals[group]; + } + + @Override + public String toString() { + StringBuilder sb = new StringBuilder("GroupTotals["); + for (int g = 0; g < totals.length; g++) { + if (g > 0) { + sb.append(", "); + } + sb.append(g).append('=').append(totals[g]); + } + return sb.append(']').toString(); + } +} diff --git a/java/src/main/java/com/adaworldapi/lancegraph/NativePattern.java b/java/src/main/java/com/adaworldapi/lancegraph/NativePattern.java index f1bd991..bbaf50c 100644 --- a/java/src/main/java/com/adaworldapi/lancegraph/NativePattern.java +++ b/java/src/main/java/com/adaworldapi/lancegraph/NativePattern.java @@ -226,6 +226,41 @@ OptionalLong maxOf(List predicates, I32Field field) { return reduceI32(predicates, field, Layouts.REDUCE_OP_MAX); } + /** + * {@code GROUP BY key SUM(value)} over the rows the plan selects, as ONE native program + * (docs/abi.md §20). No selection is built: the plan is not evaluated into a mask and then + * reduced per group, it is folded straight into the group totals tile by tile. An empty + * plan crosses as such — the ABI treats it as every row — rather than being routed through + * the cached all-rows selection {@link #sumOf} uses, because here there is nothing for a + * selection to be consumed by. + */ + GroupTotals sumByGroup(List predicates, U32Field key, I32Field value, int groups) { + requireOpen("sumByGroup()"); + synchronized (lock) { + requireOpen("sumByGroup()"); + return new GroupTotals(Engine.groupSumI32(handle, plan(predicates), + key.lane().index(), value.lane().index(), 0L, 0, groups)); + } + } + + /** + * The fk-keyed form of {@link #sumByGroup}: the group of a row is {@code viaKey} read on + * {@code via} at the row index this resource's {@code key} names — {@code SUM(line.amount) + * GROUP BY partner.country}, still one program and one crossing, with no selection on either + * resource. {@code via} may be this resource. + */ + GroupTotals sumByGroupVia(List predicates, U32Field key, NativePattern via, + U32Field viaKey, I32Field value, int groups) { + requireOpen("sumByGroupVia()"); + via.requireOpen("sumByGroupVia() via"); + synchronized (lock) { + requireOpen("sumByGroupVia()"); + return new GroupTotals(Engine.groupSumI32(handle, plan(predicates), + key.lane().index(), value.lane().index(), via.handle(), + viaKey.lane().index(), groups)); + } + } + /** Materialise a selection the caller owns and closes. */ Mask selectInto(List predicates) { requireOpen("select()"); diff --git a/java/src/main/java/com/adaworldapi/lancegraph/View.java b/java/src/main/java/com/adaworldapi/lancegraph/View.java index bdd2d04..7c21e89 100644 --- a/java/src/main/java/com/adaworldapi/lancegraph/View.java +++ b/java/src/main/java/com/adaworldapi/lancegraph/View.java @@ -114,6 +114,64 @@ public OptionalLong maxOf(I32Field field) { return owner.maxOf(predicates, field); } + /** + * {@code SELECT key, SUM(value) FROM … WHERE GROUP BY key}, for keys + * {@code 0..groups}. + * + *

One crossing, whatever the number of conditions, groups or rows. The + * chain and the grouped sum are one fused native program; nothing is selected first and + * nothing per group is asked separately. Before this existed the same question cost two + * crossings per group — a {@code sumOf} over {@code where(key.eq(g))} for each {@code g} — + * which the {@code bricks} consumer measured at 32 for 16 groups; it is now 1 for any number. + * + *

A key no selected row carries totals {@code 0}. A selected row whose key is + * {@code >= groups} belongs to no requested group and is dropped, as a {@code WHERE key < n} + * would drop it; ask for more groups to see it. + * + * @param key an unsigned 32-bit column whose values are the group keys + * @param value the signed 32-bit column to sum, widened to 64 bits + * @param groups how many keys to answer for, {@code > 0} + * @throws AbiMismatchException if the loaded library reports ABI minor < 12 + */ + public GroupTotals sumByGroup(U32Field key, I32Field value, int groups) { + java.util.Objects.requireNonNull(key, "key"); + java.util.Objects.requireNonNull(value, "value"); + requirePositiveGroups(groups); + return owner.sumByGroup(predicates, key, value, groups); + } + + /** + * {@code SELECT via.viaKey, SUM(value) FROM this JOIN via ON via.row = this.key … GROUP BY + * via.viaKey} — the grouped sum keyed through a second resource, still one + * crossing. + * + *

{@code key} holds, for each row here, the row index in {@code via} it refers to; the + * group of that row is the {@code viaKey} value found there. The lookup is fused into the + * native fold — no selection on either resource, no remapped key column. A {@code key} that + * names no row of {@code via} drops the row, as an inner join would. + * + * @param via the resource whose {@code viaKey} column supplies the group; may be this view's + * own resource + * @throws AbiMismatchException if the loaded library reports ABI minor < 12 + */ + public GroupTotals sumByGroupVia(U32Field key, NativePattern via, U32Field viaKey, + I32Field value, int groups) { + java.util.Objects.requireNonNull(key, "key"); + java.util.Objects.requireNonNull(via, "via"); + java.util.Objects.requireNonNull(viaKey, "viaKey"); + java.util.Objects.requireNonNull(value, "value"); + requirePositiveGroups(groups); + return owner.sumByGroupVia(predicates, key, via, viaKey, value, groups); + } + + private static void requirePositiveGroups(int groups) { + if (groups <= 0) { + throw new IllegalArgumentException( + "groups must be positive (the number of key values to answer for), was " + + groups); + } + } + /** * A projection of one column through this view. * diff --git a/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Downcalls.java b/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Downcalls.java index 8e8e38e..7b1cc60 100644 --- a/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Downcalls.java +++ b/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Downcalls.java @@ -491,6 +491,21 @@ private static final class Minor11 { private Minor11() {} } + /** + * ABI minor 12 symbol (docs/abi.md §20): the fused plan run straight into a grouped-sum + * terminal — one crossing returns one {@code i64} per group and no selection ever exists. + * Lazy per the minor-2..11 rule. + */ + private static final class Minor12 { + static final MethodHandle PLAN_GROUP_SUM_I32 = mh("lgj_plan_group_sum_i32", + FunctionDescriptor.of(ValueLayout.JAVA_INT, ValueLayout.JAVA_LONG, + ValueLayout.ADDRESS, ValueLayout.JAVA_INT, ValueLayout.JAVA_INT, + ValueLayout.JAVA_INT, ValueLayout.JAVA_LONG, ValueLayout.JAVA_INT, + ValueLayout.ADDRESS, ValueLayout.JAVA_LONG)); + + private Minor12() {} + } + /** * Sum one facet's 12-byte register, under {@code carving}, over the rows a mask selects. * @@ -639,6 +654,29 @@ public static long reduceI32(long res, int laneId, int reduceOp, long mask, return outValue.get(ValueLayout.JAVA_LONG, 0); } + /** + * {@code GROUP BY groupLane SUM(valLane)} over the rows the plan selects, in ONE crossing + * (docs/abi.md §20, minor 12). {@code outSums} receives {@code nGroups} widened totals; a + * selected row whose key is past {@code nGroups} is dropped, never an error. With + * {@code viaRes != 0} the key is read THROUGH {@code groupLane} into {@code viaRes}'s + * {@code viaLane} (the fk-keyed form). {@code nOps == 0} is legal and means every row. + * + *

Bulk in the §6 sense: one pass over the predicate lanes, one read of the key and value + * lanes for the selected rows, no per-row and no per-group crossing. + */ + public static void planGroupSumI32(long res, MemorySegment ops, int nOps, int groupLane, + int valLane, long viaRes, int viaLane, MemorySegment outSums, long nGroups) { + crossed(); + int st; + try { + st = (int) Minor12.PLAN_GROUP_SUM_I32.invokeExact(res, ops, nOps, groupLane, valLane, + viaRes, viaLane, outSums, nGroups); + } catch (Throwable t) { + throw wrap("lgj_plan_group_sum_i32", t); + } + Status.check("lgj_plan_group_sum_i32", st); + } + // ── row store (docs/abi.md §11, ABI minor 2) ───────────────────────────────────────────── // // Callers above this class are expected to have already checked Abi.requireMinor(2) — these diff --git a/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Engine.java b/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Engine.java index f7d59a1..4a74f37 100644 --- a/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Engine.java +++ b/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Engine.java @@ -379,6 +379,37 @@ public static ReduceOutcome reduceI32(long resource, int laneId, int reduceOp, l return new ReduceOutcome(value, present); } + /** + * The grouped sum that never builds a selection (docs/abi.md §20, ABI minor 12): + * {@code GROUP BY groupLane SUM(valLane)} over the rows {@code plan} selects, evaluated as + * ONE native program and returned as one widened total per group. Unlike every reduce + * above, no mask handle is involved at any point — the plan is not evaluated INTO anything; + * the fold reads it tile by tile and lands only the group totals. + * + *

{@code viaResource != 0} selects the fk-keyed form: the group of a row is + * {@code viaLane[groupLane[row]]}, read on {@code viaResource}, with no partner-side + * selection and no remapped key lane (the indirection is fused inside the native terminal). + * + *

An empty {@code plan} is legal here and means every row — the ABI has no destination + * mask for Java to fill on its own, so the whole-lane case crosses as a plan of zero ops + * rather than being answered locally. Requires ABI minor >= 12. + * + *

The returned array is sized by {@code groups} — the shape of the QUESTION the caller + * typed, never by the row count — which is what keeps it on the right side of the + * materialization rule (the eighth named site; see the repo's zero-copy section). + */ + public static long[] groupSumI32(long resource, List plan, int groupLane, int valLane, + long viaResource, int viaLane, int groups) { + Abi.requireMinor(12); + MemorySegment ops = plan.isEmpty() ? MemorySegment.NULL : marshal(plan); + try (Arena a = Arena.ofConfined()) { + MemorySegment out = a.allocate(ValueLayout.JAVA_LONG, groups); + Downcalls.planGroupSumI32(resource, ops, plan.size(), groupLane, valLane, viaResource, + viaLane, out, groups); + return out.toArray(ValueLayout.JAVA_LONG); + } + } + // ── mask complement + hop (docs/abi.md §13, ABI minor ≥ 4) ───────────────────────────── // // Same requireMinor-before-any-downcall discipline as the row store section above: a Java diff --git a/java/src/test/java/com/adaworldapi/lancegraph/AllTests.java b/java/src/test/java/com/adaworldapi/lancegraph/AllTests.java index 054c4a9..917e21d 100644 --- a/java/src/test/java/com/adaworldapi/lancegraph/AllTests.java +++ b/java/src/test/java/com/adaworldapi/lancegraph/AllTests.java @@ -34,6 +34,7 @@ public static void main(String[] args) { suites.put("CarvingTableTest", CarvingTableTest::run); suites.put("ColumnarStoreTest", ColumnarStoreTest::run); suites.put("MaskingOpCompletionTest", MaskingOpCompletionTest::run); + suites.put("GroupSumTest", GroupSumTest::run); if (!NativeRuntime.isAvailable()) { // ApiSurfaceTest and DoctrineFenceTest need no native library — the API's shape is a diff --git a/java/src/test/java/com/adaworldapi/lancegraph/DoctrineFenceTest.java b/java/src/test/java/com/adaworldapi/lancegraph/DoctrineFenceTest.java index 6ee74f5..3650b1f 100644 --- a/java/src/test/java/com/adaworldapi/lancegraph/DoctrineFenceTest.java +++ b/java/src/test/java/com/adaworldapi/lancegraph/DoctrineFenceTest.java @@ -100,6 +100,9 @@ private DoctrineFenceTest() {} // five named sites, mechanically: // Mask.materializeRows() → Arrays.copyOf ×2 + new long[ ×1 (the named terminal) // Engine.rowLayoutProbe → .toArray( ×1 (≤32 B diagnostic) + // Engine.groupSumI32 → .toArray( ×1 (one i64 per GROUP — + // sized by the question the caller typed, never by rows; + // the eighth named site, minor 12) // Engine.facetSumResolved → new long[ ×1 (fixed [2] pair) // Abi manifest-name read (cString) → new byte[ ×1 + MemorySegment.copy ×1 // Abi.readCarvings → new int[ ×2 (incl. empty arm) @@ -108,7 +111,7 @@ private DoctrineFenceTest() {} static { MATERIALIZATION_PINS.put("Mask.java|Arrays.copyOf", 2); MATERIALIZATION_PINS.put("Mask.java|new long[", 1); - MATERIALIZATION_PINS.put("Engine.java|.toArray(", 1); + MATERIALIZATION_PINS.put("Engine.java|.toArray(", 2); MATERIALIZATION_PINS.put("Engine.java|new long[", 1); MATERIALIZATION_PINS.put("Abi.java|new int[", 2); MATERIALIZATION_PINS.put("Abi.java|new byte[", 1); diff --git a/java/src/test/java/com/adaworldapi/lancegraph/GroupSumTest.java b/java/src/test/java/com/adaworldapi/lancegraph/GroupSumTest.java new file mode 100644 index 0000000..2d951cb --- /dev/null +++ b/java/src/test/java/com/adaworldapi/lancegraph/GroupSumTest.java @@ -0,0 +1,191 @@ +package com.adaworldapi.lancegraph; + +/** + * Minor 12 through the public facade: a {@code GROUP BY} that costs one crossing and builds no + * selection. + * + *

Three claims, each measured rather than asserted: + * + *

    + *
  1. Parity. {@link View#sumByGroup} answers exactly what the two-crossing + * path it replaces answered — {@code where(key.eq(g)).sumOf(value)} for every {@code g} — + * on a single-tile and a multi-tile resource, with and without a chain. The two paths run + * different native kernels behind different terminals; the numbers must still agree. + *
  2. Cost. One crossing for 16 groups, at 1,024 rows and at 65,536 rows, + * where the per-group path pays 32. The counter is the membrane's own. + *
  3. The keyed form is a real join. {@link View#sumByGroupVia} regroups + * through a second resource; the oracle recomputes the regrouping from that resource's + * own key column (read through the ONE named materialiser, in test code, where it belongs) + * and the local totals, and the two must differ from each other. + *
+ */ +public final class GroupSumTest { + + private GroupSumTest() {} + + private static final int GROUPS = 16; // Pattern.CLASS spans 0..15 + + public static void main(String[] args) { + System.out.println("GroupSumTest"); + if (!NativeRuntime.isAvailable()) { + System.exit(Checks.reportUnavailable("GroupSumTest")); + } + Checks c = new Checks("GroupSumTest"); + run(c); + System.exit(c.report()); + } + + public static void run(Checks c) { + if (NativeRuntime.abiMinor() < 12) { + c.that("SKIPPED: library minor " + NativeRuntime.abiMinor() + + " predates the grouped sum (minor 12); OldAbiCompatTest covers the gate", true); + return; + } + + c.section("parity: one crossing answers what sixteen pairs of crossings answered"); + for (long rows : new long[] {1_024L, 65_536L}) { + try (NativePattern data = NativePattern.open(rows, 0x51DEL)) { + View[] views = { + data.view(), + data.view().where(Pattern.VALUE.gt(100)), + data.view().where(Pattern.VALUE.gt(100)).where(Pattern.VALUE.gt(-50)), + data.view().where(Pattern.CLASS.eq(7)).where(Pattern.VALUE.gt(0)), + }; + for (View v : views) { + GroupTotals t = v.sumByGroup(Pattern.CLASS, Pattern.VALUE, GROUPS); + c.eq(rows + " rows, " + v + ": groups()", GROUPS, t.groups()); + long live = 0; + for (int g = 0; g < GROUPS; g++) { + long want = v.where(Pattern.CLASS.eq(g)).sumOf(Pattern.VALUE); + c.eq(rows + " rows, " + v + ", group " + g, want, t.total(g)); + if (want != 0) { + live++; + } + } + // Anti-vacuity: the totals are not sixteen zeros agreeing with sixteen zeros. + c.that(rows + " rows, " + v + ": at least one group is non-zero", live >= 1); + } + GroupTotals all = views[0].sumByGroup(Pattern.CLASS, Pattern.VALUE, GROUPS); + long everything = 0; + for (int g = 0; g < GROUPS; g++) { + everything += all.total(g); + } + c.eq(rows + " rows: the empty chain's groups partition the whole sum", + views[0].sumOf(Pattern.VALUE), everything); + } + } + + c.section("a narrower key domain drops, a wider one pads with zero"); + try (NativePattern data = NativePattern.open(4_096L, 0x51DEL)) { + View v = data.view().where(Pattern.VALUE.gt(0)); + GroupTotals full = v.sumByGroup(Pattern.CLASS, Pattern.VALUE, GROUPS); + GroupTotals eight = v.sumByGroup(Pattern.CLASS, Pattern.VALUE, 8); + GroupTotals twenty = v.sumByGroup(Pattern.CLASS, Pattern.VALUE, 20); + for (int g = 0; g < 8; g++) { + c.eq("groups=8 keeps group " + g, full.total(g), eight.total(g)); + } + for (int g = 0; g < GROUPS; g++) { + c.eq("groups=20 keeps group " + g, full.total(g), twenty.total(g)); + } + c.eq("groups=20: key 16 is unnamed and zero", 0, twenty.total(16)); + c.eq("groups=20: key 19 is unnamed and zero", 0, twenty.total(19)); + c.throwsUp("total(groups()) is out of range", IndexOutOfBoundsException.class, + () -> eight.total(8)); + } + + c.section("cost: one crossing for sixteen groups, at any row count"); + try (NativePattern small = NativePattern.open(1_024L, 0xC0DEL); + NativePattern large = NativePattern.open(65_536L, 0xC0DEL)) { + View sv = small.view().where(Pattern.VALUE.gt(100)); + View lv = large.view().where(Pattern.VALUE.gt(100)); + // Warm: the first terminal op on a resource may resolve lazily-held state. + sv.sumByGroup(Pattern.CLASS, Pattern.VALUE, GROUPS); + lv.sumByGroup(Pattern.CLASS, Pattern.VALUE, GROUPS); + + long a = Diagnostics.crossings(); + sv.sumByGroup(Pattern.CLASS, Pattern.VALUE, GROUPS); + long smallCost = Diagnostics.crossings() - a; + long b = Diagnostics.crossings(); + lv.sumByGroup(Pattern.CLASS, Pattern.VALUE, GROUPS); + long largeCost = Diagnostics.crossings() - b; + c.eq("1,024 rows, 16 groups: sumByGroup crosses once", 1, smallCost); + c.eq("65,536 rows, 16 groups: sumByGroup crosses once", 1, largeCost); + + // The path it replaces, measured beside it rather than remembered from bricks. + lv.where(Pattern.CLASS.eq(0)).sumOf(Pattern.VALUE); + long d = Diagnostics.crossings(); + for (int g = 0; g < GROUPS; g++) { + lv.where(Pattern.CLASS.eq(g)).sumOf(Pattern.VALUE); + } + long perGroupCost = Diagnostics.crossings() - d; + c.eq("the per-group path pays two crossings per group", 2L * GROUPS, perGroupCost); + + long e = Diagnostics.crossings(); + large.view().sumByGroup(Pattern.CLASS, Pattern.VALUE, GROUPS); + c.eq("an empty chain still crosses exactly once (the plan of zero ops is the ABI's)", + 1, Diagnostics.crossings() - e); + } + + c.section("the keyed form regroups through a second resource, in one crossing"); + try (NativePattern lines = NativePattern.open(4_096L, 0xA11CEL); + NativePattern partners = NativePattern.open(64L, 0xB0BL)) { + View v = lines.view().where(Pattern.VALUE.gt(100)); + GroupTotals local = v.sumByGroup(Pattern.CLASS, Pattern.VALUE, GROUPS); + GroupTotals via = v.sumByGroupVia(Pattern.CLASS, partners, Pattern.CLASS, + Pattern.VALUE, GROUPS); + + // Oracle: partner row k (k < 16, the only rows a line's CLASS can name) carries + // CLASS = c(k); the via total for g is the sum of the local totals over every k + // with c(k) == g. c(k) is read from `partners` through the one named materialiser. + long[] want = new long[GROUPS]; + for (int g = 0; g < GROUPS; g++) { + try (Mask m = partners.view().where(Pattern.CLASS.eq(g)).select()) { + for (long k : m.materializeRows()) { + if (k < GROUPS) { + want[g] += local.total((int) k); + } + } + } + } + boolean differs = false; + for (int g = 0; g < GROUPS; g++) { + c.eq("via group " + g, want[g], via.total(g)); + differs |= via.total(g) != local.total(g); + } + c.that("the join genuinely regrouped (via totals differ from local totals)", differs); + + long x = Diagnostics.crossings(); + v.sumByGroupVia(Pattern.CLASS, partners, Pattern.CLASS, Pattern.VALUE, GROUPS); + c.eq("sumByGroupVia crosses once", 1, Diagnostics.crossings() - x); + + GroupTotals self = v.sumByGroupVia(Pattern.CLASS, lines, Pattern.CLASS, + Pattern.VALUE, GROUPS); + c.eq("a self-join answers 16 groups", GROUPS, self.groups()); + } + + c.section("arguments are checked in Java, before any crossing"); + try (NativePattern data = NativePattern.open(256L, 1L)) { + long before = Diagnostics.crossings(); + c.throwsUp("groups = 0", IllegalArgumentException.class, + () -> data.view().sumByGroup(Pattern.CLASS, Pattern.VALUE, 0)); + c.throwsUp("groups < 0", IllegalArgumentException.class, + () -> data.view().sumByGroup(Pattern.CLASS, Pattern.VALUE, -3)); + c.throwsUp("null key", NullPointerException.class, + () -> data.view().sumByGroup(null, Pattern.VALUE, 4)); + c.eq("none of those crossed", 0, Diagnostics.crossings() - before); + + NativePattern gone = NativePattern.open(64L, 2L); + gone.close(); + c.throwsUp("a closed via resource is rejected", ClosedResourceException.class, + () -> data.view().sumByGroupVia(Pattern.CLASS, gone, Pattern.CLASS, + Pattern.VALUE, 4)); + data.close(); + c.throwsUp("a closed resource is rejected", ClosedResourceException.class, + () -> data.view().sumByGroup(Pattern.CLASS, Pattern.VALUE, 4)); + } catch (ClosedResourceException expected) { + // try-with-resources closes `data` a second time after the test closed it: that + // double close is itself an error by contract, and the one this block expects. + c.that("double close is an error, not a no-op", true); + } + } +} diff --git a/java/src/test/java/com/adaworldapi/lancegraph/OldAbiCompatTest.java b/java/src/test/java/com/adaworldapi/lancegraph/OldAbiCompatTest.java index 00a08af..5408e76 100644 --- a/java/src/test/java/com/adaworldapi/lancegraph/OldAbiCompatTest.java +++ b/java/src/test/java/com/adaworldapi/lancegraph/OldAbiCompatTest.java @@ -108,6 +108,19 @@ public static void run(Checks c) { } }); + // Minor 12 — the grouped sum that never builds a selection. Like the minor-11 reduce + // above it is built entirely on NativePattern, so it is gated unconditionally here rather + // than inside the minor-2 block: against a genuinely minor-1 library it must name minor + // 12, never die on a missing symbol. + gate(c, loaded, 12, "View.sumByGroup", () -> { + try (NativePattern p = NativePattern.open(64, 0x1234L)) { + GroupTotals t = p.view().sumByGroup(Pattern.CLASS, Pattern.VALUE, 16); + if (t.groups() != 16) { + throw new IllegalStateException("asked for 16 groups, got " + t.groups()); + } + } + }); + // Minor 4 — mask complement. Needs a minor-2 store to build masks on, so it is only // meaningful once the library has minor 2 as well. if (loaded >= 2) { diff --git a/native/lgj-abi/src/abi.rs b/native/lgj-abi/src/abi.rs index 12e6de5..2dc629d 100644 --- a/native/lgj-abi/src/abi.rs +++ b/native/lgj-abi/src/abi.rs @@ -82,7 +82,18 @@ pub const LGJ_ABI_MAJOR: u32 = 0; /// [`LgjOpDesc`] op-codes that cost no symbol at all /// ([`LGJ_OP_NE_U32`] … [`LGJ_OP_TERNARY_MATCH_U32`]). No new status and no /// manifest growth, so a minor-10 Java loads and sees none of it. -pub const LGJ_ABI_MINOR: u32 = 11; +/// +/// **Minor 12** (docs/abi.md §20): [`crate::exports::lgj_plan_group_sum_i32`] +/// — the fused plan run STRAIGHT INTO a grouped-sum terminal. Until now every +/// Java reduction paid for a selection first (`lgj_plan_eval` into a mask, +/// then `lgj_reduce_*` over it); this symbol lowers the plan and the +/// `GROUP BY key SUM(val)` as ONE `mask_risc::Program`, so the only +/// population-sized state is a tile-local scratch word and what crosses back +/// is one `i64` per group. A second table's `u32` lane may be read THROUGH a +/// key lane of this one (`via_res`/`via_lane`, the fk-keyed group sum) with +/// no partner-side mask and no second program. No new status and no manifest +/// growth, so a minor-11 Java loads and sees none of it. +pub const LGJ_ABI_MINOR: u32 = 12; /// `"LGJ_ABI\0"` read big-endian. /// diff --git a/native/lgj-abi/src/exports.rs b/native/lgj-abi/src/exports.rs index b2fd0a0..15558d5 100644 --- a/native/lgj-abi/src/exports.rs +++ b/native/lgj-abi/src/exports.rs @@ -1719,11 +1719,15 @@ fn validate_plan(pattern: &ResourceEntry, ops: &[LgjOpDesc]) -> Result<(), i32> /// **Every arm here is unreachable through the ABI**, and saying so is worth /// more than implying otherwise. `validate_plan` runs first and rejects an /// unknown opcode, a bad combine, an out-of-range lane and a kind mismatch -/// before the lowering is even built; the lowering names no input plane -/// (`Planes::masks` is `&[]`), no sum terminal, no blend and no `Pred::Range`. -/// What is left — the scratch-sizing family, `ScratchReadBeforeWrite`, -/// `GateAliasesDst`, `RangeOutOfBounds` — would be a bug in THIS file, not in -/// a caller's plan. +/// before the lowering is even built; the `Keep` lowering names no input +/// plane (`Planes::masks` is `&[]`), no sum terminal, no blend and no +/// `Pred::Range`. The grouped-sum lowering (minor 12) names a sum terminal +/// and ONE `Range` — always `0..n_rows`, so `RangeOutOfBounds` stays a bug in +/// this file; its `SumRowBound` needs more than `2^32` rows, which no pattern +/// this crate can open reaches; and its foreign-lane checks are done here, +/// before lowering, against the resolved `via` resource. What is left — the +/// scratch-sizing family, `ScratchReadBeforeWrite`, `GateAliasesDst`, +/// `RangeOutOfBounds` — would be a bug in THIS file, not in a caller's plan. /// /// So the map exists to turn such a bug into a status a caller can see /// instead of a panic, and its arms are deliberately NOT claimed to be @@ -2137,6 +2141,295 @@ pub unsafe extern "C" fn lgj_reduce_i32( }) } +// ─────────────────────────────────────────────────────────────────────────── +// Grouped reduction (ABI minor ≥ 12) — the fused plan run STRAIGHT INTO a +// grouped-sum terminal (docs/abi.md §20). +// ─────────────────────────────────────────────────────────────────────────── + +// Nine arguments because the ABI symbol has nine; a struct would be a +// second spelling of the same signature with nothing to check it against. +#[allow(clippy::too_many_arguments)] +fn plan_group_sum_impl( + res: u64, + ops: *const LgjOpDesc, + n_ops: u32, + group_lane: u32, + val_lane: u32, + via_res: u64, + via_lane: u32, + out_sums: *mut i64, + n_groups: u64, +) -> i32 { + // An empty plan is LEGAL here, unlike `lgj_plan_eval`: there is no + // destination mask a caller could fill on its own, so "every row" has to + // be a program (the whole-lane `Range`, `plan_lower::lower_group_sum`). + // `ops` may therefore be null exactly when `n_ops == 0`. + if out_sums.is_null() || (n_ops > 0 && ops.is_null()) { + return LGJ_ERR_NULL_ARGUMENT; + } + // A zero-length `out_sums` is no output buffer: no group can land + // anywhere, and `mask_risc` would refuse it as a missing sink. Reported + // as the argument defect it is rather than as the executor's refusal. + if n_groups == 0 { + return LGJ_ERR_NULL_ARGUMENT; + } + let groups = match usize::try_from(n_groups) { + Ok(g) if g <= isize::MAX as usize / std::mem::size_of::() => g, + _ => return LGJ_ERR_LENGTH_OVERFLOW, + }; + let pattern = match registry::resolve_kind(res, LGJ_RESOURCE_PATTERN) { + Ok(p) => p, + Err(e) => return e, + }; + // SAFETY: when `n_ops > 0`, `ops` is non-null (checked) and the caller + // states it points at `n_ops` contiguous `LgjOpDesc` — the same contract + // as `lgj_plan_eval`. When `n_ops == 0` no pointer is read at all. + let ops: &[LgjOpDesc] = if n_ops == 0 { + &[] + } else { + unsafe { std::slice::from_raw_parts(ops, n_ops as usize) } + }; + if !ops.is_empty() { + if let Err(e) = validate_plan(&pattern, ops) { + return e; + } + } + + // The group lane is a `U32` lane of THIS table (the key, or the foreign + // key when `via_res` is given); the value lane is an `I32` lane of this + // table. Both are checked before anything is lowered or written. + let group_lane16 = match u16::try_from(group_lane) { + Ok(l) => l, + Err(_) => return LGJ_ERR_INVALID_LANE, + }; + let val_lane16 = match u16::try_from(val_lane) { + Ok(l) => l, + Err(_) => return LGJ_ERR_INVALID_LANE, + }; + match lane_view(&pattern, group_lane) { + Ok(LaneView::U32(_)) => {} + Ok(_) => return LGJ_ERR_LANE_KIND_MISMATCH, + Err(e) => return e, + } + match lane_view(&pattern, val_lane) { + Ok(LaneView::I32(_)) => {} + Ok(_) => return LGJ_ERR_LANE_KIND_MISMATCH, + Err(e) => return e, + } + // `via_res == 0` is "no second table" — 0 is never a live handle (the + // registry's generation is never zero), so it cannot alias a resource. + let via = if via_res == 0 { + None + } else { + let entry = match registry::resolve_kind(via_res, LGJ_RESOURCE_PATTERN) { + Ok(p) => p, + Err(e) => return e, + }; + match lane_view(&entry, via_lane) { + Ok(LaneView::U32(_)) => {} + Ok(_) => return LGJ_ERR_LANE_KIND_MISMATCH, + Err(e) => return e, + } + Some(entry) + }; + + let n_rows = pattern.n_rows; + let rows = match usize::try_from(n_rows) { + Ok(r) => r, + Err(_) => return LGJ_ERR_LENGTH_OVERFLOW, + }; + // `Pred::Range` is `u32`-bounded, and so (via `MASKED_SUM_I32_MAX_ROWS`) + // is the sum terminal's carry: a pattern past `2^32 - 1` rows cannot be + // grouped by this symbol at all, and says so before lowering. + let rows32 = match u32::try_from(n_rows) { + Ok(r) => r, + Err(_) => return LGJ_ERR_LENGTH_OVERFLOW, + }; + let group = if via.is_some() { + // The one foreign lane this call supplies sits at index 0 of + // `Foreign::lanes`; the program's own lane ids are untouched. + plan_lower::GroupKey::Via { + fk: group_lane16, + key: 0, + } + } else { + plan_lower::GroupKey::Local(group_lane16) + }; + let program = match plan_lower::lower_group_sum(ops, rows32, group, val_lane16) { + Some(p) => p, + // Unreachable: `validate_plan` already rejected every unknown opcode. + None => return LGJ_ERR_UNKNOWN_OPCODE, + }; + + let fixture = match pattern.fixture() { + Some(f) => f, + None => return LGJ_ERR_WRONG_RESOURCE_KIND, + }; + // Same index order as `plan_eval_impl`, pinned by the same `const _` + // assertions: LANE_IDS, LANE_CLASSES, LANE_VALUES. + let lanes = [ + LaneRef::U64(fixture.ids()), + LaneRef::U32(fixture.classes()), + LaneRef::I32(fixture.values()), + ]; + let planes = Planes { + n_rows: rows, + masks: &[], + lanes: &lanes, + }; + // The foreign lane, borrowed from the `via` pattern for the length of + // this call. Its length is the OTHER table's row count and is never + // checked against `n_rows` — a key past it drops the row (zero fallback, + // `mask_risc`'s contract), it does not error. + let via_lanes: [LaneRef<'_>; 1]; + let foreign = match &via { + Some(entry) => { + let classes = match lane_view(entry, via_lane) { + Ok(LaneView::U32(v)) => v, + // Checked above; kept as a status rather than an unwrap. + _ => return LGJ_ERR_LANE_KIND_MISMATCH, + }; + via_lanes = [LaneRef::U32(classes)]; + Foreign { + planes: &[], + lanes: &via_lanes, + } + } + None => Foreign::NONE, + }; + + // TILED, exactly as `plan_eval_impl`: the scratch is `SLOTS × + // tile_words_for(rows)` words however many rows the resource holds, and + // the terminal adds each tile's contribution into the caller's `i64` + // buffer — the demanded sink, and the ONLY thing sized by the answer + // rather than by a tile. No `Keep`, no mask handle, no population. + let tile = tile_words_for(rows); + let need = match scratch_words_for(tile, plan_lower::SLOTS as usize) { + Some(n) => n, + None => return LGJ_ERR_LENGTH_OVERFLOW, + }; + // SAFETY: non-null (checked) and the caller states it points at + // `n_groups` contiguous, writable `i64`. `execute_into` validates the + // whole program before its first write to this slice, so a refused call + // leaves it byte-for-byte as it was; on `LGJ_OK` every element holds its + // group's sum (zero for a group no selected row names). + let out: &mut [i64] = unsafe { std::slice::from_raw_parts_mut(out_sums, groups) }; + PLAN_SCRATCH.with(|cell| { + let mut buf = cell.borrow_mut(); + if buf.len() < need { + buf.resize(need, 0); + } + let mut scratch = match Scratch::over(&mut buf[..need], tile, plan_lower::SLOTS as usize) { + Ok(s) => s, + Err(e) => return exec_error_to_status(e), + }; + match execute_into(&program, &planes, &foreign, &mut scratch, Out::I64(out)) { + Ok(Value::GroupSummed) => LGJ_OK, + // The lowering emits exactly one terminal and it is a grouped + // sum; any other value means this file built a program it did + // not intend to. + Ok(_) => LGJ_ERR_ALLOCATION_FAILED, + Err(e) => exec_error_to_status(e), + } + }) +} + +/// `GROUP BY group_lane SUM(val_lane)` over the rows the plan selects — **one +/// crossing, one program, and no selection anywhere** (ABI minor ≥ 12, +/// `docs/abi.md` §20). +/// +/// # Why this symbol exists +/// +/// Every reduction Java could ask for before minor 12 paid for a selection +/// first: `lgj_plan_eval` landed the plan's answer in a mask handle, and a +/// `lgj_reduce_*` then read that mask. The selection was never HELD by Java +/// — it lived in a native mask — but it was still a population-sized thing +/// that existed only to be consumed by the very next fold, which is the +/// intermediate materialisation the fold algebra exists to remove. A +/// `GROUP BY` made it worse by a factor of the group count: the `bricks` +/// consumer's `sumBy()` measured **32 crossings for 16 groups** (one plan +/// evaluation and one reduce per group). +/// +/// This symbol lowers the plan and the grouped sum as ONE +/// `mask_risc::Program` — the same prefix rewrite and survivor skip as +/// `lgj_plan_eval` (`plan_lower::lower_group_sum`), ending in +/// `Terminal::GroupSumI32` over the accumulator instead of `Keep`. The +/// executor runs it tile by tile; the accumulator is `TILE_WORDS` of scratch; +/// the terminal folds `val_lane[i]` into `out_sums[group_lane[i]]` for every +/// selected row of the tile and moves on. The only state sized by the answer +/// is `out_sums` itself, and the only thing that crosses back is one `i64` +/// per group. 16 groups cost one crossing; so do 10⁹ rows. +/// +/// # Semantics +/// +/// - `ops`/`n_ops` are the fused plan, exactly as `lgj_plan_eval` reads them +/// (accumulator starts as all rows; each op combines per its `combine`). +/// **`n_ops == 0` is legal here** and means every row — there is no +/// destination mask for a caller to fill itself, so the empty plan lowers +/// to a whole-lane range rather than being refused (`EMPTY_PLAN` is +/// `lgj_plan_eval`'s answer, not this symbol's). +/// - `group_lane` is a `U32` lane of `res`; `val_lane` an `I32` lane of `res`. +/// - `out_sums[g]` receives `Σ val_lane[i]` over selected rows `i` with +/// `group_lane[i] == g`, for `g < n_groups`; a selected row whose key is +/// `>= n_groups` names no group and is dropped, never an error. Every +/// element is written on `LGJ_OK` (zero for a group no row names); none is +/// written on any failure. +/// - **`via_res != 0` is the fk-keyed form**: `group_lane` is then read as a +/// foreign KEY into `via_res` (another pattern resource), and the group of +/// row `i` is `via_lane[group_lane[i]]` — `via_lane` a `U32` lane of +/// `via_res`. `SUM(line.amount) GROUP BY partner.country` in one program, +/// with no partner-side mask and no remapped key lane: the indirection is +/// fused inside `Terminal::GroupSumViaI32`. A key that names no row of +/// `via_res` drops the row (zero fallback at both hops). `via_res` may +/// equal `res`. `via_lane` is ignored when `via_res == 0`. +/// +/// # Statuses +/// +/// `NULL_ARGUMENT` for a null `out_sums`, a null `ops` with `n_ops > 0`, or +/// `n_groups == 0`; `INVALID_HANDLE` / `WRONG_RESOURCE_KIND` for `res` or a +/// non-zero `via_res` that is not a live pattern; every plan defect exactly +/// as `lgj_plan_eval` reports it; `INVALID_LANE` / `LANE_KIND_MISMATCH` for a +/// `group_lane` that is not `U32`, a `val_lane` that is not `I32`, or a +/// `via_lane` that is not `U32` on `via_res`; `LENGTH_OVERFLOW` for a pattern +/// past `2^32 - 1` rows. All of it before any write. +/// +/// # Bulk (§6) +/// +/// `O(rows)` in ONE pass over the predicate lanes plus, for the selected +/// rows only, one read of the key and value lanes. No per-row crossing, no +/// per-group crossing. +/// +/// # Safety +/// +/// A null pointer is *handled*, not UB: `NULL_ARGUMENT`. Beyond that, `ops` +/// must point to `n_ops` contiguous, initialized `LgjOpDesc` (24 bytes, align +/// 8) when `n_ops > 0`, and `out_sums` to `n_groups` writable, aligned `i64`; +/// both stay valid for the call and neither is touched after the null checks +/// except as stated above. +/// +/// `unsafe` here is a note to Rust callers linking the `rlib`. The JVM, +/// which is the real caller, has no such concept — it upholds the same +/// contract by construction, because every pointer it passes comes from a +/// `MemorySegment` whose size and alignment it derived from the manifest. +#[no_mangle] +pub unsafe extern "C" fn lgj_plan_group_sum_i32( + res: u64, + ops: *const LgjOpDesc, + n_ops: u32, + group_lane: u32, + val_lane: u32, + via_res: u64, + via_lane: u32, + out_sums: *mut i64, + n_groups: u64, +) -> i32 { + guard(|| { + plan_group_sum_impl( + res, ops, n_ops, group_lane, val_lane, via_res, via_lane, out_sums, n_groups, + ) + }) +} + // ─────────────────────────────────────────────────────────────────────────── // Graph traversal (ABI minor ≥ 4) — the first symbol gated by the // lance-graph-contract ClassView/FieldMask LAW (docs/abi.md §13). @@ -2393,6 +2686,10 @@ mod tests { use super::*; use crate::fixture::{Fixture, LANE_CLASSES, LANE_IDS, LANE_VALUES}; + /// Minor 12: the grouped sum that never builds a selection — parity + /// against the two-crossing path it replaces, the scalar oracle, the + /// fk-keyed form against a hand-walked join, and every refusal. + mod group_sum; /// The lowering differential: this crate's `plan_lower` (a flat op list) /// against `lance-graph-quack`'s `lower` (a Boolean tree), both producing /// a `mask_risc::Program`. Two implementations of one law; until this @@ -3991,6 +4288,29 @@ mod tests { } } + mod call12 { + use super::*; + + #[allow(clippy::too_many_arguments)] + pub fn plan_group_sum_i32( + res: u64, + ops: *const LgjOpDesc, + n_ops: u32, + group_lane: u32, + val_lane: u32, + via_res: u64, + via_lane: u32, + out_sums: *mut i64, + n_groups: u64, + ) -> i32 { + unsafe { + lgj_plan_group_sum_i32( + res, ops, n_ops, group_lane, val_lane, via_res, via_lane, out_sums, n_groups, + ) + } + } + } + /// Pack a `LGJ_OP_TERNARY_MATCH_U32` operand the way `abi.rs` documents: /// pattern in the low 32 bits, care in the high 32. ONE spelling in the /// tests, so a hand-packing typo cannot make a passing arm mean something diff --git a/native/lgj-abi/src/exports/tests/group_sum.rs b/native/lgj-abi/src/exports/tests/group_sum.rs new file mode 100644 index 0000000..c322b37 --- /dev/null +++ b/native/lgj-abi/src/exports/tests/group_sum.rs @@ -0,0 +1,536 @@ +//! Minor 12 — `lgj_plan_group_sum_i32`, the grouped sum that never builds a +//! selection. +//! +//! Three oracles, none of them the code under test: +//! +//! 1. **The two-crossing path it replaces.** `lgj_plan_eval` into a mask, +//! then `lgj_reduce_sum_i32` over it, once per group with `class == g` +//! appended as an AND. That is exactly what `bricks`' `sumBy()` measured +//! at 32 crossings for 16 groups; the new symbol must answer identically +//! in one. Different kernels (`masked_sum_i32` vs `masked_group_sum_i32`), +//! different lowering (`Keep` vs a sum terminal), same numbers. +//! 2. **`mask_risc`'s row-at-a-time reference executor**, run on the SAME +//! lowered program — whole-evaluator-against-whole-evaluator, the +//! discipline `lgj_plan_eval_scalar` gives the `Keep` path. +//! 3. **A hand walk over the fixture arrays** with a hand-evaluated +//! predicate, for the fk-keyed form: no kernel, no lowering, no mask — +//! `for i: if value[i] > 100 { out[b.class[a.class[i]]] += value[i] }`. +//! +//! And every refusal, each pinned to leave `out_sums` byte-for-byte alone. + +use super::*; +use lance_graph_mask_risc::reference_execute_into; + +fn open(n: u64, seed: u64) -> u64 { + let mut h = 0u64; + assert_eq!(call::pattern_open(n, seed, &mut h), LGJ_OK); + h +} + +/// Every element pre-filled with a sentinel no real sum produces, so a +/// success can be checked for having written EVERY group (zero included) and +/// a refusal for having written none. +const SENTINEL: i64 = i64::MIN + 0x5EED; + +/// Run the symbol; `ops` empty passes a NULL pointer, the way a caller with +/// no predicate would. +fn group_sum( + res: u64, + ops: &[LgjOpDesc], + group_lane: u32, + val_lane: u32, + via: (u64, u32), + groups: usize, +) -> (i32, Vec) { + let mut out = vec![SENTINEL; groups]; + let ptr = if ops.is_empty() { + std::ptr::null() + } else { + ops.as_ptr() + }; + let st = call12::plan_group_sum_i32( + res, + ptr, + ops.len() as u32, + group_lane, + val_lane, + via.0, + via.1, + out.as_mut_ptr(), + groups as u64, + ); + (st, out) +} + +/// Oracle 1: the two-crossing path, per group. +fn two_crossing_oracle(res: u64, ops: &[LgjOpDesc], groups: usize) -> Vec { + (0..groups as u32) + .map(|g| { + let mut plan = ops.to_vec(); + plan.push(op( + LGJ_OP_EQ_U32, + LANE_CLASSES, + i64::from(g), + LGJ_COMBINE_AND, + )); + let mut mask = 0u64; + assert_eq!( + call::mask_create(res, LGJ_MASK_INIT_EMPTY, &mut mask), + LGJ_OK + ); + let mut count = 0u64; + assert_eq!( + call::plan_eval(res, plan.as_ptr(), plan.len() as u32, mask, &mut count), + LGJ_OK + ); + let mut sum = 0i64; + assert_eq!( + call::reduce_sum_i32(res, LANE_VALUES, mask, &mut sum), + LGJ_OK + ); + assert_eq!(lgj_close(mask), LGJ_OK); + sum + }) + .collect() +} + +const TWO_OP: [(u32, u32, i64); 2] = [ + (LGJ_OP_EQ_U32, LANE_CLASSES, 7), + (LGJ_OP_GT_I32, LANE_VALUES, 100), +]; +const THREE_OP: [(u32, u32, i64); 3] = [ + (LGJ_OP_GT_I32, LANE_VALUES, 100), + (LGJ_OP_EQ_U32, LANE_CLASSES, 3), + (LGJ_OP_LT_I32, LANE_VALUES, 300), +]; + +/// `{AND, OR}^n` over both op sets — 4 + 8 = 12 vectors, plus the empty plan. +fn for_each_plan(mut visit: impl FnMut(String, Vec)) { + visit("n=0 []".into(), Vec::new()); + for (base, arity) in [(&TWO_OP[..], 2u32), (&THREE_OP[..], 3u32)] { + for bits in 0u32..(1 << arity) { + let ops: Vec = base + .iter() + .enumerate() + .map(|(i, &(opcode, lane, operand))| { + let combine = if bits & (1 << i) == 0 { + LGJ_COMBINE_AND + } else { + LGJ_COMBINE_OR + }; + op(opcode, lane, operand, combine) + }) + .collect(); + let names: Vec<&str> = (0..arity) + .map(|i| if bits & (1 << i) == 0 { "AND" } else { "OR" }) + .collect(); + visit(format!("n={arity} [{}]", names.join(", ")), ops); + } + } +} + +/// CAN-FIRE + parity: one crossing answers what sixteen pairs of crossings +/// answered, on a single-tile and a multi-tile population. +#[test] +fn grouped_sum_matches_the_two_crossing_path_over_the_combine_sweep() { + for (n, seed) in [(1000u64, 33u64), (65_536, 7)] { + let res = open(n, seed); + let mut distinct_answers = std::collections::HashSet::new(); + for_each_plan(|label, ops| { + let (st, got) = group_sum(res, &ops, LANE_CLASSES, LANE_VALUES, (0, 0), 16); + assert_eq!(st, LGJ_OK, "n={n} {label}"); + let want = two_crossing_oracle(res, &ops, 16); + assert_eq!( + got, want, + "n={n} {label}: one crossing vs the two-crossing path" + ); + assert!( + got.iter().all(|&v| v != SENTINEL), + "{label}: every group is written on success, zero included" + ); + distinct_answers.insert(got); + }); + // Anti-vacuity: the sweep is not one answer repeated. AND-heavy and + // OR-heavy vectors select different populations and must sum + // differently. + assert!( + distinct_answers.len() >= 4, + "n={n}: only {} distinct grouped answers across 13 plans", + distinct_answers.len() + ); + assert_eq!(lgj_close(res), LGJ_OK); + } +} + +/// The empty plan is legal here (unlike `lgj_plan_eval`) and means every +/// row; so does a plan whose every combine is OR. Both equal the per-class +/// sum over the whole fixture, hand-walked. +#[test] +fn an_empty_plan_and_an_all_or_plan_both_group_every_row() { + let (n, seed) = (4096u64, 0xC0FFEE_u64); + let f = Fixture::generate(n, seed).unwrap(); + let mut want = vec![0i64; 16]; + for (c, v) in f.classes().iter().zip(f.values()) { + want[*c as usize] += i64::from(*v); + } + let res = open(n, seed); + let (st, empty) = group_sum(res, &[], LANE_CLASSES, LANE_VALUES, (0, 0), 16); + assert_eq!(st, LGJ_OK); + assert_eq!(empty, want, "n_ops == 0 groups every row"); + let all_or = [ + op(LGJ_OP_EQ_U32, LANE_CLASSES, 7, LGJ_COMBINE_OR), + op(LGJ_OP_GT_I32, LANE_VALUES, 100, LGJ_COMBINE_OR), + ]; + let (st, ored) = group_sum(res, &all_or, LANE_CLASSES, LANE_VALUES, (0, 0), 16); + assert_eq!(st, LGJ_OK); + assert_eq!(ored, want, "an all-OR plan is dead and groups every row"); + assert!( + want.iter().filter(|&&v| v != 0).count() == 16, + "anti-vacuity: 16 live groups" + ); + assert_eq!(lgj_close(res), LGJ_OK); +} + +/// `n_groups` bounds the SINK, not the keys: a selected row whose key is past +/// it is dropped, silently, and the first `n_groups` totals are unchanged. +#[test] +fn keys_past_n_groups_are_dropped_not_errors() { + let res = open(2048, 5); + let ops = [op(LGJ_OP_GT_I32, LANE_VALUES, 0, LGJ_COMBINE_AND)]; + let (st16, full) = group_sum(res, &ops, LANE_CLASSES, LANE_VALUES, (0, 0), 16); + let (st8, eight) = group_sum(res, &ops, LANE_CLASSES, LANE_VALUES, (0, 0), 8); + let (st20, twenty) = group_sum(res, &ops, LANE_CLASSES, LANE_VALUES, (0, 0), 20); + assert_eq!((st16, st8, st20), (LGJ_OK, LGJ_OK, LGJ_OK)); + assert_eq!( + eight, + full[..8], + "a narrower sink keeps the groups it can hold" + ); + assert_eq!( + twenty[..16], + full[..], + "a wider sink changes nothing below 16" + ); + assert!( + twenty[16..].iter().all(|&v| v == 0), + "groups no key names are zero, not sentinel" + ); + assert!( + full[8..].iter().any(|&v| v != 0), + "anti-vacuity: rows WERE dropped at n_groups=8" + ); + assert_eq!(lgj_close(res), LGJ_OK); +} + +/// Oracle 2: the row-at-a-time reference executor on the same lowered +/// program, multi-tile, both key shapes. +#[test] +fn the_tiled_executor_and_the_scalar_reference_agree_on_the_lowered_program() { + let (n, seed) = (5000u64, 0xBEEF_u64); + let a = Fixture::generate(n, seed).unwrap(); + let b = Fixture::generate(64, 0xB0B).unwrap(); + let lanes = [ + LaneRef::U64(a.ids()), + LaneRef::U32(a.classes()), + LaneRef::I32(a.values()), + ]; + let planes = Planes { + n_rows: n as usize, + masks: &[], + lanes: &lanes, + }; + let via_lanes = [LaneRef::U32(b.classes())]; + let foreign = Foreign { + planes: &[], + lanes: &via_lanes, + }; + let keys = [ + plan_lower::GroupKey::Local(LANE_CLASSES as u16), + plan_lower::GroupKey::Via { + fk: LANE_CLASSES as u16, + key: 0, + }, + ]; + let mut checked = 0usize; + for_each_plan(|label, ops| { + for key in keys { + let program = + plan_lower::lower_group_sum(&ops, n as u32, key, LANE_VALUES as u16).unwrap(); + let tile = tile_words_for(n as usize); + let need = scratch_words_for(tile, plan_lower::SLOTS as usize).unwrap(); + let mut buf = vec![0u64; need]; + let mut scratch = Scratch::over(&mut buf, tile, plan_lower::SLOTS as usize).unwrap(); + assert!( + tile < n.div_ceil(64) as usize, + "the population spans several tiles" + ); + let mut simd = vec![0i64; 16]; + let mut scalar = vec![0i64; 16]; + let v = execute_into( + &program, + &planes, + &foreign, + &mut scratch, + Out::I64(&mut simd), + ) + .unwrap(); + assert!(matches!(v, Value::GroupSummed), "{label}"); + let r = + reference_execute_into(&program, &planes, &foreign, Out::I64(&mut scalar)).unwrap(); + assert!(matches!(r, Value::GroupSummed), "{label}"); + assert_eq!( + simd, scalar, + "{label} {key:?}: tiled executor vs scalar reference" + ); + checked += 1; + } + }); + assert_eq!(checked, 26); +} + +/// Oracle 3: the fk-keyed form against a hand walk — `group of row i` is +/// `b.class[a.class[i]]`, with the fixture's own arrays, a hand-evaluated +/// predicate, and no kernel anywhere. Also pins the zero fallback: a `via` +/// table shorter than the key range drops exactly the rows whose key names +/// no row of it. +#[test] +fn via_reads_the_second_table_through_the_key_lane() { + let (n, seed_a) = (4096u64, 0xA11CE_u64); + let a = Fixture::generate(n, seed_a).unwrap(); + let res = open(n, seed_a); + let ops = [op(LGJ_OP_GT_I32, LANE_VALUES, 100, LGJ_COMBINE_AND)]; + for (b_rows, seed_b) in [(64u64, 0xB0B_u64), (8, 0xB0B)] { + let b = Fixture::generate(b_rows, seed_b).unwrap(); + let via = open(b_rows, seed_b); + let mut want = vec![0i64; 16]; + let mut dropped = 0usize; + for i in 0..n as usize { + if a.values()[i] <= 100 { + continue; + } + let k = a.classes()[i] as usize; + if k >= b.classes().len() { + dropped += 1; + continue; + } + want[b.classes()[k] as usize] += i64::from(a.values()[i]); + } + let (st, got) = group_sum( + res, + &ops, + LANE_CLASSES, + LANE_VALUES, + (via, LANE_CLASSES), + 16, + ); + assert_eq!(st, LGJ_OK, "via rows={b_rows}"); + assert_eq!( + got, want, + "via rows={b_rows}: fk-keyed grouped sum vs the hand walk" + ); + // The join genuinely regrouped: the local answer must differ. + let (_, local) = group_sum(res, &ops, LANE_CLASSES, LANE_VALUES, (0, 0), 16); + assert_ne!( + got, local, + "via rows={b_rows}: the fk-keyed form is not the local form" + ); + if b_rows == 8 { + assert!( + dropped > 0, + "anti-vacuity: keys past the via table were dropped" + ); + } else { + assert_eq!( + dropped, 0, + "16 classes all name a row of a 64-row via table" + ); + // With no drops, the totals are a regrouping of the local totals. + assert_eq!(got.iter().sum::(), local.iter().sum::()); + } + assert_eq!(lgj_close(via), LGJ_OK); + } + // `via_res == res` is a legal self-join. + let (st, selfjoin) = group_sum( + res, + &ops, + LANE_CLASSES, + LANE_VALUES, + (res, LANE_CLASSES), + 16, + ); + assert_eq!(st, LGJ_OK); + let mut want = vec![0i64; 16]; + for i in 0..n as usize { + if a.values()[i] > 100 { + want[a.classes()[a.classes()[i] as usize] as usize] += i64::from(a.values()[i]); + } + } + assert_eq!(selfjoin, want); + assert_eq!(lgj_close(res), LGJ_OK); +} + +/// Every refusal is a status, reached before any write: the sentinel survives +/// in every element. +#[test] +fn every_refusal_is_a_status_and_leaves_out_sums_untouched() { + let res = open(512, 1); + let ok_ops = [op(LGJ_OP_GT_I32, LANE_VALUES, 0, LGJ_COMBINE_AND)]; + type Refusal<'a> = (&'a str, i32, (i32, Vec)); + let refusals: Vec> = vec![ + ( + "n_groups == 0", + LGJ_ERR_NULL_ARGUMENT, + group_sum(res, &ok_ops, LANE_CLASSES, LANE_VALUES, (0, 0), 0), + ), + ( + "group lane is I32", + LGJ_ERR_LANE_KIND_MISMATCH, + group_sum(res, &ok_ops, LANE_VALUES, LANE_VALUES, (0, 0), 16), + ), + ( + "group lane is U64", + LGJ_ERR_LANE_KIND_MISMATCH, + group_sum(res, &ok_ops, LANE_IDS, LANE_VALUES, (0, 0), 16), + ), + ( + "value lane is U32", + LGJ_ERR_LANE_KIND_MISMATCH, + group_sum(res, &ok_ops, LANE_CLASSES, LANE_CLASSES, (0, 0), 16), + ), + ( + "group lane out of range", + LGJ_ERR_INVALID_LANE, + group_sum(res, &ok_ops, 9, LANE_VALUES, (0, 0), 16), + ), + ( + "value lane out of range", + LGJ_ERR_INVALID_LANE, + group_sum(res, &ok_ops, LANE_CLASSES, 9, (0, 0), 16), + ), + ( + "unknown opcode in the plan", + LGJ_ERR_UNKNOWN_OPCODE, + group_sum( + res, + &[op(77, LANE_VALUES, 0, LGJ_COMBINE_AND)], + LANE_CLASSES, + LANE_VALUES, + (0, 0), + 16, + ), + ), + ( + "plan lane kind mismatch", + LGJ_ERR_LANE_KIND_MISMATCH, + group_sum( + res, + &[op(LGJ_OP_GT_I32, LANE_CLASSES, 0, LGJ_COMBINE_AND)], + LANE_CLASSES, + LANE_VALUES, + (0, 0), + 16, + ), + ), + ( + "via handle is fabricated", + LGJ_ERR_INVALID_HANDLE, + group_sum( + res, + &ok_ops, + LANE_CLASSES, + LANE_VALUES, + (0xDEAD_BEEF, LANE_CLASSES), + 16, + ), + ), + ( + "via lane is I32", + LGJ_ERR_LANE_KIND_MISMATCH, + group_sum( + res, + &ok_ops, + LANE_CLASSES, + LANE_VALUES, + (res, LANE_VALUES), + 16, + ), + ), + ( + "via lane out of range", + LGJ_ERR_INVALID_LANE, + group_sum(res, &ok_ops, LANE_CLASSES, LANE_VALUES, (res, 9), 16), + ), + ( + "res is fabricated", + LGJ_ERR_INVALID_HANDLE, + group_sum(0xDEAD_BEEF, &ok_ops, LANE_CLASSES, LANE_VALUES, (0, 0), 16), + ), + ]; + for (what, want, (st, out)) in refusals { + assert_eq!(st, want, "{what}"); + assert!( + out.iter().all(|&v| v == SENTINEL), + "{what}: out_sums was written on a refusal" + ); + } + + // A mask handle where a pattern is expected. + let mut mask = 0u64; + assert_eq!(call::mask_create(res, LGJ_MASK_INIT_ALL, &mut mask), LGJ_OK); + let (st, out) = group_sum(mask, &ok_ops, LANE_CLASSES, LANE_VALUES, (0, 0), 16); + assert_eq!(st, LGJ_ERR_WRONG_RESOURCE_KIND); + assert!(out.iter().all(|&v| v == SENTINEL)); + let (st, out) = group_sum( + res, + &ok_ops, + LANE_CLASSES, + LANE_VALUES, + (mask, LANE_CLASSES), + 16, + ); + assert_eq!( + st, LGJ_ERR_WRONG_RESOURCE_KIND, + "a mask is not a via table either" + ); + assert!(out.iter().all(|&v| v == SENTINEL)); + assert_eq!(lgj_close(mask), LGJ_OK); + + // Null pointers are statuses, not UB. + let mut sink = [SENTINEL; 4]; + assert_eq!( + call12::plan_group_sum_i32( + res, + ok_ops.as_ptr(), + 1, + LANE_CLASSES, + LANE_VALUES, + 0, + 0, + std::ptr::null_mut(), + 4 + ), + LGJ_ERR_NULL_ARGUMENT + ); + assert_eq!( + call12::plan_group_sum_i32( + res, + std::ptr::null(), + 1, + LANE_CLASSES, + LANE_VALUES, + 0, + 0, + sink.as_mut_ptr(), + 4 + ), + LGJ_ERR_NULL_ARGUMENT, + "a null plan with n_ops > 0" + ); + assert!(sink.iter().all(|&v| v == SENTINEL)); + + // And after close, the generation-checked handle refuses. + assert_eq!(lgj_close(res), LGJ_OK); + let (st, out) = group_sum(res, &ok_ops, LANE_CLASSES, LANE_VALUES, (0, 0), 16); + assert_eq!(st, LGJ_ERR_INVALID_HANDLE); + assert!(out.iter().all(|&v| v == SENTINEL)); +} diff --git a/native/lgj-abi/src/exports/tests/lowering_convergence.rs b/native/lgj-abi/src/exports/tests/lowering_convergence.rs index b9bda78..1c8ffaf 100644 --- a/native/lgj-abi/src/exports/tests/lowering_convergence.rs +++ b/native/lgj-abi/src/exports/tests/lowering_convergence.rs @@ -68,7 +68,7 @@ use super::*; use lance_graph_mask_risc::Program; -use lance_graph_quack::{Agg, Cmp, Col, Filter, Query}; +use lance_graph_quack::{Agg, Cmp, Col, Filter, Mask as QuackMask, Query}; /// `n = 1000, seed = 33` — the identical fixture /// `pr4_matrix.rs::combine_mode_products_agree_with_the_oracle` sweeps, so @@ -654,3 +654,69 @@ fn every_opcode_maps_to_the_same_predicate_in_both_lowerings() { "TCAM swapped halves at position 1 of 3", ); } + +/// The minor-12 arm of the same differential: `plan_lower::lower_group_sum` +/// against `lance_graph_quack::lower` with `Agg::GroupSumI32`, both run +/// through `execute_into` into an `Out::I64` — one law, two lowerings, now +/// for the grouped sum as well as the count. The all-rows plan is included: +/// quack spells it `Filter::Plane(alpha)` over a resident all-ones plane, +/// `plan_lower` spells it the whole-lane `Range`; they must total the same. +#[test] +fn grouped_sums_agree_with_quack_over_the_combine_sweep() { + let fixture = Fixture::generate(SWEEP_N, SWEEP_SEED).expect("fixture"); + let lanes = [ + LaneRef::U64(fixture.ids()), + LaneRef::U32(fixture.classes()), + LaneRef::I32(fixture.values()), + ]; + let words = (SWEEP_N as usize).div_ceil(64); + let mut alpha = vec![u64::MAX; words]; + let tail = (SWEEP_N as usize) % 64; + if tail != 0 { + alpha[words - 1] = (1u64 << tail) - 1; + } + let alpha_ref: &[u64] = α + let planes = Planes { + n_rows: SWEEP_N as usize, + masks: &[alpha_ref], + lanes: &lanes, + }; + let run_groups = |p: &Program| -> Vec { + let mut scratch = Scratch::for_program(p, planes.n_rows).expect("addressable"); + let mut out = vec![0i64; 16]; + let v = execute_into(p, &planes, &Foreign::NONE, &mut scratch, Out::I64(&mut out)) + .expect("runs"); + assert!(matches!(v, Value::GroupSummed), "not a grouped sum: {v:?}"); + out + }; + let mut distinct = std::collections::HashSet::new(); + let mut check = |label: String, ops: Vec| { + let ours = plan_lower::lower_group_sum( + &ops, + SWEEP_N as u32, + plan_lower::GroupKey::Local(LANE_CLASSES as u16), + LANE_VALUES as u16, + ) + .expect("lowers"); + let filter = fold_to_tree(&ops).unwrap_or(Filter::Plane(QuackMask(0))); + let q = Query { + filter, + agg: Agg::GroupSumI32 { + key: Col(LANE_CLASSES as u16), + val: Col(LANE_VALUES as u16), + }, + }; + let theirs = lance_graph_quack::lower(&q).expect("quack lowers"); + let a = run_groups(&ours); + let b = run_groups(&theirs); + assert_eq!(a, b, "{label}: plan_lower's grouped sum vs quack's"); + distinct.insert(a); + }; + check("n=0 []".into(), Vec::new()); + for_each_combine_vector(&mut check); + assert!( + distinct.len() >= 4, + "anti-vacuity: {} distinct answers", + distinct.len() + ); +} diff --git a/native/lgj-abi/src/plan_lower.rs b/native/lgj-abi/src/plan_lower.rs index d1e3743..7675001 100644 --- a/native/lgj-abi/src/plan_lower.rs +++ b/native/lgj-abi/src/plan_lower.rs @@ -42,6 +42,19 @@ use crate::abi::{ }; use lance_graph_mask_risc::{MaskOp, Operand, Pred, Program, Terminal}; +/// Which lane a grouped sum groups by — this table's own `u32` lane, or a +/// `u32` lane of ANOTHER table read through a `u32` key lane of this one. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum GroupKey { + /// `GROUP BY key` where `key` is a `U32` lane of the executing table. + Local(u16), + /// `GROUP BY other.key` read as `foreign.lanes[key][fk[i]]` — `fk` is a + /// `U32` lane of the executing table, `key` indexes the `Foreign::lanes` + /// the caller supplies. The indirection is fused inside the terminal; no + /// remapped key lane and no partner-side mask ever exist. + Via { fk: u16, key: u16 }, +} + /// Scratch slots the lowering ever names: slot 0 is the accumulator (the old /// `acc`, now living in the caller's arena rather than in a per-call `vec!`), /// slot 1 the per-op predicate destination (the old `scratch`). @@ -108,6 +121,53 @@ pub(crate) fn lower_plan(ops: &[LgjOpDesc]) -> Option { ))) } +/// Lower a validated plan STRAIGHT INTO a grouped sum — the same prefix +/// rewrite and survivor skip as [`lower_plan`], ending in +/// [`Terminal::GroupSumI32`] / [`Terminal::GroupSumViaI32`] over the +/// accumulator instead of [`Terminal::Keep`]. +/// +/// This is the shape that lets a `GROUP BY … SUM` cross the membrane once +/// and return only group totals: no `Keep` lands a population anywhere, the +/// accumulator lives in tile-local scratch, and the terminal folds `val` into +/// the caller's `i64` buffer tile by tile. +/// +/// # The all-rows arm +/// +/// [`lower_plan`] answers `AllRows` without a program because its caller can +/// fill the destination mask itself. A grouped sum has no destination mask +/// to fill, so the all-rows case — an empty plan, or one whose every combine +/// is OR — needs an accumulator that IS every row. That is +/// [`Pred::Range`]`{ lo: 0, hi: n_rows }`: a three-pass fill over the tile's +/// words (`mask_set_range`), reading no lane. It is the ONLY `Range` this +/// crate ever emits, and with `hi == n_rows` exactly it can never leave the +/// lane, which is what keeps `exec_error_to_status`'s `RangeOutOfBounds` arm +/// an internal-bug mapping — pinned by +/// `range_falsifier::the_group_sum_range_is_the_whole_lane_and_nothing_else`. +/// +/// `n_rows` is `u32` because [`Pred::Range`] is; a caller with more rows than +/// that cannot take the all-rows arm and must say so before lowering. +pub(crate) fn lower_group_sum( + ops: &[LgjOpDesc], + n_rows: u32, + group: GroupKey, + val: u16, +) -> Option { + let mask = Operand::Scratch(ACC_SLOT); + let terminal = match group { + GroupKey::Local(key) => Terminal::GroupSumI32 { mask, key, val }, + GroupKey::Via { fk, key } => Terminal::GroupSumViaI32 { mask, fk, key, val }, + }; + let program_ops = match lower_plan(ops)? { + Lowered::AllRows => vec![MaskOp::Pred { + pred: Pred::Range { lo: 0, hi: n_rows }, + under: None, + dst: ACC_SLOT, + }], + Lowered::Program(p) => p.ops, + }; + Some(Program::new(program_ops, terminal)) +} + /// One `LgjOpDesc` opcode + operand → one `Pred`. /// /// Every operand conversion is the one `kernels::eval_predicate` performs, and @@ -182,6 +242,12 @@ mod range_falsifier { /// whole purpose: it forces the author to decide, deliberately and /// visibly, what a caller should see when a range leaves the lane — rather /// than inheriting `LGJ_ERR_ALLOCATION_FAILED`, which would then be a lie. + /// + /// Minor 12 did gain ONE `Range` — [`lower_group_sum`]'s all-rows arm — and + /// decided it deliberately: that range is always `0..n_rows`, so it cannot + /// leave the lane and the internal-bug mapping stays true. It is pinned by + /// the sibling test below; this sweep still holds for every opcode through + /// `lower_plan`, which is the path a caller's operand reaches. #[test] fn no_opcode_lowers_to_pred_range() { let mut lowered_count = 0usize; @@ -220,4 +286,68 @@ mod range_falsifier { "the sweep lowered only {lowered_count} programs — it is not exercising the lowering" ); } + + /// The one `Range` this crate emits is the whole lane, in the all-rows arm + /// only. An empty plan and an all-OR plan both take it; a plan with an AND + /// emits none. Any other `Range` would make `exec_error_to_status`'s + /// `RangeOutOfBounds` arm reachable through the ABI, and this is the test + /// that would say so. + #[test] + fn the_group_sum_range_is_the_whole_lane_and_nothing_else() { + let or_op = LgjOpDesc { + op: LGJ_OP_EQ_U32, + lane_id: 1, + operand: 7, + combine: LGJ_COMBINE_OR, + _reserved: 0, + }; + let and_op = LgjOpDesc { + combine: LGJ_COMBINE_AND, + ..or_op + }; + let ranges = |ops: &[LgjOpDesc]| -> Vec<(u32, u32)> { + let p = lower_group_sum(ops, 1000, GroupKey::Local(1), 2).expect("lowers"); + p.ops + .iter() + .filter_map(|op| match op { + MaskOp::Pred { + pred: Pred::Range { lo, hi }, + .. + } => Some((*lo, *hi)), + _ => None, + }) + .collect() + }; + assert_eq!(ranges(&[]), vec![(0, 1000)], "an empty plan is every row"); + assert_eq!( + ranges(&[or_op, or_op]), + vec![(0, 1000)], + "an all-OR plan is every row" + ); + assert!( + ranges(&[and_op]).is_empty(), + "a plan with an AND needs no range" + ); + assert!(ranges(&[or_op, and_op, or_op]).is_empty()); + // And the terminal is the grouped sum over the accumulator, both shapes. + let local = lower_group_sum(&[and_op], 1000, GroupKey::Local(1), 2).unwrap(); + assert_eq!( + local.terminal, + Terminal::GroupSumI32 { + mask: Operand::Scratch(ACC_SLOT), + key: 1, + val: 2 + } + ); + let via = lower_group_sum(&[], 1000, GroupKey::Via { fk: 1, key: 0 }, 2).unwrap(); + assert_eq!( + via.terminal, + Terminal::GroupSumViaI32 { + mask: Operand::Scratch(ACC_SLOT), + fk: 1, + key: 0, + val: 2 + } + ); + } } From 303c2cf5bb6781aa0370d5a2208665ef362c6d3f Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 22 Sep 2026 04:58:24 +0000 Subject: [PATCH 2/2] =?UTF-8?q?docs:=20abi.md=20=C2=A720=20(minor=2012),?= =?UTF-8?q?=20the=20eighth=20named=20materialisation=20site,=20board=20row?= =?UTF-8?q?s?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `docs/abi.md`: minor 12 in the version history, the symbol count 29 → 30, `lgj_plan_group_sum_i32` in §7's surface table, and §20 — why the grouped sum exists (every reduction consumed a mask; a GROUP BY consumed one per group), what it does (the plan lowered as `plan_eval` lowers it, ending in a sum terminal over a tile-local accumulator), the contract, why one symbol and no scalar twin, the Java spelling, and the six-arm disable table with what each arm observed. Root `CLAUDE.md`: `Engine.groupSumI32`'s `toArray` joins the exhaustive materialisation-site list as the eighth site, sized by the key domain the caller asked for, never by rows. Board: LATEST_STATE entry (measured crossings, gates, what is still owed — the bricks consumer still runs the 32-crossing path) and a STATUS_BOARD section with D-LGJ-FOLD-4 done and D-LGJ-FOLD-5 (the bricks migration) queued. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01GXUahz73MZxtxWcfpHp9dG --- .claude/board/LATEST_STATE.md | 32 ++++++++ .claude/board/STATUS_BOARD.md | 15 ++++ CLAUDE.md | 10 ++- docs/abi.md | 144 +++++++++++++++++++++++++++++++++- 4 files changed, 193 insertions(+), 8 deletions(-) diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 3c59d8f..98d593d 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -1,3 +1,35 @@ +## 2026-09-22 — minor 12: `lgj_plan_group_sum_i32`, the grouped sum that never builds a selection (fold-distillation wave 4) + +Upstream first (lance-graph #1256 merged `99cdca38`: the tiled executor, +`Terminal::GroupSumI32`/`GroupSumViaI32`, `Out::I64`; ndarray #318 merged: +the T1 kernels; lance-graph-java #83 merged: the status arms). This is the +Java→Panama→mask-risc proof that wave: a `GROUP BY` crosses once and no +selection exists at any point. + +- **ABI:** ONE new symbol, no new status, no manifest growth — the plan + surface is reused; only the terminal changed (`Keep` → `GroupSumI32`, or + `GroupSumViaI32` when `via_res != 0`). `n_ops == 0` is legal here (the + whole-lane `Range`, the one `Range` this crate emits, pinned to be exactly + `0..n_rows`). `docs/abi.md` §20. +- **Measured through the membrane:** `View.sumByGroup` = **1 crossing** at + 1,024 and 65,536 rows for 16 groups; the two-crossing-per-group path it + replaces measured **32** beside it in the same run (the `bricks` number, + reproduced). `sumByGroupVia` (the fk-keyed form, `SUM(line) GROUP BY + partner.key`) also 1. +- **Gates:** native **191** tests (+8), clippy `-D warnings` + fmt clean; + Java **612** checks (409 + `GroupSumTest` 203) under JDK 28 + `--enable-preview` against `abi 0.12`; six disable arms red-then-green + (§20.6). `OldAbiCompatTest` gains a minor-12 leg, run BOTH ways: 13/13 + against this library, and against a minor-11 library built from `07e044f` + `View.sumByGroup` throws `AbiMismatchException` naming minor 12 — never a + missing-symbol failure. +- **The eighth named materialisation site:** `Engine.groupSumI32`'s + `toArray`, sized by `groups` (the question), pinned in `DoctrineFenceTest` + and listed in root `CLAUDE.md`. `GroupTotals` exposes no array. +- **Still owed:** the `bricks` consumer's `sumBy()` still runs the 32-crossing + path; migrating it to `sumByGroup` is a consumer-wave change, not this + one. `lgj_hop` still holds mask-sized Vecs (pre-existing, unrelated). + ## 2026-09-19 — PR #81 merged (`07aa441`): production IS the Valhalla arm; Panama × Valhalla is one membrane **The frame, because it is easy to file this wrong:** this was not a JDK diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index ad272d7..1da2a74 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -118,3 +118,18 @@ layout wired end to end. Doctrine: `E-LGJ-THE-MIDDLE-TIER-IS-DELETED-NOT-WRAPPED | D-LGJ-W5 | Three consumer examples (trades / bricks / graph) — one plan file each | **trades DONE 2026-08-17** — `consumers/trades/` (own compile unit, core consumed as a third-party would): `Trade` (schema-not-entity: zero public ctors, zero instance fields, reflection-forced construction still throws), `World.open` → the existing lazy `View` under domain names, zero new membrane surface. TradesParityTest 12/12 (chain vs transcribed-generator recomputation at 1K+64K rows; 0 crossings composing / 1 at terminal THROUGH the domain vocabulary; reflection guard). TradesAllocationTest 3/3 — **the poster's number, measured: 240 bytes/query, IDENTICAL at 64K and 1M rows** (row-count independence is the thesis assertion; 64 KiB absolute backstop). Disable-run: VENUE pointed at the wrong lane → the membrane's own LANE_KIND_MISMATCH rejected it (the binding is checked, not trusted); restored green. **bricks DONE 2026-08-17** — `consumers/bricks/` (2 Sonnet workers K1/K2 per `.claude/waves/wave-consumer-bricks.md`): mask-first RBAC where `authorize(Role)` is a real natively-evaluated predicate in the SAME lazy chain as `where(...)` (`Role.EU_ONLY` = `REGION.eq(EU)`, `DENY_ALL` = `REGION.eq(0xFFFF)` — a genuine impossible predicate, not a Java branch), fail-closed (`UnauthorizedQueryException` BEFORE any crossing; no default-allow path exists), aggregate-only egress (every public method returns `BricksQuery`/`long`/`Map` — structurally no row-shaped type). BricksAuthTest **62/62**: parity vs transcribed generator at 1K+64K; RBAC-as-predicate equivalence (EU_ONLY result == GLOBAL+explicit-where); DENY_ALL counts 0 while paying a real crossing; crossing arithmetic — count()=1, sumBy()=**32 crossings (16 groups × 2: plan_eval + lgj_reduce_sum_i32), IDENTICAL at both row counts** (the thesis: crossings ∝ groups, never rows — the measured 32 corrected K1's "1 per group" Javadoc claim, a real finding about sum-terminal cost); reflection guards. Disable-run: `requireAuthorized` short-circuited → **exactly the 3 can-fire fail-closed checks red, 59 green**; restored, 62/62. Core suite unaffected (188/188). **graph DONE 2026-08-18** — `consumers/graph/` (2 Sonnet workers G1/G2 per `.claude/waves/wave-consumer-graph.md`, dispatched only after the substrate was proven complete at all three levels: generator, ABI, public facade — D-LGJ-W6/W7): `Graph`/`Edge` (schema-not-entity `Edge`, immutable-chaining `Graph` — `from`/`hop`/`minus` each return a NEW `Graph`, mirroring `BricksQuery`'s shape rather than `View`'s laziness since every step but `hop` is already zero-cost). The row-set currency is a Java-side `long[]`, not a native `Mask` — checked and confirmed no public mask-from-row-indices constructor exists; ruled as a deliberate, documented simplification (D1 in the wave file) rather than building a FOURTH core-facade capability under time pressure. `GraphHopTest` **43/43**: hop correctness against the pinned regression (19 @ 1 hop, 29 @ 2 hops) via TWO independently-written pure-Java BFS transcriptions that never call into `Graph`; anti-vacuity; zero-serialization (structural + a reflective public-surface type check); `Edge`'s reflection guard. Disable-run: the target-decode offset corrupted by +4 → hop correctness went red exactly as required (a set-equality check caught it even though the coincidental row COUNT still matched — vindicating G2's choice to assert set equality, not just size). Core suite (204/204) and both prior consumers (trades 12+3/12+3, bricks 62/62) unaffected. **A real measured finding caught and fixed before landing, not shipped wrong:** G2's first draft asserted every hop costs an identical number of crossings; measured, hop 1 on a fresh store costs 2 (the `facetMatches` crossing plus a one-time `RowStore.rawLane()` resolution its first payload read triggers) while hop 2 onward costs exactly 1, steady-state — confirmed directly across 4 consecutive hops before touching the shipped test. Both `Graph.hop()`'s javadoc and `GraphHopTest`'s crossing assertions were corrected to state the true, now-precisely-measured relationship instead of the wrong "identical every hop" assumption | | D-LGJ-W6 | Edge-bearing row store ABI addition (`lgj_rowstore_open_with_edges`, minor 2→3, docs/abi.md §12) — the D1b-shaped "must land as its own W-tier PR before the consumer wave" the graph wave itself named | **DONE 2026-08-18** — orchestrator-authored (genuinely new ABI surface, not consumer-scope work): `registry::open_rowstore_with_edges` + `lgj_rowstore_open_with_edges` (mirrors `lgj_rowstore_open` exactly: same resource kind, same lane shape, no new mask op — purely an alternative constructor), `Engine.openRowStoreWithEdges`/`Abi.requireMinor(3)`, `RowStore.openWithEdges`. `cargo test` **93/93** (+3: registry-level open/describe, out-of-range-classid-matches-plain, radius-overflow-rejected), clippy/fmt clean, release build exports the new symbol (`nm -D`). Java: `AllTests` **194/194** (+6, all in `RowStoreParityTest`) — the strongest new result is a cross-language reproduction of the D1a hop mechanism itself: Java facet-matches + raw-lane-0 payload decode (zero new ABI op) reaches the EXACT same measured hop counts already pinned as a Rust regression (10-row seed → 19 at 1 hop → 29 at 2 hops, `n=2000, seed=0xF00D_CAFE, edge_classid=0, gate_mask=0x0, radius=25`) — proving the two sides of the membrane see identical edge structure, not merely identical classids. Two disable-runs, both red-then-green: (1) registry-level, a classid-not-threaded bug (`open_rowstore_with_edges` hardcoded classid `0`) caught by the out-of-range-parity test; (2) Java-level, the hop's classid-match condition forced to always skip → 1-hop/2-hop both went to 0 and the anti-vacuity assertion failed, exactly as expected. Caught mid-dispatch: the ABI-facing symbol did not exist before this pass (only the bare `RowStore::generate_with_edges` Rust function did, from the prior session) — the graph wave's own STOP-condition-RESOLVED note undersold what was still missing; closed here rather than discovered by G1/G2 mid-flight | | D-LGJ-W7 | Core public facade: `RowStore.classidAt`/`payloadLow64At`/`payloadHi32At` — the per-row zero-copy escape hatch a real `Graph.hop()` needs, since neither `maskOfFacetClass` nor `facetMatches` exposes payload bytes and `ApiSurfaceTest` forbids `MemorySegment`/`internal.*` in any consumer-package signature | **DONE 2026-08-18** — found while checking, concretely, how `consumers/graph` (a genuinely external compile unit per every prior consumer wave's own convention) could ever decode a matched facet's target row: it couldn't — D1a's design assumed a zero-copy raw-lane read Java-side, but nothing on the PUBLIC `RowStore` facade exposed one; only `internal.ffm.Engine.describeLane` did, off-limits to a consumer package by construction. Fixed with THREE new primitive-returning `RowStore` methods, zero new ABI surface at all (reuses `lgj_lane_describe`, already ABI minor 1 — a "lifecycle" crossing per abi.md §6, resolved once and cached; every read after is in-process, matching exports.rs's own stated doctrine "if Java wants one row it reads the MemorySegment in-process, with no crossing at all"). `AllTests` **204/204** (+10 over D-LGJ-W6: 6 in `RowStoreParityTest` — the SAME pinned hop numbers (19/29) reproduced a SECOND time, this time through the genuinely public path a real consumer has to use, plus bounds checks; 6 in `RowStoreLifetimeTest` — closed-before-first-read and closed-after-caching, both guarded). **A real self-caught redundancy, not shipped silently**: the first draft carried the closed-store guard in TWO places (`rawLane()` and `rowOffset()`); disabling one was masked by the other and produced a false-negative disable-run (30/30 green under a broken guard) — caught by re-reading WHY the disable didn't fire rather than accepting the green result, traced to Java method-call evaluation order (`rawLane()`, the receiver, evaluates before `rowOffset()`, its argument), de-duplicated to the ONE correct location, disable-run re-run and confirmed genuinely red-then-green. `ApiSurfaceTest` still 3/3 — zero FFM-typed public signature introduced. **The graph-consumer wave is now dispatchable for real**, proven at three independent levels: Rust generator (D-LGJ-W5's own entry below), ABI membrane (D-LGJ-W6), and the public core facade a consumer package can actually compile against (this row) | + +--- + +## fold-distillation wave — the Java → Panama → mask-risc grouped sum (2026-09-22) + +Upstream substrate: lance-graph #1256 (tiled executor, `GroupSumI32` / +`GroupSumViaI32` terminals, `Out::I64`), ndarray #318 (the T1 kernels), both +merged; lance-graph-java #83 (status arms, merged). Consumer witness on the +same lowering: lance-graph #1257 (SAP CATS, merged). + +| D-id | Deliverable | Status | +|---|---|---| +| D-LGJ-FOLD-4 | ABI minor 12 `lgj_plan_group_sum_i32` + `View.sumByGroup` / `sumByGroupVia` + `GroupTotals`: a `GROUP BY … SUM` as ONE program, one crossing, no selection (`docs/abi.md` §20) | **DONE 2026-09-22** — native 191 tests, Java 612 checks (JDK 28), 1 crossing measured beside the 32-crossing path, six disable arms red-then-green | +| D-LGJ-FOLD-5 | Migrate `consumers/bricks` `sumBy()` from the 32-crossing per-group path onto `sumByGroup` (re-pin BricksAuthTest's crossing arithmetic 32 → 1) | Queued | + diff --git a/CLAUDE.md b/CLAUDE.md index 708afa1..484446e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -283,10 +283,12 @@ build): not row count — verified fixed-size on both the Rust and Java sides); `Abi.java`'s `readCarvings` (bounded by `CARVING_SLOTS`, a manifest constant, not n_rows); `Engine.facetSumResolved`'s fixed `long[2]` - result pair; `View.where()`'s `List.copyOf` of the PREDICATE chain; and - `NativePattern.plan()`'s `predicates.stream()…toList()`. None of the - seven is a hidden proportional-to-n_rows population copy — keep this list - exhaustive when an eighth site is added, rather than letting the + result pair; `View.where()`'s `List.copyOf` of the PREDICATE chain; + `NativePattern.plan()`'s `predicates.stream()…toList()`; and + `Engine.groupSumI32`'s `toArray` of the group totals (minor 12 — sized by + `groups`, the key domain the caller asked for, never by rows). None of the + eight is a hidden proportional-to-n_rows population copy — keep this list + exhaustive when a ninth site is added, rather than letting the enumeration silently go stale again. > **⊘ RE-AUDITED 2026-09-16 and the list WAS stale — it claimed five and diff --git a/docs/abi.md b/docs/abi.md index 613f0e4..a58ce1e 100644 --- a/docs/abi.md +++ b/docs/abi.md @@ -62,8 +62,9 @@ cannot disagree with itself. The ABI is a **machine membrane**. It is not the product. The product is the Java semantic API (see `architecture.md`). Therefore: -- It is **small** — currently 29 symbols (minor 11's three additions are argued - in §19, and the same section argues the SEVEN capabilities it deliberately did +- It is **small** — currently 30 symbols (minor 12's one addition — the grouped + sum that never builds a selection — is argued in §20; 29 at minor 11, whose three + additions are argued in §19, and the same section argues the SEVEN capabilities it deliberately did NOT spend a symbol on; 26 at minor 10, whose one addition — the columnar constructor — is argued in §18; 25 at minor 9, whose one addition is argued in §11: a reduction Java was performing on the wrong side of the membrane, @@ -85,7 +86,7 @@ semantic API (see `architecture.md`). Therefore: ``` LGJ_ABI_MAJOR = 0 // incompatible change ⇒ bump; Java refuses to load -LGJ_ABI_MINOR = 11 // additive change ⇒ bump; older Java may still load +LGJ_ABI_MINOR = 12 // additive change ⇒ bump; older Java may still load LGJ_MAGIC = 0x4C_47_4A_5F_41_42_49_00 // "LGJ_ABI\0" big-endian-read ``` @@ -145,6 +146,17 @@ required — a gate that rejected everything would satisfy a rejection-only test ### Minor version history +- **Minor 12** (2026-09-22) — `lgj_plan_group_sum_i32` (§20): the fused plan + run STRAIGHT INTO a grouped-sum terminal. Every reduction before this minor + paid for a selection first — `lgj_plan_eval` into a mask, then a + `lgj_reduce_*` over it — and a `GROUP BY` paid it once per group (the + `bricks` consumer measured 32 crossings for 16). One symbol lowers the plan + and `GROUP BY key SUM(val)` as ONE `mask_risc::Program` whose accumulator is + tile-local scratch and whose only answer-sized state is the caller's `i64` + per group; a second table's key may be read THROUGH this table's key lane + (`via_res`/`via_lane`) with no partner-side mask. **One crossing for any + number of groups and any number of rows.** No new status; no manifest + growth; a minor-11 Java loads and sees none of it. - **Minor 11** (2026-09-14) — the masking-op completion (§19): the ndarray masking facade finished growing, and this minor consumes what it grew. **Three symbols and seven op-codes for fifteen capabilities**, which is the @@ -427,7 +439,7 @@ predicates or rows are involved. The unfused per-predicate ops are retained only so the fused path can be benchmarked *against* something and so parity can be checked predicate-by-predicate. -## 7. The function surface (29 symbols) +## 7. The function surface (30 symbols) All symbols are prefixed `lgj_`. All return `i32` status except the manifest getter. `out_*` parameters are written only on `OK`. @@ -529,6 +541,21 @@ i32 lgj_reduce_i32(u64 res, u32 lane_id, u32 reduce_op, u64 mask, Sums the `I32` lane over set mask bits into a widened `i64` (no overflow for `n_rows ≤ 2^32` on `i32` inputs). +### Grouped reduction (ABI minor ≥ 12) + +``` +i32 lgj_plan_group_sum_i32(u64 res, const LgjOpDesc* ops, u32 n_ops, + u32 group_lane, u32 val_lane, + u64 via_res, u32 via_lane, + i64* out_sums, u64 n_groups) // minor >= 12, §20 +``` + +`GROUP BY group_lane SUM(val_lane)` over the rows the plan selects, in ONE +crossing and ONE program — no selection is evaluated first and no mask handle +is involved. `out_sums[g]` for `g < n_groups`; a key past `n_groups` is +dropped. `n_ops == 0` is legal and means every row. `via_res != 0` reads the +group key THROUGH `group_lane` into `via_res`'s `via_lane` (the fk-keyed form). + ### Parity escape hatch ``` @@ -1667,3 +1694,112 @@ is textually ABSENT from the function body afterwards, not merely that a replacement occurred), and both then went red. The lesson is the one already on record and worth one more instance: assert what the disable REMOVED, never only that an edit landed. + +## 20. The grouped sum that never builds a selection (ABI minor ≥ 12) + +``` +i32 lgj_plan_group_sum_i32(u64 res, const LgjOpDesc* ops, u32 n_ops, + u32 group_lane, u32 val_lane, + u64 via_res, u32 via_lane, + i64* out_sums, u64 n_groups) +``` + +### 20.1 What was wrong before it + +Every reduction this ABI carried until minor 11 consumed a **mask**: `lgj_plan_eval` +landed the plan's answer in a mask handle, and `lgj_reduce_sum_i32` / +`lgj_reduce_i32` / the register sweeps read that handle. Java never HELD the +selection — it lived natively — but it was still a population-sized thing that +existed only to be consumed by the very next fold, which is the intermediate +materialisation the fold algebra exists to remove (lance-graph #1256's ruling: +*any intermediate population is prohibited if an addressable projection can be +consumed directly by the next fold*). A `GROUP BY` multiplied it by the group +count: `where(key.eq(g)).sumOf(value)` per `g`, measured in the `bricks` +consumer at **32 crossings for 16 groups**. + +### 20.2 What it does + +The plan is lowered exactly as `lgj_plan_eval` lowers it — the prefix rewrite +and the survivor skip of `plan_lower` — but the program ends in +`Terminal::GroupSumI32 { mask: acc, key, val }` instead of `Terminal::Keep`. The +executor runs it tile by tile over `SLOTS × tile_words_for(rows)` words of +scratch; for each tile the terminal folds `val[i]` into `out_sums[key[i]]` for +every selected row and moves on. Nothing population-sized exists at any point: +the accumulator is a tile, the sink is `n_groups` integers, and the crossing +returns when the last tile has folded. + +`via_res != 0` is the fk-keyed form. `group_lane` is then a foreign KEY into +`via_res` (another pattern resource) and the group of row `i` is +`via_lane[group_lane[i]]` — `SUM(line.amount) GROUP BY partner.country` in one +program, `Terminal::GroupSumViaI32`, with the indirection fused inside the +terminal: no partner-side mask, no remapped key lane, no second program. A key +that names no row of `via_res` drops the row (zero fallback at both hops, +`mask_risc`'s contract). `via_res` may equal `res`. + +**`n_ops == 0` is legal here** and means every row. `lgj_plan_eval` refuses an +empty plan because its caller can fill the destination mask itself; a grouped +sum has no destination mask, so the whole-lane case is a program — the one +`Pred::Range` this crate emits, always `0..n_rows`, pinned by +`the_group_sum_range_is_the_whole_lane_and_nothing_else` so that +`exec_error_to_status`'s `RangeOutOfBounds` arm stays an internal-bug mapping. + +### 20.3 Contract + +- `out_sums[g] = Σ val_lane[i]` over selected rows `i` with key `g`, for + `g < n_groups`; a selected row whose key is `>= n_groups` names no group and + is dropped, never an error. Every element is written on `OK` (zero for a + group no row names); none on any failure — `execute_into` validates the whole + program before its first write. +- `group_lane` must be a `U32` lane of `res`, `val_lane` an `I32` lane of `res`, + `via_lane` a `U32` lane of `via_res` when given. +- Statuses: `NULL_ARGUMENT` for a null `out_sums`, a null `ops` with + `n_ops > 0`, or `n_groups == 0` (a zero-length sink is no sink); + `INVALID_HANDLE` / `WRONG_RESOURCE_KIND` for `res` or a non-zero `via_res` that + is not a live pattern; every plan defect as `lgj_plan_eval` reports it; + `INVALID_LANE` / `LANE_KIND_MISMATCH` for the three lanes; `LENGTH_OVERFLOW` + past `2^32 - 1` rows (`Pred::Range` and the sum carry are both `u32`-bounded). + **No new status.** +- Bulk (§6): `O(rows)` in one pass over the predicate lanes plus one read of + the key and value lanes for the selected rows. No per-row and no per-group + crossing. + +### 20.4 Why one symbol, and why no scalar twin + +The plan surface (`LgjOpDesc`) is reused unchanged, so the whole cost of the +capability is the terminal, and the terminal's two shapes (local key / key +through a second table) are one parameter (`via_res`) rather than two symbols +— the shape §15 mandated for the reductions. There is no +`lgj_plan_group_sum_i32_scalar`: parity is falsified in the crate against +`mask_risc`'s row-at-a-time reference executor on the identical lowered +program (both key shapes, multi-tile), and through the membrane against the +two-crossing path it replaces — which runs a different kernel behind a +different terminal, so agreement is evidence rather than a tautology. + +### 20.5 The Java spelling + +`View.sumByGroup(key, value, groups)` → `GroupTotals`, and +`View.sumByGroupVia(key, via, viaKey, value, groups)`. `GroupTotals` is +addressed by key (`total(g)`, `groups()`) and exposes no array — it is sized +by the question the caller typed, never by the data, which is the eighth named +materialisation site (`Engine.groupSumI32`'s `toArray`, pinned in +`DoctrineFenceTest`). Measured: **one crossing** at 1,024 and at 65,536 rows +for 16 groups, beside the per-group path measured at **32** in the same run. + +### 20.6 The disable table (red-then-green, or it is not evidence) + +| what was disabled | test that went red | +|---|---| +| the `via` key silently downgraded to the local key | `via_reads_the_second_table_through_the_key_lane` | +| the all-rows range emptied to `0..0` | `an_empty_plan_and_an_all_or_plan_both_group_every_row`, `the_group_sum_range_is_the_whole_lane_and_nothing_else`, `grouped_sums_agree_with_quack_over_the_combine_sweep` | +| the `n_groups == 0` guard removed | `every_refusal_is_a_status_and_leaves_out_sums_untouched` (mask-risc's own refusal surfaced as `ALLOCATION_FAILED` instead) | +| Java: the `toArray` pin reverted to 1 | `DoctrineFenceTest` — *"UNFENCED MATERIALIZATION: Engine.java\|.toArray( 2 (pinned 1)"* | +| Java: `requirePositiveGroups` made a no-op | `GroupSumTest` — `groups = 0` reached the membrane and came back as the ABI's `NULL_ARGUMENT`, not the Java exception | +| Java: `Engine` passes `viaResource = 0` always | `GroupSumTest` — every `via group g` | + +Each restored from the commit (the disable cycle runs only against committed +work) and re-run green. The compatibility direction was run for real, not +assumed: `OldAbiCompatTest` against a minor-11 library built from `07e044f` +reports *"View.sumByGroup (minor 12) reports an ABI mismatch, not a missing +symbol (threw AbiMismatchException)"*, with every minor-11 feature still +working beside it. +