quack: HAVING over the K sinks, with a NULL-preserving grouped SUM

.claude/board/TECH_DEBT.md

@@ -1,3 +1,43 @@
+## TD-SYM-SUM-MERGE-IS-NOT-ADDITION-1 (2026-09-23) — OPEN, dormant
+
+**`GroupFold::SumSymI32`'s seed is not an additive identity, so two partial
+sinks must never be combined with `+`.** For MIN/MAX the seed IS the lattice
+identity (`i64::MAX` for min, `i64::MIN` for max), so partial sinks merge with
+plain `min`/`max` and an empty side is absorbed for free. For the `_sym` SUM
+the seed `ndarray::simd::SYM_EMPTY_I64` (`i64::MIN`) marks "no row reached this
+group"; `a + b` on two partials would add −2⁶³ for every group that one side
+never saw, and turn a present group into garbage or a false "empty".
+
+The correct merge is the fold's own rule, lifted:
+
+```text
+merge(a, b) = if a == SYM_EMPTY_I64 { b }
+              else if b == SYM_EMPTY_I64 { a }
+              else { a.wrapping_add(b) }
+```
+
+**Why it is dormant, not live:** nothing merges partial sinks today. The
+mask-risc executor folds every tile sequentially into ONE caller-owned
+`Out::I64` (the `Terminal::GroupReduce` arm of `exec.rs`, seeded once before
+the first tile); `lance-graph-quack` executes one program per sink. There is no
+parallel, per-segment, or per-fragment group reduction in either crate.
+
+**When it goes live:** the first parallel / multi-segment / multi-fragment
+execution of a `GroupReduce { fold: SumSymI32 }` — e.g. a rayon split of the
+tile loop, a Lance-fragment fan-out, or combining sinks across versions. That
+change must use the merge above (or an equivalent `GroupFold::merge`) and must
+carry a test whose partials include a group empty on ONE side only; a fixture
+where every group is present on both sides cannot see the defect.
+
+**Existing guard, and its limit:** `sym_sum_agrees_with_full_range_sum_plus_count`
+(`crates/lance-graph-mask-risc/tests/foreign.rs`) compares the `_sym` SUM with
+full-range SUM + COUNT over the same data and would go red on a wrong merge —
+but only once the merge path runs under it. It does not exist yet, so the test
+cannot fire on it today.
+
+Cross-ref: `.claude/board/entries/2026-09-23-quack-having-sym-sum-presence-mask.md`
+(the `_sym` decision and the presence-mask boundary); ndarray #321; lance-graph #1266.
+
 ## TD-JC-CLIPPY-RED-ON-BASE-2 (2026-09-18) — the 1.98 pre-bump lint sweep was WORKSPACE-scoped, and `jc` is workspace-EXCLUDED
 
 **`JC Substrate Proof` is RED on `main`** (run `35335429357`, head `568965e9`):

.claude/board/entries/2026-09-23-quack-having-sym-sum-presence-mask.md

@@ -0,0 +1,87 @@
+# 2026-09-23 — Quack HAVING: the `_sym` SUM, the presence-mask boundary, the twin check
+
+**Status:** MEASURED (landed on lance-graph #1266) · OPEN (see end)
+**Depends on:** ndarray #321 (`masked_group_sum_sym_i32{,_via}`, `SYM_EMPTY_I64`)
+**Supersedes:** the W-C "group existence is OPEN" item of
+`2026-09-22-quack-duckdb-parity-t0-keyed-reduction.md` (left verbatim there;
+that entry is not edited). Its instruction "decide the fused sum+count sink's
+shape before writing `HAVING`" is resolved differently below: a reserved code
+inside the fold, a mask at the boundary.
+
+## W-C HAVING landed — the empty-group decision
+
+**DECISION:** an empty group is marked by a SEED THE FOLD CANNOT REACH, not
+by a count carried beside the value. The rule is uniform: **empty ⇔ the slot
+still holds `GroupFold::seed`** (`GroupFold::is_empty_slot`). The 2026-09-22
+entry said "decide the fused sum+count sink's shape before writing
+`HAVING`"; this supersedes that, because the sentinel closes the NULL gap
+with no second sink.
+
+- Seeds: `Count` → 0 (never empty — a zero count is an answer), `MinI32` →
+  `i64::MAX`, `MaxI32` → `i64::MIN`, **new `SumSymI32` → `i64::MIN`**. MIN
+  cannot share SUM's seed (a min fold must start from its identity), so the
+  rule is "equals its own seed", not "equals `i64::MIN`".
+- **BASIS:** the same move as a 4-bit code read as −7..+7 + NaN rather than
+  −8..+7: give up one representable value to get a NULL code and a
+  range closed under negation. At 4/8 bit that also removes a median bias; at
+  i64 the bias is negligible, and what is bought is closure + NULL.
+- Cost: exactly one row of range, for this fold only.
+  `GROUP_SUM_SYM_MAX_ROWS = 2^32 − 1`, because exactly 2^32 rows of
+  `i32::MIN` sum to `i64::MIN`. `MASKED_SUM_I32_MAX_ROWS` stays `2^32`: the
+  coalescing sums may legitimately produce `i64::MIN`.
+- Kernel: ndarray #321 `masked_group_sum_sym_i32{,_via}`, first row
+  REPLACES the marker, later rows `wrapping_add`. Two named closures over the
+  shared `group_walk`; no new bit loop.
+
+**HAVING is finalization.** `lower_group_having` emits one `GroupReduce` program
+per aggregate over the same filter and key; `GroupHavingPlan::finish` is
+the O(K) pass that yields a K-bit group mask. A group survives iff it was
+REACHED (read off the first sink: count ≠ 0, or slot ≠ seed) and every
+comparison holds. There is no per-predicate NULL check. Every sink shares the
+filter and the key, so a reached group is non-empty in all of them, and a
+guard that cannot fire would be decoration.
+
+Five DuckDB cases, 34 total: HAVING on the selected aggregate, on a different
+aggregate, fk-keyed with a conjunction, and two over a filter that empties
+three groups (`HAVING COUNT(*) >= 0` and `HAVING SUM(amount) < 10000` with
+the SUM sink carrying reachedness). Disable-verified red:
+- reachedness off → both sparse cases;
+- non-count emptiness off → the SUM-first sparse case;
+- the executor routing `SumI32` through the coalescing kernel → both
+  mask-risc differentials.
+
+**Boundary refinement (same day, operator-raised):** the marker is free
+inside the fold and a silent wrong answer outside it — any full-range
+consumer would sum, sort or negate `i64::MIN`. Two consequences, both
+landed:
+- **Named, not implied.** ndarray exposes the reservation only under a
+  `_sym` suffix (`masked_group_sum_sym_i32{,_via}`, `SYM_EMPTY_I64`); every
+  unsuffixed reduction stays full two's-complement and never treats
+  `i64::MIN` specially (pinned by a test). mask-risc names the fold
+  `GroupFold::SumSymI32` with `GROUP_SUM_SYM_MAX_ROWS`. This is the 4-bit
+  analogue of choosing −7..+7 + NaN by name, never silently narrowing a
+  −8..+7 consumer.
+- **NULL leaves as a mask.** `normalize_group_sink` is the quack boundary:
+  it returns a K-bit presence mask and zeroes absent slots in place; `finish`
+  runs it over every sink and returns `{present, keep}`. The raw sink is
+  internal encoding. Row NULL and group NULL now have the same shape.
+  Disable-verified: no zeroing → the leak assertion and unit tests fail;
+  normalizing only the first sink → the multi-sink unit test fails.
+
+**Two SUM spellings are a deliberate twin, not debt** (operator, same
+day). Full-range `GroupSumI32` + `COUNT` and `_sym` `SumSymI32` answer the
+same question two independent ways, so either can check the other whenever
+something looks off. `sym_sum_agrees_with_full_range_sum_plus_count` keeps
+that check permanent: presence ⇔ count ≠ 0, equal values where present,
+full-range 0 where absent, on both key addresses. It is also the only check
+that can see a real sum colliding with the reserved code. Disable-verified:
+routing `SumSymI32` through the full-range kernel turns it red.
+
+REVISIT WHEN: a single-pass AVG is wanted — the fused sum+count sink is still
+the route to that, now for speed only, not for NULL.
+
+## What is still open (not done here)
+- **W-C:** `ORDER BY rid LIMIT n` — a first-n select, a rank/select member.
+- **W-D:** multi-key `GROUP BY` via a fused composite address.
+- **Debt:** `TD-SYM-SUM-MERGE-IS-NOT-ADDITION-1` — `_sym` partial sinks must
+  merge by the fold's rule, never `+`; dormant until a parallel reduction.

.claude/board/entries/README.md

@@ -25,10 +25,11 @@ index row, (3) no duplicate entry id. Checks 1 and 2 are deliberately
 opposite directions; the stranding this convention prevents shows up in
 exactly one of them, never both.
 
-147 entries, 2026-08-06 .. 2026-09-22.
+148 entries, 2026-08-06 .. 2026-09-23.
 
 | date | entry id | finding | file |
 |---|---|---|---|
+| 2026-09-23 | `quack-having-sym-sum-presence-mask` |  | [2026-09-23-quack-having-sym-sum-presence-mask.md](2026-09-23-quack-having-sym-sum-presence-mask.md) |
 | 2026-09-22 | `quack-duckdb-parity-t0-keyed-reduction` |  | [2026-09-22-quack-duckdb-parity-t0-keyed-reduction.md](2026-09-22-quack-duckdb-parity-t0-keyed-reduction.md) |
 | 2026-09-22 | `E-W0C-THE-ROW-BRIDGE-IS-A-DIALECT-NOT-AN-INTERPRETER-1` | a merged relational op carried as loco program data reaches the fused executor with no population crossing; the enum explosion is upstream of mask-risc | [2026-09-22-e-w0c-the-row-bridge-is-a-dialect-not-an-interpreter-1.md](2026-09-22-e-w0c-the-row-bridge-is-a-dialect-not-an-interpreter-1.md) |
 | 2026-09-22 | `E-CATS-FOLD-DOES-NOT-RETAIN-A-POPULATION-BITMAP-1` | CATS aggregate lowers to one tiled grouped terminal; bitmap realization is a requested boundary sink | [2026-09-22-e-cats-fold-does-not-retain-a-population-bitmap-1.md](2026-09-22-e-cats-fold-does-not-retain-a-population-bitmap-1.md) |

crates/lance-graph-mask-risc/src/exec.rs

@@ -30,10 +30,11 @@ use ndarray::simd::{
     mask_not_assign, mask_or, mask_or_assign, mask_scatter_or_u32, mask_set_range, mask_xor,
     mask_xor_assign, masked_group_count_u32, masked_group_count_u32_via, masked_group_max_i32,
     masked_group_max_i32_via, masked_group_min_i32, masked_group_min_i32_via, masked_group_sum_i32,
-    masked_group_sum_i32_via, masked_key_run_count_u32, masked_max_i32, masked_min_i32,
-    masked_sum_i32, ne_i32_to_mask, ne_i32_to_mask_under, ne_u32_to_mask, ne_u32_to_mask_under,
-    popcount_batch_u64, ternary_match_u32_to_mask, ternary_match_u32_to_mask_under,
-    ternary_match_u64_to_mask, ternary_match_u64_to_mask_under, KeyRunCarry,
+    masked_group_sum_i32_via, masked_group_sum_sym_i32, masked_group_sum_sym_i32_via,
+    masked_key_run_count_u32, masked_max_i32, masked_min_i32, masked_sum_i32, ne_i32_to_mask,
+    ne_i32_to_mask_under, ne_u32_to_mask, ne_u32_to_mask_under, popcount_batch_u64,
+    ternary_match_u32_to_mask, ternary_match_u32_to_mask_under, ternary_match_u64_to_mask,
+    ternary_match_u64_to_mask_under, KeyRunCarry,
 };
 
 use crate::ir::{
@@ -1042,6 +1043,21 @@ pub fn execute_into(
                                 o,
                             )
                         }
+                        (GroupKey::Lane(k), GroupFold::SumSymI32(v)) => masked_group_sum_sym_i32(
+                            m,
+                            lane_u32(planes, k, t),
+                            lane_i32(planes, v, t),
+                            o,
+                        ),
+                        (GroupKey::Via { fk, key }, GroupFold::SumSymI32(v)) => {
+                            masked_group_sum_sym_i32_via(
+                                m,
+                                lane_u32(planes, fk, t),
+                                foreign_lane_u32(foreign, key),
+                                lane_i32(planes, v, t),
+                                o,
+                            )
+                        }
                     }
                 }
             }

crates/lance-graph-mask-risc/src/ir.rs

@@ -360,10 +360,14 @@ pub enum Terminal {
     /// identity ([`GroupFold::seed`]): `0` for a count, `i64::MAX` for a
     /// minimum, `i64::MIN` for a maximum. A MIN/MAX slot still holding its
     /// seed afterwards is a group no selected row named — the SQL `NULL` of
-    /// an empty group. No per-group sum is involved, so no row bound applies.
+    /// an empty group. Only [`GroupFold::SumSymI32`] carries a row bound
+    /// ([`GROUP_SUM_SYM_MAX_ROWS`]).
     ///
-    /// `SUM` keeps its own terminals ([`Terminal::GroupSumI32`] /
-    /// [`Terminal::GroupSumViaI32`]); this one does not repeat them.
+    /// The coalescing `SUM` (empty group reads `0`) keeps its own terminals
+    /// ([`Terminal::GroupSumI32`] / [`Terminal::GroupSumViaI32`]). The
+    /// NULL-preserving `SUM` lives here as [`GroupFold::SumSymI32`], because the
+    /// empty-group rule is the same for every seeded fold: a slot still
+    /// holding [`GroupFold::seed`] is empty ([`GroupFold::is_empty_slot`]).
     GroupReduce {
         mask: Operand,
         key: GroupKey,
@@ -391,6 +395,17 @@ pub enum GroupFold {
     MinI32(u16),
     /// `MAX(lanes[val])` over an `I32` lane.
     MaxI32(u16),
+    /// `SUM(lanes[val])` over an `I32` lane in the SYMMETRIC range — the
+    /// `_sym` reading of `ndarray::simd`: real sums live in `±(2^63 − 1)`
+    /// and `ndarray::simd::SYM_EMPTY_I64` (`i64::MIN`) is reserved for "no
+    /// selected row reached this group". The first row a group sees REPLACES
+    /// the marker; later rows add. That keeps a group whose values cancel to
+    /// `0` distinct from an empty one, which the full-range
+    /// [`Terminal::GroupSumI32`] cannot tell apart. The reservation is
+    /// named, never implied: every other sum here is full range. Carries the
+    /// tighter [`GROUP_SUM_SYM_MAX_ROWS`]. The raw sink is internal encoding;
+    /// a consumer maps the marker away before treating slots as integers.
+    SumSymI32(u16),
 }
 
 impl GroupFold {
@@ -402,10 +417,30 @@ impl GroupFold {
             GroupFold::Count => 0,
             GroupFold::MinI32(_) => i64::MAX,
             GroupFold::MaxI32(_) => i64::MIN,
+            GroupFold::SumSymI32(_) => ndarray::simd::SYM_EMPTY_I64,
+        }
+    }
+
+    /// Whether a slot holding `v` after the fold is a group no selected row
+    /// reached — the SQL `NULL` of an empty group. The rule is uniform:
+    /// empty ⇔ the slot still holds [`GroupFold::seed`]. `COUNT` has no
+    /// empty groups: a zero count is a real answer, not a `NULL`.
+    pub const fn is_empty_slot(self, v: i64) -> bool {
+        match self {
+            GroupFold::Count => false,
+            _ => v == self.seed(),
         }
     }
 }
 
+/// The widest plane a NULL-preserving [`GroupFold::SumSymI32`] is defined on:
+/// `2^32 − 1` rows. One less than [`MASKED_SUM_I32_MAX_ROWS`] because the
+/// seed doubles as the empty marker: exactly `2^32` rows of `i32::MIN` sum
+/// to `i64::MIN`, a real value that would read back as `NULL`. Below that
+/// row count every real sum lies strictly above `i64::MIN`, leaving the
+/// symmetric range `±(2^63 − 1)` — closed under negation.
+pub const GROUP_SUM_SYM_MAX_ROWS: usize = (1 << 32) - 1;
+
 /// The widest plane [`Terminal::MaskedSumI32`] is defined on: `2^32` rows.
 /// The binding side is the NEGATIVE one: `2^32 · i32::MIN = −2^63 = i64::MIN`
 /// exactly, and one more row of `i32::MIN` wraps. The positive side has two

crates/lance-graph-mask-risc/src/lib.rs

@@ -124,7 +124,7 @@ pub use exec::{
 pub use fuse::{fuse, fuse_program, ternlog_imm, BoolExpr, FuseError, Fused};
 pub use ir::{
     Foreign, ForeignPlane, GroupFold, GroupKey, LaneRef, MaskOp, Operand, Planes, Pred, Program,
-    Terminal, MASKED_SUM_I32_MAX_ROWS, MAX_SCRATCH_SLOTS,
+    Terminal, GROUP_SUM_SYM_MAX_ROWS, MASKED_SUM_I32_MAX_ROWS, MAX_SCRATCH_SLOTS,
 };
 pub use reference::{
     reference_execute, reference_execute_into, reference_scratch, reference_scratch_with_foreign,

crates/lance-graph-mask-risc/src/reference.rs

@@ -18,7 +18,7 @@
 
 use crate::ir::{
     Foreign, GroupFold, GroupKey, LaneRef, MaskOp, Operand, Planes, Pred, Program, Terminal,
-    MASKED_SUM_I32_MAX_ROWS, MAX_SCRATCH_SLOTS,
+    GROUP_SUM_SYM_MAX_ROWS, MASKED_SUM_I32_MAX_ROWS, MAX_SCRATCH_SLOTS,
 };
 use crate::value::{ExecError, LaneKind, Out, Value};
 use crate::words_for;
@@ -537,6 +537,12 @@ pub(crate) fn validate(
                 GroupFold::MinI32(v) | GroupFold::MaxI32(v) => {
                     check_lane(planes, v, LaneKind::I32)?
                 }
+                GroupFold::SumSymI32(v) => {
+                    check_lane(planes, v, LaneKind::I32)?;
+                    if n > GROUP_SUM_SYM_MAX_ROWS {
+                        return Err(ExecError::SumRowBound { n_rows: n });
+                    }
+                }
             }
             match out {
                 OutShape::I64(len) if len >= 1 => Ok(()),
@@ -905,6 +911,7 @@ pub fn reference_execute_into(
                     GroupFold::Count => 0i64,
                     GroupFold::MinI32(_) => i64::MAX,
                     GroupFold::MaxI32(_) => i64::MIN,
+                    GroupFold::SumSymI32(_) => i64::MIN,
                 };
                 for x in o.iter_mut() {
                     *x = seed;
@@ -934,6 +941,15 @@ pub fn reference_execute_into(
                         GroupFold::Count => o[k] + 1,
                         GroupFold::MinI32(v) => o[k].min(i64::from(i32_at(planes, v, r))),
                         GroupFold::MaxI32(v) => o[k].max(i64::from(i32_at(planes, v, r))),
+                        // First row replaces the empty marker; later rows add.
+                        GroupFold::SumSymI32(v) => {
+                            let x = i64::from(i32_at(planes, v, r));
+                            if o[k] == i64::MIN {
+                                x
+                            } else {
+                                o[k] + x
+                            }
+                        }
                     };
                 }
             }