quack/mask-risc: DuckDB parity W-B + W-A — GroupReduce fold, AVG, SUM(CASE), NOT EXISTS, NULL-aware aggregates

[AdaWorldAPI]

What

Two parity waves. Every case's expected value is generated by DuckDB, never hand-edited. 29/29 cases match DuckDB 1.5.5, and the original 16 expected values are unchanged.

W-B: fold grouped COUNT/MIN/MAX (commit 219d4a81)

Before, GROUP BY with COUNT, MIN or MAX ran as K+1 programs over an O(N) kept mask. It now runs as one program ending in Terminal::GroupReduce { mask, key, fold } and writing a K-slot Out::I64 sink. This is the same T0 fold #1262 applied to SUM.

• mask-risc:
  • New types: GroupKey::{Lane, Via{fk,key}} and GroupFold::{Count, MinI32, MaxI32}.
  • GroupFold::seed() returns 0, i64::MAX or i64::MIN. All three are outside the i32 range, so a slot still at its seed is an empty group, i.e. SQL NULL.
  • New Value::GroupReduced.
  • exec dispatches onto the ndarray keyed-reduction family; reference.rs gets the oracle arm and the validation.
• quack:
  • New Agg::GroupReduce { key: GroupAddr::{Local, Via}, agg: GroupAgg }.
  • lower_group_by_semantic now folds Count/Min/Max; Rows stays forest residue.
  • The quack: semantically fold GROUP BY SUM before materialization #1262 test is re-pinned: each fold must match the forest group by group, with an empty group's NULL matching the seed.
• Cases: group_min_cc, group_max_cc, group_max_cc_sparse (3 empty groups → NULL), join_group_count_country and join_group_min_country. group_count_cc now runs folded, with the forest kept as a comparison METRIC.

W-A: shapes that need no new primitive (commit cb8062e0)

• SUM(CASE WHEN p THEN x ELSE 0 END): p goes into the mask and the aggregate is the plain masked SUM (sum_case_posted_cc3).
• AVG:
  • lower_avg / lower_group_avg run SUM and COUNT over the same filter, keyed Local or Via; avg_finish does the final divide outside the hot path.
  • The float digits match DuckDB exactly (avg_posted, group_avg_cc, join_group_avg_country).
  • It costs two passes. That's correct but not minimal: a fused sum+count sink would make it one.
• NOT EXISTS:
  • It lowers to Not around the factored fk predicate, as one program with no partner-side mask (anti_join_not_country3).
  • An fk naming no partner row keeps the row, which is NOT EXISTS semantics rather than <>.
• NULLs:
  • The fixture gains one derived nullable column, line.discount, with no RNG draw so no existing value moves.
  • Its validity plane is ANDed into the filter for COUNT(col), SUM(col) and AVG(col).
  • COUNT(col) is asserted to differ from COUNT(*).

Verification

• Tests pass for lance-graph-mask-risc and lance-graph-quack (30 harness tests, including the fixture drift guard), and for the dependents lance-graph-sap, r2il-mask-abi-probe and d-diamond-1-probe.
• lgj-abi passes cargo check --tests.
• clippy -D warnings and fmt are clean.
• Disable arms, each red then restored:
  • W-B: folding COUNT turned off; MIN dispatched to MAX; seed printed instead of NULL.
  • W-A: off-by-one AVG denominator; Not dropped; validity plane dropped; NULL written as 0.

Depends on

AdaWorldAPI/ndarray#320, which adds the keyed-reduction family masked_group_{count_u32,min_i32,max_i32}{,_via}. It is green and still a draft. CI here syncs ndarray master, so clippy / cats / member-tests / test fail with E0432 until #320 merges.

Board

.claude/board/entries/2026-09-22-quack-duckdb-parity-t0-keyed-reduction.md holds:

• the primitive basis, mapped facade → mask-risc → lgj-abi;
• the W-A results;
• what is still open: W-C (HAVING needs an i64 compare-to-mask; ORDER BY rid LIMIT n needs a first-n select) and W-D (multi-key GROUP BY via a composite address).

🤖 Generated with Claude Code

https://claude.ai/code/session_01GXUahz73MZxtxWcfpHp9dG

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Essentials

Run ID: 44314cbd-9488-48fe-923e-d1e7adca34fe

📥 Commits

Reviewing files that changed from the base of the PR and between 4f11b36 and da7a467.

⛔ Files ignored due to path filters (2)

• crates/lance-graph-quack/tests/duckdb/cases.tsv is excluded by !**/*.tsv
• crates/lance-graph-quack/tests/duckdb/data/line.csv is excluded by !**/*.csv

📒 Files selected for processing (12)

• .claude/board/entries/2026-09-22-quack-duckdb-parity-t0-keyed-reduction.md
• .claude/board/entries/README.md
• crates/lance-graph-mask-risc/src/exec.rs
• crates/lance-graph-mask-risc/src/ir.rs
• crates/lance-graph-mask-risc/src/lib.rs
• crates/lance-graph-mask-risc/src/reference.rs
• crates/lance-graph-mask-risc/src/value.rs
• crates/lance-graph-mask-risc/tests/foreign.rs
• crates/lance-graph-quack/src/lib.rs
• crates/lance-graph-quack/tests/duckdb/fixture.rs
• crates/lance-graph-quack/tests/duckdb/oracle.py
• crates/lance-graph-quack/tests/duckdb_differential.rs

 ______________________________________________
< Preventing the Y2K bug from happening again. >
 ----------------------------------------------
  \
   \   \
        \ /\
        ( )
      .( o ).

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

---

_{Comment @coderabbitai help to get the list of available commands.}

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_90f99800-fa90-42b1-99dc-5a96b247b740)

[AdaWorldAPI]

Blocker: clippy and cats are red because ndarray #320 hasn't merged yet. The compile jobs still in progress will fail the same way.

• What fails: error[E0432]: unresolved imports ndarray::simd::masked_group_{count_u32,min_i32,max_i32}{,_via} at crates/lance-graph-mask-risc/src/exec.rs:31.
• Why: CI syncs AdaWorldAPI/ndarray at master, and those six functions are added by simd: keyed-reduction family — group count/min/max over resident or VIA keys ndarray#320.
• No fix belongs in this repo: the missing code is the new ndarray family itself, so nothing can be ported here.
• Local results against ci: declare rustfmt + clippy as pinned-toolchain components #320's head: mask-risc, quack, the three dependents and the DuckDB differential (21/21) are all green.

The one red check on #320 is a CI cache/CPU mismatch (SIGILL), explained on that PR, and its failed job has been re-run. Once #320 is on ndarray master, this PR's CI re-runs clean with no change here.

---

Generated by Claude Code

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_1a289a76-2dbe-4222-a624-f86c77fd7b08)

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_ea8d1609-b47a-4cbf-b655-058ba46a6aaf)