Corgi reduce: one id scheme, bounded windows

[frankmcsherry]

Supersedes #895 and #896.

Reworks the corgi reduce backend around one way of naming values, and presents the key space in bounded windows. reduce.rs goes from 722 to 551 non-test lines.

Ids

• A key's id is its arrangement identifier, as before.
• A value's id names a row: (column, row) in a pool of the chunks' own value columns and the output columns the backend builds.
• Input: each key's rows are sorted by value with corgi's sort_blocks; each run of equal values is one id.
• Output: each key's desired values are netted against its current output by comparing values (compare_at); a value not seen before gets a new id.

Removed: IdPool, IdHasher, ids(), merge_present, the separate presentation paths, and the HashMap in reduce_corrections. Values are no longer hashed anywhere in the backend.

Windows: a retire's keys are searched once, then presented a few at a time, each window holding about window_size input records (default 1 << 12, as VecReduceBackend uses). A key is never split across windows.

Measurements (same-plan harness, 1 worker, median of 3; master-next -> this):

case      size, churn             load ms       churn ms/round
count2    1M rows, 1000 x 500     143 -> 88.6   4.79 -> 3.78
count2    1M rows, 10000 x 60     144 -> 88.4   31.8 -> 22.8
min2      1M rows, 1000 x 500     163 -> 96.0   4.38 -> 3.26
min2      1M rows, 10000 x 60     164 -> 96.6   31.6 -> 21.1
count     1M rows, 1000 x 500     86.3 -> 83.5  3.53 -> 3.47
distinct  1M rows, 1000 x 500     84.8 -> 82.7  2.44 -> 2.41
min       1M rows, 1000 x 500     85.5 -> 88.2  2.70 -> 2.78
reach     2M edges, 1000 x 100    680 -> 693    76.5 -> 78.0
scc       200k edges, 100 x 50    669 -> 697    35.3 -> 36.5

Peak RSS, 4M-row loads: count 1145 -> 646 MiB, count2 1397 -> 770 MiB, distinct 1145 -> 646 MiB; reach 640 -> 663 MiB.

min, reach, and scc are 2-4% slower. The cost is the input sort: chunks are already sorted by value, and this sorts again rather than merging those runs. A sort_blocks that uses presorted runs would recover it inside corgi, leaving this file as is.

Validation: outputs match compiled DD for count, count2, distinct, min, min2, reach, scc at 1 and 4 workers; AoC 33/33 on corgi; workspace tests pass; interactive and server suites pass with a one-record window.

Comments removed with their code: IdHasher's profiling note, the ids() and merge_present docs, and "Both sides ascend, so this is a merge" (key lists are now extend, sort, dedup). The module doc's ids and transcode paragraphs are rewritten for the new scheme, and Min's "reuses its input value id" no longer applies.

🤖 Generated with Claude Code