Skip to content

Corgi reduce: merge unit-valued runs without a sort - #894

Merged
frankmcsherry merged 1 commit into
master-nextfrom
corgi-unit-ids
Sep 22, 2026
Merged

frankmcsherry merged 1 commit into
master-nextfrom
corgi-unit-ids

Conversation

@frankmcsherry

@frankmcsherry frankmcsherry commented Sep 22, 2026 •

Copy link
Copy Markdown
Member

merge_present merges presented chunk runs straight into the bridge when they arrive in ((key, value_id), time) order, and otherwise consolidates. It recognized leaf values as ordered but not Unit values, though all unit rows share one id. Unit-valued presentations (distinct's output, and unit-valued inputs such as reach's join(edges, ($2 ;)) | distinct) paid for a sort. The merge also reserves its bridge up front, as the fallback's extend does, rather than growing it by doubling.

The ordered_ids line is the same as in #870, which predates the IdPool refactor.

Same-plan harness, 1 worker, median of 3 (before -> after):

reach     2M edges, 1000 x 100   load 767 -> 679 ms    churn 85.2 -> 76.4 ms/round
scc       200k edges, 100 x 50   load 674 -> 666 ms    churn 35.6 -> 35.4
distinct  1M rows, 1000 x 500    load 87.5 -> 84.9 ms  churn 2.51 -> 2.45
count     1M rows, 1000 x 500    load 88.5 -> 85.7 ms  churn 3.62 -> 3.54

Peak RSS: a 4M-row count or distinct load 1289 -> 1145 MiB (the reserve); reach 609 -> 640 MiB, not traced.

Validation: outputs match compiled DD on the harness cases (count, count2, distinct, min, min2, reach, scc) at 1 and 4 workers; workspace tests pass.

🤖 Generated with Claude Code

`merge_present` merges presented chunk runs straight into the bridge when they
arrive in `((key, value_id), time)` order, and otherwise consolidates. It
recognized leaf values as ordered but not `Unit` values, though all unit rows
share one id. Unit-valued presentations (`distinct`'s output, and unit-valued
inputs such as reach's `join(edges, ($2 ;)) | distinct`) paid for a sort. The
merge also reserves its bridge up front, as the fallback's `extend` does,
rather than growing it by doubling.

The `ordered_ids` line is the same as in #870, which predates the `IdPool`
refactor.

Same-plan harness, 1 worker, median of 3 (before -> after):

  reach     2M edges, 1000 x 100   load 767 -> 679 ms    churn 85.2 -> 76.4 ms/round
  scc       200k edges, 100 x 50   load 674 -> 666 ms    churn 35.6 -> 35.4
  distinct  1M rows, 1000 x 500    load 87.5 -> 84.9 ms  churn 2.51 -> 2.45
  count     1M rows, 1000 x 500    load 88.5 -> 85.7 ms  churn 3.62 -> 3.54

Peak RSS: a 4M-row count or distinct load 1289 -> 1145 MiB (the reserve);
reach 609 -> 640 MiB, not traced.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@frankmcsherry
frankmcsherry merged commit 105aa08 into master-next Sep 22, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant