Skip to content

Corgi reduce: present in bounded windows - #896

Closed
frankmcsherry wants to merge 1 commit into
corgi-merged-idsfrom
corgi-reduce-windows
Closed

frankmcsherry wants to merge 1 commit into
corgi-merged-idsfrom
corgi-reduce-windows

Conversation

@frankmcsherry

Copy link
Copy Markdown
Member

Stacked on #895.

Cover a retire's keys in windows of window_size input records (1 << 12 by default, as VecReduceBackend uses; with_window to choose), as the next_window contract intends, rather than presenting every key at once. The first window searches the input chunks for all of the retire's keys and counts the records each holds; each window then takes the next keys until it holds the budget, never splitting a key, and hands the presentations its share of the matches. Seeds are recorded per window, and the input id pool is cleared per window, as no input id outlives its window.

Same-plan harness, 1 worker, median of 3 (before -> after):

count     1M rows, 1000 x 500    load 81.6 -> 80.0 ms  churn 3.44 -> 3.48 ms/round
min       1M rows, 1000 x 500    load 81.8 -> 79.1 ms  churn 2.60 -> 2.68
distinct  1M rows, 1000 x 500    load 81.4 -> 78.7 ms  churn 2.34 -> 2.40
count2    1M rows, 10000 x 60    load 87.1 -> 85.0 ms  churn 21.7 -> 21.7
min2      1M rows, 10000 x 60    load 95.6 -> 95.0 ms  churn 20.1 -> 20.0
reach     2M edges, 1000 x 100   load 674 -> 683 ms    churn 75.5 -> 76.6
scc       200k edges, 100 x 50   load 667 -> 673 ms    churn 34.7 -> 35.1

Peak RSS for a 4M-row load: count 805 -> 646 MiB, count2 846 -> 770 MiB, distinct 804 -> 645 MiB; reach 638 -> 630 MiB. Forced to a single window, this code is +0.8% to +2.0% on churn against before, so most of the churn cost is the per-retire bookkeeping rather than the number of windows.

The comment this replaces recorded bounded windows as measured and rejected (scc server session at 1 << 14: 84.4s against 63.7s). That session now runs 3.57s before and 3.60s after, at 1 << 12, 1 << 14, or one window alike. Its peak RSS (284 MiB before; 313, 300, 305 MiB after) moves as much between builds with no memory change intended, so I don't read it as signal.

Other comments changed: the key list's "the novel hashes come from the scan the presentation needs anyway" is gone, as the keys now have their own scan and it is the seeds that reuse the search; the galloping-search paragraph moved from collect_present to search, with the code it describes.

Not addressed: under hashed keys, the key pool still keeps each window's presented key column until finish, so that path's key memory is not bounded by the window.

Validation: outputs match compiled DD on the harness cases at 1 and 4 workers, at the default window and at 64 records; the AoC suite passes on the corgi backend at both; the interactive and server suites pass with the window set to one record; workspace tests pass.

🤖 Generated with Claude Code

Cover a retire's keys in windows of `window_size` input records (`1 << 12` by
default, as `VecReduceBackend` uses; `with_window` to choose), as the
`next_window` contract intends, rather than presenting every key at once. The
first window searches the input chunks for all of the retire's keys and counts
the records each holds; each window then takes the next keys until it holds
the budget, never splitting a key, and hands the presentations its share of
the matches. Seeds are recorded per window, and the input id pool is cleared
per window, as no input id outlives its window.

Same-plan harness, 1 worker, median of 3 (before -> after):

  count     1M rows, 1000 x 500    load 81.6 -> 80.0 ms  churn 3.44 -> 3.48 ms/round
  min       1M rows, 1000 x 500    load 81.8 -> 79.1 ms  churn 2.60 -> 2.68
  distinct  1M rows, 1000 x 500    load 81.4 -> 78.7 ms  churn 2.34 -> 2.40
  count2    1M rows, 10000 x 60    load 87.1 -> 85.0 ms  churn 21.7 -> 21.7
  min2      1M rows, 10000 x 60    load 95.6 -> 95.0 ms  churn 20.1 -> 20.0
  reach     2M edges, 1000 x 100   load 674 -> 683 ms    churn 75.5 -> 76.6
  scc       200k edges, 100 x 50   load 667 -> 673 ms    churn 34.7 -> 35.1

Peak RSS for a 4M-row load: count 805 -> 646 MiB, count2 846 -> 770 MiB,
distinct 804 -> 645 MiB; reach 638 -> 630 MiB. Forced to a single window, this
code is +0.8% to +2.0% on churn against before, so most of the churn cost is
the per-retire bookkeeping rather than the number of windows.

The comment this replaces recorded bounded windows as measured and rejected
(scc server session at 1 << 14: 84.4s against 63.7s). That session now runs
3.57s before and 3.60s after, at 1 << 12, 1 << 14, or one window alike. Its
peak RSS (284 MiB before; 313, 300, 305 MiB after) moves as much between
builds with no memory change intended, so I don't read it as signal.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant