Skip to content

Corgi advance: keep a chunk's columns when nothing reorders - #903

Open
frankmcsherry wants to merge 1 commit into
master-nextfrom
corgi-advance-in-place
Open

frankmcsherry wants to merge 1 commit into
master-nextfrom
corgi-advance-in-place

Conversation

@frankmcsherry

Copy link
Copy Markdown
Member

CorgiChunk::advance did three things per call that it mostly didn't need to:

  • concatenated its carried trailing group with the new chunks, a copy of every row;
  • gathered every surviving row into new key and value columns, a second copy;
  • sorted each group's rows by time, one group at a time.

In scc's churn rounds only 3% of groups need re-sorting after advancing (138k of 4.4M, counted).

Now advance works on whole chunks:

  • A chunk is complete once the next chunk starts a new (key, val) group, or on done. Until then the last chunk waits, rather than being split to hold back its last group.
  • Chunks that one group spans are concatenated and advanced together, as the old code did for everything.
  • After advancing, a group whose times still ascend is left as it stands. When no group needs re-sorting, the chunk keeps its key and value columns: no copy, no gather.

One file, 72 lines added, 69 removed. Stands alone; no overlap with #902.

Corgi, 3 runs each, ms (load, then churn per round):

case master-next load this load master-next churn this churn
count 83.4–84.9 83.4–84.9 3.250–3.256 3.030–3.034
count2 (churn 10k) 87.3–88.7 87.3–88.8 22.78–22.80 21.66–21.74
min 86.1–86.6 86.1–87.6 2.546–2.569 2.394–2.418
reach 686–690 642–647 69.26–69.36 65.22–65.34
scc 674–680 654–681 32.89–33.14 31.46–31.59

Validation: interactive tests pass, including advance_owned_and_shared_nested_times_matches_reference, which feeds groups split across 1- and 3-row chunks. AoC 2023 passes 33/33 on corgi. The same-plan harness's output digests match compiled DD for count, count2, distinct, min, min2, reach and scc, at 1 and 4 workers.

🤖 Generated with Claude Code

`advance` concatenated its carried group with each new chunk (a copy), then
gathered every surviving row into new columns (a second copy), and sorted each
group's rows by time, one group at a time. In scc's churn rounds 3% of groups
need re-sorting after advancing.

Now it advances whole chunks: a chunk is complete once the next chunk starts a
new group, or on `done`, and chunks that a group spans are advanced together.
A group whose advanced times still ascend is left as it stands, and when no
group needs re-sorting the chunk keeps its key and value columns.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ABLpEcZUPN1oAUY2usfkx

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant