Corgi reduce: merge unit-valued runs without a sort - #894
Merged
Merged
Conversation
`merge_present` merges presented chunk runs straight into the bridge when they arrive in `((key, value_id), time)` order, and otherwise consolidates. It recognized leaf values as ordered but not `Unit` values, though all unit rows share one id. Unit-valued presentations (`distinct`'s output, and unit-valued inputs such as reach's `join(edges, ($2 ;)) | distinct`) paid for a sort. The merge also reserves its bridge up front, as the fallback's `extend` does, rather than growing it by doubling. The `ordered_ids` line is the same as in #870, which predates the `IdPool` refactor. Same-plan harness, 1 worker, median of 3 (before -> after): reach 2M edges, 1000 x 100 load 767 -> 679 ms churn 85.2 -> 76.4 ms/round scc 200k edges, 100 x 50 load 674 -> 666 ms churn 35.6 -> 35.4 distinct 1M rows, 1000 x 500 load 87.5 -> 84.9 ms churn 2.51 -> 2.45 count 1M rows, 1000 x 500 load 88.5 -> 85.7 ms churn 3.62 -> 3.54 Peak RSS: a 4M-row count or distinct load 1289 -> 1145 MiB (the reserve); reach 609 -> 640 MiB, not traced. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
merge_presentmerges presented chunk runs straight into the bridge when they arrive in((key, value_id), time)order, and otherwise consolidates. It recognized leaf values as ordered but notUnitvalues, though all unit rows share one id. Unit-valued presentations (distinct's output, and unit-valued inputs such as reach'sjoin(edges, ($2 ;)) | distinct) paid for a sort. The merge also reserves its bridge up front, as the fallback'sextenddoes, rather than growing it by doubling.The
ordered_idsline is the same as in #870, which predates theIdPoolrefactor.Same-plan harness, 1 worker, median of 3 (before -> after):
Peak RSS: a 4M-row count or distinct load 1289 -> 1145 MiB (the reserve); reach 609 -> 640 MiB, not traced.
Validation: outputs match compiled DD on the harness cases (count, count2, distinct, min, min2, reach, scc) at 1 and 4 workers; workspace tests pass.
🤖 Generated with Claude Code