Prototype collective differences through reduce replay - #892
Open
frankmcsherry wants to merge 4 commits into
Open
frankmcsherry wants to merge 4 commits into
frankmcsherry wants to merge 4 commits into
Conversation
Carry aligned metadata and opaque diff containers through history maintenance and correction feedback. Adapt Vec and Corgi reduce backends, preserving the timestamp schedule. Extend the composed-diff example through partial-order replay. This remains an integration experiment: scalar benchmarks regress, and flat metadata, gathered selections, and per-key scratch need further attention.
frankmcsherry
changed the title
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proxy reduce now carries opaque difference containers through presentations, history replay, accumulations, correction feedback, and emission.
The tactic has no scalar difference type or
Semigroupbound.The Vec and Corgi backends use
Vec<R>containers and retain their existing user-facing difference semantics.Keys and values retain the
Copy + Ordgeneralization landed in #891.Join still uses scalar differences, and the multiplication interface remains unintegrated.
The first commit introduces collective copying, grouped accumulation, zero selection, and multiplication without
BatchContainer.The second integrates reduce using aligned metadata and difference containers.
History advances and sorts metadata, then delegates sums and movement to the container.
Corrections enter a separate output history that is advanced, consolidated, and included at subsequent crossings.
The timestamp schedule is unchanged.
Backend-supplied empty storage preserves runtime schemas; reusable consolidation scratch and in-place retention avoid copying surviving sums a second time.
The integration exposes representation costs that remain unresolved:
(value, time)metadata replaces the old value-grouped edit list, repeating value IDs and consolidating through a sorted permutation.This is an integration experiment with measured regressions, not a performance-neutral abstraction.
Release
int_proxy_bench,MODE=proxy, default sizes, medians of three runs with alternating baseline/integration order.These measurements compare
24461268and09bc0c25:The single columnar-difference test now exercises component cancellation, incomparable timestamps, replay advancement, and correction feedback.
Validation passed: 45 DD library tests, 17 proxy integration tests, 41 Corgi unit tests, and 18 Corgi backend integration tests; one pre-existing benchmark was ignored.
Commands used the adjacent Timely checkout through a command-line Cargo patch, without dependency-file changes.