design: one ceiling and one batch for a pinned snapshot - #38932
Draft
martykulma wants to merge 1 commit into
Draft
martykulma wants to merge 1 commit into
martykulma wants to merge 1 commit into
Conversation
Step 4 is rewritten for a bound the minter commits and broadcasts to the writers rather than mints as a description. The writers group every update below it into one builder, and the minter mints nothing below an outstanding ceiling, so the whole snapshot and the catch-up behind it become one description appended once, whatever the snapshot's length. The lookahead is fixed rather than a width that doubles, which puts the tail at about one lookahead instead of growing with the snapshot's duration. That closes the tail open question, and it makes restart case 2 unreachable, since the shard upper now sits at T:c until the one description goes in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
martykulma
added this pull request to stack #38937
September 19, 2026 00:30
martykulma
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A snapshotting export pins its frontier at one time, so nothing derives a batch
description and the sink has no bounds to group under.
So the minter commits a ceiling a lookahead past the newest data and broadcasts
it to the writers. Not a description: it carries the lower of the description
that'll retire it, and it binds the minter to mint nothing below it. Writers put
everything below the ceiling into one builder, whatever the timestamp, so
snapshot rows and replication rows share a batch.
Net effect is the snapshot plus its catch-up is one description, one append. The
cost is the tail, the shard upper waits for the frontier to reach the ceiling,
about one lookahead no matter how long the snapshot ran.
Rows that outrun the ceiling write their own batch, same as everything does with
the flag off.
🤖 Generated with Claude Code