storage: group a pinned snapshot's updates into one batch - #38935
Draft
martykulma wants to merge 1 commit into
Draft
martykulma wants to merge 1 commit into
martykulma wants to merge 1 commit into
Conversation
martykulma
added this pull request to stack #38937
September 19, 2026 00:30
martykulma
changed the title
While an export snapshots, its frontier is pinned at the single time the snapshot occupies, so the minter derives no description and every update reaches write_batches with no bounds to group under, writing a batch of its own timestamp. The minter now commits a ceiling a fixed lookahead past the largest timestamp the data has reached and broadcasts it on a second output, and write_batches takes every update below it into one open builder whatever its timestamp. A ceiling is not a description. It carries the lower of the description that will retire it, since a builder declares its lower when it opens, and it binds the minter to mint nothing below it. So the whole snapshot and the catch-up behind it become one description, appended once rather than once per boundary. The cost is the tail: the shard upper waits for the frontier to reach the ceiling, which is about one lookahead however long the snapshot ran. An update that outruns the ceiling has no bound to group under and writes a batch of its own timestamp, which is also what every update does when nothing is committed ahead of the frontier. Both kinds are finished under the description that comes to cover them, on its arrival or at its readiness. max_seen_timestamps reports each worker's largest timestamp to the minting worker, since the collection is pre-sharded and the minter would otherwise pace on its own share alone. Committing is confined to an export that snapshots in this incarnation, and only while the frontier sits at the time its snapshot occupies. A collection that is keeping up lags the data by about one timestamp and has nothing to group, so it would pay the tail for nothing. CdcV2 exports are excluded, since their MZ times come from the data rather than from reclocking and no wall-clock lookahead relates to them. Defaults off in production and on in the mzcompose test configuration so the path is exercised. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
martykulma
force-pushed
the
maz-sink-group-snapshot
branch
from
September 19, 2026 01:00
fc77c10 to
05deedd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
While an export snapshots its frontier is pinned, so no description gets minted
and every update writes a batch of its own timestamp.
The minter now commits a ceiling a fixed lookahead past the newest data and
broadcasts it on a second output.
write_batchesputs everything below it intoone builder, whatever the timestamp.
A ceiling isn't a description. It carries the lower of the description that'll
retire it, since a builder declares its lower when it opens, and it binds the
minter to mint nothing below it. So the snapshot plus its catch-up is one
description, one append. The cost is the tail, the shard upper waits for the
frontier to reach the ceiling, about one lookahead however long the snapshot ran.
Rows that outrun the ceiling write their own batch, same as everything does with
the flag off. Both kinds get finished under whatever description covers them.
max_seen_timestampsreports each worker's largest timestamp to the minter,since the collection is pre-sharded and the minter would otherwise pace on its
own share.
Only for exports snapshotting in this incarnation, and only while the frontier
sits at the snapshot's time. CdcV2 is excluded, its MZ times come from the data
so a wall-clock lookahead means nothing there.
storage_persist_sink_description_lookaheaddefaults to 0, off in production,5s in mzcompose.
Tests:
write_batches_routes_updates_below_the_ceiling_into_one_builder, plusunit tests for
next_mintanddescription_lookahead.🤖 Generated with Claude Code