Conversation
A graceful ALTER CLUSTER reconfiguration cut over to its pending replicas as soon as every collection reported hydrated on them. Hydration is `as_of < output_frontier`, and the as-of is pinned to the collection's read frontier when the replica is added. A collection whose initial snapshot takes hours therefore reports hydrated the instant the snapshot lands, with everything since the as-of still to replay. The outgoing replicas keep the cluster's frontiers current until the cut-over, so the lag is invisible; at the cut-over they are dropped and the frontiers stall until the new replicas catch up. On 2026-08-28 a production cluster cut over after a 4h08m reconfiguration. The last collection had crossed an as-of from the start of that window seconds earlier, and the cluster's frontiers stalled for roughly 15 minutes while the new replica replayed the backlog. The cluster-autoscaling design chose hydration-only deliberately, calling the stronger check "a possible later refinement" that "would only avoid a brief post-cut-over latency bump". The bump is not brief: it scales with hydration time. Gate the cut-over on readiness instead. A pending replica is ready when every collection is hydrated on it and its output frontier for the collection is within `cluster_reconfiguration_allowed_lag` (60s) of the furthest output frontier among the reference replicas: the ones the cut-over will drop, which is the realized-shape set. The strategy names that set, since only it knows what a cut-over retires. A replica that survives the cut-over, such as a hydration-burst replica at a larger size, is not part of the reference: it cannot regress the cluster's frontier, so it must not raise the bar. `enable_cluster_reconfiguration_lag_gate` is the break-glass back to hydration alone. Output frontiers, not write frontiers: a materialized view's replica-reported write frontier is the persist shard upper, shared by every replica writing it, and for a REFRESH materialized view it jumps to the next refresh time. The output frontier is the meet of the write frontier and the dataflow's compute probe, so it is the replica's own progress. For indexes the two coincide. The hydration burst keeps its hydration-only signal: its durable `steady_hydrated_at` field means exactly "hydration was observed", so the reconfiguration strategy gets a separate `ready_replicas` signal rather than a redefined `hydrated_replicas`. The storage-side check is unchanged and hydration-only. Timeout semantics are unchanged: ON TIMEOUT = COMMIT still cuts over at the deadline once the target set exists and is recorded as forced. That is also how a downsize to a size that hydrates but cannot sustain the input rate, which the hydration-only gate allowed to cut over into a lagging state, now expresses that intent. The lag predicate `reference <= frontier + allowed_lag` is extracted to `mz_repr::frontier_within_lag`, replacing the two open-coded copies in the 0dt caught-up check, and saturates rather than panicking on overflow. The per-collection decision is a pure function, `classify_collection_readiness`, tested over plain frontiers. The readiness probe logs "unhydrated" and "lagging" separately; in the incident the two were indistinguishable. Linear: https://linear.app/materializeinc/issue/SQL-672 Co-Authored-By: Claude Code <noreply@anthropic.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a0b627-f434-764d-9905-738ca01fca4d Co-authored-by: Amp <amp@ampcode.com>
…ness Amp-Thread-ID: https://ampcode.com/threads/T-01a0b627-f434-764d-9905-738ca01fca4d Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a0b627-f434-764d-9905-738ca01fca4d Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a0b627-f434-764d-9905-738ca01fca4d Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a0b627-f434-764d-9905-738ca01fca4d Co-authored-by: Amp <amp@ampcode.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Refs SQL-672. Builds on #38615, retaining the original commit and attribution, with separate follow-up commits.
When enabled, graceful reconfiguration waits for the required replacement replicas to hydrate and catch up to the outgoing replicas, using per-replica output progress. The default allowance is 60 seconds. The gate defaults off in production and on in mzcompose, sqllogictest, and the Rust integration-test harness.
ON TIMEOUT = COMMITremains an escape hatch. Source ingestion and hydration-burst retirement remain hydration-only.Follow-ups to the earlier draft
The new system parameters still need LaunchDarkly registration before rollout, as noted in #38615. Hydration-burst retirement needs a separate policy change.
Docs review: https://preview.materialize.com/materialize/38940/sql/alter-cluster/#resizing-process
Verification coverage
Regression tests cover the completed-frontier bypass and reject substituting write progress for output progress. Controller tests cover retaining outgoing replicas while targets are hydrated but lagging, then cutting over once ready.