SQL-150: durable temporary objects - #37878
Closed
SangJunBak wants to merge 5 commits into
Closed
Conversation
SangJunBak
force-pushed
the
jun/move-temp-to-catalog-split
branch
3 times, most recently
from
July 27, 2026 21:10
113a967 to
de93fc8
Compare
SangJunBak
changed the title
SangJunBak
force-pushed
the
jun/move-temp-to-catalog-split
branch
7 times, most recently
from
August 7, 2026 21:43
91a42bb to
6d2cdbb
Compare
Initially the design doc stated that we wanted to persist sessions in the Catalog. Through benchmarking, it was found to create majoir regressions on CPS, even with optimizations. Thus we keep mz_sessions as a builtin table but will continue to persist durable objects. We also change the requirement of temporary schemas being durable to being a followup given we don't need them to move builtin tables.
Generic catalog migration version bump. Copies everything and is intended to make the review easier.
- We add the ephemeral_owner_session property in durable Items to indicate if an object is temporary - We remove all ephemeral items in savepoint/write open of the catalog. We can't do this for readonly mode since readonly followers would remove it in memory but then eventually panic when it sees a retraction for it
- Temporary items now write real durable Item rows, marked with ephemeral_owner_session = the creating session's UUID - The duplicated TemporaryItem is consolidated - Replaces catalog entries' `From` implementation with durable_item since we now need access to the connection <-> session mapping. - We introduce two map state variables: ephemeral_owner_conns_by_uuid and ephemeral_owner_uuids_by_conn. Both serve to create a session UUID <-> Conneciton ID mapping. These are used for two purposes: - When resolving a temporary object, routing it to its temporary schema via the `connection ID <-> Temporary schema mapping` using`temporary_schemas` - When editing an object, as opposed to creating an object, to store the new object durably, we don't have access to the session UUID but have access to the connection ID. Thus we use these maps to grab it. - In a future commit, we'll be merging temporary_schemas with these map variables since all three have the same lifecycle and are all used together
- The lifecycle and purpose of our `ephemeral_*` map state variables are very similar to `temporary_schemas`. That is, all lazily initialize in-memory temporary item metadata to resolve temporary items. Thus we unify all three in a struct `TemporaryNamespaces` - Gets rid of extraneous lazy initialization of the mz_temp schema inside `apply_item_update` from before given the lazy initialization in the inner `insert_entry` is all we need - Gets rid of eager mz_system temporary schema initialization.
SangJunBak
force-pushed
the
jun/move-temp-to-catalog-split
branch
from
August 10, 2026 22:01
6d2cdbb to
7b83fef
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Design doc for making temporary tables and views durable in the catalog
shard, so that mz_tables and mz_views can become materialized views over
mz_internal.mz_catalog_raw and temporary state survives in a multi-envd
world. Records why sessions stay in the mz_sessions builtin table
instead of becoming durable catalog records (connect latency coupling to
the catalog compare-and-append, churn contention with DDL on the
catalog single writer), and distills that lesson into the adapter guide.Remove these sections if your commit already has a good description!
Motivation
Why does this change exist? Link to a GitHub issue, design doc, Slack
thread, or explain the problem in a sentence or two. A reviewer who has
no context should understand why after reading this section.
If this implements or addresses an existing issue, it's enough to link to that:
Closes
Fixes
etc.
Description
What does this PR actually do? Focus on the approach and any non-obvious
decisions. The diff shows the code --- use this space to explain what the
diff can't tell a reviewer.
Verification
Nightly: https://buildkite.com/materialize/nightly/builds/17699