adapter: skip shardless builtins in every migration plan, not only forced ones

[bosconi]

Motivation

Since #38864 every 0dt upgrade from a catalog written before v26.41.0 to main crash-loops the new environmentd in read-only mode (SQL-713):

environmentd: fatal: cannot migrate builtin schemas from version 26.19.0 to version 26.43.0-dev.0:
missing shard ID for builtin SystemObjectDescription { schema_name: "mz_internal", object_type: Table, object_name: "mz_replica_hydration_history" }

Six jobs on nightly#18324: all four Postgres CDC (before source versioning, multi-version upgrade, ...) shards (from 0.147.0 and 26.19.0) and both Checks Self-Managed earliest to latest direct upgrade shards (from 0.147.0). The same jobs passed on #18320, the previous main nightly.

#38864 registers MigrationStep::evolution("26.43.0-dev.0", ..., "mz_replica_hydration_history"). That table first shipped in v26.41.0, so a source catalog older than that has no shard for it. plan_migration keeps every step above the source version, the plan reaches migrate_evolve_one, the shard lookup finds nothing, and the read-only branch bails. In leader mode the same case returns Ok(()), which is why the in-place Checks ... upgrade scenarios pass on the same build and only the 0dt-preflight jobs fail.

#36668 fixed this exact bail for the forced dev-to-dev plan by filtering shardless objects out of plan_forced_migration: "there is nothing to evolve or replace for an object that does not yet exist in persist." The versioned plan never got the filter, so any evolution step for a builtin younger than the source version reproduces it. #38788's step for mz_cluster_replica_metrics_history did not, only because that table predates every supported source version.

Description

Move the filter to the one place both plans pass through. Migration::run now calls drop_shardless on the selected plan, forced or versioned, before executing it; plan_forced_migration loses its private copy of the same filter. A builtin with no registered shard does not exist in persist yet, whether it was added in the target version or the source version predates it entirely, and in both cases the leader creates it at the target schema during bootstrap. The read-only bail in migrate_evolve_one stays as the fail-fast for anything that still reaches it.

This keeps the semantics #36668 chose rather than adding a new one: the read-only race the bail guards against, a leader registering the shard while a read-only process evolves it, cannot involve a builtin the source version never had.

Verification

New turmoil test test_evolution_skips_builtin_without_shard: one builtin table with a shard and a stale fingerprint (the pre-evolution schema), one without a shard at the current fingerprint (the shape catalog open produces for a builtin new to the durable catalog), steps at 0.2.0 and 0.3.0 over a 0.1.0 source, run once read-only and once as leader. Both runs must succeed, evolve the first table, and leave the second out of new_fingerprints.

• Against main's builtin_schema_migration.rs the read-only run fails with missing shard ID for builtin SystemObjectDescription { schema_name: "schema", object_type: Table, object_name: "new_table" } (s2), the production message.
• With this change both runs pass, and the existing randomized test_builtin_schema_migration harness passes.

Each turmoil test was run in its own process (cargo test -p mz-adapter --lib -- <full path> --exact), which is what nextest does in CI. Running the module's tests together in one process fails independently of this change: configure_tracing_for_turmoil installs a global subscriber under a Once, and when hydration_history_forced_migration_policy (#[mz_ore::test]) runs first the install fails and poisons the Once for every turmoil test after it. Pre-existing, not addressed here.

cargo fmt clean; cargo clippy -p mz-adapter --tests -D warnings clean.

🤖 Generated with Claude Code

[bosconi]

The in-process test-isolation failure mentioned under Verification (configure_tracing_for_turmoil unwrapping set_global_default after an #[mz_ore::test] in the same binary has installed a subscriber) is fixed separately in #38920. With it, cargo test -p mz-adapter --lib -- builtin_schema_migration passes in one process.

[def-]

QA LLM Review (Post Merge)

@bosconi — an automated review of commit d0e0e2d275 found the following potential MEDIUM+ issue(s) after this PR was merged.

1. MEDIUM -- dropping shardless objects from the plan also drops them from the fingerprint update, turning a tolerated state into a panic

src/adapter/src/catalog/open/builtin_schema_migration.rs:777

drop_shardless removes the object from plan, and migrated_objects is built from plan (L798-799), so the object no longer counts as migrated in update_fingerprints. In leader mode that is a behaviour change: a shardless object used to stay in the plan, migrate_evolve_one returned Ok(()) for it (L1002-1010), and its durable fingerprint still got refreshed. Now, if such an object's durable fingerprint is stale, update_fingerprints falls through to panic!("fingerprint mismatch for builtin ...") at L1336 and environmentd aborts at catalog open.

Details

The change is safe only under the unstated invariant no shard implies the durable fingerprint is already current. That holds for a builtin whose mapping this very open just created (add_new_remove_old_builtin_items_migration, src/adapter/src/catalog/open.rs:846-857, stamps the current fingerprint), which is the SQL-713 case the commit targets. It does not hold for a builtin whose mapping was committed at an earlier version but whose shard was never registered, and the code creates exactly that window on every upgrade that adds a builtin collection: Catalog::open commits the new mapping together with the bumped migration version (open.rs:549-551), while shards are allocated by initialize_storage_state in a separate durable transaction later in bootstrap (open.rs:693-715, via StorageCollections::initialize_state). A process that dies between those two commits leaves mapping without shard durable at version X. Rebooting at X heals it (the migration short-circuits on source_version == target_version), but booting at Y > X with an Evolution/Replacement step for that builtin now panics where it previously completed and rewrote the fingerprint. That makes rolling forward out of a bootstrap crash-loop a hard abort rather than a recovery.

The new test cannot catch this: new_object's durable fingerprint is new_builtin.fingerprint(), i.e. already current, so update_fingerprints hits the equality continue (L1310-1313) whether or not the object is in migrated_objects. Both the !fingerprints.contains_key(&new_object) assertion and the whole leader half pass against main as well; only the read-only half is load-bearing.

Suggested fix: keep the skipped objects in the set handed to update_fingerprints. They need no persist work, but their fingerprint must still advance, because the leader will create their shards at the target schema during bootstrap. It is a no-op for genuinely new builtins (their fingerprint already matches) and it also closes the same hole on the forced path, which the participates_in_forced_migration comment already calls out ("a table left out of the plan never gets its new schema registered").

-        let plan = self.drop_shardless(plan);
+        let (plan, shardless) = self.split_shardless(plan);
...
         let mut migrated_objects = BTreeSet::new();
         migrated_objects.extend(plan.evolve);
         migrated_objects.extend(plan.replace);
+        // Nothing to evolve or replace in persist for these, but the leader creates their shards
+        // at the target schema during bootstrap, so their fingerprints must still advance.
+        migrated_objects.extend(shardless);