Skip to content

feat(upgrade): boot-smoke a new server binary against a DB copy before any swap #652

Description

@aarontrowbridge

Important

Problem — The 2026-08-30 hub upgrade crash-looped 29 times against the production database because a new server binary was validated by booting it AT production: the binary carried a pending migration whose ALTER collided with drifted-but-legal database state (duplicate column name: directories). Nothing in the upgrade path stands between "built" and "serving production" — a migration that dies on the real DB is discovered by crash-looping the hub.

Approach — Make the upgrade verb's contract: build → boot-smoke against a copy of the production DB (consistent sqlite .backup snapshot, isolated XDG data dir, expect serving + migrations reconciled) → stage → atomic swap → restart → verify. The smoke harness already exists as an ad-hoc script from the incident; promote it into the verb. The hub-side stage/swap/restart/verify primitives land separately via the hub-restart ops script and are adopted as-is.

Scope — in: DB-copy smoke step in the upgrade verb, isolated-data-dir harness, honest refusal when the smoke fails · out: migration content fixes (landed separately in the fork), the client-side restart command.

Assumptions — the upgrade verb is currently gated on the pending server-binary PR; the smoke reuses the snapshot technique proven in the incident (.backup API, ~1.4 s on a 2.1 GB DB).

Acceptance Criteria

  • Upgrading a binary whose migrations would die against the current DB fails BEFORE any swap, with the migration id and the error in the refusal.
  • A clean binary passes the smoke (serves on a scratch port against the copy; journal row count >= pre-copy count) and the upgrade proceeds.
  • The smoke never touches the production DB (read-only snapshot via the backup API; the copy lives in a temp dir with an isolated data-dir env).
  • The refusal message names the recovery: fix the migration, don't retry the swap.

Metadata

Metadata

Assignees

No one assigned

    Labels

    hitlNeeds human decision or review

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions