Skip to content

Run migrations before deploying brainstore and service pods - #128

Merged
Kevin Green (KevinGreen) merged 15 commits into
mainfrom
kevingreen/inf-869-helm-migration-job
Nov 17, 2026
Merged

Kevin Green (KevinGreen) merged 15 commits into
mainfrom
kevingreen/inf-869-helm-migration-job

Conversation

@KevinGreen

@KevinGreen Kevin Green (KevinGreen) commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Order changed workloads as migrations → Brainstore → API pools, AI Gateway, and Loop Runtime. deploymentOrdering.enabled (default true) controls the whole sequence; unchanged workloads keep running.

  • Content-address migration Jobs by their image and execution configuration. Reuse successful Jobs and immutable configuration snapshots across upgrades and rollbacks; a no-op upgrade creates no Job or pod rollout.
  • Read desired Brainstore identities from a stable ConfigMap without embedding dependency identities in application pod templates. Require the expected template, observed generation, compatible images, and complete readiness; old or terminating backends cannot satisfy a changed rollout.
  • Run migrations only from api.image.repository, at the newest API pool or Brainstore release. Opaque pins require an explicit migrationImageTag and stable minimumVersion assertion. The override accepts a tag or digest, not a repository.
  • Include every rendered Brainstore role, including automation writers and draining zero-replica roles. Preserve the legacy gate ConfigMap unchanged for old ReplicaSet replacement pods. Gateway has its own configurable ServiceAccount.

This gates container startup, not Kubernetes object updates. The removed api.brainstoreStartupGate setting is ignored; existing opt-outs must set deploymentOrdering.enabled=false. See README configuration and lifecycle.

Input coverage

Input Behavior
Stable tags, with/without v, digest pins Select the newest migration release in api.image.repository; preserve digests only from that repository.
SHA pins, numeric SHAs, prerelease/custom tags Require explicit migration tag or digest and stable floor; opaque Brainstore pods must match desired image and declared version.
Migration override Accept tags, tag@digest, or @digest; reject repository references and invalid types. Missing/null/empty select automatically; numeric tags remain explicit.
Mixed service versions / isolated API pools Include all rendered pools; reject conflicting known majors and known-too-old migration bundles.
No-op / dependency-only changes Stable Job identity and workload templates; only changed templates roll, scale-only changes do not restart pods.
Ordering on/off; missing, partial, null/reused values Keep supported defaults; disabled mode omits gates and ignores unused validation.
Existing legacy gate / fresh install Preserve existing data; do not create a legacy ConfigMap on fresh installs.
Zero replicas / stale or terminating pods Require complete drain; stale templates, generations, migration state or readiness block startup.
AWS, Azure, Google, minimal; Google examples Render with ordering enabled and disabled.

Validation

  • 87 Python tests and 639 Helm cases passed. Provider/example rendering and strict Helm lint passed.
  • Temporary revision-7/revision-8 checks proved identical pod templates for all nine enabled deployments and identical migration Job execution/name (20 cases); the fixture was removed.
  • Production render comparisons proved reader-only, writer-only, API-only, migration-only, scale-only, and Brainstore-image changes affect only the intended workload templates and migration inputs. Identical inputs produced identical resources.
  • The actual rollout checker accepted completed rendered templates and rejected old healthy deployments before expected config/scale changes reached them. Opaque API, Brainstore, and numeric pins also passed.
  • AWS/Azure/Google/minimal profiles and all five Google examples rendered with ordering on/off; the autoscaling example used its required API capability flag.
  • GCP sandbox verified migration/Brainstore/API startup order, successful no-op upgrades, reader-only and API environment-only rollouts, and failure blocking. Original configuration was restored and API health checks returned HTTP 200. Gateway and Loop were not deployed. Migration naming and tag/digest override changes were verified offline only.

Resolves INF-869.

@KevinGreen Kevin Green (KevinGreen) changed the title Run Helm migrations before starting new deployment pods Run migrations and complete Brainstore rollouts before API startup Nov 11, 2026
@KevinGreen Kevin Green (KevinGreen) changed the title Run migrations and complete Brainstore rollouts before API startup Run migrations then directly verify Brainstore rollout before API startup Nov 12, 2026
@KevinGreen Kevin Green (KevinGreen) changed the title Run migrations then directly verify Brainstore rollout before API startup Run migrations before deploying brainstore and API pods Nov 12, 2026
@KevinGreen Kevin Green (KevinGreen) changed the title Run migrations before deploying brainstore and API pods Run migrations before deploying brainstore and service pods Nov 12, 2026
@KevinGreen
Kevin Green (KevinGreen) marked this pull request as ready for review November 12, 2026 22:13
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-12T22:15:43.328716Z d460dc9 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d460dc908b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

Comment thread braintrust/templates/migration-job.yaml Resolved
Comment thread braintrust/templates/_helpers.tpl Resolved
Comment thread braintrust/values.yaml Resolved
- Replace the full image override with deploymentOrdering.migrationImageTag.
- Build automatic and explicit migration images from api.image.repository.
- Validate tag and digest overrides and preserve unchanged Job identity.
@KevinGreen
Kevin Green (KevinGreen) merged commit 4cb8005 into main Nov 17, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants