Skip to content

feat(deploy): add HAProxy blue/green stack for zero-downtime relay updates - #777

Open
Ferryx349 wants to merge 1 commit into
mainfrom
feat/haproxy-blue-green
Open

Ferryx349 wants to merge 1 commit into
mainfrom
feat/haproxy-blue-green

Conversation

@Ferryx349

@Ferryx349 Ferryx349 commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Description

This pr adds an alternative production topology: two relay containers (nostream-blue,nostream-green) behind HAProxy on 127.0.0.1:8008, and a script that replaces them one at a time.

  • deploy/haproxy/haproxy.cfg :- round-robin across both relays, /readyz health checks, and option redispatch so a request that fails on a dying backend is retried on the other one rather than returned to the client.
    A resolvers docker block re-resolves backend names against Docker's embedded DNS every 2s, so a recreated container's new IP is picked up; init-addr libc,none lets HAProxy boot before the relays are resolvable.
  • deploy/docker-compose.haproxy.yml :- HAProxy plus the two relays, sharing one YAML anchor. Postgres, Redis, and the migrate job are unchanged from docker-compose.prod.yml. Relays get stop_grace_period: 45s so the
    WS_DRAIN_TIMEOUT_MS drain finishes before Docker escalates to SIGKILL.
  • deploy/rolling-relay-recreate.sh :- stops one relay, waits for its replacement to report healthy via up --wait, then moves to the second.
  • deploy/README.md — install and update procedure.docker-compose.prod.yml and the single-relay flow are untouched; operators opt in by using the new compose file.

Related Issue

Closes:- #776 and Relates to #773

Motivation and Context

How Has This Been Tested?

  • docker compose -f deploy/docker-compose.haproxy.yml config parses, and both relays correctly inherit depends_on, healthcheck, and stop_grace_period from the shared anchor
  • haproxy -c -f haproxy.cfg against haproxy:3.0-alpine exits 0. It emits two [NOTICE] lines about unresolvable backends when run outside the Compose network, which is the expected init-addr libc,none path — HAProxy parks the
    servers and boots instead of aborting
  • Not yet exercised against a live stack; needs a sandbox run of rolling-relay-recreate.sh with a WebSocket client connected to confirm no dropped requests during cutover.

Screenshots (if appropriate):

Types of changes

  • Non-functional change (docs, style, minor refactor)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my code changes.
  • I added a changeset, or this is docs-only and I added an empty changeset.
  • All new and existing tests passed.

Two relays behind HAProxy with /readyz health checks, DNS re-resolution for
recreated containers, and option redispatch. Adds a rolling recreate script
that replaces one relay at a time so a ready backend always serves traffic.
@changeset-bot

changeset-bot Bot commented Sep 15, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 967d4af

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
nostream Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant