Skip to content

operations.md

Latest commit

 

History

History
303 lines (239 loc) · 18.8 KB

operations.md

File metadata and controls

303 lines (239 loc) · 18.8 KB
Raw

Operations runbook (v0.1)

Everything an operator needs to run Rust and Spark. Commands assume the repository root and a .env copied from .env.example.

1. Topology

Service What it is Notes
api / api-prod NestJS HTTP API stateless; run several behind the proxy
worker / worker-prod BullMQ mission-resolution + reconcile worker same image, node dist/worker.js; no HTTP port
web / web-prod nginx serving the SPA and proxying /v1 to the API sets X-Forwarded-For (the API trusts it from private addresses only)
PostgreSQL 16 system of record your own instance (DATABASE_URL)
Redis 7 BullMQ queues, config cache, rate-limit counters your own instance (REDIS_URL, maxmemory-policy noeviction)
migrate / migrate-prod one-shot: prisma migrate deploy + insert-only seed api and worker wait for it

Profiles: dev (everything local), test (tmpfs Postgres/Redis on 5433/6380), prod (one server: app, worker, web and the Cloudflare tunnel against your own Postgres/Redis; see docs/deploy.md).

2. Environment reference

The API refuses to boot when a required variable is missing or invalid (zod schema in apps/api/src/common/env).

Variable Required Default Meaning
NODE_ENV yes – development | test | production
PORT no 3000 HTTP port
DATABASE_URL yes – Postgres URL
REDIS_URL yes – Redis URL
CORS_ORIGINS yes – comma-separated bare origins allowed to call the API with credentials
JWT_ACCESS_SECRET yes – ≥ 32 chars: openssl rand -base64 48
COOKIE_SECRET yes – ≥ 32 chars
ARGON2_MEMORY_KIB / ARGON2_TIME_COST / ARGON2_PARALLELISM no 19456 / 2 / 1 password hashing cost (OWASP floor)
RECONCILE_INTERVAL_MS no 30000 how often the worker sweeps for stuck missions
SHIP_RATING_INTERVAL_MS no 600000 how often the worker refreshes every pilot's ship rating (the ship power and speed boards)
BUG_REPORT_TO_EMAIL no — inbox that receives beta bug reports (POST /v1/feedback/bug); sent through Resend (RESEND_API_KEY, RESEND_FROM_EMAIL). Empty = the endpoint answers 503. Limits: 3 per 10 min and 5 per day per account, 100 per day overall
SENTRY_DSN no — Sentry DSN for the API and worker: 5xx responses and failed jobs are reported (no bodies, cookies or auth headers). Empty = off. The browser's DSN is a build argument (SENTRY_WEB_DSN in compose).
API_PORT / WEB_PORT no 3000 / 8080 host ports published on 127.0.0.1

Rotating JWT_ACCESS_SECRET signs everyone out (access tokens are short-lived; refresh cookies keep working until the next refresh fails on the new secret — expect a re-login). Rotating COOKIE_SECRET invalidates refresh cookies.

Game balance is not configuration: it lives in the database and is edited in Admin → Tuning, effective immediately, with a revision history and one-click revert.

3. Deploying

# prod: external Postgres/Redis, secrets in the environment or an env file
docker compose --profile prod config >/dev/null   # validates before touching anything
docker compose --profile prod up -d --build

Order is enforced by compose: migrate-prod (schema + insert-only seed; existing tuned values are never overwritten) → api-prod + worker-prod → web-prod. Migrations are forward-only; to roll back the application, redeploy the previous image — new columns/tables are additive.

Probes (all public, no auth):

Endpoint Use
GET /v1/health/live liveness: the process answers; touches no dependency (do not restart on Postgres/Redis outages)
GET /v1/health/ready readiness: Postgres and Redis reachable — route traffic only when 200
GET /v1/health same as ready (compose healthcheck)

The worker has no HTTP port; its health is "the queue drains": watch the admin dashboard's mission counts and Redis bull:mission:* waiting/active.

4. First admin

There is no public path to an admin account. On a fresh install, create one from the API image:

# prod (compiled CLI inside the runtime image)
docker compose --profile prod run --rm api-prod \
  node dist/admin/cli/create-admin.cli.js --email you@example.com --name Captain --password '<10–128 chars>'

# dev (from the repo)
pnpm --filter api admin:create --email you@example.com --name Captain --password '<10–128 chars>'

Log in on the web app; the Admin area appears for ADMIN accounts. Password, e-mail and name follow the same rules as registration (password 10–128 chars, name 3–24). Admin accounts cannot be banned through support actions.

4.1 Emergency account tool (server shell)

For when the web UI is not an option: someone is locked out, an account registered before you had an admin, a ban needs lifting. It runs only on the server (no web endpoint, no extra password: whoever can run Docker there already holds the database), and every change is written to the admin audit log with the actor cli.

bash scripts/account.sh                      # the full list of commands
bash scripts/account.sh show --email you@example.com
bash scripts/account.sh promote --email you@example.com      # an account that already exists becomes an admin
bash scripts/account.sh demote --email you@example.com       # refuses the last admin without --force
bash scripts/account.sh ban --email someone@example.com      # blocks and signs out everywhere (not for admins)
bash scripts/account.sh unban --email someone@example.com
bash scripts/account.sh unfreeze --player guest-a1b2c3      # a frozen guest pilot (see 6.1) comes back; --email also works
bash scripts/account.sh set-password --email you@example.com --password '<10–128 chars>'   # also signs out everywhere
bash scripts/account.sh list-admins

The script runs node dist/admin/cli/account.cli.js in the api-prod image (docker compose --env-file .env.prod --profile prod run --rm api-prod ...). Emails are matched without regard to letter case. Typing a password in the command leaves it in the shell history: start the line with a space (when HISTCONTROL=ignorespace) or change the password again from the game afterwards.

5. Backup and restore

What matters, in order:

  1. PostgreSQL — the only irreplaceable state (accounts, players, ships, parts, missions, logs, tuning history, audit log).
  2. Redis — replaceable. It holds queued/delayed mission jobs, the config cache and rate-limit counters. If it is lost, restart the worker: the reconciler re-enqueues every IN_TRANSIT mission from Postgres (D4); counters and cache rebuild themselves.
# backup (logical, consistent, safe while running)
pg_dump --format=custom --no-owner "$DATABASE_URL" > rustandspark-$(date +%F).dump

# restore into an EMPTY database, then run the app (migrate-prod is a no-op when up to date)
createdb rustandspark_restore
pg_restore --no-owner --dbname "$RESTORE_URL" rustandspark-2026-01-01.dump

Take a dump before every deploy that carries a migration and on a daily schedule; keep at least 7 dailies. Test a restore into a scratch database after the first backup and monthly after: an untested backup is a hope. Because mission resolution is deterministic (D19) and each MissionLog stores its seed, rules hash and dispatch snapshot, any mission can be replayed from a restored database (Admin → Players → report → Replay).

5.1 Keeping Admin-tuned values across a DB reset

A reset (drop/recreate DB) loses everything tuned in the Admin. Save it first, re-apply after:

# 1. BEFORE the reset: export GameConfig + every entity table (read-only)
docker compose exec -T api node dist/admin/cli/tuning-snapshot.cli.js --export /tmp/tuning.json
docker compose cp api:/tmp/tuning.json ./backups/tuning-snapshot-$(date +%F).json

# 2. reset the DB, then migrate + seed as usual (seed fills defaults only)

# 3. AFTER: dry run (default) — shows creates/updates/unchanged and rows that fail validation
docker compose cp ./backups/tuning-snapshot-<date>.json api:/tmp/tuning.json
docker compose exec -T api node dist/admin/cli/tuning-snapshot.cli.js --import /tmp/tuning.json
# 4. apply (snapshot values win over seed defaults; exits 1 if any row failed)
docker compose exec -T api node dist/admin/cli/tuning-snapshot.cli.js --import /tmp/tuning.json --apply

Notes: fields the running build no longer has (e.g. a retired column) are skipped and listed under skippedFields; null values in the snapshot ("never set") are left at whatever the seed filled. Every applied row writes an audited tuning revision (actor tuning-snapshot-cli).

The snapshot also carries the faction and location art references (file names). The image files themselves are not in it: they live on the art-data volume (section 5.2), so keep that volume across a reset — or re-upload from the Admin — or the references point at nothing.

5.2 Uploaded art (art-data volume)

Faction art (banner, logo, background) and place art (wide, square, icon) are uploaded in the Admin (Tuning → Factions / Locations). Files are stored by content hash under ART_DIR (/data/art in the containers, named volume art-data; ./data/art for local runs, see .env.example) and served publicly and immutably at GET /v1/art/:file. SVGs are scanned and sandboxed on upload. Back the volume up with the database; docker compose down -v deletes it. Public lists for the web client: GET /v1/factions (authenticated: names, descriptions, colours, art) and GET /v1/places/art.

5.3 RACE missions

A RACE is a competition against 3–5 generated rival ships; the pilot needs a fast ship (entry speed). Everything is tunable under the race.* config keys (Tuning → Config): competitors_min / competitors_max, min_mobility (entry speed), reference_mob, speed_spread, time_jitter and prize_share_1..3 (prize multipliers for places 1–3). An offer keeps the entry speed it was generated with, so a change applies to offers generated afterwards. The board and transit screens show the rivals' speeds and times; the mission report shows the standings.

5.4 Reaching the game from other computers (HTTPS)

Set WEB_BIND=0.0.0.0 in .env to publish the web port on the network. The refresh-token cookie is Secure, so other computers need HTTPS: run scripts/make-dev-cert.sh <lan-ip> localhost (creates certs/: a local CA plus a server certificate, git-ignored), recreate the web container, and open https://<lan-ip>:8443 (WEB_TLS_PORT). Trust certs/ca.crt on each client computer, otherwise the browser shows a certificate warning. Without certs/server.crt nginx serves plain HTTP only.

5.5 Playtest-round tunables (Tuning → Config unless noted)

  • ship.stat_display_scale (default 10): mobility and the speeds compared with it (mission requirements, race rivals) are shown multiplied by this; game maths is unchanged.
  • ship.cruise_deficit_floor: a ship using more cruising power than it generates still flies; its engines keep generated/used of their thrust, never less than this share.
  • economy.market_rarity_by_bridge: which part rarities are on a pilot's shelf, by the rarity of their bridge. Every market listing (new or used) is one item per port per day.
  • scavenging.nothing_chance (by zone), scavenging.tier_min_zone (rare drops only in dangerous zones), scavenging.handicap_factor (a ship that cannot fly still scavenges, at this share of the chance to find anything). The drop table itself (Tuning → Drop tables, scavenging_common) is admin data: seeds do not overwrite it, so after upgrading edit it (new default 80/17/3).
  • race.mishap_chance, race.mishap_penalty, race.overdrive_speed, race.overdrive_fuel, race.overdrive_risk: how dynamic a race is, and what pushing the engines (overdrive) costs.

Flight warnings (energy shortfalls, blocked engines/weapons, no life support) no longer ground a ship: they weaken it (see the Ship sheet warnings). Only a ship that cannot move at all is refused.

5.6 Damage and energy model

  • Layers. Every hit — a weapon's or the journey's own (space, radiation, debris) — goes through the same layers in order: shield, armor, hull, and only then the parts. Armor is a pool (combat.armor_pool_factor points of absorbed damage per armor point). Parts are otherwise hurt only by a failed roll or an overload. After a run the losses are written back as condition lost (wear.hull_to_condition for the hull, armor parts for armor), so damage persists until repaired. The Hull Frame parts (the hull family) tank wear.hull_frame_share of the wear the hull's losses cause, up to their own condition; the rest passes through to the other parts alike. wear.environment_damage_factor sizes the journey's hit; wear.mining_wear_factor eases mining. Runs stored before this model (no layered flag) replay with the old per-part wear.
  • Shield. A shield part has a shieldRegen (in its special properties): points recovered each combat round. Each recovered point is paid from combat energy (the part's energyCombat, spread over its regen); no energy, no recovery.
  • Energy. The ship's own spare power (cruising power above zero) pays for combat first; batteries cover only the rest, so a ship with enough generation needs no battery. The batteries-only mode keeps the spare power out of combat. Solar panels give a tenth of what they used to, and most parts now draw cruising power.
  • Tanks. Small 1×1 holds 300, medium 2×1 600, large 3×1 900 (epic/legendary tiers 1200/1500); fuel never exceeds what the working tanks hold.

6. Operating the game

Situation Action
Something is wrong and players should stop acting Admin → System → turn maintenance on: intents answer 503 while reads and admin keep working
Close signups (attack, capacity) Admin → System → register.open off; takes effect on the next request
Tell players something Admin → System → broadcast (English and pt-BR both required); shown in-game within a minute
Balance change Admin → Tuning; every change has a reason and a revision; revert from the history
Player stuck / wrong balance / abuse Admin → Players → support actions (each needs a reason and is audit-logged)
Who did what Admin audit log; player sheet views are logged too

Rate limits (per account when authenticated, otherwise per IP; counters in Redis so they hold across instances): reads 300/min, intents 120/min, previews 240/min, login 5/min per IP+e-mail, register 10/min per IP, new guest 10/min per IP, guest sign-back-in 30/min per IP. A 429 carries Retry-After. If Redis is down the limiter fails open.

6.1 Guest pilots

"Play as guest" starts a pilot with no email and no password. The browser keeps a random key in its localStorage (rs.guestKey); the server stores only its hash, and the key signs the guest back in (POST /v1/auth/guest/resume). Clearing the browser's data or changing device loses the pilot unless it was saved: the account menu offers "Save your pilot" (an email, a password and an optional pilot name), which turns the guest into a normal account in place.

Setting (Admin → Tuning → admin) Default What it does
admin.guest_daily_cap 300 New guests per day (UTC). 0 turns "play as guest" off. The per-IP limit (10/min) always applies
admin.guest_freeze_days 30 A guest nobody saved, idle this many days, is frozen by a worker sweep (every 6 h)
  • A closed registration (register.open off) closes guest play too.
  • Guests are named guest- plus 6 letters or digits, are not on the rankings and have no public card until saved.
  • A frozen guest keeps all its data but cannot sign in (GUEST_FROZEN). Bring one back with bash scripts/account.sh unfreeze --player guest-xxxxxx: it counts as active again, so the next sweep leaves it alone.
  • Nothing is ever deleted by this feature.

7. Graceful shutdown drill (manual)

The worker finishes the job in hand on SIGTERM (BullMQ's shutdown hook); the API drains open requests. Run this before each release:

  1. docker compose --profile prod up -d, register a player, accept and dispatch a mission with a long duration (Admin → Tuning → missions.time_scale low for the drill).
  2. While the mission is IN_TRANSIT: docker compose --profile prod stop worker-prod (sends SIGTERM, default 10 s grace). The worker log must show a clean shutdown, no unhandledRejection.
  3. docker compose --profile prod start worker-prod. Within RECONCILE_INTERVAL_MS the mission must resolve exactly once: one MissionLog, one payout event.
  4. Repeat with stop api-prod while issuing requests in a loop: in-flight requests complete, new ones fail fast, no 5xx from a half-closed connection pool.
  5. Kill Redis (docker compose stop redis) for 30 s while sending requests: reads and intents that need Redis fail with a clean error, /v1/health/live stays 200, /v1/health/ready goes 503, and everything recovers when Redis returns without a restart.

8. Troubleshooting

Symptom Look at
API exits at boot its log names the invalid env variable
Everyone gets 429 a proxy in front of nginx that is not private-network (untrusted X-Forwarded-For ⇒ all clients share the proxy's IP)
Missions never resolve worker running? docker compose logs worker-prod; Redis reachable; reconcile interval
A mission report looks wrong Admin → Players → report → Replay: matchesStored false means the engine no longer reproduces the stored run
Banned player still plays at most ~5 s on other API instances (status cache); immediate on the instance that handled the ban

9. Keeping the CI green (standing rules)

Every push to develop must end with the aggregate CI job green (quality, unit, web, integration with the coverage thresholds and the Prisma migration diff, compose-check, docker-lifecycle, security, oracle-fixtures-fresh).

  1. The browser smoke does not gate the aggregate (continue-on-error, see the comment in .github/workflows/ci.yml), so a red smoke is a real regression that goes unnoticed (it was red for four days in October after a dialogue change): read it on every run. Run it locally before pushing a UI change (recipe in the repo's e2e notes: a separate compose project on alternate ports and a fresh database; the race spec needs a fresh database).
  2. Rapid pushes cancel each other (cancel-in-progress): confirm one full run on the head before calling anything done.
  3. A CI-only failure is reproduced with a full local test:cov (cross-suite state), after checking for a stray jest process; the soak test prints the rejection reasons when it fails.
  4. Never edit baselines.json by hand (regenerate it with tools/oracle/gen_fixtures.py).
  5. A change to a config default breaks the literals in the e2e specs too: grep the old value in test, and run test:e2e as well.