ops: fleet hardening backlog from the 2026-08-30 hub incidents

[aarontrowbridge]

Important

Problem — The 2026-08-30 hub incidents left a wake of smaller infra problems recorded only in incident debriefs, where they rot. This is the backlog, promoted to tracked work so each item graduates to its own slice when picked up.

Approach — Six known items, each with its owner surface and first move. None blocks the hub-restart path that just landed; all of them erode fleet trust.

Acceptance Criteria

• Ops sudo, scoped — the ops user gets NOPASSWD for exactly the commands hub ops need (or the root requirement is dropped); the recurring failed sudo /usr/bin/true / /boot/efi/EFI probes either work or are removed (why do they exist?).
• erlich vault sync healed — resolve the unmerged-files pull failure and the detached-HEAD mount; add a sync-health check to the fleet status collector so silent rot becomes a visible alert.
• armonissima mount on erlich — read-only and partial (27 top-level dirs vs the source's 60; suspected hopper-sync failure); re-sync and re-verify, then assert completeness in the collector.
• Distiller queue mirror — 66 pending + 195 .failed on erlich; decide drain-vs-purge, and check whether the mini-side distiller process (pid 17303 at incident time) is hung or dead.
• Tailscale path + tunnel flakiness — erlich currently relays via NYC; the MacBook tunnel agent has 64 lifetime respawns (last exit 255). Copy the mini's LAN-first Match exec fast-fallback pattern if a direct path exists; alert on respawn storms.
• Scheduled chat-DB backups — the 2.1 GB canonical DB has only ad-hoc backups; a nightly sqlite .backup (live-safe, WAL-consistent, N retained) makes every future upgrade cheap to roll back. Pair with the upgrade verb's smoke test.

[aarontrowbridge]

Three more findings from the 2026-08-30 robustness sweep, folded into this backlog:

• Binary-sha drift check in the fleet-status collector — the opencode.sha256 sidecar exists (the Aug-25 recovery used it by hand) but nothing verifies disk-binary vs sidecar periodically. A one-line collector check makes binary skew/tamper a visible alert instead of a debugging session. (Verified absent: no sha/sidecar hits in the deployed collector.)
• systemd StartLimit tuning is a deliberate decision to make — the unit carries StartLimitIntervalUSec=10s, StartLimitBurst=5, which is a no-op for a 10-second-spaced crash-loop (the incident's binary looped 29×). Either tighten (burst 5 within 10 min → hub stops after 5 failed boots, alert fires) or accept the loop explicitly. With the DB-copy smoke (feat(upgrade): boot-smoke a new server binary against a DB copy before any swap #652) and the migration guards (fix(db): guard duplicate directories migrations — drift self-heals instead of boot crash-loop opencode#272), a bad binary should never reach production; this is defense-in-depth.
• Fork branch hygiene (harmoniqs/opencode) — the default branch (local/amicode) has diverged from the release line: there is no main, release tags don't sit on the default branch, and PRs base against a moving target (observed live: a generated-client check failed on cross-era diffs until the fix branch was rebased onto the real base). Consolidate so the default branch IS the release line.