Skip to content

knowledge: page when the dataops-knowledge S3 backup stops capturing HEAD - #485

Merged
alexeygrigorev merged 1 commit into
mainfrom
feat/backup-freshness-alert
Nov 18, 2026
Merged

alexeygrigorev merged 1 commit into
mainfrom
feat/backup-freshness-alert

Conversation

@alexeygrigorev

Copy link
Copy Markdown
Member

Problem

Hub intake 83a336d1: no alert for the 44-day stale knowledge repo backup. The backup-to-s3.yml workflow in dataops-knowledge (daily 03:17 UTC zip → S3) ran wholly in GitHub Actions with direct S3 uploads, so the AWS side never saw it. When its schedule silently died, nothing ran from 2026-10-02 to 2026-11-15 (44 days, confirmed via gh run list), no metric existed, and no alarm could fire.

Fix

  • Freshness check (backend/src/knowledge/repoBackupFreshness.ts): the knowledge worker gains a daily knowledge-repo-backup-check schedule (05:30 UTC). It reads {prefix}/latest/manifest.json from the backups bucket and emits DataOps/Knowledge Job=knowledge-repo-backup Success=1/0.
    • Fresh while the manifest's commit_sha equals the last mirror-reported repository HEAD, or the manifest landed within 48h.
    • The mirror heartbeat route now persists the reported HEAD to jobs/knowledge-mirror/last-report.json so the check can tell a quiet repository (current backup, stays green for weeks) from a broken backup (pages as soon as an unbacked commit ages out).
  • Alarm dataops-v1-knowledge-repo-backup-stale in infra/template.full.yaml: dead-man check mirroring KnowledgeBackupStaleAlarm (2-day sum < 1, missing data breaching), paging the shared operator topic, with OK recovery.
  • Evidence: sandbox-backup-evidence.yml now proves the new alarm exists with enabled actions and reaches a confirmed SNS subscriber.
  • Registry: the knowledge-repo-backup automation description mentions the freshness check.

Tests

  • backend/tests/knowledge-repo-backup-freshness.test.ts: freshness semantics (commit match, 48h boundary, divergent/missing manifest, no-report fallback), EMF record shape, S3 read path, unconfigured-bucket failure.
  • Publisher auth test asserts the heartbeat persists the last report.
  • Full unit suite green locally (249 files); make sam-validate passes.

…HEAD

The repository backup (dataops-knowledge backup-to-s3.yml) runs wholly in
GitHub Actions and uploads straight to the private backups bucket, so the
2026-10-02 -> 2026-11-15 scheduling outage ran 44 days with no run, no
metric and no alert - its own success even covers skip runs that upload
nothing because HEAD did not change.

The knowledge worker now runs a daily freshness check against the backup
manifest in S3: fresh while its commit_sha matches the last mirror-reported
repository HEAD (the heartbeat route now persists it to
jobs/knowledge-mirror/last-report.json) or while it landed within two days,
else Success=0 for the new KnowledgeRepoBackupStaleAlarm dead-man check on
the operator topic. A quiet repository with a current backup stays green;
only commits that never reach the backup page. The evidence workflow proves
the new alarm exists and reaches a confirmed subscriber.
@alexeygrigorev
alexeygrigorev merged commit 57d62c2 into main Nov 18, 2026
7 checks passed
@alexeygrigorev
alexeygrigorev deleted the feat/backup-freshness-alert branch November 18, 2026 15:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant