Repository navigation
knowledge: page when the dataops-knowledge S3 backup stops capturing HEAD - #485
Merged
Merged
Conversation
…HEAD The repository backup (dataops-knowledge backup-to-s3.yml) runs wholly in GitHub Actions and uploads straight to the private backups bucket, so the 2026-10-02 -> 2026-11-15 scheduling outage ran 44 days with no run, no metric and no alert - its own success even covers skip runs that upload nothing because HEAD did not change. The knowledge worker now runs a daily freshness check against the backup manifest in S3: fresh while its commit_sha matches the last mirror-reported repository HEAD (the heartbeat route now persists it to jobs/knowledge-mirror/last-report.json) or while it landed within two days, else Success=0 for the new KnowledgeRepoBackupStaleAlarm dead-man check on the operator topic. A quiet repository with a current backup stays green; only commits that never reach the backup page. The evidence workflow proves the new alarm exists and reaches a confirmed subscriber.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Hub intake 83a336d1: no alert for the 44-day stale knowledge repo backup. The
backup-to-s3.ymlworkflow in dataops-knowledge (daily 03:17 UTC zip → S3) ran wholly in GitHub Actions with direct S3 uploads, so the AWS side never saw it. When its schedule silently died, nothing ran from 2026-10-02 to 2026-11-15 (44 days, confirmed viagh run list), no metric existed, and no alarm could fire.Fix
backend/src/knowledge/repoBackupFreshness.ts): the knowledge worker gains a dailyknowledge-repo-backup-checkschedule (05:30 UTC). It reads{prefix}/latest/manifest.jsonfrom the backups bucket and emitsDataOps/KnowledgeJob=knowledge-repo-backupSuccess=1/0.commit_shaequals the last mirror-reported repository HEAD, or the manifest landed within 48h.jobs/knowledge-mirror/last-report.jsonso the check can tell a quiet repository (current backup, stays green for weeks) from a broken backup (pages as soon as an unbacked commit ages out).dataops-v1-knowledge-repo-backup-staleininfra/template.full.yaml: dead-man check mirroringKnowledgeBackupStaleAlarm(2-day sum < 1, missing data breaching), paging the shared operator topic, with OK recovery.sandbox-backup-evidence.ymlnow proves the new alarm exists with enabled actions and reaches a confirmed SNS subscriber.Tests
backend/tests/knowledge-repo-backup-freshness.test.ts: freshness semantics (commit match, 48h boundary, divergent/missing manifest, no-report fallback), EMF record shape, S3 read path, unconfigured-bucket failure.make sam-validatepasses.