ci: remove automated issue-opening monitors and drift tracker - #357
Merged
Merged
Conversation
kyle-sexton
added a commit
that referenced
this pull request
Sep 8, 2026
…sture (Phase 9b) (#359) No related issue: melodic-software/github-iac#378 tracks the ci-perf program (Phase 9b). Draft until Phase 7 steps 5 and 7 land. ## Summary This repository's README documented a routing contract that no longer exists. Its "Routing and fallback contract" section described three selector policies (`hosted-only`, `prefer-self-hosted`, `self-hosted-only`), the liveness rule that chose between them, and a recovery procedure that flips the affected repository's effective policy to `hosted-only` through the audited `github-iac` routing-control workflow and then re-runs all jobs to force a fresh selector verdict. ci-perf Phase 7 deleted every part of that. The `select-runner` reusable workflow is gone (ci-workflows#569, merged as `541ee4e90d12d77a90a3ddd72a3af9bc78634ea7`, released as v0.23.0), its grammar is out of the `runner-policy` component (melodic-software/standards#556, merged as `771a796628f325c3c418c7b397d09fb7211e2972`), and the routing-control workflow that wrote `CI_RUNNER_POLICY` is deleted (github-iac#453, merged as `4c5937e6b6067552d11b87abef04620191dda503`). **Nothing about this controller changed.** It still supplies runners inside its governed name and label namespaces, and a busy runner still queues rather than diverting work. What changed sits entirely upstream of it: an eligible private job names `melodic-ubuntu-24.04-x64` as a literal in its own `runs-on`, and nothing decides placement at run time. The decision record is github-iac ADR 0014, which supersedes ADR 0004. ## Fix `README.md`: - **Intro paragraph.** "The centrally governed routing policy decides whether local capacity is preferred, required, or bypassed" and "serve the organization's governed `self-hosted-only` routing default" both describe a policy that is gone. - **Runtime architecture mermaid.** The `Reusable workflow selector` node with its "managed route" and "hosted route" edges becomes a workflow job naming one label or the other. - **"Routing and fallback contract" becomes "Routing contract"**, rewritten. The three policies, the liveness rule, the fail-open/fail-closed selector behaviour and the two-minute `ubuntu-slim` selector control job are recorded as retired with their citing SHAs. Two consequences a reader actually needs are stated plainly: a rerun no longer recomputes anything about placement, so **Re-run all jobs** and a failed-job rerun are equivalent as far as routing is concerned; and there is no consumer-side recovery from an unavailable fleet at all, so affected jobs queue until a host returns. The unrelated stale-re-run hazard is pointed at github-iac `POSTURE.md` rather than restated. - **Rollback order.** Its first step was "set routing `hosted-only`". There is no routing variable, so the rollback now says what a fleet it cannot restore actually means: the affected jobs queue until it is back. It also states what the escape would really cost, because the escape is not one edit: widening the reason set is a governed standards change **and** each affected repository still needs its own per-job `exceptions` entry plus a `runs-on` change before anything routes differently. A reader planning a rollback window needs that stated, not silently dropped. **There is no consumer-side recovery from a fleet outage, and the doc now says so.** An earlier draft offered a `runner-policy.json` exception as the substitute for the deleted `hosted-only` flip. That path does not exist: `reason` must be a member of the closed `hostedExceptionReasons` set (`dependabot`, `docker-socket`, `job-container`, `privileged-control-plane`, `publication`, `service-container`, `windows`), enforced at `runner-policy.mjs:348`, and none of them expresses "the fleet is down". Both the routing section and the rollback order now say the affected jobs queue until a host returns, and that widening that set is a governed standards change, which on its own routes nothing. Two further precision points carried over from the sibling PRs' review, so this document does not repeat their errors: - **`CI_RUNNER_POLICY` is described as present and unread, not as deleted.** The organization variable is live at `self-hosted-only`; only its removal from the github-iac Pulumi program is decided, pending that repository's Phase 7 step 5 apply. Nothing reads it, because the selector that consumed it is deleted. - **`hosted-exception-required` is named as a finding, not a config key.** A reader following the other phrasing would write a config the schema rejects (the repository-policy schema is `additionalProperties: false` and permits no such top-level key). The real shape is an entry under `exceptions`, keyed `<workflow path>#<jobId>`, carrying a `reason` from the closed enum plus a `justification`. - **The ADR 0014 reference names github-iac#466 rather than a `main` URL**, which would 404 until that PR merges. "Supersedes ADR 0004" likewise becomes true only when #466 lands; both PRs are gated on the same Phase 7 steps. `docs/roadmap.md`: - The isolated-Linux-VM backend section required a future backend to preserve "the audited hosted-only cutoff with a full workflow rerun that recomputes selector eligibility". There is no eligibility to recompute and no cutoff to preserve. - The cost-reporting section required separating "selector spend from workload spend". The selector's `ubuntu-slim` control job is retired, so there is no selector spend. ## Verification - `npx markdownlint-cli2 README.md docs/roadmap.md`: 0 issues. - No em-dashes in added lines (`git diff -U0 | grep "^+" | grep -c "—"` prints `0`). - Every SHA cited is a full 40 characters and was resolved against the local clones before being written. - In the two **changed** files, every surviving `selector` / `CI_RUNNER_POLICY` hit is deliberate retirement prose, not an instruction. This is scoped to the changed files on purpose: `docs/queue-monitor.md` lines 104 to 108 still instruct an operator to flip `CI_RUNNER_POLICY` to `hosted-only` and expect the selector to re-run, and `.github/scripts/queue-monitor.cjs:23` ships the same text in incident output. Both are deleted by ci-runner#357 and are deliberately left to it rather than conflicted with here. **Merge-order note:** if this PR lands first, the README contradicts that shipped incident text until #357 follows. - Claims about what was deleted were checked against `origin/main` of the ci-workflows clone: `select-runner.yml`, `selector-conformance.yml` and `queue-monitor-liveness.yml` are absent from `.github/workflows/`. **Expected interaction with ci-runner#357.** That draft also edits `README.md`, but only to remove two entries from the "Further documentation" list (lines 435 onward) as it deletes the two monitors. My edits are in the intro, the mermaid block, the routing section and the rollback paragraph, so the two diffs touch disjoint hunks and merge cleanly in either order. #357 is gated on a seven-day green `pool-alert.yml` window; this PR is gated on Phase 7 steps 5 and 7. Whichever lands second may need a trivial rebase. ## Related - melodic-software/github-iac#378, the ci-perf program; github-iac#466, the Phase 9b PR carrying ADR 0014 and the completed `POSTURE.md`. - ci-workflows#569, merged as `541ee4e90d12d77a90a3ddd72a3af9bc78634ea7`: deletes `select-runner.yml`. - melodic-software/standards#556, merged as `771a796628f325c3c418c7b397d09fb7211e2972`: retires the selector grammar. - github-iac#453, merged as `4c5937e6b6067552d11b87abef04620191dda503`: deletes the routing-control workflow this README's recovery procedure pointed at. - ci-runner#357, the queue-monitor and budget-monitor retirement, gated on the pool-alert window. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
Author
Merge gate resetThe pool-alert 7-of-7 gate did not hold. Scheduled The rebase onto Owner decision needed before merge. |
Deletes queued-job-monitor.yml and actions-budget-monitor.yml along with their scripts (queue-monitor.cjs, queue-monitor.test.cjs, budget-monitor.cjs, budget-monitor.test.cjs) and their documentation pages (docs/queue-monitor.md, docs/actions-budget-monitor.md). Both workflows are superseded by pool-alert.yml in github-iac, per the ci-perf Phase 7 routing retirement plan (melodic-software/github-iac#378, sub-issue #425), step 6. incident-issue.cjs is also deleted. A grep sweep of the tree after the other deletions showed its only requirers were queue-monitor.cjs, queue-monitor.test.cjs and budget-monitor.cjs, all removed in this change, so no surviving workflow or script depends on it. README.md's two links to the deleted docs are removed, and docs/releases.md's description of the release-tag verification job is updated to say "workflow-script tests" instead of naming the now-deleted queue-monitor tests specifically. release/dependencies.json's actions/create-github-app-token entry is removed because it was minted only by queued-job-monitor.yml to read the observer App installation, and Test-ReleasePins.ps1 fails closed on any manifest entry no workflow uses. docs/topics/repo-simplify-sweep/RUN-STATE.md and docs/releases.md's historical note about the v0.1.20 tag reservation still name these files; both are historical records of past runs and are left untouched. No ci-workflows pin changed. ci-runner stays pinned at v0.22.2 (5776760254f8b63cba44e896f51604cb755350d9) by decision; this change touches none of its governed references. This PR opens as a draft. Its merge gates on pool-alert.yml showing seven green scheduled runs over seven consecutive days (currently 4 of 7). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Companion to 83d76f6, staged separately because git rm only staged the deletions themselves. - README.md: remove the two "Further documentation" links that pointed at the deleted docs/queue-monitor.md and docs/actions-budget-monitor.md. - docs/releases.md: the release-tag verification job description named "queue-monitor tests" specifically; renamed to "workflow-script tests" since queue-monitor.test.cjs no longer exists. - release/dependencies.json: removed the actions/create-github-app-token entry. Test-ReleasePins.ps1 fails closed on any githubActions manifest entry not used by a workflow, and queued-job-monitor.yml was the only caller of that action. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Owner decision 2026-09-14: no automated process opens GitHub issues on its own. Where a process has other value, keep it and remove only the issue writes. - dependency-drift.yml: remove the "Record and reconcile drift issue" step and the job's issues: write permission. The freshness check, the evidence artifact upload and the 14-day or critical drift gate are unchanged. - link-check.yml: delete this caller of the ci-workflows link-check reusable, whose scheduled failure path maintains an issue. The offline lychee lane in ci.yml still checks links on every PR. - workflow-pin-metadata.test.cjs: drop the link-check caller from the inventory. The remaining ci-workflows references still share one SHA, and repositoryPins is unchanged. - release/dependencies.json: drop actions/github-script, whose only caller was the removed drift-issue step; Test-ReleasePins.ps1 fails closed on unused githubActions entries. - README.md: drift evidence is an artifact, not an issue. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019mo9xaZ53JHHy4aBjgd8wK
kyle-sexton
force-pushed
the
ci-perf/retire-monitors
branch
from
September 14, 2026 17:51
77caf2f to
f95a6c8
Compare
kyle-sexton
changed the title
|
PR body contract — issue linkage This PR body conforms to the issue-linkage contract. Nothing to do. |
- release.yml: the script-test step no longer names the deleted queue monitor tests. - docs/roadmap.md: remove the paragraph about the deleted scheduled queue monitor's freshness; the section keeps the cost-evidence item. - README.md: remove the observer App sentence; its only consumer was the deleted queued-job-monitor.yml. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019mo9xaZ53JHHy4aBjgd8wK
kyle-sexton
marked this pull request as ready for review
September 14, 2026 18:06
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This was referenced Sep 14, 2026
kyle-sexton
added a commit
that referenced
this pull request
Sep 14, 2026
…rkflow (#363) No related issue: this PR closes nothing. ## Summary Owner direction 2026-09-14: delete stale and overengineered automation. This PR removes the host health-watch feature and the scheduled Dependency Drift workflow. It does not cut a release; nothing reaches a host until a later reviewed release is promoted. ## Fix Deleted: - `.github/workflows/dependency-drift.yml` (daily schedule). `scripts/Test-DependencyFreshness.ps1` stays and still runs as the freshness gate in `release.yml`. - `internal/healthwatch/` (alert, check, inventory, storage and their tests). - `internal/app/health_watch.go`, `health_watch_checker.go`, `health_watch_test.go`, `health_watch_install_test.go`. - `internal/host/healthwatch_task.go`, `healthwatch_task_windows.go`, `healthwatch_task_other.go`. - `internal/jobindex/snapshot_bytes_test.go`, with `SnapshotBytes`, `MaximumJobStateBytes` and `EncodedCatalogSize` in `file_store.go`. Only health-watch called them. Edited: - `internal/app/app.go`: removed the `host health-watch` case, its two usage lines, and the `Dependencies.ACL` field that only health-watch read. `bootstrap.go` no longer passes it. `ScheduledTaskCLI` stays because `host controller restart` uses it. - `internal/config/config.go`: removed the `HealthWatchdog` type, defaults and validation. Strict `KnownFields(true)` decoding is unchanged. A top-level `healthWatchdog` key is still accepted and ignored, and `Load` records a warning in `Config.Warnings`. The CLI prints it to stderr and the controller writes it to its log as `config-warning`. - `internal/config/config_test.go`: `TestLoadIgnoresLegacyHealthWatchdog` loads a config with a populated `healthWatchdog` block and asserts it loads with one warning, and that the base config has none. - `internal/controller/reconciler_test.go`, `internal/app/doctor.go`, `internal/jobindex/file_store.go`: removed the config fixture block and the comments that referred to health-watch. - `.github/actionlint.yaml`: removed the ignore entry for the deleted workflow. - `README.md`, `docs/releases.md`, `docs/worker-image.md`: freshness is now described as checked at release publication or by hand, not daily. The `#freshness-policy` anchor is kept. - `release/dependency-drift-review.json`: removed the four sentences saying the daily dependency-drift job reconciles issue 283. - `.gitignore`: comment wording only. The `dependency-drift.json` entry stays because it is the script's default output path. Left untouched: `docs/topics/repo-simplify-sweep/*`, which are historical run records. ## Host findings Checked read-only on 2026-09-14. Both hosts load `C:\Users\KyleSexton\AppData\Local\ci-runner\config.yaml`, according to the `ci-runner-fleet` scheduled task arguments. | Host | `healthWatchdog` in config | `ci-runner-health-watch` task | | --- | --- | --- | | melo-lap-001 | not present | not present | | melo-desk-001 (reached over SSH) | not present | not present | No host needs a change. The legacy-key handling covers any config this check could not see. ## Verification - `go build ./...`, `go vet ./...`, `GOOS=linux go vet ./...`: pass (go1.27.1). - `go test ./...`: all packages pass. - `golangci-lint run ./...` (v2.13.1 built with go1.27.1): 0 issues. - `actionlint -shellcheck= -pyflakes=`: exit 0. - `markdownlint-cli2 README.md docs/releases.md docs/worker-image.md`: 0 issues. - `pwsh -NoProfile -File scripts/Test-ReleasePins.ps1`: "Release and workflow pins are internally consistent." - `node --test .github/scripts/*.test.cjs`: 21 pass, 0 fail. - Fresh-context verifier (opus, given the scope but not this session's reasoning): first pass at `24fac49` FAIL on one stale sentence, where `docs/releases.md:64` still described "the daily official-release check". Every other check passed: build, vet, tests, lint, pin test, node tests, no leftover references, no dangling symbols, legacy config loads with a warning, unknown keys still rejected, release freshness step intact. Fixed in `a7ce55b`. Re-verify at `a7ce55b`: PASS. No daily or scheduled freshness wording remains, and the only `healthWatchdog` hits are the legacy-key acceptance and its test. ## Related Refs: melodic-software/github-iac#378 Refs: #357 Refs: #227 BREAKING CHANGE: `ci-runner host health-watch check` and `ci-runner host health-watch install-task` are removed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_019mo9xaZ53JHHy4aBjgd8wK --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No related issue: this PR closes nothing. #283 (the
[dependency-drift]tracker issue) is left open for the coordinator to close after merge.Summary
Owner decision 2026-09-14: remove every automated process that opens GitHub issues on its own (alerts, incidents, drift trackers, watchers). Where a process has other value (a failing check, a measurement), keep that value and remove only the issue writes.
This PR applies that to ci-runner. It retires both selector-era monitors, removes the issue writes from the dependency drift check, and deletes the scheduled link-check caller. After this change nothing in
.github/holdsissues: writeor calls an issue API.Fix
Deleted:
.github/workflows/queued-job-monitor.ymland.github/workflows/actions-budget-monitor.yml.github/scripts/queue-monitor.cjs,.github/scripts/queue-monitor.test.cjs.github/scripts/budget-monitor.cjs,.github/scripts/budget-monitor.test.cjs.github/scripts/incident-issue.cjs(its only requirers were the two monitor scripts and the queue-monitor test)docs/queue-monitor.md,docs/actions-budget-monitor.md.github/workflows/link-check.yml: the caller of the ci-workflows reusablelink-check.yml, which heldissues: writeto keep a scheduled failure issue up to date. Links are still checked on every PR by the offlinelycheelane inci.yml.Edited:
.github/workflows/dependency-drift.yml: removed the "Record and reconcile drift issue" step (issue list, create, update, comment, close) and the job'sissues: writepermission. The freshness check, the evidence artifact upload, and the "Enforce 14-day or critical drift gate" step are byte-identical. The removed step also wrote the drift table to the job summary. That summary is gone, but the same report is still uploaded as thedependency-drift-<run_id>artifact..github/scripts/workflow-pin-metadata.test.cjs: removed thelink-check.ymlentry from the ci-workflows caller inventory and dropped "link-check" from the assertion message. Every remaining ci-workflows reference still pins5776760254f8b63cba44e896f51604cb755350d9(v0.22.2), so the one-SHA rule holds.release/dependencies.json: removedactions/create-github-app-token, used only byqueued-job-monitor.yml, andactions/github-script, used only by the removed drift-issue step.scripts/Test-ReleasePins.ps1fails on anygithubActionsentry that no workflow uses.repositoryPinsis unchanged becauseci.ymlandmanaged-files-guard.ymlstill use the ci-workflows pin.README.md: removed the two links to the deleted docs and the observer App sentence, whose only consumer wasqueued-job-monitor.yml. The release section now says drift evidence is uploaded as a workflow artifact, where it used to say an issue is opened.docs/releases.md: "queue-monitor tests" is now "workflow-script tests"..github/workflows/release.yml: the script-test step is renamed from "Test queue monitor and resumable release transaction" to "Test workflow scripts and resumable release transaction".docs/roadmap.md: removed the paragraph about the deleted scheduled queue monitor's freshness, and renamed the section to "Cost evidence".scripts/Test-ReleasePins.ps1: no edit needed. It names neither link-check nor the monitors; it discovers workflows and pins dynamically.Left untouched because they are historical records: the
v0.1.20tag note indocs/releases.md,docs/topics/repo-simplify-sweep/RUN-STATE.md, and thereviewNoterows inrelease/dependency-drift-review.jsonthat say the daily job reconciles #283.docs/worker-image.mdlistslink-checkandqueue-monitor-livenessamong fleet-wide ci-workflows lanes that needgh; those lanes live in other repositories.Rebase
The branch was rebased onto
origin/mainat8ed9dd9. Main had edited four files this branch deletes: #359 changed.github/scripts/queue-monitor.cjs,queue-monitor.test.cjsanddocs/queue-monitor.md, and #358 changed.github/scripts/incident-issue.cjs. All four modify/delete conflicts were resolved by keeping the delete. #359's other edits (README.md,docs/roadmap.md) name none of the deleted files.Merge gate
pool-alert.ymlgate is dropped by the owner decision of 2026-09-14 (no automated issue alerts). This PR no longer waits on seven green scheduled runs.queued-job-monitor.ymlreferenced a repository secret that was never configured (no repository or organization secret by that name exists, checked 2026-09-14). The ping never ran, no external check was ever armed, and the owner has nothing to disarm. The workflow has beendisabled_manuallysince 2026-09-08 and is deleted here. No file in the repository references the secret after this change.Verification
node --test .github/scripts/*.test.cjs: 21 pass, 0 fail.pwsh -NoProfile -File scripts/Test-ReleasePins.ps1: "Release and workflow pins are internally consistent." (exit 0).actionlint -shellcheck= -pyflakes=: exit 0.markdownlint-cli2 README.md docs/roadmap.md: 0 issues.release.ymlstep name, the queue-monitor paragraph indocs/roadmap.md, and the observer App sentence inREADME.md), all fixed in6b38a6e. Re-verify at6b38a6e: PASS. It found no issue writes left, the drift gate and freshness steps byte-identical tomain, no unrelated hunks, and all tests green.Related
[dependency-drift]issue the removed step maintained. It is not closed by this PR.🤖 Generated with Claude Code
https://claude.ai/code/session_019mo9xaZ53JHHy4aBjgd8wK