Skip to content

Add scheduled PowerShell Gallery health check that opens status-page PRs on outages - #2086

Open
alerickson wants to merge 7 commits into
masterfrom
galleryWorkflow
Open

alerickson wants to merge 7 commits into
masterfrom
galleryWorkflow

Conversation

@alerickson

@alerickson alerickson commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

PR Summary

Adds automated PowerShell Gallery outage reporting.

  • psgallery-health-check.yml: scheduled workflow that runs every 30 minutes. It sends the same search queries PSResourceGet uses for Find-PSResource (find by name, wildcard name with -Type, and tag searches for modules, scripts and prerelease), 17 requests in total. Only hard failures count: HTTP 404, 5xx, or no response within 60 seconds. If 2 or more of the search requests fail, or find-by-name fails 3 times in a row, it triggers the outage reporter.

The requests come from the Find-PSResource CI tests that most often fail with 500s:

Probe Command it mirrors What makes it different
Find by name Find-PSResource -Name test_module FindPackagesById() with an Id eq filter
Wildcard name with type Find-PSResource -Name 'test*' -Type Module Search() with startswith(Id) and a substringof filter on Tags
Tag search, modules Find-PSResource -Tag CommandsAndResource Search() with substringof filters on Tags, $top=6000, $orderby
Tag search, scripts Find-PSResource -Tag 'Tag-Required-Script1-2.5' The separate items/psscript/Search() route
Tag search, prerelease Find-PSResource -Tag MyPSTag -Prerelease IsAbsoluteLatestVersion with includePrerelease=true

There are 5 kinds of request, but some are sent more than once per run:

Probe Sent per run
Wildcard name with type 4
Tag search, modules 4
Tag search, scripts 4
Tag search, prerelease 4
Find by name 1
Total 17

Search probes, 4 each: one bad instance only fails some requests, about 1 in 5 in my sampling, so a single request per query would usually land on a healthy instance and miss it. Sending each one 4 times gives 16 search requests. A run is flagged when 2 or more of those 16 fail, which catches a bad instance about 86% of the time per run, or about 98% within an hour.

Find by name, 1: it's only there to catch a full outage, so one request is enough. It's retried only if it fails, up to 3 tries. A healthy run sends 17 requests, and a run where find by name fails sends up to 19.

You can change the 4 with $searchSamples in psgallery-health-check.yml. Lowering it to 2 would mean 9 requests per run, but a partial outage would be caught less often per run.

  • psgallery-outage-reporter.md (+ compiled .lock.yml): an agentic workflow (gh-aw, Copilot engine) that re-checks the failing requests and, if they're still failing, opens a PR in PowerShell/PowerShellGallery. The PR adds an "Ongoing" entry to the top of psgallery_status.md for the Gallery team to review and merge. If a matching PR is already open, it comments on that PR instead. If the status page already has a matching ongoing entry, it does nothing.
  • .github/aw/actions-lock.json and .gitattributes: broilerplate files that (for actions-lock.json) pin the gh-aw setup action and (for .gitattributes) mark generated *.lock.yml files as generated.

Guardrails:

  • The agent only has read access to repositories.
  • All writes go through gh-aw safe outputs: at most 1 PR and 1 comment per run, only in PowerShell/PowerShellGallery, and the PR may only change psgallery_status.md.
  • Shell access is limited to curl, jq, sleep, date, cat and head, and network access is limited to GitHub and PowerShell Gallery domains.

Security review: this adds the repository secret PSGALLERY_STATUS_PR_TOKEN. Only gh-aw's safe-outputs job uses it, to open the PR or comment in PowerShell/PowerShellGallery; the agent itself has read-only permissions. It needs Contents, Pull requests and Issues write on PowerShell/PowerShellGallery. It also requires the existing gh-aw COPILOT_GITHUB_TOKEN secret.

PR Context

This workflow is meant to catch Gallery failures and update the status page as quickly as possible with a ready-to-merge PR.

Tested end to end on a fork (alerickson/PSResourceGet, targeting alerickson/PowerShellGallery):

PR Checklist

…Resource search queries and, when they keep returning 5xx errors or timing out, triggers an agentic workflow to open or update a service-status issue in PowerShell/PowerShellGallery
…ry_status.md in PowerShell/PowerShellGallery instead of filing an issue
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved issues can misclassify timed-out requests as healthy, duplicate ongoing outage reports, allow broader-than-stated network access, and prevent diagnostic issues from being created.

4 open issues
What changed in this PR

Adds automated PowerShell Gallery monitoring to help surface failures affecting PSResourceGet users and prepare status-page updates for the Gallery team to review.

Changes:

  • Runs scheduled probes every 30 minutes using representative Find-PSResource queries and pooled failure thresholds.
  • Adds an agentic reporter that rechecks failures and proposes a status-page PR or comments on an existing report.
  • Includes the compiled workflow, pinned setup action, and generated-file metadata.
File Description
.github/​workflows/​psgallery-outage-reporter.md Defines re-verification, incident matching, reporting, and write restrictions.
.github/​workflows/​psgallery-outage-reporter.lock.yml Compiles the reporter into executable GitHub Actions jobs.
.github/​workflows/​psgallery-health-check.yml Schedules Gallery probes and dispatches the reporter when thresholds are met.
.github/​aw/​actions-lock.json Pins the gh-aw setup action to a commit.
.gitattributes Marks compiled workflow files as generated.

Review effort: Balanced

alerickson and others added 4 commits November 13, 2026 17:08
…lthy in both the health check and the outage reporter's re-check, even when the server already sent a 200 status
….com, dropping the defaults group that pulled in unrelated sources
…ts so gh-aw's diagnostic issues in this repository use GITHUB_TOKEN instead of a token that only has access to PowerShellGallery
Updated instructions for checking existing outage reports to include all open pull requests and clarify status checks.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@alerickson

Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants