fix(impl-generate): keep infrastructure failures out of the 3-attempt cap - #11199
Conversation
… cap The cap exists to stop re-running a pair the model cannot solve. It counted every failed run alike, so the Claude outage of 2026-09-02 (03:15-03:45 UTC, every run ending in `is_error:true` with an "Internal error") burned all three attempts of 27 pairs in twenty minutes, and for the rest of the 12-hour window each re-dispatch ran without any auto-retry. The failure handler now classifies the run from step outcomes: both Claude runs dying with a provider-side signature in the execution log, a Google Cloud auth/SDK/upload failure, or a GitHub API failure at PR creation or review dispatch is an infrastructure failure. Those markers carry an extra `<!-- impl-fail-cause:infra -->` tag, are excluded from the genuine count, and are retried on a separate cap of 5 per window, after which the pair is paused without `impl:<lib>:failed`. "Implementation file not found" and a missing theme render keep counting as before. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SrKzcwZBnref1sWYtdXynu
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SrKzcwZBnref1sWYtdXynu
There was a problem hiding this comment.
🟡 Changes recommended
The workflow’s infra-cause extraction can produce an empty/low-signal “Cause” message (and one inline comment is misleading), which should be corrected to avoid confusing incident diagnostics.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Updates the impl-generate workflow to distinguish “infrastructure” failures (provider incident, cloud auth/upload, GitHub API) from genuine generation failures, so outages don’t consume the 3-attempt cap for a (spec, library) pair. It also documents the updated retry/label semantics.
Changes:
- Classify failures using step outcomes plus the Claude execution log signature, and track infra failures separately from genuine failures.
- Add an infra-only retry cap (5 per 12-hour window) that pauses the pair without setting
impl:{library}:failed. - Update workflow documentation and changelog to reflect the new label and retry behavior.
File summaries
| File | Description |
|---|---|
.github/workflows/impl-generate.yml |
Adds infra-failure classification, separate counting/tagging, and a separate infra retry cap with pause behavior. |
docs/workflows/overview.md |
Updates the meaning of impl:{library}:failed to reflect campaign-window semantics and infra exclusions. |
agentic/docs/project-guide.md |
Mirrors the updated impl:{library}:failed semantics for agent-focused docs. |
CHANGELOG.md |
Records the infra-failure retry behavior change under Fixed. |
Review details
- Files reviewed: 4/4 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
… address review - INFRA_CAUSE always carries a snippet (falls back to a generic provider-side note when only the HTTP status matched) - comment states that only the retry step's outcome is inspected Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SrKzcwZBnref1sWYtdXynu
There was a problem hiding this comment.
🔵 Needs a closer look
The new infra-classification path can inadvertently fail under Actions’ default bash -eo pipefail due to a grep pattern mismatch in the INFRA_CAUSE extraction, which can prevent retries/parking from running.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
.github/workflows/impl-generate.yml:1064
- The infra classification builds INFRA_CAUSE via a command substitution that can fail under Actions' default
bash -eo pipefail: the detection grep includes the"status": *5xxalternative, but the extraction grep does not. If the only match is the status code,grep -oreturns 1 and can abort the whole failure-handler step, preventing infra retries/parking from running.
- Files reviewed: 4/4 changed files
- Comments generated: 0 new
- Review effort level: Lite
…-excludes-infra-failures # Conflicts: # CHANGELOG.md
There was a problem hiding this comment.
🟡 Changes recommended
The updated docs describe impl:{library}:failed as only a generation-attempt cap, but the same label is also applied by impl-review.yml for post-review failure, so the documentation is currently misleading.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 4/4 changed files
- Comments generated: 2
- Review effort level: Lite
…repairs Review feedback: the label table and the project guide described only the generation path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SrKzcwZBnref1sWYtdXynu
…-excludes-infra-failures
There was a problem hiding this comment.
🟡 Changes recommended
The workflow uses a comma-separated --remove-label that can cause gh issue edit to fail to remove either label when one is missing, which undermines the new pause/retry behavior.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 4/4 changed files
- Comments generated: 1
- Review effort level: Lite
There was a problem hiding this comment.
🔵 Needs a closer look
It changes core GitHub Actions retry/labeling behavior with operational impact and can’t be fully validated here without executing real workflow runs across failure modes.
Review details
- Files reviewed: 4/4 changed files
- Comments generated: 0 new
- Review effort level: Lite
…andler Review feedback: a comma-separated --remove-label fails as a whole when one name cannot be resolved, which could leave a pair marked pending. Applies to the new infrastructure-pause branch and the existing cap branch. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SrKzcwZBnref1sWYtdXynu
There was a problem hiding this comment.
🔵 Needs a closer look
The new infra-classification logic currently treats any GCS upload step failure as “infrastructure,” which can misclassify missing-theme renders and contradict the intended attempt-counting behavior.
Review details
Suppressed comments (2)
Previously missed (2) — in code that hasn't changed since the last review.
.github/workflows/impl-generate.yml:1070
GCS_OUTCOME == failureis treated as an infrastructure failure, but theUpload to GCS Stagingstep can fail due to missing plot renders (plot-light.png/plot-dark.png), which the PR description says should still count toward the 3-attempt cap. As written, a missing-theme render would be misclassified as infra and not spend the budget.
agentic/docs/project-guide.md:792- This bullet refers to the workflows as plain text ("impl-generate", "impl-review"), while other docs in this PR use the canonical workflow filenames (e.g.,
impl-generate.yml). Using the exact filenames in backticks makes it unambiguous which workflow sets the label.
- Files reviewed: 4/4 changed files
- Comments generated: 0 new
- Review effort level: Lite
Summary
impl-generate.ymlcounted every failed run alike. During the Claude outage of 2026-09-02 (03:15–03:45 UTC) every run ended inis_error:truewith an "Internal error"; 27 pairs burned all three attempts in twenty minutes, and for the rest of the 12-hour window each re-dispatch ran without any auto-retry.steps.<id>.outcome, plus the Claude execution log for the provider-side signature): both Claude runs dying withInternal error/overloaded/rate limit/ECONNRESET, a Google Cloud auth/SDK/upload failure, or a GitHub API failure at PR creation or review dispatch is an infrastructure failure.Authenticate to GCP,Set up Cloud SDKandTrigger review workflowget ids for that.<!-- impl-fail-cause:infra -->tag in the same marker comment, excluded from the genuine count (one paginated query returns both numbers), and retried on a separate cap of 5 per window; at that cap the pair is paused with a comment and withoutimpl:<lib>:failed, because an incident is not a capability verdict. "Implementation file not found" and a missing theme render keep counting as before.dispatch_retry()and used by both branches; docs (docs/workflows/overview.md,agentic/docs/project-guide.md) describe the label semantics.Plan
N/A
Test plan
bash -n; the new jq counting filter was dry-run against a synthetic comment list (genuine=1, infra=1).Authenticate to GCPposts the:cloud:infrastructure comment, retries, and the next run's notice readsPrevious failures ...: N (plus 1 infrastructure failures, not counted).Attempt N/3and counts.🤖 Generated with Claude Code
https://claude.ai/code/session_01SrKzcwZBnref1sWYtdXynu