Pattern quality: Add DO NOT boundary tips for highest-risk archetypes missing them - #155
Merged
pelikhan merged 1 commit intoAug 24, 2026
Conversation
…atus-report archetypes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
pelikhan
marked this pull request as ready for review
August 24, 2026 11:29
pelikhan
deleted the
fix/missing-do-not-tips-high-risk-archetypes-00e650765ce00d5c
branch
August 24, 2026 11:30
Contributor
Author
There was a problem hiding this comment.
Reviewer: auto (content/data → ponytail perspective, with matt-pocock check on the JS test additions)
Reviewed the diff: three archetype JSON files gained a single new "Use DO NOT constraints..." tip entry each, and test/workflow.test.js gained two new assertions verifying the boundary-constraint line now appears in generated prompts for code-improvement, dependency-monitor, and status-report.
- Content/UX: the new tip strings are clear, actionable, and consistent in tone/format with existing tips in each file (proper JSON escaping of em dash and quotes verified).
- Correctness: relies on existing
doNotConstraints()matching/do not/i— each new tip does contain "DO NOT", so it will be picked up correctly; no generator code changes needed, which matches the PR description. - Tests: the two new tests are scoped and directly assert the intended behavior change.
No blocking issues found. This is a low-risk, well-scoped content addition with adequate regression coverage.
Generated by Specialist PR Review for #155 · auto · 13.9 AIC · ⌖ 1.59 AIC · ⊞ 7.9K
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was evaluated
Reviewed all 6 samples in
generated-patterns-and-prompts.json(status-report, issue-triage, code-improvement, documentation-updater, dependency-monitor, pr-review), each compared againstpatterns/manifest.json, per-archetype files inpatterns/archetypes/, and the prompt logic insrc/js/workflow.js.Recurring quality issue found
generateAgentPrompt()insrc/js/workflow.jsinjects a "boundary constraint" line into the generated prompt only when an archetype'stipsarray contains a tip matching/do not/i(seedoNotConstraints()). gh-aw's own research (surfaced elsewhere in this repo) shows workflows with explicit DO NOT constraints are 61% more likely to be healthy.Three of the six sampled archetypes had no DO NOT tip at all, so their generated prompts silently omitted this guardrail entirely:
code-improvement— the lowest historical success rate of any archetype (37%) and the highest risk, since it autonomously opens code-modifying pull requests. This is the archetype that most needs scope-limiting guidance (e.g. don't touch unrelated files, CI/workflow configs, or secrets), yet had zero boundary constraint.dependency-monitor— opens both issues and PRs for dependency bumps; missing a constraint on scope of auto-updates (e.g. major version bumps, auto-merge).status-report— a read-only reporting archetype; missing an explicit "don't modify code/issues" constraint to keep it strictly read-only.What changed
Added one targeted DO NOT tip to each of the three archetype files under
patterns/archetypes/:code-improvement.json: "Use DO NOT constraints to keep changes scoped — avoid touching unrelated files, CI/workflow configs, or secrets"dependency-monitor.json: "Use DO NOT constraints to avoid auto-merging updates or bumping major versions without review"status-report.json: "Use DO NOT constraints to keep the report read-only — do NOT modify code, files, or existing issues"No generator code changes were needed —
doNotConstraints()already surfaces any tip matching/do not/iinto the prompt automatically. Added regression tests intest/workflow.test.jsasserting the boundary-constraint line now appears forcode-improvement,dependency-monitor, andstatus-report.Validation
npm test: 185/185 tests passed (12 test files), including 2 new tests added for this change.npm run build: succeeded,dist/output generated without errors.Follow-ups deliberately left out
scripts/generate-patterns.pyor regeneratepatterns/from raw scan data — per task constraints, and because this is hand-curated guidance content, not scan-derived.patterns/archetypes/beyond the 6 sampled in the generated-prompts file for the same gap; a follow-up pass could check all archetypes for missing DO NOT tips.prompt-solution-simulatoragent behavior/output (it produced no usable response during this evaluation); the finding here was derived directly from static analysis of the generator logic and pattern data, which was sufficient to establish high confidence.