Skip to content

Pattern quality: Add DO NOT boundary tips for highest-risk archetypes missing them - #155

Merged
pelikhan merged 1 commit into
mainfrom
fix/missing-do-not-tips-high-risk-archetypes-00e650765ce00d5c
Aug 24, 2026
Merged

Pattern quality: Add DO NOT boundary tips for highest-risk archetypes missing them#155
pelikhan merged 1 commit into
mainfrom
fix/missing-do-not-tips-high-risk-archetypes-00e650765ce00d5c

Conversation

@github-actions

Copy link
Copy Markdown
Contributor

What was evaluated

Reviewed all 6 samples in generated-patterns-and-prompts.json (status-report, issue-triage, code-improvement, documentation-updater, dependency-monitor, pr-review), each compared against patterns/manifest.json, per-archetype files in patterns/archetypes/, and the prompt logic in src/js/workflow.js.

Recurring quality issue found

generateAgentPrompt() in src/js/workflow.js injects a "boundary constraint" line into the generated prompt only when an archetype's tips array contains a tip matching /do not/i (see doNotConstraints()). gh-aw's own research (surfaced elsewhere in this repo) shows workflows with explicit DO NOT constraints are 61% more likely to be healthy.

Three of the six sampled archetypes had no DO NOT tip at all, so their generated prompts silently omitted this guardrail entirely:

  • code-improvement — the lowest historical success rate of any archetype (37%) and the highest risk, since it autonomously opens code-modifying pull requests. This is the archetype that most needs scope-limiting guidance (e.g. don't touch unrelated files, CI/workflow configs, or secrets), yet had zero boundary constraint.
  • dependency-monitor — opens both issues and PRs for dependency bumps; missing a constraint on scope of auto-updates (e.g. major version bumps, auto-merge).
  • status-report — a read-only reporting archetype; missing an explicit "don't modify code/issues" constraint to keep it strictly read-only.

What changed

Added one targeted DO NOT tip to each of the three archetype files under patterns/archetypes/:

  • code-improvement.json: "Use DO NOT constraints to keep changes scoped — avoid touching unrelated files, CI/workflow configs, or secrets"
  • dependency-monitor.json: "Use DO NOT constraints to avoid auto-merging updates or bumping major versions without review"
  • status-report.json: "Use DO NOT constraints to keep the report read-only — do NOT modify code, files, or existing issues"

No generator code changes were needed — doNotConstraints() already surfaces any tip matching /do not/i into the prompt automatically. Added regression tests in test/workflow.test.js asserting the boundary-constraint line now appears for code-improvement, dependency-monitor, and status-report.

Validation

  • npm test: 185/185 tests passed (12 test files), including 2 new tests added for this change.
  • npm run build: succeeded, dist/ output generated without errors.

Follow-ups deliberately left out

  • Did not re-run scripts/generate-patterns.py or regenerate patterns/ from raw scan data — per task constraints, and because this is hand-curated guidance content, not scan-derived.
  • Did not audit the remaining archetypes in patterns/archetypes/ beyond the 6 sampled in the generated-prompts file for the same gap; a follow-up pass could check all archetypes for missing DO NOT tips.
  • Did not change prompt-solution-simulator agent behavior/output (it produced no usable response during this evaluation); the finding here was derived directly from static analysis of the generator logic and pattern data, which was sufficient to establish high confidence.

Generated by Pattern Quality Eval · auto · 49.3 AIC · ⌖ 15.6 AIC · ⊞ 7.5K ·

…atus-report archetypes

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@pelikhan
pelikhan marked this pull request as ready for review August 24, 2026 11:29
@pelikhan
pelikhan merged commit 1e23dc2 into main Aug 24, 2026
1 check passed
@pelikhan
pelikhan deleted the fix/missing-do-not-tips-high-risk-archetypes-00e650765ce00d5c branch August 24, 2026 11:30

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: auto (content/data → ponytail perspective, with matt-pocock check on the JS test additions)

Reviewed the diff: three archetype JSON files gained a single new "Use DO NOT constraints..." tip entry each, and test/workflow.test.js gained two new assertions verifying the boundary-constraint line now appears in generated prompts for code-improvement, dependency-monitor, and status-report.

  • Content/UX: the new tip strings are clear, actionable, and consistent in tone/format with existing tips in each file (proper JSON escaping of em dash and quotes verified).
  • Correctness: relies on existing doNotConstraints() matching /do not/i — each new tip does contain "DO NOT", so it will be picked up correctly; no generator code changes needed, which matches the PR description.
  • Tests: the two new tests are scoped and directly assert the intended behavior change.

No blocking issues found. This is a low-risk, well-scoped content addition with adequate regression coverage.

Generated by Specialist PR Review for #155 · auto · 13.9 AIC · ⌖ 1.59 AIC · ⊞ 7.9K

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant