diff --git a/.agents/logs/human_review_feedback.jsonl b/.agents/logs/human_review_feedback.jsonl index e69de29bb..c7e4b1c60 100644 --- a/.agents/logs/human_review_feedback.jsonl +++ b/.agents/logs/human_review_feedback.jsonl @@ -0,0 +1,9 @@ +{"date": "2026-06-29", "pr": "143", "skill_used": "draft_docs", "file": "src/content/docs/agent-platform/cloud-agents/agents.mdx", "feedback_type": "review_comment", "severity": "important", "comment": "This section may scan better if we separate the primary recommendation, the API reference details, and the constraints. Moving the endpoint names into a compact table also makes the legacy path feel like API reference material.", "tag": "", "pattern_category": "scannability", "resolved_by": "human_edit"} +{"date": "2026-06-29", "pr": "143", "skill_used": "draft_docs", "file": "src/content/docs/agent-platform/cloud-agents/agents.mdx", "feedback_type": "review_comment", "severity": "suggestion", "comment": "nit: more readable to use a Markdown table here?", "tag": "", "pattern_category": "scannability", "resolved_by": "human_edit"} +{"date": "2026-06-29", "pr": "143", "skill_used": "draft_docs", "file": "src/content/docs/reference/cli/api-keys.mdx", "feedback_type": "review_comment", "severity": "important", "comment": "This section is really about how the key type changes attribution, billing, and GitHub credentials. It would scan better with a more specific heading and parallel bullets for the two key types.", "tag": "", "pattern_category": "heading_specificity", "resolved_by": "human_edit"} +{"date": "2026-06-29", "pr": "143", "skill_used": "draft_docs", "file": "src/content/docs/reference/cli/api-keys.mdx", "feedback_type": "review_comment", "severity": "important", "comment": "Procedural steps did not match the live UI: corrected to 'Generate new token', the 'Type' label, and 'Warp app'. Masked-suffix and Created/Last used columns are not present in the Oz web app.", "tag": "", "pattern_category": "ui_label_accuracy", "resolved_by": "human_edit"} +{"date": "2026-06-29", "pr": "160", "skill_used": "draft_docs", "file": "src/content/docs/agent-platform/local-agents/interacting-with-agents/agent-questions.mdx", "feedback_type": "review_comment", "severity": "suggestion", "comment": "Rewrote the opening paragraph and bullets to be more concise and use the bold-term + dash list format.", "tag": "", "pattern_category": "list_format", "resolved_by": "human_edit"} +{"date": "2026-06-29", "pr": "166", "skill_used": "draft_docs", "file": "src/content/docs/agent-platform/cloud-agents/team-access-billing-and-identity.mdx", "feedback_type": "review_verdict", "severity": "important", "comment": "Be careful about embedding too many error message references throughout canonical docs pages. Lean toward a dedicated section at the bottom of the page (Troubleshooting) rather than weaving them into the main flow everywhere.", "tag": "", "pattern_category": "content_structure", "resolved_by": "human_edit"} +{"date": "2026-06-29", "pr": "189", "skill_used": "draft_docs", "file": "src/content/docs/guides/external-tools/github-mcp-summarizing-open-prs-and-creating-gh-issues.mdx", "feedback_type": "review_comment", "severity": "suggestion", "comment": "Multiple suggested edits normalizing guide section headings to a consistent, descriptive phrasing (e.g. 'Step 3. Workflow 1 - Summarize all open PRs', 'Why it's useful').", "tag": "", "pattern_category": "heading_specificity", "resolved_by": "human_edit"} +{"date": "2026-06-29", "pr": "238", "skill_used": "draft_docs", "file": "src/content/docs/support-and-community/plans-and-billing/credits.mdx", "feedback_type": "review_comment", "severity": "suggestion", "comment": "Standardized Free-plan AI-usage messaging and Warp Agent terminology, with consistent crosslinks to BYOK, custom inference endpoint, and Grok subscription pages.", "tag": "", "pattern_category": "terminology", "resolved_by": "human_edit"} +{"date": "2026-06-29", "pr": "256", "skill_used": "draft_docs", "file": "src/content/docs/agent-platform/inference/custom-routers.mdx", "feedback_type": "review_comment", "severity": "suggestion", "comment": "Link out to the model-choice page's available-models list from the examples.", "tag": "", "pattern_category": "link_quality", "resolved_by": "human_edit"} diff --git a/.agents/skills/create_pr/SKILL.md b/.agents/skills/create_pr/SKILL.md index 874653a0b..158c078d0 100644 --- a/.agents/skills/create_pr/SKILL.md +++ b/.agents/skills/create_pr/SKILL.md @@ -152,6 +152,8 @@ Exit code 0 if PR exists, 1 if not. :::caution **Always use `--body-file` instead of `--body` for PR descriptions.** Documentation PRs frequently contain backticks, quotes, and other special characters that get corrupted by shell escaping when passed inline. Write the description to a file first, then reference it. + +`--body-file` avoids shell-escaping corruption, but it does **not** catch repetition-loop degeneration — a failure mode where the model repeats a phrase or bullet several times and cuts off mid-token (e.g. a sentence ending in an unclosed inline-code span like `` because `m ``). That corrupted text is already in the generated body and survives `--body-file` unchanged. Always run the body integrity checker (`check_pr_body.py`) before creating or editing a PR. ::: ```bash @@ -167,7 +169,10 @@ Description of changes Co-Authored-By: Oz EOF -# 2. Create the PR using the file +# 2. Verify the body for corruption before submitting (exits non-zero on failure) +python3 .agents/skills/create_pr/check_pr_body.py /tmp/pr-body.md + +# 3. Create the PR using the file (only if the check passed) gh pr create --title "docs: Add feature documentation" --body-file /tmp/pr-body.md # Open in browser to fill details @@ -176,8 +181,19 @@ gh pr create --web ### Update an existing PR +When updating the body of an existing PR, make the **smallest** change rather than regenerating the whole description from memory — re-emitting a long body is what invites repetition-loop degeneration. Fetch the current body, apply a minimal or additive edit, verify it, then submit. + ```bash -# Edit body using a file (recommended — avoids shell escaping issues) +# 1. Fetch the current body to a file +gh pr view 123 --json body --jq .body > /tmp/pr-body.md + +# 2. Make a minimal/additive edit to /tmp/pr-body.md (e.g. append a new section) +# with the edit_files or create_file tools — do not rewrite untouched sections. + +# 3. Verify the body for corruption before submitting (exits non-zero on failure) +python3 .agents/skills/create_pr/check_pr_body.py /tmp/pr-body.md + +# 4. Edit the body using the file (only if the check passed) gh pr edit 123 --body-file /tmp/pr-body.md # Edit title only diff --git a/.agents/skills/create_pr/check_pr_body.py b/.agents/skills/create_pr/check_pr_body.py new file mode 100644 index 000000000..64f0269ce --- /dev/null +++ b/.agents/skills/create_pr/check_pr_body.py @@ -0,0 +1,195 @@ +#!/usr/bin/env python3 +"""Detect degeneration/corruption in a PR body before creating or editing a PR. + +Large language models occasionally emit "repetition-loop" corruption when +generating or rewriting long PR descriptions: the same phrase or bullet repeats +several times and often cuts off mid-token (for example, a sentence ending in an +unclosed inline-code span like "because `m"). This has shipped to real PRs. + +This check is COMPLEMENTARY to using `gh ... --body-file`. Writing the body to a +file avoids shell-escaping corruption, but it does NOT catch repetition-loop +degeneration that is already present in the generated text — that text survives +`--body-file` unchanged. Run this check on the body file right before +`gh pr create --body-file` / `gh pr edit --body-file`. If it reports an issue, +do NOT submit: regenerate the affected section and re-check. + +Checks performed: + * Repeated span - a long substring repeated many times anywhere in the + body (the core repetition-loop signature). Works across + line breaks and within a single Markdown line. + * Unbalanced backtick- a non-code line with an odd number of backticks, which + usually means an inline-code span was truncated + mid-token (e.g. "... because `m"). + * Duplicate heading - the same Markdown heading text appearing more than once. + * Required heading - (optional) assert specific headings are present exactly + once, for skills that emit a fixed body template. + +Usage: + python3 check_pr_body.py /tmp/pr-body.md + cat /tmp/pr-body.md | python3 check_pr_body.py - + python3 check_pr_body.py /tmp/pr-body.md \ + --require-heading "## Patterns addressed" \ + --require-heading "## Improvement targets" + +Exit codes: + 0 no issues found + 1 one or more issues found (do NOT submit; regenerate and re-check) + 2 usage / file error +""" +from __future__ import annotations + +import argparse +import re +import sys +from pathlib import Path +from typing import List, Optional, Tuple + +# Length of the sliding window (in characters) used to detect a repeated span, +# paired with the number of occurrences that trips the check. A short window must +# repeat more times; a long window only needs to repeat twice. +SHORT_WINDOW = 40 +SHORT_MIN_COUNT = 3 +LONG_WINDOW = 80 +LONG_MIN_COUNT = 2 + + +def _strip_urls(text: str) -> str: + """Remove URLs so repeated link targets don't cause false positives. + + A PR body that legitimately lists many links (e.g. a changelog) can repeat a + long URL prefix; that is not degeneration, so drop URLs before scanning. + """ + return re.sub(r"https?://\S+", " ", text) + + +def find_repeated_span(text: str) -> Optional[Tuple[str, int]]: + """Return (snippet, count) for the worst repeated span, or None if clean.""" + norm = re.sub(r"\s+", " ", _strip_urls(text)).strip() + if len(norm) < SHORT_WINDOW * 2: + return None + + worst: Optional[Tuple[str, int]] = None + for window, min_count in ((LONG_WINDOW, LONG_MIN_COUNT), (SHORT_WINDOW, SHORT_MIN_COUNT)): + if len(norm) < window * 2: + continue + counts: dict[str, int] = {} + for i in range(len(norm) - window + 1): + chunk = norm[i : i + window] + counts[chunk] = counts.get(chunk, 0) + 1 + chunk, count = max(counts.items(), key=lambda kv: kv[1]) + if count >= min_count and (worst is None or count > worst[1]): + worst = (chunk, count) + return worst + + +def _iter_non_code_lines(lines: List[str]): + """Yield (line_num, text) for lines outside fenced code blocks.""" + fence: Optional[str] = None + for line_num, line in enumerate(lines, start=1): + fence_match = re.match(r"^\s*(`{3,}|~{3,})", line) + if fence is not None: + if fence_match and fence_match.group(1)[0] == fence[0]: + fence = None + continue + if fence_match: + fence = fence_match.group(1) + continue + yield line_num, line + + +def find_unbalanced_backticks(lines: List[str]) -> List[Tuple[int, str]]: + """Return (line_num, line) for non-code lines with an odd backtick count.""" + issues = [] + for line_num, line in _iter_non_code_lines(lines): + if line.count("`") % 2 == 1: + issues.append((line_num, line.strip())) + return issues + + +def find_duplicate_headings(lines: List[str]) -> List[str]: + """Return heading texts that appear more than once (outside code fences).""" + seen: dict[str, int] = {} + for _, line in _iter_non_code_lines(lines): + m = re.match(r"^(#{1,6})\s+(.*\S)\s*$", line) + if m: + key = f"{m.group(1)} {m.group(2)}" + seen[key] = seen.get(key, 0) + 1 + return [h for h, c in seen.items() if c > 1] + + +def check_required_headings(lines: List[str], required: List[str]) -> List[str]: + """Return messages for required headings that are missing or duplicated.""" + present: dict[str, int] = {} + for _, line in _iter_non_code_lines(lines): + stripped = line.strip() + present[stripped] = present.get(stripped, 0) + 1 + problems = [] + for heading in required: + count = present.get(heading.strip(), 0) + if count == 0: + problems.append(f"missing required heading: {heading!r}") + elif count > 1: + problems.append(f"required heading appears {count}x (expected once): {heading!r}") + return problems + + +def main(argv: Optional[List[str]] = None) -> int: + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("body", help="path to the PR body file, or '-' for stdin") + parser.add_argument( + "--require-heading", + action="append", + default=[], + metavar="HEADING", + help="assert this exact heading line is present exactly once (repeatable)", + ) + args = parser.parse_args(argv) + + if args.body == "-": + text = sys.stdin.read() + else: + path = Path(args.body) + try: + text = path.read_text(encoding="utf-8") + except OSError as exc: + print(f"error: could not read {args.body}: {exc}", file=sys.stderr) + return 2 + + lines = text.splitlines() + issues: List[str] = [] + + repeated = find_repeated_span(text) + if repeated: + snippet, count = repeated + issues.append( + "repetition-loop: a span repeats " + f"{count}x (likely model degeneration).\n repeated text: {snippet!r}" + ) + + for line_num, line in find_unbalanced_backticks(lines): + issues.append( + f"unbalanced backtick on line {line_num} (possible truncated inline code): {line!r}" + ) + + for heading in find_duplicate_headings(lines): + issues.append(f"duplicate heading: {heading!r}") + + issues.extend(check_required_headings(lines, args.require_heading)) + + if issues: + print("PR body integrity check FAILED:\n", file=sys.stderr) + for issue in issues: + print(f" - {issue}", file=sys.stderr) + print( + "\nDo NOT submit this body. Regenerate the affected section " + "(or re-fetch and re-apply a minimal edit) and run this check again.", + file=sys.stderr, + ) + return 1 + + print("PR body integrity check passed.") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/.agents/skills/draft_docs/SKILL.md b/.agents/skills/draft_docs/SKILL.md index c369cfd36..8d93f9e5a 100644 --- a/.agents/skills/draft_docs/SKILL.md +++ b/.agents/skills/draft_docs/SKILL.md @@ -86,7 +86,9 @@ Use source code to verify technical behavior, understand feature implementation, These rules are frequently violated by agents. Apply them carefully during drafting: - **Sentence case for all headings (H1–H4)** — Capitalize only the first word and proper feature names. ✅ `## How it works` ❌ `## How It Works` +- **Descriptive, specific headings** — Beyond correct case, a heading should name the specific topic so readers and agents can scan the page and extract a self-contained answer. ✅ `## How key type affects billing and GitHub access` ❌ `## More details` - **Bold + dash format for list items** — `* **Term** - Description`, not `* Term: Description` +- **Tables or parallel bullets for comparison and reference data** — When you present two or more parallel items (key types, plan tiers, environments) or structured reference data (API endpoints, parameters), use a Markdown table or tightly parallel bullets instead of one dense paragraph. ✅ a table with one row per API endpoint, or parallel `**Personal API keys**` / `**Agent API keys**` bullet groups ❌ a single paragraph mixing both key types and their billing rules - **Bold for UI elements** — Use `**Save**` not `` `Save` `` after action verbs like "click" - **Bold per-segment for Settings paths** — Use `**Settings** > **AI** > **Knowledge**` not `` `Settings > AI > Knowledge` `` @@ -100,12 +102,12 @@ If this skill is running as a cloud agent producing an agent-authored PR, captur 1. Re-run with `--output /tmp/style_lint_out.json` to get machine-readable output. 2. Aggregate the `issues` array by `check` field to get violation counts per check name. -3. Print the following structured marker to stdout so the `improve-drafting-skills` collector can retrieve it from the Oz run output: +3. Include the following structured marker in your **text response** (write it as part of your agent message, not via a shell `echo` command). This ensures it appears as a `TextContentBlock` in the conversation, where `oz run get --conversation` can reliably retrieve it: ``` [SIGNAL:style-lint] {"date":"YYYY-MM-DD","pr":"NNN","branch":"BRANCH_NAME","authored_by":"agent","skill_used":"SKILL_NAME","files_scanned":N,"violations":{"check_name":count}} ``` -The `improve-drafting-skills` outer loop reads this signal from Oz run artifacts via `oz run get`. No git operations are required. +The `improve-drafting-skills` outer loop reads this signal from the conversation via `oz run get --conversation`, scanning assistant `TextContentBlock` messages for the marker. No git operations are required. Skip steps 1–3 in local/interactive sessions. diff --git a/.agents/skills/improve-drafting-skills/SKILL.md b/.agents/skills/improve-drafting-skills/SKILL.md index 2eb4d8e80..813421fc0 100644 --- a/.agents/skills/improve-drafting-skills/SKILL.md +++ b/.agents/skills/improve-drafting-skills/SKILL.md @@ -26,7 +26,7 @@ The following must be available in the cloud agent environment: Three inputs, combined during the feedback collector step: -- **Oz run artifacts** (style lint + PR review signals) — parsed from `[SIGNAL:style-lint]` and `[SIGNAL:pr-review]` markers in the stdout of drafting skill and `review-docs-pr` runs. **Primary automated signal.** No committed file needed; read directly from Oz run output via `oz run get`. +- **Oz run artifacts** (style lint + PR review signals) — parsed from `[SIGNAL:style-lint]` and `[SIGNAL:pr-review]` markers in agent text messages of drafting skill and `review-docs-pr` runs. **Primary automated signal.** No committed file needed; read directly from the conversation via `oz run get --conversation`. Note: `oz run get` without `--conversation` returns only the brief `status_message` field and does not expose conversation content or shell stdout. - **GitHub API** (human feedback) — inline review comments (`gh api repos/warpdotdev/docs/pulls/NNN/comments`), top-level reviews (`gh pr view --json reviews`), and human-authored commits after the agent's last commit. **Primary human signal.** Accumulated into `.agents/logs/human_review_feedback.jsonl` by this skill during the feedback collector step. - `.agents/logs/human_review_feedback.jsonl` — durable log written by this outer loop. Fields: `date`, `pr`, `skill_used`, `file`, `feedback_type`, `severity`, `comment`, `tag`, `resolved_by`. @@ -37,7 +37,12 @@ At the start of each monthly run, the feedback collector gathers signal data fro ### Step A: Collect style lint and PR review signals from Oz run artifacts 1. Use `oz run list` to find all Oz runs in the past 30 days whose skill name matches a drafting skill (`draft_docs`, `draft_feature_doc`, `draft_conceptual`, etc.) or `review-docs-pr`. -2. For each run, use `oz run get RUN_ID` to read the run output. +2. For each run, retrieve the full conversation and extract agent text messages: + ```bash + oz-dev run get --conversation RUN_ID --output-format json | \ + jq -r '[.. | objects | select(.role? == "assistant") | .content[]? | select(.type? == "text") | .text] | .[]' + ``` + The top-level response is `{steps: [...]}`, not `{messages: [...]}`, and steps can be nested — use recursive descent (`..`) to reach all assistant messages at any depth. Do not rely on `oz run get` without `--conversation` — that returns only the brief `status_message` field, not conversation content or shell stdout. 3. Parse any lines matching `[SIGNAL:style-lint] {JSON}` or `[SIGNAL:pr-review] {JSON}` and parse the JSON payload as the structured record. 4. Accumulate all parsed records in memory for the analysis step. 5. For `[SIGNAL:pr-review]` records, also prepend a human-readable entry to `.agents/logs/pr_review_runs.md` (using the format in that file's header). Commit the updated file directly to `main`: @@ -157,6 +162,16 @@ PR body must include: - **Patterns reviewed but not acted on** — any patterns that met the threshold but were already covered or had insufficient signal - **Open questions for human review** — any judgment calls about whether a proposed rule change is correct +Write the body to a file and verify it before opening the PR — this catches repetition-loop corruption that has reached PR descriptions before (see the `create_pr` skill for details): +```bash +python3 .agents/skills/create_pr/check_pr_body.py /tmp/pr-body.md \ + --require-heading "## Patterns addressed" \ + --require-heading "## Improvement targets" \ + --require-heading "## Patterns reviewed but not acted on" \ + --require-heading "## Open questions for human review" +``` +Run `gh pr create --draft --body-file /tmp/pr-body.md` only if the check passes. If you later edit this PR's body (for example, to record a human-review follow-up), fetch the current body first and apply a minimal, additive edit rather than regenerating it, then re-run the check — see the `create_pr` skill's "Update an existing PR" section. + Post a Slack summary to `#growth-docs`: ``` ✅ Drafting skills improvement · YYYY-MM-DD diff --git a/.agents/skills/review-docs-pr/SKILL.md b/.agents/skills/review-docs-pr/SKILL.md index 41427bdd2..6a4b46b57 100644 --- a/.agents/skills/review-docs-pr/SKILL.md +++ b/.agents/skills/review-docs-pr/SKILL.md @@ -126,9 +126,9 @@ After creating and validating `review.json` (immediately after the Validation se 1. Count comments in `review.json` by severity label (`🚨 [CRITICAL]`, `⚠️ [IMPORTANT]`, `💡 [SUGGESTION]`, `🧹 [NIT]`). 2. Identify the top 3 issue categories by frequency (use the `check` name if available from style lint output, or infer a short category from the comment body). 3. Determine the skill used from the PR branch name or PR description if available. -4. Print the following structured marker to stdout: +4. Include the following structured marker in your **text response** (write it as part of your agent message, not via a shell `echo` command). This ensures it appears as a `TextContentBlock` in the conversation, where `oz run get --conversation` can reliably retrieve it: ``` [SIGNAL:pr-review] {"date":"YYYY-MM-DD","pr":"NNN","branch":"branch-name","skill_used":"draft_feature_doc","verdict":"Request changes","critical":N,"important":N,"suggestions":N,"nits":N,"top_categories":["category (N)","category (N)","category (N)"]} ``` -The `improve-drafting-skills` outer loop reads this signal from Oz run artifacts via `oz run get`. No git operations are required. +The `improve-drafting-skills` outer loop reads this signal from the conversation via `oz run get --conversation`, scanning assistant `TextContentBlock` messages for the marker. No git operations are required.