chore: sharpen doc agent skills and add navigation audit - #383
Merged
Conversation
Reviewed the last 20 non-bot PRs (14 touched docs) against what the doc-review and doc-write skills actually ask for. doc-review found 0 Critical, 5 Improvement and ~20 Suggestion findings across those 14 PRs. Its genuinely valuable catches were all cross-file consistency checks the skill never asked for: a changelog entry documenting a nonexistent `kosli update attestation-type` (#371), a flow_template page out of sync with the schema the same PR regenerated (#296), and the one file an approvals-removal sweep missed (#345). Its weakest findings were prose polish, some already fixed at branch head. Changes: - doc-review: promote cross-file consistency, placement, redirects and anchor stability to named first-class checks; add an explicit "what not to report" bar and an 8-finding cap; stop hand-checking spelling, which the Mintlify vale-spellcheck check already enforces; drop the "what looks good" recital that the sticky comment re-renders on every push. Note that link-rot reports `skipping`, so internal links still need verifying by hand. - doc-write: add a Diátaxis-to-tab placement table. A GitHub Action reference page was authored into integrations/ and a human had to ask for the move to Reference (#305). Also add the redirects rule (3 of 14 PRs needed one), anchor stability (the CLI prints faq#boolean-flags in an error), and a table of generated paths never to hand-edit. - Both: read config/navigation.json, not docs.json, which holds only a $ref to it. - New doc-structure skill + monthly workflow: audits navigation shape and changelog coverage, which per-PR review structurally cannot see, and files issues. Its keyword check reproduces a real gap — /getting_started/attestations is the changelog's most-linked page and contained no occurrence of "summary" while three products shipped the feature (already tracked as #364, which is why the skill deduplicates against open issues before filing). - CLAUDE.md: document the automated PR checks, list the three doc skills, and drop the reference to a changelog-creator skill that does not exist.
doc-review runs on the automated `docs: update CLI and helm reference` PRs (allowed_bots includes github-actions), whose diffs are almost entirely machine-generated. Its behavior there is a coin flip. PR #375 got it right and said so: "they must be fixed in the CLI's command long-descriptions in kosli-dev/cli - editing them here would be silently reverted... I've deliberately not included 'Fix this' links." PR #374 got it wrong. It wrote "the durable fix is in the kosli-dev/cli generator - a hand-edit here is overwritten by the next release" and then, in the next sentence, emitted a "Fix this" link scoped to repo=kosli-dev/docs telling an agent to edit line 87 of the generated page. It also left 4 inline comments on regenerated files. PR #378 left another. Those line anchors are gone after the next release. Adds a "Generated pages" section to doc-review that distinguishes two failure modes rather than banning edits everywhere: - Deterministically regenerated (client_reference/kosli*.md, helm/k8s_reporter/*.mdx, schemas/, the kosli * nav groups, live-docs sections) - a hand-edit is deleted. No inline comments, no "Fix this" links scoped to this repo; report under "Upstream - does not block this merge" naming the upstream file. Never Critical: nothing in the PR under review can fix it. - Agent-synced (terraform-reference/, github-action-reference/) - an edit survives but drifts from upstream. Report both. - Hand-authored despite the directory (client_reference/overview.md, output_and_verbosity.md) - review normally. Regeneration only removes kosli*.md. Includes the filename-to-source mapping, verified against upstream: kosli_attest_sonar.md <- cmd/kosli/attestSonar.go, helm/k8s_reporter/karpenter.mdx <- charts/k8s-reporter/mintlify/ karpenter.md.gotmpl. Also records that `^` is the CLI's backtick convention in Go long descriptions (^--jq^, ^jq^), substituted by `kosli docs`. That makes the ^jq^ defect PR #374 found a generator escaping bug in the Accordion-title path, not a typo in the Go string - so a reviewer can now tell the two apart and point at the right fix. Same distinction applied to doc-write, whose table was too blunt, and to CLAUDE.md's Live docs section, which said "find the source script first" when for client_reference the source is another repo.
Counts and current-state claims go stale as PRs land, and a skill that cites a stale statistic teaches the wrong thing while looking authoritative. Swept all three skills and CLAUDE.md for them. - Redirects check no longer says "three of the last fourteen doc PRs needed one" - states the rule and why it gets forgotten instead. - link-rot guidance no longer asserts "reports skipping on most PRs". Now tells the reviewer to check `gh pr checks` and assume nothing validated the links unless it says otherwise, so it self-heals if Mintlify starts running it reliably. - doc-structure's changelog-coverage example moved to past tense: the attestation-summaries gap is a worked example of the check firing, not a claim about the page's current state or its current link rank. - doc-write no longer asserts the Tutorials group holds mostly how-to guides - that is exactly the kind of thing doc-structure exists to get fixed. Tells the author to read the group's pages first and follow the convention they set. The past-tense `Precedent:` items are kept as-is. A thing that happened stays having happened, and they are what make the checks concrete.
The doc-structure skill carried its mechanical checks as an inline python3 -c heredoc plus a find/sed/comm pipeline. Moving them into scripts/ alongside the other docs tooling buys four things: 1. Tests. pr-quality.yml already runs `pytest tests/`, so tests/test_audit_navigation.py (22 cases) pins the behavior. One of them asserts the committed navigation has integrity, which means CLAUDE.md core rule 2 - never create a page file without adding it to navigation - is now enforced deterministically on every PR instead of resting on an LLM noticing. No new workflow job needed. 2. A narrower blast radius. doc-structure.yml can allow `Bash(python3 scripts/audit_navigation.py:*)` instead of `Bash(python3:*)`, which was arbitrary code execution, and drop find/comm entirely. 3. Determinism. The same check runs identically locally, in the cron job, and in tests, rather than being retyped by a model each run. The inline version had already shipped one such bug: `sed 's|\.mdx\?$||'` is a no-op on BSD sed, so every page looked orphaned on macOS. That case is now a regression test. 4. Reuse. `--check` for CI, `--json` for the skill, readable report for humans, `--max-nesting` / `--max-group-children` to tune. Integrity findings (orphans, dangling entries) are separated from shape findings (single-child groups, deep nesting, Title Case labels, oversized groups, inconsistent icons). Only integrity can fail a build; shape is advisory, because the script cannot tell a group that should be merged from one deliberately kept separate. The Reference > CLI Reference subtree is exempt from shape checks: it is generated by update-cli-nav.py from the CLI's own command tree, so a single-child `kosli allow` group is upstream truth rather than a defect. Without that exemption the audit reported 54 findings, 21 of them proposing to reshape generated navigation - the same mistake just fixed in doc-review. It now reports 33, all hand-maintained. Writing the tests also caught a false positive in the sentence-case check: "GitHub Actions" flagged on "Actions" because only "GitHub" was allowlisted. Added multi-word product-name handling.
Restructure findings have a shape the existing backlog has no precedent
for. Coverage issues ("document X") look like #364, #169, #167 and need
no scaffold; a navigation change needs current structure, proposed
structure, reader impact, and - the part that gets missed - whether any
URLs change.
Markdown, not a YAML form, deliberately. `gh issue create --body` does
not apply templates mechanically, and YAML issue forms cannot be filled
from the CLI at all, so a form would help humans in the web UI and do
nothing for the cron job. A markdown template is something the agent can
read and follow, and humans still get it in the chooser. The skill and
the workflow prompt both say to read it and reproduce its sections.
The template's URL impact section encodes the distinction that decides
whether a restructure is safe:
- renaming a group label -> no URL change
- re-ordering within a group -> no URL change
- collapsing a group wrapper -> no URL change if the page file stays put
- moving or renaming a page -> URL changes, needs config/redirects.json
Its Out of scope section names the generated subtrees, so a proposal
cannot drift into reshaping `Reference > CLI Reference`.
Also tells the job to cluster before filing. The current audit reports 33
shape findings that reduce to about four underlying issues - Implementation
Guide alone appears under three finding kinds across both its phases.
Filing one issue per line would bury the signal.
Adds ISSUE_TEMPLATE/config.yml with blank_issues_enabled: true. The first
template makes GitHub show a chooser and de-emphasise blank issues, and
this backlog has plenty of one-line entries (#115, #134, #80) that should
stay easy to file.
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
dangrondahl
enabled auto-merge (squash)
September 3, 2026 11:06
AlexKantor87
approved these changes
Sep 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reviewed the last 20 non-bot PRs (14 touched docs) against what
doc-writeanddoc-reviewactually ask for, then fixed what the record showed was wrong. No site content changes — this is all agent tooling.Why
doc-reviewproduced 0 Critical, 5 Improvement and ~20 Suggestion findings across those 14 PRs. Its genuinely valuable catches were all cross-file consistency checks the skill never asked for:kosli update attestation-type, a command with no reference page (chore: fix ordering and a typo #371)template-reference/flow_template.mdout of sync with the schema the same PR regenerated (feat: generate flow-template and policy JSON schemas from the API #296)understand_kosli/how_kosli_works.md:22(docs: remove retired approvals from the docs #345)Its weakest findings were prose polish, some already fixed at branch head. Meanwhile its written checklist was dominated by things that always pass or are already enforced by
vale-spellcheck.Two structural gaps the PR record made obvious:
integrations/, and a human reviewer had to ask for the move — commit9e7a121 docs: move GitHub Action reference into the Reference section. That question should come from the review.kosli-dev/cligenerator — a hand-edit here is overwritten by the next release" and then emitted aFix thislink scoped torepo=kosli-dev/docstelling an agent to edit the generated file anyway, plus 4 inline comments on regenerated files.What changed
doc-review— promoted the three accidental wins to named checks, each carrying its precedent. Added placement, redirects and anchor stability. Added an explicit what not to report bar and an 8-finding cap. Stopped hand-checking spelling. Dropped the "what looks good" recital that the sticky comment re-renders on every push.Generated pages — split into three categories rather than one blanket ban: deterministically regenerated (edit is deleted), agent-synced (edit survives but drifts), and hand-authored despite the directory (
client_reference/overview.md,output_and_verbosity.md). Includes the filename→source mapping, verified upstream:kosli_attest_sonar.md←cmd/kosli/attestSonar.go.Also records that
^is the CLI's backtick convention in Go long descriptions, substituted bykosli docs. That makes the^jq^defect #374 found a generator escaping bug in the Accordion-title path, not a typo in the Go string — so a reviewer can point at the right fix. Worth filing upstream separately.doc-write— added a Diátaxis→tab placement table so #305 can't recur, plus redirects, anchor stability, and the generated-paths table.New
doc-structureskill + monthly workflow — audits navigation shape and changelog coverage, which per-PR review structurally cannot see, and files issues. Read-only against docs; issues are its only write. Capped at 8 issues, deduplicating against open issues first — its headline check reproduces the/getting_started/attestationssummary gap, which is already open as #364.New
scripts/audit_navigation.py+ 22 tests — the audit's mechanical checks, extracted from an inline heredoc. Three payoffs:pr-quality.ymlalready runspytest tests/, so CLAUDE.md core rule 2 is now enforced deterministically — a page file with nonavigationentry fails the build. No new job needed.doc-structure.ymlcan allowBash(python3 scripts/audit_navigation.py:*)instead ofBash(python3:*), which was arbitrary code execution.sed 's|\.mdx\?$||'is a no-op on BSD sed, so every page looked orphaned on macOS. That case is now a regression test.Integrity findings (orphans, dangling entries) are separated from shape findings (single-child groups, deep nesting, Title Case labels, oversized groups, inconsistent icons). Only integrity can fail a build — the script cannot tell a group that should be merged from one deliberately kept separate.
Reference ▸ CLI Referenceis exempt from shape checks:update-cli-nav.pygenerates it from the CLI's command tree, so a single-childkosli allowgroup is upstream truth. Without that exemption the audit reported 54 findings, 21 of them proposing to reshape generated navigation — the same mistake being fixed indoc-review. It now reports 33, all hand-maintained.New
docs-restructureissue template — Markdown, not a YAML form:gh issue create --bodydoesn't apply templates mechanically and YAML forms can't be filled from the CLI, so a form would help humans and do nothing for the job. Its URL-impact section encodes the distinction that decides whether a restructure is safe — a group rename changes no URLs, a moved page file needs aconfig/redirects.jsonentry.config.ymlkeeps blank issues enabled.CLAUDE.md— documented the six automated PR checks, the generated-page source map, the audit script, and the three skills. Removed the pointer to achangelog-creatorskill that doesn't exist in this repo.Anti-rot pass
Swept all three skills for dated claims. Counts ("three of the last fourteen PRs") and current-state assertions ("link-rot reports skipping on most PRs") became durable rules. The link-rot guidance now routes through
gh pr checksso it self-heals if Mintlify starts running it reliably. Past-tensePrecedent:items were kept — they're what make the checks concrete.Verification
python3 -m pytest tests/— 44 passed (22 pre-existing + 22 new)python3 scripts/audit_navigation.py --check— exit 0, integrity cleanmint broken-links— no new broken linksFollow-ups, not in this PR
tutorials/working_with_controls.mdx:24links to/getting_started/service-accounts, which has never existed — the page is/administration/authentication/service_accounts. Broken onmainsince docs: add working with controls tutorial #154;link-rotreportsskipping, which is why it went unseen.mintlify-docsplugin inkosli-pluginsstill ships near-duplicatedoc-writer/doc-revieweragents that say "updatedocs.jsonnavigation" — the pre-config/layout. Needs its own PR there.python3indoc-review.yml's--allowedToolsto validate JSON payloads and fence balance. Left alone —Bash(python3:*)is a security-surface call worth making deliberately.