Add test-gap-audit and docs-sync-audit skills - #2951
Open
specialone0007 wants to merge 2 commits into
Open
Conversation
Two repo-agnostic review skills that answer questions the existing testing and documentation skills do not. test-gap-audit asks which behaviour is not covered, rather than how to write a test in a given framework. Given no scope it audits the whole repository, inventories the testable surfaces, and reports which routes, services, jobs and contracts have no tests, too few assertions, or only indirect coverage. It bundles coverage_map.py, which detects the test framework and naming convention, then matches every source file to tests by name, by mirrored path, and by what the test files actually import, and ranks the unmatched by risk keyword and size. docs-sync-audit compares what the docs claim against what the code does. It bundles docs_drift.py, which checks documented npm scripts and make targets against the ones that exist, relative Markdown links against the filesystem, and environment variable names in both directions. It also reports a documented setting that is read only inside a module nothing imports, which is configuration that reads as working but cannot take effect. Both are read-only: they report and do not edit unless asked. Both emit the same contract, so a finding always carries a P0-P3 severity and a path:line you can open. Both scripts are Python standard library only, install nothing, and are accelerators rather than requirements, so each skill still works when the script cannot run. The existing testing and docs skills here are framework-specific, which is where most of the value is. These are the repo-agnostic complement: pytest-coverage raises coverage inside a pytest project, and this decides where coverage is missing across a repository regardless of language.
github-actions
Bot
added
new-submission
Contributor
🔒 PR Risk Scan ResultsScanned 5 changed file(s).
|
Contributor
🔍 Vally Lint Results✅ All checks passed
Summary
Full linter output |
Two CI failures on the first push. codespell flagged `testng` and `shouldBe` in coverage_map.py. Both are legitimate identifiers rather than typos: TestNG is the Java test framework the script detects by name, and shouldBe is the Kotlin and Scala assertion method matched by its assertion-detection regex. Added both to ignore-words-list with a comment each, following the convention already used for the other entries. validate-readme failed because I had reverted docs/README.agents.md. `npm start` rewrites a Dynatrace MCP URL there from re-fetched external plugin data, which is unrelated to these skills, so I had excluded it to keep the diff scoped. That was wrong: the check regenerates every generated file and compares, so the commit has to carry whatever the build produces. Restored.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two repo-agnostic review skills that answer questions the existing testing and documentation skills here don't.
What they do
test-gap-auditasks which behaviour is not covered, rather than how to write a test in a given framework. Given no scope it audits the whole repository: inventories the testable surfaces, then reports which routes, services, jobs and contracts have no tests, too few assertions, or only indirect coverage. Name a feature, PR or branch and it narrows to that.It bundles
coverage_map.py, which detects the test framework and naming convention, then matches every source file to tests three ways — by name, by mirrored path, and by what the test files actually import — and ranks the unmatched by risk keyword and size. The import scan is what makes "no tests found for X" worth reporting rather than merely plausible.docs-sync-auditcompares what the docs claim against what the code does. It bundlesdocs_drift.py, which checks documentednpm runscripts andmaketargets against the ones that exist, relative Markdown links against the filesystem, and environment variable names in both directions.It also reports a case a name-only comparison misses: a documented setting that is read by the code, but only inside a module nothing imports — configuration that reads as working and cannot take effect.
How they relate to what's already here
The existing testing and docs skills are framework-specific, which is where most of their value is. These are the repo-agnostic complement rather than an overlap:
pytest-coverageraises coverage to 100% inside a pytest project.test-gap-auditdecides where coverage is missing across a repository, in any language, and does not run tests.documentation-writerandcreate-readmewrite documentation.docs-sync-auditchecks whether existing documentation still matches the code.Neither restates something a frontier model already does well unprompted. The uplift is the discipline: a fixed report contract, and deterministic tooling for the parts a model does expensively and unreliably by hand — matching hundreds of source files to tests, or re-deriving which documented commands actually exist.
Conventions followed
P0–P3severity and apath:lineyou can open.npm run skill:validatepasses (418/418), andnpm starthas been run — the only generated change is the two new rows indocs/README.skills.md.One note on the build:
npm startalso rewrites a Dynatrace MCP URL indocs/README.agents.md, which comes from re-fetching external plugin data and is unrelated to this change. I reverted that file so this PR stays scoped to the skills.