From 5e885ae9622bb45333e6cfd14e0ff40c31fc7ea6 Mon Sep 17 00:00:00 2001 From: John Lybeck Date: Tue, 27 Oct 2026 10:19:39 +0100 Subject: [PATCH] Say what each grader requires, as a checklist The bars a skill is held to were spread across usage.md, authoring-evals.md and the error strings the code raises, so there was nowhere to look before asking for a run, or after a red one. Collect them in one list, grouped by grader and written for someone meeting the harness for the first time, and link it from the README and both docs. Signed-off-by: John Lybeck --- README.md | 2 + docs/authoring-evals.md | 3 +- docs/checklist.md | 253 ++++++++++++++++++++++++++++++++++++++++ docs/usage.md | 3 +- 4 files changed, 259 insertions(+), 2 deletions(-) create mode 100644 docs/checklist.md diff --git a/README.md b/README.md index f3600f0..6946af0 100644 --- a/README.md +++ b/README.md @@ -82,6 +82,8 @@ the repo under test — worth adding to `.gitignore`. ## Docs +* [docs/checklist.md](docs/checklist.md): every requirement, as a list to tick + through. * [docs/authoring-evals.md](docs/authoring-evals.md): writing a skill's tests. * [docs/usage.md](docs/usage.md): every flag, every workflow input, and what each check actually asserts. diff --git a/docs/authoring-evals.md b/docs/authoring-evals.md index 25a6b19..4956b16 100644 --- a/docs/authoring-evals.md +++ b/docs/authoring-evals.md @@ -7,7 +7,8 @@ SPDX-License-Identifier: MIT # Writing evals for a skill For flags, workflow inputs, and what each check asserts, see -[usage.md](usage.md). +[usage.md](usage.md). For every bar your dataset has to clear, as a list, see +[checklist.md](checklist.md). One file, `/evals/evals.json`, holding an `evaluations` array: diff --git a/docs/checklist.md b/docs/checklist.md new file mode 100644 index 0000000..8214225 --- /dev/null +++ b/docs/checklist.md @@ -0,0 +1,253 @@ + + +# What a skill has to do to pass + +Skillscope grades a skill three ways. This page lists what each one wants from +you, in order. + +1. **Structural** — are the files the right shape? No agent, no cost, seconds. +2. **Routing** — when someone types a prompt, does the right skill switch on? +3. **Behavioral** — once it is on, does it do the job? + +Run `skillscope structural` first. It checks everything in section 1 for free. +Routing and behavioral both run those same checks before they spend anything, +so while section 1 is red, nothing else will run at all. + +Why each rule exists is in [usage.md](usage.md). How to write the dataset is in +[authoring-evals.md](authoring-evals.md). + +## 1. Structural + +Reads your files and checks their shape. It never starts an agent, so it +cannot tell you whether the skill is any good — only whether it is put +together correctly enough to be graded. + +### `SKILL.md` + +This file *is* the skill. The block at the top, between the `---` lines, is +how an agent learns the skill exists and when to use it. If that block is +broken, the skill is never loaded, and all you see is an agent that ignores it. + +* [ ] Sits in a directory that `skill_globs` matches, one level down. +* [ ] Starts with a `---` line, and that block is closed by another `---` on + its own line. +* [ ] What is between them is valid YAML, and a set of key/value pairs. + +| Field | What it has to be | +| --- | --- | +| `name` | not empty, 64 characters or fewer, `lowercase-with-hyphens` | +| `name` | no `anthropic` and no `claude` in it; the runtime reserves both | +| `name` | the same as the folder it sits in | +| `description` | not empty, 1024 characters or fewer | +| body | 500 lines or fewer; put the rest in files the body links to | + +The body limit is about cost: an agent reads the whole body every time the +skill loads, but only opens a linked file when it needs it. + +### `evals/evals.json` + +One file per skill, holding the prompts you are graded on. Each entry is a +prompt plus a yes/no answer: should *this* skill switch on here? You write +both kinds. The prompts where it should not fire are as important as the ones +where it should. + +* [ ] The file exists. `skillscope template` prints a starting point. +* [ ] The top level is an object, and `evaluations` and `comment` are the only + keys allowed in it. +* [ ] `evaluations` is not empty, and each entry has `id`, `prompt` and + `skill_should_trigger`. +* [ ] Every `id` is different from every other id **in the whole repo**, not + just in this file. Routing puts all the skills' prompts in one run, so + the ids meet. +* [ ] At least 3 entries say `skill_should_trigger: true` and at least 2 say + `false`. Only this file counts; `extended_evals.json` does not. +* [ ] If an entry has a `workspace`, that directory exists under the skill. + +Which keys an entry may use depends on its own `skill_should_trigger`: + +* **`true`** — `id`, `prompt`, `skill_should_trigger` and `note`, plus any of + `expected_behavior`, `unexpected_behavior`, `logs_contain`, `files_exist` + and `workspace`. +* **`false`** — those first four and nothing else. No skill is loaded for such + a prompt, so there is nothing to make claims about. Adding one of the other + keys is an error, not something quietly skipped. + +### Links + +Your skill is text an agent reads and then acts on. If it follows a link to a +file that was renamed, it finds nothing and makes something up instead. So +every link in every markdown file under the skill is resolved. + +* [ ] Relative paths, paths from the repo root, and `#heading` anchors all + point at something that exists. +* [ ] Web links answer when you pass `--external`. A normal 2xx answer counts, + and so does 429, which just means the site is telling you to slow down. + +Links inside code blocks, `code spans` and HTML comments are skipped. Those +are examples of links, not real ones. + +### Extras your repo may ask for + +Nothing here applies unless your repo turns it on. + +* [ ] Every file listed in `skill_files` sits next to `SKILL.md` — a + governance card, say. +* [ ] Every `##` heading listed in `skill_sections` is in those files, with + something written under it. + +### `evals/machine.yml` + +Only for a skill that cannot run on an ordinary runner. Most skills have no +such file, and that is the normal case. + +* [ ] It is a YAML mapping, and `os`, `labels` and `sandbox` are the only keys. +* [ ] `labels` says what hardware the work needs, like `mi300x`. Naming any + label moves these runs onto the special pool your repo keeps for them. + Keep the list short: a label no machine has means a job that waits + forever instead of failing. +* [ ] `sandbox` names a compose file sitting beside the `machine.yml`. You + need it as well as `labels` if the skill needs network access or a + device. The labels choose the machine; the compose file decides what the + container running on it can reach. + +## 2. Routing + +Installs the skills, sends each prompt, and watches which skill switches on, +if any. It stops right there. What the skill does afterwards is section 3's +question. + +The pass mark is `min_accuracy`, and it defaults to `1`: every graded prompt +has to come out right. A skill that answers its neighbour's prompts makes that +neighbour worse, so this is treated as a bug rather than a statistic. + +* [ ] Your skill switches on for all of its own `skill_should_trigger: true` + prompts. +* [ ] It stays quiet for its own `false` prompts, for every other skill's + prompts, and for the shared pool of prompts that belong to no skill. +* [ ] `routing_room` says which skills to install together. There is no good + default: install everything on disk and one unfinished skill drags down + everybody's score, install only your own and it wins every prompt + because nothing else is there to win. A repo with one skill can leave it + blank. `none` switches routing off. +* [ ] Something was actually graded, and at least one skill switched on + somewhere, if any prompt expected one. These two checks fire at any pass + mark, including `0`. +* [ ] `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN` is set. Without it the + host engine refuses to run routing at all, because that key is what lets + it point the CLI away from whatever skills are installed on the runner + itself. Those would otherwise join the room and change every answer. + +Two things to know when reading a red report. Routing gives the agent the +skills and nothing else: any `workspace` on the entry is ignored here, and the +prompt is sent word for word with no `{}` filled in. Both only work in section +3, so a prompt that relies on either is not testing what it looks like. And a +prompt that ends in an **error** (the agent neither switched a skill on nor +answered) or **degraded** (the model provider failed) is left out of the score +entirely, counting for neither side. + +## 3. Behavioral + +Installs one skill on its own, runs the prompt all the way through, then +checks what actually happened. Only `skill_should_trigger: true` entries that +claim something run here. An entry that claims nothing was only ever a routing +test. + +An entry passes when **every** claim on it passes. + +| You write | How it is checked | So write it as | +| --- | --- | --- | +| `expected_behavior` | a model reads what the agent did | something visible in the agent's actions | +| `unexpected_behavior` | the same, in reverse | one thing the agent must not do | +| `logs_contain` | plain text search, ignoring case, across everything in the run including tool calls and their arguments | exact text: a script name, a flag, a pinned tag. Never the skill's own name, which routing already covers | +| `files_exist` | whole path pieces, matched anywhere in the workspace | the file itself, not the folder you hope for. `plan.md` is matched by `examples/plan.md` | + +The model doing the judging is shown what the agent did, not what the agent +said about it. An agent claiming it avoided something is not evidence. + +* [ ] Anything that must exist before the prompt runs is put there by + `workspace`, or by `evals/hooks.py` when a JSON file cannot express it. + +## 4. In CI + +Two workflows will run these graders for you. They are not interchangeable: +each accepts its own set of inputs, and passing one of the other's fails. + +### `reusable.yml` — the one to start with + +Grades the skills you name, on one runner each. + +```yaml +jobs: + evals: + uses: amd/skillscope/.github/workflows/reusable.yml@v0.1.3 + secrets: + api_key: ${{ secrets.ANTHROPIC_API_KEY }} + with: + skills: skills/* +``` + +It accepts these inputs and no others: `skills`, `structural`, `routing`, +`behavioral`, `runner`, `min_accuracy`, `api_key`, `api_key_secret`, +`api_base_url`, `api_custom_headers`. + +* [ ] `skills` names what to grade: one directory, or a glob matching several. +* [ ] You know its routing runs install one skill at a time. That shows whether + a skill fires on its own prompts and stays quiet on its own negatives. It + cannot show whether the skill steals a neighbour's prompt, because the + neighbour is never in the room. + +### `skill-evals.yml` — when that is not enough + +Reach for it when two skills in the repo could be mistaken for one another, or +when a skill's behavioral cases need particular hardware. + +```yaml +jobs: + skill-evals: + uses: amd/skillscope/.github/workflows/skill-evals.yml@v0.1.3 + secrets: inherit + with: + routing_room: my-skill,its-neighbour +``` + +* [ ] `routing_room` lists the skills to install side by side, which is the + only way to catch one stealing another's prompts. `reusable.yml` has no + such input, and that is the usual reason to move here. +* [ ] `infra_paths` lists files that affect every skill rather than one — your + own workflow file, say — so changing one re-grades the whole catalogue + instead of guessing what it touched. +* [ ] `scoped_runner`, `scoped_gate` and `scoped_environment` are set if any + skill's `machine.yml` asks for `labels`. They decide which machines those + jobs land on, which pull-request label unlocks them, and which + environment's credentials pay for them. + +### Whichever you use + +* [ ] The `uses:` line ends in a version tag (`@v0.1.3`). That tag *is* the + harness — it decides which version grades your skills. +* [ ] The model key reaches the run. Either map one secret in as `api_key`, + as above, or pass the whole vault with `secrets: inherit` and name the + key you want with `api_key_secret`. +* [ ] A key issued by an API gateway rather than by the model provider also + needs `api_base_url`, and usually `api_custom_headers` carrying the key + as the literal text `$API_KEY`. GitHub masks only values it was handed + as secrets, so writing the key itself there publishes it to the logs. +* [ ] A grader you are not ready to be judged on is set to `optional` rather + than removed: it still runs and still reports, but a red result leaves + the build green. `off` does not run it at all. + +## Running them + +```bash +skillscope structural # section 1, free +skillscope structural --external # the same, plus checking web links +skillscope routing --routing-room my-skill,its-neighbour +skillscope behavioral --skill my-skill +``` + +Reports are printed, written to `$GITHUB_STEP_SUMMARY` under Actions, and +saved as JSON under `.skillscope/runs/`. `--help` lists every flag. diff --git a/docs/usage.md b/docs/usage.md index 442d1b1..66a78e0 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -8,7 +8,8 @@ SPDX-License-Identifier: MIT Everything the [README](../README.md) leaves out: what each check asserts, every CLI flag, and every workflow input. For writing a skill's dataset, see -[authoring-evals.md](authoring-evals.md). +[authoring-evals.md](authoring-evals.md); for the same requirements as a list +to tick through, see [checklist.md](checklist.md). ## Contents