volen-silo · Oct 29, 2026 · Oct 29, 2026 · Oct 29, 2026 · Oct 29, 2026 · Oct 29, 2026
diff --git a/README.md b/README.md
@@ -85,6 +85,8 @@ the repo under test — worth adding to `.gitignore`.
 * [docs/authoring-evals.md](docs/authoring-evals.md): writing a skill's tests.
 * [docs/usage.md](docs/usage.md): every flag, every workflow input, and what
   each check actually asserts.
+* [docs/harnesses.md](docs/harnesses.md): what an agent harness is, and how to
+  add one.
 
 ## Development
 

diff --git a/docs/harnesses.md b/docs/harnesses.md
@@ -0,0 +1,133 @@
+<!--
+Copyright Advanced Micro Devices, Inc.
+
+SPDX-License-Identifier: MIT
+-->
+
+# Harnesses
+
+A **harness** is the agent that is handed a prompt and a workspace and does the
+work: Claude Code, GitHub Copilot CLI, Codex, and so on. It is a different
+question from where the agent runs. `--engine` says *where* (on the host, or in
+a container); the harness says *what*. The engines drive a harness as a
+subprocess, so the contract between them is small, and this page is that
+contract.
+
+Two harnesses are registered, `claude-code` and `copilot`, and the contract exists
+so that another is an adapter and a set of tests rather than a change to the
+engines. `--harness` chooses one; see [usage.md](usage.md#which-engine-grades-a-run).
+
+## The harnesses
+
+| | `claude-code` | `copilot` |
+| --- | --- | --- |
+| Binary | `claude` | `copilot` |
+| Skills directory | `.claude/skills` | `.claude/skills` |
+| A skill activates by | a `Skill` tool call | a `skill` tool call |
+| Session lists its skills | yes (`system/init`) | no; `copilot skill list --json` does, without a model call |
+| Authenticates with | `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN` | `COPILOT_GITHUB_TOKEN`, `GH_TOKEN` or `GITHUB_TOKEN` |
+| Config isolated by | `CLAUDE_CONFIG_DIR` | `COPILOT_HOME` |
+| Prompt | stdin | stdin |
+| A run is billed in | dollars | premium requests |
+
+Things to know about `copilot` before reading its numbers:
+
+* **Its room always contains built-in skills.** Copilot ships a few of its own,
+  so they are present beside the skills a run stages and can win a case.
+* **It reports its answer separately from its closing event.** The harness
+  remembers the last assistant message and attaches it to the end of the run.
+* **It reports tokens once, at the end,** and no dollar cost, so a spend cap in
+  dollars cannot bind.
+* **It runs unsandboxed with every permission granted** (`--allow-all`), as the
+  host engine's Claude Code does, because a non-interactive run cannot answer a
+  prompt. Use a throwaway runner.
+* **It uses its own model names** (`claude-sonnet-5.5`, not `sonnet`), and
+  models differ greatly in how many premium requests one run costs.
+
+## What a harness provides
+
+A harness is a subclass of `skillscope.harness.Harness`, registered in
+`skillscope/harness/__init__.py`.
+
+| Member | What it is |
+| --- | --- |
+| `name` | The name reports and `--harness` use. |
+| `binary` | The executable looked up on `PATH`. |
+| `skills_dir` | Where, relative to the workspace, the harness discovers skills (`.claude/skills` for Claude Code). The engine stages each skill there. |
+| `has_skill_tool` | Whether activating a skill is a tool call in the stream. A harness that activates a skill by reading its `SKILL.md` sets this `False`, and routing reads activation from the file access instead. |
+| `prompt_on_stdin` | Whether the prompt is written to stdin. A harness that takes it as an argument builds it into `command` and sets this `False`; stdin is closed either way, because some CLIs wait on it. |
+| `require()` | Refuse to start when the harness is not installed. The default checks `binary` is on `PATH`. |
+| `default_model`, `resolve_model(model)` | The model used when the caller names none, and how the harness names the one they did. A harness with its own names maps the generic aliases (`opus`, `sonnet`, `haiku`) onto them. |
+| `pin_model(model)` | The model a CI run is held to, so results stay comparable run to run. The default pins nothing. |
+| `credentials`, `can_isolate_room(environ)` | The environment variables a credential can come from. A routing run keeps the runner's own config and skills out of the session only when the harness authenticates from one of them. |
+| `supports_sandbox` | Whether a container leg exists (`--engine sandbox`). |
+| `cost_flags(max_budget_usd)` | The CLI's own spend controls for a case. Empty when it has none, which the report then says rather than implying a cap. |
+| `version()` | What `<binary> --version` prints, for the report. |
+| `preflight(model)` | One trivial prompt through the harness's own command, to prove before any case runs that the credential, the model name and the network work. The default is enough for most harnesses. |
+| `command(spec)` | The argv that runs one case, headless, streaming JSON events. |
+| `env(spec)` | The environment of that process, including whatever points it at `spec.config_dir` so the runner's own settings and skills stay out of the session. |
+| `normalize(event)` | Turns one event the harness printed into canonical events (below). Stateless. |
+| `normalizer()` | A translator for one run's events, which may keep state. The engine asks for a fresh one per case. Override it when an event cannot be translated on its own, for instance when the closing event does not repeat the answer and it has to be remembered from an earlier one. The default returns `normalize`. |
+
+`spec` is a `RunSpec`: the workspace, the prompt, the model and effort, an
+optional throwaway config directory, and any spend or session flags the caller
+built.
+
+## The canonical event form
+
+The engines read one event shape, and a harness translates into it. It is
+Claude Code's stream-json, reduced to what is read:
+
+| Event | Carries |
+| --- | --- |
+| `{"type": "system", "subtype": "init", "skills": [...]}` | The skills the session registered. Optional: a harness that cannot say reports nothing, and the room check says it did not run rather than reporting an empty room. |
+| `{"type": "assistant", "message": {"content": [{"type": "tool_use", "name": ..., "input": {...}}]}}` | A tool call. Activating a skill is a `tool_use` named `Skill` with `input.skill` set. |
+| `{"type": "user", "message": {"content": [{"type": "tool_result", "content": ...}]}}` | A tool result. |
+| `{"type": "result", "result": "..."}` | The end of the run, and its closing answer. A stream without one is a run that did not finish. |
+
+A harness whose stream already has this shape keeps the default `normalize`,
+which passes events through. Any other maps its events onto these and returns an
+empty list for events that carry nothing the engines read. What a harness remembers about a stream belongs in the
+translator `normalizer()` returns, never on the harness itself: one harness
+instance serves every case in a run, and cases run concurrently. Every stop rule,
+budget, score and judge sees canonical events only, never the raw ones, so each
+is written once for every harness.
+
+## Adding a harness
+
+1. Subclass `Harness` in `skillscope/harness/<name>.py`, fill in the members
+   above, and register it in `skillscope/harness/__init__.py`.
+2. Record two real runs of the CLI, headless, on a trivial skill: one where it
+   activates the skill and finishes, one where it finishes without activating
+   one. Scrub tokens and paths. Put them in
+   `tests/fixtures/harness/<name>/trigger.jsonl` and `no_trigger.jsonl`, with an
+   `expected.json` saying what the engine should read out of each (see the
+   Claude Code one) and the CLI version they were recorded with.
+3. Run `python -m unittest tests.test_harness`. The contract test runs once per
+   registered harness and fails if there are no fixtures. It checks activation,
+   completion, the closing answer, that the command carries the model and effort
+   and no credential, that the config directory is isolated, that two runs translated at the same time
+   do not share memory, and that a skill is staged where the harness looks and
+   without its `evals/` directory.
+4. Say in the pull request how the harness differs from the others in ways a
+   reader of a report should know: how it activates skills, what it cannot
+   report, and what it needs to authenticate.
+
+A report from one harness is its own result. Harnesses send different system
+prompts, offer different tools, and activate skills in different ways, so a score
+from one is not comparable with a score from another.
+
+## What is not yet behind the contract
+
+The contract covers staging, the command, the environment, the events, the model,
+the credential check, the spend controls and the preflight. These still assume
+Claude Code, and a harness other than it cannot yet use them:
+
+* **the judge** that grades `behavioral` expectations goes through an Anthropic
+  model whatever agent produced the run, so `behavioral` is refused on any other
+  harness;
+* **the room check**: Claude Code reports at session start which skills it
+  loaded, and the engine compares that with what was staged. A harness with no
+  such event leaves the check not run;
+* **`.claude/skills`** as the path routing treats as a skills inspection, rather
+  than a harness's own `skills_dir`.
diff --git a/docs/usage.md b/docs/usage.md
@@ -249,18 +249,39 @@ environment gets one matrix, labels and all.
 
 ## Which engine grades a run
 
-`--engine` chooses what actually runs the cases. The dataset, the CLI and the
-reports are identical whichever you pick; only the thing driving the agent
-changes.
+Two choices decide what runs the cases, and they are separate: `--engine` is
+*where* the agent runs, and `--harness` is *which* agent. The dataset and the
+reports are identical whichever you pick.
 
-| `--engine` | What runs | Runs in | Needs |
-| --- | --- | --- | --- |
-| `claude-code-no-sandbox` (default) | The real CLI, under `inspect_ai` | the host | the CLI on `PATH` |
-| `claude-code` | The real CLI, via `inspect_swe` | a sandbox | `skillscope[verify]`, Linux only |
+| `--engine` | Where it runs | Needs |
+| --- | --- | --- |
+| `host` (default) | the host, under `inspect_ai` | the harness's CLI on `PATH` |
+| `sandbox` | a container, via `inspect_swe` | `skillscope[verify]`, Linux only, and a harness with a sandboxed leg |
 
-Both drive the agent a skill is written for, so what differs between them
-is **where the agent runs**, not what it is. That is the axis worth choosing
-along: the host measures the machine as it is, with whatever else is installed
+`claude-code-no-sandbox` and `claude-code` are the older spellings of `host` and
+`sandbox`. They name an agent as well as a place, so they mean Claude Code, and
+combining one with a different `--harness` is refused.
+
+| `--harness` | Agent | Sandboxed leg |
+| --- | --- | --- |
+| `claude-code` (default) | Claude Code | yes |
+| `copilot` | GitHub Copilot CLI | no |
+
+`--harness` and `--engine` can also come from `$SKILLSCOPE_HARNESS` and
+`$SKILLSCOPE_ENGINE`. Each harness names models its own way and has its own
+default; `--model opus`, `sonnet` and `haiku` mean the right thing for every
+harness, and CI holds each harness to one model. A report from one harness is its
+own result and is not comparable with another's. How a harness works, and how to
+add one, is in [harnesses.md](harnesses.md).
+
+**`copilot` today runs routing only.** It authenticates from
+`COPILOT_GITHUB_TOKEN`, `GH_TOKEN` or `GITHUB_TOKEN`, needs no model credential,
+and in Actions needs `permissions: copilot-requests: write` in the calling
+workflow. `behavioral` is refused on it until the judge that grades its
+expectations is available through the harness. Its preflight sends one real
+prompt, which is billed like any other.
+
+Where the agent runs is the axis worth choosing along: the host measures the machine as it is, with whatever else is installed
 on it, and the sandbox measures the skill alone. When the two disagree, the
 disagreement is usually a fact about one of those environments rather than
 about the skill -- which is a thing one engine on its own cannot tell you.

diff --git a/pyproject.toml b/pyproject.toml
@@ -62,7 +62,7 @@ verify = ["skillscope[inspect]", "inspect-swe>=0.2.71"]
 skillscope = "skillscope.cli:main"
 
 [tool.setuptools]
-packages = ["skillscope", "skillscope.engine"]
+packages = ["skillscope", "skillscope.engine", "skillscope.harness"]
 
 [tool.setuptools.package-data]
 skillscope = ["data/*.json", "schema/*.json"]