Rebuild the eval engine on inspect_ai and retire the legacy runner

.github/workflows/selftest.yml

@@ -23,6 +23,24 @@ permissions:
   contents: read
 
 jobs:
+  imports:
+    # One rule, not a linter. `F401` finds imports nothing uses, which is how
+    # dead code announces itself: two rounds of review found functions whose
+    # last caller had been deleted, and in both cases the orphaned imports were
+    # sitting there saying so. Twelve of them had accumulated by the time
+    # anybody looked.
+    #
+    # Deliberately narrow. Adopting a full style configuration is a change to
+    # every file and a decision about house style; this is the one check that
+    # pays for itself immediately and reformats nothing.
+    name: Unused imports
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/ruff-action@v3
+        with:
+          args: check --select F401 skillscope tests tools
+
   unit:
     name: Unit tests (${{ matrix.os }}, Python ${{ matrix.python }})
     runs-on: ${{ matrix.os }}
@@ -44,6 +62,77 @@ jobs:
       - name: Run the suite
         run: python -m unittest discover -s tests -t . --verbose
 
+  sandbox:
+    name: Sandbox machinery (${{ matrix.os }})
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        # Both sandboxes a run can get. Ubuntu has Docker, which is the
+        # default and what CI uses; Windows has none, which is the other
+        # supported shape. The Windows leg is the only place the cross-platform
+        # sandbox helpers are exercised at all -- inspect's own assume a POSIX
+        # guest, and a scorer that cannot list the guest cannot grade it.
+        os: [ubuntu-latest, windows-latest]
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          # inspect-ai pulls in the order of eighty packages, so the download
+          # is most of this step. Keyed on pyproject, which is where the extra
+          # is declared and the only thing that changes what gets installed.
+          cache: pip
+          cache-dependency-path: pyproject.toml
+
+      - name: Install with the inspect extra
+        run: python -m pip install --upgrade pip && python -m pip install ".[inspect]"
+
+      # The case seeds a file and expects that same file. Nothing an agent
+      # does can satisfy it and nothing an agent does can break it, so what
+      # passes or fails is the layer underneath: the sandbox started, the
+      # fixture was staged into it, the guest was listed, and the listing was
+      # matched. That is where every cross-platform bug so far has been.
+      - name: Build a repo to test
+        shell: bash
+        run: |
+          set -euo pipefail
+          mkdir -p fixture/demo-skill/evals fixture/demo-skill/fixtures
+          cat > fixture/demo-skill/SKILL.md <<'EOF'
+          ---
+          name: demo-skill
+          description: Does demonstrable things, for a test that needs a skill.
+          ---
+          EOF
+          echo "seeded by the case, not produced by the agent" \
+            > fixture/demo-skill/fixtures/seeded.txt
+          cat > fixture/demo-skill/evals/evals.json <<'EOF'
+          {
+            "evaluations": [
+              {"id": "demo-a", "skill_should_trigger": true, "prompt": "do the demo thing",
+               "workspace": "fixtures", "files_exist": ["seeded.txt"]},
+              {"id": "demo-b", "skill_should_trigger": true, "prompt": "another way to ask"},
+              {"id": "demo-c", "skill_should_trigger": true, "prompt": "a third phrasing"},
+              {"id": "demo-d", "skill_should_trigger": false, "prompt": "something adjacent"},
+              {"id": "demo-e", "skill_should_trigger": false, "prompt": "something else entirely"}
+            ]
+          }
+          EOF
+
+      # No key, no model, no agent: a stub solver stands in for the agent so
+      # only the machinery beneath it can pass the case. That is what makes
+      # this runnable on a pull request from a fork, which is exactly where a
+      # cross-platform sandbox bug shows up.
+      - name: Exercise the sandbox, the staging and the listing
+        # The expected provider is supplied per platform rather than read back
+        # from the harness. Ubuntu must get a real container; a leg that
+        # quietly fell back to `local` would pass every check below against the
+        # host filesystem and stay green under the name "Sandbox machinery".
+        run: >
+          python tools/sandbox_smoketest.py fixture --skill demo-skill
+          --expect-sandbox ${{ matrix.os == 'windows-latest' && 'local' || 'docker' }}
+
   action:
     name: Action against a throwaway repo
     runs-on: ubuntu-latest

.gitignore

@@ -11,5 +11,10 @@ dist/
 # Reports and kept transcripts, written into the repo under test
 .skillscope/
 
+# inspect's own run transcripts, wherever a tool was invoked from. Committing
+# one is easy to do by accident and they are large, opaque and reproducible.
+*.eval
+tools/logs/
+
 # Nested checkout of this repo, used by the reusable workflows in CI
 .skillscope-action/

README.md

@@ -36,9 +36,17 @@ uv tool install --system-certs git+https://github.com/amd/skillscope
 skillscope structural                        # no agent, no tokens
 skillscope structural --external             # the same, plus checking external URLs
 skillscope behavioral --skill my-skill       # needs an authenticated `claude` CLI
-skillscope routing --routing-room my-skill,its-neighbour
+skillscope routing --routing-room my-skill,its-neighbour  # also needs a key in the environment
 ```
 
+`routing` needs `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN` set, and refuses
+to start without one — a `claude login` on its own is not enough. The key is
+what lets the run point the CLI at its own config directory; without that, the
+runner's own `~/.claude/skills` join the room for every case and the scores
+describe a room nobody asked for. `behavioral` installs one skill and grades
+what the agent produced, so a stray skill is at worst noise there, and a plain
+`claude login` is fine.
+
 ## Run it from your CI
 
 Simply add an `evals.json` file to your skill and a workflow that points to our reusable workflow. This will triger all graders to run directly from your repo.
@@ -85,9 +93,11 @@ python -m pip install -e .
 python -m unittest discover -s tests -t .
 ```
 
-Standard library only, apart from PyYAML for frontmatter, so a graded run
-installs nothing beyond this package. Tests build a throwaway repo in a temp
-directory rather than reading the tree they run in.
+Runtime dependencies are PyYAML for frontmatter, `inspect-ai` for the eval
+framework both engines are built on, and `anthropic` for the model client.
+`inspect-ai` became required when the last stdlib-only engine was removed; the
+test suite still runs on the standard library alone, and builds a throwaway
+repo in a temp directory rather than reading the tree it runs in.
 
 ## License
 

docs/authoring-evals.md

@@ -152,14 +152,49 @@ Setup a dataset cannot express: cloning a repo, tearing down a container,
 running an external scoring script. Every function is optional:
 
 ```python
-def setup_session(cache_dir): ...     # once per skill; returns {name: value} for {placeholders} in prompts
-def setup(workspace, case, ctx): ...  # before each case; may return more placeholders
-def teardown(workspace, case, ctx): ...
-def check(run, case, ctx): ...        # after each case; raise AssertionError to fail it
+def setup(workspace, case, ctx): ...     # before each case
+def teardown(workspace, case, ctx): ...  # after it, even if the agent blew up
 ```
 
-`teardown` runs even when the agent itself blew up, and a `check` that raises
-fails its case without killing the run.
+`teardown` runs even when the agent itself blew up: it is wired to inspect's
+`Task.cleanup`, which runs inside a `finally` under a shielded cancel scope, so
+an exception or a cancelled sample does not skip it. The one exception is
+`--timeout` expiring: that kills the process outright, and nothing shielded
+survives it. Keep a case inside its own `--case-timeout` if a teardown is
+load-bearing.
+
+The three arguments:
+
+| | |
+|---|---|
+| `workspace` | A fresh per-case directory **on the host**, removed after `teardown`. Yours to write in. |
+| `case` | The `Case` being run — `case.id`, `case.prompt`, and the rest of the dataset row. |
+| `ctx` | Always `{}`. It carried `setup_session`'s return value, and that entry point is gone. |
+
+`workspace` is on the host under every engine, including `claude-code`, where
+the agent itself works inside a container. It is scratch space for the hook's
+own use — fetching a source tree, holding a scoring script — not the agent's
+room. A hook that needs to put a file where the *agent* will see it should use
+[inspect's sandbox API](https://inspect.aisi.org.uk/sandboxing.html), which can
+address the guest; writing to `workspace` will not reach it.
+
+**Two entry points are no longer supported**, and a skill that defines either
+is refused rather than having it silently skipped:
+
+```python
+def setup_session(cache_dir): ...     # NOT supported
+def check(run, case, ctx): ...        # NOT supported
+```
+
+`check` was handed the retired engine's `Run` object; the engines here have
+inspect's `TaskState`, which is a different thing with different attributes.
+`setup_session` returned `{name: value}` pairs substituted into prompts as
+`{placeholders}`, and prompts are built before any hook runs -- so honouring it
+would mean constructing every case after the hook rather than before. A `setup`
+that *returns* placeholders is refused at runtime for the same reason.
+
+If you were relying on either, move the work into `setup`/`teardown`, or put
+the value in the dataset.
 
 Keep prompts and expectations in the dataset even when you use hooks, so what
 is being asserted stays readable without opening Python.
@@ -170,9 +205,9 @@ for it rather than cloning:
 ```python
 from skillscope import sources
 
-def setup_session(cache_dir):
-    source = sources.resolve("my-skill", cache_dir)
-    return {"repo": source.path}
+def setup(workspace, case, ctx):
+    source = sources.resolve("my-skill", workspace)
+    # Use source.path here; returning it as a {placeholder} is not supported.
 ```
 
 `resolve` answers with the checkout that matches the skill under test: an

docs/usage.md

@@ -60,6 +60,45 @@ whole command. Routing adds `--jobs`, `--case-timeout`, `--max-tool-calls`,
 `--max-budget-usd`, `--keep-logs`, and `--min-accuracy`. `--help` is the
 authority on all of them.
 
+### Making `--max-budget-usd` bind on `claude-code`
+
+The two engines enforce the spend cap differently, and only one of them works
+out of the box.
+
+`claude-code-no-sandbox` hands `--max-budget-usd` to the `claude` CLI, which
+knows what it bills and stops by itself. `claude-code` cannot: `inspect_swe`
+builds its command line from a closed set of arguments and takes no
+passthrough ([inspect_swe#178][swe178]), so that leg uses inspect's own
+per-sample `cost_limit` instead — and inspect computes a cost only when its
+model registry knows the rate. **It ships no rates for any model.**
+
+So without rates that leg simply runs without a spend cap: skillscope withholds
+the limit rather than passing one, because inspect validates `cost_limit` up
+front and would refuse the whole run. `meta.max_budget_can_bind` reports which
+happened.
+
+`SKILLSCOPE_MODEL_PRICING` is how. Point it at a JSON file — or pass the JSON
+directly — mapping each model to its rates in **dollars per million tokens**:
+
+```json
+{
+  "opus": {"input": 5.0, "output": 25.0, "cache_read": 0.5, "cache_write": 6.25}
+}
+```
+
+Keys may be skillscope aliases (`opus`) or inspect model strings;
+`cache_read` and `cache_write` default to `0`. A malformed table fails the
+command in the preflight rather than one sample into a graded run.
+
+skillscope deliberately ships no price list of its own. Rates change without
+notice and differ by contract, and a stale table would hold a run to a number
+nobody agreed while the report claimed the budget was enforced — worse than no
+cap at all. Whatever you configure is recorded in the report as
+`meta.max_budget_pricing`, beside `max_budget_can_bind`, so a reader can tell a
+cap backed by your figures from one that cannot fire.
+
+[swe178]: https://github.com/meridianlabs-ai/inspect_swe/issues/178
+
 ### Where the skills are
 
 Every path in the table is relative to the repository root, which is the only
@@ -188,8 +227,8 @@ Two failure modes are worth knowing before reading a report. A routing case that
 ends without the agent either activating a skill or answering is reported as an
 **error** rather than a missed trigger. And if the runner has its own skills
 installed (usually `~/.claude/skills`), they join the room for every case and
-the report says so — set `ANTHROPIC_API_KEY` so the run can use an isolated
-config dir.
+the report says so — set `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN` so the
+run can use an isolated config dir.
 
 ## Hardware a skill needs
 
@@ -208,6 +247,100 @@ Legs with a scoped environment run as a separate job, because a job's
 credentials are fixed before its matrix expands. A repo that declares no scoped
 environment gets one matrix, labels and all.
 
+## Which engine grades a run
+
+`--engine` chooses what actually runs the cases. The dataset, the CLI and the
+reports are identical whichever you pick; only the thing driving the agent
+changes.
+
+| `--engine` | What runs | Runs in | Needs |
+| --- | --- | --- | --- |
+| `claude-code-no-sandbox` (default) | The real CLI, under `inspect_ai` | the host | the CLI on `PATH` |
+| `claude-code` | The real CLI, via `inspect_swe` | a sandbox | `skillscope[verify]`, Linux only |
+
+Both drive the agent a skill is written for, so what differs between them
+is **where the agent runs**, not what it is. That is the axis worth choosing
+along: the host measures the machine as it is, with whatever else is installed
+on it, and the sandbox measures the skill alone. When the two disagree, the
+disagreement is usually a fact about one of those environments rather than
+about the skill -- which is a thing one engine on its own cannot tell you.
+
+`claude-code` is a reporting leg, never a gate. Harness runs are
+nondeterministic and the harness is not what is being graded, so a divergence
+there is a question about the skill rather than a build failure.
+
+**Routing runs on both**, and the choice matters more there than it does
+for behavioral. A stray user-level skill on the runner does not spoil one
+case's grade -- it is offered for every prompt, so it changes every decision at
+once while the run still reports a clean accuracy. `claude-code` is the only
+leg immune by construction: its guest has no `~/.claude` to contribute.
+
+The host leg cannot read the CLI's session-init event, so it has no way to
+notice such contamination or report it -- and therefore refuses to run a
+routing leg at all unless `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN` is
+set, which is what lets it redirect the CLI's config dir away from the
+runner's own. Refusing beats being
+quietly wrong about every case.
+
+Both legs stop at the decision, by different routes. `claude-code`'s calls
+cross inspect's bridge, so the one that reveals the decision is declined
+*before* it runs, and `--max-tool-calls` / `--max-inspection-calls` ride the
+same path. `claude-code-no-sandbox` drives the CLI as a subprocess with nothing
+to intercept its calls, so it reads the `stream-json` the CLI prints as it goes
+and kills the process group the moment a skill fires -- one call of overshoot,
+and the same rule. The report records which caps actually applied rather than
+which were asked for.
+
+### Where a sandboxed run is sandboxed
+
+Two separate decisions, made by different people.
+
+**Which provider** is a property of the runner, chosen with
+`SKILLSCOPE_SANDBOX`. Docker by default; `podman` on a host that has that
+instead; `local` to skip the container. `local` is for working locally rather
+than for CI, because a graded run that quietly dropped its sandbox would report
+the same numbers with none of the isolation.
+
+Podman needs three things, and each was discovered by the next one failing:
+
+* `pip install 'skillscope[podman]'`. The provider is registered by a separate
+  package through an entry point, so the podman binary alone is not enough.
+* `podman-compose`, and `INSPECT_PODMAN_COMPOSE=podman-compose`. Bare
+  `podman compose` is a shim that delegates to whichever compose provider it
+  finds, which on a host that also has Docker is Docker's -- and that then
+  talks to a daemon podman was chosen to avoid.
+* A search registry, because podman will not guess one. Docker assumes Docker
+  Hub for an image name with no registry; podman refuses, and the default
+  sandbox image is named without one. `unqualified-search-registries =
+  ["docker.io"]` in `/etc/containers/registries.conf`.
+
+Podman is worth the setup where the runner's user cannot reach the Docker
+socket, since it is daemonless and rootless and needs neither that nor group
+membership.
+
+**What the sandbox must provide** is a property of the skill, declared as
+`sandbox: compose.yaml` in its `evals/machine.yml`, resolved beside it. Skills get a container with
+no network by default; one that installs a server or pulls a model cannot run
+that way and says so. Selecting a provider does not discard what a skill asked
+for -- the compose file rides along.
+
+[`examples/skill-with-a-device/evals/`](../examples/skill-with-a-device/evals)
+is the pair, worked through: a `machine.yml` that asks for GPU runners and
+names a compose file, and the compose file that binds the devices in and
+grants egress. The two are not substitutes. `labels:` decides which machine the
+job lands on; `sandbox:` decides whether the container on it can see the
+hardware that machine has. A skill that sets only the first gets the right
+runner and a container that cannot reach its device.
+
+Windows is the exception to both: inspect's sandbox layer and every tool built
+on it assume a POSIX guest, so those legs run unsandboxed and trade isolation
+for running on the platform they are meant to test.
+
+To see what changing engine would do to your own datasets before changing it,
+[`tools/benchmark_engines.py`](../tools/benchmark_engines.py) runs the same
+cases through two engines and reports per-case agreement, measured against how
+much one engine already disagrees with itself.
+
 ## In CI: one job
 
 [`reusable.yml`](../.github/workflows/reusable.yml) grades a repo's skills with