Evals can't see what a skill's audience actually receives: add a per-harness, per-model listing check and an optional model matrix #28

Summary

skillscope evaluates a skill through one harness (Claude Code) and one model (Opus, pinned in CI). The judge is that same model. Within that setup the results are correct. What the setup can't show is whether a skill's description, which most harnesses use for routing, reaches the model at all when someone uses a different harness or model.

Here is a concrete case. With the full amd/skills catalog installed, Claude Code 2.1.305 sent every catalog skill to some models by name only, with no descriptions. Opus and Sonnet 5 got every description. CI passes, and nothing in skillscope reports the difference: routing's system/init check compares only skill names.

This issue asks skillscope to measure what each harness and model actually receives, without assuming Claude Code. Codex CLI is one concrete second harness (tracked in #4). Cursor, Gemini CLI, and others should fit the same interface.

Related:

• Add support for Codex agent harness #4 asks for a Codex harness. This issue adds evidence for it and asks that it be built as a pluggable harness interface, not a Codex-only path.
• Fail a routing run that lost a skill before it started #22 (open) would make a skill missing from session init fail routing. It still checks names, so a name-only listing passes.
• Report what a repo's skills cost in the startup listing #23 (open) estimates the catalog's cost against Claude Code's skill-listing budget from a fixed context window. The observations below show that the effective budget depends on the model id and on which other skills are installed. A static estimate can say "fits" while the model receives only names.
• Enable deterministic assertions using BDD #7 (deterministic assertions) overlaps the judge point under Proposal 4.
• Add Codex backend to skill eval harness skills#197 proposed a Codex backend for the catalog's own eval harness. It was deferred to skillscope.

Versions

• skillscope v0.1.3, which is the version amd/skills CI pins, and main at 2a1959a, after Rebuild the eval engine on inspect_ai and retire the legacy runner #19. Code references below are to main unless marked v0.1.3.
• Claude Code 2.1.305, Codex CLI 0.163.0.
• Catalog: amd/skills with 12 skills.

Observations

Method: a local endpoint captured the first request each harness sent ("hi", --max-turns 1). Claude Code ran with a fresh CLAUDE_CONFIG_DIR and --setting-sources project, and Codex with a fresh CODEX_HOME. All 12 catalog skills were installed (.claude/skills/ for Claude Code, .agents/skills/ for Codex). Model ids are as exposed by an Anthropic-Messages and OpenAI-Responses compatible gateway. The captures were repeated in separate, independently set up runs with the same results.

1. Claude Code: whether descriptions appear depends on the model id and on the other installed skills

Claude Code fits user and project skill descriptions into one listing budget. Skills are filled in listing order. When the budget runs out, the remaining skills are listed by name only. Bundled skills appear to sit outside this budget and keep their long descriptions. The budget follows the context window Claude Code assumes for the model id. For ids it doesn't recognize, it prints [claude-code:unrecognized_model] to stderr.

These measurements are empirical for 2.1.305 and are likely to change between versions:

• Unrecognized ids. The effective budget was about 230 rendered listing characters. With one short-named skill, descriptions up to about 215 characters were shown and descriptions from about 220 were dropped. YAML style (>-, |, plain) made no difference.
• The budget is shared. Two skills with 100-character descriptions: one shown. Ten 30-character skills: one shown. With the catalog installed, an extra skill with a 50-character description was also dropped.
• Settings restore descriptions. CLAUDE_CODE_MAX_CONTEXT_TOKENS=1000000, a [1m] suffix on the model id, or skillListingBudgetFraction in project settings restored the descriptions. A default install sets none of these.

So no description length is safe, and whether a skill is visible is not a property of the skill alone. It depends on the harness, the model id, the harness version and settings, and the other installed skills. That is why it needs to be measured, not linted.

2. Codex CLI sends every description, but changes other parts of the prompt by model

Codex 0.163 sent all 12 catalog descriptions verbatim to every model tried: gpt-6-luna, Claude-Haiku-4.5, Claude-Haiku-5.5, Claude-Opus-5.5, and gemini-3.5-flash-lite. It cuts descriptions over 1024 characters. The catalog's longest is 997 today. Codex also has its own model-recognition split. Models without OpenAI metadata get a fallback base prompt with none of Codex's skill-usage guidance, a different tool set, and no reasoning-effort setting. Codex also adds its own system skills to the list and loads nested SKILL.md files under a skill directory. Each harness changes what a skill's audience sees in its own way, which is the case for a harness-agnostic check.

3. Effect on routing: large once descriptions reach the model

A routing benchmark of serving-llms-on-instinct in Claude Code 2.1.305 compared the default listing with skillListingBudgetFraction: 0.05 in project settings. Request captures confirmed that this setting gives every model the full listing and changes nothing else. Each model got 10 prompts × 3 reps: 5 that should use the skill, and 5 that should not (an NVIDIA H100, a Radeon RX, a Ryzen AI NPU, an MI250X, and an HBM explainer). Runs that timed out are excluded, so some denominators are below 15.

Across these 11 models, false triggers fell from 84/164 (51%) to 12/153 (8%), and correct triggers rose from 139/165 (84%) to 149/161 (93%). Claude-Haiku-4.5 already received the description by default and didn't change. gpt-5-nano barely uses tools in either setting.

Two caveats:

• A different override showed a smaller effect. An earlier A/B restored descriptions by raising the assumed context window (CLAUDE_CODE_MAX_CONTEXT_TOKENS=1000000) instead. There gpt-6-luna went from 12/15 to 5/15 false triggers and Claude-Haiku-5.5 stayed at 8/15. That override also changes other context-dependent behavior, so the listing-only setting above is the cleaner comparison. The difference isn't explained yet.
• Codex is supporting, not controlled, evidence. Under Codex, which sends every description, false triggers were similarly low for most models. But DeepSeek-V4-Flash stayed high, correct triggers dropped, and the harness prompts differ.

So for this skill, whether the description reaches the model is the difference between mostly wrong and mostly right routing for most of these models. skillscope currently has no way to report which of those situations a given harness and model are in. The benchmark is in amd/skills#286 (the author's own PR), offered as context.

Why skillscope can't see it today

• One harness. Both engines drive only the claude CLI: claude-code-no-sandbox (skillscope/engine/no_sandbox.py:99, :456-457 builds claude -p --output-format stream-json) and claude-code (inspect_swe claude_code, skillscope/engine/verify.py). Skills are staged in .claude/skills/ (skillscope/routing.py:58, skillscope/engine/no_sandbox.py:367). action.yml installs only @anthropic-ai/claude-code. At v0.1.3 the same holds for agent._run_agent and routing.run_case.
• One model. enforce_model_policy coerces every non-Opus model to opus in CI (skillscope/agent.py:67-77). Opus has a large enough budget for the whole catalog.
• The judge is the agent's model. skillscope/engine/judge.py:275 uses get_model() (the task model) when no grader is set. At v0.1.3, agent.py:522 passes judge_model=self.model.
• The visibility check is name-only and only warns. routing.init_skills (skillscope/routing.py:227) matches names in the system/init event, and only the host engine can run it. A listing with every description dropped passes. At v0.1.3, _init_skills produced a report warning, not a failure. Neither the stream-json output nor --debug output includes the rendered listing. Only the request (or the session transcript, which routing disables) shows it.
• Credentials. Credentials are Anthropic-shaped: ANTHROPIC_*, with an optional Anthropic-compatible base URL and headers. Gateway headers apply only to anthropic/ models (skillscope/engine/models.py:205). There is no path for an OpenAI-Responses-style provider.

The Opus pin keeps paid CI runs comparable, and that's a reasonable goal. None of the proposals below would remove it.

Proposal

These are in rough order of cost. Proposal 1 is the core ask.

1. A "what was received" listing check, per harness and model id. Point each harness at a local stub endpoint that records the first request and returns a canned reply. Then report, for each installed skill, whether its name and its full description appear in the listing. The stub needs no credentials, uses no tokens, and doesn't forward upstream. In our runs it took about 1–2 s per (harness, model id). One call covers the whole installed set. Install the full catalog, not one skill at a time, because the budget is shared. This fits the credential-free structural job. It complements Report what a repo's skills cost in the startup listing #23: Report what a repo's skills cost in the startup listing #23 estimates the budget, and this check observes what was actually sent. It also catches drift, because action.yml installs an unpinned Claude Code. Claude Code's [claude-code:unrecognized_model] stderr line is an extra signal that costs nothing. Define the check through a small harness interface (stage skills, launch against a base URL, extract the listing from the captured request), so Codex, Cursor, Gemini CLI, and others can each add an adapter.
2. An optional, non-gating model matrix. Add an opt-in models input, defaulting to Opus only. Each extra model is its own pinned leg, reported per model and never gating. The judge stays fixed at the pinned model, so legs are comparable. This needs enforce_model_policy to allow declared extra legs, and the judge to be decoupled from the agent model. Non-Claude legs need a gateway route that serves them (the API-key path, not OIDC federation).
3. A pluggable harness interface for evals (see Add support for Codex agent harness #4). Codex is the obvious first adapter. It runs headlessly (codex exec --json, Responses API, CODEX_HOME isolation, skills in .agents/skills/). A $skill-name mention injects the whole SKILL.md as a user message. Codex has no Skill tool, so activation would have to come from shell reads of SKILL.md. That signal is imperfect: models sometimes name a skill without opening it. Results from a second harness should be reported as their own leg and not compared directly with Claude Code numbers, because the harness prompts differ. Codex's fallback prompt for non-OpenAI models is one example. This is more than a second runner: staging, launch, event normalization, activation detection, the listing capture above, and credentials are all per-harness today.
4. Judge independence (overlaps Enable deterministic assertions using BDD #7). Prefer deterministic checks where a case allows it. Where an LLM judge is needed, allow the judge to be set separately from the agent model, and optionally cross-check a sample with a judge from a different model family.

Reproduction (listing only)

1. Run a local HTTP endpoint that logs request bodies and returns a minimal valid response. Logging requests while forwarding them to a real endpoint also works.
2. In an empty directory, put the catalog's skills under .claude/skills/.
3. Run claude -p "hi" --model <model id> --max-turns 1 --setting-sources project, with ANTHROPIC_BASE_URL pointing at the endpoint and CLAUDE_CONFIG_DIR set to an empty directory.
4. In the captured request, find the list under "The following skills are available for use with the Skill tool". Try Opus, then an id the installed Claude Code doesn't recognize. Repeat with CLAUDE_CODE_MAX_CONTEXT_TOKENS=1000000 to see the descriptions come back.

The listing-budget behavior is Claude Code's, and its general shape is documented. A separate report to Claude Code could ask for a clearer signal when an unrecognized model id leaves every description out. This issue is only about making skillscope able to detect it.