Skip to content

Add GitHub Copilot CLI as a harness, with routing that needs no model credential - #34

Draft
volen-silo wants to merge 7 commits into
amd:mainfrom
volen-silo:harness-cli
Draft

Add GitHub Copilot CLI as a harness, with routing that needs no model credential#34
volen-silo wants to merge 7 commits into
amd:mainfrom
volen-silo:harness-cli

Conversation

@volen-silo

@volen-silo volen-silo commented Oct 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Draft, and stacked on #33. This branch contains #33's two commits first; until
that merges, review only the last three: Add GitHub Copilot CLI as a harness,
Give a harness the hooks the engines need beyond its events, and Choose the agent from the command line, and run routing on Copilot CLI. I will rebase and
mark it ready once #33 is in.

Why

Part of #4; design in this comment.

GitHub Copilot CLI is the second harness. In Actions it authenticates with the
workflow's GITHUB_TOKEN, so a repo can run graded routing evals with no model
credential at all. It reads skills from .claude/skills, where they are already
staged.

What changes

The adapter (skillscope/harness/copilot.py). Copilot's --output-format json
events share nothing with Claude Code's, so Copilot translates them:
tool.execution_start / tool.execution_complete become a tool call and its
result (activating a skill is a call to the tool named skill, which is the call
the engines already read; a call the permission rules deny is still recorded), and
the closing result event, which carries no answer, gets the last assistant
message remembered from earlier in the run. Token counts, reported once and per
model, reach the usage recorder. Streaming deltas and bookkeeping are dropped.

Hooks the engines need per harness: the default model and how the harness
names models, the CI model pin, which environment variables a credential can come
from, spend controls, version, and a preflight that sends one trivial prompt
through the harness's own command.

The command line. --harness NAME says which agent runs the cases; --engine
keeps meaning where. host and sandbox are the neutral engine names; the older
claude-code-no-sandbox / claude-code keep working and mean Claude Code, and
naming a different harness beside one is refused rather than guessed. Unnamed
still means Claude Code on the host. --model no longer defaults to opus in the
parser, since each harness has its own default and names; Claude Code keeps opus
and its CI pin.

Routing on Copilot needs no model credential. Its agent is a subprocess and
routing has no judge, so inspect is handed a model that reaches no provider
instead of an Anthropic one nothing calls. A room with no Copilot token in the
environment is refused, as Claude Code's is without its own.

Reports gain harness and harness_version. Claude Code reports its engine as
before, so earlier reports stay comparable; another harness reports the neutral
names, since the old ones would name an agent that did not run. A dollar cap a
harness cannot enforce is recorded as unable to bind.

Details worth a reviewer's eye are in docs/harnesses.md and the usage.md
engine section.

Cost, which differs a lot by model

Measured on one prompt: Sonnet 5.5 is 1 premium request, Opus 5.5 is 15. So the
default for this harness is the cheap model, and CI pins every Copilot run to it
(one fixed model keeps scores comparable). opus, sonnet and haiku map to
Copilot's own names. The preflight sends one real prompt, which is billed like any
other.

Verification

  • 604 tests pass on Windows with Python 3.12 (apart from the one flaky existing
    test described below); the 505 existing tests pass unmodified, so Claude Code is
    unchanged. ruff --select F401 is clean. CI is green on Linux and Windows,
    Python 3.10 and 3.12.
  • Real recordings. The fixtures are two sessions from Copilot CLI 1.0.105 (the
    skill activated, and a near miss), with streaming deltas, the MCP server
    listing, encrypted reasoning blobs and request ids removed and the local
    username replaced. Both pass the shared contract test from Add a harness contract, so the engines are not tied to one agent #33. Their expected
    values were computed from the raw recordings, not from the adapter.
  • End to end against a stub. New tests run a real skillscope routing command
    against a stub copilot that replays the recorded session, with no Anthropic
    variable in the environment, and check the command line it was started with, the
    model name and the CI pin, the isolated home, the skills staged where it looks,
    the preflight, and the report.
  • Deliberate breaks of the code each turn a test red: the adapter (answer
    remembered wrongly, missing flag, sub-agent message as the answer, denied call
    losing its reason, tokens dropped, failed run reported as success), the hooks
    (pin, aliases, preflight status, timeout), and the wiring (engine name in the
    report, handing inspect a real provider, the CI pin, the behavioral refusal, the
    isolation check).
  • The tests caught a real bug in this change (the unnamed-engine default was the
    Claude-named spelling, so --harness copilot alone was refused), and the
    preflight timeout test caught another: subprocess.run(timeout=) kills only the
    process it launched, and a CLI installed with npm is a shim, so on Windows a one
    second timeout took a minute. The preflight now kills the whole process tree.

Run for real, once, on Windows. skillscope routing --harness copilot against the
real rocm-doctor dataset (12 cases: its 7 plus the 5 shared negatives), with Copilot
CLI 1.0.105 on claude-sonnet-5.5, and every ANTHROPIC_* and CLAUDE_CODE_*
variable removed from the environment:
Routing accuracy 12 / 12 (5 positives activated the skill, 7 negatives routed to no skill)
Errors / degraded 0 / 0
Wall time 66 s with 3 cases at a time
Tokens 462,442 in, 5,980 out
Reported as harness: copilot, engine: host, max_budget_can_bind: false

Read this as "it works end to end", not as a score to trust: one run, no measured
run-to-run spread, and the skill's own prompts. It was run through the same Windows
code path the CI job exercises, but not in Actions. A CI run with GITHUB_TOKEN and
copilot-requests: write is the next check.

Run it on a throwaway runner. --allow-all is what lets a non-interactive agent
work, and the agents on the negative prompts used it: they ran nvidia-smi and a
torch import, wrote and ran small scripts in their temp workspaces, and one searched
the user profile directory recursively for a file. Nothing was destructive, but the
agent is unsandboxed on the host, exactly as Claude Code is on the host engine.

One existing test, TestTheHostLegActuallyKillsTheProcess.test_the_work_after_the_decision_never_happens,
is flaky on Windows and fails on a clean main too (3 of 8 runs); it is unrelated
and untouched.

Not in this PR

  • behavioral on Copilot is refused. Its expectations are graded by an LLM
    judge that still goes through an Anthropic model whatever agent produced the run;
    routing it through the harness is next.
  • The room check. Claude Code reports at session start which skills it loaded;
    Copilot does not (and always includes a few built-in skills), though
    copilot skill list --json lists them without a model call. Follows with the judge.
  • install-harness on the action and the reusable workflow inputs. In Actions the
    calling workflow needs permissions: copilot-requests: write.
  • Codex and any other harness; docs/harnesses.md is the checklist for adding one.

@volen-silo
Put the agent behind a harness contract, with Claude Code as the first
6a862cb 
The host engine assumed one agent. Its command line, environment, skills
directory and "is it installed" check were all written for the `claude` CLI
inside the solver, so adding another agent meant copying the solver.

Add `skillscope.harness`: a small `Harness` contract (binary, skills_dir,
command, env, normalize, require), a registry, and `ClaudeCode`, which is the
command line and environment the engine already used, moved out unchanged.
`normalize` translates a harness's own events into the stream-json subset the
engines already read, so stop rules, budgets, scorers and the judge are
untouched and written once for every harness.

`no_sandbox.harness_solver(harness, ...)` is the old solver, generalised.
`claude_code_no_sandbox` keeps its signature and calls it with Claude Code, so
every caller behaves as before. `_stream_until` gains an optional `normalize`
so the stop rule sees canonical events only, and `install_skill` takes the
harness's skills directory.

Nothing user-visible changes: no new flag, and the 505 existing tests pass
unmodified, including the one that drives the real solver against a stub
`claude`. Choosing a harness from the command line, and moving the preflight,
credential check, spend flags, CI model pin and judge behind the contract, are
left for the change that adds a second harness; docs/harnesses.md lists them.

New tests: the registry; a golden check that the Claude Code command and
environment equal what the engine built inline; a contract test that runs once
per registered harness against recorded output in tests/fixtures/harness and
fails when a harness ships none; and a harness whose events look nothing like
Claude Code's, driven both through `_stream_until` and through the solver under
inspect. Each was checked by breaking the code it covers and seeing it fail.

Signed-off-by: Eugene Volen <eugene.volen@amd.com>
@volen-silo
Let a harness keep state while it translates one run's events
fc1e678 
`Harness.normalize` translates each event on its own. That is not enough for
an agent whose closing event does not repeat the answer: GitHub Copilot CLI's
final `result` record carries only an exit code and usage, and the answer is
the last assistant message, printed several events earlier. A translator for
such a harness has to remember it.

Add `Harness.normalizer()`, which returns a translator for one run and may
keep state. The default returns the stateless `normalize`, so Claude Code is
unchanged. The engine asks for a fresh one per case, so nothing remembered in
one case reaches the next, and the state lives in the returned callable rather
than on the harness instance, which every concurrent case shares.

The test harness now behaves like that: its closing event carries no answer.
New tests check that each translator has its own memory, that two runs
translated at once do not share it (for every registered harness, from the
recorded fixtures), and that a case's answer does not leak into the next one
when the solver runs two cases in a row under inspect. Breaking the code two
ways -- sharing one translator across cases, and bypassing it on the
non-streaming path -- makes those tests fail.

Signed-off-by: Eugene Volen <eugene.volen@amd.com>
@volen-silo
Add GitHub Copilot CLI as a harness
82caae8 
Copilot CLI is the second agent harness. It authenticates from a token in the
environment, so a graded run needs no model key, and it discovers skills in
`.claude/skills`, where they are already staged.

Its events share nothing with Claude Code's, so `Copilot` translates them:
`tool.execution_start` and `tool.execution_complete` become a tool call and its
result (activating a skill is a call to the tool named `skill`, the call the
engines already read; a call the permission rules deny is still recorded), and
the closing `result` event, which carries no answer, gets the last assistant
message remembered from earlier in the run. Token counts, which the CLI reports
once and per model, reach the usage recorder as one usage-only assistant event.
Streaming deltas and bookkeeping are dropped.

The command runs headless with the prompt on stdin, every permission granted
(a non-interactive run cannot answer a prompt) and the built-in MCP servers off;
the environment points `COPILOT_HOME` at the throwaway config directory and
turns self-update off so the version being graded cannot change mid-run.

Fixtures are two real recordings from CLI 1.0.105, one where the skill is
activated and one near miss, with streaming deltas, the MCP server listing,
encrypted reasoning blobs and request ids removed and the local username
replaced. Both pass the shared contract test, and the expected values were
computed from the raw recordings rather than from the adapter.

Not wired to the command line yet: there is no `--harness` flag, the preflight,
credential check, model pin and judge still assume Claude, and the session's
skills are not read back for the room check. Those follow.

Tests: the command and environment, what the recording becomes (calls in order,
a denied call's reason as its result, the answer, the tokens through the real
usage recorder, a near miss), shapes the recordings lack (failed run, sub-agent
message, malformed events), and the recording replayed by a real process
through the engine's streaming loop, stopping at the skill call. Six deliberate
breaks of the adapter each turn a test red.

Signed-off-by: Eugene Volen <eugene.volen@amd.com>
@volen-silo
Give a harness the hooks the engines need beyond its events
2b91902 
The engines still ask Claude Code directly for things that differ per harness:
the model to use when none is named, what a CI run is pinned to, which
environment variables a credential can come from, the CLI's spend controls, its
version, and whether it can reach its model at all. A second harness could not
answer any of them.

Add them to the `Harness` contract, with the defaults a harness can usually
keep and Claude Code delegating to exactly what the engines did inline:
`default_model` and `resolve_model`, `pin_model`, `credentials` and
`can_isolate_room`, `cost_flags`, `supports_sandbox`, `version`, and
`preflight`, which sends one trivial prompt through the harness's own command so
what is checked is what a case will use.

Copilot CLI answers them for itself: it defaults to the model that costs one
premium request per prompt rather than the largest, which costs fifteen; maps
`opus`, `sonnet` and `haiku` onto its own names; pins CI to that one model;
authenticates from `COPILOT_GITHUB_TOKEN`, `GH_TOKEN` or `GITHUB_TOKEN`; and has
no dollar spend control.

The preflight runs the CLI through a runner that kills the whole process tree on
timeout. `subprocess.run(timeout=)` kills only the process it launched, and a
CLI installed with npm is a shim that starts the real program, so on Windows the
program kept the output pipe open and the wait after the kill lasted until it
ended on its own: a one second timeout took a minute.

Nothing calls the hooks yet. The command line, the preflight, the credential
check, the CI pin and the report still take Claude Code as given, and the next
change wires them through. All existing tests pass unmodified.

New tests run the shared hooks for every registered harness, pin that Claude
Code's behaviour is what it was, and drive Copilot's pin, aliases, version and
preflight against a stub executable, including a CLI that refuses, one that
hangs, and a deadline that has run out. Five deliberate breaks each turn a test
red, including reverting to the old timeout handling.

Signed-off-by: Eugene Volen <eugene.volen@amd.com>
@volen-silo
Choose the agent from the command line, and run routing on Copilot CLI
25709cd 
Add `--harness`, which says which agent runs the cases, and keep `--engine` for
where it runs. `host` and `sandbox` are the neutral names for the two legs; the
older `claude-code-no-sandbox` and `claude-code` keep working and mean Claude
Code, so naming a different harness beside one is refused rather than guessed.
`$SKILLSCOPE_HARNESS` and `$SKILLSCOPE_ENGINE` supply what the command line does
not. Unnamed means `claude-code` on the host, exactly as before.

The command line, the preflight, the credential check, the CI model pin, the spend
controls and the report now go through the harness instead of assuming Claude
Code. `--model` no longer defaults to `opus` in the parser: each harness has its
own default and its own names, and the default for Copilot is the model that
costs one premium request per prompt, not the one that costs fifteen. Claude Code
keeps `opus` and the CI pin it had.

A routing run on another harness reaches nothing but that harness's CLI. Its
agent is a subprocess and routing has no judge, so inspect is handed a model that
reaches no provider instead of an Anthropic one nothing calls. That is what makes
a Copilot run need no model credential, only the token the CLI authenticates
from; a room with no such token in the environment is refused, as Claude Code's
is without its own.

Reports gain `harness` and `harness_version`. Claude Code reports its engine as it
always did, so earlier reports stay comparable; any other harness reports the
neutral names, because the old ones would name an agent that did not run. A
dollar cap that a harness cannot enforce is recorded as unable to bind rather
than omitted.

`behavioral` is refused on a harness other than Claude Code: its expectations are
graded by an LLM judge that still goes through an Anthropic model. That, and the
room check for a harness whose session does not list its skills, follow.

Every existing test passes unmodified. New tests run the real command against a
stub `copilot` that replays the recorded session, with no Anthropic credential in
the environment, and check the command line it was started with, the model name
and the CI pin, the isolated home, the skills staged where it looks, the
preflight, and the report. Five deliberate breaks of the wiring each turn a test
red.

Signed-off-by: Eugene Volen <eugene.volen@amd.com>
@volen-silo
Give each stub process its own record file in the CLI tests
08201f7 
The tests that run a routing command against a stub `copilot` had every stub
process append its start record to one shared file. Inspect runs cases
concurrently, and appends from two processes to one file are not atomic on
Windows, so a record could be overwritten or cut in half: on Windows with
Python 3.12 one run read a half-written line and another saw one start where
two were expected, while the same tests passed on 3.10 and on Linux.

Each process now writes a file of its own, named so they sort in the order the
processes started, and the tests read the directory. The tests that assert the
CLI was never started now check that the directory is empty.

Signed-off-by: Eugene Volen <eugene.volen@amd.com>
@volen-silo volen-silo mentioned this pull request Oct 29, 2026
Open
@volen-silo
Name the harness in the console lines of a routing run
3ec3bd8 
A routing run on another harness still printed `engine=claude-code-no-sandbox`
in its summary line and `[claude-code-no-sandbox] routing:` before the cases,
naming an agent that did not run. The report was already right. Both console
lines now name the harness and the neutral engine; Claude Code's lines are
unchanged.

The spelling of the engine for a harness lives in one place, shared by the report
and the console.

A new test checks the progress lines of a Copilot run, and breaking either line
makes it fail.

Signed-off-by: Eugene Volen <eugene.volen@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@volen-silo