Activity

1. danielholanda

   added

   p1

   on Sep 3, 2026
2. TwelveNights commented on Oct 23, 2026

   

   TwelveNights

   on Oct 23, 2026

   More actions

   Some notes from running Codex headlessly for skill evals, in case they help whoever picks this up. Details and the broader ask (a pluggable harness interface, so Codex is one adapter among several) are in #28.

   Verified with codex-cli 0.163:

   • Launch: codex exec --json --skip-git-repo-check --ephemeral -m <model>, with stdin closed (otherwise it waits for more input). A custom provider can be passed with -c model_provider=... -c 'model_providers.<id>={base_url=..., wire_api="responses", env_key=..., env_http_headers={...}}', which keeps keys in env vars rather than files.
   • Isolation: a fresh CODEX_HOME per run. Codex refuses to create its helper binaries when CODEX_HOME is under /tmp.
   • Skills: staged in .agents/skills/ of a git-initialized workspace. A $skill-name mention in the prompt injects the whole SKILL.md as a user message, which is the equivalent of a slash invocation.
   • Activation: there is no Skill tool. Models open SKILL.md with a shell command, so the existing "read the skill's own SKILL.md" fallback in activation detection is the right signal. It is imperfect, because a model can name a skill without opening it.
   • Events: --json emits item.completed items: command_execution (command, aggregated_output, exit_code) and agent_message. turn.completed carries usage. Commands run via zsh -lc/bash -lc, so a pre-tool guard can be done with command shims on PATH.
   • Network: --sandbox workspace-write plus -c sandbox_workspace_write.network_access=true is needed if the agent has to reach another host.
   • Not neutral by model: every model gets every skill's full description (descriptions over 1024 characters are cut). But for models without OpenAI metadata, Codex uses a fallback base prompt without its skill-usage guidance, so results from a second harness should be reported as their own leg.
3. volen-silo commented on Oct 29, 2026

   

   volen-silo

   on Oct 29, 2026

   Collaborator

   More actions

   Following up on the go-ahead to add more harnesses. We would like to build the
   pluggable interface asked for in #28 and ship GitHub Copilot CLI first, then
   Codex (#4), so that Claude Code, Copilot CLI and Codex are adapters on one
   contract. Copilot CLI is the one we need: in Actions it authenticates with the
   workflow's GITHUB_TOKEN, so a repo can run graded evals with no model key.
   We have a design below and would like to settle the open questions before
   writing code.

   Proposal

   • Two axes. --harness says what drives the cases (default
     claude-code); --engine keeps meaning where it runs. Add neutral engine
     names host and sandbox; keep claude-code-no-sandbox / claude-code
     working as aliases that imply --harness claude-code. Purely additive.
   • A small Harness interface in a new skillscope/harness/ package with no
     inspect_ai import: binary and version, where to stage skills, model alias
     resolution, the command and environment for one case (including config
     isolation), a normalizer for the harness's JSON events, preflight, and a
     no-tools text completion for the judge. One adapter file per harness.
   • Canonical event form = what the core already reads (the stream-json
     subset: tool_use, tool_result, system/init, result). Adapters
     translate into it, so routing, scorers and the judge are unchanged. A Codex
     adapter would use the existing allow_body_path fallback because it has no
     Skill tool.
   • One generic host driver replacing the Claude-specific one;
     claude_code_no_sandbox stays as a thin wrapper.
   • Judge goes through the harness (default: same harness and model as the
     run, as today), so a harness that cannot reach an Anthropic provider can still
     be graded. The existing grader parameter stays available for later.
   • Per-harness CI model pin. Today CI forces opus. We would keep that
     exactly for claude-code and let each harness declare its own pin, so runs
     stay comparable within a harness. Please confirm this fits the policy.
   • Each harness is its own leg. Reports record harness, harness_version
     and model_resolved (reusing the names in Record what produced a run, not just what it scored #21), and are never compared
     across harnesses; harness prompts and tool sets differ.
   • Scalable by construction. Each harness ships recorded, token-scrubbed
     fixtures and must pass a shared contract test; a docs/harnesses.md explains
     how to add one.

   Proposed PRs

   1. Interface + claude-code adapter, with no behaviour change and no CLI change
      (the existing tests pass unmodified). This is already written; a draft is on a
      fork branch if it helps the discussion:
      https://github.com/volen-silo/skillscope/tree/harness-interface
   2. Copilot CLI adapter, with the --harness flag and the host / sandbox
      engine names, judge through the harness, per-harness model pin, and
      install-harness on the action.
   3. Codex adapter. @TwelveNights, your notes in Add support for Codex agent harness #4/Evals can't see what a skill's audience actually receives: add a per-harness, per-model listing check and an optional model matrix #28 cover this well. Would
      you like to own it on top of PR 1, or would you rather we do?
   4. --harness a,b (the "both" case), per-harness report files, a harness
      input and matrix on the reusable workflow.

   Not in scope: the listing check and model matrix from #28 (they would build on
   this), a sandbox leg for non-Claude harnesses (inspect_swe has Codex and
   Gemini agents, so it is a natural follow-up), and any cross-harness comparison.

   Questions

   1. --harness or --agent? (We lean --harness; agent.py already means
      something else.)
   2. host / sandbox as the neutral engine names, with the current names as
      permanent aliases?
   3. Is it acceptable to keep the Claude-shaped stream-json subset as the canonical
      internal form for now? The alternative is a typed event model, roughly twice
      the diff and touching routing and the scorers.
   4. Is a per-harness CI model pin consistent with the "pinned to opus" intent?
   5. Ordering with Let a skill declare its prerequisites once, for both engines #29 and Record what produced a run, not just what it scored #21: we plan to rebase on Let a skill declare its prerequisites once, for both engines #29, and to make the
      provenance fields in Record what produced a run, not just what it scored #21 harness-aware. @johnl-amd, is that OK with you?
   6. Anything in flight that this would collide with?