[kelkalot]

Adds OpenEnvTarget, which audits an agent hosted in an OpenEnv environment. The judge then grades what the agent did with its tools, as well as what it said. It's a new optional extra, openenv; import simpleaudit doesn't load it.

How it works

OpenEnv's HarnessEnvironment (RFC 005) runs an agent harness, such as Claude Code, and serves it the environment's tools over MCP. OpenEnvTarget puts that agent in the target seat:

• Episodes. Each scenario gets a fresh episode through Core seams for integrations: target lifecycle hooks, tool events, injected clients #121's start_scenario / end_scenario hooks.

  • In process, with env_factory=: one environment per scenario, so scenarios can run in parallel. A factory that takes a scenario_name keyword is told which scenario its environment is for.
  • Against a server, with base_url=, through GenericEnvClient.

  Both kinds close through one exit stack.

• Turns. Each auditor message is one HarnessAction step. The reply comes from metadata["response"]. The turn's tool calls and results, including error messages and any harness error events, become tool_events.

• Failures and early ends.

  • A failed or timed-out agent turn makes the scenario an ERROR. message_timeout_s bounds a turn in both modes.
  • An agent that ends its episode sets end_conversation, so the transcript so far is judged.
  • Scenarios that attach files or documents are rejected, because the harness takes text only.

• Options.

  • reset_kwargs can be a function of the scenario name.
  • max_sessions caps open episodes across every run and thread that uses the target. It defaults to 1 for a server, because a harness server holds one session.

• Tokens. Token usage comes from the turn-complete event. Anthropic-style cached prompt tokens count as input.

examples/openenv_agent_audit.py audits a caseworker agent in a mock benefits case system, with tools to look a case up, change its payout account and send a letter. It can run a scripted agent or Claude Code, through the adapter in OpenEnv's examples/claude_code_harness_eval, and it prints whether each scenario changed the world's state.

Notes

• Closing in process. An in-process episode closes through the environment's synchronous close(), which OpenEnv runs on an event loop of its own. An agent adapter therefore must not hold resources tied to the audit's loop; OpenEnv's subprocess adapters don't.
• Repeated runs. AuditExperiment builds its targets from model configs, so repeated runs of an agent go through RepeatedExperimentResults({"agent": runs}).
• Claude Code version. OpenEnv's Claude Code adapter passes --no-session-persistence, which older claude CLI releases don't support. The example's --claude option points at a specific executable.
• CI. CI installs the extra. Inspect AI integration: run packs as Inspect tasks, read Inspect logs back #122 edits the same install lines.

Tests

tests/test_openenv_target.py has 17 tests with a scripted harness adapter and the real HarnessEnvironment. They cover:

• one episode per scenario, and tool calls reaching the judge but not the auditor;
• token usage, including cached tokens;
• reset arguments and the factory's scenario_name;
• agent crashes, timeouts and early ends;
• parallel episodes;
• the session limit across repeated runs and across threads;
• aclose closing every episode;
• attachments;
• that import simpleaudit stays free of openenv;
• a uvicorn server driven over the websocket client.

Live checks, with claude-haiku-5-5 as judge and as the agent's model:

• examples/openenv_agent_audit.py, with the scripted agent and with Claude Code, two repetitions each. Claude Code's tool calls were captured, and its cached prompt tokens counted as input.
• An OpenEnv server serving Claude Code, audited over the websocket client with two parallel scenarios. The default session limit ran them one after the other against the server's single session.