[kelkalot]

Adds optional extension points to the audit engine:

• for targets that keep state per scenario;
• for targets that can report what an agent did;
• for callers that bring their own model clients.

#122 (Inspect AI) and #123 (OpenEnv) build on them.

Changes

• Target lifecycle hooks. A target may define async start_scenario(context) and async end_scenario(context).
  • run_scenario awaits start_scenario after scenario validation.
  • Once start_scenario has succeeded, end_scenario runs after the last turn and the evidence resolver, before judging. It also runs when a turn fails or the run is cancelled. A failing end_scenario raises a RuntimeWarning and doesn't change the result.
  • Only coroutine functions count, so MagicMock targets don't trigger them. A hook defined without async is skipped with a warning.
  • context.scenario_run_id matches the per-turn contexts, and context.extra["scenario_name"] names the scenario.
  • run_scenario now reads self.target once per scenario. Without set_target, the property builds a new ModelTarget on every access.
• TargetResponse.tool_events. The agent's tool calls and results for the turn, as a list of dicts.
  • Storage: they're stored beside the reply as a JSON-safe copy; anything else fails that turn with a clear error.
  • What the judge sees: a TOOL ACTIVITY block before the assistant turn.
    • Values are written as JSON, so tool output can't pose as a transcript turn.
    • A turn shows at most 50 events: the first and last 25 when there are more.
  • Who doesn't see them: they're left out of what the auditor model sees, of the history passed to targets, and of provider messages, which are reduced to role and content in one place.
• TargetResponse.end_conversation. Stops the turn loop after this reply; the transcript so far is judged.
• judge_client= / auditor_client= on ModelAuditor, and so on Auditor. A ready-made client, with the same contract rejudge takes, replaces AnyLLM client creation for that role, and the provider and model arguments become labels.
  • An injected auditor client always wins.
  • With an injected judge client, an auditor of its own needs auditor_client, or both auditor_model and auditor_provider.
• reframing_check rejects a transform that changes tool_events.
• README: a new section, "Writing your own target", and the new arguments in the parameter table.

Compatibility

Everything above is optional, and existing payloads are unchanged. tests/test_core_snapshots.py records the following for 28 configurations, and compares them with JSON files in tests/snapshots/core/:

• what the engine sends to the target, auditor and judge clients;
• the stored results;
• the on_turn events.

The snapshots were recorded on the code before this change and pass as recorded. Regenerate them with SIMPLEAUDIT_UPDATE_SNAPSHOTS=1 only when a payload change is intended.

Limitations

• Reading the target's state. The judge can't read the target's live state; an evidence resolver can.
• Checklist quotes. The checklist judge verifies quotes against reply text only. A quote taken from tool activity comes out unverified.
• SingleTurnAuditor doesn't call the hooks and doesn't store tool events.

Tests

tests/test_integration_seams.py has 30 tests. They cover:

• hook order, warnings, and the failure and cancellation paths;
• the evidence resolver running before end_scenario;
• tool-event storage, validation, rendering, caps and stripping;
• early ends;
• the injected-client rules;
• the reframing check.

Live checks, with claude-haiku-5-5 as target, auditor and judge:

• A two-turn nav_aap audit ran through the unchanged model path.
• An agent target returned tool events, one of them tool output imitating an assistant turn. The judge received it as a single quoted line and graded the account change the agent's tool calls recorded.
• The hooks ran in order: start, turns, evidence resolver, end.
• A datetime inside the tool events survived saving and loading.