diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 56fcd63..8d69673 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -348,11 +348,13 @@ through entry points; see the seven groups in `src/usersim/engine/core/_extensions.py`. Give a probe or scorer a name no other probe or scorer uses: registering one under a taken name raises an error naming both, and an extension that does so is skipped. Start every name -your package adds with its own prefix, such as `acme_recipes`; unprefixed +your package adds with its own prefix, such as `my_bookings`; unprefixed names belong to the built-ins, and [`architecture/plugins.md`](architecture/plugins.md) lists the names this covers. [`docs/extensions.md`](docs/extensions.md) builds a whole extension -package, a probe with its bank, a scorer and their tests, step by step. +package, a probe with its bank, a scorer and their tests, step by step. Import +from `usersim.extension` and `usersim.testing`; +[`docs/extension_api.md`](docs/extension_api.md) lists every name they export. To check your extension satisfies the contract: diff --git a/architecture/plugins.md b/architecture/plugins.md index a612744..c0b2e01 100644 --- a/architecture/plugins.md +++ b/architecture/plugins.md @@ -36,13 +36,16 @@ my_probe = "my_package.probe:MyProbe" **Every name an extension adds carries its prefix.** Probe labels, scorer names, capability ids, asset domains, command names, output columns, warning kinds and metadata keys all start with the package's own prefix, such as -`acme_recipes` or `acme_used_placeholder_recipe`. Unprefixed names belong to +`my_bookings` or `my_used_placeholder_booking`. Unprefixed names belong to the built-ins. The registries refuse a name that is already taken, so the prefix is what keeps two packages, or a package and a later built-in, from claiming the same one. [`docs/extensions.md`](../docs/extensions.md) builds a whole extension -package: a probe with its own bank, a scorer, their tests and a CI recipe. +package: a probe with a bank in each of two languages, a scorer, their tests +and a CI recipe. +[`docs/extension_api.md`](../docs/extension_api.md) lists every name an +extension imports, from `usersim.extension` and `usersim.testing`. An extension probe declares the columns it writes in its class's `output_columns`, and a run keeps them alongside the core columns. Other @@ -52,7 +55,7 @@ per-row data goes in its metadata (`state.metadata`), stored with the row as An extension's warnings carry its prefix too: a bank-backed probe names its placeholder warning `_used_placeholder_`, which the capability report counts as placeholder content, and any other warning it records is a -prefixed name such as `acme_slow_tool`. +prefixed name such as `my_slow_tool`. ## How discovery behaves @@ -119,7 +122,7 @@ The scorer check is awaitable because a scorer is. The evaluator awaits every scorer, so `register_scorer` expects an async function: ```python -from usersim.engine.evaluator.scorers import register_scorer +from usersim.extension import register_scorer async def score(row, models): return {"my_axis": 1.0} @@ -160,7 +163,7 @@ its package and names that `assets` directory on the probe class: ```python from pathlib import Path -from usersim.engine.core.probes import BaseProbe +from usersim.extension import BaseProbe class MyProbe(BaseProbe): label = "my_probe" diff --git a/architecture/probes.md b/architecture/probes.md index 908dc9a..97bbe13 100644 --- a/architecture/probes.md +++ b/architecture/probes.md @@ -30,10 +30,13 @@ Shipped probes are listed by module in `BUILTIN_PROBE_MODULES` looked up, which is when their decorators run. Out-of-tree probes are imported through the `usersim.probes` entry-point group instead. -## Hooks the loop calls +## Hooks -`core/simulation.py` calls these on the probe. Only the first two are -required; the rest have working defaults on `BaseProbe`. +The conversation loop in `core/simulation.py` calls most of these on the probe; +the table says where the others are called from. Only the first two are +required, plus `derive_task` on a `BankBackedProbe` and `execute_tool_call` +with `ToolExecutionMixin`; the rest have working defaults on `BaseProbe` or the +mixin that adds them. | Hook | When | |---|---| @@ -45,6 +48,7 @@ required; the rest have working defaults on `BaseProbe`. | `format_followup_user_instructions` | turns after the first | | `on_followup_failure` | a follow-up turn fails its gate | | `get_tools_for_assistant` | tool-using probes | +| `transform_assistant_history` | every assistant call: the copy of the history the assistant sees, which may differ from the stored transcript | | `after_assistant_turn` | after each assistant reply; may execute tools | | `format_assistant_judge_prompt` | in-sim assessment of the assistant turn | | `is_capitulation_detected` | pressure probes checking for a flip | @@ -52,13 +56,32 @@ required; the rest have working defaults on `BaseProbe`. | `user_turn_policy` | once per trajectory: whether earlier replies may be summarized for context management, whether the last follow-ups may wrap up, which filter phrases the user may write, which names the script check skips, and a check each generated opening must pass before the gate | | `seed_state_metadata` | recording bank/seed provenance on the row | | `build_result_extras` | adding probe-specific columns to the trajectory, each declared in the class's `output_columns` | +| `should_succeed` | the end of the conversation: whether the probe completed its protocol; `False` fails the row | +| `derive_task` | constructing a `BankBackedProbe`: the bank entry the row runs on | +| `run_dispatch` | once per row, from the dispatcher: runs the conversation, by default through the loop | +| `execute_tool_call` | `ToolExecutionMixin`: each tool call the assistant makes | +| `parse_tool_call` | `ToolExecutionMixin`: reading a tool call's name and arguments | +| `on_tool_round_complete` | `ToolExecutionMixin`: after each round of tool calls | +| `on_tool_call_executed` | after each tool call runs: `ToolExecutionMixin` calls it, and a probe that runs tool calls itself must | +| `set_tool_call_observer` | an external host, to watch the tool calls the probe runs | +| `should_inline_judge_user_turn` | never (see below) | +| `should_inline_judge_assistant_turn` | never (see below) | + +`should_inline_judge_user_turn` and `should_inline_judge_assistant_turn` are +defined on `BaseProbe`, but nothing calls them: every generated user turn goes +through the gate and every assistant turn through the judge, whatever they +return. + +Two helpers are for the probe to call: + +| Helper | Gives | +|---|---| +| `theme` | the row's theme, one of those `themes` lists | +| `assets_dir` | the probe's asset folder, `/