Skip to content

feat(extension): add usersim.extension and docs checked against code #115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
3mei wants to merge 6 commits into from
Open

feat(extension): add usersim.extension and docs checked against code #115

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a57313b
docs(engine): list every probe hook and declaration, checked by a test
3mei Oct 30, 2026
7c81502
test(engine): limit how often shared code names a probe
3mei Oct 31, 2026
849e295
feat(extension): add usersim.extension, one module to import the exte…
3mei Nov 1, 2026
4901764
docs(extension): import from usersim.extension in the examples
3mei Nov 1, 2026
57ed9b8
docs(extension): replace the examples with a multilingual booking probe
3mei Nov 1, 2026
40bcc16
refactor(engine): use my_ as the example extension prefix
3mei Nov 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -348,11 +348,13 @@ through entry points; see the seven groups in
`src/usersim/engine/core/_extensions.py`. Give a probe or scorer a name no
other probe or scorer uses: registering one under a taken name raises an
error naming both, and an extension that does so is skipped. Start every name
your package adds with its own prefix, such as `acme_recipes`; unprefixed
your package adds with its own prefix, such as `my_bookings`; unprefixed
names belong to the built-ins, and
[`architecture/plugins.md`](architecture/plugins.md) lists the names this
covers. [`docs/extensions.md`](docs/extensions.md) builds a whole extension
package, a probe with its bank, a scorer and their tests, step by step.
package, a probe with its bank, a scorer and their tests, step by step. Import
from `usersim.extension` and `usersim.testing`;
[`docs/extension_api.md`](docs/extension_api.md) lists every name they export.

To check your extension satisfies the contract:

Expand Down
13 changes: 8 additions & 5 deletions architecture/plugins.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,13 +36,16 @@ my_probe = "my_package.probe:MyProbe"
**Every name an extension adds carries its prefix.** Probe labels, scorer
names, capability ids, asset domains, command names, output columns, warning
kinds and metadata keys all start with the package's own prefix, such as
`acme_recipes` or `acme_used_placeholder_recipe`. Unprefixed names belong to
`my_bookings` or `my_used_placeholder_booking`. Unprefixed names belong to
the built-ins. The registries refuse a name that is already taken, so the
prefix is what keeps two packages, or a package and a later built-in, from
claiming the same one.

[`docs/extensions.md`](../docs/extensions.md) builds a whole extension
package: a probe with its own bank, a scorer, their tests and a CI recipe.
package: a probe with a bank in each of two languages, a scorer, their tests
and a CI recipe.
[`docs/extension_api.md`](../docs/extension_api.md) lists every name an
extension imports, from `usersim.extension` and `usersim.testing`.

An extension probe declares the columns it writes in its class's
`output_columns`, and a run keeps them alongside the core columns. Other
Expand All @@ -52,7 +55,7 @@ per-row data goes in its metadata (`state.metadata`), stored with the row as
An extension's warnings carry its prefix too: a bank-backed probe names its
placeholder warning `<prefix>_used_placeholder_<thing>`, which the capability
report counts as placeholder content, and any other warning it records is a
prefixed name such as `acme_slow_tool`.
prefixed name such as `my_slow_tool`.

## How discovery behaves

Expand Down Expand Up @@ -119,7 +122,7 @@ The scorer check is awaitable because a scorer is. The evaluator awaits every
scorer, so `register_scorer` expects an async function:

```python
from usersim.engine.evaluator.scorers import register_scorer
from usersim.extension import register_scorer

async def score(row, models):
return {"my_axis": 1.0}
Expand Down Expand Up @@ -160,7 +163,7 @@ its package and names that `assets` directory on the probe class:
```python
from pathlib import Path

from usersim.engine.core.probes import BaseProbe
from usersim.extension import BaseProbe

class MyProbe(BaseProbe):
label = "my_probe"
Expand Down
72 changes: 64 additions & 8 deletions architecture/probes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,13 @@ Shipped probes are listed by module in `BUILTIN_PROBE_MODULES`
looked up, which is when their decorators run. Out-of-tree probes are
imported through the `usersim.probes` entry-point group instead.

## Hooks the loop calls
## Hooks

`core/simulation.py` calls these on the probe. Only the first two are
required; the rest have working defaults on `BaseProbe`.
The conversation loop in `core/simulation.py` calls most of these on the probe;
the table says where the others are called from. Only the first two are
required, plus `derive_task` on a `BankBackedProbe` and `execute_tool_call`
with `ToolExecutionMixin`; the rest have working defaults on `BaseProbe` or the
mixin that adds them.

| Hook | When |
|---|---|
Expand All @@ -45,20 +48,40 @@ required; the rest have working defaults on `BaseProbe`.
| `format_followup_user_instructions` | turns after the first |
| `on_followup_failure` | a follow-up turn fails its gate |
| `get_tools_for_assistant` | tool-using probes |
| `transform_assistant_history` | every assistant call: the copy of the history the assistant sees, which may differ from the stored transcript |
| `after_assistant_turn` | after each assistant reply; may execute tools |
| `format_assistant_judge_prompt` | in-sim assessment of the assistant turn |
| `is_capitulation_detected` | pressure probes checking for a flip |
| `allow_early_stop_at_turn` | deciding whether the conversation may end |
| `user_turn_policy` | once per trajectory: whether earlier replies may be summarized for context management, whether the last follow-ups may wrap up, which filter phrases the user may write, which names the script check skips, and a check each generated opening must pass before the gate |
| `seed_state_metadata` | recording bank/seed provenance on the row |
| `build_result_extras` | adding probe-specific columns to the trajectory, each declared in the class's `output_columns` |
| `should_succeed` | the end of the conversation: whether the probe completed its protocol; `False` fails the row |
| `derive_task` | constructing a `BankBackedProbe`: the bank entry the row runs on |
| `run_dispatch` | once per row, from the dispatcher: runs the conversation, by default through the loop |
| `execute_tool_call` | `ToolExecutionMixin`: each tool call the assistant makes |
| `parse_tool_call` | `ToolExecutionMixin`: reading a tool call's name and arguments |
| `on_tool_round_complete` | `ToolExecutionMixin`: after each round of tool calls |
| `on_tool_call_executed` | after each tool call runs: `ToolExecutionMixin` calls it, and a probe that runs tool calls itself must |
| `set_tool_call_observer` | an external host, to watch the tool calls the probe runs |
| `should_inline_judge_user_turn` | never (see below) |
| `should_inline_judge_assistant_turn` | never (see below) |

`should_inline_judge_user_turn` and `should_inline_judge_assistant_turn` are
defined on `BaseProbe`, but nothing calls them: every generated user turn goes
through the gate and every assistant turn through the judge, whatever they
return.

Two helpers are for the probe to call:

| Helper | Gives |
|---|---|
| `theme` | the row's theme, one of those `themes` lists |
| `assets_dir` | the probe's asset folder, `<root>/<label>/`, where its bank loaders read |

Anything a probe does not override inherits a default, so a new probe declares
only what is specific to it. Its `description`, a paragraph on what it tests, is
what the per-row preview and the capability report show with its rows, and its
`preview_fields` are what the preview's Probe Inputs panel shows about the bank
entry or theme each row drew. Its `default_scorers` are the scorers
`usersim eval --scorers auto` runs when a run holds its rows.
only what is specific to it; [What a probe declares](#what-a-probe-declares)
lists each declaration.

A probe's own columns are the one part of this that is declared twice. Whatever
`build_result_extras` returns is written only if the name is a core column or
Expand Down Expand Up @@ -90,6 +113,39 @@ Six hooks can reach a model, and the loop awaits them: `get_verbatim_first_user_
`async def`. `usersim.testing.assert_probe_conforms` reports a plain `def` override,
which would otherwise raise partway through a conversation.

## What a probe declares

Everything else the core needs from a probe it reads from the class: an
attribute, or a classmethod where the answer depends on the locale or the run's
settings. Code outside `engine/probes/` writes a probe's label only as often as
[`tests/test_probe_names_in_shared_code.py`](../tests/test_probe_names_in_shared_code.py)
allows, file by file, so what a new probe needs from the core goes here rather
than into a branch on its label.

| Declaration | Default | Read by |
|---|---|---|
| `label` | none: required | the registry, and every stage that names the probe |
| `family`, `prompt_version`, `variants` | set by `@register_probe` | the row's `trajectory_id`, provenance and `probe_family` column |
| `description` | `""` | the per-row preview and the capability report, beside the probe's rows |
| `preview_fields` | `()` | the per-row preview's Probe Inputs panel |
| `default_scorers` | `()` | `usersim eval --scorers auto`, for a run that holds the probe's rows |
| `assets_root` | `None`: the packaged banks | `assets_dir()`, and through it the probe's loaders, the pre-flight and smoke |
| `locale_bank_check` | `None`: the asset folder holds bank files | the pre-flight, for each run locale |
| `bank_locale` | an India variant's own bank when it has one, else its base locale's | the pre-flight |
| `authoring_guide` | `None` | the pre-flight, when a run asks for a locale the probe has no bank for |
| `asset_problems` | no problems | `usersim smoke` and `assert_probe_conforms` |
| `themes` | `None`: no theme | run setup, which draws one per row into the `theme` column |
| `model_aliases` | `()` | run setup, which adds each alias the models file defines to every row's models |
| `output_columns` | `()` | the run's `side_effect_columns`; a column left out is dropped |
| `supports_assistant_resampling` | `True`; `False` with the tool mixins | judge-gated assistant resampling (`max_assistant_attempts > 1`) |
| `bank_loader` | none: required on a `BankBackedProbe` | constructing a `BankBackedProbe`, for the row's locale |
| `placeholder_warning_kind` | none: required on a `BankBackedProbe` | the warning a row records when its task is a placeholder, which the capability report counts |
| `bank_version_key` | `None`: one bank per locale | where provenance records the bank's version |
| `verbatim_field` | `"verbatim_text"` | `BankVerbatimMixin`: the task field turn 1 reads |
| `turn1_instruction_template` | `""` | `CustomTurn1InstructionMixin`: the instruction turn 1 is written from |
| `tool_loop_mode` | `"single"` | `ToolExecutionMixin`: one round of tool calls per turn, or with `"multi"` rounds until the assistant stops calling tools |
| `tool_max_calls_per_turn` | `8` | `ToolExecutionMixin`: the most tool calls it runs in one turn |

## Mixins

Six mixins cover the shapes that recur, all in `core/probes.py`:
Expand Down
Loading
Loading