From bf38a0193aa4cd948c66b7130d349f939a9bf3ca Mon Sep 17 00:00:00 2001 From: Yev Meyer Date: Mon, 2 Nov 2026 18:25:13 -0500 Subject: [PATCH 1/4] docs: add an extension hook request form and rules for new hooks The Extension hook form asks for the need in general terms, why the existing hooks do not cover it, a proposed shape, and whether it works when a host drives the probe. Adding a hook in CONTRIBUTING.md lists what a new hook does: it is general, keeps the current behaviour by default, is async when it can call a model, passes assert_hosted_parity, and comes with a use in the extension fixture and updates to the conformance checks, templates and docs. The feature request form and docs/extensions.md point to it. Signed-off-by: Yev Meyer --- .github/ISSUE_TEMPLATE/extension-hook.yml | 84 ++++++++++++++++++++++ .github/ISSUE_TEMPLATE/feature-request.yml | 3 +- CONTRIBUTING.md | 25 +++++++ docs/extensions.md | 4 +- 4 files changed, 114 insertions(+), 2 deletions(-) create mode 100644 .github/ISSUE_TEMPLATE/extension-hook.yml diff --git a/.github/ISSUE_TEMPLATE/extension-hook.yml b/.github/ISSUE_TEMPLATE/extension-hook.yml new file mode 100644 index 0000000..da5b723 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/extension-hook.yml @@ -0,0 +1,84 @@ +name: πŸ”Œ Extension hook +description: Ask for a hook your probe, scorer or other extension needs and User Sim does not offer +labels: ["enhancement"] +body: + - type: markdown + attributes: + value: >- + Use this form when an extension needs something no hook, class + attribute or entry point offers. A maintainer reviews the request, + agrees the hook's shape with you and adds the `triaged` label when a + pull request is wanted. + [Adding a hook](https://github.com/NVIDIA-NeMo/UserSim/blob/main/CONTRIBUTING.md#adding-a-hook) + lists what a new hook must do. + + - type: textarea + id: need + attributes: + label: What does your extension need to do? + description: >- + The need in general terms: what the extension has to read or change, + and at which stage of a run (setup, the conversation, the stored + columns, scoring or the report). The general shape is enough to design + a hook, so leave out anything confidential about your project. + validations: + required: true + + - type: textarea + id: gap + attributes: + label: Why don't the existing hooks cover it? + description: >- + The hooks, class attributes and entry points you looked at, in + [architecture/probes.md](https://github.com/NVIDIA-NeMo/UserSim/blob/main/architecture/probes.md) + and [docs/extension_api.md](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/extension_api.md), + and where each falls short. + validations: + required: true + + - type: textarea + id: shape + attributes: + label: What shape do you propose? + description: >- + A method, class attribute or entry point, its signature, and what + happens for a probe that does not define it. That default should keep + today's behaviour. + validations: + required: true + + - type: textarea + id: hosted + attributes: + label: Does it work when a host drives the probe? + description: >- + A host can run a probe from its own loop through `ProbeEpisodeRuntime`, + as + [EXTERNAL_PROBE_RUNTIME.md](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/engine/EXTERNAL_PROBE_RUNTIME.md) + describes. Say whether the hook runs there too, and whether it calls a + model. + validations: + required: true + + - type: textarea + id: investigation + attributes: + label: Prior investigation + description: >- + What you or your coding agent found in the code, documentation and + existing issues, such as a hook that covers part of this or an open + request for the same thing. If an agent investigated, paste its + findings. + placeholder: Paste the agent's findings, or describe what you checked. + + - type: checkboxes + id: checklist + attributes: + label: Checklist + options: + - label: I searched existing issues and the documentation, or had my agent do so + required: true + - label: The request describes the need in general terms and holds nothing confidential + required: true + - label: If an agent helped, its findings are under Prior investigation + required: false diff --git a/.github/ISSUE_TEMPLATE/feature-request.yml b/.github/ISSUE_TEMPLATE/feature-request.yml index e3fd988..13caf17 100644 --- a/.github/ISSUE_TEMPLATE/feature-request.yml +++ b/.github/ISSUE_TEMPLATE/feature-request.yml @@ -9,7 +9,8 @@ body: change is wanted. If you plan to build this yourself, open the pull request once the issue has that label; see [Issues before pull requests](https://github.com/NVIDIA-NeMo/UserSim/blob/main/CONTRIBUTING.md#issues-before-pull-requests). - To propose a probe you plan to build, use the New probe form instead. + To propose a probe you plan to build, use the New probe form instead, + and to ask for a hook your extension needs, the Extension hook form. - type: textarea id: problem diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8d69673..b5198f1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -445,6 +445,31 @@ To propose a probe, open the Triage settles where it lives: in this repository, as an extension, or as an extension first. +### Adding a hook + +When a probe, scorer or other extension needs something no hook, class +attribute or entry point offers, open the +[Extension hook form](https://github.com/NVIDIA-NeMo/UserSim/issues/new?template=extension-hook.yml). +[`architecture/probes.md`](architecture/probes.md) lists the hooks and +declarations a probe has, and [`docs/extension_api.md`](docs/extension_api.md) +the names an extension imports. + +A new hook: + +- **is general:** it serves probes beyond the one that asks for it, and its + name and documentation describe that general use. +- **keeps today's behaviour by default:** a probe that does not define it runs, + and stores its rows, as before. +- **is `async def` if it can call a model,** so the loop awaits it. +- **works when a host drives the probe:** `assert_hosted_parity` passes for a + probe that uses it. +- **comes with its uses:** the extension fixture in + `tests/fixtures/extension_package/` uses it, so `make check-extensions` + proves it works from an installed package. The pull request that adds it + also updates the conformance checks, `templates/probe/`, + [`architecture/probes.md`](architecture/probes.md) and + [`docs/engine/AUTHORING_A_PROBE.md`](docs/engine/AUTHORING_A_PROBE.md). + ## Code style `ruff` owns formatting and linting; there is no separate style config to diff --git a/docs/extensions.md b/docs/extensions.md index 886ece4..6d7d41e 100644 --- a/docs/extensions.md +++ b/docs/extensions.md @@ -11,7 +11,9 @@ the booking the assistant makes. point groups and how discovery behaves, [`docs/engine/AUTHORING_A_PROBE.md`](engine/AUTHORING_A_PROBE.md) covers every hook a probe can implement, and [`extension_api.md`](extension_api.md) lists -every name the package can import from `usersim`. +every name the package can import from `usersim`. When a package needs a hook +User Sim does not have, +[Adding a hook](../CONTRIBUTING.md#adding-a-hook) says how to ask for one. ## Layout From 4a85db38e5f49710ccfc0411bd02b3e416ba6684 Mon Sep 17 00:00:00 2001 From: Yev Meyer Date: Mon, 2 Nov 2026 18:25:36 -0500 Subject: [PATCH 2/4] feat(extension): export get_prompt and load_prompt_file An extension can keep its prompts in a prompts.yaml under its assets_root and read them with get_prompt, as built-in probes do: load_prompt_file reads the file from the folder the probe's banks resolve to, a declared assets_root included. docs/extension_api.md lists both. Signed-off-by: Yev Meyer --- docs/extension_api.md | 2 ++ src/usersim/extension.py | 6 ++++-- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/extension_api.md b/docs/extension_api.md index c1a1ae8..385407e 100644 --- a/docs/extension_api.md +++ b/docs/extension_api.md @@ -85,6 +85,8 @@ does not load the conversation loop. | `format_conversation_history_for_prompt` | A conversation as a text block for a judge prompt | | `format_tools_for_prompt` | Tool definitions as a readable prompt block | | `render_prompt` | Fills a template's placeholders from given values, then from the row | +| `load_prompt_file` | A probe's `prompts.yaml` as a mapping, read from the folder its banks resolve to; empty when there is none | +| `get_prompt` | One prompt from a probe's `prompts.yaml`, or the default it is given when the file or the key is missing | ### Locales and languages diff --git a/src/usersim/extension.py b/src/usersim/extension.py index f119334..d823c31 100644 --- a/src/usersim/extension.py +++ b/src/usersim/extension.py @@ -63,7 +63,7 @@ ToolExecutionMixin, register_probe, ) - from usersim.engine.core.prompt_loader import render_prompt + from usersim.engine.core.prompt_loader import get_prompt, load_prompt_file, render_prompt from usersim.engine.core.simulation import ( ConversationState, finalize_completed, @@ -119,6 +119,8 @@ "format_interaction_style_instructions", "format_persona_for_prompt", "format_tools_for_prompt", + "get_prompt", + "load_prompt_file", "persona_full_name", "persona_uuid", "render_prompt", @@ -189,7 +191,7 @@ "ToolExecutionMixin", "register_probe", ), - "usersim.engine.core.prompt_loader": ("render_prompt",), + "usersim.engine.core.prompt_loader": ("get_prompt", "load_prompt_file", "render_prompt"), "usersim.engine.core.simulation": ( "ConversationState", "finalize_completed", From a19abb422cdd8904a07d4c11e06168795949e40c Mon Sep 17 00:00:00 2001 From: Yev Meyer Date: Mon, 2 Nov 2026 18:25:44 -0500 Subject: [PATCH 3/4] docs: lead the probe guides with areas the persona explores A probe sets the area its conversations explore and the bounds a scorer checks, and the simulated user explores that area in the persona's own words. Step 0 of the authoring guide says so and starts its decision tree with the shapes in which the user model writes turn 1, keeping a verbatim turn for probes whose exact wording is what they test. Its step examples subclass BankBackedProbe alone where a mixin was incidental. Both bookings examples follow it: the bank names what can be booked, draw_booking picks each row's date, time and number of people, and the user model asks for the booking in its own words, with the gate checking turn 1. The BaseProbe template draws a topic per row from themes, and the template README says when a verbatim turn fits. Signed-off-by: Yev Meyer --- docs/engine/AUTHORING_A_PROBE.md | 205 +++++++++++++------- docs/extensions.md | 178 +++++++++++------ templates/probe/README.md | 7 + templates/probe/generator_baseprobe_only.py | 29 ++- tests/test_extension_api.py | 1 + 5 files changed, 287 insertions(+), 133 deletions(-) diff --git a/docs/engine/AUTHORING_A_PROBE.md b/docs/engine/AUTHORING_A_PROBE.md index 85ebca6..a6767b9 100644 --- a/docs/engine/AUTHORING_A_PROBE.md +++ b/docs/engine/AUTHORING_A_PROBE.md @@ -33,46 +33,56 @@ piece to pick and *how* to wire it. ## Step 0: pick your shape +A probe sets the area its conversations explore and the bounds of that area: +the topics or tasks, and the facts a scorer checks. The simulated user, played +from a persona, explores the area in its own words, so rows on the same topic +open differently and go different ways, and a run covers what its personas +bring to it. Give the user model an area and its bounds rather than a message +to send: a fixed opening starts every row in a locale the same way, and the run +explores only what that one message reaches. Send a bank's text verbatim only +when its exact wording is what the probe tests. + Use this decision tree to pick which substrate combination matches your probe: ``` -Does the probe have an asset bank (curated facts / queries / strategies / actions)? +Who writes the user's first message? β”‚ -β”œβ”€β”€ NO β†’ BaseProbe alone -β”‚ Examples: general_open_ended, general_educational -β”‚ Template: templates/probe/generator_baseprobe_only.py +β”œβ”€β”€ The user model, in the persona's voice, inside an area the probe sets +β”‚ (start here: the persona's own wording IS the feature) +β”‚ β”‚ +β”‚ β”œβ”€β”€ The areas are topics, listed per locale +β”‚ β”‚ β†’ BaseProbe, with a themes(locale) classmethod +β”‚ β”‚ (see "Give each row a theme" in Step 3) +β”‚ β”‚ Examples: general_open_ended, general_educational +β”‚ β”‚ Template: templates/probe/generator_baseprobe_only.py +β”‚ β”‚ +β”‚ └── The areas come from a curated bank, with facts a scorer checks +β”‚ β†’ BankBackedProbe, with the area and its facts in the user system prompt +β”‚ Examples: sov_ai_dynamic, the bookings worked example below +β”‚ Template: templates/probe/generator_custom_turn1.py +β”‚ β†’ or CustomTurn1InstructionMixin, when only turn 1 follows +β”‚ an instruction of its own β”‚ -└── YES β†’ BankBackedProbe + mixin (depends on turn-1 shape) - β”‚ - β”œβ”€β”€ Turn-1 should be the bank's text VERBATIM - β”‚ (false-premise wording, exact phrasing for matched-pair joins, - β”‚ or reviewer-authored content must survive unchanged) - β”‚ β”‚ - β”‚ β”œβ”€β”€ Single back-and-forth conversation - β”‚ β”‚ β†’ BankVerbatimMixin - β”‚ β”‚ Examples: sov_ai_facts, sov_ai_multilingual_parity - β”‚ β”‚ Template: templates/probe/generator_bank_verbatim.py - β”‚ β”‚ - β”‚ β”œβ”€β”€ Multi-turn with bank-driven per-turn nudges (reframings) - β”‚ β”‚ β†’ BankReframingMixin - β”‚ β”‚ Example: safety_chat_pressure - β”‚ β”‚ Template: templates/probe/generator_bank_reframing.py - β”‚ β”‚ - β”‚ └── Single user turn + multi-assistant-turn tool loop - β”‚ (NO user-LLM at all; assistant tool-calls intercepted) - β”‚ β†’ AgenticMixin (custom run_dispatch) - β”‚ Example: safety_agentic - β”‚ Template: templates/probe/generator_agentic.py - β”‚ - └── Turn-1 should be GENERATED by the user-LLM in the persona's - voice, but ANCHORED to a bank-derived hint - (paraphrasing through persona voice IS the feature) - β†’ BankBackedProbe, with the hint in the user system prompt - Example: sov_ai_dynamic - Template: templates/probe/generator_custom_turn1.py - β†’ or CustomTurn1InstructionMixin, when only turn 1 follows - an instruction of its own +└── A bank's text, VERBATIM, because its exact wording is what the probe tests + (false-premise wording, exact phrasing for matched-pair joins, + or reviewer-authored content that must survive unchanged) + β”‚ + β”œβ”€β”€ Single back-and-forth conversation + β”‚ β†’ BankVerbatimMixin + β”‚ Examples: sov_ai_facts, sov_ai_multilingual_parity + β”‚ Template: templates/probe/generator_bank_verbatim.py + β”‚ + β”œβ”€β”€ Multi-turn with bank-driven per-turn nudges (reframings) + β”‚ β†’ BankReframingMixin + β”‚ Example: safety_chat_pressure + β”‚ Template: templates/probe/generator_bank_reframing.py + β”‚ + └── Single user turn + multi-assistant-turn tool loop + (NO user-LLM at all; assistant tool-calls intercepted) + β†’ AgenticMixin (custom run_dispatch) + Example: safety_agentic + Template: templates/probe/generator_agentic.py ``` There's also a **tool-calling capability** shape (test the assistant's @@ -174,7 +184,7 @@ notifying. See `docs/engine/EXTERNAL_PROBE_RUNTIME.md`. For `BankBackedProbe` subclasses: ```python -class MyProbe(BankVerbatimMixin, BankBackedProbe): +class MyProbe(BankBackedProbe): label = "my_probe" # required bank_loader = staticmethod(_my_bank_loader) # required placeholder_warning_kind = WarningKind.USED_PLACEHOLDER_FACT # required @@ -321,7 +331,7 @@ def _bank_loader(locale: str = "en_US"): return _load_bank(locale) -class MyProbe(BankVerbatimMixin, BankBackedProbe): +class MyProbe(BankBackedProbe): bank_loader = staticmethod(_bank_loader) ``` @@ -364,7 +374,7 @@ def _check_locale_bank(locale_dir: Path) -> None: MyBank.from_yaml((locale_dir / "sample.yaml").read_text(encoding="utf-8")) -class MyProbe(BankVerbatimMixin, BankBackedProbe): +class MyProbe(BankBackedProbe): locale_bank_check = staticmethod(_check_locale_bank) authoring_guide = "https://example.com/my-probe/README.md#adding-a-locale" ``` @@ -388,7 +398,7 @@ bank fails a check rather than a run. Return one string per problem, or an empty list, and read what your loaders read without calling a model: ```python -class MyProbe(BankVerbatimMixin, BankBackedProbe): +class MyProbe(BankBackedProbe): @classmethod def asset_problems(cls) -> list[str]: problems = [] @@ -520,7 +530,7 @@ from usersim.extension import register_probe prompt_version="v1.0", # MyProbe.prompt_version variants=("default",), # MyProbe.variants ) -class MyProbe(BankVerbatimMixin, BankBackedProbe): +class MyProbe(BankBackedProbe): label = "my_probe" ... ``` @@ -631,8 +641,10 @@ the assistant has a `book_appointment` tool that takes them in one fixed form. The probe ships banks for `en_US` and `pt_BR`; the same probe, shipped in its own package, is the example in [`docs/extensions.md`](../extensions.md). -We'll use the **banked-verbatim** shape, so each date reaches the assistant -exactly as the bank writes it, with `ToolExecutionMixin` to answer the +The probe sets the area and its bounds, and the persona does the rest: the bank +names what can be booked, the probe draws a date, a time and a number of people +for each row, and the user model asks for the booking in its own words. It is a +`BankBackedProbe` with no verbatim turn, and `ToolExecutionMixin` answers the assistant's tool call. ### File 1: the banks, `src/usersim/engine/probes/bookings/assets/bookings//bookings.yaml` @@ -642,12 +654,15 @@ assistant's tool call. ```yaml bank_id: bookings_en_US bank_version: "1.0.0" -bookings: +appointments: - id: museum_tour - opening: Hi! Can you book the guided museum tour for 04/03/2027 at 3:30 pm? - date: "2027-04-03" - time: "15:30" - people: 3 + what: the guided tour of the city museum + placeholder: true + - id: restaurant + what: a table at a restaurant + placeholder: true + - id: pottery_class + what: a pottery class placeholder: true ``` @@ -656,26 +671,37 @@ bookings: ```yaml bank_id: bookings_pt_BR bank_version: "1.0.0" -bookings: +appointments: - id: museum_tour - opening: Oi! VocΓͺ pode reservar a visita guiada ao museu para 04/03/2027 Γ s 15h30? - date: "2027-03-04" - time: "15:30" - people: 3 + what: a visita guiada ao museu da cidade + placeholder: true + - id: restaurant + what: uma mesa num restaurante + placeholder: true + - id: pottery_class + what: uma aula de cerΓ’mica placeholder: true ``` -Both open with `04/03/2027`: 3 April to a user writing American English, and -4 March to one writing Brazilian Portuguese. Neither says how many people are -coming, so an assistant that books on its first reply has to guess. +Both name the same three appointments, each in its bank's language. The +banks hold no dates: `draw_booking` in File 2 draws one per row, with the day +and the month different numbers no greater than 12, so the date reads as another +date when its numbers are read in the other order. For `2027-03-06`, 6 March, a +user who writes the day first, as people in Brazil do, writes `06/03/2027`, and +one who writes the month first, as people in the United States do, writes +`03/06/2027`; an assistant that reads every date one way gets one of them +wrong. Whether the user writes the date in numbers or in words is the user +model's choice. The user says how many people are coming only when asked, so an +assistant that books on its first reply has to guess. ### File 2: `src/usersim/engine/probes/bookings/bank.py` ```python -"""Booking banks: one per locale, under the probe's asset folder.""" +"""Appointment banks: one per locale, under the probe's asset folder.""" from __future__ import annotations +import random from dataclasses import dataclass from pathlib import Path @@ -686,12 +712,9 @@ LOCALES = ("en_US", "pt_BR") @dataclass(frozen=True) -class Booking: +class Appointment: id: str - opening: str # turn 1, in the locale's language; leaves out how many people - date: str # what the user means, as the tool takes it: YYYY-MM-DD - time: str # HH:MM, 24-hour - people: int # given only when the assistant asks + what: str # what the user wants booked, in the locale's language placeholder: bool @@ -699,12 +722,40 @@ class Booking: class BookingBank: bank_id: str bank_version: str - bookings: tuple[Booking, ...] + appointments: tuple[Appointment, ...] + + +@dataclass(frozen=True) +class Booking: + id: str + what: str + date: str # what the user means, as the tool takes it: YYYY-MM-DD + time: str # HH:MM, 24-hour + people: int # given only when the assistant asks + placeholder: bool def read_booking_bank(locale_dir: Path) -> BookingBank: doc = yaml.safe_load((locale_dir / "bookings.yaml").read_text(encoding="utf-8")) - return BookingBank(doc["bank_id"], doc["bank_version"], tuple(Booking(**b) for b in doc["bookings"])) + return BookingBank(doc["bank_id"], doc["bank_version"], tuple(Appointment(**a) for a in doc["appointments"])) + + +def draw_booking(appointment: Appointment, rng: random.Random) -> Booking: + """A date, a time and a number of people for ``appointment``. + + The day and the month are different numbers no greater than 12, so the date + reads as two different dates, day first and month first. + """ + month = rng.randint(1, 12) + day = rng.choice([d for d in range(1, 13) if d != month]) + return Booking( + id=appointment.id, + what=appointment.what, + date=f"2027-{month:02d}-{day:02d}", + time=f"{rng.randint(9, 17):02d}:{rng.choice(('00', '30'))}", + people=rng.randint(2, 6), + placeholder=appointment.placeholder, + ) ``` ### File 3: `src/usersim/engine/probes/bookings/__init__.py` @@ -726,14 +777,16 @@ BOOKINGS_USER_PROMPTS = LocalePromptPack( prompts={ "en_US": ( "{persona}\n\n" - "You asked an assistant to book an appointment, giving the date and time. " - "If it asks how many people are coming, the answer is {people}; say it the way you would. " + "You want an assistant to book {what} on {date} at {time}. Ask for it in your own words, " + "and write the date and time the way you would write them yourself, not in the form they are " + "given here. Say how many people are coming only if it asks; the answer is {people}. " "Once it confirms the booking, thank it." ), "pt_BR": ( "{persona}\n\n" - "VocΓͺ pediu a um assistente para marcar um horΓ‘rio e jΓ‘ disse a data e a hora. " - "Se ele perguntar quantas pessoas vΓ£o, a resposta Γ© {people}; diga isso do seu jeito. " + "VocΓͺ quer que um assistente reserve {what} para {date} Γ s {time}. PeΓ§a do seu jeito, e " + "escreva a data e a hora como vocΓͺ mesmo escreveria, nΓ£o na forma em que aparecem aqui. " + "Diga quantas pessoas vΓ£o sΓ³ se ele perguntar; a resposta Γ© {people}. " "Quando ele confirmar a reserva, agradeΓ§a." ), }, @@ -743,6 +796,8 @@ BOOKINGS_USER_PROMPTS = LocalePromptPack( The user's instructions are written in each language the probe ships, and `LocalePromptPack` refuses, when the module is imported, a locale without them. +They give the user model what to book, with the date and time in the tool's +form, and ask it to write the date and time the way it would itself. ### File 5: `src/usersim/engine/probes/bookings/probe.py` @@ -757,11 +812,10 @@ from pathlib import Path from typing import Any from usersim.engine.core.probes import dispatch_episode -from usersim.engine.probes.bookings.bank import LOCALES, Booking, read_booking_bank +from usersim.engine.probes.bookings.bank import LOCALES, Booking, draw_booking, read_booking_bank from usersim.engine.probes.bookings.prompts import BOOKINGS_USER_PROMPTS from usersim.extension import ( BankBackedProbe, - BankVerbatimMixin, OutcomeBuilder, Provenance, ToolExecutionMixin, @@ -804,7 +858,7 @@ def _bank_loader(locale: str): prompt_version=PROMPT_VERSION, variants=tuple(PROBE_VARIANTS), ) -class BookingsProbe(ToolExecutionMixin, BankVerbatimMixin, BankBackedProbe): +class BookingsProbe(ToolExecutionMixin, BankBackedProbe): label = "bookings" assets_root = Path(__file__).parent / "assets" # holds bookings/ description = ( @@ -814,7 +868,6 @@ class BookingsProbe(ToolExecutionMixin, BankVerbatimMixin, BankBackedProbe): bank_loader = staticmethod(_bank_loader) locale_bank_check = staticmethod(read_booking_bank) placeholder_warning_kind = WarningKind.USED_PLACEHOLDER_BOOKING - verbatim_field = "opening" output_columns = ("booking_id", "booking") @classmethod @@ -829,12 +882,15 @@ class BookingsProbe(ToolExecutionMixin, BankVerbatimMixin, BankBackedProbe): def derive_task(self, persona: dict, bank: Any, *, cfg: Any) -> Booking: rng = random.Random(f"{persona_uuid(persona)}:{bank.bank_version}:{cfg.random_seed}") - return rng.choice(bank.bookings) + return draw_booking(rng.choice(bank.appointments), rng) def get_user_system_prompt(self) -> str: return BOOKINGS_USER_PROMPTS.get( self._locale, persona=format_persona_for_prompt(self._persona), + what=self._task.what, + date=self._task.date, + time=self._task.time, people=self._task.people, ) @@ -909,7 +965,12 @@ async def simulate_bookings( assistant has booked, so a user who thanks it early does not end it first. - `should_succeed` keeps its default: an assistant that never books has not broken the probe's protocol, so its row is scored like any other. -- The probe keeps `BaseProbe`'s gate prompt (Step 5). +- `derive_task` picks the row's appointment, and `draw_booking` its date, time + and number of people, with a generator seeded from the persona, the bank's + version and the run's seed, so a row draws the same booking every time it + runs. +- The user model writes every user turn, the first included, and the probe + keeps `BaseProbe`'s gate prompt (Step 5) to check each one. ### File 6: the placeholder warning diff --git a/docs/extensions.md b/docs/extensions.md index 6d7d41e..3a4005d 100644 --- a/docs/extensions.md +++ b/docs/extensions.md @@ -94,6 +94,15 @@ asyncio_mode = "auto" ## The probe +A probe sets the area its conversations explore and the bounds a scorer +checks, and the simulated user explores that area in the persona's own words. +Here the area is booking an appointment. The bank names what can be booked, +the probe draws a date, a time and a number of people for each row, and the +user model asks for the booking the way its persona would, so rows that draw +the same appointment still open differently. A bank's text goes to the +assistant verbatim only when its exact wording is what a probe tests, such as +a false premise. + ```python import json import random @@ -104,7 +113,6 @@ from pathlib import Path import yaml from usersim.extension import ( BankBackedProbe, - BankVerbatimMixin, PreviewField, ToolExecutionMixin, format_persona_for_prompt, @@ -135,12 +143,9 @@ BOOK_APPOINTMENT = { @dataclass(frozen=True) -class Booking: +class Appointment: id: str - opening: str - date: str - time: str - people: int + what: str placeholder: bool @@ -148,12 +153,22 @@ class Booking: class BookingBank: bank_id: str bank_version: str - bookings: tuple[Booking, ...] + appointments: tuple[Appointment, ...] + + +@dataclass(frozen=True) +class Booking: + id: str + what: str + date: str + time: str + people: int + placeholder: bool def read_booking_bank(locale_dir: Path) -> BookingBank: doc = yaml.safe_load((locale_dir / "bookings.yaml").read_text(encoding="utf-8")) - return BookingBank(doc["bank_id"], doc["bank_version"], tuple(Booking(**b) for b in doc["bookings"])) + return BookingBank(doc["bank_id"], doc["bank_version"], tuple(Appointment(**a) for a in doc["appointments"])) @cache @@ -161,20 +176,37 @@ def load_booking_bank(locale: str) -> BookingBank: return read_booking_bank(BookingProbe.assets_dir() / locale) +def draw_booking(appointment: Appointment, rng: random.Random) -> Booking: + """A date, a time and a number of people for ``appointment``. + + The day and the month are different numbers no greater than 12, so the date + reads as two different dates, day first and month first. + """ + month = rng.randint(1, 12) + day = rng.choice([d for d in range(1, 13) if d != month]) + return Booking( + id=appointment.id, + what=appointment.what, + date=f"2027-{month:02d}-{day:02d}", + time=f"{rng.randint(9, 17):02d}:{rng.choice(('00', '30'))}", + people=rng.randint(2, 6), + placeholder=appointment.placeholder, + ) + + @register_probe(family="my_bookings", prompt_version="v1.0", variants=("default",)) -class BookingProbe(ToolExecutionMixin, BankVerbatimMixin, BankBackedProbe): +class BookingProbe(ToolExecutionMixin, BankBackedProbe): label = "my_bookings" description = ( "Tests whether an assistant books what the user means: a date, a time and a number of people, " "written the way people write them in the user's language." ) - preview_fields = (PreviewField("my_booking_id", "Booking id"),) + preview_fields = (PreviewField("my_booking_id", "Appointment"),) default_scorers = ("my_bookings",) assets_root = Path(__file__).parent / "assets" bank_loader = staticmethod(load_booking_bank) locale_bank_check = staticmethod(read_booking_bank) placeholder_warning_kind = "my_used_placeholder_booking" - verbatim_field = "opening" output_columns = ("my_booking_id", "my_booking") @classmethod @@ -189,15 +221,18 @@ class BookingProbe(ToolExecutionMixin, BankVerbatimMixin, BankBackedProbe): def derive_task(self, persona, bank, *, cfg): rng = random.Random(f"{persona_uuid(persona)}:{bank.bank_version}:{cfg.random_seed}") - return rng.choice(bank.bookings) + return draw_booking(rng.choice(bank.appointments), rng) def get_user_system_prompt(self) -> str: + task = self._task return "\n\n".join( [ format_persona_for_prompt(self._persona), - "You asked an assistant to book an appointment, giving the date and time. " - f"If it asks how many people are coming, the answer is {self._task.people}; " - "say it the way you would. Once it confirms the booking, thank it.", + f"You want an assistant to book {task.what} on {task.date} at {task.time}. " + "Ask for it in your own words, and write the date and time the way you would write them " + "yourself, not in the form they are given here. " + f"Say how many people are coming only if it asks; the answer is {task.people}. " + "Once it confirms the booking, thank it.", language_instruction(self._language, self._locale), ] ) @@ -234,13 +269,12 @@ class BookingProbe(ToolExecutionMixin, BankVerbatimMixin, BankBackedProbe): `book_appointment` in `get_tools_for_assistant`, and `execute_tool_call` answers each call. It records the first booking in `state.metadata`, with how many user turns came before it. -- **`BankVerbatimMixin`** sends the booking's `opening`, the field - `verbatim_field` names, as turn 1, so the date reaches the assistant written - exactly as the user wrote it. -- **`get_user_system_prompt`** sets up the user model for the turns after the - first: the persona, the number of people it gives only when the assistant - asks, and `language_instruction`, which keeps it writing in the run's - language. +- **`get_user_system_prompt`** gives the user model the persona, what to book, + the date and time in the tool's form, and the number of people, to give only + if the assistant asks. It asks for the date and time written the way the user + would write them, and `language_instruction` keeps the user writing in the + run's language. The user model writes every user turn, the first included, + and the gate checks each one. - **`allow_early_stop_at_turn`** keeps the conversation going until the assistant books, so a user who adds a thank-you to the number of people does not end it before the booking is made. @@ -248,8 +282,8 @@ class BookingProbe(ToolExecutionMixin, BankVerbatimMixin, BankBackedProbe): assistant under test is told, so it answers as it would for any user. - **`description`** is what the per-row preview and the capability report show about the probe. -- **`preview_fields`** puts the row's booking id in the per-row preview's Probe - Inputs panel. +- **`preview_fields`** puts the row's appointment in the per-row preview's + Probe Inputs panel. - **`default_scorers`** names the package's scorer, so `usersim eval --scorers auto` runs it whenever a run holds the probe's rows, beside `language_compliance`, which checks that the assistant replied in the @@ -263,27 +297,32 @@ class BookingProbe(ToolExecutionMixin, BankVerbatimMixin, BankBackedProbe): bank that loads stops before any model call rather than failing every row. - **`asset_problems`** reads each bank the package ships, so `usersim smoke` and `assert_probe_conforms` report an install that is missing one. -- **`placeholder_warning_kind`** is the warning a row records when its booking - is marked `placeholder: true`. Its `_used_placeholder_` form is - what the capability report counts as placeholder content. +- **`placeholder_warning_kind`** is the warning a row records when its + appointment is marked `placeholder: true`. Its + `_used_placeholder_` form is what the capability report counts + as placeholder content. - **`output_columns`** declares the columns `build_result_extras` adds, so a run keeps them. `my_booking` holds, as JSON, what the user meant and what the assistant booked. -- **`derive_task`** picks the row's booking with a generator seeded from the - persona, the bank's version and the run's seed, never the global `random`, so - a row draws the same booking every time it runs. +- **`derive_task`** picks the row's appointment, and `draw_booking` its date, + time and number of people, with a generator seeded from the persona, the + bank's version and the run's seed, never the global `random`, so a row draws + the same booking every time it runs. The banks, `assets/my_bookings/en_US/bookings.yaml`: ```yaml bank_id: my_bookings_en_US bank_version: "1.0.0" -bookings: +appointments: - id: museum_tour - opening: Hi! Can you book the guided museum tour for 04/03/2027 at 3:30 pm? - date: "2027-04-03" - time: "15:30" - people: 3 + what: the guided tour of the city museum + placeholder: true + - id: restaurant + what: a table at a restaurant + placeholder: true + - id: pottery_class + what: a pottery class placeholder: true ``` @@ -292,21 +331,28 @@ and `assets/my_bookings/pt_BR/bookings.yaml`: ```yaml bank_id: my_bookings_pt_BR bank_version: "1.0.0" -bookings: +appointments: - id: museum_tour - opening: Oi! VocΓͺ pode reservar a visita guiada ao museu para 04/03/2027 Γ s 15h30? - date: "2027-03-04" - time: "15:30" - people: 3 + what: a visita guiada ao museu da cidade + placeholder: true + - id: restaurant + what: uma mesa num restaurante + placeholder: true + - id: pottery_class + what: uma aula de cerΓ’mica placeholder: true ``` -Both open with the same date, `04/03/2027`. A user writing American English -means 3 April and a user writing Brazilian Portuguese means 4 March. An -assistant that reads every date month first books the Brazilian user's tour a -month late, and one that reads every date day first books the American user's -a month early. Neither opening says how many people are coming, so an assistant -that books on its first reply has to guess. +Both banks name the same three appointments, each in its bank's language. +Every date `draw_booking` picks reads as two dates. For `2027-03-06`, 6 March, a +user who writes the day first, as people in Brazil do, writes `06/03/2027`, +which an assistant that reads every date month first books for 3 June. A user +who writes the month first, as people in the United States do, writes +`03/06/2027`, which an assistant that reads every date day first gets wrong the +other way. Whether the user writes the date in numbers or in words is the user +model's choice, as it would be a person's. The user says how many people are +coming only when asked, so an assistant that books on its first reply has to +guess. `placeholder: true` marks content no subject-matter expert has reviewed, which for these banks means a native speaker of the bank's language; see @@ -366,11 +412,19 @@ installed. With discovery off, the tests import the package to register it: ```python import json +import random from usersim.extension import ConversationSimulatorConfig -from usersim.testing import assert_hosted_parity, assert_probe_conforms, assert_scorer_conforms, run_episode +from usersim.testing import ( + ScriptedModel, + assert_hosted_parity, + assert_probe_conforms, + assert_scorer_conforms, + run_episode, +) import my_bookings.probe # noqa: F401 registers the probe; discovery is off under test +from my_bookings.probe import Appointment, draw_booking from my_bookings.scorer import score PERSONA = {"first_name": "Ana", "last_name": "Souza", "age": 35, "city": "Recife"} @@ -385,22 +439,31 @@ async def test_the_scorer_conforms(): await assert_scorer_conforms("my_bookings") -async def test_an_episode_opens_in_portuguese_and_records_the_booking(): +async def test_the_user_asks_for_the_drawn_booking_in_their_own_words(): run = await run_episode("my_bookings", config=CONFIG, persona=PERSONA, language="Portuguese") assert run.row["conversation_status"] is True - opening = json.loads(run.row["conversation_messages"])[0] - assert opening["content"] == "Oi! VocΓͺ pode reservar a visita guiada ao museu para 04/03/2027 Γ s 15h30?" booking = json.loads(run.row["my_booking"]) - assert booking["expected"] == {"date": "2027-03-04", "time": "15:30", "people": 3} + assert booking["expected"] == {"date": "2027-03-06", "time": "11:00", "people": 6} + instructions = run.prompts[0]["messages"][0]["content"] + assert "a visita guiada ao museu da cidade on 2027-03-06 at 11:00" in instructions + opening = json.loads(run.row["conversation_messages"])[0] + assert opening == {"role": "user", "content": ScriptedModel.CONTENT["user"]} # the user model's own words assert booking["made"]["after_user_turns"] == 1 # the scripted assistant books on its first reply warnings = json.loads(run.row["simulation_outcome"])["warnings"] assert [w["kind"] for w in warnings] == ["my_used_placeholder_booking"] +def test_every_drawn_date_reads_two_ways(): + appointment = Appointment(id="museum_tour", what="the guided tour of the city museum", placeholder=True) + for seed in range(200): + _, month, day = draw_booking(appointment, random.Random(seed)).date.split("-") + assert month != day and int(month) <= 12 and int(day) <= 12 + + async def test_a_month_first_reading_of_a_day_first_date_is_wrong(): - made = {"arguments": {"date": "2027-04-03", "time": "15:30", "people": 3}, "after_user_turns": 2} - expected = {"date": "2027-03-04", "time": "15:30", "people": 3} + made = {"arguments": {"date": "2027-06-03", "time": "11:00", "people": 6}, "after_user_turns": 2} + expected = {"date": "2027-03-06", "time": "11:00", "people": 6} result = await score({"my_booking": json.dumps({"expected": expected, "made": made})}, models={}) assert result == { @@ -422,9 +485,12 @@ async def test_a_host_reproduces_a_standalone_run(): empty fields. - **`run_episode`** runs a whole episode with scripted models, through the same generator as `usersim simulate`, with nothing patched and no provider. Here it - runs in `pt_BR`: the opening reaches the assistant as written, and the - scripted assistant, which calls a tool as soon as one is offered, books on its - first reply. + runs in `pt_BR`: the user model's instructions, the first prompt the episode + sends, hold the drawn booking, and the opening is what the user model + replied, the scripted user's stock line. The scripted assistant, which calls a + tool as soon as one is offered, books on its first reply. +- **`draw_booking`** is tested on its own over many seeds, since one episode + draws one date. - **The scorer** is tested on a row built by hand, in which the assistant read the Brazilian date month first. - **`assert_hosted_parity`** checks that a host driving the probe through diff --git a/templates/probe/README.md b/templates/probe/README.md index 942e7e9..57a3432 100644 --- a/templates/probe/README.md +++ b/templates/probe/README.md @@ -23,6 +23,13 @@ do we need to surface?* Each probe owns: `target_request_id`, `action_request_id`, …) that the matching out-of-sim scorer reads. +A probe sets the area a conversation explores, a topic or a task and the +facts a scorer checks, and the simulated user explores it in the persona's own +words. Hand the user model the area, not a message to send: a fixed opening +starts every row in a locale the same way. Send a bank's text verbatim only +when its exact wording is what the probe tests, such as a false premise; +Step 0 of the authoring tutorial says which shape fits. + If the new "probe" is just a different prompt for an existing probe, prefer a `probe_variant` over a new module. diff --git a/templates/probe/generator_baseprobe_only.py b/templates/probe/generator_baseprobe_only.py index 9490061..226f7f7 100644 --- a/templates/probe/generator_baseprobe_only.py +++ b/templates/probe/generator_baseprobe_only.py @@ -3,9 +3,12 @@ """Probe template β€” BaseProbe only, no asset bank. -Use this shape when the probe asks general questions of the assistant -without needing reviewer-curated content (the ``general_open_ended`` -and ``general_educational`` probes use this shape). +Use this shape when the probe sets topics for the simulated user to +explore, without needing reviewer-curated content (the +``general_open_ended`` and ``general_educational`` probes use this +shape). Each row draws one topic from ``themes(locale)``, and the user +explores it in the persona's own words: the probe sets the area, never +the message. What the unified ``ConversationLoop`` provides for you: - Multi-turn back-and-forth with the user-LLM. @@ -15,7 +18,10 @@ - Frustration escalation, early-stop heuristics. What you provide: -- The user-agent system prompt (how the simulated user behaves). +- ``themes(locale)``: the topics a run draws from, one per row, each a + mapping with at least a ``description``. +- The user-agent system prompt (how the simulated user behaves, and the + row's topic). - The assistant system prompt (empty per pure-capability-test policy; see docs/engine/README.md "Assistant under test"). @@ -25,6 +31,10 @@ from __future__ import annotations +from pathlib import Path + +import yaml + # After you copy this file into ``src/usersim/engine/probes//``, # also copy ``prompts.py`` next to it and import from # ``usersim.engine.probes..prompts``. @@ -44,12 +54,21 @@ class DemoGeneralProbe(BaseProbe): """Smallest viable probe β€” no asset bank, default everything else.""" label = "demo_general" + assets_root = Path(__file__).parent / "assets" # holds demo_general/, beside this module + + @classmethod + def themes(cls, locale: str) -> list[dict]: + # One file of topics per locale, written in its language. A run + # draws one per row and stores it in the row's ``theme`` column. + return yaml.safe_load((cls.assets_dir() / locale / "themes.yaml").read_text(encoding="utf-8")) def get_user_system_prompt(self) -> str: - return DEMO_USER_AGENT_PROMPTS.get( + prompt = DEMO_USER_AGENT_PROMPTS.get( self._locale, persona=self._persona.get("first_name", "the user"), ) + # The topic bounds the conversation; the persona picks the question. + return f"{prompt}\n\nTopic: {self.theme()['description']}" def get_assistant_system_prompt(self) -> str: # Pure-capability-test policy. Empty system prompt; diff --git a/tests/test_extension_api.py b/tests/test_extension_api.py index f4d700a..54c8d6d 100644 --- a/tests/test_extension_api.py +++ b/tests/test_extension_api.py @@ -56,6 +56,7 @@ ("usersim.engine.probes.YOUR_PROBE", "probe"), ("usersim.engine.probes.bookings.bank", "LOCALES"), ("usersim.engine.probes.bookings.bank", "Booking"), + ("usersim.engine.probes.bookings.bank", "draw_booking"), ("usersim.engine.probes.bookings.bank", "read_booking_bank"), ("usersim.engine.probes.bookings.prompts", "BOOKINGS_USER_PROMPTS"), ("usersim.engine.probes.my_probe.bank", "load_my_bank_default"), From b3c499c10828487bbb864eac51e7d208ca990c53 Mon Sep 17 00:00:00 2001 From: Yev Meyer Date: Mon, 2 Nov 2026 18:25:50 -0500 Subject: [PATCH 4/4] docs: add an extension package template that check-extensions tests templates/extension_package/ is a package to copy and rename: a probe that draws an area per row from themes.yaml and has the user explore it, its prompts in prompts.yaml read with get_prompt, a tool, a scorer, entry points, and tests wired to the pytest plugin. make check-extensions installs it next to the built wheel and runs its tests, and the public-import test scans it. docs/extensions.md and CONTRIBUTING.md point to it. Signed-off-by: Yev Meyer --- CONTRIBUTING.md | 4 +- Makefile | 9 +- docs/extensions.md | 4 + templates/extension_package/README.md | 70 +++++++++++++ templates/extension_package/pyproject.toml | 27 ++++++ .../src/my_probe/__init__.py | 4 + .../assets/my_probe/en_US/themes.yaml | 7 ++ .../src/my_probe/assets/my_probe/prompts.yaml | 7 ++ .../assets/my_probe/pt_BR/themes.yaml | 7 ++ .../extension_package/src/my_probe/probe.py | 97 +++++++++++++++++++ .../extension_package/src/my_probe/scorer.py | 16 +++ templates/extension_package/tests/conftest.py | 4 + .../extension_package/tests/test_my_probe.py | 57 +++++++++++ tests/test_extension_api.py | 1 + 14 files changed, 311 insertions(+), 3 deletions(-) create mode 100644 templates/extension_package/README.md create mode 100644 templates/extension_package/pyproject.toml create mode 100644 templates/extension_package/src/my_probe/__init__.py create mode 100644 templates/extension_package/src/my_probe/assets/my_probe/en_US/themes.yaml create mode 100644 templates/extension_package/src/my_probe/assets/my_probe/prompts.yaml create mode 100644 templates/extension_package/src/my_probe/assets/my_probe/pt_BR/themes.yaml create mode 100644 templates/extension_package/src/my_probe/probe.py create mode 100644 templates/extension_package/src/my_probe/scorer.py create mode 100644 templates/extension_package/tests/conftest.py create mode 100644 templates/extension_package/tests/test_my_probe.py diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b5198f1..651c349 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -352,7 +352,9 @@ your package adds with its own prefix, such as `my_bookings`; unprefixed names belong to the built-ins, and [`architecture/plugins.md`](architecture/plugins.md) lists the names this covers. [`docs/extensions.md`](docs/extensions.md) builds a whole extension -package, a probe with its bank, a scorer and their tests, step by step. Import +package, a probe with its bank, a scorer and their tests, step by step, and +[`templates/extension_package/`](templates/extension_package/) is one to copy +and fill in. Import from `usersim.extension` and `usersim.testing`; [`docs/extension_api.md`](docs/extension_api.md) lists every name they export. diff --git a/Makefile b/Makefile index fa11529..95974e9 100644 --- a/Makefile +++ b/Makefile @@ -268,7 +268,7 @@ health-check: ## Live one-row simulate + eval + report against a provider || { echo "report produced no index.html"; exit 1; }; \ echo "βœ“ simulate, eval and report all completed" -check-extensions: ## Install an out-of-tree package and prove all six seams work +check-extensions: ## Install an out-of-tree package, prove all six seams work, and test the package template @set -eu; \ WORK=$$(mktemp -d); \ EMPTY=$$(mktemp -d); \ @@ -280,10 +280,15 @@ check-extensions: ## Install an out-of-tree package and prove all six seams wor test -x "$$WORK/venv/bin/python" || { echo "venv was not created"; exit 1; }; \ $(UV) pip install --quiet --python "$$WORK/venv/bin/python" "$$WORK"/*.whl; \ cp -R tests/fixtures/extension_package "$$WORK/ext"; \ + cp -R templates/extension_package "$$WORK/template"; \ $(UV) pip install --quiet --python "$$WORK/venv/bin/python" "$$WORK/ext"; \ echo "β†’ asking the installed CLI what it can see, from $$EMPTY"; \ cp tests/fixtures/extension_package/verify_seams.py "$$EMPTY/"; \ - cd "$$EMPTY" && "$$WORK/venv/bin/python" verify_seams.py + cd "$$EMPTY" && "$$WORK/venv/bin/python" verify_seams.py; \ + echo "β†’ installing templates/extension_package and running its tests"; \ + $(UV) pip install --quiet --python "$$WORK/venv/bin/python" "$$WORK/template" \ + "pytest>=8.0" "pytest-asyncio>=1.0,<2" "pytest-env>=1.1,<2"; \ + cd "$$WORK/template" && "$$WORK/venv/bin/python" -m pytest -q -p no:cacheprovider # --- housekeeping ---------------------------------------------------------- diff --git a/docs/extensions.md b/docs/extensions.md index 3a4005d..ba249b9 100644 --- a/docs/extensions.md +++ b/docs/extensions.md @@ -35,6 +35,10 @@ my-bookings/ The banks sit under `assets/