diff --git a/.github/ISSUE_TEMPLATE/extension-hook.yml b/.github/ISSUE_TEMPLATE/extension-hook.yml new file mode 100644 index 0000000..da5b723 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/extension-hook.yml @@ -0,0 +1,84 @@ +name: πŸ”Œ Extension hook +description: Ask for a hook your probe, scorer or other extension needs and User Sim does not offer +labels: ["enhancement"] +body: + - type: markdown + attributes: + value: >- + Use this form when an extension needs something no hook, class + attribute or entry point offers. A maintainer reviews the request, + agrees the hook's shape with you and adds the `triaged` label when a + pull request is wanted. + [Adding a hook](https://github.com/NVIDIA-NeMo/UserSim/blob/main/CONTRIBUTING.md#adding-a-hook) + lists what a new hook must do. + + - type: textarea + id: need + attributes: + label: What does your extension need to do? + description: >- + The need in general terms: what the extension has to read or change, + and at which stage of a run (setup, the conversation, the stored + columns, scoring or the report). The general shape is enough to design + a hook, so leave out anything confidential about your project. + validations: + required: true + + - type: textarea + id: gap + attributes: + label: Why don't the existing hooks cover it? + description: >- + The hooks, class attributes and entry points you looked at, in + [architecture/probes.md](https://github.com/NVIDIA-NeMo/UserSim/blob/main/architecture/probes.md) + and [docs/extension_api.md](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/extension_api.md), + and where each falls short. + validations: + required: true + + - type: textarea + id: shape + attributes: + label: What shape do you propose? + description: >- + A method, class attribute or entry point, its signature, and what + happens for a probe that does not define it. That default should keep + today's behaviour. + validations: + required: true + + - type: textarea + id: hosted + attributes: + label: Does it work when a host drives the probe? + description: >- + A host can run a probe from its own loop through `ProbeEpisodeRuntime`, + as + [EXTERNAL_PROBE_RUNTIME.md](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/engine/EXTERNAL_PROBE_RUNTIME.md) + describes. Say whether the hook runs there too, and whether it calls a + model. + validations: + required: true + + - type: textarea + id: investigation + attributes: + label: Prior investigation + description: >- + What you or your coding agent found in the code, documentation and + existing issues, such as a hook that covers part of this or an open + request for the same thing. If an agent investigated, paste its + findings. + placeholder: Paste the agent's findings, or describe what you checked. + + - type: checkboxes + id: checklist + attributes: + label: Checklist + options: + - label: I searched existing issues and the documentation, or had my agent do so + required: true + - label: The request describes the need in general terms and holds nothing confidential + required: true + - label: If an agent helped, its findings are under Prior investigation + required: false diff --git a/.github/ISSUE_TEMPLATE/feature-request.yml b/.github/ISSUE_TEMPLATE/feature-request.yml index e3fd988..13caf17 100644 --- a/.github/ISSUE_TEMPLATE/feature-request.yml +++ b/.github/ISSUE_TEMPLATE/feature-request.yml @@ -9,7 +9,8 @@ body: change is wanted. If you plan to build this yourself, open the pull request once the issue has that label; see [Issues before pull requests](https://github.com/NVIDIA-NeMo/UserSim/blob/main/CONTRIBUTING.md#issues-before-pull-requests). - To propose a probe you plan to build, use the New probe form instead. + To propose a probe you plan to build, use the New probe form instead, + and to ask for a hook your extension needs, the Extension hook form. - type: textarea id: problem diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8d69673..651c349 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -352,7 +352,9 @@ your package adds with its own prefix, such as `my_bookings`; unprefixed names belong to the built-ins, and [`architecture/plugins.md`](architecture/plugins.md) lists the names this covers. [`docs/extensions.md`](docs/extensions.md) builds a whole extension -package, a probe with its bank, a scorer and their tests, step by step. Import +package, a probe with its bank, a scorer and their tests, step by step, and +[`templates/extension_package/`](templates/extension_package/) is one to copy +and fill in. Import from `usersim.extension` and `usersim.testing`; [`docs/extension_api.md`](docs/extension_api.md) lists every name they export. @@ -445,6 +447,31 @@ To propose a probe, open the Triage settles where it lives: in this repository, as an extension, or as an extension first. +### Adding a hook + +When a probe, scorer or other extension needs something no hook, class +attribute or entry point offers, open the +[Extension hook form](https://github.com/NVIDIA-NeMo/UserSim/issues/new?template=extension-hook.yml). +[`architecture/probes.md`](architecture/probes.md) lists the hooks and +declarations a probe has, and [`docs/extension_api.md`](docs/extension_api.md) +the names an extension imports. + +A new hook: + +- **is general:** it serves probes beyond the one that asks for it, and its + name and documentation describe that general use. +- **keeps today's behaviour by default:** a probe that does not define it runs, + and stores its rows, as before. +- **is `async def` if it can call a model,** so the loop awaits it. +- **works when a host drives the probe:** `assert_hosted_parity` passes for a + probe that uses it. +- **comes with its uses:** the extension fixture in + `tests/fixtures/extension_package/` uses it, so `make check-extensions` + proves it works from an installed package. The pull request that adds it + also updates the conformance checks, `templates/probe/`, + [`architecture/probes.md`](architecture/probes.md) and + [`docs/engine/AUTHORING_A_PROBE.md`](docs/engine/AUTHORING_A_PROBE.md). + ## Code style `ruff` owns formatting and linting; there is no separate style config to diff --git a/Makefile b/Makefile index fa11529..95974e9 100644 --- a/Makefile +++ b/Makefile @@ -268,7 +268,7 @@ health-check: ## Live one-row simulate + eval + report against a provider || { echo "report produced no index.html"; exit 1; }; \ echo "βœ“ simulate, eval and report all completed" -check-extensions: ## Install an out-of-tree package and prove all six seams work +check-extensions: ## Install an out-of-tree package, prove all six seams work, and test the package template @set -eu; \ WORK=$$(mktemp -d); \ EMPTY=$$(mktemp -d); \ @@ -280,10 +280,15 @@ check-extensions: ## Install an out-of-tree package and prove all six seams wor test -x "$$WORK/venv/bin/python" || { echo "venv was not created"; exit 1; }; \ $(UV) pip install --quiet --python "$$WORK/venv/bin/python" "$$WORK"/*.whl; \ cp -R tests/fixtures/extension_package "$$WORK/ext"; \ + cp -R templates/extension_package "$$WORK/template"; \ $(UV) pip install --quiet --python "$$WORK/venv/bin/python" "$$WORK/ext"; \ echo "β†’ asking the installed CLI what it can see, from $$EMPTY"; \ cp tests/fixtures/extension_package/verify_seams.py "$$EMPTY/"; \ - cd "$$EMPTY" && "$$WORK/venv/bin/python" verify_seams.py + cd "$$EMPTY" && "$$WORK/venv/bin/python" verify_seams.py; \ + echo "β†’ installing templates/extension_package and running its tests"; \ + $(UV) pip install --quiet --python "$$WORK/venv/bin/python" "$$WORK/template" \ + "pytest>=8.0" "pytest-asyncio>=1.0,<2" "pytest-env>=1.1,<2"; \ + cd "$$WORK/template" && "$$WORK/venv/bin/python" -m pytest -q -p no:cacheprovider # --- housekeeping ---------------------------------------------------------- diff --git a/docs/engine/AUTHORING_A_PROBE.md b/docs/engine/AUTHORING_A_PROBE.md index 85ebca6..a6767b9 100644 --- a/docs/engine/AUTHORING_A_PROBE.md +++ b/docs/engine/AUTHORING_A_PROBE.md @@ -33,46 +33,56 @@ piece to pick and *how* to wire it. ## Step 0: pick your shape +A probe sets the area its conversations explore and the bounds of that area: +the topics or tasks, and the facts a scorer checks. The simulated user, played +from a persona, explores the area in its own words, so rows on the same topic +open differently and go different ways, and a run covers what its personas +bring to it. Give the user model an area and its bounds rather than a message +to send: a fixed opening starts every row in a locale the same way, and the run +explores only what that one message reaches. Send a bank's text verbatim only +when its exact wording is what the probe tests. + Use this decision tree to pick which substrate combination matches your probe: ``` -Does the probe have an asset bank (curated facts / queries / strategies / actions)? +Who writes the user's first message? β”‚ -β”œβ”€β”€ NO β†’ BaseProbe alone -β”‚ Examples: general_open_ended, general_educational -β”‚ Template: templates/probe/generator_baseprobe_only.py +β”œβ”€β”€ The user model, in the persona's voice, inside an area the probe sets +β”‚ (start here: the persona's own wording IS the feature) +β”‚ β”‚ +β”‚ β”œβ”€β”€ The areas are topics, listed per locale +β”‚ β”‚ β†’ BaseProbe, with a themes(locale) classmethod +β”‚ β”‚ (see "Give each row a theme" in Step 3) +β”‚ β”‚ Examples: general_open_ended, general_educational +β”‚ β”‚ Template: templates/probe/generator_baseprobe_only.py +β”‚ β”‚ +β”‚ └── The areas come from a curated bank, with facts a scorer checks +β”‚ β†’ BankBackedProbe, with the area and its facts in the user system prompt +β”‚ Examples: sov_ai_dynamic, the bookings worked example below +β”‚ Template: templates/probe/generator_custom_turn1.py +β”‚ β†’ or CustomTurn1InstructionMixin, when only turn 1 follows +β”‚ an instruction of its own β”‚ -└── YES β†’ BankBackedProbe + mixin (depends on turn-1 shape) - β”‚ - β”œβ”€β”€ Turn-1 should be the bank's text VERBATIM - β”‚ (false-premise wording, exact phrasing for matched-pair joins, - β”‚ or reviewer-authored content must survive unchanged) - β”‚ β”‚ - β”‚ β”œβ”€β”€ Single back-and-forth conversation - β”‚ β”‚ β†’ BankVerbatimMixin - β”‚ β”‚ Examples: sov_ai_facts, sov_ai_multilingual_parity - β”‚ β”‚ Template: templates/probe/generator_bank_verbatim.py - β”‚ β”‚ - β”‚ β”œβ”€β”€ Multi-turn with bank-driven per-turn nudges (reframings) - β”‚ β”‚ β†’ BankReframingMixin - β”‚ β”‚ Example: safety_chat_pressure - β”‚ β”‚ Template: templates/probe/generator_bank_reframing.py - β”‚ β”‚ - β”‚ └── Single user turn + multi-assistant-turn tool loop - β”‚ (NO user-LLM at all; assistant tool-calls intercepted) - β”‚ β†’ AgenticMixin (custom run_dispatch) - β”‚ Example: safety_agentic - β”‚ Template: templates/probe/generator_agentic.py - β”‚ - └── Turn-1 should be GENERATED by the user-LLM in the persona's - voice, but ANCHORED to a bank-derived hint - (paraphrasing through persona voice IS the feature) - β†’ BankBackedProbe, with the hint in the user system prompt - Example: sov_ai_dynamic - Template: templates/probe/generator_custom_turn1.py - β†’ or CustomTurn1InstructionMixin, when only turn 1 follows - an instruction of its own +└── A bank's text, VERBATIM, because its exact wording is what the probe tests + (false-premise wording, exact phrasing for matched-pair joins, + or reviewer-authored content that must survive unchanged) + β”‚ + β”œβ”€β”€ Single back-and-forth conversation + β”‚ β†’ BankVerbatimMixin + β”‚ Examples: sov_ai_facts, sov_ai_multilingual_parity + β”‚ Template: templates/probe/generator_bank_verbatim.py + β”‚ + β”œβ”€β”€ Multi-turn with bank-driven per-turn nudges (reframings) + β”‚ β†’ BankReframingMixin + β”‚ Example: safety_chat_pressure + β”‚ Template: templates/probe/generator_bank_reframing.py + β”‚ + └── Single user turn + multi-assistant-turn tool loop + (NO user-LLM at all; assistant tool-calls intercepted) + β†’ AgenticMixin (custom run_dispatch) + Example: safety_agentic + Template: templates/probe/generator_agentic.py ``` There's also a **tool-calling capability** shape (test the assistant's @@ -174,7 +184,7 @@ notifying. See `docs/engine/EXTERNAL_PROBE_RUNTIME.md`. For `BankBackedProbe` subclasses: ```python -class MyProbe(BankVerbatimMixin, BankBackedProbe): +class MyProbe(BankBackedProbe): label = "my_probe" # required bank_loader = staticmethod(_my_bank_loader) # required placeholder_warning_kind = WarningKind.USED_PLACEHOLDER_FACT # required @@ -321,7 +331,7 @@ def _bank_loader(locale: str = "en_US"): return _load_bank(locale) -class MyProbe(BankVerbatimMixin, BankBackedProbe): +class MyProbe(BankBackedProbe): bank_loader = staticmethod(_bank_loader) ``` @@ -364,7 +374,7 @@ def _check_locale_bank(locale_dir: Path) -> None: MyBank.from_yaml((locale_dir / "sample.yaml").read_text(encoding="utf-8")) -class MyProbe(BankVerbatimMixin, BankBackedProbe): +class MyProbe(BankBackedProbe): locale_bank_check = staticmethod(_check_locale_bank) authoring_guide = "https://example.com/my-probe/README.md#adding-a-locale" ``` @@ -388,7 +398,7 @@ bank fails a check rather than a run. Return one string per problem, or an empty list, and read what your loaders read without calling a model: ```python -class MyProbe(BankVerbatimMixin, BankBackedProbe): +class MyProbe(BankBackedProbe): @classmethod def asset_problems(cls) -> list[str]: problems = [] @@ -520,7 +530,7 @@ from usersim.extension import register_probe prompt_version="v1.0", # MyProbe.prompt_version variants=("default",), # MyProbe.variants ) -class MyProbe(BankVerbatimMixin, BankBackedProbe): +class MyProbe(BankBackedProbe): label = "my_probe" ... ``` @@ -631,8 +641,10 @@ the assistant has a `book_appointment` tool that takes them in one fixed form. The probe ships banks for `en_US` and `pt_BR`; the same probe, shipped in its own package, is the example in [`docs/extensions.md`](../extensions.md). -We'll use the **banked-verbatim** shape, so each date reaches the assistant -exactly as the bank writes it, with `ToolExecutionMixin` to answer the +The probe sets the area and its bounds, and the persona does the rest: the bank +names what can be booked, the probe draws a date, a time and a number of people +for each row, and the user model asks for the booking in its own words. It is a +`BankBackedProbe` with no verbatim turn, and `ToolExecutionMixin` answers the assistant's tool call. ### File 1: the banks, `src/usersim/engine/probes/bookings/assets/bookings//bookings.yaml` @@ -642,12 +654,15 @@ assistant's tool call. ```yaml bank_id: bookings_en_US bank_version: "1.0.0" -bookings: +appointments: - id: museum_tour - opening: Hi! Can you book the guided museum tour for 04/03/2027 at 3:30 pm? - date: "2027-04-03" - time: "15:30" - people: 3 + what: the guided tour of the city museum + placeholder: true + - id: restaurant + what: a table at a restaurant + placeholder: true + - id: pottery_class + what: a pottery class placeholder: true ``` @@ -656,26 +671,37 @@ bookings: ```yaml bank_id: bookings_pt_BR bank_version: "1.0.0" -bookings: +appointments: - id: museum_tour - opening: Oi! VocΓͺ pode reservar a visita guiada ao museu para 04/03/2027 Γ s 15h30? - date: "2027-03-04" - time: "15:30" - people: 3 + what: a visita guiada ao museu da cidade + placeholder: true + - id: restaurant + what: uma mesa num restaurante + placeholder: true + - id: pottery_class + what: uma aula de cerΓ’mica placeholder: true ``` -Both open with `04/03/2027`: 3 April to a user writing American English, and -4 March to one writing Brazilian Portuguese. Neither says how many people are -coming, so an assistant that books on its first reply has to guess. +Both name the same three appointments, each in its bank's language. The +banks hold no dates: `draw_booking` in File 2 draws one per row, with the day +and the month different numbers no greater than 12, so the date reads as another +date when its numbers are read in the other order. For `2027-03-06`, 6 March, a +user who writes the day first, as people in Brazil do, writes `06/03/2027`, and +one who writes the month first, as people in the United States do, writes +`03/06/2027`; an assistant that reads every date one way gets one of them +wrong. Whether the user writes the date in numbers or in words is the user +model's choice. The user says how many people are coming only when asked, so an +assistant that books on its first reply has to guess. ### File 2: `src/usersim/engine/probes/bookings/bank.py` ```python -"""Booking banks: one per locale, under the probe's asset folder.""" +"""Appointment banks: one per locale, under the probe's asset folder.""" from __future__ import annotations +import random from dataclasses import dataclass from pathlib import Path @@ -686,12 +712,9 @@ LOCALES = ("en_US", "pt_BR") @dataclass(frozen=True) -class Booking: +class Appointment: id: str - opening: str # turn 1, in the locale's language; leaves out how many people - date: str # what the user means, as the tool takes it: YYYY-MM-DD - time: str # HH:MM, 24-hour - people: int # given only when the assistant asks + what: str # what the user wants booked, in the locale's language placeholder: bool @@ -699,12 +722,40 @@ class Booking: class BookingBank: bank_id: str bank_version: str - bookings: tuple[Booking, ...] + appointments: tuple[Appointment, ...] + + +@dataclass(frozen=True) +class Booking: + id: str + what: str + date: str # what the user means, as the tool takes it: YYYY-MM-DD + time: str # HH:MM, 24-hour + people: int # given only when the assistant asks + placeholder: bool def read_booking_bank(locale_dir: Path) -> BookingBank: doc = yaml.safe_load((locale_dir / "bookings.yaml").read_text(encoding="utf-8")) - return BookingBank(doc["bank_id"], doc["bank_version"], tuple(Booking(**b) for b in doc["bookings"])) + return BookingBank(doc["bank_id"], doc["bank_version"], tuple(Appointment(**a) for a in doc["appointments"])) + + +def draw_booking(appointment: Appointment, rng: random.Random) -> Booking: + """A date, a time and a number of people for ``appointment``. + + The day and the month are different numbers no greater than 12, so the date + reads as two different dates, day first and month first. + """ + month = rng.randint(1, 12) + day = rng.choice([d for d in range(1, 13) if d != month]) + return Booking( + id=appointment.id, + what=appointment.what, + date=f"2027-{month:02d}-{day:02d}", + time=f"{rng.randint(9, 17):02d}:{rng.choice(('00', '30'))}", + people=rng.randint(2, 6), + placeholder=appointment.placeholder, + ) ``` ### File 3: `src/usersim/engine/probes/bookings/__init__.py` @@ -726,14 +777,16 @@ BOOKINGS_USER_PROMPTS = LocalePromptPack( prompts={ "en_US": ( "{persona}\n\n" - "You asked an assistant to book an appointment, giving the date and time. " - "If it asks how many people are coming, the answer is {people}; say it the way you would. " + "You want an assistant to book {what} on {date} at {time}. Ask for it in your own words, " + "and write the date and time the way you would write them yourself, not in the form they are " + "given here. Say how many people are coming only if it asks; the answer is {people}. " "Once it confirms the booking, thank it." ), "pt_BR": ( "{persona}\n\n" - "VocΓͺ pediu a um assistente para marcar um horΓ‘rio e jΓ‘ disse a data e a hora. " - "Se ele perguntar quantas pessoas vΓ£o, a resposta Γ© {people}; diga isso do seu jeito. " + "VocΓͺ quer que um assistente reserve {what} para {date} Γ s {time}. PeΓ§a do seu jeito, e " + "escreva a data e a hora como vocΓͺ mesmo escreveria, nΓ£o na forma em que aparecem aqui. " + "Diga quantas pessoas vΓ£o sΓ³ se ele perguntar; a resposta Γ© {people}. " "Quando ele confirmar a reserva, agradeΓ§a." ), }, @@ -743,6 +796,8 @@ BOOKINGS_USER_PROMPTS = LocalePromptPack( The user's instructions are written in each language the probe ships, and `LocalePromptPack` refuses, when the module is imported, a locale without them. +They give the user model what to book, with the date and time in the tool's +form, and ask it to write the date and time the way it would itself. ### File 5: `src/usersim/engine/probes/bookings/probe.py` @@ -757,11 +812,10 @@ from pathlib import Path from typing import Any from usersim.engine.core.probes import dispatch_episode -from usersim.engine.probes.bookings.bank import LOCALES, Booking, read_booking_bank +from usersim.engine.probes.bookings.bank import LOCALES, Booking, draw_booking, read_booking_bank from usersim.engine.probes.bookings.prompts import BOOKINGS_USER_PROMPTS from usersim.extension import ( BankBackedProbe, - BankVerbatimMixin, OutcomeBuilder, Provenance, ToolExecutionMixin, @@ -804,7 +858,7 @@ def _bank_loader(locale: str): prompt_version=PROMPT_VERSION, variants=tuple(PROBE_VARIANTS), ) -class BookingsProbe(ToolExecutionMixin, BankVerbatimMixin, BankBackedProbe): +class BookingsProbe(ToolExecutionMixin, BankBackedProbe): label = "bookings" assets_root = Path(__file__).parent / "assets" # holds bookings/ description = ( @@ -814,7 +868,6 @@ class BookingsProbe(ToolExecutionMixin, BankVerbatimMixin, BankBackedProbe): bank_loader = staticmethod(_bank_loader) locale_bank_check = staticmethod(read_booking_bank) placeholder_warning_kind = WarningKind.USED_PLACEHOLDER_BOOKING - verbatim_field = "opening" output_columns = ("booking_id", "booking") @classmethod @@ -829,12 +882,15 @@ class BookingsProbe(ToolExecutionMixin, BankVerbatimMixin, BankBackedProbe): def derive_task(self, persona: dict, bank: Any, *, cfg: Any) -> Booking: rng = random.Random(f"{persona_uuid(persona)}:{bank.bank_version}:{cfg.random_seed}") - return rng.choice(bank.bookings) + return draw_booking(rng.choice(bank.appointments), rng) def get_user_system_prompt(self) -> str: return BOOKINGS_USER_PROMPTS.get( self._locale, persona=format_persona_for_prompt(self._persona), + what=self._task.what, + date=self._task.date, + time=self._task.time, people=self._task.people, ) @@ -909,7 +965,12 @@ async def simulate_bookings( assistant has booked, so a user who thanks it early does not end it first. - `should_succeed` keeps its default: an assistant that never books has not broken the probe's protocol, so its row is scored like any other. -- The probe keeps `BaseProbe`'s gate prompt (Step 5). +- `derive_task` picks the row's appointment, and `draw_booking` its date, time + and number of people, with a generator seeded from the persona, the bank's + version and the run's seed, so a row draws the same booking every time it + runs. +- The user model writes every user turn, the first included, and the probe + keeps `BaseProbe`'s gate prompt (Step 5) to check each one. ### File 6: the placeholder warning diff --git a/docs/extension_api.md b/docs/extension_api.md index c1a1ae8..385407e 100644 --- a/docs/extension_api.md +++ b/docs/extension_api.md @@ -85,6 +85,8 @@ does not load the conversation loop. | `format_conversation_history_for_prompt` | A conversation as a text block for a judge prompt | | `format_tools_for_prompt` | Tool definitions as a readable prompt block | | `render_prompt` | Fills a template's placeholders from given values, then from the row | +| `load_prompt_file` | A probe's `prompts.yaml` as a mapping, read from the folder its banks resolve to; empty when there is none | +| `get_prompt` | One prompt from a probe's `prompts.yaml`, or the default it is given when the file or the key is missing | ### Locales and languages diff --git a/docs/extensions.md b/docs/extensions.md index 886ece4..ba249b9 100644 --- a/docs/extensions.md +++ b/docs/extensions.md @@ -11,7 +11,9 @@ the booking the assistant makes. point groups and how discovery behaves, [`docs/engine/AUTHORING_A_PROBE.md`](engine/AUTHORING_A_PROBE.md) covers every hook a probe can implement, and [`extension_api.md`](extension_api.md) lists -every name the package can import from `usersim`. +every name the package can import from `usersim`. When a package needs a hook +User Sim does not have, +[Adding a hook](../CONTRIBUTING.md#adding-a-hook) says how to ask for one. ## Layout @@ -33,6 +35,10 @@ my-bookings/ The banks sit under `assets/