diff --git a/README.md b/README.md index 59eda9b..4d87179 100644 --- a/README.md +++ b/README.md @@ -294,6 +294,8 @@ full plan and exits before the first network call. **Extending it** +- [`docs/extensions.md`](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/extensions.md): a probe and a scorer in a package of your own, installed next to `usersim` +- [`docs/extension_api.md`](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/extension_api.md): every name an extension imports from `usersim` - [`docs/engine/AUTHORING_A_PROBE.md`](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/engine/AUTHORING_A_PROBE.md): end-to-end tutorial for a new probe - [`docs/engine/README.md`](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/engine/README.md): the substrate's full API surface - [`templates/probe/`](https://github.com/NVIDIA-NeMo/UserSim/tree/main/templates/probe/): copy-and-edit starters, one per probe shape diff --git a/architecture/probes.md b/architecture/probes.md index 07a78c2..49dd211 100644 --- a/architecture/probes.md +++ b/architecture/probes.md @@ -135,6 +135,7 @@ than into a branch on its label. | `authoring_guide` | `None` | the pre-flight, when a run asks for a locale the probe has no bank for | | `asset_problems` | no problems | `usersim smoke` and `assert_probe_conforms` | | `themes` | `None`: no theme | run setup, which draws one per row into the `theme` column | +| `seed_dataset` | `None`: no seed table | run setup, which attaches `.parquet` or `.jsonl` from the probe's asset folder and gives each row the columns of one of its rows; a mix with two such probes is refused; smoke and `assert_probe_conforms` check the table is there | | `model_aliases` | `()` | run setup, which adds each alias the models file defines to every row's models | | `output_columns` | `()` | the run's `side_effect_columns`; a column left out is dropped | | `supports_assistant_resampling` | `True`; `False` with the tool mixins | judge-gated assistant resampling (`max_assistant_attempts > 1`) | @@ -186,5 +187,5 @@ finds. See [assets.md](assets.md). A probe whose conversations vary by topic lists its themes in a `themes` classmethod, read under its asset folder. A run draws one per row into the `theme` column, and the probe reads it back with `theme()`. A probe that lists -none gets a placeholder it never reads. `tool_calling` takes its themes and -toolsets from the run's settings instead. +none gets a placeholder it never reads. `tool_calling` lists none: a row's +theme is the toolset it draws from its seed table. diff --git a/docs/engine/AUTHORING_A_PROBE.md b/docs/engine/AUTHORING_A_PROBE.md index c4d91b8..499ba75 100644 --- a/docs/engine/AUTHORING_A_PROBE.md +++ b/docs/engine/AUTHORING_A_PROBE.md @@ -1029,8 +1029,8 @@ transcript positions. ### File 8: tests at `tests/probes/bookings/test_probe.py` Copy [`templates/probe/test_probe.py.template`](../../templates/probe/test_probe.py.template) -there, beside an `__init__.py`, substitute `bookings` for `YOUR_PROBE`, fill in the 8 standard -test stubs. +there, beside an `__init__.py`, substitute `bookings` for `YOUR_PROBE`, and fill in the +marked parts: the probe's class, a column it writes and its placeholder warning. ### Run it @@ -1094,10 +1094,10 @@ lists every request the models received, in order, with its role and messages. To script other replies, subclass `ScriptedModel`, override `CONTENT`, and pass the class as `model_factory`. -A test that controls each reply turn by turn can patch `acall_llm` instead. Two -modules import it, `core.simulation` and `core.judges`, so patch both, as the -template's `_patched_call_llm` does. Those are internal modules, so a probe -shipped in its own package tests with `run_episode`. +Some built-in probes' tests patch `acall_llm` in `core.simulation` and +`core.judges`, the two modules that call it, to control each reply turn by +turn. Those are internal modules, so a new probe's tests script replies with a +`ScriptedModel` subclass instead, in tree or in its own package. ### 3. Judge replies carry an XML verdict @@ -1198,44 +1198,31 @@ def test_my_probe_conforms(): ## Test authoring [`templates/probe/test_probe.py.template`](../../templates/probe/test_probe.py.template) -is the canonical scaffold. Copy it and substitute your probe's name. Its -first test runs the probe end to end with `run_episode` (gotcha 2); then fill -in the 8 standard test stubs: - -1. **Registry registration**: the probe label is in `_PROBE_REGISTRY` and - resolves to a class whose `family`, `prompt_version` and `variants` are - set. -2. **Single-turn end-to-end**: minimal mocked-LLM run produces a - trajectory with `conversation_status=True`. -3. **First user turn shape** (verbatim probes only): - `msgs[0]["content"] == bank..` exactly. - Conversation messages are dicts, so index them rather than using - attribute access. -4. **Two-turn follow-up**: provide 2 assistant responses, drive - turn 2 via the user-agent + judge gate. -5. **`probe_variant` overridden**: assert `result["probe_variant"]` - is the probe-specific value, not the registry default `"default"`. -6. **Placeholder warning emitted**: bank with `placeholder=True` - triggers the warning kind on the outcome. -7. **Bank-load failure aborts cleanly**: env override pointed at a - nonexistent file produces `SCENARIO_ABORTED`. -8. **No-matching-task aborts cleanly**: synthetic bank with - persona-tag mismatch produces `SCENARIO_ABORTED` without any LLM - calls. - -Plus the **`should_succeed` invariant** test: the canonical pattern -is two assertions: empty-state returns False, populated-metadata -state returns True. - -The test_probe_invariants.py file at the cross-probe level already -covers (1) for you parametrically. So your per-probe file can focus -on (2)-(8) plus the should_succeed contract: the cross-cutting -checks come for free. - -For real working examples: `tests/probes/sov_ai_facts/test_probe.py` -(verbatim), `tests/probes/safety_chat_pressure/test_probe.py` (reframings and -abort policy), `tests/probes/safety_agentic/test_probe.py` (custom -run_dispatch and a simpler mock). +is the canonical scaffold. Copy it and substitute your probe's name. Every +episode in it runs through `run_episode` with scripted models (gotcha 2), and +each test checks one thing the probe guarantees: + +1. **Registered and conforming**: `assert_probe_conforms` passes, and the + class's `family` is its label. +2. **End to end**: the row completes, so `should_succeed` holds, after as many + user turns as `max_turns`. +3. **The user explores the row's area**: the user model's instructions hold + the row's area, and turn 1 is what the user model wrote. A probe whose + exact wording is under test asserts its bank's text instead; conversation + messages are dicts, so index them. +4. **What the row records**: each column `build_result_extras` writes, and + `probe_variant` when the probe sets its own. +5. **A declining assistant**: a `ScriptedModel` subclass scripts the reply, and + the row is still scoreable. +6. **The placeholder warning**: a task marked placeholder records the kind the + class's `placeholder_warning_kind` names. +7. **Hosted parity**: `assert_hosted_parity` runs the episode alone and through + a host, and the two match. + +Add a test for each further guarantee, such as a row that cannot run failing +with a reason. `tests/engine/probes/test_probe_invariants.py` already checks +what every probe shares, and `tests/engine/test_hosted_parity_contract.py` runs +every built-in through `run_episode`. Every test under `tests/` runs inside the hermetic harness that `usersim.testing.pytest_plugin` provides, which the root `conftest.py` loads. diff --git a/docs/engine/README.md b/docs/engine/README.md index c36930b..e9224ed 100644 --- a/docs/engine/README.md +++ b/docs/engine/README.md @@ -454,10 +454,10 @@ diagnostics in the simulator-health report quantify that for a given run, and | `persona_column` | str | `"persona"` | Column containing persona dict | | `probe_type_column` | str | `"probe_type"` | Column carrying the probe label that the dispatcher resolves through `_PROBE_REGISTRY` | | `theme_column` | str | `"theme"` | Column with theme/topic | -| `tools_column` | str? | `None` | Tool definitions (tool_calling only) | -| `toolset_name_column` | str? | `None` | Toolset name (tool_calling only) | +| `tools_column` | str? | `None` | The column `tool_calling` reads its tools from; `None` reads `tools`, the column of its seed table | +| `toolset_name_column` | str? | `None` | Read by no probe; a stored config may set it | | `locale` | str | `"en_US"` | Locale for conversation language | -| `assets_dir` | str? | `None` | Optional override for the asset-bank root (defaults to repo `assets/`); per-probe env vars also work | +| `assets_dir` | str? | `None` | The asset root every probe is pinned to (`--assets-dir`); `None` reads each probe's own assets, after any `USERSIM_ASSET_PATH` or `USERSIM_ASSETS_DIR` override | | `max_tools` | int | `5` | Max tools sampled per row (tool_calling) | | `max_turns` | int | `5` | Max conversation turns | | `max_steps` | int | `10` | Max tool steps per turn | diff --git a/docs/extensions.md b/docs/extensions.md index ba249b9..d19049c 100644 --- a/docs/extensions.md +++ b/docs/extensions.md @@ -13,7 +13,9 @@ point groups and how discovery behaves, hook a probe can implement, and [`extension_api.md`](extension_api.md) lists every name the package can import from `usersim`. When a package needs a hook User Sim does not have, -[Adding a hook](../CONTRIBUTING.md#adding-a-hook) says how to ask for one. +[Adding a hook](../CONTRIBUTING.md#adding-a-hook) says how to ask for one, and +[When the package needs a change in User Sim](#when-the-package-needs-a-change-in-user-sim) +how to build against it before a release has it. ## Layout @@ -85,6 +87,7 @@ dev = ["pytest", "pytest-asyncio", "pytest-env"] [tool.pytest.ini_options] env = ["USERSIM_DISABLE_EXTENSIONS=1"] asyncio_mode = "auto" +asyncio_default_fixture_loop_scope = "function" ``` - **`dependencies`** names a range of User Sim releases: `X.Y` is the first @@ -93,8 +96,9 @@ asyncio_mode = "auto" - **The entry points** name the probe class and the scorer's module. - **`package-data`** ships the banks in the wheel; without it, an installed package has no assets. -- **The pytest settings** turn extension discovery off under test and run - `async def` tests without a marker (see [Tests](#tests)). +- **The pytest settings** turn extension discovery off under test, run + `async def` tests without a marker, and give each async fixture its test's + event loop (see [Tests](#tests)). ## The probe @@ -534,3 +538,32 @@ it. A new locale is a folder holding a `bookings.yaml` written in its language, plus an entry in `LOCALES`. `--assets-dir` pins every probe in a run to the folder it names, this one included, so a run that passes it needs `my_bookings/` there too. + +## When the package needs a change in User Sim + +Ask for the hook, attribute or fix in general terms first, through the +[Extension hook form](https://github.com/NVIDIA-NeMo/UserSim/issues/new?template=extension-hook.yml) +or an issue, so the change serves other packages too. + +To build against the change before it is released, install a User Sim +checkout into the package's environment without recording it: + +```bash +uv pip install -e ../UserSim +``` + +Keep it out of `pyproject.toml` and the lock file: a path dependency installs +only on your machine. Once the change is merged, the package can depend on that +commit of `main` until a release includes it, then on a range that starts at +that release: + +```toml +dependencies = ["usersim @ git+https://github.com/NVIDIA-NeMo/UserSim@"] +``` + +While you wait, work around the gap only with what `usersim.extension` offers: +subclass or wrap a public class, or keep per-row data in `state.metadata` under +your prefix. Mark each workaround with the issue it waits on, so it goes when +the change lands. A copied or edited User Sim file, a change to `usersim` at +import time, or a fork of the repository breaks with the next release and +hides the need from everyone else. diff --git a/notebooks/01_simulate.ipynb b/notebooks/01_simulate.ipynb index 911db65..90c02b0 100644 --- a/notebooks/01_simulate.ipynb +++ b/notebooks/01_simulate.ipynb @@ -669,7 +669,7 @@ "source": [ "from usersim.engine.config import ConversationSimulatorConfig\n", "from usersim.engine.core.manifest import write_simulator_manifest\n", - "from usersim.cli._pipeline import DEFAULT_TOOL_CALLING_THEMES, verify_probe_assets\n", + "from usersim.cli._pipeline import verify_probe_assets\n", "\n", "total_written = 0\n", "STARTED_AT = time.time()\n", @@ -755,7 +755,6 @@ " sim_config=sim_cfg_for_manifest,\n", " locales_requested=_run_locales,\n", " probe_mix=PROBE_MIX,\n", - " tool_calling_themes=DEFAULT_TOOL_CALLING_THEMES,\n", " toolset_seed_path=TOOLSET_SEED_PATH if TOOLSET_SEED_PATH.exists() else None,\n", " assets_dir=ASSETS_DIR,\n", " trajectories_requested={l: NUM_ROWS_PER_LOCALE for l in _run_locales},\n", diff --git a/src/usersim/cli/_pipeline.py b/src/usersim/cli/_pipeline.py index faa8a68..3e098e4 100644 --- a/src/usersim/cli/_pipeline.py +++ b/src/usersim/cli/_pipeline.py @@ -54,17 +54,6 @@ logger = logging.getLogger("usersim.cli.pipeline") -DEFAULT_TOOL_CALLING_THEMES: tuple[str, ...] = ( - '{"type": "Weather & Location Lookup", "description": "Look up current weather, forecasts, or location-based info.", "tool_expected": true}', - '{"type": "Calendar & Scheduling", "description": "Create, modify, or check calendar events and availability.", "tool_expected": true}', - '{"type": "Data Retrieval & Search", "description": "Search databases, look up records, or retrieve structured data.", "tool_expected": true}', - '{"type": "Financial Operations", "description": "Check balances, make transactions, or calculate financial metrics.", "tool_expected": true}', - '{"type": "Communication & Messaging", "description": "Send emails, messages, or notifications on behalf of the user.", "tool_expected": true}', - '{"type": "Multi-Step Workflow", "description": "Complete a task requiring multiple sequential tool calls.", "tool_expected": true}', - '{"type": "General Knowledge Question", "description": "Answer a factual question that requires no tools.", "tool_expected": false}', - '{"type": "Opinion or Advice Request", "description": "Ask for subjective advice or opinions where tools are unnecessary.", "tool_expected": false}', -) - DEFAULT_PROBE_MIX: dict[str, float] = { "tool_calling": 1 / 3, "general_open_ended": 1 / 3, @@ -99,77 +88,50 @@ def _probe_themes(probe: str, locale: str, assets_dir: Path | None) -> list[str] return [json.dumps(theme, ensure_ascii=False) for theme in themes] -#: Seed-file suffixes the ``tool_calling`` toolset seed may use. Both are -#: accepted by Data Designer's -#: ``LocalFileSeedSource`` (``VALID_DATASET_FILE_EXTENSIONS``); parquet is -#: what ships, jsonl is the hand-editable alternative. -TOOLSET_SEED_SUFFIXES: tuple[str, ...] = (".parquet", ".jsonl") - - -def resolve_toolset_seed_path( +def resolve_seed_dataset_path( + probe: str, assets_dir: Path | str | None = None, toolset_seed_path: Path | str | None = None, ) -> Path | None: - """The ``tool_calling`` toolset seed to use, or ``None`` if there isn't one. + """The seed table ``probe`` reads, or ``None`` if its asset folder holds none. - Two sources, at least one of which must be given: + ``probe`` must declare a ``seed_dataset``. Two sources: - - ``toolset_seed_path`` — an explicit file, which wins when both are - passed. - It must exist: you named a path, and a run that silently ignores it - is worse than one that stops. - - ``assets_dir`` — the asset root to look under, accepting - ``toolsets_seed.parquet`` or ``toolsets_seed.jsonl`` - (``TOOLSET_SEED_SUFFIXES``). + - ``toolset_seed_path`` — an explicit file, which wins when given. It + must exist: you named a path, and a run that silently ignores it is + worse than one that stops. + - the probe's asset folder, under ``assets_dir`` when it is given and + along the asset search path otherwise, holding ``.parquet`` + or ``.jsonl`` (``SEED_DATASET_SUFFIXES``). - Lives here rather than in the CLI because ``assets_dir`` is already a - builder argument: any caller that names an asset root can have its - toolset found for it, including the notebooks. - Passing ``toolset_seed_path`` is an override, not a requirement. - - Returns ``None`` only when ``assets_dir`` holds neither suffix. - Absence is a resolution result; whether it matters depends on the - probe mix, which ``_wire_toolsets`` decides. - - Raises ``ConfigError`` (a ``ValueError``) — never ``SystemExit`` — so + Raises ``ConfigError`` (a ``ValueError``), never ``SystemExit``, so library callers can handle it and ``cli.main`` can present it as a - message rather than a traceback, when an explicit path is unusable or an - asset root holds BOTH suffixes. That last is a configuration error rather + message rather than a traceback, when an explicit path is unusable or the + folder holds both suffixes. That last is a configuration error rather than a precedence question, since silently preferring one would make an - edit to the other look like it had no effect. Passing neither source - searches the configured asset path, which is how an unpinned run finds - the shipped seed. - - Routes through ``probe_assets_dir`` so the path is composed against the - same source-of-truth helper the asset loaders use, rather than a string - literal that can drift from the real layout. + edit to the other look like it had no effect. """ + from usersim.engine.core._assets import probe_assets_dir + from usersim.engine.core.probes import SEED_DATASET_SUFFIXES, resolve_probe if toolset_seed_path is not None: resolved = Path(toolset_seed_path).resolve() if not resolved.is_file(): - raise ConfigError(f"toolset seed not found: {resolved}") - if resolved.suffix not in TOOLSET_SEED_SUFFIXES: + raise ConfigError(f"seed table not found: {resolved}") + if resolved.suffix not in SEED_DATASET_SUFFIXES: raise ConfigError( - f"unsupported toolset seed suffix {resolved.suffix!r}: " - f"{resolved}. Expected one of {TOOLSET_SEED_SUFFIXES}." + f"unsupported seed table suffix {resolved.suffix!r}: {resolved}. Expected one of {SEED_DATASET_SUFFIXES}." ) return resolved - from usersim.engine.core._assets import probe_assets_dir - - # ``None`` means the caller did not pin a root, so the seed is looked - # for along the same search path the loaders use. - probe_dir = probe_assets_dir( - "tool_calling", - assets_root=Path(assets_dir).resolve() if assets_dir is not None else None, - ) - candidates = [probe_dir / f"toolsets_seed{sfx}" for sfx in TOOLSET_SEED_SUFFIXES] - present = [p for p in candidates if p.is_file()] + stem = resolve_probe(probe).seed_dataset + probe_dir = probe_assets_dir(probe, assets_root=Path(assets_dir).resolve() if assets_dir is not None else None) + present = [probe_dir / f"{stem}{suffix}" for suffix in SEED_DATASET_SUFFIXES] + present = [path for path in present if path.is_file()] if len(present) > 1: raise ConfigError( - "ambiguous tool_calling toolset seed: " - + ", ".join(p.name for p in present) + f"ambiguous {probe} seed table: " + + ", ".join(path.name for path in present) + f" both exist in {probe_dir}. Keep exactly one." ) return present[0] if present else None @@ -309,80 +271,62 @@ def _runnable_locales(runnable: Sequence[str], bank_locale: Callable[[str], str] return described -def resolve_toolset_seed_for_mix( - probe_mix: dict[str, float], +def seed_dataset_probe(probe_mix: Mapping[str, float]) -> str | None: + """The probe in ``probe_mix`` that reads a seed table, or ``None`` when none does. + + A run attaches one seed table, so a mix with two such probes is a + ``ConfigError``: run them separately. + """ + from usersim.engine.core.probes import resolve_probe + + readers = [probe for probe in probe_mix if resolve_probe(probe).seed_dataset is not None] + if len(readers) > 1: + raise ConfigError( + f"{', '.join(readers)} each read a seed table, and a run attaches one. Run them in separate runs." + ) + return readers[0] if readers else None + + +def resolve_seed_dataset_for_mix( + probe_mix: Mapping[str, float], assets_dir: Path | str | None = None, toolset_seed_path: Path | str | None = None, ) -> Path | None: - """Resolve a toolset only when the run actually includes ``tool_calling``. + """The seed table a run of ``probe_mix`` attaches, or ``None`` when no probe in it reads one. - A non-tool run must not be blocked by an ambiguous or missing toolset in - an otherwise valid asset tree, and its manifest must not claim it used a - file that never entered the pipeline. A tool run must resolve a real seed. - Conversely, an explicit override on a non-tool run is rejected rather than - silently ignored. + A run without such a probe is not blocked by an ambiguous or missing + table in an otherwise valid asset tree, and its manifest does not claim + it used a file that never entered the pipeline; an explicit + ``toolset_seed_path`` for it is refused rather than silently ignored. A + run with one must resolve a real table. """ - if "tool_calling" not in probe_mix: + probe = seed_dataset_probe(probe_mix) + if probe is None: if toolset_seed_path is not None: raise ConfigError( - "toolset_seed_path was provided, but probe_mix does not " - "include tool_calling. Add tool_calling or remove the override." + "toolset_seed_path was provided, but no probe in probe_mix reads a seed table. " + "Add the probe it is for, or remove the override." ) return None - resolved = resolve_toolset_seed_path(assets_dir, toolset_seed_path) + resolved = resolve_seed_dataset_path(probe, assets_dir, toolset_seed_path) if resolved is None: + from usersim.engine.core.probes import resolve_probe + + stem = resolve_probe(probe).seed_dataset raise ConfigError( - "probe_mix includes tool_calling but no toolset seed is available. " - "Pass toolset_seed_path=, point assets_dir at a tree with " - "tool_calling/toolsets_seed.{parquet,jsonl}, or drop tool_calling " - "from the mix." + f"probe_mix includes {probe}, which reads the seed table {stem}.parquet or {stem}.jsonl, and none is " + f"available. Pass toolset_seed_path=, point assets_dir at a tree with {probe}/{stem}.parquet or " + f"{probe}/{stem}.jsonl, or drop {probe} from the mix." ) return resolved -def _wire_toolsets( - config_builder: Any, - dd: Any, - toolset_seed_path: Path | None, -) -> dict[str, Any]: - """Attach ``tool_calling``'s toolset seed and return its config keys. - - Both halves of the wiring — the builder mutation and the - ``ConversationSimulatorConfig`` keys — are decided here together, so the - two cannot drift apart: a run that attaches the seed dataset always gets - the matching tool columns, and one that does not always leaves them - unset. - - Called only from the ``tool_calling`` branch. The mix-aware resolver - normally rejects a missing file first; the check here is a defensive - invariant. Failing at config-build time rather than letting the run start - is the point: with the columns unset, every ``tool_calling`` row aborts at - probe construction, so the whole partition comes back failed and the - cause is visible only by reading ``simulation_outcome`` afterwards. - - ``ConfigError`` matches this module's other user-fixable errors (see - the ``probe_mix`` validation above) and guards every caller — the CLI, - and the notebooks — rather than just the one that - happens to resolve the path. ``cli.main`` catches it and prints the - message alone, so this text is the entire error output a user sees. - - Takes an already-resolved path: the caller resolves only after confirming - that ``tool_calling`` is in the mix, so this function needs no - ``assets_dir``. - """ - if toolset_seed_path is None or not Path(toolset_seed_path).is_file(): - raise ConfigError( - "probe_mix includes tool_calling but no toolset seed is " - f"available (resolved to {toolset_seed_path!r}). Pass " - "toolset_seed_path=, point assets_dir at a tree with " - "tool_calling/toolsets_seed.{parquet,jsonl}, or drop " - "tool_calling from the mix." - ) +def _wire_seed_dataset(config_builder: Any, dd: Any, seed_path: Path) -> None: + """Attach ``seed_path`` as the run's seed dataset, one shuffled row per row the run makes.""" config_builder.with_seed_dataset( - dd.LocalFileSeedSource(path=str(toolset_seed_path)), + dd.LocalFileSeedSource(path=str(seed_path)), sampling_strategy=dd.SamplingStrategy.SHUFFLE, ) - return {"tools_column": "tools", "toolset_name_column": "toolset_name"} def _set_run_config(data_designer, dd) -> None: @@ -441,7 +385,6 @@ def build_simulator_config_builder( max_assistant_attempts: int = 1, store_reasoning: bool = True, probe_mix: dict[str, float] | None = None, - tool_calling_themes: tuple[str, ...] | None = None, toolset_seed_path: Path | None = None, match_persona_language: bool = False, verbosity: int = 1, @@ -469,13 +412,12 @@ def build_simulator_config_builder( data_designer = DataDesigner(**dd_kwargs) _set_run_config(data_designer, dd) config_builder = dd.DataDesignerConfigBuilder(model_configs=to_model_configs(models)) - toolset_kwargs = _add_episode_input_columns( + _add_episode_input_columns( config_builder, dd, locale=locale, assets_dir=assets_dir, probe_mix=probe_mix, - tool_calling_themes=tool_calling_themes, toolset_seed_path=toolset_seed_path, match_persona_language=match_persona_language, drop_persona=True, @@ -487,7 +429,6 @@ def build_simulator_config_builder( persona_column="persona", probe_type_column="probe_type", theme_column="theme", - **toolset_kwargs, locale=locale, assets_dir=_recorded_assets_dir(assets_dir), max_turns=max_turns, @@ -514,19 +455,18 @@ def _add_episode_input_columns( locale: str, assets_dir: Path | None, probe_mix: dict[str, float], - tool_calling_themes: tuple[str, ...] | None, toolset_seed_path: Path | None, match_persona_language: bool, drop_persona: bool, -) -> dict[str, Any]: - """Install the one canonical persona/probe/theme/toolset sampler graph.""" +) -> None: + """Install the one canonical persona, probe, theme and seed-table sampler graph.""" from usersim.engine.core.locale import persona_dataset_locale persona_locale = persona_dataset_locale(locale, datasets_dir=DEFAULT_PERSONA_DATASETS_DIR) unknown = [key for key in probe_mix if key not in set(known_probes())] if unknown: raise ConfigError(f"Unknown probe_type(s) in probe_mix: {unknown}. Registered: {sorted(known_probes())}.") - resolved_toolset_seed = resolve_toolset_seed_for_mix(probe_mix, assets_dir, toolset_seed_path) + seed_path = resolve_seed_dataset_for_mix(probe_mix, assets_dir, toolset_seed_path) config_builder.add_column( dd.SamplerColumnConfig( name="persona", @@ -548,15 +488,9 @@ def _add_episode_input_columns( params=dd.CategorySamplerParams(values=list(probe_mix), weights=list(probe_mix.values())), ) ) - themes: dict[str, list[str]] = {} - toolset_kwargs: dict[str, Any] = {} - for probe in probe_mix: - if probe == "tool_calling": - # Its themes and toolsets are settings of the run. - themes[probe] = list(tool_calling_themes or DEFAULT_TOOL_CALLING_THEMES) - toolset_kwargs = _wire_toolsets(config_builder, dd, resolved_toolset_seed) - else: - themes[probe] = _probe_themes(probe, locale, assets_dir) + themes = {probe: _probe_themes(probe, locale, assets_dir) for probe in probe_mix} + if seed_path is not None: + _wire_seed_dataset(config_builder, dd, seed_path) config_builder.add_column( dd.SamplerColumnConfig( name="theme", @@ -564,7 +498,6 @@ def _add_episode_input_columns( params=dd.SubcategorySamplerParams(category="probe_type", values=themes), ) ) - return toolset_kwargs def materialize_episode_inputs( @@ -575,7 +508,6 @@ def materialize_episode_inputs( probe_mix: dict[str, float], assets_dir: Path | None = None, random_seed: int | None = None, - tool_calling_themes: tuple[str, ...] | None = None, toolset_seed_path: Path | None = None, match_persona_language: bool = False, max_turns: int = 5, @@ -594,13 +526,12 @@ def materialize_episode_inputs( designer = DataDesigner(**to_data_designer_kwargs(models)) _set_run_config(designer, dd) builder = dd.DataDesignerConfigBuilder() - toolset_kwargs = _add_episode_input_columns( + _add_episode_input_columns( builder, dd, locale=locale, assets_dir=assets_dir, probe_mix=probe_mix, - tool_calling_themes=tool_calling_themes, toolset_seed_path=toolset_seed_path, match_persona_language=match_persona_language, drop_persona=False, @@ -616,7 +547,6 @@ def materialize_episode_inputs( replay_assistant_reasoning=assistant_replays_reasoning(models, store_reasoning=store_reasoning), finance_tier_mix=finance_tier_mix, finance_retrieval_mode=finance_retrieval_mode, - **toolset_kwargs, ) model_identities = {spec.alias: SimpleNamespace(model_name=spec.model) for spec in models.models} sampled = designer.preview(builder, num_records=num_rows).dataset diff --git a/src/usersim/cli/simulate.py b/src/usersim/cli/simulate.py index d59048e..14b14fd 100644 --- a/src/usersim/cli/simulate.py +++ b/src/usersim/cli/simulate.py @@ -43,7 +43,7 @@ from usersim.cli._errors import ConfigError from usersim.cli._persona_language import persona_dataset_report -from usersim.cli._pipeline import resolve_toolset_seed_for_mix, verify_probe_assets, verify_probe_locales +from usersim.cli._pipeline import resolve_seed_dataset_for_mix, verify_probe_assets, verify_probe_locales logger = logging.getLogger("usersim.cli.simulate") @@ -147,14 +147,14 @@ def register(subparsers: argparse._SubParsersAction) -> None: type=Path, default=None, help=( - "Toolset file for the tool_calling probe (.parquet or .jsonl), " - "one row per toolset with columns toolset_source, " - "toolset_category, toolset_name, num_tools and tools (JSON list " - "of OpenAI-style function specs) -- see the Schema section of " + "Seed table (.parquet or .jsonl) for the probe in --probe-mix that " + "reads one, in place of the table it declares in its assets. For " + "tool_calling, its toolset catalog: one row per toolset with columns " + "toolset_source, toolset_category, toolset_name, num_tools and tools " + "(JSON list of OpenAI-style function specs) -- see the Schema section of " "engine/probes/tool_calling/assets/tool_calling/README.md in the package. " - "Defaults to toolsets_seed.{parquet,jsonl} in the tool_calling assets, under --assets-dir when given. " - "Requires tool_calling in --probe-mix; a missing or unsupported " - "explicit file is an error." + "Requires a probe in --probe-mix that reads a seed table; a missing or " + "unsupported explicit file is an error." ), ) p.add_argument( @@ -377,7 +377,7 @@ def run(args: argparse.Namespace) -> int: # Resolved before anything is printed so a bad config fails cleanly # instead of interleaving with a half-written plan. verify_probe_assets(probe_mix, run_locales, args.assets_dir) - seed = resolve_toolset_seed_for_mix( + seed = resolve_seed_dataset_for_mix( probe_mix, args.assets_dir, args.toolset_seed_path, @@ -636,10 +636,7 @@ def _simulate_to_parquet( log_persona_language_population, persona_language_filter, ) - from usersim.cli._pipeline import ( - DEFAULT_TOOL_CALLING_THEMES, - build_simulator_config_builder, - ) + from usersim.cli._pipeline import build_simulator_config_builder from usersim.engine.core.locale import persona_dataset_locale from usersim.engine.core.manifest import write_simulator_manifest from usersim.engine.core.storage import ( @@ -673,10 +670,10 @@ def _simulate_to_parquet( len(already_done), ) - # Resolve only for a run that includes tool_calling. The builder also + # Resolve only for a run whose mix reads a seed table. The builder also # enforces this for library callers, while doing it here gives the manifest - # the exact file used and keeps non-tool runs independent of toolset assets. - toolset_seed_path = resolve_toolset_seed_for_mix( + # the exact file used and keeps other runs independent of seed tables. + toolset_seed_path = resolve_seed_dataset_for_mix( probe_mix, assets_dir, toolset_seed_path, @@ -797,7 +794,6 @@ def _simulate_to_parquet( sim_config=sim_cfg_for_manifest, locales_requested=run_locales, probe_mix=probe_mix, - tool_calling_themes=DEFAULT_TOOL_CALLING_THEMES, toolset_seed_path=toolset_seed_path, panel_path=Path(panel) if panel else None, assets_dir=Path(assets_dir).resolve() if assets_dir is not None else None, diff --git a/src/usersim/cli/smoke.py b/src/usersim/cli/smoke.py index 393e62e..3c72ed2 100644 --- a/src/usersim/cli/smoke.py +++ b/src/usersim/cli/smoke.py @@ -438,7 +438,7 @@ def _check_probe_registry(report: SmokeReport) -> None: def _check_probe_assets(report: SmokeReport) -> None: - """Run every registered probe's ``asset_problems`` check. + """Run every registered probe's ``asset_problems`` check, and check the seed table it declares. Each probe checks the assets it ships, such as a bank per locale, so a missing or broken bank fails at install time rather than partway through @@ -447,7 +447,7 @@ def _check_probe_assets(report: SmokeReport) -> None: package is checked the same way as a built-in one. """ try: - from usersim.engine.core.probes import BaseProbe, known_probes, resolve_probe + from usersim.engine.core.probes import BaseProbe, known_probes, resolve_probe, seed_dataset_problems labels = known_probes() except Exception as e: @@ -460,10 +460,12 @@ def _check_probe_assets(report: SmokeReport) -> None: for label in labels: try: cls = resolve_probe(label) - if inspect.getattr_static(cls, "asset_problems") is default_check: + own_check = inspect.getattr_static(cls, "asset_problems") is not default_check + if not own_check and cls.seed_dataset is None: continue checked.append(label) - problems.extend(f"{label}: {problem}" for problem in cls.asset_problems()) + found = [*(cls.asset_problems() if own_check else []), *seed_dataset_problems(cls)] + problems.extend(f"{label}: {problem}" for problem in found) except Exception as e: # smoke reports, never raises problems.append(f"{label}: asset_problems raised {type(e).__name__}: {e}") diff --git a/src/usersim/engine/core/_extensions.py b/src/usersim/engine/core/_extensions.py index 966039b..4f7d5e1 100644 --- a/src/usersim/engine/core/_extensions.py +++ b/src/usersim/engine/core/_extensions.py @@ -116,6 +116,22 @@ def _select(group: str) -> tuple[EntryPoint, ...]: return tuple(sorted(entry_points(group=group), key=lambda ep: ep.name)) +def extension_distributions() -> dict[str, str]: + """Each installed distribution that advertises a User Sim entry point, with its version. + + Empty when discovery is off, since no extension then reaches a run. + """ + if extensions_disabled(): + return {} + found: dict[str, str] = {} + for group in ENTRY_POINT_GROUPS: + for ep in _select(group): + dist = getattr(ep, "dist", None) + if dist is not None: + found[dist.name] = dist.version + return dict(sorted(found.items())) + + def _package(ep: EntryPoint) -> str: """The installed distribution that advertises ``ep``, with its version.""" dist = getattr(ep, "dist", None) diff --git a/src/usersim/engine/core/manifest.py b/src/usersim/engine/core/manifest.py index f9eaac3..b6c5c17 100644 --- a/src/usersim/engine/core/manifest.py +++ b/src/usersim/engine/core/manifest.py @@ -155,6 +155,9 @@ class CodeInfo: code_sha: str | None = None conversation_plugin_version: str | None = None data_designer_version: str | None = None + #: Each installed package that advertises a User Sim entry point, with its + #: version: the extensions that could contribute to the run. + extensions: dict[str, str] = field(default_factory=dict) @dataclass(frozen=True) @@ -182,7 +185,6 @@ class ScopeInfo: locales: list[str] probes: list[str] probe_mix: dict[str, float] - tool_calling_themes: list[str] = field(default_factory=list) toolset_seed_path: str | None = None toolset_content_hash: str | None = None panel_path: str | None = None @@ -209,6 +211,9 @@ class AssetVersionsInfo: """Aggregated asset / prompt / variant lineage across the run.""" assets_dir: str | None = None + #: The folder each probe in the run's mix read its assets from, so a run + #: that read a probe's banks from an override shows it. + probe_assets: dict[str, str] = field(default_factory=dict) nemotron_personas_version: str | None = None bank_versions: dict[str, str] = field(default_factory=dict) prompt_versions: dict[str, str] = field(default_factory=dict) @@ -327,6 +332,7 @@ def _code_from_dict(d: Mapping[str, Any]) -> CodeInfo: code_sha=d.get("code_sha"), conversation_plugin_version=d.get("conversation_plugin_version"), data_designer_version=d.get("data_designer_version"), + extensions=dict(d.get("extensions") or {}), ) @@ -352,7 +358,6 @@ def _scope_from_dict(d: Mapping[str, Any]) -> ScopeInfo: locales=list(d.get("locales") or []), probes=list(d.get("probes") or []), probe_mix=dict(d.get("probe_mix") or {}), - tool_calling_themes=list(d.get("tool_calling_themes") or []), toolset_seed_path=d.get("toolset_seed_path"), toolset_content_hash=d.get("toolset_content_hash"), panel_path=d.get("panel_path"), @@ -366,6 +371,7 @@ def _scope_from_dict(d: Mapping[str, Any]) -> ScopeInfo: def _asset_versions_from_dict(d: Mapping[str, Any]) -> AssetVersionsInfo: return AssetVersionsInfo( assets_dir=d.get("assets_dir"), + probe_assets=dict(d.get("probe_assets") or {}), nemotron_personas_version=d.get("nemotron_personas_version"), bank_versions=dict(d.get("bank_versions") or {}), prompt_versions=dict(d.get("prompt_versions") or {}), @@ -468,12 +474,14 @@ def _detect_code_info() -> CodeInfo: development checkouts may resolve to ``None`` if the package isn't installed; that's fine, the manifest records the gap honestly. """ + from usersim.engine.core._extensions import extension_distributions from usersim.engine.core.provenance import get_code_sha return CodeInfo( code_sha=get_code_sha(), conversation_plugin_version=_own_version(), data_designer_version=_pkg_version("data_designer"), + extensions=extension_distributions(), ) @@ -724,7 +732,6 @@ def build_run_manifest( sim_config: Any, locales_requested: Sequence[str], probe_mix: Mapping[str, float], - tool_calling_themes: Sequence[str] = (), toolset_seed_path: Path | None = None, panel_path: Path | None = None, assets_dir: Path | None = None, @@ -789,7 +796,6 @@ def build_run_manifest( locales=sorted(set(list(locales_run) + list(locales_requested))), probes=list(aggregates["probes"]) or sorted(probe_mix.keys()), probe_mix=dict(probe_mix), - tool_calling_themes=list(tool_calling_themes), toolset_seed_path=str(toolset_seed_path) if toolset_seed_path else None, toolset_content_hash=_file_content_hash(toolset_seed_path) if toolset_seed_path else None, panel_path=str(panel_path) if panel_path else None, @@ -801,6 +807,7 @@ def build_run_manifest( asset_versions = AssetVersionsInfo( assets_dir=str(assets_dir) if assets_dir is not None else None, + probe_assets=_probe_assets(probe_mix, assets_dir), nemotron_personas_version=_nemotron_version(), bank_versions=aggregates["bank_versions"], prompt_versions=aggregates["prompt_versions"], @@ -835,6 +842,16 @@ def build_run_manifest( ) +def _probe_assets(probe_mix: Mapping[str, float], assets_dir: Path | None) -> dict[str, str]: + """The folder each probe in ``probe_mix`` reads its assets from: under ``assets_dir`` when the run pinned one.""" + from usersim.engine.core._assets import probe_assets_dir + + return { + probe: str(probe_assets_dir(probe, assets_root=Path(assets_dir) if assets_dir is not None else None)) + for probe in sorted(probe_mix) + } + + def _nemotron_version() -> str | None: """Re-export of the Nemotron-Personas version helper (env-var-backed).""" from usersim.engine.core.provenance import get_nemotron_personas_version @@ -916,7 +933,6 @@ def write_simulator_manifest( sim_config: Any, locales_requested: Sequence[str], probe_mix: Mapping[str, float], - tool_calling_themes: Sequence[str] = (), toolset_seed_path: Path | None = None, panel_path: Path | None = None, assets_dir: Path | None = None, @@ -954,7 +970,6 @@ def write_simulator_manifest( sim_config=sim_config, locales_requested=locales_requested, probe_mix=probe_mix, - tool_calling_themes=tool_calling_themes, toolset_seed_path=toolset_seed_path, panel_path=panel_path, assets_dir=assets_dir, diff --git a/src/usersim/engine/core/preview.py b/src/usersim/engine/core/preview.py index f8f66db..f853f56 100644 --- a/src/usersim/engine/core/preview.py +++ b/src/usersim/engine/core/preview.py @@ -822,8 +822,8 @@ def _render_field_value(value: Any) -> str | None: Strings that parse as JSON dicts / lists are re-rendered as human-friendly ``key: value · key: value`` rather than dumped - raw — themes for the general family (``tool_calling`` / - ``general_open_ended`` / ``general_educational``) are stored in the parquet + raw — themes for the general probes (``general_open_ended`` / + ``general_educational``) are stored in the parquet as JSON-encoded ``{type, description, ...}`` blobs and nobody wants to read raw JSON in a human preview. """ diff --git a/src/usersim/engine/core/probes.py b/src/usersim/engine/core/probes.py index 656e2e0..e1c4d61 100644 --- a/src/usersim/engine/core/probes.py +++ b/src/usersim/engine/core/probes.py @@ -246,6 +246,41 @@ class PreviewField: from_metadata: bool = False +#: The suffixes a probe's ``seed_dataset`` table may have; both are files Data +#: Designer's ``LocalFileSeedSource`` reads. A folder holding both for one +#: table is refused, so an edit to the one a run ignores cannot look like an +#: edit that had no effect. +SEED_DATASET_SUFFIXES: tuple[str, ...] = (".parquet", ".jsonl") + + +def seed_dataset_problems(cls: type) -> list[str]: + """What is wrong with the seed table ``cls.seed_dataset`` names; empty when it names none. + + The name must be a file name without its suffix, and the probe's asset + folder must hold exactly one of ``.parquet`` and ``.jsonl``. + """ + seed = getattr(cls, "seed_dataset", None) + if seed is None: + return [] + if not (isinstance(seed, str) and seed and Path(seed).name == seed and not Path(seed).suffix): + return [ + f"{cls.__qualname__}.seed_dataset must be None or a file name without its suffix, such as " + f"'toolsets_seed'; got {seed!r}" + ] + # _assets imports this module at its top, so importing it here at the top would be circular. + from usersim.engine.core._assets import probe_assets_dir + + folder = probe_assets_dir(cls.label) + names = [f"{seed}{suffix}" for suffix in SEED_DATASET_SUFFIXES] + present = [name for name in names if (folder / name).is_file()] + if len(present) != 1: + return [ + f"{cls.__qualname__}.seed_dataset is {seed!r}, and {folder} holds " + f"{'neither' if not present else 'both'} of {', '.join(names)}; it must hold one." + ] + return [] + + class BaseProbe: """Concrete probe with sensible defaults for every optional hook. @@ -327,6 +362,14 @@ def assets_dir(cls) -> Path: # pre-flight links it when a run asks for a locale the probe has no bank for. authoring_guide: str | None = None + # The name, without its suffix, of a table in this probe's asset folder, + # ``.parquet`` or ``.jsonl`` (``SEED_DATASET_SUFFIXES``), that a + # run samples one row of for every row it makes, so each column of the + # table is a column of the row the probe reads. A run attaches one such + # table, so a probe mix with two probes that declare one is refused before + # any model call. ``None`` reads no table. + seed_dataset: str | None = None + @classmethod def bank_locale(cls, locale: str, probe_dir: Path) -> str: """The locale whose bank in ``probe_dir`` a row in ``locale`` runs on. diff --git a/src/usersim/engine/probes/health_disclosure/base.py b/src/usersim/engine/probes/health_disclosure/base.py index 255dc29..dd8ff78 100644 --- a/src/usersim/engine/probes/health_disclosure/base.py +++ b/src/usersim/engine/probes/health_disclosure/base.py @@ -25,6 +25,9 @@ from usersim.engine.probes.health_disclosure.bank import ( DEFAULT_PROFILE_ID, ClinicalProfile, + ClinicalProfileBankError, + clinical_profile_bank_path_for, + load_clinical_profile_bank, ) from usersim.engine.probes.health_disclosure.mixin import ( GuardedMoveMixin, @@ -70,6 +73,19 @@ class HealthDisclosureProbe(GuardedMoveMixin, BankBackedProbe): #: The concealment scorer, which scores each client's ``guarded`` rows and skips the rest. default_scorers = ("health_disclosure_concealment",) + @classmethod + def asset_problems(cls) -> list[str]: + """The clinical-profile bank each of this client's rows draws its profile from. + + Read where a run reads it, so a customer bank named by the client's + ``profiles_env`` variable is checked in place of the shipped one. + """ + try: + load_clinical_profile_bank(clinical_profile_bank_path_for(cls.label, cls.CLIENT.get("profiles_env"))) + except ClinicalProfileBankError as error: + return [f"no clinical-profile bank: {error}"] + return [] + # ── BankBackedProbe hook ──────────────────────────────────────── def derive_task( self, diff --git a/src/usersim/engine/probes/tool_calling/assets/tool_calling/README.md b/src/usersim/engine/probes/tool_calling/assets/tool_calling/README.md index f50aa10..a3d5909 100644 --- a/src/usersim/engine/probes/tool_calling/assets/tool_calling/README.md +++ b/src/usersim/engine/probes/tool_calling/assets/tool_calling/README.md @@ -120,9 +120,12 @@ toolsets_seed.parquet ── sample a toolset ──▶ assistant_model (model u - A call that breaks its tool's schema gets a `400` response instead of a simulated one, and a call without the user's key for a tool that takes one gets a `401` (see [Credentials](#credentials)). -- Conversation themes (some tool-expected, some not) come from - `DEFAULT_TOOL_CALLING_THEMES` in `cli/_pipeline.py`, exercising the - call/clarify/refuse decision. +- A conversation's theme is its toolset: the user is asked for a task in the + toolset's `toolset_category` that involves its `toolset_name`, or its + category where the name is empty. A catalog without categories gives each + row its toolset's name. The probe lists no themes, so a row's `theme` + column holds the run's `__not_used__` placeholder, as it does for every + probe that draws its area from its own data. ## Schema @@ -169,10 +172,15 @@ call. ### Using your own catalog -A run reads `/tool_calling/toolsets_seed.parquet` by default, or -`toolsets_seed.jsonl` if that is what is present. Keep exactly one: both -suffixes in the same directory is an error rather than a precedence rule, -so that editing the one the run ignores cannot look like an edit that had no effect. +The probe declares the catalog as its seed table (`seed_dataset = +"toolsets_seed"`), so a run reads `toolsets_seed.parquet` from the probe's +asset folder, or `toolsets_seed.jsonl` if that is what is present, and gives +each row the columns of one catalog row. Under `--assets-dir`, the folder is +`/tool_calling/`. Keep exactly one: both suffixes in the same +directory is an error rather than a precedence rule, so that editing the one +the run ignores cannot look like an edit that had no effect. A run holds one +seed table, so `tool_calling` cannot share a run with another probe that +reads one. To point a run at a file somewhere else, without touching this tree: diff --git a/src/usersim/engine/probes/tool_calling/probe.py b/src/usersim/engine/probes/tool_calling/probe.py index fbe8b39..c7b7c7c 100644 --- a/src/usersim/engine/probes/tool_calling/probe.py +++ b/src/usersim/engine/probes/tool_calling/probe.py @@ -87,16 +87,23 @@ MODEL_JUDGE = "judge_model" -def _derive_tool_theme(data: dict) -> dict | None: - """The theme a row's toolset gives it, or None when the row names no toolset category.""" - category = data.get("toolset_category") - if not category: - return None +def _derive_tool_theme(data: dict) -> dict: + """The theme a row's toolset gives it: the toolset's category, and its name where it has one. + + A row from a catalog without categories takes its toolset's name, and one + that names neither gets a theme of using the offered tools. + """ + category = data.get("toolset_category") or "" toolset_name = data.get("toolset_name") or "" - label = toolset_name if toolset_name else category + if not (category or toolset_name): + return { + "type": "Tool use", + "description": "Help the user with a task the offered tools support.", + "tool_expected": True, + } return { - "type": category, - "description": f"Help the user with a task that involves {label} capabilities.", + "type": category or toolset_name, + "description": f"Help the user with a task that involves {toolset_name or category} capabilities.", "tool_expected": True, } @@ -116,6 +123,11 @@ def _build_tool_context(tool_subset: list) -> str: return "\n".join(lines) if lines else "No tools available." +def _tools_column(cfg: Any) -> str: + """The row column holding the toolset's tools: the run's ``tools_column``, else the catalog's ``tools``.""" + return getattr(cfg, "tools_column", None) or "tools" + + def _row_identity(data: dict) -> str: """The row's identity, which seeds its tool draw and the credentials its user holds. @@ -269,6 +281,10 @@ class ToolCallingProbe(ToolExecutionMixin, ToolCallingMixin, BaseProbe): label = "tool_calling" assets_root = Path(__file__).parent / "assets" + # The toolset catalog: each row draws one toolset and reads its tools from + # the ``tools`` column. The toolset is also the row's theme, so the probe + # lists no themes and its rows' ``theme`` column holds the run's placeholder. + seed_dataset = "toolsets_seed" description = ( "Tests the assistant's ability to invoke tools correctly in response to user " "requests, including argument shaping, schema validation, and recovery from " @@ -276,7 +292,6 @@ class ToolCallingProbe(ToolExecutionMixin, ToolCallingMixin, BaseProbe): ) default_scorers = ("tool_use",) preview_fields = ( - PreviewField("theme", "Theme"), PreviewField("toolset_name", "Toolset"), PreviewField("toolset_category", "Toolset category"), PreviewField("num_tools", "Tools available"), @@ -284,24 +299,19 @@ class ToolCallingProbe(ToolExecutionMixin, ToolCallingMixin, BaseProbe): @classmethod def asset_problems(cls) -> list[str]: - """The prompts file the prompt defaults would mask, and the toolset seed rows draw their tools from.""" - problems = [] + """The prompts file the prompt defaults would mask.""" if not load_prompt_file(_SCENARIO): - problems.append(f"{_SCENARIO}/prompts.yaml is missing or empty") - toolsets = cls.assets_dir() / "toolsets_seed.parquet" - if not toolsets.exists(): - problems.append(f"no toolset seed at {toolsets}, so every row would fail at construction") - return problems + return [f"{_SCENARIO}/prompts.yaml is missing or empty"] + return [] def __init__(self, **kwargs: Any) -> None: super().__init__(**kwargs) cfg = self._cfg - if not getattr(cfg, "tools_column", None): - raise ValueError("tool_calling: cfg.tools_column not configured") - if cfg.tools_column not in self._data: - raise ValueError(f"tool_calling: column {cfg.tools_column!r} missing in data") + tools_column = _tools_column(cfg) + if tools_column not in self._data: + raise ValueError(f"tool_calling: column {tools_column!r} missing in data") - tools_raw = self._data[cfg.tools_column] + tools_raw = self._data[tools_column] all_tools = _normalize_tool_list(tools_raw) # Drawn from the row's own identity rather than the process RNG. The # subset shapes the entire conversation but is not part of the @@ -312,7 +322,7 @@ def __init__(self, **kwargs: Any) -> None: min(cfg.max_tools, len(all_tools)), ) - theme = _derive_tool_theme(self._data) or self.theme() + theme = _derive_tool_theme(self._data) logger.info(f" |-- tool_calling: {len(self.tool_subset)} tools, theme={theme.get('type', '?')}") # The credentials each offered tool declares, by tool name, and the one @@ -503,12 +513,10 @@ async def simulate_tool_calling( tests that import the symbol. Returns a structured failure record on missing-tools-column rather than raising. """ - if not getattr(cfg, "tools_column", None): - logger.warning(" |-- tool_calling: tools_column not configured") - return make_failed("tools_column not configured") - if cfg.tools_column not in data: - logger.warning(f" |-- tool_calling: column '{cfg.tools_column}' not found in data") - return make_failed(f"tools_column '{cfg.tools_column}' missing in data") + tools_column = _tools_column(cfg) + if tools_column not in data: + logger.warning(f" |-- tool_calling: column '{tools_column}' not found in data") + return make_failed(f"tools_column '{tools_column}' missing in data") probe = ToolCallingProbe( persona=persona, diff --git a/src/usersim/testing/__init__.py b/src/usersim/testing/__init__.py index 6fcf942..08db550 100644 --- a/src/usersim/testing/__init__.py +++ b/src/usersim/testing/__init__.py @@ -96,6 +96,7 @@ def probe_conformance_problems(probe: str | type) -> list[str]: BaseProbe, PreviewField, known_probes, + seed_dataset_problems, ) problems: list[str] = [] @@ -220,6 +221,8 @@ def probe_conformance_problems(probe: str | type) -> list[str]: f"installed package; got {guide!r}" ) + problems.extend(seed_dataset_problems(cls)) + for klass, key in _reserved_metadata_writes(cls): problems.append( f"{klass} writes state.metadata[{key!r}], which the conversation loop owns. " diff --git a/templates/extension_package/pyproject.toml b/templates/extension_package/pyproject.toml index 79f81e0..67bd4ea 100644 --- a/templates/extension_package/pyproject.toml +++ b/templates/extension_package/pyproject.toml @@ -25,3 +25,4 @@ dev = ["pytest", "pytest-asyncio", "pytest-env"] [tool.pytest.ini_options] env = ["USERSIM_DISABLE_EXTENSIONS=1"] asyncio_mode = "auto" +asyncio_default_fixture_loop_scope = "function" diff --git a/templates/probe/README.md b/templates/probe/README.md index d8d3bd5..e5ee786 100644 --- a/templates/probe/README.md +++ b/templates/probe/README.md @@ -164,21 +164,26 @@ When porting one of the templates into a new module: absent by your scorer, which reports the trajectory as having nothing to score while the run looks fine. See step 6 of `docs/engine/AUTHORING_A_PROBE.md`. -- [ ] Add a scorer under - `evaluator/scorers/.py` and include the module - path in `DEFAULT_SCORER_MODULES`. Register it with +- [ ] Add a scorer as `scorer.py` in the probe's package and include + its module path, `usersim.engine.probes..scorer`, in + `DEFAULT_SCORER_MODULES`. Register it with `deterministic=True` when it never calls a model, and list it in the class's `default_scorers` so `usersim eval --scorers auto` runs it. -- [ ] Write tests from `test_probe.py.template`. Its first test runs - the probe end to end with `usersim.testing.run_episode` and - scripted models. The standard tests after it control each reply: - `_patched_call_llm` patches BOTH `core.simulation.acall_llm` AND - `core.judges.acall_llm`, and the alias-dispatching mock returns - XML-shaped judge verdicts. See - [`tests/probes/sov_ai_facts/test_probe.py`](../../tests/probes/sov_ai_facts/test_probe.py) - for a real reference. The tests run under `usersim.testing.pytest_plugin`, - so a model call nothing mocked fails with `BlockedNetworkCall`. +- [ ] Declare the capability rows your probe's evidence feeds in + `capabilities.py` in its package, and add the module to + `BUILTIN_CAPABILITY_MODULES` in `taxonomy/capabilities.py`; without + them the probe's rows are absent from the capability report. See + "Adding the capabilities your probe provides evidence for" in + `docs/engine/AUTHORING_A_PROBE.md`. +- [ ] Write tests from `test_probe.py.template`. Each runs the probe + through `usersim.testing.run_episode` with scripted models, with + nothing patched; a `ScriptedModel` subclass scripts the replies a + test needs. See + [`tests/engine/test_hosted_parity_contract.py`](../../tests/engine/test_hosted_parity_contract.py), + which runs every built-in this way. The tests run under + `usersim.testing.pytest_plugin`, so a model call nothing scripted + fails with `BlockedNetworkCall`. - [ ] Update `docs/engine/README.md`'s probe table and the catalog in `docs/probes.md`; `tests/test_security_defaults.py` checks the probe count there. diff --git a/templates/probe/generator_tool_calling.py b/templates/probe/generator_tool_calling.py index f74ffa3..05f20a2 100644 --- a/templates/probe/generator_tool_calling.py +++ b/templates/probe/generator_tool_calling.py @@ -20,9 +20,11 @@ ``after_assistant_turn``.) What you provide: -- ``__init__(...)`` — sample tools from ``self._data[cfg.tools_column]`` - (typically a CLI-provided tool registry), validate, store on - ``self``. +- ``seed_dataset`` — the name of the tool table in your asset folder + (``.parquet`` or ``.jsonl``); a run gives each row the + columns of one of its rows, and ``--toolset-seed-path`` swaps in another. +- ``__init__(...)`` — sample tools from the row's ``tools`` column, + validate, store on ``self``. - ``get_user_system_prompt`` — typical persona-grounded user prompt. - ``get_tools_for_assistant`` — returns the sampled OpenAI tool specs. - ``after_assistant_turn`` — intercepts tool calls, executes them @@ -58,15 +60,17 @@ class DemoToolCallingProbe(ToolCallingMixin, BaseProbe): """Tool-calling capability probe with mid-loop tool execution.""" label = "demo_tool_calling" + # assets/demo_tool_calling/toolsets.parquet (or .jsonl): one row per + # toolset, with a ``tools`` column of OpenAI-style function specs. + seed_dataset = "toolsets" def __init__(self, **kwargs: Any) -> None: super().__init__(**kwargs) - cfg = self._cfg - if not getattr(cfg, "tools_column", None): - raise ValueError("demo_tool_calling: cfg.tools_column not configured") - # Real probe: parse self._data[cfg.tools_column], validate, - # sample subset. See tool_calling/probe.py for the - # canonical tool-sampling logic. + if "tools" not in self._data: + raise ValueError("demo_tool_calling: the row has no tools column; is the seed table attached?") + # Real probe: parse self._data["tools"], validate, sample a + # subset. See tool_calling/probe.py for the canonical + # tool-sampling logic. self._tools: list[dict] = [] # populated from row def get_user_system_prompt(self) -> str: diff --git a/templates/probe/test_probe.py.template b/templates/probe/test_probe.py.template index 14b806e..9a3b704 100644 --- a/templates/probe/test_probe.py.template +++ b/templates/probe/test_probe.py.template @@ -4,286 +4,104 @@ """Test scaffold for a new probe. Copy this to ``tests/probes//test_probe.py``, beside an -``__init__.py``, and edit the marked sections. The first test runs the probe end to end -with ``usersim.testing.run_episode`` and scripted models, with nothing -patched; the 8 standard tests every probe should have are stubbed -after it. For real working examples, see: - -- ``tests/probes/sov_ai_facts/test_probe.py`` — banked-verbatim probe (the canonical - reference for the alias-dispatching mock + dual-module patching - pattern). -- ``tests/probes/safety_chat_pressure/test_probe.py`` — bank-reframing probe - (``on_followup_failure="abort"`` test case). -- ``tests/probes/safety_agentic/test_probe.py`` — agentic probe (custom ``run_dispatch``, - no user-LLM, simpler mock pattern — FIFO-only since there's only - one acall_llm caller). - -CRITICAL gotchas for the standard tests, which patch ``acall_llm`` to -control each reply: - -1. **Patch BOTH ``core.simulation.acall_llm`` AND - ``core.judges.acall_llm``** in the same context manager. Both - modules locally import ``acall_llm`` from ``core.llm`` at module - load, so patching ``core.llm.acall_llm`` doesn't reach either. - The ``_patched_call_llm`` helper below patches both at once. - The target is a coroutine function, so ``patch`` installs an - ``AsyncMock``; a plain ``def`` side_effect still works, and its - return value is what the ``await`` yields. - -2. **Mock returns must match the caller's expected shape.** For - judge calls (``judge_model``), return a string with - ``success``, ``warn`` or ``failure`` XML; - ``run_inline_judge`` parses XML, - not plain text. For user/assistant calls, return a dict with - ``content`` (and optionally ``tool_calls``). - -3. **Alias-dispatching mock**: don't FIFO-queue all responses - together. The loop calls user_model + judge_model + assistant_model - in an order that depends on probe shape and gate retries — a FIFO - queue makes the test brittle to internal refactors. Instead, - inspect the alias kwarg in the side_effect callable and pop from - the matching queue. +``__init__.py``, and edit the marked sections. Every episode runs through +``usersim.testing.run_episode``, the generator ``usersim simulate`` uses, with +scripted models: no provider, and nothing patched. ``ScriptedModel`` gives each +role a fixed reply, and a tool call when tools are offered; subclass it, as +``_DecliningAssistant`` does below, to script the replies a test needs. +``tests/engine/test_hosted_parity_contract.py`` runs every built-in probe this +way. + +Keep each test to what your probe guarantees, such as the row area reaching +the user model's instructions, the columns ``build_result_extras`` writes, a +placeholder warning, or a row that cannot run failing with a reason. """ from __future__ import annotations import json -import os -from contextlib import contextmanager -from pathlib import Path -from typing import Any, Dict, List -from unittest.mock import patch -import pytest +from usersim.extension import ConversationSimulatorConfig +from usersim.testing import ScriptedModel, assert_hosted_parity, assert_probe_conforms, run_episode -import usersim.engine.generator # noqa: F401 — bootstraps registry -from usersim.extension import ( - ConversationSimulatorConfig, ConversationState, FailureClass, - OutcomeBuilder, OutcomeStatus, Provenance, WarningKind, -) -from usersim.testing import assert_probe_conforms, run_episode - -# Replace with your probe's module + names. -from usersim.engine.probes.YOUR_PROBE import ( - probe as probe_gen, -) +# Replace with your probe's module. +from usersim.engine.probes.YOUR_PROBE import probe PROBE_LABEL = "YOUR_PROBE" +PERSONA = { + "first_name": "Ana", + "last_name": "Souza", + "age": 35, + "city": "Recife", + "education_level": "Bachelor", + "occupation": "Teacher", +} +CONFIG = ConversationSimulatorConfig(name="conversation_messages", max_turns=2, random_seed=7, locale="pt_BR") +# The row inputs your probe reads, such as its theme. A probe that draws its +# area from a bank needs none. +DATA = {"theme": {"type": "your area", "description": "What the user wants to get done."}} -# ─── Shared helpers ──────────────────────────────────────────────── - - -def _mock_call_llm( - *, - user_responses: List[Any] | None = None, - assistant_responses: List[Any] | None = None, - judge_responses: List[str] | None = None, -): - """Alias-dispatching mock for acall_llm. - - Returns a side_effect callable that pops from the right queue - based on the alias kwarg. Resilient to gate retries reordering - user/judge calls within a single user turn. - Judge responses MUST be XML-shaped strings: - '...success' - or, to fail the gate: - '...failure' +class _DecliningAssistant(ScriptedModel): + """The scripted models, with an assistant that declines.""" - See run_inline_judge in core/judges.py for the parser. - """ - user_q = list(user_responses or []) - asst_q = list(assistant_responses or []) - judge_q = list(judge_responses or []) + CONTENT = {**ScriptedModel.CONTENT, "assistant": "Sorry, I can't help with that."} - def _side_effect(*args, **kwargs): - # acall_llm signature: (models, alias, messages, **kwargs) - alias = args[1] if len(args) >= 2 else kwargs.get("alias", "") - if alias == "user_model": - queue = user_q - elif alias == "assistant_model": - queue = asst_q - elif alias == "judge_model": - queue = judge_q - else: - raise AssertionError( - f"unexpected acall_llm alias={alias!r}; queues: " - f"user={len(user_q)} assistant={len(asst_q)} " - f"judge={len(judge_q)}" - ) - if not queue: - raise AssertionError( - f"test consumed more {alias!r} responses than expected; " - f"args_head={args[:2]} kwargs_keys={list(kwargs)}" - ) - return queue.pop(0) - return _side_effect +def _messages(row: dict) -> list[dict]: + return json.loads(row["conversation_messages"]) -@contextmanager -def _patched_call_llm(side_effect): - """Patch acall_llm in BOTH core.simulation AND core.judges. - - Both modules locally re-import acall_llm from core.llm at module - load, so patching core.llm.acall_llm doesn't propagate. This - context manager patches both call sites at once. - """ - with patch( - "usersim.engine.core.simulation.acall_llm", - side_effect=side_effect, - ), patch( - "usersim.engine.core.judges.acall_llm", - side_effect=side_effect, - ): - yield - - -# ─── Fixtures ────────────────────────────────────────────────────── - - -@pytest.fixture -def en_persona() -> Dict[str, Any]: - return { - "first_name": "Sarah", "last_name": "Johnson", - "age": 35, "city": "Seattle", - "education_level": "Bachelor", - "occupation": "Software Engineer", - } - - -@pytest.fixture -def simulator_cfg(): - class _Cfg: - random_seed = 42 - max_turns = 3 - # ... add probe-specific cfg fields here - return _Cfg() +class TestYourProbe: + def test_registered_and_conforms(self) -> None: + """Registered with what @register_probe sets, and conforming to the probe contract.""" + assert_probe_conforms(PROBE_LABEL) + assert probe.YourProbe.family == PROBE_LABEL # rename to your class + async def test_runs_end_to_end(self) -> None: + run = await run_episode(PROBE_LABEL, config=CONFIG, persona=PERSONA, data=DATA, language="Portuguese") -@pytest.fixture -def models() -> Dict[str, Any]: - return { - "user_model": object(), - "assistant_model": object(), - "judge_model": object(), - } + # True only when the probe completed its protocol: should_succeed holds. + assert run.row["conversation_status"] is True + assert sum(message["role"] == "user" for message in _messages(run.row)) == CONFIG.max_turns + async def test_the_user_explores_the_row_area_in_their_own_words(self) -> None: + run = await run_episode(PROBE_LABEL, config=CONFIG, persona=PERSONA, data=DATA, language="Portuguese") -# ─── The 8 standard tests every probe should have ──────────────── + instructions = run.prompts[0]["messages"][0]["content"] + assert DATA["theme"]["description"] in instructions + # The user model writes turn 1. A probe whose exact wording is under + # test sends its bank's text instead; assert that text here. + assert _messages(run.row)[0] == {"role": "user", "content": ScriptedModel.CONTENT["user"]} + async def test_the_row_records_what_the_probe_writes(self) -> None: + run = await run_episode(PROBE_LABEL, config=CONFIG, persona=PERSONA, data=DATA, language="Portuguese") -class TestSimulateYourProbe: + # Each column in your class's output_columns, and probe_variant when + # the probe sets its own. + assert run.row["YOUR_JOIN_KEY"] # rename to a column your probe writes - # 0. End to end with scripted models: no provider, nothing patched. - async def test_runs_end_to_end_with_scripted_models(self, en_persona) -> None: + async def test_a_declining_assistant_is_still_a_scoreable_row(self) -> None: run = await run_episode( PROBE_LABEL, - config=ConversationSimulatorConfig(name="conversation_messages", max_turns=2), - persona=en_persona, - # The row inputs your probe reads, such as its theme. - data={"theme": {"type": "your topic", "description": "What the user wants."}}, - ) - assert run.row["conversation_status"] is True - - # 1. Registered, with the family, prompt version and variants - # @register_probe sets, and conforming to the probe contract. - def test_registered_and_conforms(self) -> None: - assert_probe_conforms(PROBE_LABEL) - assert probe_gen.YourProbe.family == PROBE_LABEL # rename to your class - - # 2. Single-turn end-to-end (verbatim or generate-and-gate). - async def test_single_turn_end_to_end( - self, en_persona, simulator_cfg, models, - ) -> None: - side_effect = _mock_call_llm( - assistant_responses=[ - {"role": "assistant", "content": "OK, here's an answer."}, - ], - # If verbatim turn-1: no user_responses needed. - # If generate-and-gate turn-1: provide a user response + judge XML. - user_responses=[], - judge_responses=[], + config=CONFIG, + persona=PERSONA, + data=DATA, + language="Portuguese", + model_factory=_DecliningAssistant, ) - with _patched_call_llm(side_effect): - result = await probe_gen.simulate_YOUR_PROBE( # rename to your shim - models=models, data={}, persona=en_persona, - profile={}, locale="en_US", language="English", - cfg=simulator_cfg, provenance=Provenance(), - ) - assert result["conversation_status"] is True - # 3. First user turn shape (verbatim probes only). - def test_first_user_turn_is_bank_text_verbatim( - self, en_persona, simulator_cfg, models, - ) -> None: - # For verbatim probes: assert msgs[0]["content"] == bank.task.text - # For generate-and-gate probes: skip / replace with a derive-task test. - pass - - # 4. Two-turn follow-up. - def test_two_turn_followup( - self, en_persona, simulator_cfg, models, - ) -> None: - # Provide 2 assistant responses; the loop drives turn 2 via - # the user-agent (user_responses needed) + judge gate - # (judge_responses needed with success XML). - pass - - # 5. probe_variant overridden in build_result_extras. - def test_probe_variant_overridden( - self, en_persona, simulator_cfg, models, - ) -> None: - # Assert result["probe_variant"] is the probe-specific value - # (e.g., the fact category, the strategy::harm_category combo) - # not the registry default "default". - pass - - # 6. Placeholder warning emitted. - def test_placeholder_warning_emitted( - self, en_persona, simulator_cfg, models, - ) -> None: - # Bank with placeholder=True → warning fires. - # Assert WarningKind.USED_PLACEHOLDER_* in outcome.warnings. - pass - - # 7. Bank-load failure aborts cleanly. - async def test_bank_load_failure_aborts( - self, en_persona, simulator_cfg, models, - ) -> None: - with patch.dict( - os.environ, - {"USERSIM_YOUR_BANK": "/nonexistent/bank.yaml"}, - ): - result = await probe_gen.simulate_YOUR_PROBE( - models=models, data={}, persona=en_persona, - profile={}, locale="en_US", language="English", - cfg=simulator_cfg, provenance=Provenance(), - ) - assert result["conversation_status"] is False - outcome = json.loads(result["simulation_outcome"]) - assert outcome["failure_class"] == FailureClass.SCENARIO_ABORTED.value - - # 8. No-matching-task aborts cleanly. - def test_no_matching_task_aborts( - self, en_persona, simulator_cfg, models, - ) -> None: - # Synthetic bank with persona-tag mismatch → derive_task returns None - # → __init__ raises YourProbeError → shim returns SCENARIO_ABORTED. - pass - - -# ─── should_succeed invariant (probe-specific contract) ──────────── + assert run.row["conversation_status"] is True + assert _messages(run.row)[1]["content"] == _DecliningAssistant.CONTENT["assistant"] + async def test_a_placeholder_task_records_its_warning(self) -> None: + # For a bank-backed probe: a task marked placeholder records the + # warning your class's placeholder_warning_kind names. + run = await run_episode(PROBE_LABEL, config=CONFIG, persona=PERSONA, data=DATA, language="Portuguese") -class TestShouldSucceed: - def test_returns_true_when_required_metadata_set(self) -> None: - state = ConversationState(outcome=OutcomeBuilder()) - # Set the probe's join key. - state.metadata["YOUR_REQUIRED_KEY"] = "id-001" - assert probe_gen.should_succeed(state) is True + kinds = [warning["kind"] for warning in json.loads(run.row["simulation_outcome"])["warnings"]] + assert "YOUR_PREFIX_used_placeholder_YOUR_THING" in kinds # or drop the test for a probe without one - def test_returns_false_when_required_metadata_missing(self) -> None: - state = ConversationState(outcome=OutcomeBuilder()) - assert probe_gen.should_succeed(state) is False + async def test_a_host_reproduces_a_standalone_run(self) -> None: + await assert_hosted_parity(PROBE_LABEL, config=CONFIG, persona=PERSONA, data=DATA, language="Portuguese") diff --git a/tests/_asset_audit.py b/tests/_asset_audit.py index 6606451..1a2fe77 100644 --- a/tests/_asset_audit.py +++ b/tests/_asset_audit.py @@ -29,24 +29,13 @@ import yaml from usersim.engine.core._assets import probe_assets_dir +from usersim.engine.core.locale import SHIPPED_LOCALES from usersim.engine.core.probes import known_probes YAML_LOADER = getattr(yaml, "CSafeLoader", yaml.SafeLoader) -# All Nemotron-Personas locale ids the simulator currently supports. -SUPPORTED_LOCALES: tuple[str, ...] = ( - "en_US", - "en_IN", - "en_SG", - "en_GB", - "pt_BR", - "fr_FR", - "de_DE", - "ja_JP", - "hi_Deva_IN", - "hi_Latn_IN", - "ko_KR", -) +#: The shipped locales; a mapping keyed by them is a locale-keyed rendering. +SUPPORTED_LOCALES: tuple[str, ...] = SHIPPED_LOCALES # Regex patterns that indicate template-fill / placeholder text masquerading # as bank content. Each entry is ``(name, pattern)``; the pattern is matched diff --git a/tests/engine/core/test_episode_input.py b/tests/engine/core/test_episode_input.py index 52141a1..15a8007 100644 --- a/tests/engine/core/test_episode_input.py +++ b/tests/engine/core/test_episode_input.py @@ -143,6 +143,14 @@ def test_a_general_probe_draws_its_theme_from_its_seeds(probe_type: str, kind: s assert json.loads(row["theme"]) in load_seeds("en_US", probe_type, kind) +def test_a_tool_calling_row_stores_no_drawn_theme() -> None: + """A tool_calling row's theme is its toolset, so the run draws none and the row holds the placeholder.""" + (row,) = materialize_episode_inputs(locale="en_US", num_rows=1, probe_mix={"tool_calling": 1.0}, random_seed=5) + + assert row["theme"] == "__not_used__" + assert row["toolset_category"] + + def _themes_from_its_folder(cls: type, locale: str) -> list[dict]: return json.loads((cls.assets_dir() / locale / "themes.json").read_text(encoding="utf-8")) diff --git a/tests/engine/core/test_extensions.py b/tests/engine/core/test_extensions.py index 1bc69df..2f99e66 100644 --- a/tests/engine/core/test_extensions.py +++ b/tests/engine/core/test_extensions.py @@ -17,6 +17,7 @@ ENTRY_POINT_GROUPS, PROBES, clear_extension_cache, + extension_distributions, extension_records, extensions_disabled, load_extensions, @@ -194,3 +195,20 @@ def test_clearing_the_cache_clears_the_records(self, monkeypatch, discovery_enab load_extensions(PROBES) clear_extension_cache() assert extension_records() == () + + +class TestExtensionDistributions: + """The packages a run's manifest records as the extensions that could reach it.""" + + def test_none_while_discovery_is_off(self, monkeypatch) -> None: + _fake_entry_points(monkeypatch, ("p", "json:dumps")) + assert extension_distributions() == {} + + def test_each_advertising_distribution_with_its_version(self, monkeypatch, discovery_enabled) -> None: + class _Dist: + name, version = "my-bookings", "0.1.0" + + ep = EntryPoint(name="my_bookings", value="json:dumps", group=PROBES) + object.__setattr__(ep, "dist", _Dist()) + monkeypatch.setattr(_extensions, "entry_points", lambda group: [ep] if group == PROBES else []) + assert extension_distributions() == {"my-bookings": "0.1.0"} diff --git a/tests/engine/core/test_manifest.py b/tests/engine/core/test_manifest.py index 6447049..cd6385b 100644 --- a/tests/engine/core/test_manifest.py +++ b/tests/engine/core/test_manifest.py @@ -158,7 +158,6 @@ def test_manifest_to_dict_and_from_dict_roundtrip(self) -> None: locales=["en_US", "ja_JP"], probes=["tool_calling", "general_open_ended"], probe_mix={"tool_calling": 0.5, "general_open_ended": 0.5}, - tool_calling_themes=["weather", "calendar"], trajectories_requested={"en_US": 2, "ja_JP": 2}, trajectories_completed={"en_US": 2, "ja_JP": 2}, ), @@ -284,6 +283,16 @@ def test_loads_what_writer_wrote(self, tmp_path: Path) -> None: loaded = load_run_manifest("1700000000", tmp_path) assert loaded == m + def test_loads_a_manifest_with_a_scope_key_it_no_longer_reads(self, tmp_path: Path) -> None: + """A manifest whose scope holds a key the loader does not read, such as tool_calling_themes, still loads.""" + m = _minimal_manifest("1700000000") + write_run_manifest(m, tmp_path) + path = manifest_path(tmp_path, "1700000000") + payload = json.loads(path.read_text()) + payload["scope"]["tool_calling_themes"] = ['{"type": "Weather & Location Lookup"}'] + path.write_text(json.dumps(payload)) + assert load_run_manifest("1700000000", tmp_path) == m + def test_returns_none_for_run_without_manifest(self, tmp_path: Path) -> None: # No manifest written. Old-run handling depends on this contract. assert load_run_manifest("1700000000", tmp_path) is None @@ -557,7 +566,6 @@ def test_assembles_all_blocks(self) -> None: sim_config=_StubSimConfig(random_seed=42), locales_requested=["en_US"], probe_mix={"tool_calling": 1.0}, - tool_calling_themes=("weather",), assets_dir=Path("/repo/assets"), trajectories_requested={"en_US": 1}, trajectory_df=traj, @@ -572,12 +580,48 @@ def test_assembles_all_blocks(self) -> None: assert manifest.scope.trajectories_completed == {"en_US": 1} assert manifest.outcome_summary.status_counts == {"ok": 1} assert manifest.asset_versions.prompt_versions == {"tool_calling": "v1.0"} + # The run pinned /repo/assets, so the probe read its assets there. + assert manifest.asset_versions.probe_assets == {"tool_calling": str(Path("/repo/assets") / "tool_calling")} identity = manifest.models["assistant_model"] assert identity.model == "nvidia/nemotron-3-super-v3" assert identity.endpoint == "https://gateway.example/v1" assert identity.api_key_env_var == "GATEWAY_API_KEY" assert manifest.replay == ReplayInfo() # reserved fields default null + def test_an_unpinned_run_records_each_probe_s_own_assets(self) -> None: + from usersim.engine.core._assets import probe_assets_dir + + manifest = build_run_manifest( + run_id="1700000005", + started_at_epoch=1700000005, + finished_at_epoch=1700000006, + models_config=_StubModelsConfig(), + sim_config=_StubSimConfig(), + locales_requested=["en_US"], + probe_mix={"tool_calling": 0.5, "general_open_ended": 0.5}, + trajectory_df=None, + ) + assert manifest.asset_versions.assets_dir is None + assert manifest.asset_versions.probe_assets == { + probe: str(probe_assets_dir(probe)) for probe in ("general_open_ended", "tool_calling") + } + + def test_records_the_installed_extensions(self, monkeypatch: pytest.MonkeyPatch) -> None: + from usersim.engine.core import _extensions + + monkeypatch.setattr(_extensions, "extension_distributions", lambda: {"my-bookings": "0.1.0"}) + manifest = build_run_manifest( + run_id="1700000007", + started_at_epoch=1700000007, + finished_at_epoch=1700000008, + models_config=_StubModelsConfig(), + sim_config=_StubSimConfig(), + locales_requested=["en_US"], + probe_mix={}, + trajectory_df=None, + ) + assert manifest.code.extensions == {"my-bookings": "0.1.0"} + def test_empty_trajectory_df_renders_zeros(self) -> None: manifest = build_run_manifest( run_id="1700000002", diff --git a/tests/engine/core/test_preview.py b/tests/engine/core/test_preview.py index e5a8206..77e8927 100644 --- a/tests/engine/core/test_preview.py +++ b/tests/engine/core/test_preview.py @@ -665,23 +665,22 @@ def test_safety_agentic_inputs_panel(self) -> None: assert "send_email_internal" in out assert "transfer_funds_external" in out - def test_general_family_includes_theme_in_inputs_panel(self) -> None: - # tool_calling / general_open_ended / general_educational don't carry - # curated-asset side channels but they ARE seeded by a - # ``theme`` (sampled from the locale's seed YAML or a - # toolset bundle). The Probe Inputs panel surfaces theme - # + toolset metadata for the general family so readers - # can answer "what was this probe seeded with?" without - # cross-referencing the seed parquet. + def test_tool_calling_inputs_panel_shows_its_toolset(self) -> None: + # A tool_calling row is seeded by the toolset it draws, which is + # also its theme, so the Probe Inputs panel shows the toolset and + # not the run's theme placeholder: readers can answer "what was + # this probe seeded with?" without cross-referencing the catalog. row = _general_row() - row["theme"] = '{"name": "weather"}' + row["theme"] = "__not_used__" row["toolset_name"] = "weather_tools" row["toolset_category"] = "weather" row["num_tools"] = 3 out = _record([row]) assert "Probe Inputs" in out - assert "Theme" in out + assert "Toolset category" in out assert "weather_tools" in out + assert "Theme" not in out + assert "__not_used__" not in out def test_multilingual_consistency_inputs_panel(self) -> None: row = _general_row() diff --git a/tests/fixtures/extension_package/pyproject.toml b/tests/fixtures/extension_package/pyproject.toml index f3e92ee..63ba43d 100644 --- a/tests/fixtures/extension_package/pyproject.toml +++ b/tests/fixtures/extension_package/pyproject.toml @@ -33,4 +33,4 @@ fixture-cmd = "usersim_extension_fixture.command:register" include = ["usersim_extension_fixture*"] [tool.setuptools.package-data] -usersim_extension_fixture = ["assets/**/*.yaml"] +usersim_extension_fixture = ["assets/**/*.yaml", "assets/**/*.jsonl"] diff --git a/tests/fixtures/extension_package/usersim_extension_fixture/assets/fixture_probe/fixture_seed.jsonl b/tests/fixtures/extension_package/usersim_extension_fixture/assets/fixture_probe/fixture_seed.jsonl new file mode 100644 index 0000000..fd297d3 --- /dev/null +++ b/tests/fixtures/extension_package/usersim_extension_fixture/assets/fixture_probe/fixture_seed.jsonl @@ -0,0 +1 @@ +{"fixture_seed_value": "seeded"} diff --git a/tests/fixtures/extension_package/usersim_extension_fixture/probe.py b/tests/fixtures/extension_package/usersim_extension_fixture/probe.py index e4f472a..85553c9 100644 --- a/tests/fixtures/extension_package/usersim_extension_fixture/probe.py +++ b/tests/fixtures/extension_package/usersim_extension_fixture/probe.py @@ -22,6 +22,7 @@ class FixtureProbe(BaseProbe): default_scorers = ("fixture_scorer",) locale_bank_check = staticmethod(_check_locale_bank) authoring_guide = "https://example.com/fixture-probe/locales" + seed_dataset = "fixture_seed" @classmethod def model_aliases(cls, cfg) -> tuple[str, ...]: diff --git a/tests/fixtures/extension_package/verify_seams.py b/tests/fixtures/extension_package/verify_seams.py index e7731a0..de7f712 100644 --- a/tests/fixtures/extension_package/verify_seams.py +++ b/tests/fixtures/extension_package/verify_seams.py @@ -138,6 +138,17 @@ def preflight_refusal(locales: list[str]) -> str: str(themes), ) +# A contributed probe's seed table resolves from its installed assets, as a run +# that holds the probe attaches it. +from usersim.cli._pipeline import resolve_seed_dataset_for_mix # noqa: E402 + +seed = resolve_seed_dataset_for_mix({"fixture_probe": 1.0}) +check( + "seed_dataset resolves a contributed seed table", + seed is not None and seed.name == "fixture_seed.jsonl" and "usersim_extension_fixture" in seed.parts, + str(seed), +) + # A model a contributed probe declares joins the models every row resolves. from usersim.engine.core.probes import declared_model_aliases # noqa: E402 diff --git a/tests/probes/health_disclosure/test_bank.py b/tests/probes/health_disclosure/test_bank.py index 87b76ea..3ac6ae2 100644 --- a/tests/probes/health_disclosure/test_bank.py +++ b/tests/probes/health_disclosure/test_bank.py @@ -149,3 +149,29 @@ def test_loader_caches_by_client(self, tmp_path, monkeypatch): first = load_clinical_profile_bank_for_client("health_therapy_disclosure", "MY_CUSTOM_PROFILES") second = load_clinical_profile_bank_for_client("health_therapy_disclosure", "MY_CUSTOM_PROFILES") assert first is second # cached instance + + +class TestAssetCheck: + """Each client's asset check loads the bank its rows draw from, where a run reads it.""" + + @pytest.mark.parametrize( + "label", + [ + "health_decision_support_disclosure", + "health_general_disclosure", + "health_therapy_disclosure", + "health_triage_disclosure", + ], + ) + def test_the_shipped_bank_passes(self, label: str) -> None: + from usersim.engine.core.probes import resolve_probe + + assert resolve_probe(label).asset_problems() == [] + + def test_a_customer_bank_that_does_not_load_is_reported(self, monkeypatch, tmp_path: Path) -> None: + from usersim.engine.core.probes import resolve_probe + + cls = resolve_probe("health_triage_disclosure") + monkeypatch.setenv(cls.CLIENT["profiles_env"], str(tmp_path / "missing.yaml")) + (problem,) = cls.asset_problems() + assert "file not found" in problem and "missing.yaml" in problem diff --git a/tests/probes/tool_calling/test_probe.py b/tests/probes/tool_calling/test_probe.py index 1dd9c51..20ce24f 100644 --- a/tests/probes/tool_calling/test_probe.py +++ b/tests/probes/tool_calling/test_probe.py @@ -170,13 +170,7 @@ async def _run_concrete_tool_probe( persona = {"first_name": "A", "last_name": "User", "age": 35} data = { "tools": sample_tools, - "theme": json.dumps( - { - "type": "Weather & Location Lookup", - "description": "Look up weather.", - "tool_expected": True, - } - ), + "theme": "__not_used__", "persona_uuid": "persona-1", "disclosure_style": "upfront", "user_interaction_style": "neutral", @@ -335,7 +329,7 @@ def _subset(**data_overrides: object) -> list[str]: persona = {"first_name": "A", "last_name": "User", "age": 35} data = { "tools": [_tool_entry(i) for i in range(12)], - "theme": json.dumps({"type": "T", "description": "d", "tool_expected": True}), + "theme": "__not_used__", "persona_uuid": "persona-1", "trajectory_id": "traj-1", "disclosure_style": "upfront", @@ -376,6 +370,82 @@ def test_separate_rows_draw_separately(self) -> None: assert a != b, f"two rows drew the same subset {a}, so the draw is not derived from the row" +class TestThemeFollowsTheToolset: + """A row's theme is its toolset, so the probe lists no themes for a run to draw.""" + + @staticmethod + def _user_prompt(**data_overrides: object) -> str: + from usersim.engine.core.behavioral import compute_behavioral_profile + + persona = {"first_name": "A", "last_name": "User", "age": 35} + data = { + "tools": [_tool_entry(i) for i in range(3)], + "theme": "__not_used__", + "persona_uuid": "persona-1", + "trajectory_id": "traj-1", + "disclosure_style": "upfront", + "user_interaction_style": "neutral", + } + data.update(data_overrides) + cfg = types.SimpleNamespace( + tools_column="tools", + theme_column="theme", + max_tools=3, + max_turns=2, + max_query_attempts=1, + context_compression=False, + compression_window=1, + enforce_user_language=False, + ) + probe = ToolCallingProbe( + persona=persona, + locale="en_US", + language="English", + models={}, + cfg=cfg, + provenance=Provenance(), + profile=compute_behavioral_profile(persona), + data=data, + outcome_builder=OutcomeBuilder(provenance=Provenance()), + ) + return probe.get_user_system_prompt() + + def test_the_probe_lists_no_themes(self) -> None: + assert ToolCallingProbe.themes("en_US") is None + + def test_a_row_takes_its_toolsets_category_and_name(self) -> None: + prompt = self._user_prompt(toolset_category="Weather", toolset_name="openweathermap") + + assert '"type": "Weather"' in prompt + assert "a task that involves openweathermap capabilities" in prompt + + def test_a_toolset_without_a_name_gives_its_category(self) -> None: + prompt = self._user_prompt(toolset_category="Weather", toolset_name=None) + + assert "a task that involves Weather capabilities" in prompt + + def test_a_catalog_without_categories_gives_the_toolsets_name(self) -> None: + prompt = self._user_prompt(toolset_name="openweathermap") + + assert '"type": "openweathermap"' in prompt + assert "a task that involves openweathermap capabilities" in prompt + + def test_a_row_naming_no_toolset_is_about_the_offered_tools(self) -> None: + prompt = self._user_prompt() + + assert '"type": "Tool use"' in prompt + assert "a task the offered tools support" in prompt + + def test_the_rows_theme_column_never_reaches_the_prompt(self) -> None: + prompt = self._user_prompt( + toolset_category="Weather", + theme=json.dumps({"type": "Calendar & Scheduling", "description": "Book a meeting."}), + ) + + assert "Calendar" not in prompt + assert "Book a meeting" not in prompt + + class TestCredentials: """The user holds the key a tool declares, and only a call carrying it reaches the simulated service.""" diff --git a/tests/test_cli.py b/tests/test_cli.py index 7c30cda..b78de53 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -1232,10 +1232,11 @@ def test_missing_toolset_seed_raises_config_error(self, tmp_path): """The real path: an explicit ``--toolset-seed-path`` that is not there is a user error, not a crash.""" from usersim.cli._errors import ConfigError - from usersim.cli._pipeline import resolve_toolset_seed_path + from usersim.cli._pipeline import resolve_seed_dataset_path - with pytest.raises(ConfigError, match="toolset seed not found"): - resolve_toolset_seed_path( + with pytest.raises(ConfigError, match="seed table not found"): + resolve_seed_dataset_path( + "tool_calling", toolset_seed_path=tmp_path / "absent.parquet", ) diff --git a/tests/test_extension_api.py b/tests/test_extension_api.py index b0d2825..24e656f 100644 --- a/tests/test_extension_api.py +++ b/tests/test_extension_api.py @@ -63,9 +63,6 @@ ("usersim.engine.probes.my_probe.bank", "load_my_bank_default"), ("usersim.engine.templates.probe.prompts", "DEMO_GATE_PROMPT_TEMPLATE"), ("usersim.engine.templates.probe.prompts", "DEMO_USER_AGENT_PROMPTS"), - # The test template's standard tests, which load the probe registry before - # patching the loop's model calls. - ("usersim.engine.generator", None), # The fixture's probe that layers a spec over a shipped probe's. ("usersim.engine.probes.identity_disclosure.probe", "IdentityDisclosureProbe"), } diff --git a/tests/test_probe_names_in_shared_code.py b/tests/test_probe_names_in_shared_code.py index bcaf21c..83f7fa0 100644 --- a/tests/test_probe_names_in_shared_code.py +++ b/tests/test_probe_names_in_shared_code.py @@ -28,7 +28,7 @@ class instead (``architecture/probes.md``, "What a probe declares"). #: How many built-in probe labels each file outside ``engine/probes/`` may #: write as strings, by path under ``src/usersim/``. _ALLOWED: dict[str, int] = { - "cli/_pipeline.py": 7, + "cli/_pipeline.py": 4, "cli/smoke.py": 2, "engine/core/preview.py": 3, "engine/evaluator/axes.py": 6, diff --git a/tests/test_simulator_pipeline_e2e.py b/tests/test_simulator_pipeline_e2e.py index d7a2d7d..e252fdd 100644 --- a/tests/test_simulator_pipeline_e2e.py +++ b/tests/test_simulator_pipeline_e2e.py @@ -4,21 +4,19 @@ """End-to-end wiring test: every shipped probe survives construction + single-turn dispatch when driven through ``cli/_pipeline.build_simulator_config_builder``. -This is the test that would have caught the -``cfg.tools_column not configured`` regression where every -``tool_calling`` trajectory failed at probe construction. The -existing per-probe tests (``tests/probes/test_*.py``) all use -synthetic ``ConversationSimulatorConfig`` fixtures that hard-code the -correct ``tools_column`` / ``persona_column`` / etc. values; they -therefore never exercise the wiring chain +A probe whose construction needs a setting or a column the run's wiring +does not provide fails here, at PR time, rather than on every row of a +live run. The per-probe tests (``tests/probes/