feat(engine): declare seed tables and close out the extension work

README.md

@@ -294,6 +294,8 @@ full plan and exits before the first network call.
 
 **Extending it**
 
+- [`docs/extensions.md`](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/extensions.md): a probe and a scorer in a package of your own, installed next to `usersim`
+- [`docs/extension_api.md`](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/extension_api.md): every name an extension imports from `usersim`
 - [`docs/engine/AUTHORING_A_PROBE.md`](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/engine/AUTHORING_A_PROBE.md): end-to-end tutorial for a new probe
 - [`docs/engine/README.md`](https://github.com/NVIDIA-NeMo/UserSim/blob/main/docs/engine/README.md): the substrate's full API surface
 - [`templates/probe/`](https://github.com/NVIDIA-NeMo/UserSim/tree/main/templates/probe/): copy-and-edit starters, one per probe shape

architecture/probes.md

@@ -135,6 +135,7 @@ than into a branch on its label.
 | `authoring_guide` | `None` | the pre-flight, when a run asks for a locale the probe has no bank for |
 | `asset_problems` | no problems | `usersim smoke` and `assert_probe_conforms` |
 | `themes` | `None`: no theme | run setup, which draws one per row into the `theme` column |
+| `seed_dataset` | `None`: no seed table | run setup, which attaches `<seed_dataset>.parquet` or `.jsonl` from the probe's asset folder and gives each row the columns of one of its rows; a mix with two such probes is refused; smoke and `assert_probe_conforms` check the table is there |
 | `model_aliases` | `()` | run setup, which adds each alias the models file defines to every row's models |
 | `output_columns` | `()` | the run's `side_effect_columns`; a column left out is dropped |
 | `supports_assistant_resampling` | `True`; `False` with the tool mixins | judge-gated assistant resampling (`max_assistant_attempts > 1`) |
@@ -186,5 +187,5 @@ finds. See [assets.md](assets.md).
 A probe whose conversations vary by topic lists its themes in a `themes`
 classmethod, read under its asset folder. A run draws one per row into the
 `theme` column, and the probe reads it back with `theme()`. A probe that lists
-none gets a placeholder it never reads. `tool_calling` takes its themes and
-toolsets from the run's settings instead.
+none gets a placeholder it never reads. `tool_calling` lists none: a row's
+theme is the toolset it draws from its seed table.

docs/engine/AUTHORING_A_PROBE.md

@@ -1029,8 +1029,8 @@ transcript positions.
 ### File 8: tests at `tests/probes/bookings/test_probe.py`
 
 Copy [`templates/probe/test_probe.py.template`](../../templates/probe/test_probe.py.template)
-there, beside an `__init__.py`, substitute `bookings` for `YOUR_PROBE`, fill in the 8 standard
-test stubs.
+there, beside an `__init__.py`, substitute `bookings` for `YOUR_PROBE`, and fill in the
+marked parts: the probe's class, a column it writes and its placeholder warning.
 
 ### Run it
 
@@ -1094,10 +1094,10 @@ lists every request the models received, in order, with its role and messages.
 To script other replies, subclass `ScriptedModel`, override `CONTENT`, and pass
 the class as `model_factory`.
 
-A test that controls each reply turn by turn can patch `acall_llm` instead. Two
-modules import it, `core.simulation` and `core.judges`, so patch both, as the
-template's `_patched_call_llm` does. Those are internal modules, so a probe
-shipped in its own package tests with `run_episode`.
+Some built-in probes' tests patch `acall_llm` in `core.simulation` and
+`core.judges`, the two modules that call it, to control each reply turn by
+turn. Those are internal modules, so a new probe's tests script replies with a
+`ScriptedModel` subclass instead, in tree or in its own package.
 
 ### 3. Judge replies carry an XML verdict
 
@@ -1198,44 +1198,31 @@ def test_my_probe_conforms():
 ## Test authoring
 
 [`templates/probe/test_probe.py.template`](../../templates/probe/test_probe.py.template)
-is the canonical scaffold. Copy it and substitute your probe's name. Its
-first test runs the probe end to end with `run_episode` (gotcha 2); then fill
-in the 8 standard test stubs:
-
-1. **Registry registration**: the probe label is in `_PROBE_REGISTRY` and
-   resolves to a class whose `family`, `prompt_version` and `variants` are
-   set.
-2. **Single-turn end-to-end**: minimal mocked-LLM run produces a
-   trajectory with `conversation_status=True`.
-3. **First user turn shape** (verbatim probes only):
-   `msgs[0]["content"] == bank.<your_task>.<verbatim_field>` exactly.
-   Conversation messages are dicts, so index them rather than using
-   attribute access.
-4. **Two-turn follow-up**: provide 2 assistant responses, drive
-   turn 2 via the user-agent + judge gate.
-5. **`probe_variant` overridden**: assert `result["probe_variant"]`
-   is the probe-specific value, not the registry default `"default"`.
-6. **Placeholder warning emitted**: bank with `placeholder=True`
-   triggers the warning kind on the outcome.
-7. **Bank-load failure aborts cleanly**: env override pointed at a
-   nonexistent file produces `SCENARIO_ABORTED`.
-8. **No-matching-task aborts cleanly**: synthetic bank with
-   persona-tag mismatch produces `SCENARIO_ABORTED` without any LLM
-   calls.
-
-Plus the **`should_succeed` invariant** test: the canonical pattern
-is two assertions: empty-state returns False, populated-metadata
-state returns True.
-
-The test_probe_invariants.py file at the cross-probe level already
-covers (1) for you parametrically. So your per-probe file can focus
-on (2)-(8) plus the should_succeed contract: the cross-cutting
-checks come for free.
-
-For real working examples: `tests/probes/sov_ai_facts/test_probe.py`
-(verbatim), `tests/probes/safety_chat_pressure/test_probe.py` (reframings and
-abort policy), `tests/probes/safety_agentic/test_probe.py` (custom
-run_dispatch and a simpler mock).
+is the canonical scaffold. Copy it and substitute your probe's name. Every
+episode in it runs through `run_episode` with scripted models (gotcha 2), and
+each test checks one thing the probe guarantees:
+
+1. **Registered and conforming**: `assert_probe_conforms` passes, and the
+   class's `family` is its label.
+2. **End to end**: the row completes, so `should_succeed` holds, after as many
+   user turns as `max_turns`.
+3. **The user explores the row's area**: the user model's instructions hold
+   the row's area, and turn 1 is what the user model wrote. A probe whose
+   exact wording is under test asserts its bank's text instead; conversation
+   messages are dicts, so index them.
+4. **What the row records**: each column `build_result_extras` writes, and
+   `probe_variant` when the probe sets its own.
+5. **A declining assistant**: a `ScriptedModel` subclass scripts the reply, and
+   the row is still scoreable.
+6. **The placeholder warning**: a task marked placeholder records the kind the
+   class's `placeholder_warning_kind` names.
+7. **Hosted parity**: `assert_hosted_parity` runs the episode alone and through
+   a host, and the two match.
+
+Add a test for each further guarantee, such as a row that cannot run failing
+with a reason. `tests/engine/probes/test_probe_invariants.py` already checks
+what every probe shares, and `tests/engine/test_hosted_parity_contract.py` runs
+every built-in through `run_episode`.
 
 Every test under `tests/` runs inside the hermetic harness that
 `usersim.testing.pytest_plugin` provides, which the root `conftest.py` loads.

docs/engine/README.md

@@ -454,10 +454,10 @@ diagnostics in the simulator-health report quantify that for a given run, and
 | `persona_column` | str | `"persona"` | Column containing persona dict |
 | `probe_type_column` | str | `"probe_type"` | Column carrying the probe label that the dispatcher resolves through `_PROBE_REGISTRY` |
 | `theme_column` | str | `"theme"` | Column with theme/topic |
-| `tools_column` | str? | `None` | Tool definitions (tool_calling only) |
-| `toolset_name_column` | str? | `None` | Toolset name (tool_calling only) |
+| `tools_column` | str? | `None` | The column `tool_calling` reads its tools from; `None` reads `tools`, the column of its seed table |
+| `toolset_name_column` | str? | `None` | Read by no probe; a stored config may set it |
 | `locale` | str | `"en_US"` | Locale for conversation language |
-| `assets_dir` | str? | `None` | Optional override for the asset-bank root (defaults to repo `assets/`); per-probe env vars also work |
+| `assets_dir` | str? | `None` | The asset root every probe is pinned to (`--assets-dir`); `None` reads each probe's own assets, after any `USERSIM_ASSET_PATH` or `USERSIM_ASSETS_DIR` override |
 | `max_tools` | int | `5` | Max tools sampled per row (tool_calling) |
 | `max_turns` | int | `5` | Max conversation turns |
 | `max_steps` | int | `10` | Max tool steps per turn |

docs/extensions.md

@@ -13,7 +13,9 @@ point groups and how discovery behaves,
 hook a probe can implement, and [`extension_api.md`](extension_api.md) lists
 every name the package can import from `usersim`. When a package needs a hook
 User Sim does not have,
-[Adding a hook](../CONTRIBUTING.md#adding-a-hook) says how to ask for one.
+[Adding a hook](../CONTRIBUTING.md#adding-a-hook) says how to ask for one, and
+[When the package needs a change in User Sim](#when-the-package-needs-a-change-in-user-sim)
+how to build against it before a release has it.
 
 ## Layout
 
@@ -85,6 +87,7 @@ dev = ["pytest", "pytest-asyncio", "pytest-env"]
 [tool.pytest.ini_options]
 env = ["USERSIM_DISABLE_EXTENSIONS=1"]
 asyncio_mode = "auto"
+asyncio_default_fixture_loop_scope = "function"
 ```
 
 - **`dependencies`** names a range of User Sim releases: `X.Y` is the first
@@ -93,8 +96,9 @@ asyncio_mode = "auto"
 - **The entry points** name the probe class and the scorer's module.
 - **`package-data`** ships the banks in the wheel; without it, an installed
   package has no assets.
-- **The pytest settings** turn extension discovery off under test and run
-  `async def` tests without a marker (see [Tests](#tests)).
+- **The pytest settings** turn extension discovery off under test, run
+  `async def` tests without a marker, and give each async fixture its test's
+  event loop (see [Tests](#tests)).
 
 ## The probe
 
@@ -534,3 +538,32 @@ it. A new locale is a folder holding a `bookings.yaml` written in its language,
 plus an entry in `LOCALES`. `--assets-dir` pins every probe in a run to the
 folder it names, this one included, so a run that passes it needs
 `my_bookings/` there too.
+
+## When the package needs a change in User Sim
+
+Ask for the hook, attribute or fix in general terms first, through the
+[Extension hook form](https://github.com/NVIDIA-NeMo/UserSim/issues/new?template=extension-hook.yml)
+or an issue, so the change serves other packages too.
+
+To build against the change before it is released, install a User Sim
+checkout into the package's environment without recording it:
+
+```bash
+uv pip install -e ../UserSim
+```
+
+Keep it out of `pyproject.toml` and the lock file: a path dependency installs
+only on your machine. Once the change is merged, the package can depend on that
+commit of `main` until a release includes it, then on a range that starts at
+that release:
+
+```toml
+dependencies = ["usersim @ git+https://github.com/NVIDIA-NeMo/UserSim@<commit>"]
+```
+
+While you wait, work around the gap only with what `usersim.extension` offers:
+subclass or wrap a public class, or keep per-row data in `state.metadata` under
+your prefix. Mark each workaround with the issue it waits on, so it goes when
+the change lands. A copied or edited User Sim file, a change to `usersim` at
+import time, or a fork of the repository breaks with the next release and
+hides the need from everyone else.

notebooks/01_simulate.ipynb

@@ -669,7 +669,7 @@
    "source": [
     "from usersim.engine.config import ConversationSimulatorConfig\n",
     "from usersim.engine.core.manifest import write_simulator_manifest\n",
-    "from usersim.cli._pipeline import DEFAULT_TOOL_CALLING_THEMES, verify_probe_assets\n",
+    "from usersim.cli._pipeline import verify_probe_assets\n",
     "\n",
     "total_written = 0\n",
     "STARTED_AT = time.time()\n",
@@ -755,7 +755,6 @@
     "        sim_config=sim_cfg_for_manifest,\n",
     "        locales_requested=_run_locales,\n",
     "        probe_mix=PROBE_MIX,\n",
-    "        tool_calling_themes=DEFAULT_TOOL_CALLING_THEMES,\n",
     "        toolset_seed_path=TOOLSET_SEED_PATH if TOOLSET_SEED_PATH.exists() else None,\n",
     "        assets_dir=ASSETS_DIR,\n",
     "        trajectories_requested={l: NUM_ROWS_PER_LOCALE for l in _run_locales},\n",