refactor(engine)!: keep every built-in's assets in its own folder

architecture/assets.md

@@ -2,8 +2,8 @@
 
 Probe content (fact banks, query banks, pressure banks, clinical profiles,
 finance corpora, tool catalogs, action taxonomies) ships as **package
-data** under `usersim/engine/assets/<probe>/`, or, for a built-in probe kept
-in its own folder, under that folder's `assets/<probe>/`.
+data** in each built-in probe's own folder, under
+`usersim/engine/probes/<folder>/assets/<probe>/`.
 
 They ship inside the package so a plain `pip install` gives you working
 probes with no extra download step. `make check-dist` fails if the wheel
@@ -12,40 +12,37 @@ Markdown outside the banks.
 
 ## Resolution
 
-Assets resolve along an ordered **search path**, highest priority first:
+Each probe keeps its banks in the `<label>/` folder of the root its class
+declares as `assets_root`. A built-in probe declares its own folder in the
+package, `engine/probes/<folder>/assets/`. A probe shipped in its own package
+points it inside that package, such as `Path(__file__).parent / "assets"`, so
+the probe finds its banks wherever the package is installed, with no
+environment variable to set. Registration refuses a relative root. The lookup
+finds a built-in's folder before the probe's module is imported: a prompt read
+while the module loads, or a scorer in a process that never imports the probe,
+reaches the probe's assets.
+
+Ahead of a probe's own root, assets resolve along an ordered **search path**
+of deliberate overrides, highest priority first:
 
 1. the per-context override set by `set_runtime_assets_dir`
 2. each entry of `$USERSIM_ASSET_PATH` (`os.pathsep`-separated)
 3. `$USERSIM_ASSETS_DIR`
-4. the packaged `usersim/engine/assets/` directory
-
-`probe_assets_dir()` walks that path and takes the first root holding the
-probe. The packaged directory is always last, so prepending a root never
-hides the shipped banks, which is what lets a package contribute one
-probe's assets without restating the rest.
-
-A probe class can also declare `assets_root`, the directory that holds its
-`<label>/` folder. A probe shipped in its own package points it inside that
-package, such as `Path(__file__).parent / "assets"`, so the probe finds its
-banks wherever the package is installed, with no environment variable to
-set. Registration refuses a relative root. That root is searched for that
-probe only, after the first three entries and before the packaged directory,
-so a deliberate override wins over it. A built-in probe kept in its own
-folder declares one too, that folder's `assets/`, and the lookup finds that
-folder before the probe's module is imported: a prompt read while the module
-loads, or a scorer in a process that never imports the probe, reaches the
-probe's assets. A probe's bank loader reads
-under its class's `assets_dir()`, the folder this search finds for the probe,
-and its `prompts.yaml` is read from that folder too.
+
+`probe_assets_dir()` walks those roots, then the probe's own, and takes the
+first that holds the probe. An override without a probe's folder leaves that
+probe's own banks in reach, which is what lets a package contribute one
+probe's assets without restating the rest. A probe's bank loader reads under
+its class's `assets_dir()`, the folder this search finds for the probe, and
+its `prompts.yaml` is read from that folder too.
 
 An explicit `--assets-dir` is **not** part of the search. It pins resolution
 to exactly that directory, for every probe in the run, including one that
 declares `assets_root`, because a user who named a location should be told
-when the assets are not there rather than quietly served another copy.
-Pinning the packaged directory pins the assets that ship with User Sim, so it
-also reaches a built-in probe whose banks are in its own folder.
-`simulate --inputs` pins the folder its rows were stored with: the packaged
-directory, for rows stored by a run that named no folder and set no override.
+when the assets are not there rather than quietly served another copy. A run
+without it records no asset folder. `simulate --inputs` reads stored rows
+from the folder they were stored with, or from each probe's own assets when
+they name none.
 
 There is no fallback to the working directory. Resolving against wherever the
 process happened to start makes a broken install look healthy from a source

architecture/plugins.md

@@ -171,8 +171,8 @@ class MyProbe(BaseProbe):
 ```
 
 The probe then finds its banks wherever the package is installed. That root
-is searched after `$USERSIM_ASSET_PATH` and before the packaged banks, so a
-deployment can override it and the shipped banks remain reachable.
+is searched after `$USERSIM_ASSET_PATH`, so a deployment can override it, and
+an override without the probe's folder leaves its shipped banks in reach.
 `@register_probe` refuses a relative `assets_root`, which would resolve against
 the working directory. The probe's bank loader reads under `MyProbe.assets_dir()`,
 its folder on that search, so an override reaches the loader too.

architecture/probes.md

@@ -129,7 +129,7 @@ than into a branch on its label.
 | `description` | `""` | the per-row preview and the capability report, beside the probe's rows |
 | `preview_fields` | `()` | the per-row preview's Probe Inputs panel |
 | `default_scorers` | `()` | `usersim eval --scorers auto`, for a run that holds the probe's rows |
-| `assets_root` | `None`: the packaged banks | `assets_dir()`, and through it the probe's loaders, the pre-flight and smoke |
+| `assets_root` | `None`: the override roots only | `assets_dir()`, and through it the probe's loaders, the pre-flight and smoke |
 | `locale_bank_check` | `None`: the asset folder holds bank files | the pre-flight, for each run locale |
 | `bank_locale` | an India variant's own bank when it has one, else its base locale's | the pre-flight |
 | `authoring_guide` | `None` | the pre-flight, when a run asks for a locale the probe has no bank for |
@@ -167,9 +167,9 @@ the turn had no side effects.
 
 ## Assets
 
-A probe's content lives at `engine/assets/<label>/`, or, for a probe kept in
-its own folder, at `engine/probes/<folder>/assets/<label>/`, where the folder
-is named after the probe or, as `health_disclosure` is, shared by several. It
+A built-in probe's content lives in its own folder, at
+`engine/probes/<folder>/assets/<label>/`, where the folder is named after the
+probe or, as `health_disclosure` is, shared by several. It
 is resolved through `probe_assets_dir()`. A run fails pre-flight, before any
 model call, if a selected probe's bank is missing or, for the probes whose banks
 are split by locale, doesn't load for a run locale (a finance bank without

docs/engine/AUTHORING_A_PROBE.md

@@ -259,12 +259,11 @@ resolve against the working directory. In this repository the banks are then
 at `src/usersim/engine/probes/<folder>/assets/<label>/`, as for
 `general_educational`, `general_open_ended`, the three `sov_ai_*` probes,
 `safety_chat_pressure`, `safety_agentic`, `identity_disclosure`,
-`financial_services` and the four `health_*` probes, which share the
-`health_disclosure/` folder. They keep their bank loader (`bank.py`,
-`taxonomy.py` for `sov_ai_dynamic`, `spec.py` for `identity_disclosure`) and
-their scorer (`scorer.py`) in that folder too, and their tests in
-`tests/probes/<folder>/`. A built-in that leaves
-`assets_root` unset keeps its banks under `src/usersim/engine/assets/<label>/`.
+`financial_services`, `tool_calling` and the four `health_*` probes, which
+share the `health_disclosure/` folder: every built-in probe. They keep their
+bank loader (`bank.py`, `taxonomy.py` for `sov_ai_dynamic`, `spec.py` for
+`identity_disclosure`) and their scorer (`scorer.py`) in that folder too, and
+their tests in `tests/probes/<folder>/`.
 
 Its bank loader reads under `MyProbe.assets_dir()`, the probe's own folder,
 resolved as the built-in loaders resolve theirs, so a folder `--assets-dir` or

docs/engine/EXTERNAL_PROBE_RUNTIME.md

@@ -109,7 +109,8 @@ print(result.run_id, result.written, result.skipped)
   stored with. The rows in one call must share their settings apart from
   locale, asset folder and that replay setting.
   `--assets-dir` replaces the asset folder the rows were stored with, and is
-  required when that folder doesn't exist on this machine.
+  required when that folder doesn't exist on this machine. Rows stored by a
+  run that named no folder read each probe's own assets.
 - **Concurrent writers.** Writes into `--out` take a lock, so two calls can
   share it. On a filesystem without file locks, or whose locks don't reach
   every node (Lustre mounted without `flock` or with `localflock`, NFS with

docs/engine/README.md

@@ -589,7 +589,7 @@ The tables describe the main files.
 | `test_locale.py` | `SHIPPED_LOCALES` parity across language-name / display-name / script-range maps; new locales caught at CI time |
 | `test_language_detection.py` | Lazy `lingua` detector loading, `script_compliance_fraction`, `script_dominance` helpers |
 | `test_default_asset_paths.py` | Every shipped probe's default asset path resolves to a real file across every supported locale |
-| `test_assets_discovery.py` | Runtime assets-dir resolution (`set_runtime_assets_dir`, `probe_assets_dir`), env-override precedence, a probe's declared `assets_root`, and every registered probe owning a packaged folder |
+| `test_assets_discovery.py` | Runtime assets-dir resolution (`set_runtime_assets_dir`, `probe_assets_dir`), env-override precedence, a probe's declared `assets_root`, and every registered probe owning a folder in the package |
 | `test_config.py` | `ConversationSimulatorConfig` defaults, optional fields, side-effect-columns contract (the core columns plus every registered probe's `output_columns`) |
 | `test_pipeline.py` | Pipeline-level config validation |
 | `test_generator.py` | Dispatcher failed-result structure, probe-variant resolution, structured `SCENARIO_ABORTED` outcomes |
@@ -603,13 +603,11 @@ The tables describe the main files.
 | `test_analysis.py` | Score extraction, demographic preservation, edge cases used by the deep-dive analysis script |
 | `test_probing_taxonomy.py` | Probing-taxonomy YAML schema validator, loader, lookup helpers, locale-keyed shared cache |
 
-**`tests/engine/probes/`: the probe generators, verifier and invariants**
+**`tests/engine/probes/`: cross-probe invariants**
 
 | Test File | Coverage |
 |-----------|----------|
 | `test_probe_invariants.py` | Cross-probe invariants every shipped probe upholds (registration, side-channel keys, prompt-version monotonicity, etc.) |
-| `test_tool_calling.py` | Tool-calling probe: tool sampling, gate check |
-| `test_verifiers.py` | Tool-call JSON-schema validator |
 
 **`tests/engine/evaluator/`: judges, scorers, registry**
 
@@ -659,3 +657,6 @@ The tables describe the main files.
 | `health_disclosure/test_bank.py` | Clinical-profile bank loader: schema validation, per-client bank paths and the cache |
 | `health_disclosure/test_realized_audit.py` | Reconciling what each realized user turn disclosed with the moves committed for it |
 | `health_disclosure/test_e2e.py` | The probe's real columns through the store to the concealment scorer |
+| `tool_calling/test_probe.py` | `tool_calling` probe: prompt templates, the tool helpers, the simulated API response, the row-derived tool draw, and conversations through the loop with mocked models |
+| `tool_calling/test_verifiers.py` | Tool-call JSON-schema validator |
+| `tool_calling/test_catalog.py` | The shipped tool catalog: every spec is valid JSON Schema that providers accept, with typed items, defaults, examples and enums, and every tool and parameter described |

notebooks/01_simulate.ipynb

@@ -97,7 +97,7 @@
     "from usersim.cli._pipeline import build_simulator_config_builder, known_probes, sample_trajectories\n",
     "\n",
     "# --- Engine (asset + storage + preview helpers) ---\n",
-    "from usersim.engine.core._assets import packaged_assets_dir, probe_assets_dir\n",
+    "from usersim.engine.core._assets import probe_assets_dir\n",
     "from usersim.engine.core.preview import (\n",
     "    filter_preview_locale,\n",
     "    preview_row,\n",
@@ -128,9 +128,9 @@
     "# Trajectories land as a directory-partitioned dataset rooted at\n",
     "# TRAJECTORY_PATH (Hive-style: locale=X/probe_family=Y/part-N.parquet).\n",
     "REPO_ROOT = Path.cwd().parent if Path.cwd().name == \"notebooks\" else Path.cwd()\n",
-    "# Banks ship inside the installed package, so resolve them from there rather\n",
-    "# than from the checkout. This works the same from a wheel install.\n",
-    "ASSETS_DIR = packaged_assets_dir()\n",
+    "# None reads each probe's banks from its own folder in the installed package,\n",
+    "# the same from a checkout or a wheel. A folder here pins every probe to it.\n",
+    "ASSETS_DIR = None\n",
     "OUTPUT_DIR = REPO_ROOT / \"output\"\n",
     "OUTPUT_DIR.mkdir(parents=True, exist_ok=True)\n",
     "TRAJECTORY_PATH = OUTPUT_DIR / \"trajectories\"\n",

pyproject.toml

@@ -116,7 +116,7 @@ dev = [
     # has only through this backport.
     "exceptiongroup>=1.0; python_version < '3.11'",
     # Validates each tool spec in the tool_calling catalog against the JSON
-    # Schema 2020-12 meta-schema, in tests/engine/probes/test_tool_catalog.py.
+    # Schema 2020-12 meta-schema, in tests/probes/tool_calling/test_catalog.py.
     "jsonschema>=4.0,<5",
 ]
 # Empty on purpose: the gate runs a pinned pip-licenses through `uv tool run`,
@@ -168,15 +168,10 @@ include = ["usersim*"]
     "models_*.toml",
 ]
 "usersim.engine" = [
-    # The asset banks. setuptools ships package data only when it is named,
-    # and a wheel without the banks fails every probe at asset resolution.
-    # `make check-dist` fails if the wheel misses any bank file.
-    "assets/**/*.yaml",
-    "assets/**/*.yml",
-    "assets/**/*.json",
-    "assets/**/*.parquet",
-    "assets/**/*.md",
-    # The banks a probe keeps in its own folder, under probes/<label>/assets/.
+    # The asset banks, each in its probe's own folder, under probes/<folder>/assets/.
+    # setuptools ships package data only when it is named, and a wheel without the
+    # banks fails every probe at asset resolution. `make check-dist` fails if the
+    # wheel misses any bank file.
     "probes/*/assets/**/*.yml",
     "probes/*/assets/**/*.json",
     "probes/*/assets/**/*.parquet",

scripts/check_dist.py

@@ -36,13 +36,11 @@
 def in_a_bank(path: str) -> bool:
     """Whether a package file is in an asset bank.
 
-    Banks live under ``usersim/engine/assets/``, or in the ``assets/`` folder
-    of a probe kept in its own folder, ``usersim/engine/probes/<label>/assets/``.
+    Banks live in the ``assets/`` folder of each built-in probe's own folder,
+    ``usersim/engine/probes/<folder>/assets/``.
     """
     parts = path.split("/")
-    return path.startswith("usersim/engine/assets/") or (
-        parts[:3] == ["usersim", "engine", "probes"] and parts[4:5] == ["assets"]
-    )
+    return parts[:3] == ["usersim", "engine", "probes"] and parts[4:5] == ["assets"]
 
 
 def ships(path: str) -> bool:

src/usersim/cli/_inputs.py

@@ -72,8 +72,8 @@ class EpisodeInputs:
     #: The ``usersim_config`` every row shares, apart from ``_PER_ROW_SETTINGS``
     #: and ``_RUN_SETTINGS``, which the run settles (see :func:`_run_config`).
     config: dict[str, Any]
-    #: The asset root the rows run against.
-    assets_dir: Path
+    #: The asset root the rows run against, or ``None`` to read each probe's own.
+    assets_dir: Path | None
     #: The file the rows came from, when they came from one.
     source: Path | None = None
 
@@ -208,22 +208,29 @@ def setup(row: dict[str, Any]) -> dict[str, Any]:
     return dict(rows[0]["usersim_config"])
 
 
-def _resolve_assets_dir(rows: list[dict[str, Any]], given: str | Path | None) -> Path:
-    """``--assets-dir`` when given, else the one asset root the rows were stored with."""
-    from usersim.engine.core._assets import packaged_assets_dir
+def _resolve_assets_dir(rows: list[dict[str, Any]], given: str | Path | None) -> Path | None:
+    """``--assets-dir`` when given, else the asset root the rows were stored with.
 
+    ``None`` when the rows name none: they were stored by a run that named no
+    folder, so each probe reads its own assets.
+    """
     if given is not None:
         path = Path(given).resolve()
         if not path.is_dir():
             raise ValueError(f"--assets-dir {path} is not a directory")
         return path
     stored = {row["usersim_config"].get("assets_dir") for row in rows}
-    hint = f"pass --assets-dir (the assets that ship with User Sim are at {packaged_assets_dir()})"
-    if len(stored) != 1 or None in stored:
-        raise ValueError(f"the rows do not name one asset folder; {hint}")
-    path = Path(stored.pop())
+    if len(stored) != 1:
+        raise ValueError("the rows were stored with different asset folders; pass --assets-dir to name one")
+    folder = stored.pop()
+    if folder is None:
+        return None
+    path = Path(folder)
     if not path.is_dir():
-        raise ValueError(f"the asset folder the rows were stored with, {path}, does not exist here; {hint}")
+        raise ValueError(
+            f"the asset folder the rows were stored with, {path}, does not exist here; pass --assets-dir to "
+            "name one, or store the rows again without --assets-dir so each probe reads its own assets"
+        )
     return path
 
 
@@ -288,7 +295,7 @@ def _model_settings(models: Any, aliases: list[str]) -> tuple[dict[str, Any], di
     return readable, hashed
 
 
-def _asset_roots_digest(assets_dir: Path) -> list[str]:
+def _asset_roots_digest(assets_dir: Path | None) -> list[str]:
     """The contents of every asset root a run can read, in priority order.
 
     The search path, then the ``assets_root`` each registered probe declares.
@@ -520,7 +527,11 @@ def _column_config(inputs: EpisodeInputs, locale: str, models: Any):
     from usersim.engine.config import ConversationSimulatorConfig
 
     return ConversationSimulatorConfig.model_validate(
-        {**_run_config(inputs, models), "locale": locale, "assets_dir": str(inputs.assets_dir)}
+        {
+            **_run_config(inputs, models),
+            "locale": locale,
+            "assets_dir": str(inputs.assets_dir) if inputs.assets_dir is not None else None,
+        }
     )