feat(engine): read run setup from the probe class

architecture/assets.md

@@ -45,17 +45,18 @@ checkout.
 ## Pre-flight
 
 Before any model call, a run verifies that every selected probe's assets
-exist, including the locale subtree for the locale-scoped probes:
-`financial_services`, `sov_ai_facts` and `sov_ai_dynamic`. A locale that runs
-on another locale's bank is checked against that bank, the one the probe loads
-in its place: a registered India variant with no subtree of its own runs on
-`en_IN`'s. For `financial_services` a subtree counts once it has tasks, and
-`hi_Deva_IN` and `hi_Latn_IN` run on `en_IN`'s until theirs does. Every
-missing path is reported at once, with what to do: drop the probe or the
-locale from the run, or point `--assets-dir` somewhere else. When a probe
-can't run one of the run's locales, the refusal also names the locales it can
-run and links the feature-request form, and for `financial_services` the guide
-to authoring a bank.
+exist. A probe whose banks are split by locale declares a `locale_bank_check`
+on its class, and the run loads the bank each run locale runs on with it:
+`financial_services`, `sov_ai_facts` and `sov_ai_dynamic` declare one. The
+class's `bank_locale` names that bank, the one the probe loads in the locale's
+place when it has none of its own: a registered India variant with no subtree
+of its own runs on `en_IN`'s. For `financial_services` a subtree counts once it
+has tasks, and `hi_Deva_IN` and `hi_Latn_IN` run on `en_IN`'s until theirs
+does. Every missing path is reported at once, with what to do: drop the probe
+or the locale from the run, or point `--assets-dir` somewhere else. When a
+probe can't run one of the run's locales, the refusal also names the locales it
+can run and the guide the class's `authoring_guide` links, if any, and for a
+probe that ships with User Sim, the feature-request form.
 
 Without this, a missing bank surfaces from inside a loader partway through a
 run, after the other probes in the mix have already been paid for, while

architecture/cli.md

@@ -43,10 +43,13 @@ them: `api_response_model` for tool-calling probes and `summary_model` for the
 loop's light work: summaries for context management, early-stop checks, the
 pressure probe's capitulation check, and translating India-variant openings
 and `financial_services` search queries.
-Without it, `user_model` does that work. `eval` requires `evaluator_model`,
-kept distinct from `judge_model` so the in-sim gate and the post-hoc rubric
-can be sized independently. A missing required alias fails at config
-construction, not mid-run.
+Without it, `user_model` does that work. A probe that calls a model of its own
+names the alias in `model_aliases` on its class, as `financial_services` does
+for the model that embeds its search queries: each declared alias the file
+defines reaches the probe, and one it lacks is left out. `eval` requires
+`evaluator_model`, kept distinct from `judge_model` so the in-sim gate and the
+post-hoc rubric can be sized independently. A missing required alias fails at
+config construction, not mid-run.
 
 `usersim.cli.model_catalog` holds per-model inference defaults: temperature,
 top_p, max_tokens, and any model-family-specific `extra_body`. A TOML row

architecture/probes.md

@@ -111,7 +111,14 @@ A probe's content lives at `engine/assets/<label>/`, resolved through
 `probe_assets_dir()`. A run fails pre-flight, before any model call, if a
 selected probe's bank is missing or, for the probes whose banks are split by
 locale, doesn't load for a run locale (a finance bank without tasks, for
-instance). The refusal names the probe, the locale and the locales that can
-run, rather than failing from inside a loader mid-run. `simulate` and its
-`--dry-run`, `--materialize-inputs`, `--inputs` and notebook 01 all run the
-check. See [assets.md](assets.md).
+instance). Those probes declare a `locale_bank_check` on the class, which the
+pre-flight calls on the bank the class's `bank_locale` names. The refusal names
+the probe, the locale and the locales that can run, rather than failing from
+inside a loader mid-run. `simulate` and its `--dry-run`, `--materialize-inputs`,
+`--inputs` and notebook 01 all run the check. See [assets.md](assets.md).
+
+A probe whose conversations vary by topic lists its themes in a `themes`
+classmethod, read under its asset folder. A run draws one per row into the
+`theme` column, and the probe reads it back with `theme()`. A probe that lists
+none gets a placeholder it never reads. `tool_calling` takes its themes and
+toolsets from the run's settings instead.

docs/engine/AUTHORING_A_PROBE.md

@@ -235,6 +235,28 @@ def load_my_bank(locale: str) -> MyBank:
 [`docs/extensions.md`](../extensions.md) builds a whole package around such a
 probe.
 
+### Call a model of your own
+
+A probe that calls a model beyond the conversation's own (the user, assistant,
+tool-response, judge and summary models) names its alias in the models file in
+a `model_aliases` classmethod:
+
+```python
+class MyProbe(BaseProbe):
+    @classmethod
+    def model_aliases(cls, cfg) -> tuple[str, ...]:
+        return ("my_probe_critic_model",)
+```
+
+Each row's `models` holds every declared alias the models file defines, so the
+probe reads its model as `self._models.get("my_probe_critic_model")`. An alias
+the file lacks is left out, and the probe handles its absence: `get` returns
+`None`, and the probe can fall back or raise an error that names the alias. A
+run's setup fingerprint counts the settings of each declared model. `cfg` is
+the run's `ConversationSimulatorConfig`, for an alias that is itself a setting:
+`financial_services` declares the model named by `finance_embedding_model_alias`,
+which embeds its search queries, and searches lexically without it.
+
 ---
 
 ## Step 3: wire the asset bank
@@ -302,6 +324,63 @@ or `None` if nothing matched. The substrate:
 Your probe then reads `self._task` from the prompt-formatting hooks
 and from `should_succeed` / `build_result_extras`.
 
+### Check each locale's bank before a run
+
+Before any model call, a run checks that each selected probe's asset folder
+exists and holds bank files. A probe whose banks are split by locale also sets
+`locale_bank_check`, and the run then loads the bank each run locale runs on
+and refuses a locale whose bank is missing or does not load:
+
+```python
+def _check_locale_bank(locale_dir: Path) -> None:
+    MyBank.from_yaml((locale_dir / "sample.yaml").read_text(encoding="utf-8"))
+
+
+class MyProbe(BankVerbatimMixin, BankBackedProbe):
+    locale_bank_check = staticmethod(_check_locale_bank)
+    authoring_guide = "https://example.com/my-probe/README.md#adding-a-locale"
+```
+
+- **`locale_bank_check`** takes one locale's folder, `<asset folder>/<locale>/`,
+  and raises whatever would stop the bank loader. The refusal quotes the error.
+- **`bank_locale(locale, probe_dir)`**, a classmethod, names the locale whose
+  bank a row in `locale` runs on, as the bank loader picks it. By default, an
+  India language variant runs on its own bank when it has one and on `en_IN`'s
+  otherwise; any other locale runs on its own. `financial_services` overrides
+  it, since its bank for a locale counts only once it has tasks.
+- **`authoring_guide`** is an `https://` link to how to build the probe's bank
+  for another locale. The refusal links it when a run asks for a locale the
+  probe has no bank for.
+
+### Give each row a theme
+
+A probe whose conversations vary by topic lists the topics in a `themes`
+classmethod. A run draws one per row, stores it in the row's `theme` column,
+and the probe reads it back with `self.theme()`:
+
+```python
+class MyProbe(BaseProbe):
+    @classmethod
+    def themes(cls, locale: str) -> list[dict]:
+        path = cls.assets_dir() / locale / "topics.json"
+        return json.loads(path.read_text(encoding="utf-8"))
+
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+        self._topic = self.theme().get("description", "a topic of their choice")
+```
+
+- **`themes(locale)`** returns the themes as mappings, read under
+  `assets_dir()`, so the run's asset root reaches them as it reaches the bank
+  loader. A run refuses a probe that returns none for a run locale.
+  `general_open_ended` and `general_educational` read theirs from their seed
+  files.
+- **`self.theme()`** returns the row's theme as a mapping, whether the row
+  holds it as JSON text, as a run stores it, or as the mapping a test passes
+  to `run_episode`.
+- **`None`**, the default `themes`, is a probe that reads no theme. Its rows
+  hold a placeholder.
+
 ---
 
 ## Step 4: write your `LocalePromptPack`

docs/engine/EXTERNAL_PROBE_RUNTIME.md

@@ -148,6 +148,7 @@ as a Data Designer model facade.
 runtime_models["api_response_model"] = my_tool_response_model
 ```
 
+A probe names any other model it calls in `model_aliases` on its class.
 `financial_services` optionally embeds its `kb_search` queries with the model
 named by `finance_embedding_model_alias` (`embedding_model` by default, anything
 exposing `agenerate_text_embeddings()`); without one, dense retrieval falls back

docs/extensions.md

@@ -113,18 +113,22 @@ class RecipeBank:
     recipes: tuple[Recipe, ...]
 
 
+def read_recipe_bank(locale_dir: Path) -> RecipeBank:
+    doc = yaml.safe_load((locale_dir / "recipes.yaml").read_text(encoding="utf-8"))
+    return RecipeBank(doc["bank_id"], doc["bank_version"], tuple(Recipe(**r) for r in doc["recipes"]))
+
+
 @cache
 def load_recipe_bank(locale: str) -> RecipeBank:
-    path = RecipeProbe.assets_dir() / locale / "recipes.yaml"
-    doc = yaml.safe_load(path.read_text(encoding="utf-8"))
-    return RecipeBank(doc["bank_id"], doc["bank_version"], tuple(Recipe(**r) for r in doc["recipes"]))
+    return read_recipe_bank(RecipeProbe.assets_dir() / locale)
 
 
 @register_probe(family="acme_recipes", prompt_version="v1.0", variants=("default",))
 class RecipeProbe(BankBackedProbe):
     label = "acme_recipes"
     assets_root = Path(__file__).parent / "assets"
     bank_loader = staticmethod(load_recipe_bank)
+    locale_bank_check = staticmethod(read_recipe_bank)
     placeholder_warning_kind = "acme_used_placeholder_recipe"
     output_columns = ("acme_recipe_id",)
 
@@ -147,6 +151,8 @@ class RecipeProbe(BankBackedProbe):
 - **`load_recipe_bank`** reads under `RecipeProbe.assets_dir()`, the probe's
   folder on the asset search path, so a folder `--assets-dir` or
   `USERSIM_ASSET_PATH` names reaches it too.
+- **`locale_bank_check`** reads one locale's bank, so a run in a locale with no
+  bank that loads stops before any model call rather than failing every row.
 - **`placeholder_warning_kind`** is the warning a row records when its recipe
   is marked `placeholder: true`. Its `<prefix>_used_placeholder_<thing>` form is
   what the capability report counts as placeholder content.
@@ -272,5 +278,7 @@ usersim simulate --locale en_US --num-rows 5 \
   --probe-mix "acme_recipes=1.0" --out output/trajectories
 ```
 
-`--assets-dir` pins every probe in a run to the folder it names, this one
-included, so a run that passes it needs `acme_recipes/` there too.
+The package has a bank for `en_US` only, so a run with `--locale fr_FR` stops
+before any model call and says the probe has no bank for it. `--assets-dir`
+pins every probe in a run to the folder it names, this one included, so a run
+that passes it needs `acme_recipes/` there too.

src/usersim/cli/_inputs.py

@@ -368,9 +368,10 @@ def setup_fingerprint(models: Any, inputs: EpisodeInputs) -> tuple[str, dict[str
 
     Returns the fingerprint and the run record that names it. Covered:
 
-    - every simulator model alias as Data Designer sends it, provider settings
-      included (``api_key``, a provider's name and ``max_parallel_requests``
-      are left out: they don't change what a model says);
+    - every simulator model alias, and every alias a registered probe names in
+      ``model_aliases``, as Data Designer sends it, provider settings included
+      (``api_key``, a provider's name and ``max_parallel_requests`` are left
+      out: they don't change what a model says);
     - the shared ``usersim_config`` as loaded, defaults applied, with whether
       the assistant's reasoning is replayed as the run's models file sets it;
     - the contents of every asset root, a probe's declared ``assets_root``
@@ -382,13 +383,14 @@ def setup_fingerprint(models: Any, inputs: EpisodeInputs) -> tuple[str, dict[str
     """
     from usersim.engine.config import MODEL_ALIASES, ConversationSimulatorConfig
     from usersim.engine.core.manifest import _own_version
+    from usersim.engine.core.probes import declared_model_aliases
     from usersim.engine.core.storage import RUN_KIND_STORED
 
-    simulation = ConversationSimulatorConfig.model_validate(_run_config(inputs, models)).model_dump(mode="json")
+    config = ConversationSimulatorConfig.model_validate(_run_config(inputs, models))
+    simulation = config.model_dump(mode="json")
     for key in _PER_ROW_SETTINGS:
         simulation.pop(key, None)
-    aliases = [*MODEL_ALIASES, simulation.get("finance_embedding_model_alias") or ""]
-    readable, hashed = _model_settings(models, [alias for alias in aliases if alias])
+    readable, hashed = _model_settings(models, [*MODEL_ALIASES, *declared_model_aliases(config)])
     fingerprint = _digest(
         {
             "models": hashed,