feat(engine): declare probe descriptions and which scorers run

CONTRIBUTING.md

@@ -144,8 +144,8 @@ uv run usersim smoke        # offline self-check; no API key, no network
 ```
 
 `smoke` is the fastest way to confirm an install is sound. It resolves the
-model catalog, every probe's assets, and the plugin entry points without
-making a single model call.
+model catalog, runs each probe's check of the assets it ships, and loads the
+plugin entry points without making a single model call.
 
 ### Choosing models and providers
 

architecture/assets.md

@@ -62,6 +62,16 @@ Without this, a missing bank surfaces from inside a loader partway through a
 run, after the other probes in the mix have already been paid for, while
 `--dry-run` would report the plan and exit 0.
 
+## Install check
+
+`usersim smoke` runs each registered probe's `asset_problems` classmethod,
+which checks the assets the probe ships without a model call. For example,
+`sov_ai_facts` and `sov_ai_dynamic` look for a bank in every shipped locale,
+`financial_services` loads its `en_US` bank and that bank's dynamic-tier
+taxonomy, and `identity_disclosure` loads its spec through every layer. A probe
+from another package is checked the same way, and `assert_probe_conforms`
+reports what its check finds.
+
 ## Generating a bank
 
 `usersim gen-assets --domain <d>` generates one; `validate-assets` vets it

architecture/evaluator.md

@@ -13,7 +13,9 @@ ensemble never rests on one model family. A single judge is allowed.
 
 **Scorers** are per-probe functions over stored columns, each returning a
 structured verdict rather than a free-form opinion. Three are fully
-deterministic (`language_compliance`, `refusal_basics`, `response_shape`); the
+deterministic (`language_compliance`, `refusal_basics`, `response_shape`) and
+register with `deterministic=True`, which is what lets `usersim rescore`
+recompute them on a stored evaluation; the
 other nine call a model to reach a judgment the stored columns cannot settle on
 their own, such as whether a retrieved fact is actually supported or what a
 free-form turn actually disclosed (`health_disclosure_concealment`, which falls
@@ -35,8 +37,12 @@ Twelve ship:
 | `response_shape` | structural expectations |
 | `refusal_basics` | refusal presence and form |
 
-Scorers are opt-in per run. A capability whose scorer was not dispatched
-renders as *untested* rather than as a failure, and
+Scorers are opt-in per run. `usersim eval --scorers auto` runs the three
+deterministic scorers, which apply to every probe, plus the `default_scorers`
+each probe in the run declares, found by the `probe_family` its rows are stored
+under; `auto` combines with scorer names. Over a run holding every built-in
+probe it runs every scorer the capability report reads. A capability whose
+scorer was not dispatched renders as *untested* rather than as a failure, and
 `capabilities_missing_scorer()` exists so a run can warn about that instead
 of silently producing an empty column.
 
@@ -62,6 +68,8 @@ Scorers register through `register_scorer` at import, and are discovered
 out-of-tree through the `usersim.scorers` entry point. A scorer receives the
 trajectory row as a dict in which every empty cell is `None`, apart from
 `conversation_messages`, `persona`, `probe_family`, `locale` and `language`,
-which the evaluator always fills. `usersim.testing` publishes a conformance
+which the evaluator always fills: `probe_family` is an empty string for a row
+that names no probe, and the judges score such a row only on the axes every
+probe shares. `usersim.testing` publishes a conformance
 check, including that a scorer tolerates a row missing any given field, since
 the evaluator calls it on incomplete trajectories. See [plugins.md](plugins.md).

architecture/plugins.md

@@ -133,7 +133,8 @@ failing later inside a run.
 The row a scorer receives holds every column of the trajectory, and an empty
 cell arrives as `None` whichever way pandas held it. The evaluator fills
 `conversation_messages` (a list), `persona` (a dict), `probe_family`, `locale`
-and `language` itself, so those are never `None`. A default passed to
+and `language` itself, so those are never `None`; `probe_family` is an empty
+string for a row that names no probe. A default passed to
 `row.get(key, default)` covers only an absent key, so pass a row shaped like
 your probe's trajectories to the check as well:
 `assert_scorer_conforms("my_scorer", sample_row=row)` also scores that row with

architecture/probes.md

@@ -54,7 +54,11 @@ required; the rest have working defaults on `BaseProbe`.
 | `build_result_extras` | adding probe-specific columns to the trajectory, each declared in the class's `output_columns` |
 
 Anything a probe does not override inherits a default, so a new probe declares
-only what is specific to it.
+only what is specific to it. Its `description`, a paragraph on what it tests, is
+what the per-row preview and the capability report show with its rows, and its
+`preview_fields` are what the preview's Probe Inputs panel shows about the bank
+entry or theme each row drew. Its `default_scorers` are the scorers
+`usersim eval --scorers auto` runs when a run holds its rows.
 
 A probe's own columns are the one part of this that is declared twice. Whatever
 `build_result_extras` returns is written only if the name is a core column or
@@ -115,7 +119,10 @@ instance). Those probes declare a `locale_bank_check` on the class, which the
 pre-flight calls on the bank the class's `bank_locale` names. The refusal names
 the probe, the locale and the locales that can run, rather than failing from
 inside a loader mid-run. `simulate` and its `--dry-run`, `--materialize-inputs`,
-`--inputs` and notebook 01 all run the check. See [assets.md](assets.md).
+`--inputs` and notebook 01 all run the check. A probe's `asset_problems`
+classmethod checks the assets it ships without a model call: `usersim smoke`
+runs it for each registered probe, and `assert_probe_conforms` reports what it
+finds. See [assets.md](assets.md).
 
 A probe whose conversations vary by topic lists its themes in a `themes`
 classmethod, read under its asset folder. A run draws one per row into the

docs/engine/AUTHORING_A_PROBE.md

@@ -206,6 +206,32 @@ warnings a probe adds with `add_warning` take the same prefixed form, such as
 
 For `BaseProbe`-only subclasses, just `label` is required.
 
+Every probe also sets `description`, a paragraph on what it tests. The per-row
+preview and the capability report's evidence show it with each of the probe's
+rows:
+
+```python
+class MyProbe(BaseProbe):
+    label = "my_probe"
+    description = "Tests whether the assistant ..."
+```
+
+A probe whose rows draw from a bank also lists, in `preview_fields`, what the
+per-row preview's Probe Inputs panel shows about the entry each row drew. A
+`PreviewField` names a column of the row, or with `from_metadata=True` a key
+the probe writes to `state.metadata`, and the label it is shown with. The panel
+leaves out a field whose value is empty, and is left out when every field is:
+
+```python
+from usersim.engine.core.probes import PreviewField
+
+class MyProbe(BankBackedProbe):
+    preview_fields = (
+        PreviewField("my_probe_item_id", "Item id"),
+        PreviewField("item_category", "Category", from_metadata=True),
+    )
+```
+
 A probe shipped in its own package, rather than in this repository, also
 sets `assets_root` to the directory that holds its `<label>/` asset folder,
 so it finds its banks wherever the package is installed:
@@ -352,6 +378,29 @@ class MyProbe(BankVerbatimMixin, BankBackedProbe):
   for another locale. The refusal links it when a run asks for a locale the
   probe has no bank for.
 
+### Check the assets you ship
+
+`usersim smoke` runs each registered probe's `asset_problems` classmethod, and
+`assert_probe_conforms` reports what it finds, so an install that is missing a
+bank fails a check rather than a run. Return one string per problem, or an
+empty list, and read what your loaders read without calling a model:
+
+```python
+class MyProbe(BankVerbatimMixin, BankBackedProbe):
+    @classmethod
+    def asset_problems(cls) -> list[str]:
+        problems = []
+        for locale in ("en_US", "fr_FR"):
+            try:
+                _check_locale_bank(cls.assets_dir() / locale)
+            except Exception as error:
+                problems.append(f"{locale}: {type(error).__name__}: {error}")
+        return problems
+```
+
+The default checks nothing. An exception the check raises is reported like a
+problem it returns.
+
 ### Give each row a theme
 
 A probe whose conversations vary by topic lists the topics in a `themes`
@@ -551,6 +600,9 @@ is the most likely cause. If it shows up but `family missing` fires, the
 `@register_probe` decorator didn't run (check for an import-time
 exception in your module).
 
+A probe that checks its assets is named on the `probe assets` line, which
+fails with the first problems its `asset_problems` finds.
+
 For a probe in its own package, the `extensions` line names the package and
 fails with the reason when an entry point contributes nothing, such as an
 import error or a name another package already uses.
@@ -1078,7 +1130,10 @@ register_scorer("your_probe", score_your_probe_trajectory)
 
 `register_scorer` takes the name and the function; it is not a decorator.
 Give it a name no other scorer uses: a different function under a taken name
-raises `ScorerRegistrationError`. The
+raises `ScorerRegistrationError`. A scorer that never calls or looks up a model
+also passes `deterministic=True`, so `usersim rescore` can recompute it on a
+stored evaluation; `assert_scorer_conforms` reports one that looks up a model
+anyway. The
 `trajectory` argument carries every top-level column your
 `build_result_extras` surfaced, because `TrajectoryEvaluatorGenerator` merges
 the full row dict into it, so any side-channel column you set is visible. A
@@ -1089,10 +1144,22 @@ when you pass it a `sample_row`.
 Add the module path to
 [`evaluator/scorers/__init__.py::DEFAULT_SCORER_MODULES`](../../src/usersim/engine/evaluator/scorers/__init__.py)
 so the auto-load path picks it up. After that, name it at evaluation time
-(`usersim eval --scorers your_probe`). The evaluator calls each named scorer on
-every row, so return early for rows from other probes, for example when your
-probe's join key is missing, as the `safety_agentic` scorer does with
-`action_request_id`.
+(`usersim eval --scorers your_probe`), or list it in the probe's
+`default_scorers`:
+
+```python
+class MyProbe(BaseProbe):
+    label = "my_probe"
+    default_scorers = ("your_probe",)
+```
+
+`usersim eval --scorers auto` then runs it whenever the run holds the probe's
+rows, beside the rule-based `language_compliance`, `refusal_basics` and
+`response_shape`, which `auto` runs for every probe. `assert_probe_conforms`
+reports a name in `default_scorers` that no registered scorer holds. The
+evaluator calls each scorer it runs on every row, so return early for rows from
+other probes, for example when your probe's join key is missing, as the
+`safety_agentic` scorer does with `action_request_id`.
 
 The scorer is its own design exercise (axis definition, rubric
 authoring, threshold gating). See

docs/engine/README.md

@@ -419,9 +419,10 @@ Quick orientation:
    discarded on the way to storage (with a warning in the log), and the
    scorer that reads it back reports nothing to score; see the
    authoring guide's step 6.
-7. Run `usersim smoke` to confirm registration; copy
-   `templates/probe/test_probe.py.template` for the per-probe
-   regression test.
+7. Run `usersim smoke` to confirm registration and, when your probe
+   checks its banks in an `asset_problems` classmethod, that they are
+   found; copy `templates/probe/test_probe.py.template` for the
+   per-probe regression test.
 
 For the substrate's API surface (every class, every mixin, every
 hook), see the source-of-truth docstrings in
@@ -432,7 +433,8 @@ hook), see the source-of-truth docstrings in
 new axes to `evaluator/axes.py::PROBE_SCORES` if needed; add a
 scorer under `evaluator/scorers/your_probe.py` and include it in
 `DEFAULT_SCORER_MODULES` to register it on import; name it in `--scorers` to
-run it.
+run it, or list it in the probe's `default_scorers` so `--scorers auto` runs
+it.
 
 ## Scope
 

docs/extensions.md

@@ -96,7 +96,7 @@ from pathlib import Path
 import yaml
 
 from usersim.engine.core.identity import persona_uuid
-from usersim.engine.core.probes import BankBackedProbe, register_probe
+from usersim.engine.core.probes import BankBackedProbe, PreviewField, register_probe
 
 
 @dataclass(frozen=True)
@@ -126,12 +126,23 @@ def load_recipe_bank(locale: str) -> RecipeBank:
 @register_probe(family="acme_recipes", prompt_version="v1.0", variants=("default",))
 class RecipeProbe(BankBackedProbe):
     label = "acme_recipes"
+    description = "Tests cooking help: a home cook asks about a recipe from the bank."
+    preview_fields = (PreviewField("acme_recipe_id", "Recipe id"),)
+    default_scorers = ("acme_recipes",)
     assets_root = Path(__file__).parent / "assets"
     bank_loader = staticmethod(load_recipe_bank)
     locale_bank_check = staticmethod(read_recipe_bank)
     placeholder_warning_kind = "acme_used_placeholder_recipe"
     output_columns = ("acme_recipe_id",)
 
+    @classmethod
+    def asset_problems(cls) -> list[str]:
+        try:
+            read_recipe_bank(cls.assets_dir() / "en_US")
+        except Exception as error:
+            return [f"en_US: {type(error).__name__}: {error}"]
+        return []
+
     def derive_task(self, persona, bank, *, cfg):
         rng = random.Random(f"{persona_uuid(persona)}:{bank.bank_version}:{cfg.random_seed}")
         return rng.choice(bank.recipes)
@@ -146,13 +157,21 @@ class RecipeProbe(BankBackedProbe):
         return {**super().build_result_extras(state), "acme_recipe_id": self._task.id}
 ```
 
+- **`description`** is what the per-row preview and the capability report
+  show about the probe.
+- **`preview_fields`** puts the row's recipe id in the per-row preview's Probe
+  Inputs panel.
+- **`default_scorers`** names the package's scorer, so
+  `usersim eval --scorers auto` runs it whenever a run holds the probe's rows.
 - **`assets_root`** is the absolute directory holding `acme_recipes/`, so the
   probe finds its banks wherever the package is installed.
 - **`load_recipe_bank`** reads under `RecipeProbe.assets_dir()`, the probe's
   folder on the asset search path, so a folder `--assets-dir` or
   `USERSIM_ASSET_PATH` names reaches it too.
 - **`locale_bank_check`** reads one locale's bank, so a run in a locale with no
   bank that loads stops before any model call rather than failing every row.
+- **`asset_problems`** reads the bank the package ships, so `usersim smoke` and
+  `assert_probe_conforms` report an install that is missing it.
 - **`placeholder_warning_kind`** is the warning a row records when its recipe
   is marked `placeholder: true`. Its `<prefix>_used_placeholder_<thing>` form is
   what the capability report counts as placeholder content.
@@ -189,12 +208,14 @@ async def score(row, models):
     return {"acme_recipe_grounded": 1.0 if row.get("acme_recipe_id") else 0.0}
 
 
-register_scorer("acme_recipes", score)
+register_scorer("acme_recipes", score, deterministic=True)
 ```
 
 The evaluator awaits a scorer once per trajectory, with the row and the
 models, and merges the mapping it returns into the row's evaluation. A field
-the row holds empty arrives as `None`.
+the row holds empty arrives as `None`. `deterministic=True` says the scorer
+never calls a model, so `usersim rescore` can recompute it on a stored
+evaluation.
 
 ## Tests
 

src/usersim/cli/evaluate.py

@@ -112,7 +112,9 @@ def register(subparsers: argparse._SubParsersAction) -> None:
             "Comma-separated scorer names to run beside the judges, for "
             "example tool_use,language_compliance. Most scorers use an LLM "
             "judge; language_compliance, response_shape and refusal_basics "
-            "are rule-based. Default: none."
+            "are rule-based. 'auto' stands for those three plus the "
+            "default_scorers of each probe in the run, and combines with "
+            "names. Default: none."
         ),
     )
     p.add_argument(
@@ -179,6 +181,36 @@ def _parse_csv_or_none(spec: str | None) -> list[str] | None:
     return items or None
 
 
+def _resolve_scorers(requested: list[str], families: Iterable[str]) -> list[str]:
+    """``requested`` with ``auto`` replaced by the generic scorers and the ``default_scorers`` of the run's probes.
+
+    ``families`` are the ``probe_family`` values of the rows being evaluated.
+    Each name appears once, where it first occurs. A family no installed probe
+    stores its rows under adds no scorer of its own, and a warning names it. A
+    blank family names no probe, so it adds none and goes unnamed.
+    """
+    from usersim.engine.core.probes import default_scorers_by_family
+    from usersim.engine.evaluator.scorers import AUTO_SCORERS, GENERIC_SCORERS
+
+    if AUTO_SCORERS not in requested:
+        return requested
+    by_family = default_scorers_by_family()
+    present = {family for family in families if family}
+    unclaimed = sorted(present - set(by_family))
+    if unclaimed:
+        logger.warning(
+            "no installed probe stores its rows under probe_family %s, so --scorers %s adds none of "
+            "its scorers for those rows.",
+            ", ".join(unclaimed),
+            AUTO_SCORERS,
+        )
+    auto = [*GENERIC_SCORERS, *(name for family, names in by_family.items() if family in present for name in names)]
+    resolved: dict[str, None] = {}
+    for name in requested:
+        resolved.update(dict.fromkeys(auto if name == AUTO_SCORERS else (name,)))
+    return list(resolved)
+
+
 def _choose_rows(args: argparse.Namespace, seed_path: Path) -> tuple[Any, Any, int, dict[str, Any]]:
     """The trajectory rows this pass evaluates, and how they were chosen.
 
@@ -350,6 +382,11 @@ def run(args: argparse.Namespace) -> int:
             temp_sample_seed = seed_source_path = Path(tmp.name)
             sampled.to_parquet(temp_sample_seed, index=False)
 
+        resolved = _resolve_scorers(scorers, seed_df["probe_family"].head(num_records).dropna().astype(str))
+        if resolved != scorers:
+            print(f"  scorers:       {', '.join(resolved)}")
+        scorers = resolved
+
         data_designer, builder = build_evaluator_config_builder(
             models=models,
             trajectory_parquet=seed_source_path,