diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5fc609a3..56fcd63b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -144,8 +144,8 @@ uv run usersim smoke # offline self-check; no API key, no network ``` `smoke` is the fastest way to confirm an install is sound. It resolves the -model catalog, every probe's assets, and the plugin entry points without -making a single model call. +model catalog, runs each probe's check of the assets it ships, and loads the +plugin entry points without making a single model call. ### Choosing models and providers diff --git a/architecture/assets.md b/architecture/assets.md index 707de136..8986f98c 100644 --- a/architecture/assets.md +++ b/architecture/assets.md @@ -62,6 +62,16 @@ Without this, a missing bank surfaces from inside a loader partway through a run, after the other probes in the mix have already been paid for, while `--dry-run` would report the plan and exit 0. +## Install check + +`usersim smoke` runs each registered probe's `asset_problems` classmethod, +which checks the assets the probe ships without a model call. For example, +`sov_ai_facts` and `sov_ai_dynamic` look for a bank in every shipped locale, +`financial_services` loads its `en_US` bank and that bank's dynamic-tier +taxonomy, and `identity_disclosure` loads its spec through every layer. A probe +from another package is checked the same way, and `assert_probe_conforms` +reports what its check finds. + ## Generating a bank `usersim gen-assets --domain ` generates one; `validate-assets` vets it diff --git a/architecture/evaluator.md b/architecture/evaluator.md index e499c6df..12f1034e 100644 --- a/architecture/evaluator.md +++ b/architecture/evaluator.md @@ -13,7 +13,9 @@ ensemble never rests on one model family. A single judge is allowed. **Scorers** are per-probe functions over stored columns, each returning a structured verdict rather than a free-form opinion. Three are fully -deterministic (`language_compliance`, `refusal_basics`, `response_shape`); the +deterministic (`language_compliance`, `refusal_basics`, `response_shape`) and +register with `deterministic=True`, which is what lets `usersim rescore` +recompute them on a stored evaluation; the other nine call a model to reach a judgment the stored columns cannot settle on their own, such as whether a retrieved fact is actually supported or what a free-form turn actually disclosed (`health_disclosure_concealment`, which falls @@ -35,8 +37,12 @@ Twelve ship: | `response_shape` | structural expectations | | `refusal_basics` | refusal presence and form | -Scorers are opt-in per run. A capability whose scorer was not dispatched -renders as *untested* rather than as a failure, and +Scorers are opt-in per run. `usersim eval --scorers auto` runs the three +deterministic scorers, which apply to every probe, plus the `default_scorers` +each probe in the run declares, found by the `probe_family` its rows are stored +under; `auto` combines with scorer names. Over a run holding every built-in +probe it runs every scorer the capability report reads. A capability whose +scorer was not dispatched renders as *untested* rather than as a failure, and `capabilities_missing_scorer()` exists so a run can warn about that instead of silently producing an empty column. @@ -62,6 +68,8 @@ Scorers register through `register_scorer` at import, and are discovered out-of-tree through the `usersim.scorers` entry point. A scorer receives the trajectory row as a dict in which every empty cell is `None`, apart from `conversation_messages`, `persona`, `probe_family`, `locale` and `language`, -which the evaluator always fills. `usersim.testing` publishes a conformance +which the evaluator always fills: `probe_family` is an empty string for a row +that names no probe, and the judges score such a row only on the axes every +probe shares. `usersim.testing` publishes a conformance check, including that a scorer tolerates a row missing any given field, since the evaluator calls it on incomplete trajectories. See [plugins.md](plugins.md). diff --git a/architecture/plugins.md b/architecture/plugins.md index 5ca16fab..a6127447 100644 --- a/architecture/plugins.md +++ b/architecture/plugins.md @@ -133,7 +133,8 @@ failing later inside a run. The row a scorer receives holds every column of the trajectory, and an empty cell arrives as `None` whichever way pandas held it. The evaluator fills `conversation_messages` (a list), `persona` (a dict), `probe_family`, `locale` -and `language` itself, so those are never `None`. A default passed to +and `language` itself, so those are never `None`; `probe_family` is an empty +string for a row that names no probe. A default passed to `row.get(key, default)` covers only an absent key, so pass a row shaped like your probe's trajectories to the check as well: `assert_scorer_conforms("my_scorer", sample_row=row)` also scores that row with diff --git a/architecture/probes.md b/architecture/probes.md index 7f9bddfb..908dc9ae 100644 --- a/architecture/probes.md +++ b/architecture/probes.md @@ -54,7 +54,11 @@ required; the rest have working defaults on `BaseProbe`. | `build_result_extras` | adding probe-specific columns to the trajectory, each declared in the class's `output_columns` | Anything a probe does not override inherits a default, so a new probe declares -only what is specific to it. +only what is specific to it. Its `description`, a paragraph on what it tests, is +what the per-row preview and the capability report show with its rows, and its +`preview_fields` are what the preview's Probe Inputs panel shows about the bank +entry or theme each row drew. Its `default_scorers` are the scorers +`usersim eval --scorers auto` runs when a run holds its rows. A probe's own columns are the one part of this that is declared twice. Whatever `build_result_extras` returns is written only if the name is a core column or @@ -115,7 +119,10 @@ instance). Those probes declare a `locale_bank_check` on the class, which the pre-flight calls on the bank the class's `bank_locale` names. The refusal names the probe, the locale and the locales that can run, rather than failing from inside a loader mid-run. `simulate` and its `--dry-run`, `--materialize-inputs`, -`--inputs` and notebook 01 all run the check. See [assets.md](assets.md). +`--inputs` and notebook 01 all run the check. A probe's `asset_problems` +classmethod checks the assets it ships without a model call: `usersim smoke` +runs it for each registered probe, and `assert_probe_conforms` reports what it +finds. See [assets.md](assets.md). A probe whose conversations vary by topic lists its themes in a `themes` classmethod, read under its asset folder. A run draws one per row into the diff --git a/docs/engine/AUTHORING_A_PROBE.md b/docs/engine/AUTHORING_A_PROBE.md index 5e753a7f..58d3d75c 100644 --- a/docs/engine/AUTHORING_A_PROBE.md +++ b/docs/engine/AUTHORING_A_PROBE.md @@ -206,6 +206,32 @@ warnings a probe adds with `add_warning` take the same prefixed form, such as For `BaseProbe`-only subclasses, just `label` is required. +Every probe also sets `description`, a paragraph on what it tests. The per-row +preview and the capability report's evidence show it with each of the probe's +rows: + +```python +class MyProbe(BaseProbe): + label = "my_probe" + description = "Tests whether the assistant ..." +``` + +A probe whose rows draw from a bank also lists, in `preview_fields`, what the +per-row preview's Probe Inputs panel shows about the entry each row drew. A +`PreviewField` names a column of the row, or with `from_metadata=True` a key +the probe writes to `state.metadata`, and the label it is shown with. The panel +leaves out a field whose value is empty, and is left out when every field is: + +```python +from usersim.engine.core.probes import PreviewField + +class MyProbe(BankBackedProbe): + preview_fields = ( + PreviewField("my_probe_item_id", "Item id"), + PreviewField("item_category", "Category", from_metadata=True), + ) +``` + A probe shipped in its own package, rather than in this repository, also sets `assets_root` to the directory that holds its `