refactor(engine): keep each built-in's capabilities in its own package

architecture/plugins.md

@@ -100,7 +100,10 @@ report**, because the capability matrix is built from capability definitions
 rather than from trajectories. The rows exist; nothing asks about them.
 
 `usersim.capabilities` closes that. If you contribute a probe, contribute the
-capability it provides evidence for.
+capability it provides evidence for. A built-in probe declares its capabilities
+in its package's `capabilities.py`, which `BUILTIN_CAPABILITY_MODULES` in
+`taxonomy/capabilities.py` lists in report order, after the capabilities every
+probe feeds.
 
 ## Validating an extension
 

docs/engine/AUTHORING_A_PROBE.md

@@ -1251,8 +1251,8 @@ covers the setup.
 ## Wiring the out-of-sim scorer
 
 The probe simulator only PRODUCES trajectories. The matching scorer
-that JUDGES the assistant's responses lives separately under
-[`evaluator/scorers/<your_probe>.py`](../../src/usersim/engine/evaluator/scorers/).
+that JUDGES the assistant's responses runs separately, after the simulation,
+and lives in the probe's own package as `src/usersim/engine/probes/<your_probe>/scorer.py`.
 This is the in-sim / out-of-sim boundary the architecture
 deliberately enforces.
 
@@ -1277,7 +1277,7 @@ column that is empty on a row arrives present and `None`, so handle `None`
 wherever you read one; `usersim.testing.assert_scorer_conforms` checks that
 when you pass it a `sample_row`.
 
-Add the module path to
+Add the module path, `usersim.engine.probes.<your_probe>.scorer`, to
 [`evaluator/scorers/__init__.py::DEFAULT_SCORER_MODULES`](../../src/usersim/engine/evaluator/scorers/__init__.py)
 so the auto-load path picks it up. After that, name it at evaluation time
 (`usersim eval --scorers your_probe`), or list it in the probe's
@@ -1297,18 +1297,58 @@ evaluator calls each scorer it runs on every row, so return early for rows from
 other probes, for example when your probe's join key is missing, as the
 `safety_agentic` scorer does with `action_request_id`.
 
-A scorer registered under your probe's label writes that label in a module
-outside `engine/probes/`.
+A scorer in your probe's package may write its label;
 [`tests/test_probe_names_in_shared_code.py`](../../tests/test_probe_names_in_shared_code.py)
-allows each such file a fixed number of probe labels, so add the new module to
-its `_ALLOWED` in the same change. Anything else the core needs from your probe
-goes on the class: see
+counts labels only in shared code outside `engine/probes/`. Anything else the
+core needs from your probe goes on the class: see
 [What a probe declares](../../architecture/probes.md#what-a-probe-declares).
 
 The scorer is its own design exercise (axis definition, rubric
 authoring, threshold gating). See
 [`probes/sov_ai_facts/scorer.py`](../../src/usersim/engine/probes/sov_ai_facts/scorer.py)
-or `safety_chat_pressure.py` for canonical examples.
+or [`probes/safety_chat_pressure/scorer.py`](../../src/usersim/engine/probes/safety_chat_pressure/scorer.py)
+for canonical examples.
+
+## Adding the capabilities your probe provides evidence for
+
+The capability report is built from capability definitions, not from
+trajectories, so a probe without one simulates and evaluates and is then absent
+from the report. Declare the rows your probe's evidence feeds in
+`src/usersim/engine/probes/<your_probe>/capabilities.py`, as `CAPABILITIES`, a
+tuple of `CapabilityDefinition`:
+
+```python
+from usersim.extension import CapabilityDefinition, EvidenceSource
+
+CAPABILITIES: tuple[CapabilityDefinition, ...] = (
+    CapabilityDefinition(
+        id="booking_argument_accuracy",
+        label="Booking arguments",
+        description="Does the assistant book the date and time the user asked for?",
+        sources=(EvidenceSource(probe="your_probe", scorer="your_probe", axes=("your_probe.argument_match_rate",)),),
+        threshold=0.9,
+        scale="rate",
+        aggregation_policy="mean",
+        next_action="Read the failing rows and compare the user's words with the tool call.",
+    ),
+)
+
+#: What each rate axis your scorer computes measures, in a reviewer's terms.
+AXIS_DESCRIPTIONS: dict[str, str] = {
+    "your_probe.argument_match_rate": "Share of bookings whose date, time and party size match what the user asked for.",
+}
+```
+
+Then add the module, `usersim.engine.probes.<your_probe>.capabilities`, to
+`BUILTIN_CAPABILITY_MODULES` in
+[`taxonomy/capabilities.py`](../../src/usersim/taxonomy/capabilities.py), at
+the place its rows belong in the report. The module holds definitions only and
+imports nothing from the probe, so the report reads it without importing your
+probe. `AXIS_DESCRIPTIONS` is optional: it explains a rate axis in an evidence
+card, where a judge axis carries its own reasoning. A probe shipped as an
+extension contributes the same definitions through the `usersim.capabilities`
+entry point instead
+([`architecture/plugins.md`](../../architecture/plugins.md#capabilities-the-seam-that-is-easy-to-miss)).
 
 ---
 

docs/extension_api.md

@@ -94,6 +94,8 @@ does not load the conversation loop.
 | Name | What it is |
 |---|---|
 | `probe_assets_dir` | The folder a probe's assets resolve to: the first override root that holds them, else the root its class declares |
+| `probe_asset_roots` | The roots searched for a probe's assets, highest priority first, for assets layered across them |
+| `asset_roots_in_use` | Every root an asset lookup can read, for keying a cache by the roots a run uses |
 | `load_seeds` | A probe's seeds of one kind for a locale, such as its topics, from `<probe>/<locale>/<kind>.json`, or the `en_US` file when the locale has none |
 | `LocalizedText` | Bank text with a rendering per locale, read with `for_locale`, which falls back to `en_US` |
 | `LocalizedTextError` | Raised when a bank value cannot be read as `LocalizedText` |
@@ -134,6 +136,7 @@ does not load the conversation loop.
 |---|---|
 | `CapabilityDefinition` | A row of the capability report and the evidence it is scored from |
 | `EvidenceSource` | One probe, scorer and set of axes a capability reads |
+| `ALL_PROBES` | The probe an `EvidenceSource` names to read the rows of every probe |
 | `AssetDomain` | A domain `usersim gen-assets` can build banks for |
 | `to_model_configs` | The Data Designer model configs for the models an asset domain receives |
 | `to_data_designer_kwargs` | The `DataDesigner` keyword arguments for those models' providers |

src/usersim/engine/core/behavioral.py

@@ -144,7 +144,7 @@
 _ED_LOW_KEYWORDS = ["illiterate", "no_formal", "less_than", "below_primary"]
 
 
-def _education_ordinal(education_level: str, locale: str = "en_US") -> float:
+def education_ordinal(education_level: str, locale: str = "en_US") -> float:
     """Map an education_level string to a [0, 1] ordinal score.
 
     Uses exact per-locale lookup first, then keyword fallback, then 0.5.
@@ -289,7 +289,7 @@ def _infer_tech_literacy(persona: dict[str, Any], locale: str = "en_US") -> floa
     score = 0.5
 
     education = str(persona.get("education_level", ""))
-    ed_ord = _education_ordinal(education, locale)
+    ed_ord = education_ordinal(education, locale)
     if ed_ord >= 0.7:
         score += 0.15
     elif ed_ord <= 0.15:
@@ -329,7 +329,7 @@ def compute_behavioral_profile(
     tech_literacy = _infer_tech_literacy(persona, locale)
 
     education = str(persona.get("education_level", ""))
-    ed_ord = _education_ordinal(education, locale)
+    ed_ord = education_ordinal(education, locale)
     error_proneness = 0.3
     if ed_ord <= 0.15:
         error_proneness += 0.3

src/usersim/engine/probes/financial_services/README.md

@@ -521,7 +521,7 @@ Related code elsewhere:
 | Inner tool-loop scaffold (shared) | `core/probes.py` (`ToolExecutionMixin`) |
 | Query embedding (dense retrieval) | `probes/financial_services/embeddings.py` |
 | Scorer | `probes/financial_services/scorer.py` |
-| Capabilities | `taxonomy/capabilities.py` |
+| Capabilities | `probes/financial_services/capabilities.py` |
 | Training-export columns | `selection/schemas.py` (`finance` group) |
 | Assets + schema | `assets/financial_services/<locale>/` + `assets/financial_services/SCHEMA.md` |
 | Offline generation | `asset_gen/` + `asset_gen/notebooks/finance_assets.ipynb` |

src/usersim/engine/probes/financial_services/capabilities.py

@@ -0,0 +1,249 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""The capability rows ``financial_services``'s rows provide evidence for."""
+
+from __future__ import annotations
+
+from usersim.extension import CapabilityDefinition, EvidenceSource
+
+# financial_services institution types that get their OWN verifiable-task-success
+# capability row (a per-institution-type breakdown of financial_task_success).
+# Add a type here when a new institution type ships; a type absent from a given
+# locale's bank just renders as a no-data cell for that locale. Kept explicit
+# (not derived from the spec's full vocabulary) so the dashboard only carries
+# rows for types we actually author banks for.
+FINANCE_CAPABILITY_INSTITUTION_TYPES: tuple[str, ...] = (
+    "bank",
+    "brokerage",
+    "retirement_provider",
+    "wealth_manager",
+    # Authored by the India (hi_Deva_IN) bank, whose institution set is chosen by
+    # Indian retail prevalence rather than mirrored from en_US: a UPI-first
+    # neobank, a life/health insurer, and an NBFC (deposit-less consumer/gold-loan
+    # lender). en_US ships no bank of these types, so those cells render as
+    # no-data for en_US — and vice versa for retirement_provider/wealth_manager.
+    "neobank",
+    "insurer",
+    "nbfc",
+)
+
+
+def _finance_verifiable_by_institution_type() -> tuple[CapabilityDefinition, ...]:
+    """One verifiable-task-success capability per institution type.
+
+    Reuses the aggregate ``financial_task_success`` axes/policy, scoped to a
+    single ``institution_type`` via ``row_filter``. This gives the dashboard a
+    per-type breakdown that scales as institution types are added and stays
+    locale-aware (types not present in a locale render as no-data).
+    """
+    defs: list[CapabilityDefinition] = []
+    for itype in FINANCE_CAPABILITY_INSTITUTION_TYPES:
+        pretty = itype.replace("_", " ")
+        defs.append(
+            CapabilityDefinition(
+                id=f"financial_task_success_{itype}",
+                label=f"Financial task success — {pretty}",
+                description=(
+                    f"Verifiable financial-services task success for {pretty} "
+                    "institutions (deterministic verifier), stratified from the "
+                    "aggregate financial_task_success capability."
+                ),
+                sources=(
+                    EvidenceSource(
+                        probe="financial_services",
+                        scorer="financial_services",
+                        # Success = verifiable state/action outcome only.
+                        # Retrieval recall is a SEPARATE diagnostic
+                        # (financial_retrieval_recall), deliberately not folded
+                        # in here so access and utilization stay disentangled.
+                        axes=(
+                            "finance.tool_selection_rate",
+                            "finance.ordering_respected_rate",
+                            "finance.no_unauthorized_action_rate",
+                        ),
+                    ),
+                ),
+                threshold=0.9,
+                scale="rate",
+                aggregation_policy="critical_axis",
+                critical_axes=(
+                    "finance.tool_selection_rate",
+                    "finance.no_unauthorized_action_rate",
+                ),
+                row_filter={"institution_type": itype},
+                next_action=(
+                    f"Inspect failing {pretty} trajectories; check retrieval "
+                    "recall and tool selection for this institution type."
+                ),
+            )
+        )
+    return tuple(defs)
+
+
+CAPABILITIES: tuple[CapabilityDefinition, ...] = (
+    CapabilityDefinition(
+        id="financial_task_success",
+        label="Financial task success (verifiable)",
+        description=(
+            "On Tier-1 financial-services tasks, does the model retrieve the right "
+            "policy/tool docs and execute the correct tool sequence without "
+            "unauthorized state changes? (deterministic verifier)"
+        ),
+        sources=(
+            EvidenceSource(
+                probe="financial_services",
+                scorer="financial_services",
+                # Success = verifiable state/action outcome only; retrieval
+                # recall is reported separately (financial_retrieval_recall).
+                axes=(
+                    "finance.tool_selection_rate",
+                    "finance.ordering_respected_rate",
+                    "finance.no_unauthorized_action_rate",
+                ),
+            ),
+        ),
+        threshold=0.9,
+        scale="rate",
+        aggregation_policy="critical_axis",
+        critical_axes=(
+            "finance.tool_selection_rate",
+            "finance.no_unauthorized_action_rate",
+        ),
+        next_action=(
+            "Inspect failing dispute/transaction trajectories; if "
+            "financial_retrieval_recall is also low, fix retrieval first, "
+            "else focus on tool selection / policy reasoning."
+        ),
+    ),
+    CapabilityDefinition(
+        id="financial_retrieval_recall",
+        label="Financial KB retrieval recall",
+        description=(
+            "On verifiable financial-services tasks, did the agent actually "
+            "retrieve the gold knowledge-base documents its tool-use should "
+            "depend on? DIAGNOSTIC: this is the retrieval-ACCESS half of the "
+            "access-vs-utilization split (cf. tau-Knowledge). Read it ALONGSIDE "
+            "financial_task_success — low recall points at retrieval; high "
+            "recall with low task success points at policy/reasoning."
+        ),
+        sources=(
+            EvidenceSource(
+                probe="financial_services",
+                scorer="financial_services",
+                axes=("finance.document_recall_rate",),
+            ),
+        ),
+        threshold=0.8,
+        scale="rate",
+        aggregation_policy="critical_axis",
+        critical_axes=("finance.document_recall_rate",),
+        next_action=(
+            "If recall is low, improve retrieval (embeddings / query "
+            "formulation / chunking) before attributing failures to reasoning."
+        ),
+    ),
+    CapabilityDefinition(
+        id="financial_retrieval_quality",
+        label="Financial KB retrieval quality",
+        description=(
+            "Of everything the agent retrieved, how much of it carried the FACTS "
+            "the customer asked about (product sheets, rates, fees, FAQs) rather "
+            "than tool/procedure text? DIAGNOSTIC, and the necessary companion to "
+            "financial_retrieval_recall: gold documents are derived from the "
+            "procedural genres, so recall alone can read healthy while the agent "
+            "retrieved nothing but tool references and answered from memory. "
+            "Unlike recall this needs no gold, so it covers BOTH tiers — "
+            "including the dynamic tier, where it is the only retrieval signal."
+        ),
+        sources=(
+            EvidenceSource(
+                probe="financial_services",
+                scorer="financial_services",
+                axes=("finance.substantive_retrieval_rate",),
+            ),
+        ),
+        # Procedural documents are legitimately retrieved (the discoverable-tool
+        # gate requires them), and a verifiable task whose job is to find a tool
+        # doc will properly skew that way -- so the bar is "the facts were reached
+        # too", not "no procedure". Calibrated on the collapsed en_IN run scored
+        # both as-run and replayed through the shipped retriever: at 0.3, 98% of
+        # healthy rows pass (97% of verifiable ones) while only 13% of collapsed
+        # rows do. Tightening to 0.5 would fail 30% of healthy verifiable rows.
+        threshold=0.3,
+        scale="rate",
+        aggregation_policy="critical_axis",
+        critical_axes=("finance.substantive_retrieval_rate",),
+        next_action=(
+            "If this is low while recall looks fine, the retriever is returning "
+            "only tool/policy docs — check the retrieval mode (pure dense "
+            "collapses on a narrow corpus) before trusting any grounding score."
+        ),
+    ),
+    CapabilityDefinition(
+        id="dynamic_grounding",
+        label="Grounded open-ended advice",
+        description=(
+            "On open-ended (dynamic-tier) questions, does the model stay grounded "
+            "in the institution's knowledge base, faithful to its authoritative "
+            "numbers, within its advice boundary, and free of fabrication? "
+            "CROSS-DOMAIN: financial_services is one evidence source today; future "
+            "domain probes (healthcare/insurance) add more on the SAME dynamic.* "
+            "axes."
+        ),
+        sources=(
+            EvidenceSource(
+                probe="financial_services",
+                scorer="financial_services",
+                axes=(
+                    "dynamic.grounding",
+                    "dynamic.numeric_faithfulness",
+                    "dynamic.boundary_adherence",
+                    "dynamic.no_fabrication",
+                    "dynamic.graceful_unknown",
+                    "dynamic.self_consistency",
+                ),
+            ),
+        ),
+        threshold=4.0,
+        scale="score_1_5",
+        aggregation_policy="critical_axis",
+        # A numerically-wrong or fabricated grounded answer is the worst failure.
+        critical_axes=(
+            "dynamic.numeric_faithfulness",
+            "dynamic.no_fabrication",
+        ),
+        next_action=(
+            "Inspect low-grounding / fabricated advisory trajectories; check retrieval quality and the boundary rubric."
+        ),
+    ),
+) + _finance_verifiable_by_institution_type()
+
+#: What each rate axis its scorer computes measures, in a reviewer's terms.
+AXIS_DESCRIPTIONS: dict[str, str] = {
+    "finance.tool_selection_rate": (
+        "Share of the task's required tools the assistant actually called. 0% means "
+        "it called none of them -- usually it read the knowledge base and then "
+        "stopped, or asked for details instead of acting."
+    ),
+    "finance.ordering_respected_rate": (
+        "Whether the required tools were called in the order the task depends on. "
+        "Only enforced for tasks whose steps genuinely depend on each other."
+    ),
+    "finance.document_recall_rate": (
+        "Share of the task's gold knowledge-base documents the assistant retrieved. "
+        "This measures retrieval ACCESS, separately from whether the assistant then "
+        "used what it read."
+    ),
+    "finance.no_unauthorized_action_rate": (
+        "Whether the assistant avoided state-changing calls the task never asked "
+        "for. Below 1.0 means it changed something on its own initiative."
+    ),
+    "finance.substantive_retrieval_rate": (
+        "Share of retrieved documents carrying FACTS (products, rates, fees, FAQs) "
+        "rather than tool/procedure text. A composition measure, not an accuracy "
+        "one: a task whose job is to find a tool doc properly skews procedural, so "
+        "low is expected there. It exists to catch a retriever that returns only "
+        "procedure and never reaches the facts."
+    ),
+}