Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,10 @@ data/
artifacts/
output/
# Dense-retrieval vectors `gen-assets` writes beside a bank stay local.
src/usersim/engine/assets/financial_services/**/embeddings.parquet
src/usersim/engine/probes/financial_services/assets/financial_services/**/embeddings.parquet
# So do the judge scorecards `evaluate-assets` writes there.
src/usersim/engine/assets/financial_services/**/_eval_scorecard.parquet
src/usersim/engine/assets/financial_services/**/_scorecards/
src/usersim/engine/probes/financial_services/assets/financial_services/**/_eval_scorecard.parquet
src/usersim/engine/probes/financial_services/assets/financial_services/**/_scorecards/

# IDE
.vscode/
Expand Down
8 changes: 5 additions & 3 deletions architecture/assets.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,9 +92,11 @@ Each domain's region specs (`<locale>.yaml`, one per locale it can generate)
ship as package data, so all three commands find them from an installed
package without `--region-spec`.

Domains are discovered through the `usersim.asset_domains` entry point, so a
new domain does not require modifying this project. See
[plugins.md](plugins.md).
A built-in domain's generator lives beside the probe whose banks it builds:
`financial_services` keeps its generator, region specs and asset notebook in
its folder's `asset_gen/`. Other domains are discovered through the
`usersim.asset_domains` entry point, so a new domain does not require
modifying this project. See [plugins.md](plugins.md).

## Every entry declares its provenance

Expand Down
13 changes: 7 additions & 6 deletions architecture/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ the arrow pointing one direction, and a test enforces it.
| `usersim.reporting` | Turning parquets into a capability dashboard | [reporting.md](reporting.md) |
| `usersim.selection` | Curating trajectories into exportable datasets | [selection.md](selection.md) |
| `usersim.taxonomy` | The shared vocabulary of capabilities and evidence | above |
| `usersim.asset_gen` | Generating new asset banks for a domain | [assets.md](assets.md) |
| `usersim.asset_gen` | Dispatching asset-bank generation to a domain's generator | [assets.md](assets.md) |
| `usersim.testing` | Conformance checks and the test harness for extensions you write | [plugins.md](plugins.md) |

Probes get their own page: [probes.md](probes.md) for the substrate,
Expand Down Expand Up @@ -84,16 +84,17 @@ throughout, so a name tells you which one you are reading about.
src/usersim/
├── cli/ console driver; models_*.toml catalogs
├── engine/
│ ├── core/ the conversation loop, probe base classes, banks,
│ │ identity, outcomes, storage
│ ├── core/ the conversation loop, probe base classes, asset lookup,
│ │ shared bank loaders, identity, outcomes, storage
│ ├── evaluator/ judges, axes, and the scorer registry
│ ├── probes/ the shipped probes
│ ├── assets/ per-probe YAML and parquet banks
│ ├── probes/ the shipped probes, most with their banks, loaders and
│ │ scorer in their own folder
│ ├── assets/ the banks of the probes that keep none of their own
│ └── plugin.py the two Data Designer entry points
├── reporting/ capability dashboard and model comparison
├── selection/ trajectory curation and export
├── taxonomy/ capability definitions, evaluation-cell decoding
├── asset_gen/ asset-bank generation
├── asset_gen/ the asset-domain registry the asset commands dispatch through
└── testing/ conformance suites for extension authors

architecture/ this directory
Expand Down
9 changes: 5 additions & 4 deletions docs/engine/AUTHORING_A_PROBE.md
Original file line number Diff line number Diff line change
Expand Up @@ -258,10 +258,11 @@ It must be absolute: `@register_probe` refuses a relative path, which would
resolve against the working directory. In this repository the banks are then
at `src/usersim/engine/probes/<label>/assets/<label>/`, as for
`general_educational`, `general_open_ended`, the three `sov_ai_*` probes,
`safety_chat_pressure`, `safety_agentic` and `identity_disclosure`, which keep
their bank loader (`bank.py`, `taxonomy.py` for `sov_ai_dynamic`, `spec.py` for
`identity_disclosure`) and their scorer (`scorer.py`) in that folder too, and
their tests in `tests/probes/<label>/`. A built-in that leaves
`safety_chat_pressure`, `safety_agentic`, `identity_disclosure` and
`financial_services`, which keep their bank loader (`bank.py`, `taxonomy.py`
for `sov_ai_dynamic`, `spec.py` for `identity_disclosure`) and their scorer
(`scorer.py`) in that folder too, and their tests in `tests/probes/<label>/`.
A built-in that leaves
`assets_root` unset keeps its banks under `src/usersim/engine/assets/<label>/`.

Its bank loader reads under `MyProbe.assets_dir()`, the probe's own folder,
Expand Down
7 changes: 4 additions & 3 deletions docs/engine/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ different questions of the model.
| --- | --- | --- | --- | --- |
| Fact bank | `probes/sov_ai_facts/bank.py` | One fact per entry (with `question`, `false_premises`, `ground_truth`) | One file per locale, content authored separately per locale | `sov_ai_facts` |
| Probing / dynamic-tier taxonomy | `core/probing_taxonomy.py` | One topic category per entry (`invitation`, `subtopic_hints`, optional `applies_to_types` + `persona_affinities`); shared `select_category` derivation | One file per locale (`categories.yaml` for sov_ai_dynamic; `dynamic.yaml` for financial_services) | `sov_ai_dynamic`, `financial_services` |
| Financial-services bank | `core/finance_bank.py` (+ `finance_tasks.py`, `embeddings.py`) | Per-institution corpus + tools + tasks + embeddings; `Institution` / `FinanceBank` typed views; sim-time gold derivation | One dir tree per locale (`<type>/<institution>/`) | `financial_services` |
| Financial-services bank | `probes/financial_services/bank.py` (+ `tasks.py`, `embeddings.py`) | Per-institution corpus + tools + tasks + embeddings; `Institution` / `FinanceBank` typed views; sim-time gold derivation | One dir tree per locale (`<type>/<institution>/`) | `financial_services` |
| Query bank | `probes/sov_ai_multilingual_parity/bank.py` | One **conceptual question** per entry (with N locale `renderings` + `concern`) | **One file shared across locales; N renderings per entry joined by `query_id`** | `sov_ai_multilingual_parity` |
| Clinical-profile bank | `core/clinical_profile_bank.py` | One clinical profile per entry (versioned concealment ground truth) | One bank per client label | the four `health_disclosure` clients |
| Identity spec | `probes/identity_disclosure/spec.py` | Developers, ordered expected-identity rules, competitors, tactics and strategies; a spec layer can extend another | One file shared across locales; localized text per tactic | `identity_disclosure` |
Expand Down Expand Up @@ -549,7 +549,7 @@ what each one carries:
| `core/_assets.py` | Resolves the runtime assets directory (`set_runtime_assets_dir` / `probe_assets_dir`) so every loader uses the same root |
| `probes/sov_ai_facts/bank.py` | Per-locale fact-bank loader, schema validator, persona-tag matcher (consumed by `sov_ai_facts`) |
| `core/probing_taxonomy.py` | Per-locale dynamic-tier taxonomy loader + schema validator + the shared `select_category` derivation (optional `applies_to_types` scoping + `persona_affinities` soft-weighting); consumed by `sov_ai_dynamic` and `financial_services` |
| `core/finance_bank.py` | Per-locale multi-institution financial-services bank loader (`Institution` / `FinanceBank`, scope-integrity, embeddings sidecar); `core/finance_tasks.py` (sim-time instantiation + gold derivation) + `core/embeddings.py` (query embedding); consumed by `financial_services` |
| `probes/financial_services/bank.py` | Per-locale multi-institution financial-services bank loader (`Institution` / `FinanceBank`, scope-integrity, embeddings sidecar); `probes/financial_services/tasks.py` (sim-time instantiation + gold derivation) + `probes/financial_services/embeddings.py` (query embedding); consumed by `financial_services` |
| `probes/sov_ai_multilingual_parity/bank.py` | Cross-locale query-bank loader, schema validator with rendering × locale cross-product check, persona-tag matcher (consumed by `sov_ai_multilingual_parity`) |
| `probes/safety_chat_pressure/bank.py` | Pressure-strategy bank loader (Strategy / Reframing / TargetRequest), template-placeholder validator, deterministic per-turn reframing rotation, persona-tag matcher (consumed by `safety_chat_pressure`) |
| `probes/safety_agentic/bank.py` | Agentic-action bank loader (SubProtocol / ToolSpec / ActionRequest), JSON-Schema tool-spec validator, mock-response cross-reference validator, **action-taxonomy name resolution at load time** (rejects banks that reference an unknown action), persona-tag matcher (consumed by `safety_agentic`) |
Expand Down Expand Up @@ -608,7 +608,6 @@ The tables describe the main files.
| `test_probe_invariants.py` | Cross-probe invariants every shipped probe upholds (registration, side-channel keys, prompt-version monotonicity, etc.) |
| `test_tool_calling.py` | Tool-calling probe: tool sampling, gate check |
| `test_verifiers.py` | Tool-call JSON-schema validator |
| `test_financial_services.py` | `financial_services` probe: registration, multi-institution bank load and scope integrity, deterministic task instantiation and gold derivation, institution-scoped retrieval with no cross-institution leakage, the discoverable-tool gate through `run_dispatch` (mocked assistant), and the verifier scorer (good, unauthorized, skipped and dynamic-tier rows) |
| `test_health_disclosure.py` | `health_disclosure` family: the Guard's pacing gates, the `commit_move` tool schema, move extraction, and the per-client config (topics, risk, archetype, prompts, axes) |

**`tests/engine/evaluator/`: judges, scorers, registry**
Expand Down Expand Up @@ -653,3 +652,5 @@ The tables describe the main files.
| `identity_disclosure/test_spec.py` | Identity spec loader: loading, the env override and the cache; model-id resolution (a family name after any separator, declarations over patterns, hosts never deciding); name folding; layering down the asset search path and the roots probes declare (inheritance, in-place patches, probes built on top, cycles); merging and `select` / `exclude` narrowing; the content digest; validation reporting every problem at once |
| `identity_disclosure/test_capabilities.py` | The identity capability rows: their definitions, and cells built from real scorer output |
| `identity_disclosure/test_contradictions.py` | `identity_contradictions`: the turns the identity scorer labelled contradictions, with the question each answered |
| `financial_services/test_probe.py` | `financial_services` probe: registration, multi-institution bank load and scope integrity, deterministic task instantiation and gold derivation, institution-scoped retrieval with no cross-institution leakage, the discoverable-tool gate through `run_dispatch` (mocked assistant), and the verifier scorer (good, unauthorized, skipped and dynamic-tier rows) |
| `financial_services/test_e2e.py` | A finance trajectory of each tier through evaluation, selection and the capability report, with the judge mocked |
3 changes: 3 additions & 0 deletions docs/extension_api.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,9 @@ does not load the conversation loop.
| `CapabilityDefinition` | A row of the capability report and the evidence it is scored from |
| `EvidenceSource` | One probe, scorer and set of axes a capability reads |
| `AssetDomain` | A domain `usersim gen-assets` can build banks for |
| `to_model_configs` | The Data Designer model configs for the models an asset domain receives |
| `to_data_designer_kwargs` | The `DataDesigner` keyword arguments for those models' providers |
| `require_aliases` | Raises a configuration error naming the aliases a domain needs that the models lack |
| `ExecutionBackend` | Somewhere a run can be submitted, for `--backend` |
| `JobSpec` | What to run, independent of where |
| `JobHandle` | A backend's reference to work it accepted |
Expand Down
2 changes: 1 addition & 1 deletion docs/notebooks.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ to shape training records. `tests/test_extraction_example.py` pins the schema co
pairwise shapes, confirming the trajectory and evaluation columns are
sufficient to drive them.

**[`finance_assets.ipynb`](../src/usersim/asset_gen/notebooks/finance_assets.ipynb)**
**[`finance_assets.ipynb`](../src/usersim/engine/probes/financial_services/asset_gen/notebooks/finance_assets.ipynb)**
generates and vets a `financial_services` asset bank: it drives
`usersim gen-assets`, `validate-assets` and `evaluate-assets`, then previews
the result. Its generate and evaluate cells make live, billable model calls;
Expand Down
2 changes: 1 addition & 1 deletion notebooks/01_simulate.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -418,7 +418,7 @@
"its own, task set included, before you add it to `LOCALES` (see\n",
"[Authoring a new locale](../src/usersim/engine/probes/financial_services/README.md#authoring-a-new-locale)).\n",
"The banks are produced offline (see\n",
"[`src/usersim/asset_gen/notebooks/finance_assets.ipynb`](../src/usersim/asset_gen/notebooks/finance_assets.ipynb))\n",
"[`src/usersim/engine/probes/financial_services/asset_gen/notebooks/finance_assets.ipynb`](../src/usersim/engine/probes/financial_services/asset_gen/notebooks/finance_assets.ipynb))\n",
"and managed with three CLI commands:\n",
"\n",
"```bash\n",
Expand Down
14 changes: 5 additions & 9 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -178,24 +178,20 @@ include = ["usersim*"]
"probes/*/assets/**/*.json",
"probes/*/assets/**/*.parquet",
"probes/*/assets/**/*.md",
# Every other YAML, the region specs an asset domain beside its probe reads
# among them: without those an installed package reports no locales to generate.
"**/*.yaml",
]
"usersim.asset_gen" = [
# The region specs gen-assets, validate-assets and evaluate-assets read
# when no --region-spec is given. Without them an installed package
# reports no locales to generate.
"*/region_spec/*.yaml",
]

[tool.setuptools.exclude-package-data]
# Retrieval embeds each query with the model that produced the corpus vectors,
# so shipped vectors would tie every install to one embedding model. The
# vectors `gen-assets` writes beside a bank stay local, as do the judge
# scorecards `evaluate-assets` writes there.
"usersim.engine" = [
"assets/financial_services/**/embeddings.parquet",
"assets/financial_services/**/_eval_scorecard.parquet",
"assets/financial_services/**/_scorecards/*",
"probes/financial_services/assets/financial_services/**/embeddings.parquet",
"probes/financial_services/assets/financial_services/**/_eval_scorecard.parquet",
"probes/financial_services/assets/financial_services/**/_scorecards/*",
]

[tool.pytest.ini_options]
Expand Down
23 changes: 6 additions & 17 deletions src/usersim/asset_gen/__init__.py
Original file line number Diff line number Diff line change
@@ -1,22 +1,11 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

"""asset_gen — offline Data Designer generation of financial_services assets.
"""asset_gen: the generic dispatch for building asset banks offline.

Repo-root package (parallels ``reporting/`` and ``selection/``) that authors the
committed ``assets/financial_services/<locale>/`` banks the runtime probe/scorer
consume. It imports the plugin (for the shared gold-derivation resolver + bank
schema) but the plugin never imports it — the dependency arrow is one-way.

Layout:

- ``asset_gen/financial_services/spec.py`` — the region_spec contract as Pydantic
models (the shared vocabulary + validation).
- ``asset_gen/financial_services/pipeline.py`` — the whole Data Designer flow:
schemas -> prompt -> seed plan -> column DAG -> explode -> serialize.
- ``asset_gen/financial_services/region_spec/*.yaml`` — the committed structured
seed per region (the generation input; data, not code).

See the probe design doc:
``src/usersim/engine/probes/financial_services/README.md``.
``registry.py`` holds :class:`AssetDomain` and the built-in domains, and
``usersim gen-assets``, ``validate-assets`` and ``evaluate-assets`` dispatch
through it. A domain's generator lives beside the probe whose banks it builds:
``usersim.engine.probes.financial_services.asset_gen`` authors the committed
``assets/financial_services/<locale>/`` banks the probe and its scorer read.
"""
Loading
Loading