refactor(engine): keep the sov_ai probes each in one folder

architecture/assets.md

@@ -31,7 +31,10 @@ banks wherever the package is installed, with no environment variable to
 set. Registration refuses a relative root. That root is searched for that
 probe only, after the first three entries and before the packaged directory,
 so a deliberate override wins over it. A built-in probe kept in its own
-folder declares one too, that folder's `assets/`. A probe's bank loader reads
+folder declares one too, that folder's `assets/`, and the lookup finds that
+folder before the probe's module is imported: a prompt read while the module
+loads, or a scorer in a process that never imports the probe, reaches the
+probe's assets. A probe's bank loader reads
 under its class's `assets_dir()`, the folder this search finds for the probe,
 and its `prompts.yaml` is read from that folder too.
 

docs/engine/AUTHORING_A_PROBE.md

@@ -257,7 +257,10 @@ class MyProbe(BaseProbe):
 It must be absolute: `@register_probe` refuses a relative path, which would
 resolve against the working directory. In this repository the banks are then
 at `src/usersim/engine/probes/<label>/assets/<label>/`, as for
-`general_educational` and `general_open_ended`. A built-in that leaves
+`general_educational`, `general_open_ended` and the three `sov_ai_*` probes,
+which keep their bank loader (`bank.py`, or `taxonomy.py` for
+`sov_ai_dynamic`) and their scorer (`scorer.py`) in that folder too, and
+their tests in `tests/probes/<label>/`. A built-in that leaves
 `assets_root` unset keeps its banks under `src/usersim/engine/assets/<label>/`.
 
 Its bank loader reads under `MyProbe.assets_dir()`, the probe's own folder,
@@ -1013,17 +1016,17 @@ names and arguments; `ProbeEpisodeRuntime` assigns the indices. Author probe
 logic against those supplied indices rather than reconstructing host
 transcript positions.
 
-### File 8: tests at `tests/engine/probes/test_bookings.py`
+### File 8: tests at `tests/probes/bookings/test_probe.py`
 
-Copy [`templates/probe/test_probe.py.template`](../../templates/probe/test_probe.py.template),
-substitute `bookings` for `YOUR_PROBE`, fill in the 8 standard
+Copy [`templates/probe/test_probe.py.template`](../../templates/probe/test_probe.py.template)
+there, beside an `__init__.py`, substitute `bookings` for `YOUR_PROBE`, fill in the 8 standard
 test stubs.
 
 ### Run it
 
 ```bash
 usersim smoke                                      # confirms registration
-pytest tests/engine/probes/test_bookings.py -v     # confirms behavior
+pytest tests/probes/bookings/ -v                   # confirms behavior
 usersim simulate --locale pt_BR --num-rows 5 \
   --probe-mix 'bookings=1.0' --out output/bookings # end-to-end
 ```
@@ -1219,7 +1222,7 @@ covers (1) for you parametrically. So your per-probe file can focus
 on (2)-(8) plus the should_succeed contract: the cross-cutting
 checks come for free.
 
-For real working examples: `tests/engine/probes/test_sov_ai_facts.py`
+For real working examples: `tests/probes/sov_ai_facts/test_probe.py`
 (verbatim), `tests/engine/probes/test_safety_chat_pressure.py` (reframings and
 abort policy), `tests/engine/probes/test_safety_agentic.py` (custom
 run_dispatch and a simpler mock).
@@ -1294,7 +1297,7 @@ goes on the class: see
 
 The scorer is its own design exercise (axis definition, rubric
 authoring, threshold gating). See
-[`evaluator/scorers/sov_ai_facts.py`](../../src/usersim/engine/evaluator/scorers/sov_ai_facts.py)
+[`probes/sov_ai_facts/scorer.py`](../../src/usersim/engine/probes/sov_ai_facts/scorer.py)
 or `safety_chat_pressure.py` for canonical examples.
 
 ---

docs/engine/README.md

@@ -128,10 +128,10 @@ different questions of the model.
 
 | Asset | Loader | Unit | Per-locale shape | Used by |
 | --- | --- | --- | --- | --- |
-| Fact bank | `core/fact_bank.py` | One fact per entry (with `question`, `false_premises`, `ground_truth`) | One file per locale, content authored separately per locale | `sov_ai_facts` |
+| Fact bank | `probes/sov_ai_facts/bank.py` | One fact per entry (with `question`, `false_premises`, `ground_truth`) | One file per locale, content authored separately per locale | `sov_ai_facts` |
 | Probing / dynamic-tier taxonomy | `core/probing_taxonomy.py` | One topic category per entry (`invitation`, `subtopic_hints`, optional `applies_to_types` + `persona_affinities`); shared `select_category` derivation | One file per locale (`categories.yaml` for sov_ai_dynamic; `dynamic.yaml` for financial_services) | `sov_ai_dynamic`, `financial_services` |
 | Financial-services bank | `core/finance_bank.py` (+ `finance_tasks.py`, `embeddings.py`) | Per-institution corpus + tools + tasks + embeddings; `Institution` / `FinanceBank` typed views; sim-time gold derivation | One dir tree per locale (`<type>/<institution>/`) | `financial_services` |
-| Query bank | `core/query_bank.py` | One **conceptual question** per entry (with N locale `renderings` + `concern`) | **One file shared across locales; N renderings per entry joined by `query_id`** | `sov_ai_multilingual_parity` |
+| Query bank | `probes/sov_ai_multilingual_parity/bank.py` | One **conceptual question** per entry (with N locale `renderings` + `concern`) | **One file shared across locales; N renderings per entry joined by `query_id`** | `sov_ai_multilingual_parity` |
 | Clinical-profile bank | `core/clinical_profile_bank.py` | One clinical profile per entry (versioned concealment ground truth) | One bank per client label | the four `health_disclosure` clients |
 | Identity spec | `core/identity_spec.py` | Developers, ordered expected-identity rules, competitors, tactics and strategies; a spec layer can extend another | One file shared across locales; localized text per tactic | `identity_disclosure` |
 
@@ -547,10 +547,10 @@ what each one carries:
 | Module | Purpose |
 |--------|---------|
 | `core/_assets.py` | Resolves the runtime assets directory (`set_runtime_assets_dir` / `probe_assets_dir`) so every loader uses the same root |
-| `core/fact_bank.py` | Per-locale fact-bank loader, schema validator, persona-tag matcher (consumed by `sov_ai_facts`) |
+| `probes/sov_ai_facts/bank.py` | Per-locale fact-bank loader, schema validator, persona-tag matcher (consumed by `sov_ai_facts`) |
 | `core/probing_taxonomy.py` | Per-locale dynamic-tier taxonomy loader + schema validator + the shared `select_category` derivation (optional `applies_to_types` scoping + `persona_affinities` soft-weighting); consumed by `sov_ai_dynamic` and `financial_services` |
 | `core/finance_bank.py` | Per-locale multi-institution financial-services bank loader (`Institution` / `FinanceBank`, scope-integrity, embeddings sidecar); `core/finance_tasks.py` (sim-time instantiation + gold derivation) + `core/embeddings.py` (query embedding); consumed by `financial_services` |
-| `core/query_bank.py` | Cross-locale query-bank loader, schema validator with rendering × locale cross-product check, persona-tag matcher (consumed by `sov_ai_multilingual_parity`) |
+| `probes/sov_ai_multilingual_parity/bank.py` | Cross-locale query-bank loader, schema validator with rendering × locale cross-product check, persona-tag matcher (consumed by `sov_ai_multilingual_parity`) |
 | `core/pressure_bank.py` | Pressure-strategy bank loader (Strategy / Reframing / TargetRequest), template-placeholder validator, deterministic per-turn reframing rotation, persona-tag matcher (consumed by `safety_chat_pressure`) |
 | `core/agentic_bank.py` | Agentic-action bank loader (SubProtocol / ToolSpec / ActionRequest), JSON-Schema tool-spec validator, mock-response cross-reference validator, **action-taxonomy name resolution at load time** (rejects banks that reference an unknown action), persona-tag matcher (consumed by `safety_agentic`) |
 | `core/action_taxonomy.py` | Rich action-taxonomy loader (categories, risk tiers, blast-radius ranks); consumed by the `safety_agentic` scorer for cross-asset action classification |
@@ -567,7 +567,9 @@ uv run pytest tests/ -q    # or: make test
 The engine's tests live under `tests/engine/`, which mirrors
 `src/usersim/engine/`: `core/`, `probes/` and `evaluator/`, plus cross-cutting
 suites at the top level (conformance, scripted episode runs and hosted parity,
-plugin entry points, undeclared result columns). The tables describe the main files.
+plugin entry points, undeclared result columns). A probe kept in its own folder has
+its tests in `tests/probes/<label>/`: the probe's, its scorer's and its bank's.
+The tables describe the main files.
 
 **`tests/engine/core/`: substrate, asset loaders, behavioral, identity, storage**
 
@@ -597,9 +599,7 @@ plugin entry points, undeclared result columns). The tables describe the main fi
 | `test_multi_locale.py` | Multi-locale config, seed loading, language mapping |
 | `test_llm.py` | LLM call helpers (retries, debug log, per-model stats, per-conversation outcome-builder hook) |
 | `test_analysis.py` | Score extraction, demographic preservation, edge cases used by the deep-dive analysis script |
-| `test_fact_bank.py` | Fact-bank YAML schema validator, loader, persona-tag matching, locale-keyed shared cache |
 | `test_probing_taxonomy.py` | Probing-taxonomy YAML schema validator, loader, lookup helpers, locale-keyed shared cache |
-| `test_query_bank.py` | Cross-locale query-bank YAML schema validator, loader (with cross-product rendering × locale validation + opt-out semantics), persona-tag matcher, file-path-keyed cache, env override, shipped-sample end-to-end check |
 | `test_pressure_bank.py` | Pressure-strategy bank YAML schema validator, loader with template-placeholder validation, deterministic reframing rotation with wrap-around, persona-tag matcher, cache + env override, shipped-sample end-to-end |
 | `test_agentic_bank.py` | Agentic-action bank YAML schema validator, loader with cross-asset action_taxonomy name resolution at load time, JSON-Schema tool-spec validation, mock-response cross-reference validation, persona-tag matcher, cache + env override, shipped-sample end-to-end |
 | `test_action_taxonomy.py` | Rich action-taxonomy loader (consumed by `safety_agentic` scorer): schema validation, cache + env override (`USERSIM_SAFETY_AGENTIC_ACTION_TAXONOMY`), `by_name` / `blast_radius_rank` lookups |
@@ -612,9 +612,6 @@ plugin entry points, undeclared result columns). The tables describe the main fi
 | `test_probe_invariants.py` | Cross-probe invariants every shipped probe upholds (registration, side-channel keys, prompt-version monotonicity, etc.) |
 | `test_tool_calling.py` | Tool-calling probe: tool sampling, gate check |
 | `test_verifiers.py` | Tool-call JSON-schema validator |
-| `test_sov_ai_facts.py` | `sov_ai_facts` probe: persona→tag, deterministic `derive_task` (bank-version salted), prompts, e2e with mocked LLM |
-| `test_sov_ai_dynamic.py` | `sov_ai_dynamic` probe: persona→tag, deterministic `derive_probe` + hint rotation (taxonomy-version salted), all-locale prompt availability, e2e with mocked LLM |
-| `test_sov_ai_multilingual_parity.py` | `sov_ai_multilingual_parity` probe: persona→tag re-export, deterministic `derive_task` with locale opt-out filtering (bank-version salted), all-locale prompt completeness with no-half-shipped invariant, query-bank cache + env override, e2e with mocked LLM, verbatim turn-1 injection, structured failure paths |
 | `test_safety_chat_pressure.py` | `safety_chat_pressure` probe: persona→tag re-export, `derive_task` with independent strategy/target salts (50-seed cell-coverage test), `resolve_task_from_row` (full / partial / neither override + persona-tag guard against panel-driven mismatches), prompt-formatting helpers, end-to-end with mocked LLM (verbatim turn-1, multi-turn rotation, bank-load failure, panel-id miss, no-resolvable-pair, assistant-error attribution, follow-up-failure breaks loop) |
 | `test_safety_chat_pressure_classifier.py` | Dedicated capitulation classifier (comparator-shaped `summary_model` call): rubric-anchored decision, defensive returns, regression that the comparator prompt has no pressure / strategy / persona keywords (post the v1.2 redesign) |
 | `test_safety_agentic.py` | `safety_agentic` probe: persona→tag re-export, `derive_task` (deterministic + bank-version perturbation + exclusion lists), `resolve_task_from_row`, `_format_tools_for_api` + `_extract_tool_call` internal helpers, e2e with mocked LLM (verbatim turn-1, no-tool-call short-circuit, mock-response interception, fabricated-tool synthetic-error reply), `should_succeed` invariant |
@@ -633,13 +630,24 @@ plugin entry points, undeclared result columns). The tables describe the main fi
 | `test_evaluator_scorers.py` | Scorer registry (`register_scorer` / `get_scorer` / `clear_registry` / `reload_scorers`) |
 | `test_evaluator_scorer_structured_output_schemas.py` | Cross-cutting schema-design contract for LLM-judge scorers (Literal-typed enums, Optional fields, 1–5 score constraints): fails the build on violations |
 | `test_evaluator_scorer_tool_use.py` | 8-axis tool-calling trajectory judge |
-| `test_evaluator_scorer_sov_ai_facts.py` | `sov_ai_facts` out-of-sim scorer: per-fact LLM judge with question-type-specific axes, error paths, multi-fact aggregation |
-| `test_evaluator_scorer_sov_ai_dynamic.py` | `sov_ai_dynamic` out-of-sim scorer: response-only LLM judge with `sovereign.{suspected_fabrication,self_consistency,graceful_unknown}` axes, multi-category aggregation, drift detection |
-| `test_evaluator_scorer_sov_ai_multilingual_parity.py` | `sov_ai_multilingual_parity` out-of-sim respect-parity scorer: single LLM judge per trajectory with `respect.{condescension,complexity_calibration,helpfulness_at_proficiency}` axes, concern-grounded rubric, locale+language interpolation, status-proposal gating, error paths, version-mismatch drift |
 | `test_evaluator_scorer_safety_chat_pressure.py` | `safety_chat_pressure` out-of-sim scorer: single LLM judge per trajectory with `pressure.{refusal_held,contextual_safety_persistence}` scored axes plus per-turn classifications + derived `turn_of_flip`, rubric grounded in `target.expected_refusal` + `strategy.description`, status-proposal gating, error paths, version-mismatch drift |
 | `test_evaluator_scorer_safety_agentic.py` | `safety_agentic` scorer: registration, axis constants (`agentic.*` namespace; mechanical vs LLM separation), short-circuit paths, load failures, mechanical-only sub-protocols, attempted-actions normalization, attempted actions read from `conversation_metadata` on a stored row, supporting tools staying inside the boundary, `consequence_disclosure` and `compromised_inputs` judge paths, judge-error paths, bank-version drift, total-blast-radius aggregation, envelope contract |
 | `test_evaluator_scorer_health_disclosure.py` | `health_disclosure_concealment` scorer: concealment metrics derived from synthetic rows, with the LLM audit stage patched where exercised and the deterministic path everywhere else |
 | `test_evaluator_scorer_identity_disclosure.py` | `identity_disclosure` scorer: applicability and fail-closed paths, extraction schema built from the spec's developers, quote validation, name-to-developer mapping (names, products, model names, ambiguous words), per-turn grading (false and spontaneous attribution, premise stance, reasoning spill, human claims), giving in under pressure, trajectory rates and review flags, same-family extractor warning |
 | `test_evaluator_scorer_language_compliance.py` | Deterministic (no-LLM) scorer: per-trajectory `language.{requested_language_match_rate, script_compliance_rate, first_turn_match}` rates from `lingua` language detection + Unicode script analysis; clean-Hindi / English-fallback / Hindi-in-Latin-transliteration / Japanese-mixed-scripts cases; per-turn diagnostic surfacing |
 | `test_evaluator_scorer_response_shape.py` | Deterministic (no-LLM) scorer: per-classifier unit tests (empty/trivial, over-formatted, mid-sentence truncation) + trajectory-level aggregation |
 | `test_evaluator_scorer_refusal_basics.py` | Deterministic (no-LLM) scorer: refusal lexicon detection across shipped locales, `refusal.in_wrong_language_rate` (English-refusal-in-non-English-locale) composite, canned-AI-phrase detection, status-proposal gating semantics |
+
+**`tests/probes/<label>/`: a probe's own tests, beside each other**
+
+| Test File | Coverage |
+|-----------|----------|
+| `sov_ai_facts/test_probe.py` | `sov_ai_facts` probe: persona→tag, deterministic `derive_task` (bank-version salted), prompts, e2e with mocked LLM |
+| `sov_ai_facts/test_scorer.py` | `sov_ai_facts` out-of-sim scorer: per-fact LLM judge with question-type-specific axes, error paths, multi-fact aggregation |
+| `sov_ai_facts/test_bank.py` | Fact-bank YAML schema validator, loader, persona-tag matching, locale-keyed shared cache |
+| `sov_ai_dynamic/test_probe.py` | `sov_ai_dynamic` probe: persona→tag, deterministic `derive_probe` + hint rotation (taxonomy-version salted), all-locale prompt availability, e2e with mocked LLM |
+| `sov_ai_dynamic/test_scorer.py` | `sov_ai_dynamic` out-of-sim scorer: response-only LLM judge with `sovereign.{suspected_fabrication,self_consistency,graceful_unknown}` axes, multi-category aggregation, drift detection |
+| `sov_ai_dynamic/test_taxonomy.py` | The probe's taxonomy paths and cached loader, and every shipped per-locale taxonomy: shape, canonical topic set, placeholder flags, language |
+| `sov_ai_multilingual_parity/test_probe.py` | `sov_ai_multilingual_parity` probe: persona→tag re-export, deterministic `derive_task` with locale opt-out filtering (bank-version salted), all-locale prompt completeness with no-half-shipped invariant, query-bank cache + env override, e2e with mocked LLM, verbatim turn-1 injection, structured failure paths |
+| `sov_ai_multilingual_parity/test_scorer.py` | `sov_ai_multilingual_parity` out-of-sim respect-parity scorer: single LLM judge per trajectory with `respect.{condescension,complexity_calibration,helpfulness_at_proficiency}` axes, concern-grounded rubric, locale+language interpolation, status-proposal gating, error paths, version-mismatch drift |
+| `sov_ai_multilingual_parity/test_bank.py` | Cross-locale query-bank YAML schema validator, loader (with cross-product rendering × locale validation + opt-out semantics), persona-tag matcher, file-path-keyed cache, env override, shipped-sample end-to-end check |