[3mei]

What changed and why

sov_ai_facts, sov_ai_dynamic and sov_ai_multilingual_parity each live in one folder: the probe module probe.py, its bank loader, its scorer scorer.py and its banks in assets/<label>/, the folder the class names as its assets_root. Each probe's tests are in tests/probes/<label>/. That is the layout a probe shipped in its own package uses; the probes' capability entries stay in taxonomy/capabilities.py. Labels, scorer names, banks, stored columns and the caches each probe shares with its scorer are unchanged.

Builds on #120.

• A built-in's own folder before its class registers (first commit). A built-in kept in its own folder declares usersim/engine/probes/<label>/assets/. Until its class registers, the asset lookup takes that folder as the declared root when it holds the probe's folder. So a prompt read while the probe's module loads, as general_educational and general_open_ended read theirs, and a scorer in a process that never imports the probe reach the probe's assets.
• One folder per probe (second to fourth commits). One commit per probe: generator.py renamed to probe.py, the bank loader moved in as bank.py, the scorer as scorer.py, the banks into assets/<label>/, and the tests into tests/probes/<label>/ as test_probe.py, test_scorer.py and test_bank.py. BUILTIN_PROBE_MODULES, DEFAULT_SCORER_MODULES, the capability source references and the docs name the new modules, and the shared-code label allowances drop the moved files.
  • core/probing_taxonomy.py stays shared, since financial_services uses its format, select_category and its cache. Only the three helpers that name sov_ai_dynamic's files move, to sov_ai_dynamic/taxonomy.py, and the shared module names no probe. Their tests and the checks of every shipped taxonomy move to tests/probes/sov_ai_dynamic/test_taxonomy.py.
  • The scorers' log lines read <label> scorer:.
• Where a new probe's tests go (fifth commit). The test template, the probe template README and the authoring guide's worked example put a new probe's tests in tests/probes/<label>/, beside an __init__.py.

How it was verified

At each commit, on that commit's tree:

$ make check-all && make test && make smoke
All checks passed!
354 files already formatted
4653 passed in 92.98s (0:01:32)
All checks passed.

The test count is 4653 at every commit.

At the head:

$ make check-license-headers
🎉 All files have correct license headers!
$ make check-doc-links
✓ every relative link, and every link into this repository's main branch, resolves
$ make check-notebooks-clean
OK 4 notebook(s) carry no outputs
$ make check-dist
✓ installed package runs from a clean environment
$ make check-extensions
all extension seams work against an installed distribution

• The first commit's test fails without the fix:

  E         FileNotFoundError: Seed file not found for probe='general_educational' kind='subjects' locale='en_US': checked .../src/usersim/engine/assets/general_educational/en_US/subjects.json and .../src/usersim/engine/assets/general_educational/en_US/subjects.json
  FAILED tests/engine/core/test_assets_discovery.py::TestBeforeTheProbeIsImported::test_its_banks_and_prompts_are_found_in_a_fresh_process
  

• make health-check ran against a custom provider config for each moved probe, two turns each: sov_ai_facts in pt_BR, sov_ai_dynamic in ja_JP and sov_ai_multilingual_parity in fr_FR. Each ended with ✓ simulate, eval and report all completed, and every model's requests show failed=0.

• usersim simulate, then usersim eval naming each probe's scorer, ran against the same config, sov_ai_facts in de_DE, sov_ai_dynamic in ko_KR and sov_ai_multilingual_parity in hi_Deva_IN:

  judge: scored
  sov_ai_facts: ok (scores=['sovereign.accuracy', 'sovereign.completeness', 'sovereign.fabrication'], error=None)
  judge: scored
  sov_ai_dynamic: ok (scores=['sovereign.graceful_unknown', 'sovereign.self_consistency', 'sovereign.suspected_fabrication'], error=None)
  judge: scored
  sov_ai_multilingual_parity: ok (scores=['respect.complexity_calibration', 'respect.condescension', 'respect.helpfulness_at_proficiency'], error=None)
  

Results impact

Conversations, prompts and scores are unchanged. The setup fingerprint of a simulate --inputs run hashes the contents of every asset root, so it changes, as after any bank edit, and such a run gives its rows new trajectory ids rather than skipping rows it wrote before this change.