[3mei]

What changed and why

A probe declares the seed table its rows sample, so the run setup names no probe; a tool_calling row's theme is its toolset; new probes get a test template that runs through run_episode with nothing patched; and the run manifest records the extensions installed and where each probe read its assets.

Builds on #133.

• tool_calling's themes in its assets (first commit). assets/tool_calling/en_US/themes.json holds them, read by the probe's themes() with load_seeds, as the general probes read their topics. build_simulator_config_builder and materialize_episode_inputs take no tool_calling_themes, and the run manifest's scope records none; a manifest that holds the key still loads. The catalog README says the themes reach a row only when its toolset has no category: the probe takes a row's theme from its toolset's category and name, and every shipped toolset has one.
• Seed tables (second commit). BaseProbe.seed_dataset names a table in the probe's asset folder, <name>.parquet or <name>.jsonl, that a run attaches as its seed dataset, so each row gets the columns of one of its rows. tool_calling declares its toolset catalog, toolsets_seed, and reads its tools from the row's tools column unless the run sets tools_column. resolve_seed_dataset_for_mix finds the table of the probe in the mix that declares one and refuses a mix with two; --toolset-seed-path and toolset_seed_path= keep their names and override that table. usersim smoke and assert_probe_conforms check the table is there, the extension fixture declares one so make check-extensions resolves it from an installed package, and the tool-calling template declares one. The CLI pipeline names tool_calling only in the default probe mix.
• The test template (third commit). Every episode runs through usersim.testing.run_episode with scripted models, nothing patched: conformance, a run end to end, the row's area in the user model's instructions, the columns the probe writes, a declining assistant scripted by a ScriptedModel subclass, the placeholder warning, and hosted parity. AUTHORING_A_PROBE.md and the template README describe the same tests, and the README's checklist says a probe's scorer and capabilities live in its own package.
• Docs (fourth commit). The README links docs/extensions.md and docs/extension_api.md, and docs/extensions.md says how a package builds against a change in User Sim before a release has it.
• The run manifest (fifth commit). code.extensions records each installed package that advertises a User Sim entry point, with its version, and asset_versions.probe_assets the folder each probe in the mix read its assets from.
• Small fixes (sixth to eighth commits). The health probes' asset_problems loads each client's clinical-profile bank, so usersim smoke checks all 14 probes' assets; the extension template and guide set asyncio_default_fixture_loop_scope, so the template's tests run without a deprecation warning; and tests/_asset_audit.py reads SHIPPED_LOCALES.
• A tool_calling row's theme is its toolset (ninth commit). The user prompt takes a row's theme from its toolset's category and name. Every shipped toolset has a category, so a theme drawn from themes.json reaches no conversation, while the row stores it in its theme column, where the preview and exports show it beside an unrelated toolset. The probe lists no themes, so its rows hold the run's __not_used__ placeholder, as the rows of every probe that draws its area from its own data do, and themes.json is removed. A catalog without categories gives each row its toolset's name; the preview shows the toolset and its category. No prompt, judge or scorer reads the theme column of these rows, so shipped conversations, scores and trajectory ids stay the same.

Changes for callers

• build_simulator_config_builder and materialize_episode_inputs in usersim.cli._pipeline, and build_run_manifest and write_simulator_manifest in usersim.engine.core.manifest, take no tool_calling_themes, and a run manifest's scope has no tool_calling_themes key.
• resolve_toolset_seed_path, resolve_toolset_seed_for_mix and TOOLSET_SEED_SUFFIXES in usersim.cli._pipeline become resolve_seed_dataset_path, resolve_seed_dataset_for_mix and SEED_DATASET_SUFFIXES (in usersim.engine.core.probes). DEFAULT_TOOL_CALLING_THEMES has no counterpart, since a tool_calling row's theme is its toolset.
• A tool_calling row's theme column holds __not_used__; its toolset_category and toolset_name columns say what the conversation is about.

How it was verified

At each commit, on that commit's tree:

$ make check-all && make test && make smoke
All checks passed!
381 files already formatted
4707 passed in 103.70s (0:01:43)
All checks passed.

That is the first commit's tree. The later ones pass with 4711, 4711, 4711, 4715, 4724, 4724, 4724 and 4731, as each adds its tests. The pre-commit hooks pass on every file the commits touch, at each commit.

At the head:

$ make check-license-headers
🎉 All files have correct license headers!
$ make check-doc-links
✓ every relative link, and every link into this repository's main branch, resolves
$ make check-notebooks-clean
OK 4 notebook(s) carry no outputs
$ make check-dist
✓ installed package runs from a clean environment
$ make check-extensions
      ok  seed_dataset resolves a contributed seed table
all extension seams work against an installed distribution

• The probe test template, filled in for general_open_ended in a scratch folder (the placeholder-warning test left out, since that probe has no bank), passed its six tests.

• make health-check ran against a custom provider config for tool_calling in en_US and for the default probe, sov_ai_dynamic in ja_JP. Both ended with ✓ simulate, eval and report all completed, and every model's requests show failed=0.

• usersim simulate with the default mix (tool_calling and the two general probes), then usersim eval --scorers auto, against the same config: each tool_calling row drew a toolset from the attached catalog, stores __not_used__ as its theme, and opens with a request in its toolset's area; the manifest records each probe's asset folder and the seed table and no tool_calling_themes; the judge scored every row and every scorer reported ok.

  tool_calling | status True | theme __not_used__ | toolset <NA> / Fitness Tracker
      first user turn: 'Hey, can you check my workout history and tell me what I've actually been consistent with lately? ...'
  tool_calling | status True | theme __not_used__ | toolset news search / News
      first user turn: "Could you look up what's going on around Perth Amboy and Middlesex County this week? ..."
  judge: scored on 4 of 4 rows
  language_compliance, refusal_basics, response_shape, tool_use: ok on 4 of 4 rows