[3mei]

What changed and why

An extension imports from one module, usersim.extension, and its tests import from usersim.testing. A reference page lists every name the two export. Tests keep the probe reference tables, that page and the examples an author copies in step with the code, and limit how often shared code names a built-in probe, so what a probe needs from the core goes on its class.

Builds on #112.

• Every probe hook and declaration documented (first commit). architecture/probes.md lists every public method of BaseProbe, BankBackedProbe and the six mixins with where it is called, and every class attribute and classmethod with its default and what reads it.

  • tests/test_reference_docs.py fails when a public name has no row, when a row names nothing the classes define, when a method sits outside the hook table, or when a documented default differs from the class.
  • should_inline_judge_user_turn and should_inline_judge_assistant_turn are defined on BaseProbe, but nothing calls them. The tables, the authoring guide, the docstrings and the agentic template say so.

• Probe labels in shared code (second commit). tests/test_probe_names_in_shared_code.py counts the built-in probe labels written as strings in src/usersim/ outside engine/probes/, docstrings excluded, file by file: 88 in 24 files.

  • A file over its allowance fails with each occurrence listed. An allowance above the count fails too, so a label taken out cannot come back.
  • architecture/probes.md points at the test, and the authoring guide says a scorer registered under its probe's label adds its module to the allowances.

• usersim.extension (third commit). 59 names:

  • the probe classes, mixins and register_probe;
  • the conversation and outcome types;
  • the user, assistant and judge model aliases, acall_llm and the run's settings;
  • prompt, locale and tool-call helpers;
  • register_scorer and the empty-cell helpers;
  • the capability, asset-domain and backend types.

  Each name is imported from the module that defines it on first use. usersim --help imports every installed asset domain and backend to list them, and an extension that imports AssetDomain from here loads the asset-domain registry, not the conversation loop or Data Designer. Type checkers read ordinary imports and a literal __all__.

  • docs/extension_api.md lists every name in usersim.extension and usersim.testing, and says the listed names can change in any release.
  • tests/test_extension_api.py checks both modules against the page, each name against the module that defines it, the type-checker imports against the runtime map, and, in a subprocess, that importing the asset-domain and backend types leaves the loop unloaded.

• Examples import the public API (fourth commit). docs/extensions.md, architecture/plugins.md, the authoring guide, the probe templates and the make check-extensions fixture import from usersim.extension and usersim.testing, and every import in them is absolute, as ruff requires.

  • The authoring guide's worked example loads its bank through the probe's assets_dir(), from a folder named for its label.
  • The test template checks registration with assert_probe_conforms.
  • A test fails when one of these examples imports another usersim module, apart from a listed few: the modules of the in-tree example, the test template's standard tests, which patch the loop's model calls, and the fixture's probe built on identity_disclosure.
  • CONTRIBUTING.md points at the reference page.

• A multilingual tool-calling example (fifth commit). docs/extensions.md builds an extension package, my_bookings, and the authoring guide's worked example builds the same probe in tree, as bookings. A simulated user asks an assistant that has a book_appointment(date, time, people) tool to book a tour.

  • The en_US and pt_BR banks both open with 04/03/2027, which is 3 April to the first user and 4 March to the second. The user gives the number of people only when the assistant asks.
  • The probe answers the tool call through ToolExecutionMixin, records the first booking with how many user turns came before it, and lets a conversation end only once the assistant has booked.
  • The extension's scorer checks each argument against what the user meant, and whether the assistant asked before booking. Its tests run an episode in pt_BR offline and score a month-first reading of the Brazilian date as wrong.
  • The in-tree version adds what a built-in probe needs: the user's prompts in each language in a LocalePromptPack, a WarningKind member, the BUILTIN_PROBE_MODULES entry and an asset_problems check.

• One example prefix (sixth commit). Every example of a name an extension adds starts with my_, the prefix of the package docs/extensions.md builds: the warning kind in architecture/plugins.md and the authoring guide, the error for a malformed warning kind, and the warning kinds, metadata keys and scorer names in the tests that stand in for an extension.

How it was verified

At the head commit:

$ make check-all && make test && make smoke
All checks passed!
349 files already formatted
4631 passed in 47.68s
All checks passed.

$ make check-license-headers
🎉 All files have correct license headers!
$ make check-doc-links
✓ every relative link, and every link into this repository's main branch, resolves
$ make check-notebooks-clean
OK 4 notebook(s) carry no outputs
$ make check-dist
✓ both release archives pass
✓ installed package runs from a clean environment
$ make check-extensions
verifying extension seams against an installed distribution
      ok  usersim.probes
      ok  usersim.scorers
      ok  deterministic scorer is rescorable
      ok  usersim.asset_domains
      ok  asset_domains: built-in survives
      ok  usersim.backends
      ok  usersim.capabilities
      ok  capability reaches the report
      ok  usersim.commands
      ok  commands: built-ins survive
      ok  no asset path set
      ok  assets_root resolves a contributed bank
      ok  output_columns joins the run's columns
      ok  pre-flight passes a locale with a bank
      ok  pre-flight refuses a locale without one, linking the probe's guide
      ok  themes come from the contributed assets
      ok  model_aliases joins the run's models
      ok  default_scorers join --scorers auto
 ────────────────────────────────────────── Probe Inputs ──────────────────────────────────────────
  Marker: yes
 ──────────────────────────────────────────────────────────────────────────────────────────────────
      ok  preview_fields reach the preview
      ok  usersim.testing conformance
      ok  probe built on identity_disclosure
      ok  identity spec layer loads
      ok  identity layer extends the shipped spec
      ok  identity layer adds its own model
      ok  identity layer inherits the shipped rules
      ok  identity probe conformance
      ok  usersim smoke reports the package
      ok  usersim smoke runs the contributed asset checks

all extension seams work against an installed distribution

• Each commit passes make check-all && make test && make smoke on its own: 4622 tests after the first, 4624 after the second, 4630 after the third, and 4631 after each of the last three.

• Each new test fails on the breakage it guards against, checked by breaking a copy:

  • a missing, stale or misfiled row in the probe tables, and a wrong default;
  • a file over its probe-label allowance, an allowance above its count, a new file naming a probe, and a wrong source root (a label in a docstring is not counted);
  • a name missing from the reference page, a row naming nothing, a name mapped to the wrong module, a type-checker import missing, and an import that loads the loop up front;
  • a private import put back in a guide, a usersim import in a code block that does not parse, a listed exception no example makes, and a name usersim.extension does not export.

• The new tests pass three times in parallel, and serially after the tests that register and restore probes.

• The example package in docs/extensions.md matches the guide's code blocks byte for byte, and its five tests pass offline. Installed next to usersim, it is listed by usersim smoke as the guide shows:

  [OK]   extensions            my-bookings 0.1.0: 2 entry point(s)
  

• The in-tree example, copied file for file from the authoring guide into a copy of the source, passes assert_probe_conforms, runs an episode in each locale with scripted models and through a host with the same result, and reports a missing pt_BR bank from asset_problems. A --dry-run in another locale stops before any model call:

  - bookings can't run fr_FR: no bank. Locales that can run: en_US, pt_BR
  

• The in-tree example ran live against a custom provider config: three pt_BR rows at four turns, every request succeeding, each row recording the booking the assistant made.

• make health-check was not run: no shipped model, provider, prompt, scorer or stored data changes.