[3mei]

What changed and why

A probe declares on its own class what a run reads from it before any row runs: which locale banks the pre-flight loads, which themes its rows draw from, and which models it calls beyond the conversation's own. A probe shipped in its own package gets the same pre-flight, themes and models as a built-in one, with no shared module naming it.

• Locale pre-flight on the class (first commit). Three optional declarations on BaseProbe:

  • locale_bank_check loads one locale's bank folder and raises when no row could run on it. When it is set, the pre-flight loads the bank each run locale runs on, before any model call. financial_services, sov_ai_facts and sov_ai_dynamic declare theirs.
  • bank_locale(locale, probe_dir), a classmethod, names that bank. The default runs an India language variant on its own bank when it has one and on en_IN's otherwise; financial_services overrides it, since its bank for a locale counts once it has tasks.
  • authoring_guide is the link a refusal gives for building the probe's bank in another locale.

  A refusal links the feature-request form when a probe that ships with User Sim is missing a locale, and each probe's own guide when it has one. assert_probe_conforms checks the three declarations. docs/engine/AUTHORING_A_PROBE.md (step 3), architecture/assets.md, architecture/probes.md and templates/probe/ describe them. In docs/extensions.md, the example probe declares its check.

• Themes on the class (second commit). BaseProbe.themes(locale), a classmethod, returns the themes a probe's rows draw from, as mappings.

  • A run samples one per row into the theme column. theme() reads the row's theme back as a mapping, whether the row holds JSON text, as a run stores it, or the mapping a test passes to run_episode.
  • general_open_ended and general_educational read theirs from their seed files. tool_calling takes its themes and toolsets from the run's settings. A probe that returns None, the default, reads no theme and gets a placeholder.
  • A probe's themes are read with the run's asset root installed, as its rows read their banks, so a probe whose assets ship in its own package finds them under assets_root.
  • A run refuses a probe that lists no themes for a run locale.

• Models on the class (third commit). BaseProbe.model_aliases(cfg), a classmethod, names the aliases of the models a probe calls beyond the user, assistant, tool-response, judge and summary models.

  • Every row's models holds each declared alias the models file defines. An alias the file lacks is left out, and the probe handles its absence; financial_services searches lexically without its embedding model.
  • The --inputs setup fingerprint counts the settings of each declared model.
  • financial_services declares the model that embeds its search queries, named by its finance_embedding_model_alias setting, so the generator and the setup fingerprint name no probe.
  • assert_probe_conforms checks that it is a classmethod returning a tuple of names.

All three are additions with defaults that keep the current behaviour; no existing hook or attribute changes. The make check-extensions fixture probe declares a locale check, a guide, themes and a model, and the check verifies each from an installed package.

How it was verified

$ make check-all && make test && make smoke
All checks passed!
343 files already formatted
4562 passed in 56.13s
[OK]   probe registry        14 probe(s) registered: ['financial_services', 'general_educational', 'general_open_ended', 'health_decision_support_disclosure', 'health_general_disclosure', 'health_therapy_disclosure', 'health_triage_disclosure', 'identity_disclosure', 'safety_agentic', 'safety_chat_pressure', 'sov_ai_dynamic', 'sov_ai_facts', 'sov_ai_multilingual_parity', 'tool_calling']
All checks passed.

$ make check-license-headers
🎉 All files have correct license headers!
$ make check-doc-links
✓ every relative link, and every link into this repository's main branch, resolves
$ make check-notebooks-clean
OK 4 notebook(s) carry no outputs
$ make check-dist
✓ both release archives pass
✓ installed package runs from a clean environment
$ make check-extensions
      ok  pre-flight passes a locale with a bank
      ok  pre-flight refuses a locale without one, linking the probe's guide
      ok  themes come from the contributed assets
      ok  model_aliases joins the run's models
      ok  usersim.testing conformance
all extension seams work against an installed distribution

• Each commit passes make check-all && make test && make smoke on its own: 4547 tests after the first, 4556 after the second.

• New tests:

  • A probe registered from outside the package gets the locale pre-flight from its class, with its own guide and no feature-request link.
  • The pre-flight checks the bank bank_locale picks.
  • A probe's themes, read under its own asset folder, reach its rows; a probe without themes gets the placeholder; an empty list is refused.
  • theme() reads JSON text and a mapping alike.
  • A row gets each declared model the registry has, and one the registry lacks is left out.
  • Changing a declared model's settings changes the setup fingerprint.
  • assert_probe_conforms reports each unusable declaration.
  • The guide-link test reads the links from the probe classes.

  The tests for a probe's own pre-flight, themes and models fail against the code before the commit that adds them.

• For the built-in probes, the declared aliases are the alias the setup fingerprint already counted, so --inputs fingerprints are unchanged; this was checked for the default, a renamed and an empty finance_embedding_model_alias.

• The example package in docs/extensions.md matches the guide's code blocks byte for byte. Its four tests pass, and a run in fr_FR is refused before any model call with acme_recipes can't run fr_FR: no bank. Locales that can run: en_US.

• make health-check against a custom provider config:

  • with HEALTH_PROBE=general_open_ended, a two-turn row drew its theme and completed;
  • with HEALTH_PROBE=financial_services, a two-turn row completed.

  Both evaluations reported failed=0, and both reports rendered.

• The financial_services scorer, run by hand with --scorers financial_services on an eight-turn row against the same config:

  judge: scored
  financial_services: ok (scores=['finance.document_recall_rate', 'finance.no_unauthorized_action_rate', 'finance.ordering_respected_rate', 'finance.substantive_retrieval_rate', 'finance.tool_selection_rate'], error=None)
  

• A live financial_services run against the same config (two rows, three turns) made 12 kb_search calls with no lexical-fallback warning, so the query embeddings came from the declared model.

Results impact

None. The built-in probes draw the same themes from the same seed files, the pre-flight passes and refuses the same runs for them, every row gets the same models, and the pinned trajectory IDs and --inputs setup fingerprints are unchanged.