Skip to content

feat(extension): export the names built-in probes share - #132

Open
3mei wants to merge 2 commits into
yev/feat/tool-calling-credentialsfrom
yev/feat/shared-names-in-extension-api
Open

3mei wants to merge 2 commits into
yev/feat/tool-calling-credentialsfrom
yev/feat/shared-names-in-extension-api

Conversation

@3mei

@3mei 3mei commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

What changed and why

A built-in probe imports what it shares with the other probes from usersim.extension, as an extension does, so six of the eleven built-in packages import nothing from internal usersim modules.

Builds on #131.

  • Shared helpers in the public API (first commit). usersim.extension exports the nine names that two or more built-in probes take from internal modules: dispatch_episode, probe_assets_dir, load_seeds, LocalizedText, LocalizedTextError, persona_to_tags, asset_locale, india_variant and recorded_value. docs/extension_api.md lists each with what it is for, among them a section for banks and assets, and the worked example in AUTHORING_A_PROBE.md imports dispatch_episode from usersim.extension with the rest of its names.
  • Built-ins import them there (second commit). Each built-in package imports from usersim.extension every name that usersim.extension exports from the module the package imported it from, at the top of a module and inside a function alike. general_educational, general_open_ended, health_disclosure, safety_chat_pressure, sov_ai_facts and sov_ai_multilingual_parity import nothing from other usersim modules, and the allowances in tests/test_probe_private_imports.py are each package's count: financial_services 12, identity_disclosure 6, safety_agentic 3, sov_ai_dynamic 7 and tool_calling 1. The names left are each used by one probe, apart from the probing taxonomy that financial_services and sov_ai_dynamic share.

No behaviour changes: the same objects are imported under the same names.

How it was verified

At each commit, on that commit's tree:

$ make check-all && make test && make smoke
All checks passed!
374 files already formatted
4704 passed in 99.20s (0:01:39)
All checks passed.

That is the first commit's tree; the second passes with the same count.

At the head:

$ make check-license-headers
🎉 All files have correct license headers!
$ make check-doc-links
✓ every relative link, and every link into this repository's main branch, resolves
$ make check-notebooks-clean
OK 4 notebook(s) carry no outputs
$ make check-dist
✓ installed package runs from a clean environment
$ make check-extensions
all extension seams work against an installed distribution
  • Each of the 70 modules in the built-in probe packages imports cleanly on its own in a fresh interpreter, so no import order in the suite hides a cycle through usersim.extension.
  • make health-check ran against a custom provider config for the default probe, sov_ai_dynamic in ja_JP. It ended with ✓ simulate, eval and report all completed, and every model's requests show failed=0.

3mei added 2 commits November 4, 2026 08:46
usersim.extension exports dispatch_episode, probe_assets_dir, load_seeds,
LocalizedText, LocalizedTextError, persona_to_tags, asset_locale,
india_variant and recorded_value: each name that two or more built-in
probes take from an internal module. docs/extension_api.md lists each in
its section, among them a section for banks and assets, and the worked
example in AUTHORING_A_PROBE.md imports dispatch_episode from
usersim.extension with the rest of its names.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
Each built-in probe package imports from usersim.extension every name that
usersim.extension exports from the module the package imported it from,
at the top of a module and inside a function alike. general_educational,
general_open_ended, health_disclosure, safety_chat_pressure, sov_ai_facts
and sov_ai_multilingual_parity import nothing from other usersim modules,
and the allowances in test_probe_private_imports.py are each package's
count: financial_services 12, identity_disclosure 6, safety_agentic 3,
sov_ai_dynamic 7 and tool_calling 1. AUTHORING_A_PROBE.md says that a
package importing everything from usersim.extension has an allowance of 0.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant