Repository navigation
Conversation
usersim.extension exports dispatch_episode, probe_assets_dir, load_seeds, LocalizedText, LocalizedTextError, persona_to_tags, asset_locale, india_variant and recorded_value: each name that two or more built-in probes take from an internal module. docs/extension_api.md lists each in its section, among them a section for banks and assets, and the worked example in AUTHORING_A_PROBE.md imports dispatch_episode from usersim.extension with the rest of its names. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
Each built-in probe package imports from usersim.extension every name that usersim.extension exports from the module the package imported it from, at the top of a module and inside a function alike. general_educational, general_open_ended, health_disclosure, safety_chat_pressure, sov_ai_facts and sov_ai_multilingual_parity import nothing from other usersim modules, and the allowances in test_probe_private_imports.py are each package's count: financial_services 12, identity_disclosure 6, safety_agentic 3, sov_ai_dynamic 7 and tool_calling 1. AUTHORING_A_PROBE.md says that a package importing everything from usersim.extension has an allowance of 0. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed and why
A built-in probe imports what it shares with the other probes from
usersim.extension, as an extension does, so six of the eleven built-in packages import nothing from internalusersimmodules.Builds on #131.
usersim.extensionexports the nine names that two or more built-in probes take from internal modules:dispatch_episode,probe_assets_dir,load_seeds,LocalizedText,LocalizedTextError,persona_to_tags,asset_locale,india_variantandrecorded_value.docs/extension_api.mdlists each with what it is for, among them a section for banks and assets, and the worked example inAUTHORING_A_PROBE.mdimportsdispatch_episodefromusersim.extensionwith the rest of its names.usersim.extensionevery name thatusersim.extensionexports from the module the package imported it from, at the top of a module and inside a function alike.general_educational,general_open_ended,health_disclosure,safety_chat_pressure,sov_ai_factsandsov_ai_multilingual_parityimport nothing from otherusersimmodules, and the allowances intests/test_probe_private_imports.pyare each package's count:financial_services12,identity_disclosure6,safety_agentic3,sov_ai_dynamic7 andtool_calling1. The names left are each used by one probe, apart from the probing taxonomy thatfinancial_servicesandsov_ai_dynamicshare.No behaviour changes: the same objects are imported under the same names.
How it was verified
At each commit, on that commit's tree:
That is the first commit's tree; the second passes with the same count.
At the head:
usersim.extension.make health-checkran against a custom provider config for the default probe,sov_ai_dynamicinja_JP. It ended with✓ simulate, eval and report all completed, and every model's requests showfailed=0.