Repository navigation
Conversation
test_evaluator_scorers.py checks that reload_scorers re-runs a module that registers no scorer with a module it writes to tmp_path and removes afterwards. Reloading the scorer package itself replaces register_scorer and the registry for every later test in the same process, and the test comparing usersim.extension's names with their home modules fails after such a reload, so its result depends on how the suite is split across workers. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
usersim.extension exports ALL_PROBES, the probe an EvidenceSource names to read the rows of every probe, and probe_asset_roots and asset_roots_in_use: the roots searched for a probe's assets, highest priority first, and every root an asset lookup can read, for assets layered across roots. docs/extension_api.md lists each. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
Each built-in probe package with capability rows of its own declares them in its capabilities.py, with the descriptions of its scorer's rate axes: sov_ai_facts (with sovereign_local_knowledge, which also reads sov_ai_dynamic's rows), sov_ai_multilingual_parity, tool_calling, safety_chat_pressure, safety_agentic, identity_disclosure and financial_services. taxonomy/capabilities.py keeps the capabilities every probe feeds and reads the rest from BUILTIN_CAPABILITY_MODULES, which lists the modules in the order the report shows their rows. The modules hold definitions only, so reading the capabilities imports no probe. The definitions, their order, their implementation refs and the axis descriptions are unchanged, and the taxonomy names no probe. AUTHORING_A_PROBE.md describes a built-in's capabilities.py, and architecture/plugins.md says where a built-in declares its capabilities. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
tool_calling falls back to BaseProbe.theme() for a row without a toolset category, which reads the theme as the probe did, and imports nothing from other usersim modules. identity_disclosure reads the asset search roots from usersim.extension, and financial_services derives literacy from education_ordinal in core.behavioral, named without a leading underscore. No built-in imports a private module or name from core, and the allowances in test_probe_private_imports.py are identity_disclosure 4 and tool_calling 0. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
AUTHORING_A_PROBE.md says a built-in's scorer lives in its own package as scorer.py, with its module path in DEFAULT_SCORER_MODULES, and that test_probe_names_in_shared_code.py counts labels in shared code only, so a scorer in the probe's package may write its label. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed and why
Each built-in probe keeps the capability rows its evidence feeds in its own package, beside its scorer, so adding or removing a built-in no longer edits the shared capability registry. No built-in imports a private module or name from core.
Builds on #132.
test_reload_scorers_reruns_a_module_that_registers_nothingreloads a module it writes totmp_path. Reloading the scorer package itself replacesregister_scorerfor every later test in the same process, andtest_extension_api.py's check that each exported name is its home module's object fails after such a reload, so its result depends on how the suite is split across workers. Run in one process after the scorer registry tests, it fails at feat(extension): export the names built-in probes share #132's head and passes here.usersim.extensionexportsALL_PROBES, the probe anEvidenceSourcenames to read the rows of every probe, andprobe_asset_rootsandasset_roots_in_use, the roots searched for a probe's assets and every root an asset lookup can read, for assets layered across roots.docs/extension_api.mdlists each.sov_ai_facts(withsovereign_local_knowledge, which also readssov_ai_dynamic's rows),sov_ai_multilingual_parity,tool_calling,safety_chat_pressure,safety_agentic,identity_disclosureandfinancial_serviceseach declare theirs in acapabilities.py, with the descriptions of their scorer's rate axes.taxonomy/capabilities.pykeeps the eight capabilities every probe feeds and reads the rest fromBUILTIN_CAPABILITY_MODULES, which lists the modules in the order the report shows their rows. The modules hold definitions only, so reading the capabilities imports no probe.AUTHORING_A_PROBE.mdhas a section on a built-in'scapabilities.py, andarchitecture/plugins.mdsays where built-ins declare theirs.tool_callingfalls back toBaseProbe.theme()for a row without a toolset category,identity_disclosurereads the asset search roots fromusersim.extension, andfinancial_servicesderives literacy fromeducation_ordinalincore.behavioral, named without a leading underscore.tool_callingimports nothing from otherusersimmodules.AUTHORING_A_PROBE.mdsays a built-in's scorer isscorer.pyin its package, and that the probe-label count covers shared code only.The capability report is unchanged: the same 46 definitions in the same order, with the same implementation refs and axis descriptions.
How it was verified
At each commit, on that commit's tree:
That is the first commit's tree. The second passes with the same count, and the last three with 4706, with the two tests of the capability modules.
At the head:
Before and after the move,
capability_definitions()gives the same 46 definitions in the same order, andimplementation_refs_for_capabilityandaxis_descriptiongive the same values for each:make health-checkran against a custom provider config fortool_callinginen_USand for the default probe,sov_ai_dynamicinja_JP, each through simulate, eval and the report. Both ended with✓ simulate, eval and report all completed, and every model's requests showfailed=0.