Skip to content

refactor(engine): keep each built-in's capabilities in its own package - #133

Open
3mei wants to merge 5 commits into
yev/feat/shared-names-in-extension-apifrom
yev/refactor/capabilities-in-probe-packages
Open

3mei wants to merge 5 commits into
yev/feat/shared-names-in-extension-apifrom
yev/refactor/capabilities-in-probe-packages

Conversation

@3mei

@3mei 3mei commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

What changed and why

Each built-in probe keeps the capability rows its evidence feeds in its own package, beside its scorer, so adding or removing a built-in no longer edits the shared capability registry. No built-in imports a private module or name from core.

Builds on #132.

  • A scorer registry test that leaves the registry alone (first commit). test_reload_scorers_reruns_a_module_that_registers_nothing reloads a module it writes to tmp_path. Reloading the scorer package itself replaces register_scorer for every later test in the same process, and test_extension_api.py's check that each exported name is its home module's object fails after such a reload, so its result depends on how the suite is split across workers. Run in one process after the scorer registry tests, it fails at feat(extension): export the names built-in probes share #132's head and passes here.
  • Public names (second commit). usersim.extension exports ALL_PROBES, the probe an EvidenceSource names to read the rows of every probe, and probe_asset_roots and asset_roots_in_use, the roots searched for a probe's assets and every root an asset lookup can read, for assets layered across roots. docs/extension_api.md lists each.
  • Capabilities in each package (third commit). sov_ai_facts (with sovereign_local_knowledge, which also reads sov_ai_dynamic's rows), sov_ai_multilingual_parity, tool_calling, safety_chat_pressure, safety_agentic, identity_disclosure and financial_services each declare theirs in a capabilities.py, with the descriptions of their scorer's rate axes. taxonomy/capabilities.py keeps the eight capabilities every probe feeds and reads the rest from BUILTIN_CAPABILITY_MODULES, which lists the modules in the order the report shows their rows. The modules hold definitions only, so reading the capabilities imports no probe. AUTHORING_A_PROBE.md has a section on a built-in's capabilities.py, and architecture/plugins.md says where built-ins declare theirs.
  • No private core names in the built-ins (fourth commit). tool_calling falls back to BaseProbe.theme() for a row without a toolset category, identity_disclosure reads the asset search roots from usersim.extension, and financial_services derives literacy from education_ordinal in core.behavioral, named without a leading underscore. tool_calling imports nothing from other usersim modules.
  • Where a scorer lives (fifth commit). AUTHORING_A_PROBE.md says a built-in's scorer is scorer.py in its package, and that the probe-label count covers shared code only.

The capability report is unchanged: the same 46 definitions in the same order, with the same implementation refs and axis descriptions.

How it was verified

At each commit, on that commit's tree:

$ make check-all && make test && make smoke
All checks passed!
374 files already formatted
4704 passed in 87.72s (0:01:27)
All checks passed.

That is the first commit's tree. The second passes with the same count, and the last three with 4706, with the two tests of the capability modules.

At the head:

$ make check-license-headers
🎉 All files have correct license headers!
$ make check-doc-links
✓ every relative link, and every link into this repository's main branch, resolves
$ make check-notebooks-clean
OK 4 notebook(s) carry no outputs
$ make check-dist
✓ installed package runs from a clean environment
$ make check-extensions
all extension seams work against an installed distribution
  • Before and after the move, capability_definitions() gives the same 46 definitions in the same order, and implementation_refs_for_capability and axis_description give the same values for each:

    ids identical
    defs identical
    refs identical
    axes identical
    46
    
  • make health-check ran against a custom provider config for tool_calling in en_US and for the default probe, sov_ai_dynamic in ja_JP, each through simulate, eval and the report. Both ended with ✓ simulate, eval and report all completed, and every model's requests show failed=0.

3mei added 5 commits November 4, 2026 09:40
test_evaluator_scorers.py checks that reload_scorers re-runs a module
that registers no scorer with a module it writes to tmp_path and removes
afterwards. Reloading the scorer package itself replaces register_scorer
and the registry for every later test in the same process, and the test
comparing usersim.extension's names with their home modules fails after
such a reload, so its result depends on how the suite is split across
workers.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
usersim.extension exports ALL_PROBES, the probe an EvidenceSource names
to read the rows of every probe, and probe_asset_roots and
asset_roots_in_use: the roots searched for a probe's assets, highest
priority first, and every root an asset lookup can read, for assets
layered across roots. docs/extension_api.md lists each.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
Each built-in probe package with capability rows of its own declares them
in its capabilities.py, with the descriptions of its scorer's rate axes:
sov_ai_facts (with sovereign_local_knowledge, which also reads
sov_ai_dynamic's rows), sov_ai_multilingual_parity, tool_calling,
safety_chat_pressure, safety_agentic, identity_disclosure and
financial_services. taxonomy/capabilities.py keeps the capabilities every
probe feeds and reads the rest from BUILTIN_CAPABILITY_MODULES, which
lists the modules in the order the report shows their rows. The modules
hold definitions only, so reading the capabilities imports no probe. The
definitions, their order, their implementation refs and the axis
descriptions are unchanged, and the taxonomy names no probe.
AUTHORING_A_PROBE.md describes a built-in's capabilities.py, and
architecture/plugins.md says where a built-in declares its capabilities.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
tool_calling falls back to BaseProbe.theme() for a row without a toolset
category, which reads the theme as the probe did, and imports nothing
from other usersim modules. identity_disclosure reads the asset search
roots from usersim.extension, and financial_services derives literacy
from education_ordinal in core.behavioral, named without a leading
underscore. No built-in imports a private module or name from core, and
the allowances in test_probe_private_imports.py are identity_disclosure 4
and tool_calling 0.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
AUTHORING_A_PROBE.md says a built-in's scorer lives in its own package as
scorer.py, with its module path in DEFAULT_SCORER_MODULES, and that
test_probe_names_in_shared_code.py counts labels in shared code only, so a
scorer in the probe's package may write its label.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant