[3mei]

What changed and why

A scorer declares whether usersim rescore may rerun it, and a probe declares the description its rows are shown with, what the preview shows of each row, the scorers that belong to it and the check of the assets it ships, so a scorer or probe shipped in its own package gets all five without a shared module naming it. The rules a simulated user writes by are chosen by name, and a row with no probe family is scored as belonging to no probe.

Builds on #111.

• Rescore eligibility on the scorer (first commit). register_scorer(name, fn, deterministic=True) marks a scorer that never calls or looks up a model.
  • usersim rescore reruns exactly the scorers registered that way and refuses any other, naming why and listing the allowed ones.
  • language_compliance, refusal_basics and response_shape declare it.
  • assert_scorer_conforms reports a scorer registered as deterministic that looks up a model.
  • docs/engine/AUTHORING_A_PROBE.md, architecture/evaluator.md, templates/probe/README.md and the docs/extensions.md example scorer describe it.
• Descriptions on the probe (second commit). BaseProbe.description is the paragraph the per-row preview and the capability report show with a probe's rows.
  • Each built-in probe carries its own description text.
  • Both consumers read it through probe_description(label), which gives an empty string for a label no installed probe holds, so a stored row from a probe that is not installed renders without one.
  • assert_probe_conforms checks that it is a string. The authoring guide, architecture/probes.md, the template checklist and the docs/extensions.md example probe describe it.
• Chat-realism rules by name (third commit). format_behavioral_profile_for_prompt takes chat_realism, a ChatRealism member, and no probe label.
  • SHARING_DETAILS lets the simulated user give IDs, reference numbers and dates; ASKING, the default, keeps the user to what the assistant asks for.
  • tool_calling asks for SHARING_DETAILS. The prompts each probe builds are unchanged.
• Default scorers on the probe (fourth commit). BaseProbe.default_scorers names the registered scorers that belong to a probe.
  • usersim eval --scorers auto runs language_compliance, refusal_basics and response_shape, which apply to every probe, plus the default_scorers of each probe whose rows the pass evaluates, found by the probe_family the rows are stored under.
  • auto combines with scorer names and each name runs once. A family no installed probe stores its rows under is named in a warning.
  • Each built-in probe with a scorer of its own declares it, so auto over a run holding every built-in probe runs every scorer the capability report reads.
  • Without auto, usersim eval runs exactly the scorers named.
  • register_scorer refuses the name auto, and assert_probe_conforms reports a default scorer that no registered scorer holds.
  • The authoring guide, architecture/evaluator.md, architecture/probes.md, docs/engine/README.md, the template checklist and the docs/extensions.md example probe describe it.
• Rows with no probe family (fifth commit). A row whose probe_family cell is empty names no probe, so the evaluator attributes it to none.
  • The judges score it on the axes every probe shares, and its scorers see an empty probe_family.
  • --scorers auto adds no probe's scorers for it, and leaves it out of the warning about families no installed probe stores rows under.
  • architecture/evaluator.md and architecture/plugins.md say what a scorer receives for such a row.
• Asset checks on the probe (sixth commit). BaseProbe.asset_problems() returns each problem with the assets a probe ships, found without a model call. The default checks nothing.
  • usersim smoke runs it for every registered probe. Its probe assets line names each probe that checks and fails with the first problems found; an exception a check raises is reported the same way.
  • The checks smoke made for the built-in probes move onto those probes: a fact bank and a probing taxonomy for every shipped locale, the general probes' topics, subjects and prompts files, tool_calling's prompts file and toolset seed, the parity, pressure and agentic-action banks with the action taxonomy, the en_US finance bank with its dynamic-tier taxonomy, and the identity spec. cli/smoke.py names no probe's assets.
  • The finance check reads its bank and taxonomy from disk rather than from the loaders' caches, so a check in a long-running process sees what is installed.
  • assert_probe_conforms reports what the check finds, and a check that is not a classmethod, raises, or returns something other than a list of strings.
  • The authoring guide, architecture/assets.md, architecture/probes.md, docs/engine/README.md, CONTRIBUTING.md, the template checklist and the docs/extensions.md example probe describe it.
• Preview fields on the probe (seventh commit). BaseProbe.preview_fields lists what the per-row preview's Probe Inputs panel shows of each of the probe's rows, top to bottom, such as the bank entry the row drew.
  • Each PreviewField names a column of the row, or with from_metadata=True a key the probe writes to state.metadata, and the label it is shown with. The panel leaves out an empty field, and is left out when every field is.
  • The panel reads them through probe_preview_fields(label), which gives none for a probe that is not installed, so preview.py holds no table of probes.
  • The nine built-in probes the panel covered declare the same fields with the same labels.
  • assert_probe_conforms reports preview_fields that are not a tuple of PreviewField, each with a text key and label.
  • The authoring guide, architecture/probes.md, the template checklist and the docs/extensions.md example probe describe it.

The make check-extensions fixture registers its scorer as deterministic, names it in its probe's default_scorers, checks its probe's bank in asset_problems, and lists a column in preview_fields. From an installed package, the check confirms that usersim rescore accepts the scorer, that --scorers auto finds it by the probe's family, which differs from its label, that usersim smoke runs the fixture probe's asset check and the spec check its identity probe inherits, and that the preview's Probe Inputs panel shows the fixture probe's field.

How it was verified

$ make check-all && make test && make smoke
All checks passed!
344 files already formatted
4618 passed in 65.25s
[OK]   probe registry        14 probe(s) registered: ['financial_services', 'general_educational', 'general_open_ended', 'health_decision_support_disclosure', 'health_general_disclosure', 'health_therapy_disclosure', 'health_triage_disclosure', 'identity_disclosure', 'safety_agentic', 'safety_chat_pressure', 'sov_ai_dynamic', 'sov_ai_facts', 'sov_ai_multilingual_parity', 'tool_calling']
[OK]   probe assets          10 probe(s) checked their assets: financial_services, general_educational, general_open_ended, identity_disclosure, safety_agentic, safety_chat_pressure, sov_ai_dynamic, sov_ai_facts, sov_ai_multilingual_parity, tool_calling
All checks passed.

$ make check-license-headers
🎉 All files have correct license headers!
$ make check-doc-links
✓ every relative link, and every link into this repository's main branch, resolves
$ make check-notebooks-clean
OK 4 notebook(s) carry no outputs
$ make check-dist
✓ both release archives pass
✓ installed package runs from a clean environment
$ make check-extensions
      ok  usersim.scorers
      ok  deterministic scorer is rescorable
      ok  default_scorers join --scorers auto
      ok  preview_fields reach the preview
      ok  usersim smoke runs the contributed asset checks
all extension seams work against an installed distribution

• Each commit passes make check-all && make test && make smoke on its own: 4565 tests after the first, 4567 after the second, 4570 after the third, 4587 after the fourth, 4593 after the fifth, 4612 after the sixth.

• New tests:

  • usersim rescore refuses a scorer registered without the flag.
  • Exactly the three rule-based scorers are registered as deterministic.
  • assert_scorer_conforms reports a deterministic scorer that looks up a model, and not a scorer without the flag that does.
  • Every built-in probe has a description; a probe registered from outside the package is described through its class; an unknown label has none.
  • assert_probe_conforms reports a description that is not a string.
  • The user that tool_calling simulates may share structured details and the general_open_ended user may not, checked on the system prompt each user model receives.
  • usersim eval --scorers tool_use,auto reaches the evaluator as tool_use, the three rule-based scorers and the default scorers of the run's probes, each once.
  • A family no installed probe stores its rows under is named in a warning, and a probe is found by its family, not its label.
  • auto over every built-in probe leaves no capability without its scorer, and auto over each probe runs the scorers its capabilities read.
  • register_scorer refuses auto. assert_probe_conforms reports default scorers that are not a tuple of names, or that name no registered scorer.
  • A row whose probe_family is None, NaN, pd.NA or blank reaches its scorers with an empty family, and is judged on the shared axes alone. --scorers auto gives a blank family the three rule-based scorers and no warning.
  • Each of the ten built-in asset checks reports a probe whose asset folder is empty, and none of those probes falls back to the default check.
  • The probe assets line names a probe that checks and leaves out one that does not, fails on a problem a check returns or an exception it raises, and names the first three problems with a count of the rest.
  • assert_probe_conforms reports an asset check that is not a classmethod, returns a bare string, raises, or finds a problem.
  • A probe registered from outside the package shows its preview fields, from the row and from the conversation metadata, in the order it lists them and without the empty ones; a row from a probe that is not installed gets no Probe Inputs panel.
  • assert_probe_conforms reports preview fields given as one field, as plain tuples, or without a text label.

• The example package in docs/extensions.md matches the guide's code blocks byte for byte, and its four tests pass with the deterministic scorer and the description.

• make health-check against a custom provider config, with HEALTH_PROBE=tool_calling, with the default sov_ai_dynamic row in ja_JP after the fifth commit, and with HEALTH_PROBE=financial_services after the sixth: each time a two-turn row completed, the evaluator reported failed=0, and the report rendered.

• The three rule-based scorers, run by hand with --scorers language_compliance,refusal_basics,response_shape on an eight-turn tool_calling row against the same config:

  judge: scored
  language_compliance: ok (scores=['language.first_turn_match', 'language.requested_language_match_rate', 'language.script_compliance_rate', 'language.script_integrity_rate'], error=None)
  refusal_basics: ok (scores=['refusal.canned_phrase_rate', 'refusal.detected_rate', 'refusal.in_wrong_language_rate'], error=None)
  response_shape: ok (scores=['shape.empty_or_trivial_rate', 'shape.over_formatted_rate', 'shape.truncation_suspicion_rate'], error=None)
  

  usersim rescore then reran all three on that evaluation with no model call, and refused tool_use:

  cannot rescore:
    'tool_use' is not registered as deterministic, so it may call a model; refused: rescore is LLM-free
  allowed: language_compliance, response_shape, refusal_basics
  

• usersim eval --scorers auto on a new eight-turn tool_calling row against the same config resolved to the three rule-based scorers plus tool_use, the probe's default scorer. The evaluator reported failed=0, and each scorer returned a result:

    scorers:       language_compliance, refusal_basics, response_shape, tool_use
  judge: scored
  envelope scorers: ['language_compliance', 'refusal_basics', 'response_shape', 'tool_use']
  language_compliance: ok (scores=['language.first_turn_match', 'language.requested_language_match_rate', 'language.script_compliance_rate', 'language.script_integrity_rate'], error=None)
  refusal_basics: ok (scores=['refusal.canned_phrase_rate', 'refusal.detected_rate', 'refusal.in_wrong_language_rate'], error=None)
  response_shape: ok (scores=['shape.empty_or_trivial_rate', 'shape.over_formatted_rate', 'shape.truncation_suspicion_rate'], error=None)
  tool_use: ok (scores=['architecture_leaking', 'argument_quality', 'information_gathering', 'overall', 'role_adherence', 'task_completion', 'tool_selection', 'unnecessary_tool_use'], error=None)
  

• After the sixth commit, usersim eval --scorers auto on an eight-turn financial_services row against the same config resolved to the three rule-based scorers plus financial_services. The evaluator reported failed=0, and each scorer returned a result:

    scorers:       language_compliance, refusal_basics, response_shape, financial_services
  judge: scored
  envelope scorers: ['language_compliance', 'refusal_basics', 'response_shape', 'financial_services']
  language_compliance: ok (scores=['language.first_turn_match', 'language.requested_language_match_rate', 'language.script_compliance_rate', 'language.script_integrity_rate'], error=None)
  refusal_basics: ok (scores=['refusal.canned_phrase_rate', 'refusal.detected_rate', 'refusal.in_wrong_language_rate'], error=None)
  response_shape: ok (scores=['shape.empty_or_trivial_rate', 'shape.over_formatted_rate', 'shape.truncation_suspicion_rate'], error=None)
  financial_services: ok (scores=['finance.document_recall_rate', 'finance.no_unauthorized_action_rate', 'finance.ordering_respected_rate', 'finance.substantive_retrieval_rate', 'finance.tool_selection_rate'], error=None)
  

Results impact

None. Without auto, usersim eval runs the scorers it is given, so evaluation cells and their envelopes are unchanged. The same three scorers are rescorable, every probe shows the same description, and every probe's user prompt is unchanged.

usersim smoke runs the same asset checks for the built-in probes, on a line named probe assets that lists the probes it checked. Runs and their outputs are unchanged.

The per-row preview shows the same Probe Inputs fields, with the same labels, for every built-in probe.

A row with no probe_family is judged on the shared axes, which are all the judges score for a general_open_ended row, so its evaluation and envelope are unchanged. A scorer that reads probe_family sees an empty string on such a row; no shipped scorer reads that field.