[3mei]

What changed and why

In a run that mixes probes, each probe's scorer judges only its own probe's rows, and each row stores only the inputs its own probe used.

Builds on #134.

• tool_use skips a row that offers no tools (first commit). usersim eval runs every scorer it is given on every row, and --scorers auto gives it the default scorers of every probe in the run, so a run that mixes tool_calling with other probes hands tool_use their rows too. Each probe's own scorer skips a row without its probe's columns. Without that check, tool_use judges every row, and the capability report, which counts any row whose scorer result is ok, takes the general probes' conversations as evidence for the two tool-use capabilities. tool_use skips a row with no tool_subset, before any model call, with the no ... scorer skipped error that the report counts as not applicable. tests/engine/evaluator/test_scorers_skip_other_probes.py runs every built-in probe's scorer on a general_open_ended row with no models and checks that each skips it; the authoring guide and architecture/evaluator.md say how a scorer skips a row.
• A seed table's columns stay with the probe that reads it (second commit). Data Designer gives every row of a run the columns of one seed-table row, whatever the row's probe, so in a mix with tool_calling the other probes' rows store a toolset their conversations never offered (toolset_source, toolset_category, toolset_name, num_tools, tools). clear_seed_columns_of_other_probes empties the attached table's columns on the rows of the probes that do not declare it, resolving the table as the run does; usersim simulate, notebook 01 and materialize_episode_inputs apply it before they store or return rows. The reading probe's rows keep their values and types.
• A probe's scorer in its own package, in two docs (third commit). The probe template README's out-of-sim judges note and the engine README's evaluation paragraph put a new probe's scorer in evaluator/scorers/<your_probe>.py; they say engine/probes/<your_probe>/scorer.py and give the module path to add to DEFAULT_SCORER_MODULES, as the template's checklist and AUTHORING_A_PROBE.md do.

Changes for callers

• Evaluating a mixed run gives tool_use results, and the tool-use capability numbers, for its rows that offered tools only. An evaluation that holds tool_use results on other probes' rows keeps them until the run is evaluated again; usersim rescore does not rerun tool_use, since it calls a model.
• A stored or materialized row of a probe that reads no seed table holds the seed table's columns empty.

How it was verified

At each commit, on that commit's tree:

$ make check-all
All checks passed!
382 files already formatted
$ pre-commit run --files <the commit's files>
(every hook Passed or Skipped)
$ uv run pytest --collect-only -q
(exit 0)

At the head:

$ make test && make smoke
4749 passed in 84.36s (0:01:24)
All checks passed.
$ make check-license-headers
🎉 All files have correct license headers!
$ make check-doc-links
✓ every relative link, and every link into this repository's main branch, resolves
$ make check-notebooks-clean
OK 4 notebook(s) carry no outputs
$ make check-dist
✓ installed package runs from a clean environment
$ make check-extensions
      ok  seed_dataset resolves a contributed seed table
all extension seams work against an installed distribution

That set ran on the second commit's tree. The third commit changes two Markdown files only; on its tree the quick set passes, as do make check-doc-links and the test files that read the docs (test_packaging_contract, test_security_defaults, test_check_doc_links, test_extension_api, test_cli: 163 passed).

• With tool_use as feat(engine): declare seed tables and close out the extension work #134 has it, the new scorer test fails: run on a general_open_ended row with no models, it tries to call the judge (ValueError: Model alias 'judge_model' not found in models dict) and returns an error, not a skip.

• With the call removed from usersim simulate, the new write-path test fails on the general_open_ended row's seed columns.

• make health-check for tool_calling against a custom provider config ended with ✓ simulate, eval and report all completed, and every model's requests show failed=0.

• usersim simulate with tool_calling=0.5,general_open_ended=0.5, six rows, then usersim eval --scorers auto and usersim report, against the same config:

  general_open_ended | toolset None/None | num_tools None | tools None | tool_use=not_applicable   (4 rows)
  tool_calling       | toolset <NA>/Train | num_tools 86 | tool_use=ok
  tool_calling       | toolset API NBA/Competition | num_tools 9 | tool_use=ok
  capability_matrix: tool_use_correctness n=2 of 6 rows, tool_use_restraint n=2 of 6 rows