[3mei]

What changed and why

A user turn gated with the default rubric takes one judge request: BaseProbe.format_gate_prompt asks for the reply run_inline_judge parses, an <explanation> and a <rating> of success or failure. financial_services, sov_ai_dynamic, sov_ai_facts and sov_ai_multilingual_parity gate their user turns with it.

Builds on #115.

• The default gate prompt (first commit). The rubric is the same; the prompt names the rating for each case and ends with the reply format the other gate prompts use.
  • tests/engine/test_hosted_parity_contract.py runs every registered probe offline with run_episode and checks that each request it sends to the judge asks for an <explanation> and a <rating> of success or failure. safety_agentic, which has no simulated user, sends none, and the test checks that too.
• The examples (second commit). Step 5 of docs/engine/AUTHORING_A_PROBE.md and the demo gate prompt in templates/probe/prompts.py ask for the same reply. The guide says which ratings run_inline_judge accepts, that success and warn accept a turn, and that a reply without a valid rating gets one reformat request before the turn counts as failed.

How it was verified

At each commit, on that commit's tree:

$ make check-all && make test && make smoke
All checks passed!
349 files already formatted
4645 passed in 91.51s (0:01:31)
All checks passed.

(4645 passed in 55.29s at the second commit.)

At the head:

$ make check-license-headers
🎉 All files have correct license headers!
$ make check-doc-links
✓ every relative link, and every link into this repository's main branch, resolves
$ make check-notebooks-clean
OK 4 notebook(s) carry no outputs

• The new test fails without the first commit's prompt change, for the four probes that gate with the default rubric, and passes for all 14 with it:

  FAILED tests/engine/test_hosted_parity_contract.py::test_every_judge_request_asks_for_a_rating_the_parser_reads[financial_services]
  FAILED tests/engine/test_hosted_parity_contract.py::test_every_judge_request_asks_for_a_rating_the_parser_reads[sov_ai_dynamic]
  FAILED tests/engine/test_hosted_parity_contract.py::test_every_judge_request_asks_for_a_rating_the_parser_reads[sov_ai_facts]
  FAILED tests/engine/test_hosted_parity_contract.py::test_every_judge_request_asks_for_a_rating_the_parser_reads[sov_ai_multilingual_parity]
  

• make health-check ran against a custom provider config (ja_JP, sov_ai_dynamic, two turns):

  |-- sov_ai_dynamic setup: 0.0s (query + gate)
  |-- sov_ai_dynamic turn 1/2: 32.5s (assistant + asst_judge + user + user_judge)
  |-- sov_ai_dynamic turn 2/2: 19.0s (assistant + asst_judge, last turn)
  ...
  ✓ simulate, eval and report all completed
  

  The judge model, which serves no other alias, received 4 requests (success=4, failed=0) for the run's 4 verdicts: the opening gate, one follow-up gate and two assistant judgments. The evaluator's requests show failed=0.

• usersim simulate and usersim eval --scorers auto ran against the same config for two probes that gate with the default rubric, two turns each. sov_ai_dynamic (ja_JP):

  judge: scored
  language_compliance: ok (error=None)
  refusal_basics: ok (error=None)
  response_shape: ok (error=None)
  sov_ai_dynamic: ok (error=None)
  

  financial_services (en_US):

  judge: scored
  financial_services: ok (error=None)
  language_compliance: ok (error=None)
  refusal_basics: ok (error=None)
  response_shape: ok (error=None)
  

  The judge model received one request per judge verdict in each row's simulation_traces: 4 for sov_ai_dynamic (success=4, failed=0) and 3 for financial_services (success=3, failed=0), whose opening gate is bypassed. The evaluator's requests show failed=0 in both.

Results impact

For financial_services and the three sov_ai probes, each gated user turn takes one judge request, and its verdict and the explanation stored with it come from the judge's first reply. Where a first reply rates a turn differently from a reformatted one, the user turns a run keeps and its role-violation counts can shift, and with them the conversations the assistant is judged and scored on. The assistant-quality prompt and the scorers are unchanged.