Send Tool Call Success transcripts to the judge, and fix Decision reasons and Task adherence's threshold

sdk/evaluation/azure-ai-evaluation/CHANGELOG.md

@@ -1,5 +1,20 @@
 # Release History
 
+## 1.18.12 (Unreleased)
+
+### Features Added
+
+### Breaking Changes
+
+### Bugs Fixed
+
+- `_ToolCallSuccessEvaluator` sends a list of text messages to the judge model, rather than reporting it as not applicable, when its text looks like a tool transcript. Such text contains `tool_call`, `tool_result`, `tool_response`, `function_call` or `function_result` in any letter case, as tags such as `<tool_call>`, `[TOOL_CALLS]` and this evaluator's `[TOOL_RESULT]` do, Llama's `<|python_tag|>` or `<function=` tags or the gpt-oss `<|call|>` token, or has a line that starts with a label such as `Action:`, `Observation:`, `Tool call:` or `Tool result:`, as ReAct agents write, also after a bullet or a list marker such as `1)`, `(2)` or `a)`. In 1.18.11 only text with `[TOOL_CALL]` or `[TOOL_RESULT]` was sent. Such a list is sent as its text, with each message's role and every tool definition, as a string is, because a Decision model can miss ReAct steps inside a list of messages. The text of a message that holds an MCP approval request is not counted, because it is left out of every judge input. Pass a transcript in any other text format as a string for the judge model to assess it.
+- A Decision evaluator that the judge model rates as not applicable gives a readable reason, for example `Not applicable: the judge model's applicability probability 0.2000 is below the 0.5000 cutoff, so the evaluator was skipped. No model rationale is available.` It used to read `Status completion probability 0.2000 is below 0.5000; evaluation skipped.`, which named an internal question.
+- `TaskAdherenceEvaluator` with a Decision model reports a threshold of 1 when none is requested, the only score at which its binary result passes, as `_TaskCompletionEvaluator` does. It reported 0. A requested threshold is still reported as given and still does not decide whether a binary result passes. The evaluator's `threshold` attribute now also defaults to 1 for chat models; chat results are unchanged.
+- A Decision reason names a question by the part of its key after the last `/`, so a namespaced key such as `grade/confirm_details`, which the hosted evaluation service uses for rubric dimensions, reads `Confirm details probability 0.1645 is below the 0.7500 cutoff.` It used to read `Grade/confirm details probability ...`.
+
+### Other Changes
+
 ## 1.18.11 (2026-11-03)
 
 ### Features Added
@@ -8,13 +23,13 @@
 - `QAEvaluator` accepts a Decision model configuration and passes `credential` and the Decision execution controls to its model-based evaluators, which run on separate threads, so the credential must be synchronous. With a chat model they are not passed, as before.
 - The binary evaluators (task adherence, task completion, tool call success, tool input accuracy, tool output utilization and tool selection) accept a `probability_threshold` keyword. For Decision models it sets the probability at or above which the score is 1 (default 0.75) and overrides the value in `model_config`; chat models ignore it with a warning.
 - Conversation inputs are handled as before with a Decision model: evaluators that accept them score each user and assistant turn separately and aggregate the results. The catalog's whole-conversation rubrics are not included.
-- A failed Decision request names a recognized reason in `decision_model.error_kind` of the error's properties, which is kept on failed result rows, and in the error message, for example `The decision model returned HTTP 422 (context_window_exceeded).` The provider states these reasons for an HTTP 422 only in text, which is never exposed: `context_window_exceeded` when the evidence and one question exceed the model's context window, `request_token_limit_exceeded` when the evidence and all questions exceed the model's request limit, `question_limit_exceeded` and `invalid_question`. A request over `max_request_bytes` or `max_state_question_bytes`, which is not sent, has the kind `request_bytes_exceeded`.
+- A failed Decision request names a recognized reason in `decision_model.error_kind` of the error's properties, which is kept on failed result rows. The error message also names it when the provider returns no error code, for example `The decision model returned HTTP 422 (context_window_exceeded).`, so read the property rather than the message. The provider states these reasons for an HTTP 422 only in text, which is never exposed: `context_window_exceeded` when the evidence and one question exceed the model's context window, `request_token_limit_exceeded` when the evidence and all questions exceed the model's request limit, `question_limit_exceeded` and `invalid_question`. A request over `max_request_bytes` or `max_state_question_bytes`, which is not sent, has the kind `request_bytes_exceeded`, and its error message is unchanged.
 
 ### Bugs Fixed
 
-- `_ToolCallSuccessEvaluator` no longer scores successful tool calls 0 when a chat judge model, such as gpt-4.1 or gpt-4o, puts `score` and `status` inside `properties`. The example outputs in its prompt are now valid JSON, and `score`, `status` and `reason` are read from `properties` when the output has none at the top level. A missing score is no longer reported as 0, and any score other than 0 or 1 raises a system-classified `EvaluationException`, as `_ToolInputAccuracyEvaluator` does.
+- `_ToolCallSuccessEvaluator` no longer scores successful tool calls 0 when a chat judge model, such as gpt-4.1 or gpt-4o, puts `score` and `status` inside `properties`. The example outputs in its prompt are now valid JSON, and `score`, `status` and `reason` are read from `properties` when the output has none at the top level. A missing score is no longer reported as 0, and a score that does not read as 0 or 1 raises a system-classified `EvaluationException`. Values that read as 0 or 1, such as `True`, `"1"` and `1.0`, are still accepted.
 - `_ToolCallSuccessEvaluator` reports a list of messages that holds no tool calls, such as one that contains only text or only MCP approval messages, as not applicable without calling the chat or Decision judge model. A `tool_calls`, `function_call` or `tool_call_id` field that is None or empty, as in OpenAI's `model_dump()` of a reply without tool calls, does not count as a tool call. Strings, text that contains `[TOOL_CALL]` or `[TOOL_RESULT]`, and messages with any other content are still sent to the judge model.
-- `evaluate()` computes `<evaluator>.binary_aggregate` over the rows whose result is `pass` or `fail`. Other rows, such as not applicable or errored rows, used to count as failures, although the mean score leaves them out. An evaluator without `pass` or `fail` rows, including a custom evaluator whose results have other values, has no `binary_aggregate`, as one without scores has no mean.
+- `evaluate()` computes `<evaluator>.binary_aggregate` over the rows whose result is `pass` or `fail`. Other rows, such as not applicable or errored rows, used to count as failures, although the mean score leaves them out. An evaluator without `pass` or `fail` rows, including a custom evaluator whose results have other values, has no `binary_aggregate`, as one without scores has no mean. With several result columns, the last one whose values can be counted decides, as before, so an evaluator whose `pass` and `fail` rows are only in an earlier column also has no `binary_aggregate`.
 - An evaluator's inputs keep the order in which its `__call__` overloads declare them, instead of the order of a Python `set`, which varies with the process's hash seed. An evaluator that sends these inputs to a Decision model unchanged, as several Microsoft Foundry catalog evaluators do, could list the same fields in a different order in each process, and the Decision model could answer the same input differently. The built-in evaluators in this package already send their Decision inputs in a fixed order.
 
 ## 1.18.10 (2026-10-22)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_decision_strategy.py

@@ -57,8 +57,8 @@ def decision_skip_reason(
     if probability >= cutoff:
         return None
     return (
-        f"Status completion probability {probability:.4f} is below {cutoff:.4f}; "
-        "evaluation skipped. No model rationale is available."
+        f"the judge model's applicability probability {probability:.4f} is below the {cutoff:.4f} cutoff, "
+        "so the evaluator was skipped. No model rationale is available."
     )
 
 
@@ -74,7 +74,9 @@ def decision_token_counts(response: Dict[str, Any]) -> Dict[str, Any]:
 
 
 def _label(result_key: str) -> str:
-    return result_key.replace("_", " ")
+    # A caller can namespace a question key, as in "grade/confirm_details"; the label names only the question.
+    leaf = result_key.rsplit("/", 1)[-1]
+    return (leaf or result_key).replace("_", " ")
 
 
 def _digest(question: Dict[str, Any]) -> str:

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_task_adherence/_task_adherence.py

@@ -76,7 +76,8 @@ class TaskAdherenceEvaluator(PromptyEvaluatorBase[Union[str, float]]):
     _supports_decision_models = True
     _binary_decision = True
 
-    _DEFAULT_TASK_ADHERENCE_SCORE = 0
+    # A binary result passes only at a score of 1, so Decision results report 1 unless a threshold is requested.
+    _DEFAULT_TASK_ADHERENCE_SCORE = 1
 
     _validator: ValidatorInterface
 

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_tool_call_success/_tool_call_success.py

@@ -4,8 +4,9 @@
 import math
 import os
 import logging
+import re
 import reprlib
-from typing import Any, Dict, Union, List, Optional
+from typing import Any, Dict, Union, List, Optional, Tuple
 from typing_extensions import overload, override
 from azure.ai.evaluation._exceptions import (
     EvaluationException,
@@ -16,7 +17,7 @@
 from azure.ai.evaluation._evaluators._common import PromptyEvaluatorBase
 from azure.ai.evaluation._evaluators._common._decision_rubric import rubric_strategy, with_probability_threshold
 from azure.ai.evaluation._evaluators._common._validators import ToolDefinitionsValidator, ValidatorInterface
-from azure.ai.evaluation._common.utils import _log_safe_summary, _stringify_tool_result
+from azure.ai.evaluation._common.utils import _drop_mcp_approval_messages, _log_safe_summary, _stringify_tool_result
 from azure.ai.evaluation._common._experimental import experimental
 
 logger = logging.getLogger(__name__)
@@ -26,7 +27,22 @@
 _TEXT_CONTENT_TYPES = frozenset({"text", "input_text", "output_text"})
 _CONVERSATION_ROLES = frozenset({"system", "developer", "user", "assistant"})
 _TOOL_CALL_KEYS = ("tool_calls", "function_call")
-_TOOL_TRACE_MARKERS = ("[TOOL_CALL]", "[TOOL_RESULT]")
+# Tool calls and results written as text: this evaluator's [TOOL_CALL] and [TOOL_RESULT] markers, tags such as
+# <tool_call>, [TOOL_CALLS] and <function_calls>, and the tool tokens of Llama and gpt-oss models.
+_TOOL_TRACE_MARKER = re.compile(
+    r"tool_call|tool_result|tool_response|function_call|function_result|<\|python_tag\|>|<function=|<\|call\|>",
+    re.IGNORECASE,
+)
+# A line that starts with a tool step label, as ReAct agents ("Action:", "Observation 2:") and LangChain's printed
+# messages ("Tool Calls:") write it, also in a Markdown list, quote, heading or emphasis, after a bullet or a list
+# marker such as "1)", "(2)" or "a)", and with a fullwidth colon. The character classes hold no line breaks and the
+# list marker is bounded, so a search takes time linear in the length of the text.
+_TOOL_STEP_LABEL = re.compile(
+    r"^[ \t\u00a0>*_#+`.\u2022\d-]*(?:\(?(?:\d{1,3}|[a-z])\)[ \t\u00a0]*)?"
+    r"(?:action|observation|(?:tool|function)[ _-]?(?:call|result|response|output)s?)"
+    r"[ \t\d*_#.()`]*[:\uff1a]",
+    re.IGNORECASE | re.MULTILINE,
+)
 _APPROVAL_TYPES = frozenset({"mcp_approval_request", "mcp_approval_response"})
 
 
@@ -151,7 +167,12 @@ def __call__(
 
         A list of messages that holds only text has no tool calls, so it is reported as not applicable without
         calling the judge model. A tool_calls, function_call or tool_call_id field that is None or empty does not
-        count as a tool call.
+        count as a tool call. A list of text messages that looks like a tool transcript still goes to the judge model,
+        as text with each message's role and every tool definition: text that contains tool_call, tool_result,
+        tool_response, function_call or function_result in any letter case, as tags such as <tool_call> and
+        [TOOL_CALLS] do, Llama's <|python_tag|> or <function= tags or the gpt-oss <|call|> token, or a line that starts
+        with a label such as Action:, Observation:, Tool call: or Tool result:, as ReAct agents write. Pass a transcript
+        in any other text format as a string so the judge model assesses it.
 
         :keyword response: The response being evaluated, either a string or a list of messages (full agent
             response potentially including tool calls)
@@ -213,12 +234,17 @@ async def _do_eval(self, eval_input: Dict) -> Dict[str, Union[str, float]]:  # t
                 target=ErrorTarget.TOOL_CALL_SUCCESS_EVALUATOR,
             )
 
+        transcript = None
         if isinstance(eval_input.get("response"), list):
-            # Decided before preprocessing, which drops whole messages that hold an MCP approval.
+            # Decided on the raw list, so a list of only MCP approvals stays not applicable; the text of a message
+            # that preprocessing drops for holding an MCP approval is not counted.
             if _has_no_tool_calls(eval_input["response"]):
                 return self._return_not_applicable_result(_NO_TOOL_CALLS_MESSAGE, self._threshold)
             eval_input["response"] = _preprocess_messages(eval_input["response"])
-            eval_input["tool_calls"] = _reformat_tool_calls_results(eval_input["response"], logger)
+            transcript = _tool_transcript(eval_input["response"])
+            eval_input["tool_calls"] = (
+                transcript if transcript is not None else _reformat_tool_calls_results(eval_input["response"], logger)
+            )
         # If response is a string, pass directly without reformatting
         elif isinstance(eval_input["response"], str):
             eval_input["tool_calls"] = eval_input["response"]
@@ -236,8 +262,8 @@ async def _do_eval(self, eval_input: Dict) -> Dict[str, Union[str, float]]:  # t
         # If tool definitions are string, pass directly without reformatting, else format it.
         if "tool_definitions" in eval_input and not isinstance(eval_input["tool_definitions"], str):
             tool_definitions = eval_input["tool_definitions"]
-            # Only if response is not a string, we filter tool definitions to only tools needed.
-            if not isinstance(eval_input["response"], str):
+            # Keep only the tools that were called, unless the calls are in text: a string or a transcript.
+            if not isinstance(eval_input["response"], str) and transcript is None:
                 tool_definitions = _filter_to_used_tools(
                     tool_definitions=tool_definitions,
                     msgs_list=eval_input["response"],
@@ -322,18 +348,20 @@ def _is_one_of(value: Any, names: frozenset) -> bool:
     return isinstance(value, str) and value in names
 
 
-def _has_no_tool_calls(messages) -> bool:
-    """Return True only for a list of text messages and MCP approvals that holds no tool calls, results or traces.
+def _texts_of_text_messages(messages) -> Optional[List[Tuple[str, str]]]:
+    """Return the role and text of each text in a list of text messages and MCP approvals, or None for any other value.
 
     A tool_calls, function_call or tool_call_id field that holds nothing, such as the None that OpenAI's model_dump()
     writes, counts as absent. A list in any other format may hold tool activity this evaluator does not parse, so the
-    judge assesses it.
+    judge assesses it. The text of a message that preprocessing drops for holding an MCP approval is left out, as no
+    judge input includes it.
     """
     if not isinstance(messages, list) or not messages:
-        return False
+        return None
+    texts = []
     for message in messages:
         if not isinstance(message, dict) or not all(_holds_nothing(message.get(key)) for key in _TOOL_CALL_KEYS):
-            return False
+            return None
         content = message.get("content")
         items = content
         if isinstance(content, list):
@@ -346,22 +374,47 @@ def _has_no_tool_calls(messages) -> bool:
                 # Only MCP approval requests or responses, which are not tool calls; a response names its request
                 # in tool_call_id.
                 continue
-        if not _is_one_of(message.get("role"), _CONVERSATION_ROLES) or not _holds_nothing(message.get("tool_call_id")):
-            return False
+        role = message.get("role")
+        if not _is_one_of(role, _CONVERSATION_ROLES) or not _holds_nothing(message.get("tool_call_id")):
+            return None
         if isinstance(content, str):
-            texts = [content]
+            message_texts = [content]
         elif isinstance(items, list) and all(
             isinstance(item, dict) and _is_one_of(item.get("type"), _TEXT_CONTENT_TYPES) for item in items
         ):
-            texts = [item.get("text") for item in items]
+            message_texts = [item.get("text") for item in items]
         else:
-            return False
-        # Text can carry tool calls in this evaluator's own trace format.
-        if not all(
-            isinstance(text, str) and not any(marker in text for marker in _TOOL_TRACE_MARKERS) for text in texts
-        ):
-            return False
-    return True
+            return None
+        if not all(isinstance(text, str) for text in message_texts):
+            return None
+        if _drop_mcp_approval_messages([message]):
+            texts.extend((role, text) for text in message_texts)
+    return texts
+
+
+def _has_no_tool_calls(messages) -> bool:
+    """Return True only for a list of text messages and MCP approvals that holds no tool calls, results or traces.
+
+    Text that looks like a tool transcript, such as ReAct steps or <tool_call> tags, counts as tool activity.
+    """
+    texts = _texts_of_text_messages(messages)
+    return texts is not None and not any(_holds_tool_transcript(text) for _, text in texts)
+
+
+def _tool_transcript(messages) -> Optional[str]:
+    """Return the text of a list of text messages that holds a tool transcript, with each message's role, or None.
+
+    A Decision model can miss ReAct steps in a list of messages that it recognizes in the same text on its own.
+    """
+    texts = _texts_of_text_messages(messages)
+    if texts is None or not any(_holds_tool_transcript(text) for _, text in texts):
+        return None
+    return "\n\n".join(f"{role}: {text}" for role, text in texts if text)
+
+
+def _holds_tool_transcript(text: str) -> bool:
+    """Return True when the text records tool calls or results in a text format this evaluator recognizes."""
+    return _TOOL_TRACE_MARKER.search(text) is not None or _TOOL_STEP_LABEL.search(text) is not None
 
 
 def _filter_to_used_tools(tool_definitions, msgs_list, logger=None):

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_version.py

@@ -3,4 +3,4 @@
 # ---------------------------------------------------------
 # represents upcoming version
 
-VERSION = "1.18.11"
+VERSION = "1.18.12"