Fix Tool Call Success scoring, binary_aggregate and Decision input order, and name Decision failure reasons

sdk/evaluation/azure-ai-evaluation/CHANGELOG.md

@@ -8,6 +8,14 @@
 - `QAEvaluator` accepts a Decision model configuration and passes `credential` and the Decision execution controls to its model-based evaluators, which run on separate threads, so the credential must be synchronous. With a chat model they are not passed, as before.
 - The binary evaluators (task adherence, task completion, tool call success, tool input accuracy, tool output utilization and tool selection) accept a `probability_threshold` keyword. For Decision models it sets the probability at or above which the score is 1 (default 0.75) and overrides the value in `model_config`; chat models ignore it with a warning.
 - Conversation inputs are handled as before with a Decision model: evaluators that accept them score each user and assistant turn separately and aggregate the results. The catalog's whole-conversation rubrics are not included.
+- A failed Decision request names a recognized reason in `decision_model.error_kind` of the error's properties, which is kept on failed result rows, and in the error message, for example `The decision model returned HTTP 422 (context_window_exceeded).` The provider states these reasons for an HTTP 422 only in text, which is never exposed: `context_window_exceeded` when the evidence and one question exceed the model's context window, `request_token_limit_exceeded` when the evidence and all questions exceed the model's request limit, `question_limit_exceeded` and `invalid_question`. A request over `max_request_bytes` or `max_state_question_bytes`, which is not sent, has the kind `request_bytes_exceeded`.
+
+### Bugs Fixed
+
+- `_ToolCallSuccessEvaluator` no longer scores successful tool calls 0 when a chat judge model, such as gpt-4.1 or gpt-4o, puts `score` and `status` inside `properties`. The example outputs in its prompt are now valid JSON, and `score`, `status` and `reason` are read from `properties` when the output has none at the top level. A missing score is no longer reported as 0, and any score other than 0 or 1 raises a system-classified `EvaluationException`, as `_ToolInputAccuracyEvaluator` does.
+- `_ToolCallSuccessEvaluator` reports a list of messages that holds no tool calls, such as one that contains only text or only MCP approval messages, as not applicable without calling the chat or Decision judge model. A `tool_calls`, `function_call` or `tool_call_id` field that is None or empty, as in OpenAI's `model_dump()` of a reply without tool calls, does not count as a tool call. Strings, text that contains `[TOOL_CALL]` or `[TOOL_RESULT]`, and messages with any other content are still sent to the judge model.
+- `evaluate()` computes `<evaluator>.binary_aggregate` over the rows whose result is `pass` or `fail`. Other rows, such as not applicable or errored rows, used to count as failures, although the mean score leaves them out. An evaluator without `pass` or `fail` rows, including a custom evaluator whose results have other values, has no `binary_aggregate`, as one without scores has no mean.
+- An evaluator's inputs keep the order in which its `__call__` overloads declare them, instead of the order of a Python `set`, which varies with the process's hash seed. An evaluator that sends these inputs to a Decision model unchanged, as several Microsoft Foundry catalog evaluators do, could list the same fields in a different order in each process, and the Decision model could answer the same input differently. The built-in evaluators in this package already send their Decision inputs in a fixed order.
 
 ## 1.18.10 (2026-10-22)
 

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluate/_batch_run/_run_submitter_client.py

@@ -22,7 +22,7 @@
 from ..._evaluate._utils import _has_aggregator
 from ..._constants import Prefixes, PF_BATCH_TIMEOUT_SEC
 from ..._exceptions import EvaluationException
-from ..._evaluators._common._model_runners import _is_finite_number, validated_execution
+from ..._evaluators._common._model_runners import DECISION_ERROR_KINDS, _is_finite_number, validated_execution
 
 from .._utils import get_int_env_var as get_int
 
@@ -56,6 +56,7 @@ def _decision_error_metadata(exception: EvaluationException) -> Dict[str, Any]:
             ("backend", ("foundry_decision", "system_one")),
             ("protocol", ("shared_sdk", "system_one")),
             ("result_usage_scope", ("reported_attempts_only", "final_response_only", "reported_turns_only")),
+            ("error_kind", DECISION_ERROR_KINDS),
         ):
             value = native.get(key)
             if isinstance(value, str) and value in accepted:

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluate/_evaluate.py

@@ -273,7 +273,10 @@ def _aggregation_binary_output(df: pd.DataFrame) -> Dict[str, float]:
     """
     Aggregate binary output results (pass/fail) from evaluation dataframe.
 
-    For each evaluator, calculates the proportion of "pass" results.
+    For each evaluator, calculates the proportion of "pass" results among the rows whose result is "pass" or "fail".
+    Other rows, such as not applicable or errored rows, are left out, as they are from the mean score, and a result
+    column without "pass" or "fail" rows gives no binary aggregate. An evaluator with several result columns reports
+    its last one.
 
     :param df: The dataframe of evaluation results.
     :type df: ~pandas.DataFrame
@@ -312,14 +315,16 @@ def _aggregation_binary_output(df: pd.DataFrame) -> Dict[str, float]:
                 )
                 continue
 
-            # Calculate the proportion of EVALUATION_PASS_FAIL_MAPPING[True] results
-            total_rows = len(df)
             pass_count = value_counts.get(EVALUATION_PASS_FAIL_MAPPING[True], 0)
-            proportion = pass_count / total_rows if total_rows > 0 else 0.0
+            judged_rows = pass_count + value_counts.get(EVALUATION_PASS_FAIL_MAPPING[False], 0)
 
             # Set the result with the evaluator name as the key
             result_key = f"{evaluator_name}.{BINARY_AGGREGATE_SUFFIX}"
-            results[result_key] = round(proportion, 2)
+            if judged_rows == 0:
+                # The evaluator's last result column decides, so an earlier column's rate is not kept.
+                results.pop(result_key, None)
+                continue
+            results[result_key] = round(pass_count / judged_rows, 2)
 
     return results
 

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_base_eval.py

@@ -249,13 +249,14 @@ def _get_matching_overload_inputs(self, **kwargs) -> List[str]:
     def _get_all_singleton_inputs(self) -> List[str]:
         """Get a flattened list of all possible singleton inputs across all overloads.
 
+        Names keep the order in which the overloads declare them, so an evaluator's inputs, and the state of a decision
+        request that sends them unchanged, have the same order in every process. A set's order would vary with the
+        process's hash seed.
+
         :return: Flattened list of all singleton input names.
         :rtype: List[str]
         """
-        all_inputs = set()
-        for inputs in self._singleton_inputs:
-            all_inputs.update(inputs)
-        return list(all_inputs)
+        return list(dict.fromkeys(name for inputs in self._singleton_inputs for name in inputs))
 
     def _derive_conversation_converter(
         self,

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_model_runners.py

@@ -45,7 +45,6 @@ def value(self) -> str:
 DECISION_MODEL_TYPE = "decision"
 _DECISION_PATH = "/providers/microsoft/v1/systemone"
 _DECISION_TOKEN_SCOPE = "https://cognitiveservices.azure.com/.default"
-_DECISION_RETRY_STATUS_CODES = [408, 429, 500, 502, 503, 504, 529]
 LOGGER = logging.getLogger(__name__)
 _PREFIX = "azure.ai.evaluation.decision."
 _DECISION_CONFIG_KEYS = {
@@ -59,6 +58,29 @@ def value(self) -> str:
 _DEFAULT_PROBABILITY_THRESHOLD = 0.75
 # The service adds api_version to every resolved model configuration; the decision endpoint does not use it.
 _IGNORED_CONFIG_KEYS = {"api_version"}
+_ERROR_CODE = re.compile(r"[A-Za-z][A-Za-z0-9_.-]{0,127}")
+# A 422 states its reason only in a ``detail`` sentence, which is never surfaced. These reasons are recognized by the
+# provider's fixed sentence openings, so a quoted question identifier later in the sentence cannot change the kind, and
+# reported as a bounded ``decision_model.error_kind``; any other sentence is ignored. Only the sentence's first
+# characters are read, which hold every opening, so a long question identifier cannot hide the reason.
+_PROVIDER_DETAIL_KINDS = (
+    (
+        re.compile(r"invalid model request: Question prompt exceeds the \d+-token context limit\b"),
+        "context_window_exceeded",
+    ),
+    (
+        re.compile(
+            r"invalid TypeSafe request: request is about \d+ tokens; the limit for the state plus all questions is \d+"
+        ),
+        "request_token_limit_exceeded",
+    ),
+    (re.compile(r"invalid TypeSafe request: request has \d+ questions; the limit is \d+"), "question_limit_exceeded"),
+    (re.compile(r"invalid TypeSafe request: invalid [a-z_]+ for question \""), "invalid_question"),
+)
+_MAX_PROVIDER_DETAIL_CHARS = 1024
+# Reported when a request exceeds the caller's max_request_bytes or max_state_question_bytes and is not sent.
+_REQUEST_BYTES_EXCEEDED = "request_bytes_exceeded"
+DECISION_ERROR_KINDS = tuple(kind for _, kind in _PROVIDER_DETAIL_KINDS) + (_REQUEST_BYTES_EXCEEDED,)
 
 
 def is_decision_model_config(model_config: Any) -> bool:
@@ -340,6 +362,26 @@ def _is_finite_number(value: Any) -> bool:
         return False
 
 
+def _provider_error_code(payload: Any) -> Optional[str]:
+    """Return the provider's nested ``error.code`` when it is a bounded identifier, otherwise None."""
+    provider_error = payload.get("error") if isinstance(payload, dict) else None
+    code = provider_error.get("code") if isinstance(provider_error, dict) else None
+    return code if isinstance(code, str) and _ERROR_CODE.fullmatch(code) is not None else None
+
+
+def _provider_error_kind(status: int, payload: Any) -> Optional[str]:
+    """Name a 422 that states its reason only in a ``detail`` sentence, from the sentence's fixed opening.
+
+    For example ``{"detail": "invalid model request: Question prompt exceeds the 32768-token context limit"}`` is
+    ``context_window_exceeded``. Any other status or unrecognized sentence returns None and is never surfaced.
+    """
+    detail = payload.get("detail") if status == 422 and isinstance(payload, dict) else None
+    if not isinstance(detail, str):
+        return None
+    opening = detail[:_MAX_PROVIDER_DETAIL_CHARS]
+    return next((kind for phrase, kind in _PROVIDER_DETAIL_KINDS if phrase.match(opening)), None)
+
+
 def _probability(value: Any) -> float:
     if not _is_finite_number(value):
         raise ValueError("Decision probabilities must be finite numbers.")
@@ -415,6 +457,10 @@ class DecisionModelRunner:
     it stops awaiting synchronous credentials but cannot stop their worker threads. Byte limits measure
     compact unescaped UTF-8 JSON, with ``max_state_question_bytes`` applying to the largest individual pair.
     ``max_response_bytes`` limits acceptance before parsing, not transport allocation.
+    A failed request names a recognized reason in ``decision_model.error_kind`` of the error's properties:
+    ``context_window_exceeded`` when the evidence and one question exceed the model's context window,
+    ``request_token_limit_exceeded``, ``question_limit_exceeded`` or ``invalid_question`` for the provider's other
+    rejections, and ``request_bytes_exceeded`` for a request over the byte limits, which is not sent.
     """
 
     def __init__(
@@ -583,6 +629,7 @@ async def __call__(self, *, state: Any, questions: Dict[str, Dict[str, Any]], ti
             if (self._max_request_bytes is not None and len(content) > self._max_request_bytes) or (
                 self._max_state_question_bytes is not None and state_question_bytes > self._max_state_question_bytes
             ):
+                properties["decision_model"]["error_kind"] = _REQUEST_BYTES_EXCEEDED
                 raise _failed("The decision request exceeds the configured byte budget.", ErrorBlame.USER_ERROR)
             try:
                 payload = await asyncio.wait_for(
@@ -670,13 +717,14 @@ async def _post(self, content: bytes, *, deadline: float, execution: DecisionExe
 
         policies: Dict[str, Any] = {
             "custom_hook_policy": execution,
+            # azure-core retries 408, 429, 500, 502, 503, 504 and 506 through 998 by default; POST is opted in
+            # per request below.
             "retry_policy": AsyncRetryPolicy(
                 retry_total=self._max_retries,
                 retry_connect=self._max_retries,
                 retry_read=self._max_retries,
                 retry_status=self._max_retries,
                 retry_backoff_factor=0.8,
-                retry_on_status_codes=_DECISION_RETRY_STATUS_CODES,
             ),
             "redirect_policy": AsyncRedirectPolicy(permit_redirects=False),
         }
@@ -719,15 +767,13 @@ async def _post(self, content: bytes, *, deadline: float, execution: DecisionExe
                 payload = None
         if status != 200:
             blame = ErrorBlame.USER_ERROR if status in (400, 401, 403, 404, 422) else ErrorBlame.SYSTEM_ERROR
-            provider_error = payload.get("error") if isinstance(payload, dict) else None
-            code = provider_error.get("code") if isinstance(provider_error, dict) else None
-            diagnostic = (
-                f" ({code})"
-                if isinstance(code, str) and re.fullmatch(r"[A-Za-z][A-Za-z0-9_.-]{0,127}", code) is not None
-                else ""
-            )
+            code = _provider_error_code(payload)
+            kind = _provider_error_kind(status, payload)
+            diagnostic = f" ({code or kind})" if code or kind else ""
             error = _failed(f"The decision model returned HTTP {status}{diagnostic}.", blame)
             error.properties = self._response_properties(payload)
+            if kind is not None:
+                error.properties["decision_model"]["error_kind"] = kind
             raise error
         return payload