Add Azure OpenAI Decisions protocol and evaluation metrics to evaluation runners

sdk/evaluation/azure-ai-evaluation/CHANGELOG.md

@@ -1,5 +1,17 @@
 # Release History
 
+## 1.18.12 (Unreleased)
+
+### Features Added
+
+- Added `decision_protocol="openai"` for Decision evaluators, which calls Azure OpenAI's `/openai/v1/decisions` operation on the account's OpenAI endpoint. The Microsoft Decision protocol remains the default, and model names never select a protocol. When an Azure OpenAI Decisions request fails with `model_not_found`, the user error says to set `decision_protocol` to `microsoft`, or remove it, if the deployment is a Microsoft Decision model. Error messages on the Microsoft Decision protocol are unchanged.
+- Each protocol is a small adapter that translates the shared Boolean and ordinal questions to its provider contract and binds answers back to the same identifiers, so evaluators keep their scoring, applicability and probability cutoffs. OpenAI score levels are labeled `Level 1` to `Level n`, matching rubric text such as "imply level 2". An OpenAI answer of any other type is rejected as an invalid response, and score probabilities that are not a list of distinct levels fail when the evaluation reads them. The shared transport, authentication, retries, deadlines, cancellation and byte budgets are reused; no OpenAI client upgrade is required. Byte budgets measure the same canonical request on every protocol, so OpenAI admits the same evidence as Microsoft Decision.
+- Decision provenance names the protocol in `decision_model.decision_protocol` (`microsoft` or `openai`), and the per-request metrics and execution logs carry it as `protocol`. Provider cache and reasoning counters are kept under `decision_model.usage_details` without being added to token totals, and only when the token count they describe part of is known. Split conversations sum each counter over the turns that report that token count, so the counters never cover requests that the token totals omit.
+- An OpenAI question can be refused. A refusal fails an evaluation only when the evaluation reads that answer, so a skipped evaluation never depends on its grade. The failure is a user-classified `EvaluationException` with known usage and `decision_model.error_kind` set to `model_refusal`.
+- A provider content filter's rejection (HTTP 400 with code `content_filter`) keeps its user classification and sets `decision_model.error_kind` to `content_filter`. Failed result rows keep both new kinds, as they keep the kinds added in 1.18.11.
+- Evidence that is not finite JSON data fails with the same user-classified error on every protocol.
+- Each `DecisionEvaluationStrategy.evaluate` call, which requests and interprets one Decision evaluation, records the optional OpenTelemetry metrics `azure.ai.evaluation.decision.evaluations` (count) and `azure.ai.evaluation.decision.evaluation.duration` (ms), with `protocol`, `metric`, `outcome` (`completed`, `skipped`, `error` or `cancelled`, including cancellation by the cancellation callback), `blame` (`none`, `user` or `system`) and `error_kind` (`none` or the failure's `decision_model.error_kind`, such as `context_window_exceeded`, `content_filter` or `model_refusal`). `metric` is the result key of this package's and the Microsoft Foundry catalog's Decision evaluators, and `custom` for any other evaluator, so a customer-defined name never becomes a metric dimension. The metrics complement the per-request metrics: an HTTP 200 response that fails validation or is not applicable is visible here. A conversation scored turn by turn counts each turn. Inputs found not applicable before any request are not counted, and evaluators that interpret a shared request without `evaluate`, such as composite evaluators, are covered by the per-request metrics only. A metric failure, including a broken OpenTelemetry installation, never changes the evaluation result or error.
+
 ## 1.18.11 (2026-11-03)
 
 ### Features Added

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluate/_batch_run/_run_submitter_client.py

@@ -22,7 +22,13 @@
 from ..._evaluate._utils import _has_aggregator
 from ..._constants import Prefixes, PF_BATCH_TIMEOUT_SEC
 from ..._exceptions import EvaluationException
-from ..._evaluators._common._model_runners import DECISION_ERROR_KINDS, _is_finite_number, validated_execution
+from ..._evaluators._common._decision_protocols import DECISION_PROTOCOLS
+from ..._evaluators._common._model_runners import (
+    DECISION_ERROR_KINDS,
+    _is_finite_number,
+    usage_details,
+    validated_execution,
+)
 
 from .._utils import get_int_env_var as get_int
 
@@ -55,6 +61,7 @@ def _decision_error_metadata(exception: EvaluationException) -> Dict[str, Any]:
         for key, accepted in (
             ("backend", ("foundry_decision", "system_one")),
             ("protocol", ("shared_sdk", "system_one")),
+            ("decision_protocol", tuple(DECISION_PROTOCOLS)),
             ("result_usage_scope", ("reported_attempts_only", "final_response_only", "reported_turns_only")),
             ("error_kind", DECISION_ERROR_KINDS),
         ):
@@ -63,6 +70,9 @@ def _decision_error_metadata(exception: EvaluationException) -> Dict[str, Any]:
                 provenance[key] = value
         if isinstance(native.get("usage_available"), bool):
             provenance["usage_available"] = native["usage_available"]
+        details = usage_details(native.get("usage_details"))
+        if details:
+            provenance["usage_details"] = details
         execution = validated_execution(native.get("execution"))
         if execution is not None:
             provenance["execution"] = execution

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_base_model_eval.py

@@ -4,7 +4,7 @@
 
 import asyncio
 import math
-from typing import Any, Callable, Dict, List, Optional, TypeVar, Union
+from typing import Any, Callable, Dict, Iterable, List, Optional, Tuple, TypeVar, Union
 
 from azure.core.credentials import TokenCredential
 from azure.ai.evaluation._constants import EVALUATION_PASS_FAIL_MAPPING
@@ -13,8 +13,10 @@
 from ._model_runners import (
     DecisionModelRunner,
     PromptyModelRunner,
+    USAGE_DETAIL_TOKENS,
     _is_finite_number,
     is_decision_model_config,
+    usage_details,
 )
 
 T = TypeVar("T")
@@ -68,7 +70,9 @@ class ModelEvaluatorBase(EvaluatorBase[T]):
     :keyword cancellation_requested: Optional Decision cancellation callback, polled during authentication,
         requests and retry waits. Cancelling stops awaiting synchronous credential calls, not their worker threads.
     :paramtype cancellation_requested: Optional[Callable[[], bool]]
-    :keyword max_request_bytes: Optional limit on the compact UTF-8 serialized Decision request.
+    :keyword max_request_bytes: Optional limit on the compact UTF-8 JSON of the canonical Decision request
+        (deployment, evidence and questions), measured before protocol encoding so every protocol admits the same
+        evidence.
     :paramtype max_request_bytes: Optional[int]
     :keyword max_state_question_bytes: Optional limit on the largest individual state/question pair.
     :paramtype max_state_question_bytes: Optional[int]
@@ -202,10 +206,12 @@ def _decision_model_runner(self) -> DecisionModelRunner:
     def _aggregate_results(self, per_turn_results: List[Dict[str, T]]) -> Dict[str, Any]:
         """Aggregate scores normally, but sum independently known Decision usage across turns.
 
-        A single shared model is promoted for attribution. Mixed or missing model identities remain per-turn
-        rather than attributing all usage to the last model. Root properties use the same usage and model,
-        without a single-request execution summary; other diagnostics retain their final-turn semantics.
-        Prompty aggregation is unchanged.
+        Token totals are each summed over the turns that reported them, and cache or reasoning counters over the
+        turns that reported the token count they describe part of. A single shared model is promoted for
+        attribution. Mixed or missing model identities remain per-turn rather than attributing all usage to the last
+        model. When the final turn has properties, root properties use the same usage and model, without a
+        single-request execution summary; other diagnostics retain their final-turn semantics. Prompty aggregation is
+        unchanged.
         """
         aggregated: Dict[str, Any] = super()._aggregate_results(per_turn_results)
         if not self._uses_decision_model:
@@ -235,11 +241,15 @@ def _aggregate_results(self, per_turn_results: List[Dict[str, T]]) -> Dict[str,
         properties_key = f"{self._result_key}_properties"
         properties = aggregated.get(properties_key)
         if len(per_turn_results) > 1 and isinstance(properties, dict):
-            usage = self._known_turn_usage(
-                aggregated, ("prompt_tokens", "completion_tokens", "total_tokens"), prefix=f"{self._result_key}_"
+            prefix = f"{self._result_key}_"
+            fields = ("prompt_tokens", "completion_tokens", "total_tokens")
+            usage = self._known_turn_usage(aggregated, fields, prefix=prefix)
+            details = self._summed_usage_details(
+                (turn.get(properties_key), self._known_turn_usage(turn, fields, prefix=prefix))
+                for turn in per_turn_results
             )
             aggregated[properties_key] = self._reconcile_decision_properties(
-                properties, usage, aggregated.get(f"{self._result_key}_model")
+                properties, usage, aggregated.get(f"{self._result_key}_model"), details
             )
         return aggregated
 
@@ -248,24 +258,30 @@ def _on_evaluation_error(
     ) -> None:
         """Preserve known Decision turn usage without changing the failure or Prompty behavior.
 
-        Only the request-owning result's usage is accumulated. A failed turn's execution summary cannot
-        describe all these turns, so it is omitted and the wider usage scope is explicit.
+        Only the request-owning result's usage is accumulated, and cache or reasoning counters only from the turns
+        that reported the token count they describe part of. A failed turn's execution summary cannot describe all
+        these turns, so it is omitted and the wider usage scope is explicit. When no earlier turn counted usage, the
+        failure's own properties already describe every counted request and are left unchanged.
         """
         if not self._uses_decision_model or not per_turn_results:
             return
         fields = ("prompt_tokens", "completion_tokens", "total_tokens")
+        properties_key = f"{self._result_key}_properties"
         contributions = []
+        reported = []
         for result in per_turn_results:
             usage = self._known_turn_usage(result, fields, prefix=f"{self._result_key}_")
             if usage:
                 contributions.append((usage, result.get(f"{self._result_key}_model")))
+                reported.append((result.get(properties_key), usage))
         if not contributions:
             return
         error_properties = getattr(error, "properties", None)
         properties = dict(error_properties) if isinstance(error_properties, dict) else {}
         failing_usage = self._known_turn_usage(properties, fields)
         if failing_usage:
             contributions.append((failing_usage, properties.get("model")))
+            reported.append((properties, failing_usage))
         usage = {}
         for field in fields:
             known = [item[field] for item, _ in contributions if field in item]
@@ -275,11 +291,39 @@ def _on_evaluation_error(
         common_model = models[0] if isinstance(models[0], str) and models[0].strip() else None
         if not all(model == common_model for model in models):
             common_model = None
-        setattr(error, "properties", self._reconcile_decision_properties(properties, usage, common_model))
+        details = self._summed_usage_details(reported)
+        setattr(error, "properties", self._reconcile_decision_properties(properties, usage, common_model, details))
+
+    @staticmethod
+    def _summed_usage_details(turns: Iterable[Tuple[Any, Dict[str, int]]]) -> Dict[str, Dict[str, int]]:
+        """Sum each known cache or reasoning counter over the turns whose matching token count is counted.
+
+        Input counters are summed over the turns that report prompt tokens and output counters over the turns that
+        report completion tokens, so the counters never cover requests that the token totals omit.
+
+        :param turns: The properties and known token usage of each turn whose Decision response is accounted for.
+        :type turns: Iterable[Tuple[Any, Dict[str, int]]]
+        :return: The summed counters grouped like the provider's usage object; empty when no turn reported any.
+        :rtype: Dict[str, Dict[str, int]]
+        """
+        totals: Dict[str, Dict[str, int]] = {}
+        for properties, usage in turns:
+            native = properties.get("decision_model") if isinstance(properties, dict) else None
+            details = usage_details(native.get("usage_details")) if isinstance(native, dict) else {}
+            for group, counters in details.items():
+                if USAGE_DETAIL_TOKENS[group] not in usage:
+                    continue
+                summed = totals.setdefault(group, {})
+                for counter, value in counters.items():
+                    summed[counter] = summed.get(counter, 0) + value
+        return totals
 
     @staticmethod
     def _reconcile_decision_properties(
-        properties: Dict[str, Any], usage: Dict[str, int], model: Optional[str]
+        properties: Dict[str, Any],
+        usage: Dict[str, int],
+        model: Optional[str],
+        details: Optional[Dict[str, Dict[str, int]]] = None,
     ) -> Dict[str, Any]:
         reconciled = dict(properties)
         for field in ("prompt_tokens", "completion_tokens", "total_tokens"):
@@ -294,6 +338,10 @@ def _reconcile_decision_properties(
         native = reconciled.get("decision_model")
         native = dict(native) if isinstance(native, dict) else {}
         native.pop("execution", None)
+        if details:
+            native["usage_details"] = details
+        else:
+            native.pop("usage_details", None)
         if "model" in reconciled:
             native["model"] = reconciled["model"]
         else: