You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Fix Tool Call Success scoring, binary_aggregate and Decision input order, and name Decision failure reasons - #49804
Fixes for azure-ai-evaluation 1.18.11 (unreleased). The Tool Call Success scoring bug also affects chat evaluators in 1.18.10, the binary_aggregate behavior is older, and the Decision input order bug has affected Decision evaluations since 1.18.10, the first release with Decision models.
Token reuse moved to its own PR, #49815. The OpenAI Decisions protocol stays in #49695.
1. Tool Call Success scoring
Problem: all 14 example outputs in tool_call_success.prompty had a trailing comma after "failed_tools": "", which is invalid JSON. gpt-4.1 and gpt-4o then put score and status inside properties, and float(llm_output.get("score", 0)) scored successful tool calls 0. On main, gpt-4o scored all 3 successful tool-call message rows 0, and gpt-4.1 and gpt-4o scored all 3 successful tool traces 0.
Fix:
The examples are valid JSON. With this, every chat judge in the live runs below returned a top-level score.
score, status and reason fall back to properties when the top-level value is missing.
A score other than 0 or 1 raises a system-classified EvaluationException, as _ToolInputAccuracyEvaluator does, instead of becoming 0. Values that already read as 0 or 1, such as True, "1" and 1.0, still do.
2. Tool Call Success applicability
Problem: on main, a response with no tool calls went to the judge, and gpt-4o-mini, gpt-5-mini and o3-mini scored text-only responses as passed.
Fix: a list of messages that holds only text or MCP approvals, with no tool calls, results or [TOOL_CALL]/[TOOL_RESULT] traces, is now not applicable without a judge call, for chat and Decision models. A tool_calls, function_call or tool_call_id field counts only when it holds something: None, "", [] and {}, as in OpenAI's model_dump() of a reply without tool calls, are treated as absent, and any other value, such as [None] or False, still counts. A role or content type that is not a string, such as a list or a dict, is left to the judge or to the input validator, as before. The decision is made before preprocessing. Strings, text that carries tool traces, and any other content still go to the judge.
3. Decision failure reasons
Problem: a Decision request the provider refused with HTTP 422 failed as "The decision model returned HTTP 422." The provider states the reason only in a detail sentence, so callers could not tell a context window overflow from an invalid question.
Fix:
decision_model.error_kind in the error's properties names a recognized reason, _decision_error_metadata keeps it on failed result rows, and the message adds it, for example The decision model returned HTTP 422 (context_window_exceeded).
context_window_exceeded: the evidence and one question exceed the model's context window.
request_token_limit_exceeded: the evidence and all questions exceed the model's request limit.
question_limit_exceeded and invalid_question.
request_bytes_exceeded: a request over max_request_bytes or max_state_question_bytes, which is not sent.
A reason is read from the fixed opening of the provider's sentence, within its first 1,024 characters, so a quoted question identifier cannot change it. The sentence itself is never surfaced.
The Decision retry policy no longer passes retry_on_status_codes. azure-core adds its defaults to that list, and they already include every status it named, so retries are unchanged.
4. binary_aggregate
Problem: evaluate() reported <evaluator>.binary_aggregate as the passes divided by all rows, so not applicable and errored rows counted as failures, while the mean score leaves them out. Rows that passed, failed and were not applicable gave a mean of 0.5 and a binary_aggregate of 0.33.
Fix: binary_aggregate is the passes divided by the rows whose result is pass or fail. With several _result columns, the last one decides, as before. When that column has no pass or fail rows, for example a custom evaluator whose results use other values, the evaluator has no binary_aggregate instead of 0.0, as one without scores has no mean. Azure OpenAI graders' <grader>.pass_rate uses the same denominator, although it reports 0.0 when nothing was judged. The evaluation service stores this metric as reported and does not read it. The metric on a run's result asset now agrees with the Foundry portal's pass-rate card, which counts only passed and failed rows: a Tool Call Success run with one passed, one failed and one not applicable row reads 0.5, where it read 0.33.
5. Decision input order
Problem: EvaluatorBase._get_all_singleton_inputs flattened the overloads' input names through a set, so _convert_kwargs_to_eval_input built each evaluator's inputs in the set's order, which varies with the process's hash seed. An evaluator that sends these inputs to a Decision model unchanged, as the Microsoft Foundry catalog's task completion, tool output utilization, output quality and tool use quality evaluators do, listed the same fields in a different order in each process, and the Decision model could answer the same row differently: one task completion row scored P(true) 0.18 in one process and 0.84 in another.
Fix: the names keep the order in which the overloads declare them, so the inputs, and the state of a Decision request that sends them unchanged, have the same order in every process. For those four catalog evaluators the order is query, response, tool_definitions, the order this package's own task completion and tool output utilization evaluators already send. The built-in evaluators in this package build their Decision state in a fixed order and are unchanged, and chat prompts place inputs by name. Catalog evaluators pick this up from the SDK installed in the evaluation worker.
Validation
Rows marked as measured before the split ran while this PR also carried token reuse, which changes token counts, not results.
Check
main
This PR
tests/unittests
2338 passed, 15 skipped
2481 passed, 15 skipped
Decision input order, offline with the catalog assets: state order sent across 8 hash seeds by task completion, tool output utilization, output quality and tool use quality
4 distinct orders for each
One order for each: query, response, tool_definitions
Decision input order, live on Decision-1 through Vienna master's evaluation worker with registry assets, 4 rows per affected asset, hash seeds 0 to 3
3 orders per asset, and borderline rows moved with the process: task completion 0.08 to 0.62, output quality 0.06 to 0.49, tool use quality 0.56 to 0.95
Byte-identical requests in every process; probabilities within 0.03
Decision input order, labeled smoke on Decision-1, 12 rows per asset (6 pass, 6 fail), one run each
Task completion 11 of 12 with each of the 4 orders old processes produced; output quality 12 of 12
Task completion 10 of 12: one row labeled fail passes only with the declared order (0.85, against 0.20 to 0.68 with the old orders); output quality 12 of 12. The sample is too small to rank orders; see the larger study below
Decision input order, larger labeled study on Decision-1, run after the push: task completion 60 rows (30 pass, 30 fail, 36 of them borderline) and tool use quality 32 rows, each scored with every order old processes produced and twice with the declared order
Task completion 58 of 60 with each of the 4 old orders, mean probability of the expected label 0.937 to 0.950; tool use quality 29 to 32 of 32
Task completion 58 of 60, mean probability 0.942; paired against the average old order, -0.003 (95% interval -0.026 to 0.014) and no accuracy difference. Tool use quality 30 of 32; its two misses are rows whose tool arguments resolve "tomorrow" without a stated date, which every order passes once the date is given. The order flips 3 of 60 borderline task completion rows, and the declared order is right on 2 of them. No sign that the declared order is worse
Live Tool Call Success applicability and binary_aggregate on gpt-4o-mini and Decision-1, 16 checks: the review's three null or empty-field messages, an OpenAI model_dump() text reply, OpenAI tool calls that succeed and fail, and evaluate() over pass, fail and text rows and over text-only rows
4 of 16, measured on 4d7a193, whose _evaluate.py matches main and whose earlier shortcut these inputs bypassed: every empty-field and text reply reached the judge, gpt-4o-mini passed the text reply in evaluate(), binary_aggregate was 0.67 on gpt-4o-mini and 0.33 on Decision-1 (means 0.67 and 0.5), and 0.0 for text-only rows
16 of 16: empty-field and text replies are not applicable with no judge call, OpenAI tool calls still reach the judge and pass or fail as expected, binary_aggregate is 0.5 and equal to the mean on both judges, and text-only rows produce neither metric and no judge call
Malformed messages and normal inputs for Tool Call Success, live on gpt-4o-mini and Decision-1, 28 checks: a list or dict role, a dict or list content type, MCP approvals, empty tool-call fields, OpenAI text and tool calls, and evaluate() with one malformed row among good rows
Before the guard in 0028aa3, 22 of 28: a list or dict role raised TypeError
28 of 28: a list or dict role reaches the judge, a dict or list content type gets the input validator's user error with no judge call, the other inputs are unchanged, and the other rows of evaluate() are unaffected
Every helper this PR adds, and the Tool Call Success call path, with 20,000 random JSON-like inputs each (lists, dicts, None, NaN, integers up to 400 digits, nested values)
Not run
No exception other than EvaluationException. Checked after the merge: a score of up to 4,300 digits gets EvaluationException, and a longer one raises ValueError while the error message is built (before this PR, OverflowError). A judge's JSON output can't hold such a number, since json.loads rejects integers over 4,300 digits
Live Decision-1 refusals: a window overflow, a request over the token limit, 256 questions, empty instructions, a question id that quotes the window sentence, a request over max_request_bytes, and CoherenceEvaluator with a window overflow
Each failed as "HTTP 422" with no reason
Each named with its kind; the quoted id stays invalid_question; the request over max_request_bytes made no attempt
Live failed later conversation turn on Decision-1, through _decision_error_metadata
No reason
error_kindcontext_window_exceeded kept with result_usage_scopereported_turns_only
Live Tool Call Success, measured before the split: 7 judges (gpt-4.1, gpt-4o, gpt-4o-mini, gpt-5-mini, o3-mini, gpt-6-luna, Microsoft-Decision-1), 5 cases, 3 runs each
gpt-4o scored successful tool calls 0; gpt-4.1 and gpt-4o scored successful traces 0; three judges passed text-only responses
Every run matched the expected result, except 2 of 105 Decision-1 runs that hit HTTP 429; the other runs of those cases matched
Live built-in evaluators, measured before the split: the 15 Decision-capable built-ins and QAEvaluator on Decision-1, gpt-4o-mini and gpt-6-luna
Baseline
Same result and status for every pair, except luna's QAEvaluator similarity, which main skipped and this run scored; two Decision-1 429s matched main when run alone
Evaluation worker at every level, in process: Vienna master's worker code with registry assets on Decision-1 and gpt-4o-mini; turn level with strings, context, ground truth and tool messages; conversation level; trace level from GenAI, hosted, OpenInference and OpenLLMetry spans at turn and conversation level; Decision failures
Baseline
Same outcomes, apart from judge variation confirmed by reruns with identical requests and one Decision task completion row whose verdict follows the input order (pass with the declared order, fail with the order that process used before); failed Decision rows add error_kind, and binary_aggregate changes as described in section 4
Vienna evaluation worker tests (13.7k) on master bade101a with this wheel
3 failures and 2 collection errors, unrelated to this PR
Identical outcomes
azureml-assets catalog Decision and behavior tests (test_decision_models, test_evaluators_behavior), 4741 tests
1155 passed on Vienna 2640626; the same 2 guard tests fail as on any build since #49781
Hosted smoke, local RAISvc stack with Vienna 2640626's worker and the on-behalf-of exchange not bypassed: Tool Call Success through the portal route on gpt-4o-mini and Decision-1 with pass, fail and text-only rows; a Decision-1 coherence run with one context-window overflow row; and the five catalog evaluators whose Decision input order changed (task_completion, tool_output_utilization, output_quality, tool_use_quality, deflection_rate), registry and project package, on Decision-1
On 4d7a193, whose _evaluate.py matches main: the result asset's binary_aggregate was 0.33 for the Tool Call Success runs and 0.67 for the overflow run
13 of 13 runs as expected. Tool Call Success is pass, fail and not applicable on both judges, and the result asset's binary_aggregate is 0.5, equal to the mean and to the portal's 1 of 2. The overflow row fails with the 422 context-window message, and binary_aggregate is 1 for the two scored rows. All 10 order-affected catalog runs pass. One package run was repeated after the local on-behalf-of token service returned 403 while saving its status, as also happened once on 4d7a193
Hosted, measured before the split: local RAISvc stack, 96 registry (portal route), project package and rubric cases on Decision-1, gpt-4o-mini and gpt-6-luna
Not run
95 of 96. The remaining failure is luna marking the good Similarity row not applicable, a judge behavior also seen with main's SDK and with the released 1.18.10 worker
Hosted, measured before the split: local RAISvc stack with the on-behalf-of exchange not bypassed; registry built-ins, project packages, rubrics, F1 and size cases
Not run
26 of 26. A window overflow is the caller's 422 named context_window_exceeded; a request over the worker's 256,000-byte limit is never sent
The evaluation worker's uptake of this release: its two guard tests change with it.
The catalog's builtin.tool_call_success asset, which the portal route runs, keeps its own copy of this evaluator; the Tool Call Success fixes, including the empty-field rule, reach the portal only with a matching azureml-assets change.
_drop_mcp_approval_messages drops a whole message that mixes an MCP approval with a tool call. This is existing behavior shared by every evaluator; the judge still gets the remaining messages.
Other binary evaluators that default a missing score to 0. Their prompts are valid JSON, and no nested output has been seen from them.
…ss scoring
Token reuse (from Azure#49695, without the OpenAI Decisions protocol):
- Decision requests, and Azure OpenAI chat requests that use Microsoft Entra ID, reuse a credential's token until
five minutes before it expires instead of requesting one per request.
- Concurrent requests that share a synchronous credential wait for one token request, off the event loop; requests
queued behind a token that is not cached use it.
- A 401 drops the token, and an acquisition that began before the rejection is not stored.
- The default chat credential shares one credential and token across concurrent first requests.
- A Decision credential that returns no string token is a user error, not a timeout.
Tool Call Success:
- The prompt's 14 example outputs are now valid JSON, which stops gpt-4.1 and gpt-4o nesting score and status
in properties and scoring successful tool calls 0.
- score, status and reason fall back to properties, and a score other than 0 or 1 raises a system error instead of
becoming 0.
- A list of messages without tool calls (only text or MCP approvals) is not applicable without a judge call;
strings, tool traces in text and any other content still go to the judge.
The reason will be displayed to describe this comment to others. Learn more.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
Azure Pipelines:
Successfully started running 1 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.
…ant retry list
Decision requests now request tokens for https://ai.azure.com/.default, the scope
Azure OpenAI chat requests use, so a credential shared by chat and Decision
evaluators needs one token instead of one per scope. Decision endpoints accept
both audiences. The explicit retry status list is removed: azure-core retries
those statuses by default and unions an explicit list with its defaults, so
retry behavior is unchanged. _get_token documents its absolute monotonic
deadline, and the token cache module records why get_bearer_token_provider is
not used in its place.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Shared cached token-acquisition failures are replayed with incorrect exception types across chat and Decision callers, misclassifying errors and preventing expected credential retries.
… authentication failures with waiting requests
A failed Decision request names a recognized reason in decision_model.error_kind of the error's properties, which is
kept on failed result rows, and in the error message, for example "The decision model returned HTTP 422
(context_window_exceeded).". The provider states these reasons for an HTTP 422 only in text, which is never exposed:
context_window_exceeded, request_token_limit_exceeded, question_limit_exceeded and invalid_question, recognized by the
sentence's fixed opening so a quoted question identifier cannot change the kind. A request over max_request_bytes or
max_state_question_bytes, which is not sent, has the kind request_bytes_exceeded.
Requests that waited on a token request now share its failure only when it failed authentication, as on the chat
path; after any other failure each waiting request makes its own attempt, so a transient failure fails only the
request that hit it.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
A shared token-acquisition failure causes Decision waiters to reuse raw chat authentication diagnostics, exposing credential details and incorrectly classifying the error instead of applying Decision-specific sanitization.
When a chat token acquisition fails authentication while a Decision request waits on the same credential and scope, the Decision waiter reconstructs EvaluationException from chat's raw message rather than its own sanitized error metadata. This produces an InternalError with unknown blame/category and exposes credential diagnostics instead of the expected user-classified token error. Store a caller-independent failure description, apply the waiting caller's formatter when replaying it, and add a mixed chat/Decision regression test.
… within each kind of request
Decision endpoints accept both audiences, but the hosted evaluation service selects the identity for Decision calls by token audience: for https://ai.azure.com it forwards the caller's own token, and for the Cognitive Services audience it exchanges that token or uses the project managed identity. With the chat scope, single-tenant projects would call Decision as the end user instead of the project managed identity, and the service's worker tests fail. Decision requests go back to https://cognitiveservices.azure.com/.default and their own cached token, and the redundant retry status list stays removed.
The token cache records a failure per failure formatter and exception type, with the attempt that ended in it, so a request shares only a failure of its own kind that ended after it queued, and a successful token request clears them. A request could otherwise replay another kind's failure with its own exception type: a Decision description raised as a chat ClientAuthenticationError, or a chat authentication failure raised as an EvaluationException with an unknown blame.
A credential that implements AsyncTokenCredential but whose get_token is a plain function returning an awaitable was asked on a worker thread, where a get_token that needs the running event loop failed. A credential whose get_token is a coroutine function, or that implements AsyncTokenCredential, is now asked on the caller's event loop, as released chat requests did.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Null or empty tool-call fields bypass the no-tool-call check, allowing text-only messages to reach a judge and potentially pass instead of being marked not_applicable.
A text-only message with tool_calls: null, tool_calls: [], or function_call: null bypasses the no-tool-call check solely because the key exists. These inputs pass the conversation validator and still reach the chat or Decision judge, which can incorrectly mark them as passed instead of not_applicable. Treat null or empty call fields as absent while retaining the check for populated calls.
…rotocol on a worker thread
A synchronous credential whose class also implements the asynchronous context manager protocol matched the structural AsyncTokenCredential check and was asked on the event loop, which blocked the loop, lost the shared token request and broke a get_token that runs its own event loop. A credential that implements the synchronous context manager protocol is again asked on a worker thread. A plain get_token that needs the caller's running loop must belong to an AsyncTokenCredential, as the CHANGELOG and docstrings now state, and fetch documents that its failure formatter and exception type key the shared failures and must be module-level.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because the new lock breaks default-backed evaluator serialization, while transient authentication outages can fail every queued evaluation.
Token reuse for Decision and chat requests is not part of the Tool Call
Success and Decision failure reason fixes, so it moves to a separate pull
request. This restores the base _prompty.py and _async_token_provider.py,
removes _token_cache.py and its tests, and returns DecisionModelRunner to
requesting a token for each call, with the Cognitive Services scope as
before. The error_kind reasons, the retry policy cleanup and the Tool Call
Success fixes are unchanged.
A tool_calls, function_call or tool_call_id field that is None or an empty
string, list or dict no longer sends a text-only response to the judge.
OpenAI's model_dump() writes tool_calls: None and function_call: None on every
reply, so those replies reached the chat or Decision judge and could pass.
Populated fields and values of any other type still go to the judge.
evaluate() divided an evaluator's passes by every row, so not applicable and
errored rows counted as failures and binary_aggregate disagreed with the mean
score. It now divides by the rows that passed or failed, and an evaluator
without such rows gets no binary_aggregate, as one without scores gets no mean.
_get_all_singleton_inputs flattened the overloads' input names through a
set, so an evaluator's converted inputs followed the set's order, which
varies with the process's hash seed. An evaluator that sends these inputs
to a Decision model unchanged, as several Microsoft Foundry catalog
evaluators do, could therefore list the same fields in a different order
in each process, and the Decision model could answer the same input
differently. The names now keep the order in which the overloads declare
them. The built-in evaluators in this package build their Decision state
in a fixed order and are unchanged.
An evaluator with several _result columns reports its last one, as before.
A last column without pass or fail rows now leaves no binary_aggregate
instead of an earlier column's rate. The CHANGELOG says only pass and fail
count, which also affects custom evaluators that return other values.
Tests cover all-fail and boolean result columns, several result columns,
and OpenAI tool calls on the Decision judge.
python -c puts the working directory first on the path, so in CI jobs that
install the built wheel and run pytest from the package directory, the
child processes imported the checkout instead of the installed package.
They now start in an empty temporary directory, and three seeds keep the
test short.
bonanza89
changed the title
Fix Tool Call Success scoring and name Decision failure reasons
Fix Tool Call Success scoring, binary_aggregate and Decision input order, and name Decision failure reasons
Nov 4, 2026
The reason will be displayed to describe this comment to others. Learn more.
Re-reviewed d100061. The previously reported P2 applicability issue is fixed: null/empty tool-call fields no longer trigger a judge call, while populated calls remain eligible. No additional actionable, high-confidence findings in the current diff, including the pass-rate denominator and deterministic input-order changes.
Review included isolated source-level checks. Full package unit tests, pandas integration tests, and live tests were not independently run.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The new applicability check raises TypeError for unhashable message type or role values, bypassing the existing formatter fallback instead of leaving unsupported formats for the judge.
The no-tool-call check compared role and content type values with sets,
so an unhashable value, such as a list role or a dict type, raised
TypeError. A value that is not a string now fails the check, so the
message goes to the judge or the input validator, as it did before the
check existed.
… the judge
Since Azure#49804, a list of text messages was reported as not applicable without a judge call unless its text held this evaluator's [TOOL_CALL] or [TOOL_RESULT] markers. A transcript in another text format, such as ReAct Action and Observation lines or <tool_call> tags, was left out of the pass rate, together with its failed tool calls.
Text that contains tool_call, tool_result, tool_response, function_call or function_result in any letter case, Llama's <|python_tag|> or <function= tags, the gpt-oss <|call|> token, or a line that starts with a label such as Action:, Observation:, Tool call: or Tool result: now counts as a tool transcript. Such a list goes to the judge model as its text, with each message's role and every tool definition, as a string does, because Decision-1 answered not applicable for ReAct steps inside a list of messages but judged the same text correctly.
Also corrects the 1.18.11 CHANGELOG entries for error_kind, the score check and binary_aggregate.
Keeps both sides' additions where Azure#49804 and this change touched the same lines of _model_runners.py
and test_decision_model_support.py, and moves the token reuse entry from the released 1.18.11 section
to a 1.18.12 section with the version bump.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
EvaluationIssues related to the client library for Azure AI Evaluation
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
github-actions
Bot
added
the
Evaluation
Sashank Kolli (sakoll)
left a comment
Fixes for azure-ai-evaluation 1.18.11 (unreleased). The Tool Call Success scoring bug also affects chat evaluators in 1.18.10, the
binary_aggregatebehavior is older, and the Decision input order bug has affected Decision evaluations since 1.18.10, the first release with Decision models.Token reuse moved to its own PR, #49815. The OpenAI Decisions protocol stays in #49695.
1. Tool Call Success scoring
Problem: all 14 example outputs in
tool_call_success.promptyhad a trailing comma after"failed_tools": "", which is invalid JSON. gpt-4.1 and gpt-4o then putscoreandstatusinsideproperties, andfloat(llm_output.get("score", 0))scored successful tool calls 0. On main, gpt-4o scored all 3 successful tool-call message rows 0, and gpt-4.1 and gpt-4o scored all 3 successful tool traces 0.Fix:
score,statusandreasonfall back topropertieswhen the top-level value is missing.EvaluationException, as_ToolInputAccuracyEvaluatordoes, instead of becoming 0. Values that already read as 0 or 1, such asTrue,"1"and1.0, still do.2. Tool Call Success applicability
Problem: on main, a response with no tool calls went to the judge, and gpt-4o-mini, gpt-5-mini and o3-mini scored text-only responses as passed.
Fix: a list of messages that holds only text or MCP approvals, with no tool calls, results or
[TOOL_CALL]/[TOOL_RESULT]traces, is now not applicable without a judge call, for chat and Decision models. Atool_calls,function_callortool_call_idfield counts only when it holds something: None,"",[]and{}, as in OpenAI'smodel_dump()of a reply without tool calls, are treated as absent, and any other value, such as[None]orFalse, still counts. Aroleor contenttypethat is not a string, such as a list or a dict, is left to the judge or to the input validator, as before. The decision is made before preprocessing. Strings, text that carries tool traces, and any other content still go to the judge.3. Decision failure reasons
Problem: a Decision request the provider refused with HTTP 422 failed as "The decision model returned HTTP 422." The provider states the reason only in a
detailsentence, so callers could not tell a context window overflow from an invalid question.Fix:
decision_model.error_kindin the error's properties names a recognized reason,_decision_error_metadatakeeps it on failed result rows, and the message adds it, for exampleThe decision model returned HTTP 422 (context_window_exceeded).context_window_exceeded: the evidence and one question exceed the model's context window.request_token_limit_exceeded: the evidence and all questions exceed the model's request limit.question_limit_exceededandinvalid_question.request_bytes_exceeded: a request overmax_request_bytesormax_state_question_bytes, which is not sent.retry_on_status_codes. azure-core adds its defaults to that list, and they already include every status it named, so retries are unchanged.4. binary_aggregate
Problem:
evaluate()reported<evaluator>.binary_aggregateas the passes divided by all rows, so not applicable and errored rows counted as failures, while the mean score leaves them out. Rows that passed, failed and were not applicable gave a mean of 0.5 and abinary_aggregateof 0.33.Fix:
binary_aggregateis the passes divided by the rows whose result ispassorfail. With several_resultcolumns, the last one decides, as before. When that column has nopassorfailrows, for example a custom evaluator whose results use other values, the evaluator has nobinary_aggregateinstead of 0.0, as one without scores has no mean. Azure OpenAI graders'<grader>.pass_rateuses the same denominator, although it reports 0.0 when nothing was judged. The evaluation service stores this metric as reported and does not read it. The metric on a run's result asset now agrees with the Foundry portal's pass-rate card, which counts only passed and failed rows: a Tool Call Success run with one passed, one failed and one not applicable row reads 0.5, where it read 0.33.5. Decision input order
Problem:
EvaluatorBase._get_all_singleton_inputsflattened the overloads' input names through aset, so_convert_kwargs_to_eval_inputbuilt each evaluator's inputs in the set's order, which varies with the process's hash seed. An evaluator that sends these inputs to a Decision model unchanged, as the Microsoft Foundry catalog's task completion, tool output utilization, output quality and tool use quality evaluators do, listed the same fields in a different order in each process, and the Decision model could answer the same row differently: one task completion row scored P(true) 0.18 in one process and 0.84 in another.Fix: the names keep the order in which the overloads declare them, so the inputs, and the state of a Decision request that sends them unchanged, have the same order in every process. For those four catalog evaluators the order is query, response, tool_definitions, the order this package's own task completion and tool output utilization evaluators already send. The built-in evaluators in this package build their Decision state in a fixed order and are unchanged, and chat prompts place inputs by name. Catalog evaluators pick this up from the SDK installed in the evaluation worker.
Validation
Rows marked as measured before the split ran while this PR also carried token reuse, which changes token counts, not results.
tests/unittestsbinary_aggregateon gpt-4o-mini and Decision-1, 16 checks: the review's three null or empty-field messages, an OpenAImodel_dump()text reply, OpenAI tool calls that succeed and fail, andevaluate()over pass, fail and text rows and over text-only rows_evaluate.pymatches main and whose earlier shortcut these inputs bypassed: every empty-field and text reply reached the judge, gpt-4o-mini passed the text reply inevaluate(),binary_aggregatewas 0.67 on gpt-4o-mini and 0.33 on Decision-1 (means 0.67 and 0.5), and 0.0 for text-only rowsbinary_aggregateis 0.5 and equal to the mean on both judges, and text-only rows produce neither metric and no judge callrole, a dict or list contenttype, MCP approvals, empty tool-call fields, OpenAI text and tool calls, andevaluate()with one malformed row among good rowsroleraisedTypeErrorrolereaches the judge, a dict or list contenttypegets the input validator's user error with no judge call, the other inputs are unchanged, and the other rows ofevaluate()are unaffectedEvaluationException. Checked after the merge: a score of up to 4,300 digits getsEvaluationException, and a longer one raisesValueErrorwhile the error message is built (before this PR,OverflowError). A judge's JSON output can't hold such a number, sincejson.loadsrejects integers over 4,300 digitsmax_request_bytes, andCoherenceEvaluatorwith a window overflowinvalid_question; the request overmax_request_bytesmade no attempt_decision_error_metadataerror_kindcontext_window_exceededkept withresult_usage_scopereported_turns_onlyQAEvaluatoron Decision-1, gpt-4o-mini and gpt-6-lunaQAEvaluatorsimilarity, which main skipped and this run scored; two Decision-1 429s matched main when run aloneerror_kind, andbinary_aggregatechanges as described in section 4test_decision_models,test_evaluators_behavior), 4741 tests_evaluate.pymatches main: the result asset'sbinary_aggregatewas 0.33 for the Tool Call Success runs and 0.67 for the overflow runbinary_aggregateis 0.5, equal to the mean and to the portal's 1 of 2. The overflow row fails with the 422 context-window message, andbinary_aggregateis 1 for the two scored rows. All 10 order-affected catalog runs pass. One package run was repeated after the local on-behalf-of token service returned 403 while saving its status, as also happened once on 4d7a193context_window_exceeded; a request over the worker's 256,000-byte limit is never sentNot in this PR
builtin.tool_call_successasset, which the portal route runs, keeps its own copy of this evaluator; the Tool Call Success fixes, including the empty-field rule, reach the portal only with a matching azureml-assets change._drop_mcp_approval_messagesdrops a whole message that mixes an MCP approval with a tool call. This is existing behavior shared by every evaluator; the judge still gets the remaining messages.