[bonanza89]

Follow-up to #49804. 1.18.11 shipped without this change, so it targets 1.18.12.

#49804 made _ToolCallSuccessEvaluator report a list of messages that holds only text as not applicable without calling the judge model. Text with this evaluator's own [TOOL_CALL] or [TOOL_RESULT] markers still went to the judge, but a tool transcript in any other text format did not. A ReAct agent's Action: and Observation: steps, or a model's raw <tool_call> tags, inside a message list came back not applicable, so a failed tool call written that way was left out of the pass rate. In 1.18.10 these lists went to the judge.

Changes

• Text counts as a tool transcript when it contains tool_call, tool_result, tool_response, function_call or function_result in any letter case, which covers tags such as <tool_call>, <tool_response>, [TOOL_CALLS] and <function_calls> as well as this evaluator's own markers. Llama's <|python_tag|> and <function= tags, the gpt-oss <|call|> token, and a line that starts with a label such as Action:, Observation:, Tool call: or Tool result: also count; the labels may carry step numbers and Markdown decoration. Messages of every role are checked, so a tool result passed as a user message counts.
• A label also counts after a bullet or a list marker such as 1), (2) or a), after indentation with non-breaking spaces, and with a fullwidth colon. Prose such as (action: see below), a) The next action: and Reaction: still does not count.
• A list of text messages with such text goes to the judge model as its text, one role: text block per message, with every tool definition, as a string does. Before, the judge got the raw list. In live checks, Decision-1 answered not applicable for a ReAct transcript inside a list that also held the user's question, both for a failed and a successful call, but judged the same text correctly. Messages that hold an MCP approval are still left out, as preprocessing leaves them out of every judge input. Their text no longer counts as a transcript either, so a list whose only transcript text is in such a message stays not applicable without a judge call.
• _has_no_tool_calls and the new _tool_transcript share one helper, _texts_of_text_messages, so the rules for what counts as a text message stay in one place.
• A Decision evaluator that the judge model rates as not applicable gives a readable reason: Not applicable: the judge model's applicability probability 0.2000 is below the 0.5000 cutoff, so the evaluator was skipped. No model rationale is available. It used to read Status completion probability 0.2000 is below 0.5000; evaluation skipped., which named an internal question. Only the wording changes; the cutoff, the numbers and the not applicable result are the same.
• Decision reasons name a question by the part of its key after the last /. The evaluation service grades Boolean rubric dimensions through BinaryScoring with keys such as grade/confirm_details, so its reasons read Grade/confirm details probability 0.1645 is below the 0.7500 cutoff.; they now read Confirm details probability 0.1645 is below the 0.7500 cutoff. The SDK's own question keys have no /, so its reasons are unchanged.
• TaskAdherenceEvaluator with a Decision model reports threshold 1 when none is requested, the only score at which its binary result passes, as _TaskCompletionEvaluator and the other binary evaluators do. It reported 0, below the catalog evaluator's own minimum of 1. A requested threshold is still reported as given and does not decide whether a binary result passes. The threshold attribute also defaults to 1 for chat models; chat results still report 3.
• CHANGELOG and version: adds a 1.18.12 (Unreleased) section with the transcript, skip reason, reason label and Task adherence threshold entries, bumps _version.py to 1.18.12, and keeps the 1.18.11 Tool Call Success entry as it shipped. It also corrects three other 1.18.11 entries found in review. The error_kind entry now says when the kind appears in the error message, the score check entry no longer says it matches _ToolInputAccuracyEvaluator, and the binary_aggregate entry says the last result column whose values can be counted decides.

Strings, structured tool calls and lists in other formats are unchanged. The Microsoft Foundry portal and the evaluation service run their own copy of this evaluator, so they are not affected. The service's copy of Task adherence in azureml-assets also defaults to 0 and needs the same change there. The service takes the reason label fix when it moves to 1.18.12, since it calls BinaryScoring from this package.

Behavior

Text in a list of messages	1.18.10	1.18.11 (#49804)	This PR
ReAct Action: and Observation: steps, <tool_call> tags, [TOOL_CALLS], LangChain's printed "Tool Calls:"	Judged, raw list	Not applicable, no judge call	Judged, as text with roles
Plain text, including "Next action: ..." or "Action items:"	Judged, raw list	Not applicable, no judge call	Not applicable, no judge call
Text with [TOOL_CALL] or [TOOL_RESULT]	Judged, raw list	Judged, raw list	Judged, as text with roles
A system prompt that describes one of these formats, and no tool call	Judged, raw list	Not applicable, no judge call	Judged; both judges returned not applicable in live checks

Testing

Check	Result
Unit tests, tests/unittests	2553 passed, 15 skipped
New tests run against main's evaluator (ce8639d)	The 54 new transcript cases fail; the 11 new plain-text cases pass on both
black	Clean
Decision unit tests (test_builtin_decision_evaluators.py, test_decision_*.py) after the skip reason change	739 passed; the updated reason check fails on the old wording in all 15 cases
black 24.4.0 at line length 120 on the two changed Python files	Clean
Detection time on adversarial inputs of 0.25 to 2 million characters: long runs of spaces, line breaks and Markdown decoration, labels without colons, near-miss markers	Linear, about 8 times as long for 8 times the input; slowest 0.4 s at 2 million characters
The line-label pattern against a line-by-line reference, 200,000 random texts	0 mismatches
50,000 random message lists with malformed roles, types, texts and fields, against the helper on main	Only the 807 lists that hold a transcript change, each gets a transcript, and nothing raises
Live, gpt-4o-mini (2024-07-18) and Microsoft-Decision-1 judges: ReAct failure and success, Hermes failure, LangChain-printed failure	Right verdict on both judges; main reports all four as not applicable without a judge call
Live: plain text, and "Next action: ..." with "Action items: none."	Not applicable on both judges, no judge call
Live evaluate() with ReAct failure, ReAct success, plain text and Hermes failure rows	fail, pass, not applicable and fail on both judges; binary_aggregate 0.33
Live with tool definitions, transcript as string content and as a text item	Same verdict for both on both judges
Live: plain replies with no tool activity that mention tool_calls, have an "Action:" line, or have a Markdown "Observation:" bullet	One judge call each, where main makes none; both judges return not applicable
Second run on the built wheel: unit suite, black, and 28 live checks of malformed roles and content types, model_dump() fields, MCP approval rows and evaluate() with mixed rows	Same as before this change: 28 of 28 on both judges, binary_aggregate 0.5
Spelling (cspell 10.2.2 with the repo config) and the docs build (Sphinx 8.2.0 with -W)	No new findings; _ToolCallSuccessEvaluator is not in the generated docs

Added with the reason label and Task adherence threshold changes (e9fb612):

Check	Result
Unit tests, tests/unittests	2567 passed, 31 skipped
New tests: requested thresholds 0 and 2 on Task adherence and Task completion, the chat default, and Binary and Likert reasons for namespaced keys	11 cases pass
The label helper on 20,010 random and edge-case keys: empty, /, grade/, 100,000 characters, non-ASCII	Never raises, keys without / are unchanged, and non-string keys raise as before
black 24.4.0 at line length 120	Clean
Live, Microsoft-Decision-1: TaskAdherenceEvaluator without a threshold, with threshold=0, and on a plainly adherent answer	Threshold 1, 0 and 1; a result passes only at score 1

Added with the list marker and MCP approval text changes (93ec9a0):

Check	Result
Unit tests, tests/unittests	2586 passed, 31 skipped
New tests: labels after 1), (2), a) and a bullet, with non-breaking space indentation and a fullwidth colon; prose such as (action: see below), a) The next action: and Reaction:; MCP approval text alone and beside a user turn	Pass; the two MCP approval cases fail without the change
Detection time on adversarial inputs of up to 2 million characters	Linear; slowest under 0.6 s
black 24.4.0 at line length 120	Clean
Live, Microsoft-Decision-1: numbered 1) and bulleted transcripts in message lists	Both go to the judge, where 1.18.11 reported both not applicable; the numbered failure is judged fail
Live: MCP approval text alone, and plain answers	Not applicable, no judge call

Not changed

• A list with structured tool calls and a text transcript in other messages sends only the structured calls to the judge, as in 1.18.10.
• A transcript in another text format still needs to be passed as a string; the docstring and CHANGELOG say so.
• A structured tool call in the same message as an MCP approval is still left out with that message by the shared preprocessing.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🟢 Approval recommended

No active review issues remain for the changes restoring evaluation of recognized free-text tool transcripts.

No open issues 🎉

What changed in this PR

Restores judging of recognized free-text tool transcripts inside message lists in azure-ai-evaluation, addressing the applicability regression introduced by #49804 while preserving the no-judge path for plain text.

Changes:

• Detects common tool transcript markers and step labels, and sends matching text-only message lists to the judge with roles and all tool definitions.
• Adds regression coverage for transcript formats, message shapes, judge inputs, and aggregate scoring.
• Documents the supported formats and corrects existing release-note descriptions.

File	Description
sdk/​evaluation/​azure-ai-evaluation/​tests/​unittests/​test_tool_call_success_evaluator.py	Adds transcript detection, formatting, and scoring regression tests.
sdk/​evaluation/​azure-ai-evaluation/​CHANGELOG.md	Documents transcript support and clarifies related release notes.
sdk/​evaluation/​azure-ai-evaluation/​azure/​ai/​evaluation/​_evaluators/​_tool_call_success/​_tool_call_success.py	Shares text extraction logic and routes recognized transcripts to the judge as role-labeled text.

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

No active review issues remain for the transcript-handling and Decision evaluator skip-reason updates.

No open issues 🎉

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

The transcript parser misses Markdown list prefixes such as 1), causing failed tool calls to bypass judging and be excluded from the pass rate.

1 open issue

• Recognize ) ordered-list prefixes when detecting tool calls · New

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

No active review issues are reported for the tool-transcript evaluation and Decision metadata fixes.

1 open issue

• Recognize ) ordered-list prefixes when detecting tool calls

_{Review effort: Balanced}