Repository navigation
Conversation
github-actions
Bot
added
the
Evaluation
|
Azure Pipelines: Successfully started running 1 pipeline(s). 10 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
5 of 6 tasks
Contributor
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
bonanza89
changed the title
Contributor
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
Contributor
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
Contributor
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
Contributor
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
Contributor
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
added 6 commits
November 2, 2026 22:32
Replace the OpenAI-specific runner path with a small DecisionProtocol adapter layer. The runner selects the Microsoft or OpenAI adapter from decision_protocol, and each adapter only translates the canonical request and binds answers back to the canonical shape, so evaluators keep their scoring, applicability and cutoffs unchanged. - OpenAI score levels are labeled Level 1 to Level n, matching rubric text such as "imply level 2"; answers still bind by zero-based level index. - Byte budgets measure the canonical request on every protocol, and invalid evidence fails with the same user error on every protocol. - A refusal fails only when an evaluator reads it (error_kind model_refusal), and a provider content filter rejection is marked error_kind content_filter. - Provenance and execution metrics name the protocol and keep provider cache and reasoning counters under usage_details. Split conversations sum each counter over the turns whose token usage is counted, on success and on failure, so the counters never cover requests the token totals omit. - Each evaluation run through DecisionEvaluationStrategy.evaluate records the optional OpenTelemetry metrics azure.ai.evaluation.decision.evaluations and azure.ai.evaluation.decision.evaluation.duration with protocol, metric, outcome, blame and error_kind. The runner marks a cancellation by the cancellation callback, so it is recorded as cancelled even when its execution summary is omitted. A metric sink or OpenTelemetry failure never changes the result or the error.
The sample now imports CoherenceEvaluator from azure.ai.evaluation, which accepts a Decision model configuration in 1.18.11, instead of requiring an azureml-assets checkout. It also notes the error for a deployment that serves the other protocol, and that GPT-6 Luna used as a chat judge needs is_reasoning_model=True.
…tocol A Decision deployment that serves the other protocol now fails with a hint that names the decision_protocol value to use: Microsoft Decision answers such a deployment with unknown_model and Azure OpenAI Decisions with model_not_found. Other errors are unchanged.
bonanza89
force-pushed
the
users/aqardahji/openai-decisions
branch
from
November 3, 2026 06:55
ee99b73 to
c0b6931
Compare
bonanza89
changed the title
bonanza89
added a commit
that referenced
this pull request
Nov 4, 2026
…der, and name Decision failure reasons (#49804) * Reuse credential tokens across model requests and fix Tool Call Success scoring Token reuse (from #49695, without the OpenAI Decisions protocol): - Decision requests, and Azure OpenAI chat requests that use Microsoft Entra ID, reuse a credential's token until five minutes before it expires instead of requesting one per request. - Concurrent requests that share a synchronous credential wait for one token request, off the event loop; requests queued behind a token that is not cached use it. - A 401 drops the token, and an acquisition that began before the rejection is not stored. - The default chat credential shares one credential and token across concurrent first requests. - A Decision credential that returns no string token is a user error, not a timeout. Tool Call Success: - The prompt's 14 example outputs are now valid JSON, which stops gpt-4.1 and gpt-4o nesting score and status in properties and scoring successful tool calls 0. - score, status and reason fall back to properties, and a score other than 0 or 1 raises a system error instead of becoming 0. - A list of messages without tool calls (only text or MCP approvals) is not applicable without a judge call; strings, tool traces in text and any other content still go to the judge. * Share the chat token scope with Decision requests and drop the redundant retry list Decision requests now request tokens for https://ai.azure.com/.default, the scope Azure OpenAI chat requests use, so a credential shared by chat and Decision evaluators needs one token instead of one per scope. Decision endpoints accept both audiences. The explicit retry status list is removed: azure-core retries those statuses by default and unions an explicit list with its defaults, so retry behavior is unchanged. _get_token documents its absolute monotonic deadline, and the token cache module records why get_bearer_token_provider is not used in its place. * Name the reason a failed Decision request was refused, and share only authentication failures with waiting requests A failed Decision request names a recognized reason in decision_model.error_kind of the error's properties, which is kept on failed result rows, and in the error message, for example "The decision model returned HTTP 422 (context_window_exceeded).". The provider states these reasons for an HTTP 422 only in text, which is never exposed: context_window_exceeded, request_token_limit_exceeded, question_limit_exceeded and invalid_question, recognized by the sentence's fixed opening so a quoted question identifier cannot change the kind. A request over max_request_bytes or max_state_question_bytes, which is not sent, has the kind request_bytes_exceeded. Requests that waited on a token request now share its failure only when it failed authentication, as on the chat path; after any other failure each waiting request makes its own attempt, so a transient failure fails only the request that hit it. * Keep Decision on the Cognitive Services scope and keep token failures within each kind of request Decision endpoints accept both audiences, but the hosted evaluation service selects the identity for Decision calls by token audience: for https://ai.azure.com it forwards the caller's own token, and for the Cognitive Services audience it exchanges that token or uses the project managed identity. With the chat scope, single-tenant projects would call Decision as the end user instead of the project managed identity, and the service's worker tests fail. Decision requests go back to https://cognitiveservices.azure.com/.default and their own cached token, and the redundant retry status list stays removed. The token cache records a failure per failure formatter and exception type, with the attempt that ended in it, so a request shares only a failure of its own kind that ended after it queued, and a successful token request clears them. A request could otherwise replay another kind's failure with its own exception type: a Decision description raised as a chat ClientAuthenticationError, or a chat authentication failure raised as an EvaluationException with an unknown blame. A credential that implements AsyncTokenCredential but whose get_token is a plain function returning an awaitable was asked on a worker thread, where a get_token that needs the running event loop failed. A credential whose get_token is a coroutine function, or that implements AsyncTokenCredential, is now asked on the caller's event loop, as released chat requests did. * Format the Decision error-kind patterns and tests with black The package enforces black, and these lines were not formatted. No code changes. * Format the Decision batch persistence test with black The Analyze job's black check failed on this file. No code changes. * Ask credentials that also implement the synchronous context manager protocol on a worker thread A synchronous credential whose class also implements the asynchronous context manager protocol matched the structural AsyncTokenCredential check and was asked on the event loop, which blocked the loop, lost the shared token request and broke a get_token that runs its own event loop. A credential that implements the synchronous context manager protocol is again asked on a worker thread. A plain get_token that needs the caller's running loop must belong to an AsyncTokenCredential, as the CHANGELOG and docstrings now state, and fetch documents that its failure formatter and exception type key the shared failures and must be module-level. * Move token reuse to its own pull request Token reuse for Decision and chat requests is not part of the Tool Call Success and Decision failure reason fixes, so it moves to a separate pull request. This restores the base _prompty.py and _async_token_provider.py, removes _token_cache.py and its tests, and returns DecisionModelRunner to requesting a token for each call, with the Cognitive Services scope as before. The error_kind reasons, the retry policy cleanup and the Tool Call Success fixes are unchanged. * Treat empty tool call fields as absent in Tool Call Success A tool_calls, function_call or tool_call_id field that is None or an empty string, list or dict no longer sends a text-only response to the judge. OpenAI's model_dump() writes tool_calls: None and function_call: None on every reply, so those replies reached the chat or Decision judge and could pass. Populated fields and values of any other type still go to the judge. * Count only passed and failed rows in binary_aggregate evaluate() divided an evaluator's passes by every row, so not applicable and errored rows counted as failures and binary_aggregate disagreed with the mean score. It now divides by the rows that passed or failed, and an evaluator without such rows gets no binary_aggregate, as one without scores gets no mean. * Keep evaluator inputs in their declared order _get_all_singleton_inputs flattened the overloads' input names through a set, so an evaluator's converted inputs followed the set's order, which varies with the process's hash seed. An evaluator that sends these inputs to a Decision model unchanged, as several Microsoft Foundry catalog evaluators do, could therefore list the same fields in a different order in each process, and the Decision model could answer the same input differently. The names now keep the order in which the overloads declare them. The built-in evaluators in this package build their Decision state in a fixed order and are unchanged. * Keep the last result column deciding binary_aggregate An evaluator with several _result columns reports its last one, as before. A last column without pass or fail rows now leaves no binary_aggregate instead of an earlier column's rate. The CHANGELOG says only pass and fail count, which also affects custom evaluators that return other values. Tests cover all-fail and boolean result columns, several result columns, and OpenAI tool calls on the Decision judge. * Run the hash seed test's processes from an empty directory python -c puts the working directory first on the path, so in CI jobs that install the built wheel and run pytest from the package directory, the child processes imported the checkout instead of the installed package. They now start in an empty temporary directory, and three seeds keep the test short. * Leave unsupported role and content type values to the judge The no-tool-call check compared role and content type values with sets, so an unhashable value, such as a list role or a dict type, raised TypeError. A value that is not a string now fails the check, so the message goes to the judge or the input validator, as it did before the check existed. --------- Co-authored-by: Ahmad Qardahji <aqardahji@microsoft.com>
added 3 commits
November 6, 2026 14:11
Keep the failure reasons, retry policy and changelog of released 1.18.11, and report content filter and refusal kinds through the same error kind list.
…lt-path errors unchanged - Evaluation metrics report any bounded decision_model.error_kind, including the five that 1.18.11 names, and failed rows keep them through one list. - The metric label is the result key of a known SDK or catalog evaluator, or custom, so customer evaluator names are never metric dimensions. Labels are computed inside the guarded metric call. - Microsoft Decision errors carry no protocol hint, as in 1.18.11; the OpenAI model_not_found hint is conditional. - Provenance names the protocol in decision_model.decision_protocol, leaving decision_model.protocol to the hosted route values. A runner without a protocol counts as microsoft and an unrecognized one as unknown. - Cache and reasoning counters are kept only with the token count they describe, on failed requests and when summing split conversations. - OpenAI answers of an unexpected type are rejected, and malformed level probabilities fail when read. - Changelog entries move to 1.18.12 (Unreleased), the version is bumped, and the sample names the first release with decision_protocol.
…etric names to the Decision evaluators - Tests fail if a successful split conversation sums a counter without its token count, if decision_model.protocol is written for SDK provenance, or if a Decision evaluator in this package is missing from the metric names. - quality_grader is chat only, so it is not a metric name. - The changelog says error messages on the Microsoft Decision protocol are unchanged, and the runner docstring notes that the OpenAI encoding is larger than the canonical request the byte limits measure.
bonanza89
marked this pull request as draft
November 7, 2026 00:46
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Existing Decision evaluators speak the Microsoft Decision contract. This adds a small protocol adapter layer to the shared runner from #49486, so
decision_protocol="openai"calls Azure OpenAI's/openai/v1/decisions, and existing configurations keep the Microsoft route.DecisionProtocolhas three operations: the operation URL, encoding the canonical request, and binding answers back to the canonical shape.MicrosoftDecisionProtocolis the identity;OpenAIDecisionProtocolturns Boolean questions into predicates that state both criteria, ordinal criteria into levels labeledLevel 1toLevel n(matching rubric text such as "imply level 2"), and binds answers by name and zero-based level index. Evaluators keep their scoring, applicability and cutoffs.error_kindmodel_refusal), so skipped evaluations never depend on their grade. A provider content filter's 400 is markedcontent_filter. Failed rows keep both kinds, as they keep the kinds that 1.18.11 names.model_not_founderror says to setdecision_protocoltomicrosoft, or remove it, if the deployment is a Microsoft Decision model. Microsoft Decision error messages are the same as in 1.18.11.decision_model.decision_protocol, anddecision_model.protocolkeeps the route values that the hosted worker writes. Request metrics and execution logs carry a boundedprotocoldimension. Cache and reasoning counters stay underdecision_model.usage_details, only with the token count they describe part of, and split conversations sum them over the turns that report that count.DecisionEvaluationStrategy.evaluatecall recordsazure.ai.evaluation.decision.evaluationsandazure.ai.evaluation.decision.evaluation.durationwithprotocol,metric,outcome(completed,skipped,errororcancelled),blameanderror_kind.error_kindis any bounded kind of the failure, including the five that 1.18.11 names.metricis the result key of an SDK or Microsoft Foundry catalog Decision evaluator, orcustom, so customer evaluator names are never metric dimensions. A conversation scored turn by turn counts each turn; composite evaluators that interpret one shared request are covered by the request metrics. A metric failure never changes the result or the error.This branch is merged with main after the 1.18.11 release, so the entries are under 1.18.12 (Unreleased) and the version is 1.18.12. Decision support in the built-in evaluators (#49781) and the named failure reasons (#49804) are already on main. The vienna RAISvc worker on master selects only the Microsoft protocol, already accepts these metric attributes, and reads
usage_details.Validation at 4ea0574:
tests/unittestsCoherenceEvaluatordecision_protocolmicrosoft; error text unchangedLive OpenAI Decisions runs (gpt-6-luna) were done on an earlier head; the OpenAI changes since then are the hint wording, stricter answer validation and the provenance field name, which the unit tests cover. No package release is performed by this PR. API contract.