[bonanza89]

Fix for azure-ai-evaluation 1.18.12. 1.18.11 shipped without it. The token bug affects chat evaluators in 1.18.10 and 1.18.11, and the Decision runner added in 1.18.11.

Token reuse was part of #49804, and before that #49695. Neither carried it, so it lives only in this PR. #49804 shipped the Tool Call Success fixes and the Decision failure reasons in 1.18.11, and #49695 keeps the OpenAI Decisions protocol.

Problem

Each Decision request, and each Azure OpenAI chat request that authenticates with Microsoft Entra ID, asked the credential for a new token. With AzureCliCredential, which starts a process per request, concurrent evaluations were slow and could fail. On main, 16 concurrent chat rows made 16 token requests (about 39 s of token time), and one row failed with a ClientAuthenticationError from az.

Fix

• New _legacy/_common/_token_cache.py keeps each credential's token per scope until five minutes before it expires, as azure-core's bearer token policy does.
  • Credentials are keyed by identity through a weak reference, so the cache never keeps one alive.
  • Concurrent requests that share a synchronous credential wait for one token request. That request runs on a worker thread, so event loops are not blocked, and abandoned or past-deadline waiters stop within 50 ms.
  • A token the cache does not keep, because of its expiry or because it was issued during a rejection, is handed to the requests queued behind it instead of each requesting another.
• A 401 drops the token. An acquisition that began before the rejection is not stored, and the cache decision and hand-off happen under one lock, so a rejection cannot land between them.
• Chat (_prompty.py): _ReusedTokenProvider replaces the per-request wrapper. The default credential that a chat evaluator creates when no credential is passed is resolved to its inner synchronous credential. AsyncAzureTokenProvider.get_credential() keeps the first credential created, so concurrent first rows share one credential and one token request, and a credential created while the provider closes is closed.
• Decision (_model_runners.py): _get_token(deadline) uses the cache. A credential that returns no string token is the user error "Could not acquire a token for the decision model." rather than a timeout. Decision requests keep the https://cognitiveservices.azure.com/.default scope and their own cached token. Decision endpoints also accept the chat scope, but the hosted evaluation service picks the identity for Decision calls by token audience, so the chat scope would make single-tenant projects call Decision as the end user instead of the project managed identity.
• Requests that waited on a token request share its failure only when it failed authentication. After any other failure, such as a token service outage, each waiting request makes its own attempt, so a transient failure fails only the request that hit it. Decision and chat behave the same way. A request replays only a failure that a request of its own kind described, so a chat request never raises a Decision failure or the reverse, and a successful token request ends the sharing.
• A credential whose get_token is a coroutine function, or that implements AsyncTokenCredential but not the synchronous context manager protocol, is asked for its token on the caller's event loop; any other credential on a worker thread, so a plain get_token that needs the running loop must belong to an AsyncTokenCredential.

No service change is needed. The hosted evaluation worker does not depend on this cache, and Decision keeps the scope the worker relies on.

Validation

Live runs use gpt-4o-mini and Microsoft-Decision-1 deployments, AzureCliCredential, and _ToolCallSuccessEvaluator rows with a tool call.

Check	main (f39c84b)	This PR
tests/unittests	2338 passed, 15 skipped	2421 passed, 15 skipped
Live token requests, 2 waves of 8 concurrent chat rows	17	1, all 16 scored
Same, Microsoft-Decision-1	16	1, all 16 scored
Same, default chat credential (requests that reach its AzureCliCredential)	16 to 18 in three runs	1 in three runs, all scored
evaluate() over 8 rows, chat and Decision-1 sharing one credential	10 chat and 8 Decision	1 per scope, all scored
A first token that the service rejects with 401, then 2 more sequential requests, chat and Decision-1	3 token requests each	2 each; the first request fails with the 401 and the next two pass
4 chat and 4 Decision-1 calls sharing one credential	8	1 per scope (https://ai.azure.com/.default for chat, https://cognitiveservices.azure.com/.default for Decision), all 8 scored
8 concurrent Decision-1 calls sharing one credential	8	1, all scored
4 concurrent calls with a credential that fails authentication, Decision-1 then chat	4 token requests each	1 each; every Decision call raises the user error "Could not acquire a token for the decision model." and every chat call ClientAuthenticationError
Vienna evaluation worker Decision qualification gate	Not run	1155 passed on Vienna 2640626; the same 2 guard tests fail as on any build since #49781

Main merged in (04b36ed), after the 1.18.11 release:

Check	Result
tests/unittests	2569 passed, 31 skipped
black 24.4.0 at line length 120, package and both test files	Clean
Live evaluate() over 8 rows with a Microsoft-Decision-1 judge and AzureCliCredential, 1.18.11 against this PR	8 token requests against 1; all 8 rows scored on both

Merging

Main is merged in (04b36ed). #49804 and this change both added helpers after _is_finite_number in _model_runners.py and imports in test_decision_model_support.py; the merge keeps both. The CHANGELOG entry moved from the released 1.18.11 section to 1.18.12 (Unreleased), and _version.py is 1.18.12, as in #49821. Whichever of the two merges second needs a CHANGELOG merge that keeps both entries in the one 1.18.12 section.

The earlier docs build failure was Sphinx failing to download the requests intersphinx inventory (an SSL error from requests.kennethreitz.org), not this change.

Not in this PR

• The Tool Call Success fixes and the Decision failure reasons (Fix Tool Call Success scoring, binary_aggregate and Decision input order, and name Decision failure reasons #49804, shipped in 1.18.11).
• The OpenAI Decisions protocol (Add Azure OpenAI Decisions protocol and evaluation metrics to evaluation runners #49695).

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
10 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🟡 Changes recommended

The new threading.Lock prevents provider pickling and deepcopy, also breaking serialization of enclosing chat flows; it must be excluded from serialized state and recreated on restore.

1 open issue

• Exclude the provider lock from serialized state · New

What changed in this PR

This PR adds credential-token reuse to azure-ai-evaluation chat and Decision requests, reducing repeated authentication calls while keeping their token scopes separate.

Changes:

• Adds a shared token cache with expiry handling, concurrent synchronous acquisition, and HTTP 401 invalidation.
• Integrates caching into both model paths and coordinates default chat credential creation.
• Adds regression coverage and documents the fix for 1.18.11.

File	Description
sdk/​evaluation/​azure-ai-evaluation/​tests/​unittests/​test_prompty_token_reuse.py	Tests chat caching, concurrency, authentication failures, and credential lifecycle.
sdk/​evaluation/​azure-ai-evaluation/​tests/​unittests/​test_decision_model_support.py	Tests Decision caching, scope isolation, deadlines, cancellation, and serialization.
sdk/​evaluation/​azure-ai-evaluation/​CHANGELOG.md	Documents token reuse and credential behavior.
sdk/​evaluation/​azure-ai-evaluation/​azure/​ai/​evaluation/​_legacy/​prompty/​_prompty.py	Reuses chat tokens and invalidates rejected tokens.
sdk/​evaluation/​azure-ai-evaluation/​azure/​ai/​evaluation/​_legacy/​_common/​_token_cache.py	Implements shared, identity-keyed token caching and acquisition coordination.
sdk/​evaluation/​azure-ai-evaluation/​azure/​ai/​evaluation/​_legacy/​_common/​_async_token_provider.py	Coordinates default credential creation and cleanup.
sdk/​evaluation/​azure-ai-evaluation/​azure/​ai/​evaluation/​_evaluators/​_common/​_model_runners.py	Integrates Decision token reuse, deadline handling, and rejection invalidation.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The new threading lock makes the default token provider unpicklable, breaking serialization and deepcopy of chat evaluators, including API-key configurations and batch workflows.

1 open issue

• Exclude the provider lock from serialized state

_{Review effort: Balanced}