Repository navigation
Support Decision models in the built-in evaluators - #49781
Conversation
Fifteen built-in evaluators accept a Decision model configuration (type="decision") and ask the same rubrics as their Microsoft Foundry catalog versions. Chat model behavior is unchanged. - Each evaluator packages its catalog rubric as a .decision.yaml file beside its prompt and builds a Decision strategy only for a Decision model. PromptyEvaluatorBase sends validated input to a _do_decision_eval hook before any Prompty call, and the Decision state holds exactly the inputs the evaluator's prompt renders. - The binary evaluators accept a probability_threshold keyword that sets the Decision cutoff (default 0.75) and overrides model_config; chat models ignore it with a warning. - Groundedness selects its rubric for each call and reports a plain text response without context as not applicable rather than grounded in itself. - Conversations are still evaluated turn by turn. - QAEvaluator passes credential and the Decision execution controls to its model-based evaluators for a Decision model. - Offline tests cover the request each evaluator sends, its results, thresholds, conversations and rubric packaging, and 64 golden cases pin the chat path.
The test expected one token request per QA member. With Decision token reuse the members share one token, so the test now checks that every member's request carries the forwarded credential's token and that every token request uses the Decision scope.
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
github-actions
Bot
added
the
Evaluation
|
Azure Pipelines: Successfully started running 1 pipeline(s). 10 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
NLTK 3.10.4 refuses, on POSIX, data that another account can write to. The Linux CI agents create the prepared test data group-writable, so NLTK refused it and test_tokenization_uses_prepared_data_with_no_proxy failed when the tokenizer fell back to the downloader. The fixture now removes group and other write access from the prepared tree after verifying it, and a POSIX test checks the tree stays private and loadable under a permissive umask. Windows is unaffected.
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
NLTK 3.10.4 reads data only under nltk.data.path and a temp directory that is private to the user. The macOS temp directory is private, so the permission test passed there, but Linux's shared /tmp is not, so finding the prepared data with an explicit paths argument was refused on Linux. The test now puts the data on nltk.data.path, as the fixture does, and finds the directory the tokenizer reads. Verified on Ubuntu 24.04 with the real conftest under umask 002: the module passes on NLTK 3.10.3 and 3.10.4, the previous version of the test fails with the CI error, and removing the permission fix fails both this test and the tokenization test.
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
…tion With a Decision model, TaskAdherenceEvaluator reported the base default threshold of 3 instead of the requested threshold, because it does not pass threshold to its base, and _TaskCompletionEvaluator reported 3 where its catalog version reports 1. Scores and pass or fail results were unaffected. Decision configurations now pass the requested threshold (default 0) for task adherence and a default of 1 for task completion, as the catalog evaluators do. Chat results are unchanged, as the chat golden tests confirm. The Decision test cases now expect the catalog thresholds, and a new test checks that a requested threshold is reported.
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
The autouse no_network fixture in test_aoai_default_mapping.py replaced socket.socket.connect for every socket family with pytest.fail. promptflow's e2e tests leave a multiprocessing manager that only a full garbage collection cleans up. When that collection ran during this module, the manager's shutdown could not connect to its server process over a local socket, the server kept running, and pytest hung at interpreter exit after every test had passed. That intermittently hung the Python 3.10 CI jobs, including on other PRs. The fixture now refuses only IPv4 and IPv6 connections and lets local sockets through. Reproduced on Python 3.10 with CI's test arguments and two concurrent suites: 2 of 9 runs hung without the change and 0 of 5 with it. With a collection forced inside the module, 4 of 4 hung without it and 0 of 4 with it. A hidden connection to 127.0.0.1 or ::1 from the mapping code still fails the module's tests.
There was a problem hiding this comment.
Copilot encountered an error: Your billing is not configured or you have Copilot licenses from multiple standalone organizations or enterprises. To use premium requests, select a billing entity via the GitHub site, under Settings > Copilot > Features.
Description
Customers who run the built-in evaluators with the SDK can now use a Decision model as the judge. Fifteen built-in evaluators accept a Decision model configuration (
type="decision",azure_endpoint,azure_deployment) and ask the same rubrics as their Microsoft Foundry catalog versions. Chat model behavior is unchanged.Evaluators: coherence, fluency, groundedness, intent resolution, relevance, response completeness, retrieval, similarity, tool call accuracy, task adherence, task completion, tool call success, tool input accuracy, tool output utilization and tool selection.
QAEvaluatoruses Decision through its model-based evaluators..decision.yamlfile beside its prompt, byte-identical to the catalog, and builds a Decision strategy only for a Decision model.PromptyEvaluatorBasesends validated input to a_do_decision_evalhook before any Prompty call, and the Decision state holds exactly the inputs the evaluator's prompt renders.probability_thresholdkeyword (default 0.75) that overridesmodel_config. Chat models ignore it with a warning and never raise.TaskAdherenceEvaluatorreports the requestedthreshold(default 0) and_TaskCompletionEvaluatora threshold of 1, as their catalog versions do. Chat results report the thresholds they did before.QAEvaluatorpassescredentialand the Decision execution controls to its model-based evaluators. The credential must be synchronous because the evaluators run on separate threads.Relation to #49695: the two PRs change different code and should ship together in 1.18.11. With #49695, the built-in evaluators also accept
decision_protocol="openai"for Azure OpenAI Decisions deployments such as GPT-6 Luna, record the evaluation metrics and summedusage_details, and reuse Decision and chat tokens; its sample uses the publicCoherenceEvaluatorthat this PR enables. Both add a 1.18.11 CHANGELOG section, so whichever merges second keeps both lists.CI: the Linux test jobs were failing on
test_nltk_data_setup.pyfor every evaluation PR. NLTK 3.10.4 refuses data that another account can write to, and the agents create the prepared test data group-writable. The test fixture now makes that data private to its owner; #49695 carries the same change. Reproduced and verified on Ubuntu 24.04 with NLTK 3.10.4. The Python 3.10 test jobs could also hang after every test had passed: promptflow's e2e tests leave a multiprocessing manager, and when a garbage collection ran duringtest_aoai_default_mapping.py, its network guard blocked the local socket the manager's shutdown needs. The guard now blocks only IPv4 and IPv6 connections. #49695 carries the same change. Reproduced on Python 3.10: 2 of 9 runs hung before the change and 0 of 5 after.Validation:
evaluate()run of every evaluator andQAEvaluatorwith an Entra credential completed on a Microsoft Decision judge and a chat judge, and the OpenTelemetry evaluation counts matched every call.decision_protocol="openai"against a GPT-6 Luna Decisions deployment.is_reasoning_model=True, as before.evaluate()with every built-in evaluator andQAEvaluatorpassed on the Microsoft Decision judge and on gpt-4.1, gpt-4o, gpt-5-mini and o3-mini. Chat results match 1.18.10 key for key, and the catalog evaluator classes send the same Decision requests and get the same scores as the built-ins.probability_thresholdincident cases on Decision and chat judges.All SDK Contribution checklist:
General Guidelines and Best Practices
Testing Guidelines