Skip to content

Python: Respect FOUNDRY_MODEL in FoundryEvals - #9981

Merged
Eduard van Valkenburg (eavanvalkenburg) merged 1 commit into
microsoft:mainfrom
ZedingZhang:fix/foundry-evals-model-precedence-9951
Nov 4, 2026
Merged

Eduard van Valkenburg (eavanvalkenburg) merged 1 commit into
microsoft:mainfrom
ZedingZhang:fix/foundry-evals-model-precedence-9951

Conversation

@ZedingZhang

@ZedingZhang ZedingZhang (ZedingZhang) commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Motivation & Context

FoundryEvals() passes "gpt-4o" as an explicit model when creating its own FoundryChatClient. This silently overrides FOUNDRY_MODEL, so evaluations use the default deployment even when another deployment is configured.

Description & Review Guide

  • What are the major changes? Resolve the model through the existing load_settings and FoundrySettings before applying the "gpt-4o" fallback. Document the precedence and add seven regression cases covering explicit values, environment values, unset/empty environment values, and both injected-client paths.
  • What is the impact of these changes? Auto-created clients now use an explicit model, then FOUNDRY_MODEL, then "gpt-4o". Model resolution for injected clients keeps its existing behavior.
  • What do you want reviewers to focus on? The model fallback is confined to the auto-created-client branch. The environment-only regression fails on the original implementation and passes with the fix.

Related Issue

Fixes #9951. The issue has the ready-for-implementation label; no competing open PR was found before submission.

AI Assistance

  • No material AI assistance was used.
  • This is an AI-assisted contribution.

AI assistance details: Codex assisted with issue investigation, implementation, regression tests, the docstring update, and this PR description. Codex executed the reported checks; I reviewed all submitted content and take responsibility for it.

Contribution Checklist

  • The code builds clean without any errors or warnings — Foundry sdist and wheel built successfully.
  • All unit tests pass, and I have added new tests where possible — Foundry unit suite: 473 passed, 29 integration tests deselected; seven new regression cases. Ruff formatting/lint, strict source Pyright, and all five test type checkers also pass. No live Foundry integration tests were run.
  • The PR follows the Contribution Guidelines
  • This PR links to an agreed issue with no competing open PR, or the Related Issue section documents a trivial-change or repository-automation exception.
  • This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are recorded for the change that makes auto-created FoundryEvals clients honor FOUNDRY_MODEL.

No open issues 🎉

What changed in this PR

Fixes #9951 by making auto-created FoundryEvals clients respect FOUNDRY_MODEL instead of overriding it with the default deployment.

Changes:

  • Resolves the model in order: explicit argument, FOUNDRY_MODEL, then "gpt-4o".
  • Documents the fallback behavior while preserving injected-client behavior.
  • Adds seven regression cases covering model precedence, unset/empty environment values, and injected clients.
File Description
python/​packages/​foundry/​tests/​test_foundry_evals.py Adds model-precedence and injected-client regression tests.
python/​packages/​foundry/​agent_framework_foundry/​_foundry_evals.py Resolves settings before applying the default model and documents the behavior.

Review effort: Balanced

Merged via the queue into microsoft:main with commit cd45c58 Nov 4, 2026
56 of 59 checks passed
@ZedingZhang
ZedingZhang (ZedingZhang) deleted the fix/foundry-evals-model-precedence-9951 branch November 4, 2026 09:29

This branch was previously deployed

1 inactive deployment
github-app-auth — 6fab3356 Deployed Nov 2, 2026 by ZedingZhang via add_label #26861
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python: [Bug]: FoundryEvals default model overrides FOUNDRY_MODEL

3 participants