[donei003]

Summary

This change implements the handler routing rules of the shared spec, TESTING.md §8, in launchdarkly/ai-sdks-monorepo#52. It lets one evaluation run use different providers or modes for generation and for judges. For example, generation uses OpenAI and a judge uses Anthropic, or generation uses an agent handler and a judge uses a messages handler.

This is a breaking change to the experimental evaluations API.

handler or handlers

• run() takes one handler as handler or a list as handlers, not both. judge_handlers is removed, with no alias.
• Every handler must declare provides_for. Before this change, a handler without it received every judge config with no warning.
• Two handlers with the same provider and mode raise an error.

One selection rule

• Generation and every judge select a handler with utils.select_handler, the same rule as config(). The evals code has no copy of it.
• There is no mode fallback. A messages-mode judge never runs on an agent handler, so collapse_messages is removed from the evals path. The online judges.py path does not change.

generation.mode

• mode is required: "completion" or "agent". There is no default.
• Other values, such as "messages", "judge" or a typo, raise before any network I/O. They do not go through normalize_mode.
• GenerationConfig stays a TypedDict with Required[...], so callers pass a plain dict and import no type. The new GenerationOverrides TypedDict is the type for overrides with ai_config. Two @overload signatures on run() select between them.
• The GenerationConfig docstring and the README tell the user how to choose a mode.
• The mode is not sent to LaunchDarkly.

Prompt must match the handler mode

• A messages handler with instructions raises: "the generation prompt needs messages".
• An agent handler with messages raises: "the generation prompt needs instructions".
• The same check applies to each judge. All judge problems are reported in one error, before any record is created.

Judges get no tools

A judge call gets an empty tool map, and tools is removed from the judge config.

ai_config

• The run first reads GET ai-configs/<key> to get the mode. The mode is on the AI Config, not on the variation.
• Then it reads the variation, and then the model config.
• Each check runs right after the read that supplies its input:
  • mode coverage: after 1 GET
  • prompt: after 2 GETs
  • provider: after the model config read
• The mode selects the variation's prompt field: instructions in agent mode, messages in completion and judge mode.
• When the mode's field is empty, the other field is kept, so that the prompt check rejects it.
• Overrides cannot set mode.

Migration

# Before
await evals.run(..., handler=openai_handler, judge_handlers=[claude_handler],
                generation={"provider": "OpenAI", "model": "gpt-4o"})

# After
await evals.run(..., handlers=[openai_handler, claude_handler],
                generation={"provider": "OpenAI", "model": "gpt-4o", "mode": "completion"})

To evaluate your own code, wrap it: create_handler(("OpenAI", "messages"), my_app).

Tests

• The existing tests now use tagged handlers and set mode. The judge fixtures now use messages, which is what judge configs carry.
• New tests cover:
  • handler argument validation
  • mode validation
  • prompt and mode mismatches
  • selection by provider and mode, and exact provider over wildcard
  • no mode fallback for judges
  • one error for all judge problems
  • judges with no tools
  • the request count at each ai_config failure
  • prompt field selection by mode
• make test after the merge with main: 2583 passed, 14 skipped. make lint, make format-check and make typecheck are clean.

Review fixes

• utils.select_handler now compares a handler's provider and its mode after normalize_mode. Before this change, a handler tagged ("OpenAI", "completion") or with a list passed validation but was never selected. The fix is in the shared function, so the online config() path is fixed too. Tests for both paths fail without the fix.
• GenerationMode and GenerationOverrides are exported from launchdarkly_ai_server.experimental.evaluations, which is the public path since feat(evaluations)!: mark evaluations from code as experimental #157. They are also in its surface test.
• The branch is merged with main. The packages/ai/README.md conflict takes the side of main.

Out of scope

• The spec's tools map shape and its GET ai-tools step at run time. Python uses list[EvalTool] and tools.get(), which was an earlier divergence.

BEGIN_COMMIT_OVERRIDE
feat(evaluations)!: route handlers by provider and mode
fix(client): match a handler's provides_for after normalizing its mode
END_COMMIT_OVERRIDE

🤖 Generated with Claude Code

---

Note

Overview
Breaking change to the experimental offline evaluations API: handler routing now follows the same provider/mode rules as online config(), with stricter validation up front.

run() accepts either handler or handlers (not both); judge_handlers is removed. Every handler must declare provides_for, and duplicates for the same provider+mode are rejected. Generation and each judge pick a handler via shared select_handler / handler_key (with completion and messages treated as one mode). There is no mode fallback in evals—a messages judge never runs on an agent handler, and judge message collapsing is removed on this path.

generation.mode is required ("completion" or "agent", no default). Prompt shape must match the selected handler (messages vs instructions), for generation and judges; judge setup failures are aggregated into one error before any run records are created. Judges run with no tools (empty tool map, tools stripped from judge config).

ai_config runs read AI Config mode first, then variation, then linked model config, validating handler coverage, prompt, and provider after each step. Overrides use new GenerationOverrides (no mode override). Types GenerationMode and GenerationOverrides are exported from the experimental evaluations module. Docs and tests are updated throughout; online judges.py gains normalized provides_for matching but keeps agent fallback for messages judges.

^{Reviewed by Cursor Bugbot for commit be009fe. Bugbot is set up for automated code reviews on this repo. Configure here.}

[jeffdupont]

Reviewed at 26eafe03, against spec §8.4 as merged in monorepo #52. The routing rules match: one select_handler rule for generation and judges, no mode fallback, generation.mode checked before normalize_mode (module.py:852-856, with 'judge' and 'agnet' tested), prompt/mode checks, and judges with no tools. make test: 1565 passed, 11 skipped. This also closes the judge_handlers divergence that monorepo #56 had deleted the note for.

1. A handler tagged "completion", or with a list, is accepted and then never selected (inline at runner.py:117; same finding as the open Devin and Bugbot threads).
   • _validate_handlers normalizes the tag, so ("OpenAI", "completion") counts as ("OpenAI", "messages"). test_evaluations_run.py:4300 even relies on this to reject a duplicate.
   • But utils.select_handler compares the raw tag: h.provides_for == (provider, mode) (utils.py:516), and [1] == mode for wildcards (:524).
   • I ran it. create_handler(("OpenAI", "completion"), fn) and ["OpenAI", "messages"] both pass validation, but select_eval_handler returns None. The error then lists the handler as ('OpenAI', 'messages'), the very pair it needed.
   • This PR tells users to write mode: "completion", so that tag is the natural one to reach for. Online config() misses it the same way, because it shares select_handler.
   • Fix it once in select_handler (utils.py, not in this diff): compare against (h.provides_for[0], normalize_mode(h.provides_for[1])), and use the same normalized value in the wildcard branch. That's about 3 lines, and it fixes config() too. A test with a lone ("OpenAI", "completion") handler would cover it.
2. The two new types need to reach #157's public path (inline at evaluations/__init__.py:36). I trial-merged this onto #157. The only conflict is packages/ai/README.md: take #157's side, since it removes that example. The merged suite passes (2418 passed). But GenerationMode and GenerationOverrides end up only in launchdarkly_ai_server.evaluations, which #157 makes internal. They aren't in launchdarkly_ai_server.experimental.evaluations. Whichever of the two lands second should add both names to experimental/evaluations.py __all__ and to #157's public-surface test. I'll add + GenerationMode and + GenerationOverrides to the API list in monorepo #51.

Not checked: the ai_config GET order (I went by the tests, I didn't probe it).

[jeffdupont]

Matches TESTING.md §8 from monorepo #52, and all bot threads are addressed. Please close #118 and #120 once this merges.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit be009fe. Configure here.}

[cursor]

Judge prompt check rejects dual fields

High Severity

_resolve_judges runs prompt_mode_error on the raw flag-delivery judge config. Generation first keeps only the mode's prompt field via from_api, but judges skip that step, so a config that has both instructions and messages fails the check even when the mode-appropriate field is present and the run never starts.

Additional Locations (2)

• packages/client/src/launchdarkly_ai_server/evaluations/runner.py#L132-L142
• packages/client/src/launchdarkly_ai_server/evaluations/types.py#L197-L206

 

^{Reviewed by Cursor Bugbot for commit be009fe. Configure here.}