feat(AIC-3439): let wildcard handlers accept only named providers

AGENTS.md

@@ -374,7 +374,7 @@ Returns a `ConfigInstance` with:
 **Behavior when `.invoke()` is called:**
 
 1. Fetches and validates the `AiConfigRep` variation from LaunchDarkly using `key` and the supplied `context`. Raises if the variation is disabled or invalid.
-2. Selects the handler by matching on `[config.provider.name, normalized mode]`. Selection priority: (a) exact provider match, (b) wildcard `['*', mode]` fallback for multi-provider adapters (e.g. LangChain). Raises if no matching handler is found.
+2. Selects the handler by matching on `[config.provider.name, normalized mode]`. Selection priority: (a) exact provider match, ignoring any `providers` list; (b) a wildcard `['*', mode]` whose `providers` list contains the name, shortest list first and earlier registration on a tie; (c) an unscoped wildcard `['*', mode]`. Raises if no matching handler is found.
 3. Invokes the selected handler with the config, user input, tool handlers, variables, and history. The `context` passed to `.invoke()` is automatically merged into `variables` under the key `ldContext`, so templates can reference `{{ldContext.key}}`, `{{ldContext.email}}`, etc. If `history` is provided, it is passed to the handler as the 5th positional argument — messages-mode handlers splice it into the messages array; agent-mode handlers append it to the system prompt.
 4. Emits LaunchDarkly telemetry events: duration (`$ld:ai:duration:total`), outcome (`$ld:ai:generation:success` / `$ld:ai:generation:error`), and token counts (`$ld:ai:tokens:*`).
 5. If `judgeConfiguration` is present:
@@ -409,7 +409,7 @@ registry = Registry(
 )
 ```
 
-`.register(handlers=[], tools={})` can be called multiple times to add more handlers or tools. Duplicate `provides_for` keys or tool names produce a warning and the last registration wins.
+`.register(handlers=[], tools={})` can be called multiple times to add more handlers or tools. Handler identity is `provides_for` plus the provider set: two wildcards of the same mode with different `providers` lists both stay registered, while the same set (order-independent; repeated names collapse) or two unscoped wildcards warn and the last registration wins. Duplicate tool names do the same.
 
 `global_registry` is a pre-constructed singleton `Registry` instance.
 
@@ -460,7 +460,7 @@ async def handler(
 handler.provides_for = [provider_name: str, mode: Literal["agent", "messages"]]
 ```
 
-The `provides_for` list is how `config()` routes to the correct handler at runtime. The mode element must exactly match the normalized `meta.mode`. The provider element must either exactly match `config.provider.name` **or** be the wildcard `'*'`. A wildcard handler is chosen only when no handler with an exact provider name matches — it acts as a fallback for multi-provider adapters like LangChain. **Always attach `provides_for` using `create_handler` rather than direct attribute assignment.**
+The `provides_for` list is how `config()` routes to the correct handler at runtime. The mode element must exactly match the normalized `meta.mode`. The provider element must either exactly match `config.provider.name` **or** be the wildcard `'*'`. A wildcard handler is chosen only when no handler with an exact provider name matches — it acts as a fallback for multi-provider adapters like LangChain. Pass `providers=["Bedrock"]` to `create_handler` to limit which `config.provider.name` values that wildcard accepts; names must match exactly, and an empty list or a blank name is rejected. Omit `providers` to accept every provider. **Always attach `provides_for` using `create_handler` rather than direct attribute assignment.**
 
 ### Factory Function
 

packages/client/README.md

@@ -184,7 +184,7 @@ Two limits keep a trajectory from spending the judge's context window: at most 5
 
 A tool result is now judge-prompt input. It stays literal for the same reason the generated output does: the judge config is handed to the handler unrendered and the handler makes exactly one template pass, so a `{{...}}` sequence coming back from a tool is never expanded into the judge prompt.
 
-**Judges are independent AI Configs, so handlers are routed per judge.** A judge may resolve to a different provider or mode than `generation`, and a handler built for one provider cannot execute another's config. `handler` runs a judge when it provides for that judge's provider; pass handlers for any other providers in `judge_handlers`. Selection prefers a handler naming the judge's provider outright over a wildcard multi-provider adapter, and an agent-mode handler can serve a messages-mode judge with its messages collapsed into one instructions block. A plain callable that declares no `provides_for` routes itself, exactly as it already does for the generation config.
+**Judges are independent AI Configs, so handlers are routed per judge.** A judge may resolve to a different provider or mode than `generation`, and a handler built for one provider cannot execute another's config. `handler` runs a judge when it provides for that judge's provider; pass handlers for any other providers in `judge_handlers`. Selection prefers a handler naming the judge's provider outright over a wildcard multi-provider adapter, and an agent-mode handler can serve a messages-mode judge with its messages collapsed into one instructions block. A wildcard's `providers` list, when set, accepts only those names. A plain callable that declares no `provides_for` routes itself, exactly as it already does for the generation config.
 
 Judges are resolved through flag delivery, and handlers are matched to them, **before** any evaluation records are created — a missing judge or one no handler covers fails the run up front rather than after the generation spend. After that point a criterion failure never aborts the run: an unparseable judge response, an out-of-range score, a raising handler or scorer, and a row whose generation errored each become a per-criterion `ERROR` event with a cause code (`invalid_judge_output`, `invalid_score`, `handler_raised`, `scorer_raised`, `generation_incomplete`) and a top-level `errorMessage`. Event *delivery* is different: the backend needs one result per `(row, criterion)` to finish row accounting, so if tracking a criterion event fails, every remaining result is still attempted and flushed and then `run()` raises — rather than polling to its timeout with the cause hidden.
 

packages/client/src/launchdarkly_ai_server/evaluations/runner.py

@@ -96,11 +96,19 @@ def _provides_for(
     return None
 
 
-def _covers_provider(
-    provides_for: tuple[str, Literal["agent", "messages"]],
-    provider: str | None,
-) -> bool:
-    return provides_for[0] == provider or provides_for[0] == "*"
+def _covers_provider(handler: EvalHandler, provider: str | None) -> bool:
+    """Exact provider name, or a wildcard whose allowlist contains it."""
+    provides_for = _provides_for(handler)
+    if provides_for is None or provider is None:
+        return False
+    if provides_for[0] == provider:
+        return True
+    if provides_for[0] != "*":
+        return False
+    allow = getattr(handler, "providers", None)
+    if allow is None:
+        return True
+    return provider in tuple(allow)
 
 
 def _find_judge_handler(
@@ -115,19 +123,37 @@ def _find_judge_handler(
     ``config()`` already applies to a generation config. Searching in one pass
     would instead let the order the caller happened to list its handlers in
     decide, sending an OpenAI judge through a LangChain adapter that was merely
-    listed first.
+    listed first. Among wildcards that accept the name, the shortest allowlist
+    wins and an unscoped ``*`` is the last resort.
     """
-    for exact in (True, False):
-        for candidate in judge_handlers:
-            provides_for = _provides_for(candidate)
-            if provides_for is None or provides_for[1] != mode:
-                continue
-            if exact:
-                if provides_for[0] == provider:
-                    return candidate
-            elif provides_for[0] == "*":
-                return candidate
-    return None
+    for candidate in judge_handlers:
+        provides_for = _provides_for(candidate)
+        if (
+            provides_for is not None
+            and provider is not None
+            and provides_for[0] == provider
+            and provides_for[1] == mode
+        ):
+            return candidate
+
+    scoped: list[tuple[int, int, EvalHandler]] = []
+    unscoped: EvalHandler | None = None
+    for index, candidate in enumerate(judge_handlers):
+        provides_for = _provides_for(candidate)
+        if provides_for is None or provides_for[0] != "*" or provides_for[1] != mode:
+            continue
+        allow = getattr(candidate, "providers", None)
+        if allow is None:
+            if unscoped is None:
+                unscoped = candidate
+            continue
+        names = tuple(allow)
+        if provider is not None and provider in names:
+            scoped.append((len(names), index, candidate))
+    if scoped:
+        scoped.sort(key=lambda item: (item[0], item[1]))
+        return scoped[0][2]
+    return unscoped
 
 
 def _select_judge_handler(
@@ -164,7 +190,7 @@ def _select_judge_handler(
     generation_provides_for = _provides_for(handler)
     if generation_provides_for is None:
         return JudgeExecution(resolved=resolved, handler=handler)
-    if _covers_provider(generation_provides_for, resolved.provider):
+    if _covers_provider(handler, resolved.provider):
         return JudgeExecution(
             resolved=resolved,
             handler=handler,

packages/client/src/launchdarkly_ai_server/judges.py

@@ -27,17 +27,20 @@
 )
 from .utils import (
     normalize_mode,
+    select_mode_handler,
     to_ld_context,
     to_usage_dict,
+    wildcard_covers,
 )
 
 
 def _provider_matches(handler: ProviderHandler, provider: str | None) -> bool:
-    """Returns True when the handler covers the given provider or is a wildcard."""
-    return bool(
-        handler.provides_for
-        and (handler.provides_for[0] == provider or handler.provides_for[0] == "*")
-    )
+    """True when the handler names the provider or its wildcard allowlist contains it."""
+    if not handler.provides_for:
+        return False
+    if provider is not None and handler.provides_for[0] == provider:
+        return True
+    return wildcard_covers(handler, provider)
 
 
 logger = logging.getLogger(__name__)
@@ -108,29 +111,11 @@ async def run_judges(
             # config, collapse messages into a single instructions block.
             judge_handler: ProviderHandler = handler
             collapse_messages = False
-            if handlers:
-                exact = next(
-                    (
-                        h
-                        for h in handlers
-                        if _provider_matches(h, judge_provider)
-                        and h.provides_for
-                        and h.provides_for[1] == judge_mode
-                    ),
-                    None,
-                )
+            if handlers is not None:
+                exact = select_mode_handler(handlers, judge_provider, judge_mode)
                 agent_fallback = (
-                    next(
-                        (
-                            h
-                            for h in handlers
-                            if _provider_matches(h, judge_provider)
-                            and h.provides_for
-                            and h.provides_for[1] == "agent"
-                        ),
-                        None,
-                    )
-                    if not exact and judge_mode == "messages"
+                    select_mode_handler(handlers, judge_provider, "agent")
+                    if exact is None and judge_mode == "messages"
                     else None
                 )
                 if exact:
@@ -145,6 +130,15 @@ async def run_judges(
                         and handler.provides_for is not None
                         and handler.provides_for[1] == "agent"
                     )
+                else:
+                    # No compatible handler — skip rather than calling a provider
+                    # whose allowlist does not include this judge.
+                    logger.warning(
+                        "Judge %r skipped: no handler covers provider %r",
+                        judge_key,
+                        judge_provider,
+                    )
+                    continue
 
             effective_judge_config = (
                 _collapse_messages_to_instructions(judge_ai_config)
@@ -291,29 +285,11 @@ async def _build_judge_tasks(
             )
 
             collapse_messages = False
-            if handlers:
-                exact = next(
-                    (
-                        h
-                        for h in handlers
-                        if _provider_matches(h, judge_provider)
-                        and h.provides_for
-                        and h.provides_for[1] == judge_mode
-                    ),
-                    None,
-                )
+            if handlers is not None:
+                exact = select_mode_handler(handlers, judge_provider, judge_mode)
                 agent_fallback = (
-                    next(
-                        (
-                            h
-                            for h in handlers
-                            if _provider_matches(h, judge_provider)
-                            and h.provides_for
-                            and h.provides_for[1] == "agent"
-                        ),
-                        None,
-                    )
-                    if not exact and judge_mode == "messages"
+                    select_mode_handler(handlers, judge_provider, "agent")
+                    if exact is None and judge_mode == "messages"
                     else None
                 )
                 if exact:
@@ -328,6 +304,11 @@ async def _build_judge_tasks(
                     )
                 else:
                     # No compatible handler — skip, same as run_judges.
+                    logger.warning(
+                        "Judge %r skipped: no handler covers provider %r",
+                        judge_key,
+                        judge_provider,
+                    )
                     continue
 
             evaluation_metric_key = (
@@ -379,27 +360,10 @@ async def run_judge(
     report_usage("client.runJudge")
     from .tracking import execute_and_track
 
-    def _matches(h: ProviderHandler) -> bool:
-        return _provider_matches(h, task.judge_provider)
-
-    exact = next(
-        (
-            h
-            for h in handlers
-            if _matches(h) and h.provides_for and h.provides_for[1] == task.judge_mode
-        ),
-        None,
-    )
+    exact = select_mode_handler(handlers, task.judge_provider, task.judge_mode)
     agent_fallback = (
-        next(
-            (
-                h
-                for h in handlers
-                if _matches(h) and h.provides_for and h.provides_for[1] == "agent"
-            ),
-            None,
-        )
-        if task.judge_mode == "messages" and not exact
+        select_mode_handler(handlers, task.judge_provider, "agent")
+        if task.judge_mode == "messages" and exact is None
         else None
     )
 

packages/client/src/launchdarkly_ai_server/registry.py

@@ -9,6 +9,23 @@
 logger = logging.getLogger(__name__)
 
 
+def _registration_key(
+    handler: ProviderHandler,
+) -> tuple[tuple[str, str], tuple[str, ...] | None]:
+    """
+    ``provides_for``, plus the allowlist as a set on a ``"*"`` handler. Order and
+    repeated names do not make a new identity. Routing ignores the list on an
+    exact handler, so it is not part of that handler's identity.
+    """
+    provides_for = handler.provides_for
+    if provides_for is None:
+        raise ValueError("handler is missing provides_for")
+    providers = handler.providers
+    if provides_for[0] != "*" or providers is None:
+        return (provides_for, None)
+    return (provides_for, tuple(sorted(set(providers))))
+
+
 class Registry:
     """
     Manages handlers and tools for ``routed_model`` and ``graph``.
@@ -42,13 +59,15 @@ def register(
         if handlers:
             for handler in handlers:
                 if handler.provides_for is not None:
-                    # Check for duplicate by providesFor key
-                    key = handler.provides_for
+                    # Identity is provides_for plus the provider set. Two scoped
+                    # wildcards of the same mode stay registered when their lists differ.
+                    key = _registration_key(handler)
                     existing_idx = next(
                         (
                             i
                             for i, h in enumerate(self._handlers)
-                            if h.provides_for == key
+                            if h.provides_for is not None
+                            and _registration_key(h) == key
                         ),
                         None,
                     )

packages/client/src/launchdarkly_ai_server/types.py

@@ -1,6 +1,6 @@
 from __future__ import annotations
 
-from collections.abc import AsyncGenerator, Awaitable, Callable
+from collections.abc import AsyncGenerator, Awaitable, Callable, Sequence
 from dataclasses import dataclass, field
 from typing import Any, Generic, Literal, TypeVar
 
@@ -156,6 +156,7 @@ class ProviderHandler:
     - ``__call__`` — blocking invocation
     - ``stream``   — optional async-generator streaming (may be ``None``)
     - ``provides_for`` — ``(provider_name, mode)`` tuple or ``None``
+    - ``providers`` — optional allowlist consulted only when ``provides_for`` names ``"*"``
     - ``capture_content`` — the effective content-capture decision for this handler
 
     ``capture_content`` is declared here so the client core can apply the handler's own content
@@ -166,18 +167,24 @@ class ProviderHandler:
     """
 
     provides_for: tuple[str, Literal["agent", "messages"]] | None
+    providers: tuple[str, ...] | None
 
     def __init__(
         self,
         fn: _HandlerFn,
         provides_for: tuple[str, Literal["agent", "messages"]] | None = None,
         stream_fn: _StreamFn | None = None,
         capture_content: bool | None = None,
+        *,
+        providers: Sequence[str] | None = None,
     ) -> None:
         self._fn = fn
         self.provides_for = provides_for
         self._stream_fn = stream_fn
+        from .utils import normalize_providers
+
         self._capture_content_override = capture_content
+        self.providers = normalize_providers(providers)
 
     @property
     def capture_content(self) -> bool: