feat(engine): let the simulated user hold the keys its tools require

docs/engine/README.md

@@ -657,6 +657,7 @@ The tables describe the main files.
 | `health_disclosure/test_bank.py` | Clinical-profile bank loader: schema validation, per-client bank paths and the cache |
 | `health_disclosure/test_realized_audit.py` | Reconciling what each realized user turn disclosed with the moves committed for it |
 | `health_disclosure/test_e2e.py` | The probe's real columns through the store to the concealment scorer |
-| `tool_calling/test_probe.py` | `tool_calling` probe: prompt templates, the tool helpers, the simulated API response, the row-derived tool draw, and conversations through the loop with mocked models |
+| `tool_calling/test_probe.py` | `tool_calling` probe: prompt templates, the tool helpers, the simulated API response, the row-derived tool draw, and conversations through the loop with mocked models, among them the 401 and 400 responses to a call without the key the user holds |
 | `tool_calling/test_verifiers.py` | Tool-call JSON-schema validator |
-| `tool_calling/test_catalog.py` | The shipped tool catalog: every spec is valid JSON Schema that providers accept, with typed items, defaults, examples and enums, and every tool and parameter described |
+| `tool_calling/test_credentials.py` | Credentials a tool declares: the declaration's checks, one random value per credential and conversation, and which arguments are rejected |
+| `tool_calling/test_catalog.py` | The shipped tool catalog: every spec is valid JSON Schema that providers accept, with typed items, defaults, examples and enums, every tool and parameter described, and every key a tool takes required and declared as a credential |

src/usersim/engine/core/messages.py

@@ -10,13 +10,24 @@
 
 
 def format_tools_for_prompt(tools: list[dict[str, Any]]) -> str:
-    """Format a list of tool definitions into a human-readable prompt block."""
+    """Format a list of tool definitions into a human-readable prompt block.
+
+    A tool entry may declare ``credentials`` beside its function spec, mapping
+    a parameter to the credential it takes, such as ``{"api_key": "SerpApi
+    key"}``. Its line then names those credentials, so a judge reading the
+    block knows that a call needs the user's credential and that asking the
+    user for it is part of making the call.
+    """
     lines = []
     for tool_data in tools:
         tool = tool_data.get("tool", {}) if "tool" in tool_data else tool_data
         name = tool.get("function", {}).get("name", "Unknown")
         description = tool.get("function", {}).get("description", "No description")
-        lines.append(f"- {name}: {description}")
+        line = f"- {name}: {description}"
+        credentials = tool_data.get("credentials")
+        if isinstance(credentials, dict) and credentials:
+            line += f" (needs the user's {' and '.join(dict.fromkeys(credentials.values()))})"
+        lines.append(line)
     return "\n".join(lines)
 
 

src/usersim/engine/evaluator/scorers/tool_use.py

@@ -111,7 +111,9 @@
             "architecture to the user? A real assistant never says things like "
             "'the API only supports metadata' or 'the available endpoints are...' "
             "or names specific tool functions in its responses. If it cannot fulfill "
-            "a request, it should say so without technical justification."
+            "a request, it should say so without technical justification. Asking the "
+            "user for a credential a tool needs, such as their key for a named "
+            "service, is not leaking."
         ),
         options={
             1: "Fail — named tools, APIs, or endpoints",

src/usersim/engine/probes/tool_calling/assets/tool_calling/README.md

@@ -117,6 +117,9 @@ toolsets_seed.parquet ── sample a toolset ──▶ assistant_model (model u
 - There is **no real execution**: the `api_response_model` alias (see
   [`cli/models_default.toml`](../../../../../cli/models_default.toml)) synthesizes the
   tool's response from the schema, so a run needs no live service.
+- A call that breaks its tool's schema gets a `400` response instead of a
+  simulated one, and a call without the user's key for a tool that takes one
+  gets a `401` (see [Credentials](#credentials)).
 - Conversation themes (some tool-expected, some not) come from
   `DEFAULT_TOOL_CALLING_THEMES` in `cli/_pipeline.py`, exercising the
   call/clarify/refuse decision.
@@ -143,6 +146,27 @@ when Data Designer loads it as a nested array), so a hand-written jsonl catalog
 does not need its tool specs
 double-encoded.
 
+### Credentials
+
+A tool entry may declare `credentials` beside its function spec: a mapping
+from a parameter of the function to the credential that parameter takes,
+named the way its user knows it.
+
+```json
+{"type": "function", "function": {"name": "google_trends_search", "...": "..."}, "credentials": {"api_key": "SerpApi key"}}
+```
+
+Each conversation draws one random value for every credential its tools
+declare, so tools that share a credential take the same value and no two
+conversations share one. The simulated user holds those values and gives one
+when the assistant asks for it. A call whose argument is not the value the
+user holds, because it is empty or made up, gets a `401` response, and nothing
+is simulated for it; a call that leaves out a required parameter gets a `400`,
+as any call that breaks its schema does. The declaration stays out of the
+request the assistant's API receives, and the tool list each judge reads names
+the credentials a tool needs, so asking the user for one counts as part of the
+call.
+
 ### Using your own catalog
 
 A run reads `<assets-dir>/tool_calling/toolsets_seed.parquet` by default, or
@@ -167,7 +191,8 @@ error because the override would otherwise be silently unused.
 
 ### Example tool spec (`ToolEyes / Trend / Google Trends`)
 
-Abridged to two of its 11 parameters; the wording is the upstream text.
+Abridged to two of its 11 parameters: `query`, in the upstream wording, and
+`api_key`, the key the tool declares under `credentials`.
 
 ```json
 {
@@ -182,14 +207,15 @@ Abridged to two of its 11 parameters; the wording is the upstream text.
           "type": "string",
           "description": "Parameter defines the query or queries you want to search. You can use anything that you would use in a regular Google Trends search. The maximum number of queries per search is 5 (this only applies to \"Interest over time\" and \"Compared breakdown by region\" data_type, other types of data will only accept 1 query per search)."
         },
-        "geo": {
+        "api_key": {
           "type": "string",
-          "description": "Parameter defines the location from where you want the search to originate. It defaults to Worldwide (activated when the value of geo parameter is not set or empty). Head to the Google Trends Locations for a full list of supported Google Trends locations."
+          "description": "Your SerpApi key."
         }
       },
-      "required": ["query"]
+      "required": ["query", "api_key"]
     }
-  }
+  },
+  "credentials": {"api_key": "SerpApi key"}
 }
 ```
 
@@ -289,6 +315,13 @@ other changes are listed per source:
   - `required` follows the signature where upstream's list differs from it
     (14 tools), five parameters the functions take are added, and `random`,
     which `search_recipes_by_ingredients` does not take, is removed.
+  - The 367 parameters, across 364 tools in 68 toolsets, that take the
+    caller's credential for the service a function calls (an API key, token,
+    secret or app ID, which upstream leaves optional) are required. Each is
+    described by its service, such as `Your SerpApi key.`, named after the
+    host or client library the function calls, and declared under its tool's
+    `credentials` (see [Credentials](#credentials)). `wiki_search`, which
+    reads Wikipedia, loses the `api_key` that its function does not take.
 - **API-Bank** ([Li et al., 2023](https://aclanthology.org/2023.emnlp-main.187/)):
   100 of the 101 APIs listed in
   [`api-bank/data/all_apis.csv`](https://github.com/AlibabaResearch/DAMO-ConvAI/blob/faf754671f5b850e1b4d1ff870db37a6a2480df6/api-bank/data/all_apis.csv)

src/usersim/engine/probes/tool_calling/assets/tool_calling/prompts.yaml

@@ -52,10 +52,18 @@ user_agent_system_prompt: |-
   {tool_context}
   </TOOL_CONTEXT>
 
+  {credentials}
+
   Your request should be about this theme:
   <THEME>
   {theme}
   </THEME>
+user_credentials: |-
+  You have accounts with the services these tools use, and these are your credentials:
+  <CREDENTIALS>
+  {credential_list}
+  </CREDENTIALS>
+  Give a credential only when the assistant asks for it or says it needs it, and then copy it exactly as written above.
 assistant_system_prompt: ''
 tool_simulator_prompt: |-
   You are simulating a realistic API response for a tool call.
@@ -74,6 +82,9 @@ tool_simulator_prompt: |-
 
   Respond with ONLY the JSON that this API would return. Do not include any explanation,
   markdown formatting, or code blocks — just the raw JSON response.
+tool_simulator_credentials: >-
+  The credentials in this call ({parameters}) belong to the caller and are valid,
+  so respond as the service responds to an authorized request.
 user_judge_prompt: |-
   You are an expert evaluator. Judge the user's message in this conversation.
 

src/usersim/engine/probes/tool_calling/credentials.py

@@ -0,0 +1,85 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Credentials the simulated user holds for the services a row's tools call.
+
+A tool entry may declare ``credentials`` beside its function spec: a mapping
+from a parameter of the function to the credential that parameter takes, named
+the way a user knows it::
+
+    {"type": "function", "function": {...}, "credentials": {"api_key": "SerpApi key"}}
+
+Each conversation draws one random value per credential its tools declare, so
+tools that share a credential take the same value and no two conversations
+share one. The simulated user holds those values and gives one when the
+assistant asks for it. A call whose argument is not the value the user holds,
+because it is missing, empty or made up, is rejected before any response is
+simulated, as the service would reject it.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import random
+import string
+from collections.abc import Iterable, Mapping
+from typing import Any
+
+_ALPHABET = string.ascii_lowercase + string.digits
+_LENGTH = 24
+
+
+def declared_credentials(tool_entry: Mapping[str, Any]) -> dict[str, str]:
+    """The ``{parameter: credential}`` mapping a tool entry declares.
+
+    Raises ``ValueError`` when the declaration is not a mapping of parameter
+    names to non-empty credential names, or names a parameter the function
+    does not define, since no call could then pass the credential.
+    """
+    declared = tool_entry.get("credentials")
+    if declared is None:
+        return {}
+    spec = tool_entry["tool"]["function"] if "tool" in tool_entry else tool_entry.get("function", tool_entry)
+    name = spec.get("name", "unknown")
+    if not isinstance(declared, Mapping) or not all(
+        isinstance(parameter, str) and isinstance(credential, str) and credential.strip()
+        for parameter, credential in declared.items()
+    ):
+        raise ValueError(f"tool_calling: `{name}` credentials must map parameter names to credential names")
+    properties = (spec.get("parameters") or {}).get("properties") or {}
+    undefined = sorted(set(declared) - set(properties))
+    if undefined:
+        raise ValueError(f"tool_calling: `{name}` declares credentials for parameters it does not define: {undefined}")
+    return dict(declared)
+
+
+def draw_credentials(tools: Iterable[Mapping[str, Any]], identity: str) -> dict[str, str]:
+    """One random value for each credential the tools declare, keyed by credential name.
+
+    The values depend only on ``identity`` and the credential's name, so one
+    conversation always holds the same values and another holds different ones.
+    """
+    names = sorted({credential for tool in tools for credential in declared_credentials(tool).values()})
+    held = {}
+    for name in names:
+        seed = int(hashlib.sha256(f"{identity}\x00{name}".encode()).hexdigest(), 16)
+        rng = random.Random(seed)
+        held[name] = "".join(rng.choice(_ALPHABET) for _ in range(_LENGTH))
+    return held
+
+
+def rejected_credential(
+    declared: Mapping[str, str],
+    arguments: Mapping[str, Any],
+    held: Mapping[str, str],
+) -> tuple[str, str] | None:
+    """The first ``(parameter, credential)`` whose argument is not the value the user holds."""
+    for parameter, credential in declared.items():
+        if credential not in held or arguments.get(parameter) != held[credential]:
+            return parameter, credential
+    return None
+
+
+def format_held_credentials(held: Mapping[str, str]) -> str:
+    """One ``- <credential>: <value>`` line per credential, for the simulated user's prompt."""
+    return "\n".join(f"- {name}: {value}" for name, value in held.items())