[3mei]

What changed and why

A tool_calling tool that takes the caller's key for a service declares it, the simulated user holds a random key for each such service, and only a call carrying that key reaches the simulated service.

Builds on #130.

• Credentials the user holds (first commit). A tool entry may declare credentials beside its function spec, mapping a parameter to the credential it takes, such as {"api_key": "SerpApi key"}. Each conversation draws one random 24-character value per declared credential from its trajectory identity, so tools that share a credential take the same value and no two conversations share one. The simulated user holds the values and gives one when the assistant asks for it. A call that passes the schema check without the value the user holds gets {"error": "Unauthorized: the SerpApi key is missing or invalid.", "status": 401}, and nothing is simulated for it; a call that leaves out a required key gets the 400 any schema error gets. The simulator is told that the credentials of a call it answers are valid. The declaration stays out of the request the assistant's API receives. format_tools_for_prompt names the credentials a tool needs, so the user-turn gate and the tool_use judge know that asking for one is part of the call, and the architecture_leaking rubric says that asking for a credential is not a leak. The probe's prompt version is v1.2.
• Keys required and declared in the catalog (second commit). The 367 parameters, across 364 ToolEyes tools in 68 toolsets, that take the caller's credential for the service a function calls (an API key, token, secret or app ID, which upstream leaves optional) are required, described by their service (Your SerpApi key.) and declared under their tool's credentials. The service is the one whose host or client library the function in the toolset's tool.py calls: RapidAPI for 138 parameters, Spoonacular for 55, SerpApi for 48, Alpha Vantage for 33, Travelpayouts for 19, and 22 other services for the rest. wiki_search, which reads Wikipedia, loses an api_key that its function does not take. Parameters named key that select a lookup field, and API-Bank's token inputs, which its own login tool issues, are unchanged. The catalog README documents credentials and records the change, and the catalog tests require every declared credential to be a required string its description names, and every key parameter to be declared.

How it was verified

At each commit, on that commit's tree:

$ make check-all && make test && make smoke
All checks passed!
374 files already formatted
4701 passed in 83.52s (0:01:23)
All checks passed.

That is the first commit's tree. The second passes with 4704: it adds the catalog tests of the credentials.

At the head:

$ make check-license-headers
🎉 All files have correct license headers!
$ make check-doc-links
✓ every relative link, and every link into this repository's main branch, resolves
$ make check-notebooks-clean
OK 4 notebook(s) carry no outputs
$ make check-dist
✓ installed package runs from a clean environment
$ make check-extensions
all extension seams work against an installed distribution

• The two credential tests of the catalog fail on the catalog before the second commit:

  E       AssertionError: the shipped catalog declares no credentials
  E       AssertionError: key parameters whose credential the user would not hold: ['ToolEyes/Weather/openweathermap:current_weather.apikey', 'ToolEyes/Search/EngineSearch:google_autocomplete_search.api_key', ...]
  

• make health-check ran against a custom provider config for tool_calling in en_US and for the default probe, sov_ai_dynamic in ja_JP. Both ended with ✓ simulate, eval and report all completed, and every model's requests show failed=0.

• usersim simulate --toolset-seed-path with only the weatherapi toolset, every tool of which takes the WeatherAPI.com key, then usersim eval --scorers auto, against the same config: each call carrying a key other than the one the user holds got the 401 and no simulated response, and the call carrying the key the user gave was simulated. Every scorer reported ok:

  verifier: failure  `api_key` is not the WeatherAPI.com API key the user holds.
  verifier: success  current_weather validated
  judge: scored
  language_compliance: ok (error=None)
  refusal_basics: ok (error=None)
  response_shape: ok (error=None)
  tool_use: ok (error=None)
  

Results impact

• On the 68 ToolEyes toolsets whose tools take a key, a call needs the key the user holds: one with a made-up or missing key gets an error response instead of a simulated result. tool_use scores on those rows (argument quality, information gathering, task completion) move accordingly, so they do not compare like for like with earlier runs.
• Every tool_calling trajectory id changes with the prompt version, so resuming an earlier run simulates its tool_calling rows again.