[3mei]

What changed and why

Each tool spec in the catalog describes the call its operation takes: a type for every parameter, an item type for every array, descriptions of the operation and parameters the spec actually has, and the parameters a call needs. The assistant learns each tool's contract from its spec, so a spec that misdescribes an operation teaches a wrong contract and records it in every conversation that samples it.

Builds on #128, which labels the catalog's sources and fixes the specs providers reject.

Every change was checked against the source of truth for its tool, and the README's Sources and licenses section describes each source's changes:

• ToolEyes (first commit). Each spec matches the function of the same name in its toolset's tool.py, the code that calls the API. 669 parameters that upstream types string take the type the function's signature gives them, 80 arrays declare their item type, and 228 descriptions that contradict the function or are empty are written from the endpoint the function calls (all nine CoinMarketCap tools shared one tool's text, Ergast described its ids as names, and 13 search tools had no description). Fourteen required lists follow the signatures, five parameters are added and one the function does not take is removed.
• UltraTool (second commit). UltraTool's tools have no implementation, so each spec is checked against its description, the dev.json plan steps that call it and the results of the tools that feed it. 179 arrays declare their item type, 98 parameters take the number, boolean or array type their plans show, and 103 parameters a call needs, such as the origin and destination of book_best_flight, are added after the matching parameter of a related tool. Keywords and formats are JSON Schema's, every nested field is described, and 72 descriptions are rewritten.
• AutoTools and RestGPT (third commit). An AutoTools parameter is required when it fills the URL or when upstream's in field marks it needed (34 tools), and its description ends with where it is sent. Defaults have their parameter's type, copied parameter descriptions are written from the endpoint, and 13 RestGPT parameters that an operation's description promises are added.
• API-Bank (fourth commit). A tool requires the inputs its signature in all_apis.csv takes without a default (82 tools), eight tools gain the token input their signature takes, and 15 descriptions are corrected against the class's code and the CSV.
• Custom (fifth commit). Five tools keep their optional inputs optional.
• Catalog test (sixth commit). tests/engine/probes/test_tool_catalog.py validates every parameters schema against the JSON Schema 2020-12 meta-schema with jsonschema, a dev dependency, and checks that every array declares its items, schemas use only JSON Schema keywords and formats, enum, default and example values have their type, enum values are distinct, and every tool and parameter is described.

Tool names are unchanged, the catalog holds the same 143 toolsets and 3,406 tools, and API-key parameters are as upstream has them.

How it was verified

At each commit, on that commit's tree:

$ make check-all && make test && make smoke
All checks passed!
371 files already formatted
4670 passed in 96.64s (0:01:36)
All checks passed.

That is the first commit's tree; the next four pass with the same count, and the sixth with 4676, the catalog test's six new tests included.

At the head, with check-dependency-licenses for the new dev dependency:

$ make check-license-headers
🎉 All files have correct license headers!
$ make check-doc-links
✓ every relative link, and every link into this repository's main branch, resolves
$ make check-notebooks-clean
OK 4 notebook(s) carry no outputs
$ make check-dist
✓ installed package runs from a clean environment
$ make check-extensions
all extension seams work against an installed distribution
$ make check-dependency-licenses
All dependency licenses satisfy the Apache-2.0 compatibility policy.

• The catalog test's new tests fail on the base catalog, five of its fourteen:

  E  arrays without an item schema: ['UltraTools/Music:event_atmosphere_match.music_list', 'UltraTools/Music:filter_music_by_weather.music_list', 'UltraTools/Music:style_similarity_f
  E  keywords outside JSON Schema: ['UltraTools/Movie:ai_search_tool.region: example', 'UltraTools/Movie:ai_search_tool.start_date: example', 'UltraTools/Movie:ai_search_tool.end_dat
  E  defaults or examples of another type: ['AutoTools/Food:get_search_recipes_by_nutrients.maxVitaminB2: string 50', 'AutoTools/Food:get_search_recipes_by_nutrients.minVitaminB3: st
  E  enums that repeat a value: ['UltraTools/Document:special_recipe_search.cooking_method', 'UltraTools/Travel:special_recipe_search.cooking_method', 'UltraTools/Restaurant:special_
  E  tools without a description: ['ToolEyes/Search/EngineSearch:baidu_search', 'ToolEyes/Search/EngineSearch:google_search', 'ToolEyes/Search/EngineSearch:bing_search', 'ToolEyes/Se
  

• A model reviewed every spec, one request per batch of up to 40 tools of one toolset, for descriptions of another operation, parameters that contradict their operation, parameters a call needs that are missing or unknowable, and garbled text. Each flag was checked against the tool's upstream source (ToolEyes' tool.py, UltraTool's dev.json plans and results, the RestGPT and AutoTools specs, API-Bank's classes and all_apis.csv) and either fixed or kept, with that evidence. The review ran again after each round of fixes, and every flag of the five passes has a verdict:

  pass 1: 486 flags, 433 fixed,  53 not a defect
  pass 2: 149 flags,  72 fixed,  77 not a defect
  pass 3: 128 flags,  41 fixed,  87 not a defect
  pass 4: 118 flags,  38 fixed,  80 not a defect
  pass 5:  85 flags,  11 fixed,  74 not a defect
  systematic: 73 tools without parameters and 56 strings described as lists, 41 of 140 copies fixed
  

• Every spec of the head's catalog, sent in batches of up to 64 per toolset through a custom provider config, was accepted by all three routes:

  == OpenAI route: 171 requests, 3406 tools, 0 not accepted
  == Anthropic route: 171 requests, 3406 tools, 0 not accepted
  == NVIDIA route: 171 requests, 3406 tools, 0 not accepted
  

• usersim simulate --probe-mix tool_calling=1.0 --toolset-seed-path <catalog>, then usersim eval --scorers auto, ran against the same config in en_GB on a catalog holding four changed toolsets (ToolEyes CoinMarketCap, UltraTool Flight, AutoTools Music, API-Bank Finance & Banking): 4 rows, with every model's requests at failed=0. Every tool call the rows record passes only parameters its spec defines, and each row's evaluation reads:

  judge: scored
  language_compliance: ok (error=None)
  refusal_basics: ok (error=None)
  response_shape: ok (error=None)
  tool_use: ok (error=None)
  

• make health-check ran against the same config for tool_calling in en_GB on the head's catalog. It ended with ✓ simulate, eval and report all completed, and every model's requests show failed=0.

Results impact

Every run that samples a changed toolset gives the assistant different specs: typed parameters, item types, descriptions of what each tool does, and the parameters each call needs. 126 of the 143 toolsets and 1,166 of the 3,406 tools change, so tool_calling conversations and tool_use scores move for most rows.