Repository navigation
Conversation
Each tool in the Game toolset has the parameters that game.tools.json in the AutoTools repository, at commit 09c88df, lists for its endpoint. They are converted as the Food, Music and Weather tools' parameters are: the type in lower case, the description ending with the parameter's location, and a parameter named in the endpoint's URL required. That file lists them under params, where the other three use parameters. Eleven of the 18 tools take parameters, 20 in all; seven require an id or a year. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
Each TMDB tool whose endpoint names a path parameter, such as movie_id
in /movie/{movie_id}, has that parameter and requires it. RestGPT's
tmdb_oas.json, at commit d990128, declares these parameters on the path
item, which applies them to every operation under it. Upstream gives
them no description, so each of the 12 has a short one written for this
catalog. 34 of the 54 tools gain parameters, 44 in all.
Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
A Spotify tool requires a parameter only when RestGPT's spotify_oas.json, at commit d990128, marks it required. The spec writes that mark as the string true or false, and false leaves a parameter such as market, limit or offset optional. The fields of an operation's JSON request body are parameters as well, so the tool post_users_user_id_playlists takes the name of the playlist it creates, and the type parameter of get_search keeps the values upstream lists for its items. 24 of the 40 tools change. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
Each tool name is unique within its toolset, and each parameter name fits the pattern providers accept: a request that repeats a tool name or carries an invalid parameter name is rejected. Three ToolEyes toolsets repeat a name in their upstream config. In API F1 and Anti Malware the second spec describes another function in the toolset's tool.py, teams and blacklist_url, and in spoonacular products the repeat is identical and appears once. In Execute, code and return_err_output are the names the functions' signatures use. The catalog holds 3,406 tools, and both READMEs give that count. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
Every parameter schema in the catalog declares a JSON Schema type, and enum values have that type. population_growth_rate in the UltraTool analyze_impact_on_economy tool has type number, where upstream writes float, which providers reject. with_status and with_type in TMDB's get_discover_tv list their values as strings, and the value parameter of the Custom set_device_setting tool has type string. The results schema that two UltraTool tools nest in their arguments is removed. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
tests/engine/probes/test_tool_catalog.py checks the catalog the tool_calling probe ships. num_tools counts each toolset's specs. Tool names are valid and unique within a toolset, and parameter names, types and enum values are ones JSON Schema and providers accept. Every required name is a declared parameter, no parameters schema carries a results schema, and each placeholder in a tool's endpoint is a required parameter. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
Each row's toolset_source names the project its specs come from. The Spotify and TMDB toolsets are RestGPT, from RestBench's OpenAPI specs, and the Food, Game, Music and Weather toolsets are AutoTools, from the tools folder of the AutoTools repository. The commit changes no tool spec and no count. The tool_calling README has a Sources and licenses section. For each source it gives the upstream revision the specs match, the changes made to them, the license and a citation. It carries the MIT notices of RestGPT and DAMO-ConvAI and a note on the trademarks the specs name. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
The example tool spec is the catalog's google_trends_search spec, abridged to two of its 11 parameters, with the upstream wording and its required list. The README describes each tool response as a model writes it: shaped by the tool's schema, with no live service behind it. The schema table notes that only the ToolEyes rows set toolset_name, and the long tail holds 64 categories. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed and why
Every spec in the catalog is one the tested providers accept, the AutoTools and RestGPT tools take the parameters their upstream declares, and every toolset names the project its specs come from, with the revision they match, the changes made to them, the license and a citation.
Builds on #127.
game.tools.jsonat commit09c88dflists for its endpoint. That file lists them underparams, where the Food, Music and Weather files useparameters. Eleven of the 18 tools take parameters, 20 in all; seven require anidor ayear.movie_idin/movie/{movie_id}, has that parameter and requires it. RestGPT'stmdb_oas.jsondeclares them on the path item, which applies them to every operation under it. 34 of the 54 tools gain 44 parameters, and the 12 distinct ones, which have no upstream description, each have a short one.spotify_oas.jsonmarks it required. The spec writes that mark as the string"true"or"false", and"false"leaves 50 parameters, such asmarket,limitandoffset, optional. The fields of a JSON request body are parameters as well (14 fields on four tools), sopost_users_user_id_playliststakes thenameof the playlist it creates. 24 of the 40 tools change.driversin API F1 isteamsand the secondblacklist_ipin Anti Malware isblacklist_url, the functions their specs describe in the toolset's upstreamtool.py; spoonacular products keeps one of its two identicalclassify_grocery_productspecs; and Execute'scode:andreturn_err_outputpersistent=Falsearecodeandreturn_err_output, as in the functions' signatures. The catalog holds 3,406 tools, and both READMEs give that count.floatisnumberin one UltraTool tool, the integer enum values of TMDB'swith_statusandwith_typeare strings, and the Customset_device_setting.valuehas typestring. Theresultsschema two UltraTool tools nest in their arguments is removed.tests/engine/probes/test_tool_catalog.pychecks the invariants above for every spec the probe ships.toolset_sourceisRestGPTfor the Spotify and TMDB toolsets andAutoToolsfor the Food, Game, Music and Weather toolsets. The README's Sources and licenses section gives, for each of the six sources, the upstream revision the specs match, every change made to them, the license and a citation, with the MIT notices of RestGPT and DAMO-ConvAI and a trademarks note.google_trends_search, abridged, in its upstream wording. The README describes each tool response as a model writes it from the tool's schema, notes that only theToolEyesrows settoolset_name, and gives the long tail as 64 categories.How it was verified
At each commit, on that commit's tree:
That is the first commit's tree; the next four pass with the same count, and the last three with 4670 tests, the catalog test's eight included.
At the head:
The catalog test fails on the base catalog, six of its eight tests:
Every spec was compared with its upstream at the pinned commits. Rebuilding the AutoTools and RestGPT specs under the README's rules gives the shipped specs, except
get_game_reviewed_today, whose upstreamurlfield holds its description, and TMDB'sget_discover_tv, whose enum values are strings:The UltraTool, ToolEyes and API-Bank specs equal upstream's, apart from the changes the README lists:
Between the base and the head, the catalog differs in these cells:
Against a custom provider config, requests carrying the base catalog's
floatparameter were rejected by the OpenAI and Anthropic routes, and requests carrying a toolset with a repeated name or withcode:by the Anthropic route:Every spec of the head's catalog, sent in batches of up to 64 per toolset, was accepted by all three routes:
usersim simulate --probe-mix tool_calling=1.0 --toolset-seed-path <catalog>, thenusersim eval --scorers auto, ran against the same config inen_GB: 3 rows on a catalog holding only the Game row and 4 rows on one holding only the Spotify and TMDB rows, with every model's requests atfailed=0. Every tool call the rows record passes only parameters its spec defines, and each of the seven rows' evaluations reads:make health-checkran against the same config fortool_callinginen_GBon the head's catalog. It ended with✓ simulate, eval and report all completed, and every model's requests showfailed=0.Results impact
tool_usescores for those rows move. A row offers the assistant five tools sampled from its toolset, so rows whose sample drewanalyze_impact_on_economy(rejected on the OpenAI and Anthropic routes), both copies of a repeated name or an Execute tool with a malformed key (rejected on the Anthropic route) complete.toolset_source. Rows that sample the six RestGPT and AutoTools toolsets recordRestGPTorAutoTools, where runs of earlier releases recordAuto-generated, so a comparison grouped bytoolset_sourceacross releases splits that group.