diff --git a/README.md b/README.md index ca11081..59eda9b 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ > trajectories: evaluation signal, failure analysis, and curated training data, > before you have production traffic to learn from. -**14 probes · 11 shipped locales across 9 countries (+ 22 preview India-language variants) · 8M+ census-grounded personas · 3,407 tools across 143 toolsets for agentic runs · 12 evaluator scorers · 4,000+ tests · offline `usersim smoke` in seconds** +**14 probes · 11 shipped locales across 9 countries (+ 22 preview India-language variants) · 8M+ census-grounded personas · 3,406 tools across 143 toolsets for agentic runs · 12 evaluator scorers · 4,000+ tests · offline `usersim smoke` in seconds** NeMo User Sim: Population-Grounded User Simulation. Simulated users in the USA, Brazil, France, India and Japan, each speaking their own language under a different probe, on a world map beside headline counts for people represented, languages, locales and probes. diff --git a/src/usersim/engine/assets/tool_calling/README.md b/src/usersim/engine/assets/tool_calling/README.md index f69ac75..c84de03 100644 --- a/src/usersim/engine/assets/tool_calling/README.md +++ b/src/usersim/engine/assets/tool_calling/README.md @@ -13,46 +13,34 @@ authoritative per-tool descriptions live inside each tool's JSON spec. ## Catalog at a glance -**143 toolsets · 3,407 tools · 79 categories.** A "toolset" is a named bundle +**143 toolsets · 3,406 tools · 79 categories.** A "toolset" is a named bundle of related tools (e.g. `openweathermap` → `current_weather`); `num_tools` counts the functions in that bundle. ## Tools by source -Specs are drawn from public tool-calling benchmarks plus a small curated / -auto-generated set. Counts: +Specs come from five public tool-calling benchmarks, plus a set written for +this project. Counts: | Source | Toolsets | Tools | |---|---:|---:| | `UltraTools` | 22 | 2,328 | -| `ToolEyes` | 95 | 568 | +| `ToolEyes` | 95 | 567 | | `API-Bank` | 10 | 225 | -| `Auto-generated` | 6 | 177 | | `Custom` | 10 | 109 | -| **Total** | **143** | **3,407** | +| `RestGPT` | 2 | 94 | +| `AutoTools` | 4 | 83 | +| **Total** | **143** | **3,406** | > `UltraTools` dominates the raw tool count; `ToolEyes` contributes the most > distinct toolsets (95). Per-toolset size ranges from 1 to 354 functions > (mean ≈ 24). -### Provenance - -`toolset_source` records where each row came from, and the label is worth -reading precisely: - -- **`Custom`**: authored for this project. Generic domains (Inventory - Warehouse, Hospital Ward Manager, Recipe Manager, and so on) with no - external derivation. -- **`Auto-generated`**: despite the name, these describe publicly - documented third-party APIs: Spoonacular, OpenCritic and Genius via - RapidAPI, plus Spotify, TMDB and Meteosource. The descriptions retain the - vendors' own wording and endpoint URLs. -- **`ToolEyes` / `UltraTools` / `API-Bank`**: drawn from the public - tool-calling benchmarks of those names. - -Only the schemas are bundled; nothing here calls a live API, and the -catalog exists to exercise a model's tool-selection behaviour rather than to -reach any of these services. +`toolset_source` records where each row came from. [Sources and +licenses](#sources-and-licenses) lists each upstream project, the revision +the specs match, what changed and the license. Only the schemas are bundled; +nothing here calls a live API, and the catalog exists to exercise a model's +tool-selection behaviour rather than to reach any of these services. ## Tools by category (top 15 of 79) @@ -71,10 +59,10 @@ reach any of these services. | Competition | 95 | | Account | 87 | | Train | 86 | -| Food | 85 | +| Food | 84 | | Alarm | 70 | -The remaining ~64 categories (Calendar, Database, Smart Home & IoT, Health & +The remaining 64 categories (Calendar, Database, Smart Home & IoT, Health & Medical, Music, News, Translation, Stock, Weather, Hospital Ward Manager, Library Management System, …) make up the long tail. @@ -84,10 +72,11 @@ Library Management System, …) make up the long tail. ``` UltraTools ████████████████████████████████████████ 2328 -ToolEyes ██████████ 568 +ToolEyes ██████████ 567 API-Bank ████ 225 -Auto-generated ███ 177 Custom ██ 109 +RestGPT ██ 94 +AutoTools █ 83 ``` **Toolsets by size** (how many functions each toolset bundles): @@ -112,7 +101,7 @@ toolsets_seed.parquet ── sample a toolset ──▶ assistant_model (model u │ emits tool call(s) ▼ api_response_model synthesizes a - deterministic, schema-shaped response + schema-shaped response │ ▼ tool_use scorer grades the trajectory @@ -127,7 +116,7 @@ toolsets_seed.parquet ── sample a toolset ──▶ assistant_model (model u decides whether/which to call. - There is **no real execution**: the `api_response_model` alias (see [`cli/models_default.toml`](../../../cli/models_default.toml)) synthesizes the - tool's response from the schema, so behavior is reproducible. + tool's response from the schema, so a run needs no live service. - Conversation themes (some tool-expected, some not) come from `DEFAULT_TOOL_CALLING_THEMES` in `cli/_pipeline.py`, exercising the call/clarify/refuse decision. @@ -140,9 +129,9 @@ per line, with these keys: | Column | Type | Description | |---|---|---| -| `toolset_source` | str | Provenance (`ToolEyes`, `API-Bank`, `UltraTools`, `Custom`, `Auto-generated`). | +| `toolset_source` | str | Provenance (`UltraTools`, `ToolEyes`, `API-Bank`, `Custom`, `RestGPT`, `AutoTools`). | | `toolset_category` | str | Coarse domain label (e.g. `Weather`, `Finance`). | -| `toolset_name` | str | Bundle name (e.g. `openweathermap`). | +| `toolset_name` | str | Bundle name (e.g. `openweathermap`). The shipped catalog sets it on the `ToolEyes` rows and leaves it empty on the others. | | `num_tools` | int | Number of functions in the bundle. | | `tools` | str (JSON) or list | List of OpenAI-style function specs (`{"type": "function", "function": {"name", "description", "parameters"}}`). | @@ -178,6 +167,8 @@ error because the override would otherwise be silently unused. ### Example tool spec (`ToolEyes / Trend / Google Trends`) +Abridged to two of its 11 parameters; the wording is the upstream text. + ```json { "type": "function", @@ -189,13 +180,14 @@ error because the override would otherwise be silently unused. "properties": { "query": { "type": "string", - "description": "The query or queries to search (up to 5 for some data types)." + "description": "Parameter defines the query or queries you want to search. You can use anything that you would use in a regular Google Trends search. The maximum number of queries per search is 5 (this only applies to \"Interest over time\" and \"Compared breakdown by region\" data_type, other types of data will only accept 1 query per search)." }, "geo": { "type": "string", - "description": "Two-letter country code to scope the trend (e.g. US)." + "description": "Parameter defines the location from where you want the search to originate. It defaults to Worldwide (activated when the value of geo parameter is not set or empty). Head to the Google Trends Locations for a full list of supported Google Trends locations." } - } + }, + "required": ["query"] } } } @@ -223,3 +215,175 @@ json.loads(df.iloc[0]["tools"]) - The **agentic** safety surface uses a *different*, hand-authored mock-tool mechanism (per-action tool specs + canned responses) documented in [`probes/safety_agentic/assets/safety_agentic/SCHEMA.md`](../../probes/safety_agentic/assets/safety_agentic/SCHEMA.md). + +## Sources and licenses + +The catalog combines tool specs from five public research benchmarks with +toolsets written for NeMo User Sim. Only tool schemas (name, description and +parameters) are included: no code, dialogues, test cases or recorded API +responses from these projects. + +### Research benchmarks + +| `toolset_source` | Toolsets | Tools | Upstream | License | +|---|---:|---:|---|---| +| `UltraTools` | 22 | 2,328 | [UltraTool](https://github.com/JoeYing1019/UltraTool) | [Apache-2.0](https://github.com/JoeYing1019/UltraTool/blob/main/LICENSE) | +| `ToolEyes` | 95 | 567 | [ToolEyes](https://github.com/Junjie-Ye/ToolEyes) | [Apache-2.0](https://github.com/Junjie-Ye/ToolEyes/blob/main/LICENSE) | +| `API-Bank` | 10 | 225 | [API-Bank](https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/api-bank), part of DAMO-ConvAI | [Apache-2.0](https://github.com/AlibabaResearch/DAMO-ConvAI/blob/main/api-bank/LICENSE) for the `api-bank` folder, [MIT](https://github.com/AlibabaResearch/DAMO-ConvAI/blob/main/LICENSE) for the repository | +| `RestGPT` (categories `Spotify`, `TMDB`) | 2 | 94 | [RestGPT](https://github.com/Yifan-Song793/RestGPT) (RestBench) | [MIT](https://github.com/Yifan-Song793/RestGPT/blob/main/LICENSE) | +| `AutoTools` (categories `Food`, `Game`, `Music`, `Weather`) | 4 | 83 | [AutoTools](https://github.com/mangopy/AutoTools) | [Apache-2.0](https://github.com/mangopy/AutoTools/blob/main/LICENSE) | + +Every spec is wrapped in the OpenAI function-calling envelope +(`{"type": "function", "function": {...}}`) and placed in a toolset. The +other changes are listed per source: + +- **UltraTool** ([Huang et al., 2024](https://aclanthology.org/2024.findings-acl.259/)): + every distinct tool spec in the English `dev.json` split of + [`data/English-dataset`](https://github.com/JoeYing1019/UltraTool/tree/5d340adc26d5881725308398dd5dfc6441eb0156/data/English-dataset) + (commit `5d340ad`), with one toolset per upstream `domain` label. Tool names + and descriptions are unchanged. The `arguments` field is renamed + `parameters`, and the `results` (output) schema is removed, including where + upstream nests it in the arguments of `seat_preference_query` and + `select_and_change_ticket`. Argument schemas are otherwise unchanged, except + that `population_growth_rate` in `analyze_impact_on_economy` has type + `number`, where upstream writes `float`, which is not a JSON Schema type. +- **ToolEyes** ([Ye et al., 2025](https://aclanthology.org/2025.coling-main.12/)): + all 95 toolsets in + [`Tool_Library`](https://github.com/Junjie-Ye/ToolEyes/tree/c818fcebe017951570a25e552a7984e40093fd52/Tool_Library) + (commit `c818fce`), read from each toolset's `config_gpt4.json` and named + after its folders. Specs are unchanged and keep their upstream order. The two + agent-control functions that every upstream toolset includes, `ask_to_user` + and `finish`, are removed. Three toolsets repeat a name upstream: the second + `drivers` in `API F1` is `teams` and the second `blacklist_ip` in + `Anti Malware` is `blacklist_url`, the functions their descriptions match in + the toolset's `tool.py`, and `spoonacular products` keeps one of its two + identical `classify_grocery_product` specs. In `Execute`, two parameter names + are the ones the functions' signatures in `tool.py` use: `code` in + `execute_pycode` (upstream `code:`) and `return_err_output` in + `execute_bash_code` (upstream `return_err_outputpersistent=False`). +- **API-Bank** ([Li et al., 2023](https://aclanthology.org/2023.emnlp-main.187/)): + 100 of the 101 APIs listed in + [`api-bank/data/all_apis.csv`](https://github.com/AlibabaResearch/DAMO-ConvAI/blob/faf754671f5b850e1b4d1ff870db37a6a2480df6/api-bank/data/all_apis.csv) + (commit `faf7546`), all except `ImageCaption`. Descriptions and parameter + descriptions are unchanged. Tool names are the upstream class names in lower + case (`AddAlarm` becomes `addalarm`); parameter types are written as JSON + Schema types (`str` as `string`, `int` as `integer`, `datetime` as `string`, + `List[str]` as `array`); 32 tools gain a `required` list; and output + parameters are removed. The `All` toolset holds every API, and nine topical + toolsets repeat subsets of them. +- **RestGPT** ([Song et al., 2023](https://arxiv.org/abs/2306.06624)): + all 40 Spotify and 54 TMDB operations in RestBench's OpenAPI specs under + [`specs`](https://github.com/Yifan-Song793/RestGPT/tree/d99012860cb45323975bf5780152d7aa44f390f5/specs) + (commit `d990128`), one tool per operation, named after its HTTP method and + path. A tool's parameters are the operation's own, those declared on its + path, and the fields of its JSON request body; a body field that repeats a + query parameter appears once, as the query parameter. A parameter is + required when upstream marks it so, and the Spotify spec writes that mark as + the string `"true"` or `"false"`. Parameter schemas keep their type, `enum`, + `default` and `items`. Descriptions are joined onto one line; each tool + description ends with the operation's method and URL, and each parameter + description ends with its location (for example `Location: query` or + `Location: body`). One operation and 16 parameters that have no upstream + description were given a short one: four query parameters and the 12 TMDB + path parameters. In TMDB's `get_discover_tv`, the values that `with_status` + and `with_type` list for a string parameter are written as strings. +- **AutoTools** ([Shi et al., 2025](https://arxiv.org/abs/2405.16533)): + all 83 endpoints in the `food`, `game`, `music` and `weather` files under + [`tools`](https://github.com/mangopy/AutoTools/tree/09c88df0959b2baf03e570aeff4365ea9f4d8b64/tools) + (commit `09c88df`), one toolset per file and one tool per endpoint. They + document APIs listed on RapidAPI: Spoonacular, OpenCritic, Genius and + Meteosource. Tool names are the upstream names in lower case + (`GET_menu_item_information` becomes `get_menu_item_information`). + Parameters come from each endpoint's `parameters` list, which the `game` + file names `params`. Descriptions are trimmed; each tool description ends + with the endpoint's method and URL, and each parameter description ends with + its location (for example `Location: query`). Parameter types are written in + lower case, `float` as `number` and an empty type as `string`; a parameter + named in the endpoint's URL is required; and the sample responses are + removed. One `game` endpoint, `get_game_reviewed_today`, has its description + in the upstream `url` field, so its tool description ends with that text and + the API's host. + +### Toolsets written for this project + +`Custom` (10 toolsets, 109 tools) covers generic domains such as Inventory +Warehouse, Hospital Ward Manager and Recipe Manager. These toolsets were +written for NeMo User Sim and are covered by this project's Apache-2.0 +license. + +### License notices + +UltraTool, ToolEyes, AutoTools and the `api-bank` folder of DAMO-ConvAI are +licensed under the Apache License, Version 2.0. None of them includes a NOTICE file. +The full license text is in this project's +[LICENSE](https://github.com/NVIDIA-NeMo/UserSim/blob/main/LICENSE) file. + +RestGPT and the DAMO-ConvAI repository are licensed under the MIT License. +Their notices follow. + +```text +MIT License + +Copyright (c) 2023 Yifan Song + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. +``` + +```text +MIT License + +Copyright (c) 2022 Alibaba Research + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. +``` + +### Trademarks and third-party services + +Product and service names in the catalog, such as Spotify, TMDB, Spoonacular, +OpenWeatherMap and Google Trends, are trademarks of their owners and identify +the APIs that the specs describe. The catalog holds schemas only: nothing in +it calls a live API, and using any of these services is subject to its +provider's terms. + +### Citation + +If you publish results obtained with this catalog, please also cite the +benchmarks it draws on: + +- Shijue Huang et al. 2024. [Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios](https://aclanthology.org/2024.findings-acl.259/). In *Findings of the Association for Computational Linguistics: ACL 2024*. +- Junjie Ye et al. 2025. [ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios](https://aclanthology.org/2025.coling-main.12/). In *Proceedings of the 31st International Conference on Computational Linguistics (COLING 2025)*, pages 156-187. +- Minghao Li et al. 2023. [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://aclanthology.org/2023.emnlp-main.187/). In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023)*. +- Yifan Song et al. 2023. [RestGPT: Connecting Large Language Models with Real-World RESTful APIs](https://arxiv.org/abs/2306.06624). arXiv:2306.06624. +- Zhengliang Shi et al. 2025. [Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agents](https://arxiv.org/abs/2405.16533). WWW 2025. arXiv:2405.16533. diff --git a/src/usersim/engine/assets/tool_calling/toolsets_seed.parquet b/src/usersim/engine/assets/tool_calling/toolsets_seed.parquet index 10abd39..bf9874c 100644 Binary files a/src/usersim/engine/assets/tool_calling/toolsets_seed.parquet and b/src/usersim/engine/assets/tool_calling/toolsets_seed.parquet differ diff --git a/tests/engine/probes/test_tool_catalog.py b/tests/engine/probes/test_tool_catalog.py new file mode 100644 index 0000000..311203f --- /dev/null +++ b/tests/engine/probes/test_tool_catalog.py @@ -0,0 +1,157 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Tests for the tool catalog the tool_calling probe ships. + +Each row's tools reach the assistant model unchanged, as OpenAI-style function +specs, so every spec in ``toolsets_seed.parquet`` must be one a provider accepts. +""" + +import json +import re +from collections import Counter +from collections.abc import Iterator + +import pandas as pd +import pytest + +from usersim.engine.core.missing import is_missing +from usersim.engine.probes.tool_calling.generator import ToolCallingProbe + +JSON_SCHEMA_TYPES = frozenset({"string", "number", "integer", "boolean", "array", "object", "null"}) +_PYTHON_TYPES = { + "string": str, + "number": (int, float), + "integer": int, + "boolean": bool, + "array": list, + "object": dict, + "null": type(None), +} +_FUNCTION_NAME = re.compile(r"^[a-zA-Z0-9_-]{1,64}$") +_PARAMETER_NAME = re.compile(r"^[a-zA-Z0-9_.-]{1,64}$") +#: The method and URL a REST tool's description ends with. +_ENDPOINT = re.compile(r" HTTP [A-Z]+ (\S+)$") + + +@pytest.fixture(scope="module") +def toolsets() -> list[tuple[str, int, list[dict]]]: + """Each toolset as (label, num_tools, tool specs).""" + df = pd.read_parquet(ToolCallingProbe.assets_dir() / "toolsets_seed.parquet") + out = [] + for row in df.itertuples(): + name = "" if is_missing(row.toolset_name) else f"/{row.toolset_name}" + out.append((f"{row.toolset_source}/{row.toolset_category}{name}", row.num_tools, json.loads(row.tools))) + assert out, "the shipped tool catalog holds no toolsets" + return out + + +def _schemas(schema: dict, path: str) -> Iterator[tuple[str, dict]]: + """Each property and item schema inside a parameters schema, with its path.""" + for name, sub in (schema.get("properties") or {}).items(): + yield f"{path}.{name}", sub + yield from _schemas(sub, f"{path}.{name}") + if isinstance(schema.get("items"), dict): + yield f"{path}[]", schema["items"] + yield from _schemas(schema["items"], f"{path}[]") + + +def _tools(toolsets) -> Iterator[tuple[str, dict]]: + for label, _, specs in toolsets: + for spec in specs: + yield f"{label}:{spec['function']['name']}", spec["function"] + + +def _fits(value, json_type: str) -> bool: + if isinstance(value, bool) and json_type in ("number", "integer"): + return False + return isinstance(value, _PYTHON_TYPES[json_type]) + + +def test_num_tools_counts_each_toolsets_specs(toolsets): + wrong = [(label, n, len(specs)) for label, n, specs in toolsets if n != len(specs)] + assert not wrong, f"num_tools differs from the number of specs (toolset, num_tools, specs): {wrong}" + + +def test_every_tool_is_a_function_spec_with_a_name_unique_in_its_toolset(toolsets): + bad_shape = [ + f"{label}: {sorted(spec)}" + for label, _, specs in toolsets + for spec in specs + if spec.get("type") != "function" or set(spec["function"]) - {"name", "description", "parameters"} + ] + bad_names = [where for where, f in _tools(toolsets) if not _FUNCTION_NAME.match(f["name"])] + not_objects = [where for where, f in _tools(toolsets) if f["parameters"].get("type") != "object"] + repeated = [ + f"{label}:{name}" + for label, _, specs in toolsets + for name, count in Counter(s["function"]["name"] for s in specs).items() + if count > 1 + ] + assert not bad_shape, f"specs outside the function envelope: {bad_shape[:10]}" + assert not bad_names, f"names a provider rejects: {bad_names[:10]}" + assert not not_objects, f"parameters that are not an object schema: {not_objects[:10]}" + assert not repeated, f"names repeated within a toolset, which a provider rejects: {repeated}" + + +def test_parameter_names_are_ones_a_provider_accepts(toolsets): + wrong = [ + f"{where}{path}" + for where, f in _tools(toolsets) + for path, _ in _schemas(f["parameters"], "") + if not path.endswith("[]") and not _PARAMETER_NAME.match(path.rsplit(".", 1)[-1]) + ] + assert not wrong, f"parameter names a provider rejects: {wrong[:10]}" + + +def test_every_schema_declares_a_json_schema_type(toolsets): + """A missing type leaves the assistant guessing; a type outside JSON Schema fails the request.""" + wrong = [ + f"{where}{path}: {schema.get('type')!r}" + for where, f in _tools(toolsets) + for path, schema in _schemas(f["parameters"], "") + if schema.get("type") not in JSON_SCHEMA_TYPES + ] + assert not wrong, f"schemas without a JSON Schema type: {wrong[:10]}" + + +def test_enum_values_have_the_declared_type(toolsets): + wrong = [ + f"{where}{path}: {schema['type']} {schema['enum']}" + for where, f in _tools(toolsets) + for path, schema in _schemas(f["parameters"], "") + if "enum" in schema and not all(_fits(v, schema["type"]) for v in schema["enum"]) + ] + assert not wrong, f"enum values of another type: {wrong[:10]}" + + +def test_every_required_name_is_a_declared_parameter(toolsets): + wrong = [ + f"{where}{path}: {sorted(set(schema['required']) - set(schema.get('properties') or {}))}" + for where, f in _tools(toolsets) + for path, schema in [("", f["parameters"]), *_schemas(f["parameters"], "")] + if isinstance(schema.get("required"), list) + and not set(schema["required"]) <= set(schema.get("properties") or {}) + ] + assert not wrong, f"required names that are not parameters: {wrong[:10]}" + + +def test_parameters_hold_no_output_schema(toolsets): + wrong = [where for where, f in _tools(toolsets) if "results" in f["parameters"]] + assert not wrong, f"parameters schemas carrying a results schema: {wrong}" + + +def test_each_placeholder_in_an_endpoint_is_a_required_parameter(toolsets): + """A tool for ``GET .../movie/{movie_id}`` can only be called with ``movie_id``.""" + wrong, checked = [], 0 + for where, f in _tools(toolsets): + endpoint = _ENDPOINT.search(f["description"]) + if not endpoint: + continue + checked += 1 + params = f["parameters"] + for name in re.findall(r"\{(\w+)\}", endpoint.group(1)): + if name not in (params.get("properties") or {}) or name not in params.get("required", []): + wrong.append(f"{where}: {name}") + assert checked, "no tool description ends with an endpoint, so no placeholder was checked" + assert not wrong, f"endpoint placeholders that are not required parameters: {wrong[:10]}"