[domfarolino]

This proposal includes three concrete API changes to satisfy great feedback from Mozilla, largely captured in #236 and #237.

Please take a look @jakearchibald, @martinthomson, @bvandersloot-mozilla.

[beaufortfrancois]

Suggested change

	model: {
	naturalLanguageProperties: {

[beaufortfrancois]

It's not clear if exposedTo behavior is changing from what we currently in WebMCP spec.

[bvandersloot-mozilla]

I think this states that it does. And I agree with the direction

[khushalsagar]

The behaviour of exposedTo remains unchanged. It controls which execution contexts have visibility into a tool.

But your comment makes me think that there are 2 concepts here which are combined and its worth spelling that out, in case there's a better way to approach this in the API:

1. The author needs to be able to say, "this tool is just an API between my same-origin frames or between me and other.com, its not for any agent. Don't show it to the browser agent".
2. For a tool to be usable by the browser agent (or any agent), it needs to have these natural language properties.

The API is combining these concepts by saying that the presence of naturalLanguageProperties implies the tool is meant for the browser agent.

An alternative we considered was a separate value in exposedTo called "native-agent", "browser" or "user-agent" to say that this tool should be exposed to consumers outside the web platform. And say that if these consumers are using AI models then they also need the fields in naturalLanguageProperties. So now an author needs to set the bit in exposedTo and naturalLanguageProperties to have the tool be added to the browser's native AI agent.

We didn't go for this because the external consumer use-case is limited to AI consumers but I can see that splitting the 2 is more forward compatible.

[beaufortfrancois]

Is it important to distinguish following cases?

await document.modelContext.registerTool({
  name: 'fake_operation_error',
  description: 'desc',
  execute: () => { throw new DOMException('Schema validation failed', 'OperationError') }
});
const [tool] = await document.modelContext.getTools();
await document.modelContext.executeTool(tool);
// throws with a fake OperationError: Schema validation failed

vs

await document.modelContext.registerTool({
  name: 'true_operation_error',
  description: 'desc',
  execute: () => { result: 'string instead of number' } ,
  outputSchema: {
    type: 'object',
    properties: { result: { type: "number" }, }
  }
});
const [tool] = await document.modelContext.getTools();
await document.modelContext.executeTool(tool);
// throws with a real OperationError: Schema validation failed

If so, we may want to introduce a ToolExecutionError returned by executeTool().

[khushalsagar]

Good point. The fundamental question is whether the consumer needs to tell if the error came from the browser's own validation vs. from userland code. From a security perspective, I don't think userland code can do anything malicious by replicating a browser error, though I'm curious if you have a case in mind.

The other reason is ergonomic, maybe the harness wants to handle all class of userland errors differently vs schema validation errors in the browser. This becomes a convenient way of doing that.

[Kulikowski]

I understand that naturalLanguageProperties are just "properties" as they are not specific to "natural language consumers".

For example: annotations are not concern of a model, they are concern of a harness. Similarly the outputSchema.

I also think about cases when we will have debugging tools, for automation, and I need inputSchema and outputSchema mostly and there might be no natural language consumer in picture.

If you agree with above I would generalize "To make tool registrations useful ..." content. In current form it reads to me as naturalLanguageProperties are SPECIFIC to AI Agents and even more - natural language consumers (reads as models (Decision models, LLMs)?)

[khushalsagar]

I also think about cases when we will have debugging tools, for automation, and I need inputSchema and outputSchema mostly and there might be no natural language consumer in picture.

can you describe how this use-case would work more. I rationalized to myself that its only AI use-cases that can read a natural language description of an API and its parameters at runtime and invoke them. That's why parameters which are necessary for an AI system to make use of an API should be bucketed under a naturalLanguageProperties.

[andreban]

It would be possible to build a deterministic IFTTT system connecting outputs from one tool to the input of other using schemas provided by the tool developer to design those connections (I haven't seen this in practice, but would theoretically be possible).

My .02 cents is that naturalLanguageProperties describes who the information is for (agents) rather than what the information is (tool metadata), and describing what it is makes usage less prescriptive.

[fallaciousreasoning]

Additionally, at the moment we validate input against the inputSchema so I feel like naturalLanguageProperties is slightly misleading

[martinthomson]

This looks directionally good from my recollection of the discussion.

[martinthomson]

I'm not sure about this framing for this.

A model will want to invoke tools, which means that inputs might need to have fromJSON methods and outputs can use toJSON. That means lowering on the outputs, but lifting (?) on the inputs, right?

[khushalsagar]

You're right. For inputs to a tool, the model will generate JSON which will be converted to a web type when invoking the tool.

[khushalsagar]

A subtlety worth mentioning: this doesn't imply that the serialized JSON is added to a model's context exactly. Its meant to be an interoperable contract between the browser and agent harnesses, the format should err towards reducing the payload size. The agent harness is responsible for optimally adding the JSON to the model's context and ensuring its semantics are preserved.

For example, if the tool output is this:

{
  products: [
    { id: "sku-1", thumbnail: await createImageBitmap(lampImg) },
    { id: "sku-2", thumbnail: await createImageBitmap(chairImg) },
  ],
};

The serialized JSON could be:

{
  "products": [
    {
      "id": "sku-1",
      "thumbnail": { "mimeType": "image/jpeg", "data": "/9j/4AAQSkZJRgABAQ..." }
    },
    {
      "id": "sku-2",
      "thumbnail": { "mimeType": "image/jpeg", "data": "/9j/4AAQSkZJRgABAQ..." }
    }
  ]
}

The agent harness can show this to the model in any way as long as it understands which thumbnail is associated with which product id.

[martinthomson]

Didn't we talk about the idea that maybe the values returned by getTools() is transferable as well? I realize that this could add a layer of complexity, but it makes things much, much more useful.

Maybe your intention here was to cover that with this comment, but I thought I'd flag it here.

[martinthomson]

Forgiving is only feasible to the extent that the specific shape of any forgiving-ness can be fully specified. I'd start with strict treatment and strong feedback about what failed, then build from there.

[martinthomson]

I assume that this is with respect to how they are represented in JSON?

[martinthomson]

Rather than say "versioning", I'd prefer that this was about how it might be possible to evolve how things are transformed into a model-compatible form.

We discussed versioning at some length when we met and it seems like there was a bit of a disconnect about expectations. None of that made it into this document, which is good, but I'd like to keep that solid.

One way to manage changes in serialization is the same way you would manage a new, incompatible version of a tool: define a new tool. If the tool takes a different type of object, then that different type can have different transforms to and from JSON.

[martinthomson]

I wanted to flag that it might be feasible to defer this question in the short term without loss of function. Or, to incrementally expand support, at least.

A tool that exposes a JSON-native interface (null, bool, number, dictionary, array) could be made usable without really grappling with this issue. It's only when you want to pass objects that don't have a clean JSON.parse/.serialize round trip that you need to contend with this. Narrower than that really: when the transformation doesn't work in the direction you care about; passing an input object that has a toJSON probably works fine.

That doesn't mean that this isn't worth doing, but I'd treat this as something incremental rather than a core prerequisite. Whatever isn't solved now will mean it is not accessible to agents... for now.

[bvandersloot-mozilla]

Also directionally correct from where I sit!

[bvandersloot-mozilla]

I was arguing for (2) at TPAC, but I've been convinced this is right. I'd rename it to toolManager rather than toolRegistry personally (managers having writing power makes sense, whereas I might try go to a registry to read all of the tools). But, I defer naming if you've thought hard on it already.

[khushalsagar]

I think toolRegistry happened because were thinking about customElementRegistry when discussing this with jake. i'm good with toolManager as well.

[bvandersloot-mozilla]

Another one that is pretty important: tool coalescing and permissions policy.

Right now, tools can only be registered by frames that have permission and are automatically included in the top's getTools result. This is great for agent-intended tools, but makes much less sense for a message event equivalent.

I think we need to change the meaning of the permission granted from "can register tool" to "tool coalesces upward." This preserves the security property of preventing top level tool-space pollution while allowing generic tools. Maybe rename the policy key from "tools" to "include-tools" to make it a little clearer.

It may also make more sense to have getTools just give the subtree's coalesced tools, moving the current behavior to window.top.tools.getTools.

With both of these together, we can look at this case:

<body> <!-- top -->
 <iframe id="frameDisallowFoo" allow="tools 'none'">
   <!--registers tool foo -->
  </iframe>
 <iframe id="frameAllowBar" allow="tools">
   <!--registers tool bar -->
  </iframe>
  <script>
   <!--registers tool baz -->
  </script>
</body>

Currently in the spec,document.modelContext.getTools() would return bar and baz. And we still get that from top.tools.getTools() here! But now the top frame can also call frameDisallowFoo.tools.getTools() and get foo.

Without this change, there is no way to use this API as an RPC primitive with a frame and also not trust that frame to publish tools on your window.

[khushalsagar]

The exact API for reading tools aside (because that's changing with #227), there's 2 fundamental constraints that an embedder can impose on an iframe:

1. Can the iframe expose its tools to other Documents in the same browsing context group and the browser agent.
2. Can the iframe read tools from other Documents in the same browsing context group.

Right now the tools policy is a boolean control on both: without the policy the embedded Document can't register any tools (which implicitly means it can't expose tools to any Document) and it also can't read tools from any other Document. We should consider a couple of things for simplifying this:

a. Should tool exposure and tool read be separate permissions? I think yes. Because its a common case to want your tools to only be exposed to a trusted agent (like the browser agent), so you want to prevent the iframe from reading it. But you're ok with the iframe also exposing tools to the browser agent for the functionality its adding to your side.

b. Is the current boolean syntax good enough or should the permission policy mirror the syntax for exposedOrigins? This one I'm not sure about. I think we can start with a bool and add support for exposedOrigins without any compat issues.

[bvandersloot-mozilla]

there's 2 fundamental constraints that an embedder can impose on an iframe:

I disagree. I think exposing tools ban be split into two modes (a) included in a direct getTools on that window or (b) included in a getTools on an ancestor frame.

a. Should tool exposure and tool read be separate permissions?

Yes, and I believe read shouldn't require a permission at all unless we have a concrete use case.

b. Is the current boolean syntax good enough

If you mean permissions policy, they are not boolean. See MDN: https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Permissions_Policy#allowlists.

[bvandersloot-mozilla]

I think this states that it does. And I agree with the direction