BYOK generic-chat-completion-api: parallel tool calls with image results create an orphan role:"tool" message -> 400 from strict upstreams

[marovole]

Summary

With a BYOK custom model configured as provider: "generic-chat-completion-api" + noImageSupport: false, an assistant turn that makes 2 or more tool calls in one message where a non-final tool result carries an image generates an invalid request: a role: "tool" message ends up with no preceding assistant message declaring its id.

Strict OpenAI-compatible upstreams reject it with:

400 invalid_request_error: Messages with role 'tool' must be a response to a preceding message with 'tool_calls'

The broken shape is written into the session history, so every later turn in that session fails identically until the model is switched to one whose images get stripped.

Environment

• Droid CLI 0.215.1 (droid --version), also observed via the desktop worker reporting 0.216.0
• macOS 24.6.0 (darwin), Apple Silicon
• Custom model: provider: "generic-chat-completion-api", noImageSupport: false
• First occurrence upstream: an OpenAI-compatible third-party gateway (https://opencode.ai/zen/go/v1, model deepseek-flash)

The bug is client-side, not upstream-specific: it reproduces against a local mock endpoint with no third party involved.

Steps to reproduce

1. Run a minimal OpenAI-compatible mock that answers the first turn with two parallel Read tool calls and logs every request body:

#!/usr/bin/env python3
"""Mock endpoint: first turn returns two parallel Read tool calls, then records what Droid sends."""
import json, time
from http.server import BaseHTTPRequestHandler, HTTPServer

def sse(p): return ("data: " + json.dumps(p) + "\n\n").encode()
def chunk(d, f=None):
    return {"id": "chatcmpl-mock", "object": "chat.completion.chunk", "created": 0,
            "model": "mock-model", "choices": [{"index": 0, "delta": d, "finish_reason": f}]}

class H(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"
    def log_message(self, *a): pass
    def do_POST(self):
        body = json.loads(self.rfile.read(int(self.headers.get("Content-Length", 0))))
        with open("/tmp/droid-repro/payloads.jsonl", "a") as f:
            f.write(json.dumps({"ts": time.time(), "body": body}) + "\n")
        turn1 = not any(m.get("role") == "tool" for m in body.get("messages", []))
        self.send_response(200)
        self.send_header("Content-Type", "text/event-stream")
        self.send_header("Transfer-Encoding", "chunked")
        self.end_headers()
        def send(b):
            self.wfile.write(("%X\r\n" % len(b)).encode() + b + b"\r\n"); self.wfile.flush()
        if turn1:
            calls = [{"index": i, "id": f"call_mock_{c}", "type": "function",
                      "function": {"name": "Read",
                                   "arguments": json.dumps({"file_path": f"/tmp/droid-repro/{c}.png"})}}
                     for i, c in enumerate(("a", "b"))]
            send(sse(chunk({"role": "assistant", "content": None, "tool_calls": calls[:1]})))
            send(sse(chunk({"tool_calls": calls[1:]})))
            send(sse(chunk({}, "tool_calls")))
        else:
            send(sse(chunk({"role": "assistant", "content": "done"})))
            send(sse(chunk({}, "stop")))
        send(b"data: [DONE]\n\n"); send(b""); self.wfile.write(b"0\r\n\r\n"); self.wfile.flush()

HTTPServer(("127.0.0.1", 8731), H).serve_forever()

2. Point a custom model at it (project-scope ./.factory/settings.json is enough):

{
  "customModels": [
    {
      "model": "mock-model",
      "id": "custom:mock-0",
      "index": 0,
      "baseUrl": "http://127.0.0.1:8731/v1",
      "apiKey": "dummy",
      "displayName": "Mock",
      "maxOutputTokens": 4096,
      "noImageSupport": false,
      "provider": "generic-chat-completion-api"
    }
  ]
}

3. Put two tiny PNGs (a.png, b.png) in the working directory and run:

droid exec --cwd "$PWD" --auto high -m custom:mock-0 \
  "Read both a.png and b.png in one message (two Read calls at once)."

4. Read the second recorded payload.

Evidence

Captured second request (system prompt, context reminders and tool definitions elided). Note the ordering and the missing call_mock_b in the assistant message:

[
  {"role": "user", "content": "hi"},
  {"role": "assistant", "content": null,
   "tool_calls": [{"id": "call_mock_a", "type": "function", "function": {"name": "Read", "...": "..."}}]},
  {"role": "tool", "tool_call_id": "call_mock_a",
   "content": "Image file: a.png (original size: 1.2 KB). Image quality: default"},
  {"role": "user", "content": [
      {"type": "text", "text": "Image content from tool result:"},
      {"type": "image_url", "image_url": {"url": "data:image/png;base64,...", "detail": "auto"}}]},
  {"role": "tool", "tool_call_id": "call_mock_b",
   "content": "Image file: b.png (original size: 1.2 KB). Image quality: default"},
  {"role": "user", "content": [
      {"type": "text", "text": "Image content from tool result:"},
      {"type": "image_url", "image_url": {"url": "data:image/png;base64,...", "detail": "auto"}}]}
]

Two independent defects are visible:

1. messages[3] (a user message) breaks the run of tool messages after the assistant turn.
2. call_mock_b is not declared by any assistant message even though Droid executed it and produced messages[4].

Live log line from a real session (droid 0.216.0 desktop worker), for the same shape:

WARN: [Chat route failure] | {"sessionId":"<redacted>","reason":"invalidRequest",
"error":{"message":"400 Error from provider (Console Go): Upstream request failed:
[invalid_request_error] Messages with role 'tool' must be a response to a preceding
message with 'tool_calls'"},"modelId":"deepseek-flash","apiProvider":"notYetSet",
"tags":{"modelId":"custom:...","isByok":"true","version":"0.216.0"}}

Trigger matrix verified against a live deepseek-flash endpoint (raw HTTP, no Droid involved):

Request shape	Result
1 tool call + trailing image user message	200
2 tool calls, image on the first result only	400
2 tool calls, image on the second result only	200
2 tool calls, all tool messages first, then all images	200
user message immediately followed by a tool message	400 (same error text)

So the failure requires an assistant turn with 2+ tool calls where a non-final tool result carries an image. With noImageSupport: true (images stripped) or text-only tool results, the ordering stays valid, which is why this only shows up for vision-capable BYOK models.

Root cause

Extracted from the shipped bundle, the generic-chat-completion-api conversion emits, per tool result, first the role:"tool" message and then a separate role:"user" message carrying the images:

if ($.role === "user" || $.role === "tool") {
  let I = [];
  for (let w of O) if (w.type === "tool_result") {
      let U = { role: "tool", content: /* text parts only */, tool_call_id: f(w.toolUseId) };
      if (h.push(U), R && n.supportsImages(R) && Array.isArray(w.content)) {
        let F = w.content.filter((e) => e.type === "image" && ... );
        if (F.length > 0) {
          let e = F.map((z) => ({ type: "image_url", image_url: { url: `data:...;base64,...`, detail: "auto" } }));
          h.push({ role: "user", content: [{ type: "text", text: "Image content from tool result:" }, ...e] });
        }
      }
  }
  ...
}

A later pass then trims each assistant message's tool_calls down to the ids found in the contiguous run of following tool messages:

function NWu(T) { for (let R = 0; R < T.length - 1; R++) {
    let H = T[R]; if (H.role !== "assistant" || !H.tool_calls) continue;
    let A = new Set;
    for (let t = R + 1; t < T.length; t++) { let _ = T[t]; if (_.role !== "tool") break; A.add(_.tool_call_id) }
    let h = H.tool_calls.filter((t) => A.has(t.id));
    if (h.length === H.tool_calls.length) continue;
    let n = H; if (h.length > 0) { n.tool_calls = h; continue }
    ...
} }

Because the inserted image user message interrupts that run, the second (and any later) tool call is deleted from the assistant message while its tool message stays in the array — producing an orphan role:"tool" message that no provider can accept.

Suggested fix

Emit image content only after the entire run of tool messages for the assistant turn, not after each individual result. For example, collect the image parts while iterating and append the user message(s) once the loop over that assistant turn's tool results is done. That produces tool, tool, user[images...], which is accepted (verified 200 above).

Optionally, harden the second pass so it cannot leave orphans: when it removes an id from an assistant's tool_calls, it should also drop the now-unmatched role:"tool" message.

Impact

• Any vision-capable BYOK model on generic-chat-completion-api breaks permanently as soon as an assistant turn parallel-reads two images (a common pattern: reading multiple screenshots/frames in one message).
• The failure is sticky: the invalid ordering is persisted in the session, so every subsequent turn re-sends it and fails, and retrying the prompt does not recover the session.
• Workarounds are lossy: set noImageSupport: true (loses vision) or avoid parallel image reads.

Related but distinct from #24 (that one reports images being stripped despite noImageSupport: false; this one is about message ordering when images are sent).