Skip to content

[Other] A tool call written as text is shown as not run - #4447

Closed
xvcezrqpbxtpzblrt wants to merge 1 commit into
Niko1221:mainfrom
xvcezrqpbxtpzblrt:agent/text-call-027
Closed

xvcezrqpbxtpzblrt wants to merge 1 commit into
Niko1221:mainfrom
xvcezrqpbxtpzblrt:agent/text-call-027

Conversation

@xvcezrqpbxtpzblrt

Copy link
Copy Markdown
Contributor

Problem

A model can put a tool call in the text channel instead of calling the tool. A real session (session-82188d88, 4 turns, 373 steps) has this at seq 1328:

{"type":"assistant/message","seq":1328,"data":{"message":{"role":"assistant",
 "content":[{"type":"reasoning","text":"Now compile..."},
            {"type":"text","text":"<tool_call>\n<function=compile>\n<parameter=command>\ncd \"D:\\下载器\\gui\"; python ..."}]}}}

Then step/end and urn/end follow: the command never ran and the turn was over. The chat showed the raw XML as the answer, so nothing told the person that the work was not done.

The same session has 3,772 ordinary ool/call events, so this is the model's slip, not a broken parser.

Result

  • A call written as text is taken out of the answer and shown as an ordinary tool card whose verb is "Not run — written as text", with the command and the tool's own name in it. The raw <tool_call> never reaches the chat, whether it arrives as one chunk or split across many.
  • A block that is not a well-formed call (for example an answer quoting the format) is left as text.
  • The turn ends with one plain sentence: it did not run, and saying "continue" sends it as a real call. That sentence is in all nine catalogs, as are the card's verb, effect and result.
  • The app's standing prompt gains one rule: call a tool, never write the call as text.

Validation

  • node --test tests/agent-events.test.mjs → 18 pass (3 new: whole call, split across chunks plus a quoted block, a half-written tag at the end of a message).
  • node --test tests/agent-turn-guard.test.mjs with the installed Harness, against a scripted model that answers with the same text → 10 pass, including the new end-to-end case: no raw XML in the deltas, one card with args.command = "python -m py_compile a.py", the note is the last delta, the turn ends normally.
  • node --test tests/languages-and-catalog.test.mjs tests/french-catalog.test.mjs tests/locale-gaps.test.mjs tests/functional-fixes.test.mjs → pass; the catalogs have the same 3,532 keys in every language and French keeps its typography.
  • Full host suite: 1,292 tests, 1,182 pass, 2 fail (the known Windows symlink EPERM cases, unchanged), 108 skipped.

Limits

  • The prompt rule is wording; the model can still ignore it. What does not depend on the model is the display and the sentence at the end of the turn.
  • The app does not run a call that arrives as text. Executing text the model wrote would bypass the approval card and the sandbox's own decision, so it is shown and the person is told.
  • The test ceiling for the prompt moves from 1,600 to 2,600 characters (1,688 now). Another open change that also raises it should take the higher number when the two meet.

@xvcezrqpbxtpzblrt

Copy link
Copy Markdown
Contributor Author

Superseded by #4606, which carries this change with the rest of the 0.2.7 branch. Closing so the review has one place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant