Gemma4 native tool calling: parser, lazy generation and chat-template hardening

[DassaultFalconKing]

🛠 Summary

JIRA/Issue if applicable.
Describe the changes.

🧪 Checklist

• Unit tests added.
• The documentation updated.
• Change follows security best practices.
  ``## Summary

This PR hardens native Gemma4 tool calling in OVMS across three layers that must work together:

1. tool output parsing
2. tool generation constraints
3. chat-template compatibility for multi-turn tool conversations

The work was driven by live Gemma4 failures observed with OpenAI-compatible agent clients and then cross-checked against the native Gemma4 protocol, existing OVMS/OpenVINO structured-generation primitives, vLLM parsing behavior, and llama.cpp lazy tool activation semantics.

The implementation remains native to OVMS/OpenVINO.

No vLLM or llama.cpp parser/generator implementation code is transplanted into OVMS.

---

Motivation

The existing Gemma4 path could fail in several distinct ways that looked similar from an agent client's perspective:

• the model emitted a valid native Gemma4 tool call that the parser failed to reconstruct;
• streaming boundaries split native syntax in ways the parser did not preserve correctly;
• nested Gemma4 arguments were flattened or incompletely normalized;
• large numeric tool arguments could lose lexical precision;
• tool_choice=required or a named tool could silently lose its generation constraint after structured-output validation failure;
• tool_choice=auto was either effectively unguided or risked becoming equivalent to required;
• multi-turn tool history could become incompatible with the canonical Gemma4 Jinja template;
• malformed or stray call: text could be over-promoted into executable tool calls.

These are separate Parser, Generator, and input-template problems. Fixing only the parser is insufficient if generation never emits the tool structure in the first place, and fixing only generation is insufficient if the emitted structure cannot be recovered safely.

A parser cannot recover a tool call that the Generator never caused the model to emit.

---

1. Gemma4 parser hardening

The Gemma4 tool parser now understands the native protocol recursively rather than treating tool arguments as a mostly flat serialization problem.

The parser handles:

• native <|tool_call>call:name...<tool_call|> boundaries;
• Gemma4's native string delimiter <|"|>;
• nested objects;
• nested arrays;
• JSON strings;
• booleans;
• null;
• numeric values;
• parenthesized and braced argument containers;
• streaming prefixes and incomplete chunks.

Lossless numeric handling

Tool arguments are API payload text, not values that OVMS needs to perform arithmetic on.

Large integers and precise decimals therefore should not be unnecessarily routed through uint64_t or double during normalization.

The parser preserves numeric lexical representation while converting native Gemma4 argument syntax into JSON.

This avoids avoidable precision loss in values such as identifiers, counters, hashes represented numerically, or high-precision decimal fields.

Bounded malformed-call handling

Malformed tool syntax is kept bounded to the current candidate instead of poisoning the remainder of the stream.

A later valid tool call can still be recovered after a malformed one.

Request tool registry validation

When request tool schemas are available, executable tool calls are checked against the request's actual tool registry.

Unknown tools are not promoted into executable calls.

This is particularly important for recovery paths.

Guarded bare call: recovery

Live Gemma4 traces showed a narrow variant where the model may close a reasoning channel and continue directly with:

call:tool_name{...}

without repeating <|tool_call>.

The parser can recover this observed shape, but only under bounded conditions:

• call: must begin at a logical line/phase boundary;
• optional indentation is allowed;
• the tool name must be syntactically sane;
• when a request registry is available, the tool must exist in that registry;
• the arguments still pass the normal structural parser.

Arbitrary prose such as:

Documentation example: call:foo{...}

is deliberately not treated as an executable call.

This recovery is intentionally narrow rather than a general search for call: substrings.

---

2. Parser provenance: vLLM as an independent behavioral reference

While hardening the parser, the Gemma4 implementation in vLLM was used as an independent protocol reference.

The useful comparison points were:

• native Gemma4 tool-call framing;
• the <|"|> string delimiter;
• recursive object/array argument structure;
• distinction between structural delimiters and delimiters occurring inside strings;
• interaction between reasoning and tool-call channels.

This is not a source-code port of the vLLM parser.

The useful part carried into OVMS is the protocol-aware parsing model and its edge cases, implemented inside OVMS's existing parser interfaces and streaming response machinery.

---

3. Explicit Gemma4 Generator policy

This PR also adds an explicit Gemma4 tool-constraint policy to generation.

The implementation separates tool behavior into:

Disabled
Auto
Hard

Disabled

Used when:

• there are no tools; or
• tool_choice=none.

No tool grammar is imposed.

Hard

Used for:

• tool_choice=required;
• named tool choices.

Hard choices are represented using native OpenVINO GenAI structural tags.

They:

• require at least one tool call;
• restrict named choices to the requested tool;
• respect parallel_tool_calls;
• remain fail-closed if grammar validation fails.

A hard OpenAI API contract must not silently turn into unconstrained generation because structured-output validation failed.

Optional reasoning before a hard tool call

Gemma4 may emit a reasoning channel before selecting a required or named tool.

The hard grammar therefore accepts either:

tool call(s)

or:

thought channel
followed by
tool call(s)

The optional path is represented as a Union, rather than using an empty ConstString.

This matters because xgrammar rejects empty ConstString nodes.

---

4. Lazy tool_choice=auto

auto must not mean required.

The model must remain able to answer with normal text when no tool is necessary, while any tool call it does initiate must conform to the available tool registry and JSON schema.

Gemma4 auto therefore uses OpenVINO GenAI TriggeredTags:

normal generation
        |
        | model emits <|tool_call>
        v
structured tool grammar activates

The trigger is the native Gemma4 tool marker:

<|tool_call>

Once triggered, the generated call is constrained to the request's available tool names and schemas.

Before the trigger, normal assistant text remains legal.

llama.cpp provenance

llama.cpp was used as an independent behavioral reference for this lazy/optional activation model.

Only the semantics were ported.

OVMS does not use llama.cpp's PEG grammar, sampler, or parser implementation.

The implementation here uses the existing native OpenVINO GenAI structured-output abstraction.

A concise description is:

behavioral port, native OVMS implementation.

---

5. Why TriggeredTags fits OVMS architecture

This PR does not introduce an external generation architecture into OVMS.

The layering is:

Gemma4-specific tool policy
        |
        v
OVMS GenerationConfigBuilder
        |
        v
OpenVINO GenAI StructuredOutputConfig / TriggeredTags
        |
        v
xgrammar structural-tag enforcement

TriggeredTags already exists in OpenVINO GenAI and is already used by OVMS for other model-specific generation behavior.

This patch applies that existing abstraction to Gemma4's native protocol.

---

6. parallel_tool_calls

Both lazy and hard modes respect the OpenAI parallel_tool_calls request field.

When parallel tool calls are disabled:

stop_after_first = true

When they are enabled, subsequent compatible tool calls remain legal.

This behavior is covered for:

• auto;
• required;
• named tool choices.

---

7. Tool schema and name validation

The Generator rejects active Gemma4 tool configurations that cannot be represented safely.

Examples include:

• hard choices with no tool schemas;
• named choices referring to unavailable tools;
• empty tool schemas;
• unsupported tool-name characters;
• conflicting response_format and active tool generation constraints.

The same tool-name shape accepted by the Generator is compatible with what the parser can safely recognize and execute.

---

8. Google Gemma4 chat-template compatibility

The work was also tested against the canonical Google Gemma4 tool protocol and chat-template behavior.

A multi-turn incompatibility was found in OVMS input adaptation.

OVMS may parse JSON text from a role: "tool" message into an object before rendering the chat template.

That is not universally safe.

The canonical Gemma4 template includes a path that iterates message content parts and calls:

part.get('type')

If a JSON object has already replaced the original content string, Jinja iterates the object's keys, producing strings rather than part objects.

The resulting failure is equivalent to:

'str object' has no attribute 'get'

Capability-driven fix

The chat-template analyzer now distinguishes between templates where converting tool-response JSON into an object is safe and templates where a parts scan makes that transformation unsafe.

A capability controls the history adaptation.

JSON tool-response content is converted to an object only where the detected template semantics support it.

This preserves the existing upstream tool-definition adaptation behavior as well.

---

9. Template adaptation is conservative

The tool-response conversion itself is intentionally narrow.

Only:

• messages with role == "tool";
• whose content is a string;
• whose string parses as a JSON object

are eligible.

Arrays, scalars, invalid JSON, user messages, and assistant messages retain their original string semantics.

This avoids changing general OpenAI message behavior merely because content happens to look like JSON.

---

10. Regression coverage

The PR adds contract tests covering the failure modes that motivated the work.

Generation contracts

Coverage includes:

• no-tools and tool_choice=none;
• response_format preservation when tools are inactive;
• Gemma4-specific validation fallback policy;
• mandatory tool selection after optional reasoning;
• hard choices remaining fail-closed;
• lazy auto using TriggeredTags;
• auto remaining optional;
• named tool restriction;
• parallel_tool_calls;
• validation of lazy grammar using a Gemma4 tokenizer;
• nested schema preservation;
• invalid tool names;
• empty schemas;
• invalid hard choices;
• avoidance of empty ConstString.

Parser contracts

Coverage includes:

• recursive arrays of objects;
• nested scalar types;
• native strings;
• parenthesized arguments;
• colon/name variants;
• direct calls after reasoning;
• guarded bare-call recovery;
• rejection of unknown tools;
• rejection of embedded or quoted call: examples;
• malformed-call bounding;
• recovery of a later valid tool call.

Chat-template contracts

Coverage includes:

• Gemma4 template detection;
• disabling tool-response JSON conversion when the template performs a part.get(...) scan;
• retaining conversion for mapping-safe templates;
• preserving opaque values such as commit SHAs during JSON conversion;
• leaving arrays and non-JSON strings unchanged;
• leaving non-tool messages untouched;
• preserving the existing upstream tool-definition response adaptation.

---

11. Real-world validation background

This patch grew out of live Gemma4 deployment work using OVMS on Windows with Intel Arc hardware and OpenAI-compatible agent clients.

The local integration environment also includes:

• explicit tool_parser: gemma4;
• explicit reasoning_parser: gemma4;
• guided generation enabled;
• long-context profiles;
• correctness-gated performance tuning;
• Google-template provenance tracking;
• multi-turn agent/tool-loop tests.

An earlier exact-source runtime baseline in that environment produced:

100 / 100 tool-call executions

through the OVMS REST path.

That result is useful supporting evidence for the architecture, but it is not claimed as exact-head validation of this PR, because additional parser, lazy-generator, and template-compatibility changes were made afterward.

The authoritative validation for this PR head should be the upstream CI results and any exact-head runtime testing performed during review.

---

12. External implementation references and code provenance

The implementation was informed by several independent sources, each for a different purpose:

Google Gemma4

Authority for:

• native tool-call syntax;
• tool-response syntax;
• reasoning/tool channel behavior;
• chat serialization semantics.

vLLM

Independent behavioral reference for:

• protocol-aware Gemma4 parsing;
• recursive native argument handling;
• reasoning/tool parsing boundaries.

No vLLM parser code is copied into this patch.

llama.cpp

Independent behavioral reference for:

• optional/lazy tool activation under auto.

No llama.cpp grammar, sampler, or parser code is copied into this patch.

OpenVINO GenAI / OVMS

Native implementation mechanism:

• StructuredOutputConfig;
• TriggeredTags;
• structural tag grammar;
• xgrammar-backed validation and enforcement.

---

13. What this PR intentionally does not include

The development fork contains additional deployment and diagnostics work used to validate Gemma4 on Windows and Intel Arc hardware.

That includes launch profiles, acceptance harnesses, long-context experiments, and local chat-template deployment helpers.

Those are intentionally not included in this upstream core PR.

This PR is limited to the reusable OVMS runtime changes:

• Gemma4 parser correctness;
• Gemma4 generation policy;
• lazy auto constraints;
• hard tool-choice enforcement;
• multi-turn chat-template compatibility;
• focused regression tests.

---

Current PR coordinates

Upstream base:

openvinotoolkit/model_server
main
b935fe8b96a0445f3746297f872b55ed202fa6e5

Proposed head:

DassaultFalconKing/model_server
fix/gemma4-native-tool-calling-upstream-v2
b45d2aa41046089d98861266eb07f276df752035

The branch is based directly on the upstream commit above and contains only the Gemma4 runtime and contract-test scope described in this PR.

[mzegla]

Style check fails:

[2026-09-08T22:25:22.761Z] ## No header files detected:

[2026-09-08T22:25:22.761Z] ./src/test/llm/generation_config/BUILD

[2026-09-08T22:25:22.761Z] make: *** [Makefile:300: license-headers] Error 1

Also I tried to build it, but it does not compile:

ERROR: /ovms/src/llm/BUILD:355:16: Compiling src/llm/io_processing/gemma4/gemma4_tool_parser.cpp failed: (Exit 1): gcc failed: error executing command (from target //src/llm:io_processing_gemma4_tool_parser) /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 106 arguments skipped)
In file included from src/llm/io_processing/gemma4/gemma4_tool_parser.cpp:11:
src/llm/io_processing/gemma4/gemma4_tool_parser.hpp: In static member function 'static ovms::OutputParsingConfig ovms::Gemma4ToolParser::defaultParsingConfig()':
src/llm/io_processing/gemma4/gemma4_tool_parser.hpp:58:13: error: 'struct ovms::OutputParsingConfig' has no member named 'ownsToolCallBoundaries'
   58 |         cfg.ownsToolCallBoundaries = true;
      |             ^~~~~~~~~~~~~~~~~~~~~~
Target //src:ovms failed to build

[DassaultFalconKing]

@mzegla saw the Compiling error yesterday, in the night.

The problem is that this fork/PR is based on the version 2026.4, the release is the ver. 2026.3.1, and the upstream is the ver. 2026.5. My mistake was the very bad versioning work.

after the successful testing of the 2026.4 based variant i even lost the reproducibly good binary and had to do the archaeology to recover it, while porting the changes to the Version 2026.5 core.

As a result, i got the working .5 binary and recovered the 2026.4 but didnt prove the workability yet.
the compile fix is now landing.