fix(python-backends): re-attach media markers under use_tokenizer_template (#11621)

[Anai-Guo]

Description

Fixes #11621.

With template.use_tokenizer_template: true, an image_url content part is silently ignored on the sglang and vllm backends — HTTP 200, nothing in the log, and the model answers as if no image had been attached.

The pixels are forwarded correctly. What is missing is the media placeholder in the prompt:

• _build_prompt() (sglang) / the templating block in _predict() (vllm) pass string content to tokenizer.apply_chat_template().
• A chat template only emits the model's own media tokens — <|vision_start|><|image_pad|><|vision_end|> for the Qwen-VL family — when the message content is a list of parts.
• Both engines locate the attached media by scanning the rendered prompt for that token (sglang/srt/multimodal/processors/qwen_vl.py; vLLM substitutes multi_modal_data into it). With no placeholder present, the media are dropped without a word.

This is option (a) from the issue: backend-local, no protocol change.

Change

• backend/python/common/python_utils.py — new attach_media_parts(messages_dicts, n_images, n_videos). It rebuilds the last user turn as [{"type": "image"} * n, {"type": "video"} * n, {"type": "text", "text": ...}] and returns a new list, or None when there is nothing to attach (no media, no user turn, content already a list of parts). It never mutates its input.
• backend/python/sglang/backend.py, backend/python/vllm/backend.py — call it just before templating. The pixels keep travelling out of band via image_data / multi_modal_data exactly as before.
• backend/python/common/python_utils_test.py — 7 new cases for the helper.

Backwards compatibility

• Text-only requests are untouched. With no images and no videos the helper returns None on its first line and the existing string-content path runs verbatim.
• Text-only templates still work. If a template cannot iterate content parts, the parts render is caught and the request falls back to the previous string-content prompt (with a note on stderr) instead of failing. The pre-existing except TypeError retry is preserved underneath.

Verification

Helper unit tests (stdlib only, no backend venv — the target the Makefile already runs):

$ cd backend/python/common && python3 -m unittest python_utils_test
..............
Ran 14 tests in 0.000s
OK

End-to-end on the real code path, without loading a model: the _messages_to_dicts / _build_prompt methods are lifted out of the actual backend.py files with ast and driven with a tokenizer stub that renders the real chat_template.json of Qwen/Qwen2-VL-7B-Instruct through jinja2.

request	before	after
1 image	0 <|vision_start|><|image_pad|><|vision_end|>	1
2 images	0	2
1 video	0 <|vision_start|><|video_pad|><|vision_end|>	1
text only	<|im_start|>user\nWie hoch steht das Wasser?<|im_end|>	identical
text-only template + 1 image	renders, no crash	renders, no crash (falls back)

Rendered user turn after the change, 1 image:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
<|vision_start|><|image_pad|><|vision_end|>Wie hoch steht das Wasser?<|im_end|>
<|im_start|>assistant

Both backends produce the same before/after table.

Scope

Option (b) from the issue — carrying media per message in backend.proto — is deliberately not attempted here. That is the only way to get images in different turns of a multi-turn conversation right, and it belongs in its own protocol-level change. This PR covers every single-image and last-turn request, which is effectively all real vision traffic today.

Notes

• The branch is based on an older master commit rather than the current tip — the token used here has no workflow scope, so GitHub refuses to sync the fork. The diff is nonetheless exactly the four files above, and it merges into the current master cleanly (GitHub reports the PR as mergeable). I re-ran the before/after table on the auto-merged trees of both backends to confirm the merge is semantically right, not just conflict-free.

🤖 Generated with Claude Code

[localai-org-maint-bot]

Good to merge. The shared helper reconstructs the tokenizer content parts without mutating the original messages, both Python backends use it only for media-bearing tokenizer-template requests, and text-only/template-incompatible paths preserve the prior fallback. I verified the focused common helper suite (14 tests), Python compilation for the helper plus both backends, and diff hygiene; DCO is green. @mudler

[Anai-Guo]

Friendly ping @mudler — this one is approved and still MERGEABLE. Is there anything else needed before it can go in? Happy to rebase if you'd prefer.