Permanent LLMExecutor wedge after CL_OUT_OF_RESOURCES on discrete GPU: servable stays AVAILABLE but every request returns 400 until restart (VLM legacy pipeline, public model)

[SyueYiLiao]

Describe the bug

Reproduced with the stock openvino/model_server:2026.2-gpu and 2026.3-gpu images and the public OpenVINO/Qwen3.5-9B-int8-ov export — no vendor components involved. Under a sustained agent-style workload, the VLM legacy pipeline enters a permanent failure state on a discrete Intel GPU. One request triggers CL_OUT_OF_RESOURCES inside pipe->generate(); from that point on every subsequent request — including a 60-token one with no tools — fails immediately with

400 {"code":"ovms_400","message":"Mediapipe execution failed. MP status - INVALID_ARGUMENT: CalculatorGraph::Run() failed:
Calculator::Process() for node \"LLMExecutor\" failed: Request processing failed, check its correctness.","type":"upstream_error"}

The process stays alive, /v1/config still reports the servable AVAILABLE, and the llm_executor heartbeat ([legacy_executor.cpp:76] All requests: N;) goes silent. Only restarting the server recovers it.

Two things look wrong on the OVMS side, independent of whatever the driver is doing:

1. A context-fatal GPU error is not treated as fatal. After CL_OUT_OF_RESOURCES the OCL context/queue is unusable, but the servable is neither marked unavailable nor rebuilt, so it serves errors indefinitely while advertising itself as ready. A load balancer or orchestrator has no way to route around it.
2. The error is reported as a client error. INVALID_ARGUMENT / HTTP 400 with "check its correctness" points the user at their own request, while the actual cause is a server-side unrecoverable GPU failure. HTTP 503 (or at least a distinct code) would be correct.

To Reproduce

1. Models repository — the public export, unmodified:

huggingface-cli download OpenVINO/Qwen3.5-9B-int8-ov --local-dir ./repro/ir

./repro/config.json:

{ "model_config_list": [],
  "mediapipe_config_list": [ { "name": "qwen3.5-9b-public", "base_path": "/models/graph" } ] }

./repro/graph/graph.pbtxt (the parts that matter; LLMExecutor node wiring is the standard one):

node_options: {
  [type.googleapis.com / mediapipe.LLMCalculatorOptions]: {
    pipeline_type: VLM,
    tool_parser: "qwen3coder",
    reasoning_parser: "qwen3",
    enable_tool_guided_generation: true,
    models_path: "/models/ir",
    plugin_config: '{"CACHE_DIR":"/opt/cache"}',
    enable_prefix_caching: true,
    cache_size: 2,
    max_num_seqs: 256,
    device: "GPU.1",
  }
}

2. OVMS launch command

docker run -d --name ovms-pub --device /dev/dri --network host \
  -v $PWD/repro:/models -v $PWD/repro/cache:/opt/cache \
  openvino/model_server:2026.2-gpu \
  --config_path /models/config.json --rest_port 8123 \
  --cache_dir /opt/cache --file_system_poll_wait_seconds 1 --log_level DEBUG

3. Client — the attached four JSON bodies posted sequentially to /v3/chat/completions, one connection at a time, no think time, in this cycle:

step	prompt tokens	tools	note
tiny	148	0	short "generate a conversation title" request
big01	~12.0k	46	tool-calling turn, one tool round trip in history
big02	~12.6k	47	two round trips
big03	~14.3k	47	three round trips, last tool result is a ~2.7 KB structured dump

ENDPOINT=http://127.0.0.1:8123/v3/chat/completions ./wedge-loop.sh 150

All four bodies carry stream: true, stream_options.include_usage: true and chat_template_kwargs: {"enable_thinking": false}.

4. See error — the loop stops at the first non-200.

With the public export on the stock image (the configuration above): wedged after 25 requests, on the tiny step.

With the same payloads against our own export of the same base model (9 runs total, fresh server process each): wedged after 13, 21, 21, 25, 29, 37, 57, 65 and 89 requests, median 29. Every one of the nine failed on the tiny step — the small tool-less request that immediately follows the large tool-calling turns.

After the failure, the server keeps reporting itself ready:

$ curl -s http://127.0.0.1:8123/v1/config
{"qwen3.5-9b-public":{"model_version_status":[{"version":"1","state":"AVAILABLE","status":{"error_code":"OK","error_message":"OK"}}]}}

$ curl -s -d '{"model":"qwen3.5-9b-public","max_tokens":8,"messages":[{"role":"user","content":"1+1?"}]}' \
       -H 'Content-Type: application/json' http://127.0.0.1:8123/v3/chat/completions
{"error":"Mediapipe execution failed. MP status - INVALID_ARGUMENT: CalculatorGraph::Run() failed: \nCalculator::Process() for node \"LLMExecutor\" failed: Request processing failed, check its correctness."}

Expected behavior

• A GPU error that invalidates the inference context should mark the servable unavailable and rebuild the pipeline (or terminate the process so the orchestrator can restart it) instead of serving INVALID_ARGUMENT forever while reporting AVAILABLE.
• The health endpoints should reflect it. In the wedged state we measured, every discovery and health endpoint still answers 200 — /v3/models, /v1/config ("state":"AVAILABLE", "error_code":"OK"), /v2/health/ready, /v2/health/live and /v2/models/<name>/ready — while an 8-token inference request returns 400. A readiness probe that returns 200 for a servable that cannot serve anything makes every orchestrator, load balancer and monitor blind to the failure; today the only way to detect it is to send a real inference request.
• The response for a server-side fatal error should not be a 4xx that blames the request.
• Ideally, the workload above should not provoke the GPU error in the first place.

Logs

Server, at the moment of the first failure (--log_level DEBUG):

[llm_executor][error][legacy_executor.cpp:46] VLM pipeline generation failed: Exception from src/inference/src/cpp/infer_request.cpp:224:
[GPU] clFinish, error code: -5 CL_OUT_OF_RESOURCES
[GPU] CL_OUT_OF_RESOURCES exception.
        The CL_OUT_OF_RESOURCES error typically occurs in two cases:
        ...

Immediately after, the executor heartbeat stops:

[llm_executor][info][legacy_executor.cpp:76] All requests: 1
[llm_executor][info][legacy_executor.cpp:76] All requests: 0     <-- last line, then silence

Every later request (including {"messages":[{"role":"user","content":"1+1?"}],"max_tokens":8}) returns the 400 quoted above, with no further ERROR lines.

Kernel side, same instant:

xe 0000:05:00.0: [drm] Tile0: GT0: Engine reset: engine_class=ccs, logical_mask: 0x1, guc_id=2
xe 0000:05:00.0: [drm] Xe device coredump has been created
xe 0000:05:00.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data

Configuration

1. OVMS version: reproduced on both unmodified upstream images (heartbeat line legacy_executor.cpp:76 in both):

   • openvino/model_server:2026.2-gpu — wedged after 25 requests
   • openvino/model_server:2026.3-gpu (sha256:2a52cd2b…, OpenVINO backend 2026.3.0-22451-8a17657b995-releases/2026/3) — 11 runs, 11 wedges, after 5–13 requests (9,9,9,5,9,5,13,9,9,9,13), same model, same graph, same payloads, identical prompt-token counts
     The exception text is identical apart from the OCL source line (ocl_stream.cpp:395 on 2026.3). On 2026.3 the failure is close to deterministic — the three observed values are exactly the boundary request of cycles 2, 3 and 4 — which makes it the configuration we suggest starting from. We are not claiming 2026.3 is worse than 2026.2: we only have one 2026.2 run with the public export, and the two were not interleaved.

2. config.json: single mediapipe_config_list entry pointing at the graph above.

3. Accelerator and driver stack. The workload runs on GPU.1; an integrated GPU is present as GPU.0 and stays idle.

   Host:

   GPU	Intel discrete, PCI 8086:E223 rev 0 (subsystem 8086:1701), Battlemage class, 32.5 GB, arch=v20.2.0, 256 EUs
   Kernel	Linux 6.6.129 x86_64
   DRM driver	xe (not i915)
   GuC firmware	xe/bmg_guc_70.bin, release version 70.58.0
   HuC firmware	not loaded (status: N/A) — this workload does not use it

   Inside the OVMS container (the image ships its own compute runtime):

   package	version
   intel-opencl-icd	26.18.38308.1-0
   libze-intel-gpu1	26.18.38308.1-0 (libze_intel_gpu.so.1.15.38308)
   intel-ocloc	26.18.38308.1-0
   intel-igc-core-2 / intel-igc-opencl-2	2.34.4
   libze1 (Level Zero loader)	1.27.0-124.04ppa2

4. Model repository: /models/<model>/{ir/,g_gpu.1/graph.pbtxt} plus config.json.

5. Model: OpenVINO/Qwen3.5-9B-int8-ov (public, Apache-2.0), downloaded unmodified from the Hugging Face hub. The failure was first seen with our own export of the same base model and then reproduced with this public one, so nothing vendor-specific is required to hit it.

Additional context

What we ruled out (each experiment is a full run of the cycle above with one variable changed; "no wedge" means the run completed without a single failure):

change	result
cache_size 2 → 8 GB	still wedges (after 101 and 245 requests) — not a capacity problem
remove the tiny step, keep only the tool-calling turns	180 requests, no wedge
remove the tool-calling turns, alternate tiny with a 16.8k-token tool-less request	240 requests, no wedge
give the boundary request 3 small tools (659 tokens)	240 requests, no wedge
keep the boundary request tool-less but enlarge it to 1.9k tokens	240 requests, no wedge
boundary request the same size but sharing the tool turns' prefix	240 requests, no wedge

So the trigger needs both halves: large tool-schema-heavy prefills alternating with a very small tool-less prefill. Enlarging the KV cache does not remove it.

One more observation that may narrow it down: we built a second payload set with the same token counts, same tool counts and same request structure, but whose text is prose instead of machine data (the real payloads are API schemas plus structured result dumps — short indented lines, opaque ids, timestamps; roughly 10 % punctuation and 13 % digits in the tool results). The prose variant ran 1,440 requests with no wedge; the data-shaped variant wedges in tens of requests. Pushing punctuation density well above the original (18–23 %) also stopped reproducing (600 requests each, twice). That suggests the trigger depends on the shape of the prefill, not only on its length or on the number of tools.

Timing note for anyone reproducing: on our host the public export is slow and erratic per request on both images (big requests 4–40 s, versus ~3–4 s for our own export of the same base model), with a cold plugin cache. This changes how long a run takes but not the outcome.

One more thing you may hit while reproducing: with these payloads (no max_tokens, tool-guided generation on, 46–47 tools) the model occasionally enters a degenerate loop, streaming function_calls tokens for tens of minutes. That is not the wedge — the heartbeat still shows All requests: 1, there is no exception and no engine reset. Our driver caps each request at 300 s to keep the loop moving.

Impact: a single occurrence takes the servable out of service permanently while it continues to advertise itself as ready; recovery requires a restart plus model reload (~3 minutes for us). In production we observed six occurrences, each after 12–129 requests.

Notes: the wedge itself may well be a driver/firmware issue (the engine reset points that way), and we are raising that separately. What we would like from OVMS regardless is the handling: detect CL_OUT_OF_RESOURCES as context-fatal, stop reporting the servable as AVAILABLE, and return a 5xx rather than a 400 that blames the caller.

mock-tiny.json
mock-big01.json
mock-big02.json
mock-big03.json

wedge-loop.sh

[snailium]

Adding a data point: this also reproduces on the newest dev image and with a larger model, so it is not fixed in the current weekly either.

Config (all stock, no vendor components):

• Image: openvino/model_server:weekly (dev 2026.4), sha256:de1d2eed43e2a7c6c8e4802788e13c6c0a47371fe4984f9cce98d0d47c88af41, created 2026-08-21T17:33+02:00. OpenVINO 2026.4.0 + GenAI 2026.4.0.0.
• Model: OpenVINO/Qwen3.8-27B-int4-ov (the 27B variant, larger than the 9B used here). VLM legacy servable via --model_path ... --task text_generation --pipeline_type VLM --cache_size 8.
• Host GPU: same discrete Intel Battlemage 8086:E223 (32 GB), Linux xe driver.

Repro: a sustained multi-turn agent workload (dsh) with deep context and tool-calling turns in history, requests issued sequentially. After a burst of large tool-schema-heavy prefills, a request fails with the exact same trace:

[llm_executor][error][legacy_executor.cpp:46] VLM pipeline generation failed: Exception from src/inference/src/cpp/infer_request.cpp:224:
Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_stream.cpp:395:
[GPU] clFinish, error code: -5 CL_OUT_OF_RESOURCES

...followed by the identical wedge: the servable keeps reporting AVAILABLE, but every later request (down to a trivial 8-token one) returns HTTP 400 Mediapipe execution failed. ... Calculator::Process() for node "LLMExecutor" failed: Request processing failed, check its correctness. until a server restart.

Additional data points beyond the report:

• Not fixed in the latest weekly (2026.4 dev) — only GA 2026.2 / 2026.3 were covered here.
• Reproduces with the 27B model, confirming it is not specific to the 9B export.

One practical note for anyone reproducing: the kernel-side xe device coredump (/sys/class/drm/card1/device/devcoredump/data) auto-deletes after roughly an hour, so snapshot it immediately after a recurrence if you want the full dump.

Fully agreeing with the OVMS-side handling you propose — treat CL_OUT_OF_RESOURCES as context-fatal (mark servable unavailable / rebuild or restart), and return a 5xx instead of a 400 that blames the caller. Happy to share logs or a captured coredump on request.

[snailium]

Follow-up with kernel-level evidence, plus a more concerning regression in the failure mode.

Captured the xe device coredump (this occurrence created a kernel coredump; earlier occurrences did not):

**** Xe Device Coredump ****
Reason: Timedout job - seqno=540, lrc_seqno=540, guc_id=2, flags=0x20
Process: ovms [464054]
PCI ID: 0xe223 (Arc Pro B70, Battlemage), GT0, IP ver 20.2.0
VM state: [0].error: -19 (ENOENT)

preceded by 11 consecutive:

xe 0000:03:00.0: [drm] Tile0: GT0: Fault response: Unsuccessful -ENOENT
xe 0000:03:00.0: [drm] Xe device coredump has been created

So in this case the userspace clFinish CL_OUT_OF_RESOURCES is the tail of a kernel-level GPU page fault (-ENOENT) + job timeout, not an OVMS logic error. The full coredump (504 KB) is available on request.

---

Update — deterministic trigger isolated, and a correction to my restart note.

With --log_level DEBUG and a packet capture on :8000, the exact trigger is isolated: under a sustained multi-turn agent workload, the failure recurs specifically on the agent's context-compaction / checkpoint request — the agent condenses a long conversation into one large message plus a long system prompt, producing a ~290 KB, tens-of-thousands-of-token prefill, after which GPU generation faults ~5 s later:

15:08:44  POST /v1/chat/completions        content-length: 291691
          messages: [huge system prompt, "automatically generated CHECKPOINT condensing ... free up context ... Continue the task"]
15:08:44  InputProcessor Tokenization      took 402 ms
15:08:49  [llm_executor] VLM pipeline generation failed ... [GPU] clFinish, error code: -5 CL_OUT_OF_RESOURCES

So on B70 the practical trigger is a single huge prefill at deep (near-max) context — the compaction request itself.

Correction to the restart claim above: on this occurrence a fresh OVMS restart did recover it (a plain request passed immediately after the restart). The failure then recurred deterministically on the very next compaction/deep-context request. So it is recurring from the same trigger, not a permanent post-restart wedge — each context compaction reliably knocks the servable over again.

Sanitized evidence (coredump reason, dmesg faults, trigger shape; no agent conversation payload or auth material) — gist: https://gist.github.com/snailium/72c510a4a4ab5e166b063a4bb8c52eef

Full artifacts (504 KB xe coredump, packet capture, OVMS DEBUG logs) are retained locally and available on request.