From 943555f24b348a9aaa9ccdd4227a5894d4317f8f Mon Sep 17 00:00:00 2001
From: Trawinski <dariusz.trawinski@intel.com>
Date: Wed, 2 Sep 2026 01:13:14 +0200
Subject: [PATCH 01/23] mtp demo

---
 .../speculative_decoding/README.md            | 265 +++++++++---------
 1 file changed, 126 insertions(+), 139 deletions(-)

diff --git a/demos/continuous_batching/speculative_decoding/README.md b/demos/continuous_batching/speculative_decoding/README.md
index 92810bf5fc..b58b78b3ab 100644
--- a/demos/continuous_batching/speculative_decoding/README.md
+++ b/demos/continuous_batching/speculative_decoding/README.md
@@ -6,17 +6,110 @@ OpenVINO GenAI implements three drafting strategies, all exposed through the sam
 
 | Strategy | How it drafts | Best for | Extra model required |
 |---|---|---|---|
+| **MTP** | Built-in multi-token prediction head | Models with bundled MTP heads (e.g. Qwen3.8-27B) | No — head bundled with the main model |
 | **Fast Draft** | Small off-the-shelf LLM | General-purpose; any target/draft pair | Yes — smaller LLM sharing target's tokenizer |
 | **EAGLE3** | Draft head conditioned on target's hidden states | Highest acceptance rate; code and reasoning; supports tree drafting | Yes — EAGLE3 head trained on the target family |
-| **MTP** | Built-in multi-token prediction head | Models with bundled MTP heads (e.g. Qwen3.8-27B) | No — head bundled with the main model |
 
 All three strategies share the same server API — only the generation parameters differ.
 
 ## Prerequisites
 
-**Model preparation**: Python 3.9 or higher with pip and HuggingFace account
+**Model preparation**: Python 3.9 or higher with pip and a Hugging Face account
+
+**Model Server deployment**: Docker Engine or the OVMS binary package installed according to the [bare-metal deployment guide](../../../docs/deploying_server_baremetal.md)
+
+# MTP (Multi-Token Prediction)
+
+MTP replaces the separate draft model with a lightweight prediction head bundled inside the main model weights — no additional download is needed. The head is auto-detected by OVMS when `openvino_mtp_model.xml` is present in the draft model directory. Because it shares the main model's weights, the draft cost is minimal and acceptance rates are high for the same model family.
+
+## Model considerations
+
+For this demo we use [OpenVINO/Qwen3.8-27B-int4-ov](https://huggingface.co/OpenVINO/Qwen3.8-27B-int4-ov), which has a bundled MTP head and is exported in INT4 precision.
+
+> **Note:** This model requires OVMS 2026.4 or weekly pre-release build. See the model card for compatibility details. Prefix caching is not currently supported in MTP mode.
+
+## Server Deployment
+
+:::{dropdown} **Deploying with Docker**
+```bash
+export GPU_ARGS=$(if ls /dev/dri/render* >/dev/null 2>&1; then echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)"; fi)
+docker run -d --rm ${GPU_ARGS} -p 8000:8000 -v ${HOME}/models:/models:rw openvino/model_server:weekly \
+  --rest_port 8000 \
+  --model_repository_path /models \
+  --source_model OpenVINO/Qwen3.8-27B-int4-ov \
+  --draft_model_path . \
+  --enable_prefix_caching false
+```
+:::
+
+:::{dropdown} **Deploying on Bare Metal**
+```bat
+ovms --rest_port 8000 --model_repository_path c:\models --source_model OpenVINO/Qwen3.8-27B-int4-ov --draft_model_path . --enable_prefix_caching false
+```
+:::
+
+## Request Generation
+
+The API is identical to other speculative decoding strategies:
+
+```python
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
+
+response = client.chat.completions.create(
+    model="OpenVINO/Qwen3.8-27B-int4-ov",
+    messages=[{"role": "user", "content": "Explain the transformer attention mechanism."}],
+    temperature=0,
+    extra_body={"num_assistant_tokens": 5},
+)
+print(response.choices[0].message)
+```
+
+`num_assistant_tokens` controls how many MTP candidates are proposed per target step. The default is `5` if not specified.
+
+## Check performance
+
+Check the deployed model's performance by using the vLLM benchmark script and the Sonnet dataset.
+
+Install vLLM and download the Sonnet dataset:
+```bash
+pip install vllm --index-url https://wheels.vllm.ai/nightly/cpu --extra-index-url https://pypi.org/simple
+curl https://raw.githubusercontent.com/vllm-project/vllm/refs/heads/main/benchmarks/sonnet.txt -o sonnet.txt
+```
+
+Run benchmark with 100 requests sent sequentially:
+```bash
+vllm bench serve --dataset-name sonnet --dataset-path sonnet.txt --backend openai-chat --host localhost --port 8000 --endpoint /v3/chat/completions --max-concurrency 1 --model OpenVINO/Qwen3.8-27B-int4-ov --num-prompts 10
+```
+```
+============ Serving Benchmark Result ============
+Successful requests:                     10
+Failed requests:                         0
+Maximum request concurrency:             1
+Benchmark duration (s):                  27.85
+Total input tokens:                      5405
+Total generated tokens:                  1500
+Request throughput (req/s):              0.36
+Output token throughput (tok/s):         53.86
+Peak output token throughput (tok/s):    72.00
+Peak concurrent requests:                2.00
+Total token throughput (tok/s):          247.95
+---------------Time to First Token----------------
+Mean TTFT (ms):                          496.63
+Median TTFT (ms):                        433.09
+P99 TTFT (ms):                           897.54
+-----Time per Output Token (excl. 1st token)------
+Mean TPOT (ms):                          15.35
+Median TPOT (ms):                        15.15
+P99 TPOT (ms):                           17.88
+---------------Inter-token Latency----------------
+Mean ITL (ms):                           16.02
+Median ITL (ms):                         0.02
+P99 ITL (ms):                            54.90
+==================================================
+```
 
-**Model Server deployment**: Installed Docker Engine or OVMS binary package according to the [baremetal deployment guide](../../../docs/deploying_server_baremetal.md)
 
 # EAGLE3
 
@@ -48,8 +141,12 @@ mkdir models
 
 Run `export_model.py` script to download and quantize the model:
 
-```console
-python export_model.py text_generation --source_model Qwen/Qwen3-8B --draft_source_model AngelSlim/Qwen3-8B_eagle3 --draft_eagle3_mode --weight-format int4 --config_file_path models/config.json --model_repository_path models
+```bat
+python export_model.py text_generation --source_model Qwen/Qwen3-8B --draft_source_model AngelSlim/Qwen3-8B_eagle3 --draft_eagle3_mode --weight-format int4 --model_repository_path c:\models
+```
+or
+```bash
+python export_model.py text_generation --source_model Qwen/Qwen3-8B --draft_source_model AngelSlim/Qwen3-8B_eagle3 --draft_eagle3_mode --weight-format int4 --model_repository_path ${HOME}/models
 ```
 
 Draft model inherits all scheduler properties from the main model.
@@ -57,7 +154,6 @@ Draft model inherits all scheduler properties from the main model.
 You should have a model folder like below:
 ```
 models
-├── config.json
 └── Qwen
     └── Qwen3-8B
         ├── added_tokens.json
@@ -90,78 +186,24 @@ models
 
 :::{dropdown} **Deploying with Docker**
 ```bash
-docker run -d --rm $(test -d /dev/dri && echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)") \
-  -p 8000:8000 -v $(pwd)/models:/workspace:ro openvino/model_server:weekly \
-  --rest_port 8000 --config_path /workspace/config.json
+export GPU_ARGS=$(if ls /dev/dri/render* >/dev/null 2>&1; then echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)"; fi)
+docker run ${GPU_ARGS} -d --rm -p 8000:8000 -v ${HOME}/models:/models:ro openvino/model_server:weekly \
+    --model_path /models/Qwen/Qwen3-8B \
+    --model_name Qwen/Qwen3-8B \
+    --rest_port 8000
 ```
 
-OVMS auto-detects the best available device at startup. To target a specific device explicitly, pass `--target_device GPU` (or `NPU`, `HETERO:GPU,CPU`, etc.) to `export_model.py` and follow the [AI accelerators guide](../../../docs/accelerators.md) for additionally required docker parameters.
 :::
 
 :::{dropdown} **Deploying on Bare Metal**
 
-Assuming you have unpacked model server package, make sure to:
-
-- **On Windows**: run `setupvars` script
-- **On Linux**: set `LD_LIBRARY_PATH` and `PATH` environment variables
-
-as mentioned in [deployment guide](../../../docs/deploying_server_baremetal.md), in every new shell that will start OpenVINO Model Server.
-
-Depending on how you prepared models in the first step of this demo, they are deployed to either CPU or GPU (it's defined in `config.json`). If you run on GPU make sure to have appropriate drivers installed, so the device is accessible for the model server.
+Install OVMS as described in the [deployment guide](../../../docs/deploying_server_baremetal.md).
 
 ```bat
-ovms --rest_port 8000 --config_path ./models/config.json
+ovms --rest_port 8000 --model_path c:\models\Qwen\Qwen3-8B --model_name Qwen/Qwen3-8B
 ```
 :::
 
-## Check performance
-
-Let's check how the deployed model is doing by running performance test. For that purpose we can use vLLM benchmark script and sonnet dataset.
-
-Install vLLM and download sonnet dataset: 
-```bash
-pip install vllm --index-url https://wheels.vllm.ai/nightly/cpu --extra-index-url https://pypi.org/simple
-curl https://raw.githubusercontent.com/vllm-project/vllm/refs/heads/main/benchmarks/sonnet.txt -o sonnet.txt
-```
-
-Run benchmark with 100 requests sent sequentially:
-```bash
-vllm bench serve --dataset-name sonnet --dataset-path sonnet.txt --backend openai-chat --host localhost --port 8000 --endpoint /v3/chat/completions --max-concurrency 1 --tokenizer Qwen/Qwen3-8B --model Qwen/Qwen3-8B --num_prompts 100
-
-Starting initial single prompt test run...
-Skipping endpoint ready check.
-Starting main benchmark run...
-Traffic request rate: inf
-Burstiness factor: 1.0 (Poisson process)
-Maximum request concurrency: 1
-100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [06:59<00:00,  4.19s/it]
-tip: install termplotlib and gnuplot to plot the metrics
-============ Serving Benchmark Result ============
-Successful requests:                     100
-Failed requests:                         0
-Maximum request concurrency:             1
-Benchmark duration (s):                  419.00
-Total input tokens:                      54256
-Total generated tokens:                  15000
-Request throughput (req/s):              0.24
-Output token throughput (tok/s):         35.80
-Peak output token throughput (tok/s):    16.00
-Peak concurrent requests:                2.00
-Total token throughput (tok/s):          165.29
----------------Time to First Token----------------
-Mean TTFT (ms):                          426.71
-Median TTFT (ms):                        424.97
-P99 TTFT (ms):                           635.37
------Time per Output Token (excl. 1st token)------
-Mean TPOT (ms):                          25.25
-Median TPOT (ms):                        25.09
-P99 TPOT (ms):                           29.22
----------------Inter-token Latency----------------
-Mean ITL (ms):                           66.29
-Median ITL (ms):                         66.75
-P99 ITL (ms):                            72.11
-==================================================
-```
 
 ## Chain drafting
 
@@ -184,7 +226,7 @@ print(response.choices[0].message.content)
 
 Increase `num_assistant_tokens` until the tokens-per-step figure plateaus, then back off — past the plateau, rejected draft tokens are pure overhead.
 
-Setting `num_assistant_tokens: 0` disables drafting for that request; only the target model runs.
+`num_assistant_tokens` must be greater than `0`; OVMS rejects a value of `0` for EAGLE3 decoding.
 
 ## Tree drafting
 
@@ -222,10 +264,10 @@ In this demo:
   - [meta-llama/CodeLlama-7b-hf](https://huggingface.co/meta-llama/CodeLlama-7b-hf) as a main model
   - [AMD-Llama-135m](https://huggingface.co/amd/AMD-Llama-135m) as a draft model
 
-both in FP16 precision.
+both in INT8 precision.
 
 ## Model preparation
-Here, the original Pytorch LLM models and the tokenizers will be converted to IR format and optionally quantized.
+Here, the original PyTorch LLM models and tokenizers are converted to IR format and quantized.
 That ensures faster initialization time, better performance and lower memory consumption.
 LLM engine parameters will be defined inside the `graph.pbtxt` file.
 
@@ -238,10 +280,14 @@ mkdir models
 
 Run `export_model.py` script to download and quantize the model:
 
-> **Note:** Before downloading the CodeLlama model, access must be requested. Follow the instructions on the [meta-llama/CodeLlama-7b-hf](https://huggingface.co/meta-llama/CodeLlama-7b-hf) to request access. When access is granted, create an authentication token in the HuggingFace account -> Settings -> Access Tokens page. Issue the following command and enter the authentication token. Authenticate via `huggingface-cli login`.
+> **Note:** Before downloading the CodeLlama model, request access by following the instructions on the [meta-llama/CodeLlama-7b-hf](https://huggingface.co/meta-llama/CodeLlama-7b-hf) model page. After access is granted, create an authentication token under Hugging Face **Settings > Access Tokens**, run `huggingface-cli login`, and enter the token when prompted.
 
-```console
-python export_model.py text_generation --source_model meta-llama/CodeLlama-7b-hf --draft_source_model amd/AMD-Llama-135m --weight-format fp16 --kv_cache_precision u8 --config_file_path models/config.json --model_repository_path models
+```bat
+python export_model.py text_generation --source_model meta-llama/CodeLlama-7b-hf --draft_source_model amd/AMD-Llama-135m --weight-format int8 --model_repository_path c:\models
+```
+or
+```bash
+python export_model.py text_generation --source_model meta-llama/CodeLlama-7b-hf --draft_source_model amd/AMD-Llama-135m --weight-format int8 --model_repository_path ${HOME}/models
 ```
 
 Draft model inherits all scheduler properties from the main model.
@@ -249,7 +295,6 @@ Draft model inherits all scheduler properties from the main model.
 You should have a model folder like below:
 ```
 models
-├── config.json
 └── meta-llama
     └── CodeLlama-7b-hf
         ├── amd-AMD-Llama-135m
@@ -285,25 +330,21 @@ models
 
 :::{dropdown} **Deploying with Docker**
 ```bash
-docker run -d --rm -p 8000:8000 -v $(pwd)/models:/workspace:ro openvino/model_server:latest --rest_port 8000 --config_path /workspace/config.json
+export GPU_ARGS=$(if ls /dev/dri/render* >/dev/null 2>&1; then echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)"; fi)
+docker run -d ${GPU_ARGS} --rm -p 8000:8000 -v ${HOME}/models:/models:ro openvino/model_server:weekly \
+    --rest_port 8000 \
+    --model_path /models/meta-llama/CodeLlama-7b-hf \
+    --model_name meta-llama/CodeLlama-7b-hf
 ```
 
-OVMS auto-detects the best available device at startup. To target a specific device explicitly, pass `--target_device GPU` (or `NPU`, `HETERO:GPU,CPU`, etc.) to `export_model.py` and follow the [AI accelerators guide](../../../docs/accelerators.md) for additionally required docker parameters.
 :::
 
 :::{dropdown} **Deploying on Bare Metal**
 
-Assuming you have unpacked model server package, make sure to:
-
-- **On Windows**: run `setupvars` script
-- **On Linux**: set `LD_LIBRARY_PATH` and `PATH` environment variables
-
-as mentioned in [deployment guide](../../../docs/deploying_server_baremetal.md), in every new shell that will start OpenVINO Model Server.
-
-Depending on how you prepared models in the first step of this demo, they are deployed to either CPU or GPU (it's defined in `config.json`). If you run on GPU make sure to have appropriate drivers installed, so the device is accessible for the model server.
+Install OVMS as described in the [deployment guide](../../../docs/deploying_server_baremetal.md).
 
 ```bat
-ovms --rest_port 8000 --config_path ./models/config.json
+ovms --rest_port 8000 --model_path c:\models\meta-llama\CodeLlama-7b-hf --model_name meta-llama/CodeLlama-7b-hf
 ```
 :::
 
@@ -364,61 +405,7 @@ for chunk in stream:
 
 `num_assistant_tokens` does not have to be sent on every request — see [Setting default generation parameters](#setting-default-generation-parameters) to configure a deployment-level default.
 
-# MTP (Multi-Token Prediction)
-
-MTP replaces the separate draft model with a lightweight prediction head bundled inside the main model weights — no additional download is needed. The head is auto-detected by OVMS when `openvino_mtp_model.xml` is present in the draft model directory. Because it shares the main model's weights, the draft cost is minimal and acceptance rates are high for the same model family.
-
-## Model considerations
-
-For this demo we use [OpenVINO/Qwen3.8-27B-int8-ov](https://huggingface.co/OpenVINO/Qwen3.8-27B-int8-ov) — a vision-language model with a bundled MTP head exported in INT8 precision.
-
-> **Note:** This model requires OpenVINO nightly builds and is marked experimental. See the model card for compatibility details.
-> **Note:** Prefix caching is not yet supported in this mode.
-
-## Server Deployment
-
-:::{dropdown} **Deploying with Docker**
-```bash
-docker run -d --rm $(test -d /dev/dri && echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)") \
-  -p 8000:8000 -v ${HOME}/models:/models:rw openvino/model_server:weekly \
-  --rest_port 8000 \
-  --model_repository_path /models \
-  --source_model OpenVINO/Qwen3.8-27B-int4-ov \
-  --draft_model_path . \
-  --enable_prefix_caching false
-```
-:::
-
-:::{dropdown} **Deploying on Bare Metal**
-```console
-ovms --rest_port 8000 \
-  --model_repository_path c:\models \
-  --source_model OpenVINO/Qwen3.8-27B-int4-ov \
-  --draft_model_path . \
-  --enable_prefix_caching false
-```
-:::
-
-## Request Generation
 
-The API is identical to other speculative decoding strategies:
-
-```python
-from openai import OpenAI
-
-client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
-
-response = client.chat.completions.create(
-    model="OpenVINO/Qwen3.8-27B-int4-ov",
-    messages=[{"role": "user", "content": "Explain the transformer attention mechanism."}],
-    temperature=0,
-    max_tokens=200,
-    extra_body={"num_assistant_tokens": 5},
-)
-print(response.choices[0].message.content)
-```
-
-`num_assistant_tokens` controls how many MTP candidates are proposed per target step. The default is `5` if not specified.
 
 # Setting Default Generation Parameters
 

From 42d1f4d474a2dec4c3207faf4a6290cb6806a3e5 Mon Sep 17 00:00:00 2001
From: Trawinski <dariusz.trawinski@intel.com>
Date: Wed, 2 Sep 2026 01:34:33 +0200
Subject: [PATCH 02/23] fix

---
 demos/continuous_batching/speculative_decoding/README.md | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/demos/continuous_batching/speculative_decoding/README.md b/demos/continuous_batching/speculative_decoding/README.md
index b58b78b3ab..fa42c43b74 100644
--- a/demos/continuous_batching/speculative_decoding/README.md
+++ b/demos/continuous_batching/speculative_decoding/README.md
@@ -33,7 +33,7 @@ For this demo we use [OpenVINO/Qwen3.8-27B-int4-ov](https://huggingface.co/OpenV
 :::{dropdown} **Deploying with Docker**
 ```bash
 export GPU_ARGS=$(if ls /dev/dri/render* >/dev/null 2>&1; then echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)"; fi)
-docker run -d --rm ${GPU_ARGS} -p 8000:8000 -v ${HOME}/models:/models:rw openvino/model_server:weekly \
+docker run -d --rm ${GPU_ARGS} --user $(id -u):$(id -g) -p 8000:8000 -v ${HOME}/models:/models:rw openvino/model_server:weekly \
   --rest_port 8000 \
   --model_repository_path /models \
   --source_model OpenVINO/Qwen3.8-27B-int4-ov \
@@ -187,7 +187,7 @@ models
 :::{dropdown} **Deploying with Docker**
 ```bash
 export GPU_ARGS=$(if ls /dev/dri/render* >/dev/null 2>&1; then echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)"; fi)
-docker run ${GPU_ARGS} -d --rm -p 8000:8000 -v ${HOME}/models:/models:ro openvino/model_server:weekly \
+docker run -d ${GPU_ARGS} --user $(id -u):$(id -g) --rm -p 8000:8000 -v ${HOME}/models:/models:ro openvino/model_server:weekly \
     --model_path /models/Qwen/Qwen3-8B \
     --model_name Qwen/Qwen3-8B \
     --rest_port 8000
@@ -331,7 +331,7 @@ models
 :::{dropdown} **Deploying with Docker**
 ```bash
 export GPU_ARGS=$(if ls /dev/dri/render* >/dev/null 2>&1; then echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)"; fi)
-docker run -d ${GPU_ARGS} --rm -p 8000:8000 -v ${HOME}/models:/models:ro openvino/model_server:weekly \
+docker run -d ${GPU_ARGS} --user $(id -u):$(id -g) --rm -p 8000:8000 -v ${HOME}/models:/models:ro openvino/model_server:weekly \
     --rest_port 8000 \
     --model_path /models/meta-llama/CodeLlama-7b-hf \
     --model_name meta-llama/CodeLlama-7b-hf

From e0774a7e3c574952e0a41d9b23e5a20a333c0db7 Mon Sep 17 00:00:00 2001
From: Trawinski <dariusz.trawinski@intel.com>
Date: Wed, 2 Sep 2026 01:41:26 +0200
Subject: [PATCH 03/23] dummy change

---
 src/BUILD | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/BUILD b/src/BUILD
index 1e2368327b..97d0d5e00b 100644
--- a/src/BUILD
+++ b/src/BUILD
@@ -2939,4 +2939,4 @@ ovms_cc_library(
             "//src:libovms_ov_utils",],
     visibility = ["//visibility:public"],
     alwayslink = 1,
-)
+)
\ No newline at end of file

From c0210212cd9d9153d013b9fa09a670d1c1cf2d5a Mon Sep 17 00:00:00 2001
From: Trawinski <dariusz.trawinski@intel.com>
Date: Wed, 2 Sep 2026 09:03:02 +0200
Subject: [PATCH 04/23] set max_tokens

---
 demos/continuous_batching/speculative_decoding/README.md | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/demos/continuous_batching/speculative_decoding/README.md b/demos/continuous_batching/speculative_decoding/README.md
index fa42c43b74..18777f6f84 100644
--- a/demos/continuous_batching/speculative_decoding/README.md
+++ b/demos/continuous_batching/speculative_decoding/README.md
@@ -26,7 +26,7 @@ MTP replaces the separate draft model with a lightweight prediction head bundled
 
 For this demo we use [OpenVINO/Qwen3.8-27B-int4-ov](https://huggingface.co/OpenVINO/Qwen3.8-27B-int4-ov), which has a bundled MTP head and is exported in INT4 precision.
 
-> **Note:** This model requires OVMS 2026.4 or weekly pre-release build. See the model card for compatibility details. Prefix caching is not currently supported in MTP mode.
+> **Note:** This model requires OVMS 2026.4 or weekly pre-release build. See the model card for compatibility details. Prefix caching is not currently supported in MTP mode. It is also required to set max_tokens parameter
 
 ## Server Deployment
 
@@ -61,6 +61,7 @@ response = client.chat.completions.create(
     model="OpenVINO/Qwen3.8-27B-int4-ov",
     messages=[{"role": "user", "content": "Explain the transformer attention mechanism."}],
     temperature=0,
+    max_tokens=10000,
     extra_body={"num_assistant_tokens": 5},
 )
 print(response.choices[0].message)
@@ -218,7 +219,7 @@ response = client.chat.completions.create(
     model="Qwen/Qwen3-8B",
     messages=[{"role": "user", "content": "What is OpenVINO?"}],
     temperature=0,
-    max_tokens=200,
+    max_tokens=2000,
     extra_body={"num_assistant_tokens": 5},
 )
 print(response.choices[0].message.content)
@@ -390,7 +391,7 @@ stream = client.completions.create(
     model="meta-llama/CodeLlama-7b-hf",
     prompt="<s>def quicksort(numbers):",
     temperature=0,
-    max_tokens=100,
+    max_tokens=2000,
     extra_body={"num_assistant_tokens": 5},
     stream=True,
 )

From a08392bc4a19730678dadcde0391102018e83490 Mon Sep 17 00:00:00 2001
From: "Trawinski, Dariusz" <dariusz.trawinski@intel.com>
Date: Wed, 2 Sep 2026 11:57:06 +0200
Subject: [PATCH 05/23] Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---
 demos/continuous_batching/speculative_decoding/README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/demos/continuous_batching/speculative_decoding/README.md b/demos/continuous_batching/speculative_decoding/README.md
index 18777f6f84..59e770c5a4 100644
--- a/demos/continuous_batching/speculative_decoding/README.md
+++ b/demos/continuous_batching/speculative_decoding/README.md
@@ -227,7 +227,7 @@ print(response.choices[0].message.content)
 
 Increase `num_assistant_tokens` until the tokens-per-step figure plateaus, then back off — past the plateau, rejected draft tokens are pure overhead.
 
-`num_assistant_tokens` must be greater than `0`; OVMS rejects a value of `0` for EAGLE3 decoding.
+Setting `num_assistant_tokens: 0` disables drafting for that request; only the target model runs.
 
 ## Tree drafting
 

From 99bf4365bc569b3396e1efa7a38f5e7c05cb36e3 Mon Sep 17 00:00:00 2001
From: Trawinski <dariusz.trawinski@intel.com>
Date: Wed, 2 Sep 2026 23:26:15 +0200
Subject: [PATCH 06/23] instal opoenai

---
 demos/continuous_batching/speculative_decoding/README.md | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/demos/continuous_batching/speculative_decoding/README.md b/demos/continuous_batching/speculative_decoding/README.md
index 18777f6f84..1dcdb7192e 100644
--- a/demos/continuous_batching/speculative_decoding/README.md
+++ b/demos/continuous_batching/speculative_decoding/README.md
@@ -51,7 +51,9 @@ ovms --rest_port 8000 --model_repository_path c:\models --source_model OpenVINO/
 ## Request Generation
 
 The API is identical to other speculative decoding strategies:
-
+```console
+pip install openai
+```
 ```python
 from openai import OpenAI
 

From 6a3b1e130dfe0b6867238709b9e51832e5778c1b Mon Sep 17 00:00:00 2001
From: Trawinski <dariusz.trawinski@intel.com>
Date: Thu, 3 Sep 2026 11:41:06 +0200
Subject: [PATCH 07/23] test

---
 demos/common/export_models/requirements.txt              | 2 +-
 demos/continuous_batching/speculative_decoding/README.md | 4 +++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/demos/common/export_models/requirements.txt b/demos/common/export_models/requirements.txt
index 54c76788cf..f567b2137d 100644
--- a/demos/common/export_models/requirements.txt
+++ b/demos/common/export_models/requirements.txt
@@ -1,7 +1,7 @@
 --extra-index-url "https://download.pytorch.org/whl/cpu"
 --extra-index-url "https://storage.openvinotoolkit.org/simple/wheels/nightly"
 --extra-index-url "https://storage.openvinotoolkit.org/simple/wheels/pre-release"
-optimum-intel@git+https://github.com/huggingface/optimum-intel.git@fd9499006ef4c3f35d5c6647851b2e9fa8711912
+optimum-intel
 accelerate
 datasets
 diffusers  # for image generation
diff --git a/demos/continuous_batching/speculative_decoding/README.md b/demos/continuous_batching/speculative_decoding/README.md
index 7a7d505fe3..29b5cee3b8 100644
--- a/demos/continuous_batching/speculative_decoding/README.md
+++ b/demos/continuous_batching/speculative_decoding/README.md
@@ -63,7 +63,6 @@ response = client.chat.completions.create(
     model="OpenVINO/Qwen3.8-27B-int4-ov",
     messages=[{"role": "user", "content": "Explain the transformer attention mechanism."}],
     temperature=0,
-    max_tokens=10000,
     extra_body={"num_assistant_tokens": 5},
 )
 print(response.choices[0].message)
@@ -236,6 +235,9 @@ Setting `num_assistant_tokens: 0` disables drafting for that request; only the t
 Tree drafting adds two `GenerationConfig` fields. Setting `tree_depth > 0` switches from chain to tree mode:
 
 ```python
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:8000/v3", api_key="unused")
 response = client.chat.completions.create(
     model="Qwen/Qwen3-8B",
     messages=[{"role": "user", "content": "What is OpenVINO?"}],

From 8045ba01733a22247982706803127ac77c279a61 Mon Sep 17 00:00:00 2001
From: Trawinski <dariusz.trawinski@intel.com>
Date: Thu, 3 Sep 2026 11:49:57 +0200
Subject: [PATCH 08/23] WA for github

---
 ci/build_test_OnCommit.groovy | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/ci/build_test_OnCommit.groovy b/ci/build_test_OnCommit.groovy
index 5573d55ce6..526535c770 100644
--- a/ci/build_test_OnCommit.groovy
+++ b/ci/build_test_OnCommit.groovy
@@ -38,6 +38,13 @@ pipeline {
     agent {
       label 'ovmsbuilder'
     }
+    environment {
+      // Force HTTP/1.1: some proxies mangle git's HTTP/2 smart-protocol responses,
+      // causing "fatal: expected flush after ref listing" on fetch.
+      GIT_CONFIG_COUNT = "1"
+      GIT_CONFIG_KEY_0 = "http.version"
+      GIT_CONFIG_VALUE_0 = "HTTP/1.1"
+    }
     options {
       timeout(time: 4, unit: 'HOURS')
     }

From 3b629156b4baf158f17e6295e73c1e25909b210c Mon Sep 17 00:00:00 2001
From: Trawinski <dariusz.trawinski@intel.com>
Date: Thu, 3 Sep 2026 14:27:44 +0200
Subject: [PATCH 09/23] tune parameters

---
 .../speculative_decoding/README.md            | 170 +-----------------
 1 file changed, 8 insertions(+), 162 deletions(-)

diff --git a/demos/continuous_batching/speculative_decoding/README.md b/demos/continuous_batching/speculative_decoding/README.md
index 29b5cee3b8..b02cd93bba 100644
--- a/demos/continuous_batching/speculative_decoding/README.md
+++ b/demos/continuous_batching/speculative_decoding/README.md
@@ -82,7 +82,7 @@ curl https://raw.githubusercontent.com/vllm-project/vllm/refs/heads/main/benchma
 
 Run benchmark with 100 requests sent sequentially:
 ```bash
-vllm bench serve --dataset-name sonnet --dataset-path sonnet.txt --backend openai-chat --host localhost --port 8000 --endpoint /v3/chat/completions --max-concurrency 1 --model OpenVINO/Qwen3.8-27B-int4-ov --num-prompts 10
+vllm bench serve --dataset-name sonnet --dataset-path sonnet.txt --backend openai-chat --host localhost --port 8000 --endpoint /v1/chat/completions --max-concurrency 1 --model OpenVINO/Qwen3.8-27B-int4-ov --num-prompts 10
 ```
 ```
 ============ Serving Benchmark Result ============
@@ -214,7 +214,7 @@ Send `num_assistant_tokens` to control how many candidates the draft head propos
 ```python
 from openai import OpenAI
 
-client = OpenAI(base_url="http://localhost:8000/v3", api_key="unused")
+client = OpenAI(base_url="http://localhost:8000/1", api_key="unused")
 
 response = client.chat.completions.create(
     model="Qwen/Qwen3-8B",
@@ -237,179 +237,25 @@ Tree drafting adds two `GenerationConfig` fields. Setting `tree_depth > 0` switc
 ```python
 from openai import OpenAI
 
-client = OpenAI(base_url="http://localhost:8000/v3", api_key="unused")
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
 response = client.chat.completions.create(
     model="Qwen/Qwen3-8B",
     messages=[{"role": "user", "content": "What is OpenVINO?"}],
     temperature=0,
-    max_tokens=200,
+    max_tokens=2000,
     extra_body={
-        "num_assistant_tokens": 15,   # candidates verified per step
-        "branching_factor": 8,        # top-k expansions per tree layer
-        "tree_depth": 4,              # draft head iterations
+        "num_assistant_tokens": 5,    # candidates verified per step
+        "branching_factor": 4,        # top-k expansions per tree layer
+        "tree_depth": 2,              # draft head iterations
     },
 )
+print(response.choices[0].message.content)
 ```
 
 `total_draft_tokens = branching_factor² × (tree_depth − 1) + branching_factor` must be ≥ `num_assistant_tokens`. A reasonable starting point is `branching_factor=4..8`, `tree_depth=3..4`.
 
 Tree drafting is EAGLE3-only; it cannot be combined with beam search or multinomial sampling.
 
-# Fast Draft
-
-Fast Draft is the classic two-model setup: a smaller off-the-shelf LLM that shares the target's tokenizer proposes tokens autoregressively, and the target model verifies them. It works with any target/draft pair without retraining. The speedup depends on how often the small model's distribution agrees with the large one.
-
-## Model considerations
-
-Both models must share the same tokenizer so draft token IDs map correctly to target token IDs.
-
-Performance gain depends heavily on the model pair and workload — the optimal combination should be found empirically. Model sizes and precisions both factor in.
-
-In this demo:
-  - [meta-llama/CodeLlama-7b-hf](https://huggingface.co/meta-llama/CodeLlama-7b-hf) as a main model
-  - [AMD-Llama-135m](https://huggingface.co/amd/AMD-Llama-135m) as a draft model
-
-both in INT8 precision.
-
-## Model preparation
-Here, the original PyTorch LLM models and tokenizers are converted to IR format and quantized.
-That ensures faster initialization time, better performance and lower memory consumption.
-LLM engine parameters will be defined inside the `graph.pbtxt` file.
-
-Download export script, install its dependencies and create directory for the models:
-```console
-curl https://raw.githubusercontent.com/openvinotoolkit/model_server/refs/heads/main/demos/common/export_models/export_model.py -o export_model.py
-pip3 install -r https://raw.githubusercontent.com/openvinotoolkit/model_server/refs/heads/main/demos/common/export_models/requirements.txt
-mkdir models 
-```
-
-Run `export_model.py` script to download and quantize the model:
-
-> **Note:** Before downloading the CodeLlama model, request access by following the instructions on the [meta-llama/CodeLlama-7b-hf](https://huggingface.co/meta-llama/CodeLlama-7b-hf) model page. After access is granted, create an authentication token under Hugging Face **Settings > Access Tokens**, run `huggingface-cli login`, and enter the token when prompted.
-
-```bat
-python export_model.py text_generation --source_model meta-llama/CodeLlama-7b-hf --draft_source_model amd/AMD-Llama-135m --weight-format int8 --model_repository_path c:\models
-```
-or
-```bash
-python export_model.py text_generation --source_model meta-llama/CodeLlama-7b-hf --draft_source_model amd/AMD-Llama-135m --weight-format int8 --model_repository_path ${HOME}/models
-```
-
-Draft model inherits all scheduler properties from the main model.
-
-You should have a model folder like below:
-```
-models
-└── meta-llama
-    └── CodeLlama-7b-hf
-        ├── amd-AMD-Llama-135m
-        │   ├── config.json
-        │   ├── generation_config.json
-        │   ├── openvino_detokenizer.bin
-        │   ├── openvino_detokenizer.xml
-        │   ├── openvino_model.bin
-        │   ├── openvino_model.xml
-        │   ├── openvino_tokenizer.bin
-        │   ├── openvino_tokenizer.xml
-        │   ├── special_tokens_map.json
-        │   ├── tokenizer_config.json
-        │   ├── tokenizer.json
-        │   └── tokenizer.model
-        ├── config.json
-        ├── generation_config.json
-        ├── graph.pbtxt
-        ├── openvino_detokenizer.bin
-        ├── openvino_detokenizer.xml
-        ├── openvino_model.bin
-        ├── openvino_model.xml
-        ├── openvino_tokenizer.bin
-        ├── openvino_tokenizer.xml
-        ├── special_tokens_map.json
-        ├── tokenizer_config.json
-        ├── tokenizer.json
-        └── tokenizer.model
-
-```
-
-## Server Deployment
-
-:::{dropdown} **Deploying with Docker**
-```bash
-export GPU_ARGS=$(if ls /dev/dri/render* >/dev/null 2>&1; then echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)"; fi)
-docker run -d ${GPU_ARGS} --user $(id -u):$(id -g) --rm -p 8000:8000 -v ${HOME}/models:/models:ro openvino/model_server:weekly \
-    --rest_port 8000 \
-    --model_path /models/meta-llama/CodeLlama-7b-hf \
-    --model_name meta-llama/CodeLlama-7b-hf
-```
-
-:::
-
-:::{dropdown} **Deploying on Bare Metal**
-
-Install OVMS as described in the [deployment guide](../../../docs/deploying_server_baremetal.md).
-
-```bat
-ovms --rest_port 8000 --model_path c:\models\meta-llama\CodeLlama-7b-hf --model_name meta-llama/CodeLlama-7b-hf
-```
-:::
-
-## Readiness Check
-
-Wait for the model to load. You can check the status with a simple command:
-```console
-curl http://localhost:8000/v1/config
-```
-```json
-{
-    "meta-llama/CodeLlama-7b-hf": {
-        "model_version_status": [
-            {
-                "version": "1",
-                "state": "AVAILABLE",
-                "status": {
-                    "error_code": "OK",
-                    "error_message": "OK"
-                }
-            }
-        ]
-    }
-}
-```
-
-## Request Generation
-
-Models used in this demo — `meta-llama/CodeLlama-7b-hf` and `AMD-Llama-135m` — are base (non-chat) models, so we use the `completions` endpoint.
-
-```console
-pip3 install openai
-```
-```python
-from openai import OpenAI
-
-client = OpenAI(
-  base_url="http://localhost:8000/v3",
-  api_key="unused"
-)
-
-stream = client.completions.create(
-    model="meta-llama/CodeLlama-7b-hf",
-    prompt="<s>def quicksort(numbers):",
-    temperature=0,
-    max_tokens=2000,
-    extra_body={"num_assistant_tokens": 5},
-    stream=True,
-)
-for chunk in stream:
-    if chunk.choices[0].text is not None:
-        print(chunk.choices[0].text, end="", flush=True)
-```
-
-**`num_assistant_tokens`** controls how many tokens the draft model proposes before the main model validates them. High values pay off when the draft frequently agrees with the target; low values reduce wasted work when it doesn't. `5` is a good starting point.
-
-**`assistant_confidence_threshold`** is an alternative stopping criterion: the draft keeps proposing while its token probability exceeds the threshold, then hands off to the target. It is mutually exclusive with `num_assistant_tokens`. Supported on the Continuous Batching backend only (`LM_CB`) — the stateful backend (`pipeline_type: LM`) does not implement dynamic-length drafting.
-
-`num_assistant_tokens` does not have to be sent on every request — see [Setting default generation parameters](#setting-default-generation-parameters) to configure a deployment-level default.
-
 
 
 # Setting Default Generation Parameters

From 9854b28a9c6cbab124a7025acf8370ca60654625 Mon Sep 17 00:00:00 2001
From: Trawinski <dariusz.trawinski@intel.com>
Date: Thu, 3 Sep 2026 14:29:19 +0200
Subject: [PATCH 10/23] tune

---
 demos/continuous_batching/speculative_decoding/README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/demos/continuous_batching/speculative_decoding/README.md b/demos/continuous_batching/speculative_decoding/README.md
index b02cd93bba..2cec1e9e11 100644
--- a/demos/continuous_batching/speculative_decoding/README.md
+++ b/demos/continuous_batching/speculative_decoding/README.md
@@ -61,7 +61,7 @@ client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
 
 response = client.chat.completions.create(
     model="OpenVINO/Qwen3.8-27B-int4-ov",
-    messages=[{"role": "user", "content": "Explain the transformer attention mechanism."}],
+    messages=[{"role": "user", "content": "Explain briefly the transformer attention mechanism."}],
     temperature=0,
     extra_body={"num_assistant_tokens": 5},
 )

From d8da8c5564f76d833fd858bdac7ca814881bcccf Mon Sep 17 00:00:00 2001
From: Trawinski <dariusz.trawinski@intel.com>
Date: Fri, 4 Sep 2026 09:23:30 +0200
Subject: [PATCH 11/23] fix url

---
 demos/continuous_batching/speculative_decoding/README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/demos/continuous_batching/speculative_decoding/README.md b/demos/continuous_batching/speculative_decoding/README.md
index 2cec1e9e11..c74427c877 100644
--- a/demos/continuous_batching/speculative_decoding/README.md
+++ b/demos/continuous_batching/speculative_decoding/README.md
@@ -214,7 +214,7 @@ Send `num_assistant_tokens` to control how many candidates the draft head propos
 ```python
 from openai import OpenAI
 
-client = OpenAI(base_url="http://localhost:8000/1", api_key="unused")
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
 
 response = client.chat.completions.create(
     model="Qwen/Qwen3-8B",

From 81d090e96631beb2cdf6b4a75473db9d3459bd94 Mon Sep 17 00:00:00 2001
From: Trawinski <dariusz.trawinski@intel.com>
Date: Fri, 4 Sep 2026 13:30:50 +0200
Subject: [PATCH 12/23] temporary switch to CPU in eagle3

---
 demos/continuous_batching/speculative_decoding/README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/demos/continuous_batching/speculative_decoding/README.md b/demos/continuous_batching/speculative_decoding/README.md
index c74427c877..76f710f501 100644
--- a/demos/continuous_batching/speculative_decoding/README.md
+++ b/demos/continuous_batching/speculative_decoding/README.md
@@ -148,7 +148,7 @@ python export_model.py text_generation --source_model Qwen/Qwen3-8B --draft_sour
 ```
 or
 ```bash
-python export_model.py text_generation --source_model Qwen/Qwen3-8B --draft_source_model AngelSlim/Qwen3-8B_eagle3 --draft_eagle3_mode --weight-format int4 --model_repository_path ${HOME}/models
+python export_model.py text_generation --source_model Qwen/Qwen3-8B --draft_source_model AngelSlim/Qwen3-8B_eagle3 --draft_eagle3_mode --weight-format int4 --target_device CPU --model_repository_path ${HOME}/models
 ```
 
 Draft model inherits all scheduler properties from the main model.

From e9a4d5f8892681a8e02b1056a290f237920e20bd Mon Sep 17 00:00:00 2001
From: Trawinski <dariusz.trawinski@intel.com>
Date: Fri, 4 Sep 2026 13:34:11 +0200
Subject: [PATCH 13/23] revert unrelated changes

---
 ci/build_test_OnCommit.groovy               | 7 -------
 demos/common/export_models/requirements.txt | 2 +-
 src/BUILD                                   | 2 +-
 3 files changed, 2 insertions(+), 9 deletions(-)

diff --git a/ci/build_test_OnCommit.groovy b/ci/build_test_OnCommit.groovy
index 526535c770..5573d55ce6 100644
--- a/ci/build_test_OnCommit.groovy
+++ b/ci/build_test_OnCommit.groovy
@@ -38,13 +38,6 @@ pipeline {
     agent {
       label 'ovmsbuilder'
     }
-    environment {
-      // Force HTTP/1.1: some proxies mangle git's HTTP/2 smart-protocol responses,
-      // causing "fatal: expected flush after ref listing" on fetch.
-      GIT_CONFIG_COUNT = "1"
-      GIT_CONFIG_KEY_0 = "http.version"
-      GIT_CONFIG_VALUE_0 = "HTTP/1.1"
-    }
     options {
       timeout(time: 4, unit: 'HOURS')
     }
diff --git a/demos/common/export_models/requirements.txt b/demos/common/export_models/requirements.txt
index f567b2137d..54c76788cf 100644
--- a/demos/common/export_models/requirements.txt
+++ b/demos/common/export_models/requirements.txt
@@ -1,7 +1,7 @@
 --extra-index-url "https://download.pytorch.org/whl/cpu"
 --extra-index-url "https://storage.openvinotoolkit.org/simple/wheels/nightly"
 --extra-index-url "https://storage.openvinotoolkit.org/simple/wheels/pre-release"
-optimum-intel
+optimum-intel@git+https://github.com/huggingface/optimum-intel.git@fd9499006ef4c3f35d5c6647851b2e9fa8711912
 accelerate
 datasets
 diffusers  # for image generation
diff --git a/src/BUILD b/src/BUILD
index 97d0d5e00b..1e2368327b 100644
--- a/src/BUILD
+++ b/src/BUILD
@@ -2939,4 +2939,4 @@ ovms_cc_library(
             "//src:libovms_ov_utils",],
     visibility = ["//visibility:public"],
     alwayslink = 1,
-)
\ No newline at end of file
+)

From 0fe84bbd40729aa45ccdd3ca7418b76ecd6be08f Mon Sep 17 00:00:00 2001
From: Trawinski <dariusz.trawinski@intel.com>
Date: Fri, 4 Sep 2026 13:40:50 +0200
Subject: [PATCH 14/23] was for github

---
 ci/build_test_OnCommit.groovy | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/ci/build_test_OnCommit.groovy b/ci/build_test_OnCommit.groovy
index 5573d55ce6..526535c770 100644
--- a/ci/build_test_OnCommit.groovy
+++ b/ci/build_test_OnCommit.groovy
@@ -38,6 +38,13 @@ pipeline {
     agent {
       label 'ovmsbuilder'
     }
+    environment {
+      // Force HTTP/1.1: some proxies mangle git's HTTP/2 smart-protocol responses,
+      // causing "fatal: expected flush after ref listing" on fetch.
+      GIT_CONFIG_COUNT = "1"
+      GIT_CONFIG_KEY_0 = "http.version"
+      GIT_CONFIG_VALUE_0 = "HTTP/1.1"
+    }
     options {
       timeout(time: 4, unit: 'HOURS')
     }

From f570bec28e5ea4a255ab715b173c1a596438085f Mon Sep 17 00:00:00 2001
From: Trawinski <dariusz.trawinski@intel.com>
Date: Fri, 4 Sep 2026 14:48:33 +0200
Subject: [PATCH 15/23] WA for github issues

---
 Makefile | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index aebb6bbdf8..7df8fc2c1b 100644
--- a/Makefile
+++ b/Makefile
@@ -244,7 +244,10 @@ BUILD_ARGS = --build-arg http_proxy=$(HTTP_PROXY)\
 	--build-arg VERBOSE_LOGS=$(VERBOSE_LOGS)\
 	--build-arg KONFLUX=$(KONFLUX)\
 	--build-arg ESPEAK=$(ESPEAK)
-
+	-e GIT_CONFIG_COUNT="1"\
+	-e GIT_CONFIG_KEY_0="http.version"\
+	-e GIT_CONFIG_VALUE_0="HTTP/1.1"
+# WA for github issues with anonymous access
 
 .PHONY: default docker_build \
 

From 3249d40afc1daceced7a3dbacc8ee2ac4b47751c Mon Sep 17 00:00:00 2001
From: Trawinski <dariusz.trawinski@intel.com>
Date: Fri, 4 Sep 2026 14:55:42 +0200
Subject: [PATCH 16/23] WA for github issues

---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 7df8fc2c1b..ffe415839c 100644
--- a/Makefile
+++ b/Makefile
@@ -243,7 +243,7 @@ BUILD_ARGS = --build-arg http_proxy=$(HTTP_PROXY)\
 	--build-arg CAPI_FLAGS=$(CAPI_FLAGS)\
 	--build-arg VERBOSE_LOGS=$(VERBOSE_LOGS)\
 	--build-arg KONFLUX=$(KONFLUX)\
-	--build-arg ESPEAK=$(ESPEAK)
+	--build-arg ESPEAK=$(ESPEAK)\
 	-e GIT_CONFIG_COUNT="1"\
 	-e GIT_CONFIG_KEY_0="http.version"\
 	-e GIT_CONFIG_VALUE_0="HTTP/1.1"

From 48399bd392e8222f8ad4bfe80d14d4f02eafc1e0 Mon Sep 17 00:00:00 2001
From: Trawinski <dariusz.trawinski@intel.com>
Date: Fri, 4 Sep 2026 19:36:56 +0200
Subject: [PATCH 17/23] fix

---
 Makefile | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/Makefile b/Makefile
index ffe415839c..6ecc8ef3cf 100644
--- a/Makefile
+++ b/Makefile
@@ -244,10 +244,6 @@ BUILD_ARGS = --build-arg http_proxy=$(HTTP_PROXY)\
 	--build-arg VERBOSE_LOGS=$(VERBOSE_LOGS)\
 	--build-arg KONFLUX=$(KONFLUX)\
 	--build-arg ESPEAK=$(ESPEAK)\
-	-e GIT_CONFIG_COUNT="1"\
-	-e GIT_CONFIG_KEY_0="http.version"\
-	-e GIT_CONFIG_VALUE_0="HTTP/1.1"
-# WA for github issues with anonymous access
 
 .PHONY: default docker_build \
 

From f4aeff51c747be2fed479f55098d87094a59545d Mon Sep 17 00:00:00 2001
From: Dariusz Trawinski <dariusz.trawinski@intel.com>
Date: Tue, 8 Sep 2026 10:55:25 +0200
Subject: [PATCH 18/23] update tests

---
 .../continuous_batching/speculative_decoding/README.md | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/demos/continuous_batching/speculative_decoding/README.md b/demos/continuous_batching/speculative_decoding/README.md
index 76f710f501..5fc3ac7425 100644
--- a/demos/continuous_batching/speculative_decoding/README.md
+++ b/demos/continuous_batching/speculative_decoding/README.md
@@ -133,7 +133,7 @@ both in INT4 precision.
 ## Model preparation
 
 Python environment setup:
-```console
+```bat
 # Install regular requirements for OVMS export script
 curl https://raw.githubusercontent.com/openvinotoolkit/model_server/refs/heads/main/demos/common/export_models/export_model.py -o export_model.py
 pip3 install -r https://raw.githubusercontent.com/openvinotoolkit/model_server/refs/heads/main/demos/common/export_models/requirements.txt
@@ -144,11 +144,11 @@ mkdir models
 Run `export_model.py` script to download and quantize the model:
 
 ```bat
-python export_model.py text_generation --source_model Qwen/Qwen3-8B --draft_source_model AngelSlim/Qwen3-8B_eagle3 --draft_eagle3_mode --weight-format int4 --model_repository_path c:\models
+python export_model.py text_generation --model_repository_path c:\models --source_model Qwen/Qwen3-8B --draft_source_model AngelSlim/Qwen3-8B_eagle3 --draft_eagle3_mode --weight-format int4
 ```
 or
-```bash
-python export_model.py text_generation --source_model Qwen/Qwen3-8B --draft_source_model AngelSlim/Qwen3-8B_eagle3 --draft_eagle3_mode --weight-format int4 --target_device CPU --model_repository_path ${HOME}/models
+```text
+python export_model.py text_generation --model_repository_path ${HOME}/models --source_model Qwen/Qwen3-8B --draft_source_model AngelSlim/Qwen3-8B_eagle3 --draft_eagle3_mode --weight-format int4
 ```
 
 Draft model inherits all scheduler properties from the main model.
@@ -187,7 +187,7 @@ models
 ## Server Deployment
 
 :::{dropdown} **Deploying with Docker**
-```bash
+```text
 export GPU_ARGS=$(if ls /dev/dri/render* >/dev/null 2>&1; then echo "--device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1)"; fi)
 docker run -d ${GPU_ARGS} --user $(id -u):$(id -g) --rm -p 8000:8000 -v ${HOME}/models:/models:ro openvino/model_server:weekly \
     --model_path /models/Qwen/Qwen3-8B \

From ffe0429208a2c0d96cad063070255b64377a75c5 Mon Sep 17 00:00:00 2001
From: Dariusz Trawinski <dariusz.trawinski@intel.com>
Date: Tue, 8 Sep 2026 10:58:05 +0200
Subject: [PATCH 19/23] fix

---
 Makefile                      | 2 +-
 ci/build_test_OnCommit.groovy | 7 -------
 2 files changed, 1 insertion(+), 8 deletions(-)

diff --git a/Makefile b/Makefile
index 6ecc8ef3cf..49b7abfab2 100644
--- a/Makefile
+++ b/Makefile
@@ -243,7 +243,7 @@ BUILD_ARGS = --build-arg http_proxy=$(HTTP_PROXY)\
 	--build-arg CAPI_FLAGS=$(CAPI_FLAGS)\
 	--build-arg VERBOSE_LOGS=$(VERBOSE_LOGS)\
 	--build-arg KONFLUX=$(KONFLUX)\
-	--build-arg ESPEAK=$(ESPEAK)\
+	--build-arg ESPEAK=$(ESPEAK)
 
 .PHONY: default docker_build \
 
diff --git a/ci/build_test_OnCommit.groovy b/ci/build_test_OnCommit.groovy
index 4af0470b41..1952349275 100644
--- a/ci/build_test_OnCommit.groovy
+++ b/ci/build_test_OnCommit.groovy
@@ -38,13 +38,6 @@ pipeline {
     agent {
       label 'ovmsbuilder'
     }
-    environment {
-      // Force HTTP/1.1: some proxies mangle git's HTTP/2 smart-protocol responses,
-      // causing "fatal: expected flush after ref listing" on fetch.
-      GIT_CONFIG_COUNT = "1"
-      GIT_CONFIG_KEY_0 = "http.version"
-      GIT_CONFIG_VALUE_0 = "HTTP/1.1"
-    }
     options {
       timeout(time: 4, unit: 'HOURS')
     }

From 2a16689152bcd15d2eebced63c32f370e3be170b Mon Sep 17 00:00:00 2001
From: Dariusz Trawinski <dariusz.trawinski@intel.com>
Date: Tue, 8 Sep 2026 10:58:48 +0200
Subject: [PATCH 20/23] fix

---
 Makefile | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Makefile b/Makefile
index 49b7abfab2..903df474b6 100644
--- a/Makefile
+++ b/Makefile
@@ -244,6 +244,7 @@ BUILD_ARGS = --build-arg http_proxy=$(HTTP_PROXY)\
 	--build-arg VERBOSE_LOGS=$(VERBOSE_LOGS)\
 	--build-arg KONFLUX=$(KONFLUX)\
 	--build-arg ESPEAK=$(ESPEAK)
+	
 
 .PHONY: default docker_build \
 

From b59dd360a3f4f50a384b485b9f21da7f3589774a Mon Sep 17 00:00:00 2001
From: Dariusz Trawinski <dariusz.trawinski@intel.com>
Date: Tue, 8 Sep 2026 10:59:12 +0200
Subject: [PATCH 21/23] fix

---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 903df474b6..aebb6bbdf8 100644
--- a/Makefile
+++ b/Makefile
@@ -244,7 +244,7 @@ BUILD_ARGS = --build-arg http_proxy=$(HTTP_PROXY)\
 	--build-arg VERBOSE_LOGS=$(VERBOSE_LOGS)\
 	--build-arg KONFLUX=$(KONFLUX)\
 	--build-arg ESPEAK=$(ESPEAK)
-	
+
 
 .PHONY: default docker_build \
 

From cf79349f94663858e58c51dc83dac95a15371079 Mon Sep 17 00:00:00 2001
From: Dariusz Trawinski <dariusz.trawinski@intel.com>
Date: Tue, 8 Sep 2026 11:31:17 +0200
Subject: [PATCH 22/23] skip long tests

---
 .../speculative_decoding/README.md                   | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)

diff --git a/demos/continuous_batching/speculative_decoding/README.md b/demos/continuous_batching/speculative_decoding/README.md
index 5fc3ac7425..009117f5ca 100644
--- a/demos/continuous_batching/speculative_decoding/README.md
+++ b/demos/continuous_batching/speculative_decoding/README.md
@@ -133,7 +133,7 @@ both in INT4 precision.
 ## Model preparation
 
 Python environment setup:
-```bat
+```text
 # Install regular requirements for OVMS export script
 curl https://raw.githubusercontent.com/openvinotoolkit/model_server/refs/heads/main/demos/common/export_models/export_model.py -o export_model.py
 pip3 install -r https://raw.githubusercontent.com/openvinotoolkit/model_server/refs/heads/main/demos/common/export_models/requirements.txt
@@ -143,10 +143,6 @@ mkdir models
 
 Run `export_model.py` script to download and quantize the model:
 
-```bat
-python export_model.py text_generation --model_repository_path c:\models --source_model Qwen/Qwen3-8B --draft_source_model AngelSlim/Qwen3-8B_eagle3 --draft_eagle3_mode --weight-format int4
-```
-or
 ```text
 python export_model.py text_generation --model_repository_path ${HOME}/models --source_model Qwen/Qwen3-8B --draft_source_model AngelSlim/Qwen3-8B_eagle3 --draft_eagle3_mode --weight-format int4
 ```
@@ -201,7 +197,7 @@ docker run -d ${GPU_ARGS} --user $(id -u):$(id -g) --rm -p 8000:8000 -v ${HOME}/
 
 Install OVMS as described in the [deployment guide](../../../docs/deploying_server_baremetal.md).
 
-```bat
+```text
 ovms --rest_port 8000 --model_path c:\models\Qwen\Qwen3-8B --model_name Qwen/Qwen3-8B
 ```
 :::
@@ -211,7 +207,7 @@ ovms --rest_port 8000 --model_path c:\models\Qwen\Qwen3-8B --model_name Qwen/Qwe
 
 Send `num_assistant_tokens` to control how many candidates the draft head proposes per target step:
 
-```python
+```text
 from openai import OpenAI
 
 client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
@@ -234,7 +230,7 @@ Setting `num_assistant_tokens: 0` disables drafting for that request; only the t
 
 Tree drafting adds two `GenerationConfig` fields. Setting `tree_depth > 0` switches from chain to tree mode:
 
-```python
+```text
 from openai import OpenAI
 
 client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")

From 3f516e620be9458bf464ce5ffc4ba4b08acf1eb2 Mon Sep 17 00:00:00 2001
From: "Trawinski, Dariusz" <dariusz.trawinski@intel.com>
Date: Thu, 10 Sep 2026 14:20:40 +0200
Subject: [PATCH 23/23] Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---
 demos/continuous_batching/speculative_decoding/README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/demos/continuous_batching/speculative_decoding/README.md b/demos/continuous_batching/speculative_decoding/README.md
index 009117f5ca..67563d4157 100644
--- a/demos/continuous_batching/speculative_decoding/README.md
+++ b/demos/continuous_batching/speculative_decoding/README.md
@@ -80,7 +80,7 @@ pip install vllm --index-url https://wheels.vllm.ai/nightly/cpu --extra-index-ur
 curl https://raw.githubusercontent.com/vllm-project/vllm/refs/heads/main/benchmarks/sonnet.txt -o sonnet.txt
 ```
 
-Run benchmark with 100 requests sent sequentially:
+Run benchmark with 10 requests sent sequentially:
 ```bash
 vllm bench serve --dataset-name sonnet --dataset-path sonnet.txt --backend openai-chat --host localhost --port 8000 --endpoint /v1/chat/completions --max-concurrency 1 --model OpenVINO/Qwen3.8-27B-int4-ov --num-prompts 10
 ```