Skip to content

Dummy LLM and VLM models for testing - #4524

Open
mzegla wants to merge 6 commits into
mainfrom
dummy_genai_models
Open

Dummy LLM and VLM models for testing#4524
mzegla wants to merge 6 commits into
mainfrom
dummy_genai_models

Conversation

@mzegla

@mzegla mzegla commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

No description provided.

@mzegla
mzegla requested a review from rasapala September 8, 2026 15:05
@mzegla
mzegla force-pushed the dummy_genai_models branch from b83bc50 to d9270e2 Compare September 9, 2026 11:44
@mzegla
mzegla marked this pull request as ready for review September 9, 2026 11:50
Copilot AI lite review requested due to automatic review settings September 9, 2026 11:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A few changes introduce concrete reliability issues in scripts/tests (e.g., Windows download helper swallowing failures and redundant parameterized structured-output tests), which should be fixed before merging.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR introduces deterministic “dummy cyclic” LLM/VLM model fixtures (hosted on HuggingFace as pre-converted OpenVINO IR) and migrates a large portion of OVMS LLM/VLM tests and model-prep scripts to use them, reducing reliance on heavyweight real models for integration testing.

Changes:

  • Switch LLM/VLM test graphs and C++ tests to use mzeglars/dummy-cyclic-gpt2-ov and mzeglars/dummy-cyclic-llava-ov.
  • Update Linux/Windows model preparation scripts to download the dummy model artifacts from HuggingFace.
  • Add src/test/dummy_genai/ with scripts/docs to build, export, and verify the dummy models.
File summaries
File Description
windows_prepare_llm_models.bat Downloads dummy HF-hosted OV artifacts for Windows test setup.
src/test/llm/visual_language_model/vlm_legacy_regular.pbtxt Points VLM legacy test graph to dummy VLM model path.
src/test/llm/visual_language_model/vlm_cb_regular.pbtxt Points VLM CB test graph to dummy VLM model path.
src/test/llm/visual_language_model/initialization_test.cpp Updates initialization tests to use dummy VLM model paths.
src/test/llm/tokenize_endpoint_test.cpp Migrates tokenize tests to GPT-2-based dummy tokenizer ids/pad token.
src/test/llm/lm_legacy_regular.pbtxt Points LM legacy test graph to dummy LLM model path.
src/test/llm/lm_cb_regular.pbtxt Points LM CB test graph to dummy LLM model path.
src/test/llm/lm_cb_regular_queue.pbtxt Points queue-graph LM CB fixture to dummy LLM model path.
src/test/llm/llmnode_test.cpp Migrates many HTTP tests to dummy model behavior/deterministic stop tokens.
src/test/dummy_genai/verify_model.py Adds verification for HF + OpenVINO dummy model generation behavior.
src/test/dummy_genai/test.py Minimal manual sanity script for LLMPipeline/VLMPipeline.
src/test/dummy_genai/requirements.txt Local build/export deps for dummy model generation and verification.
src/test/dummy_genai/README.md Documents purpose, behavior, and regeneration workflow for dummy models.
src/test/dummy_genai/model_card_vlm.md Model card content for publishing the dummy VLM to HF.
src/test/dummy_genai/model_card_llm.md Model card content for publishing the dummy LLM to HF.
src/test/dummy_genai/export_to_ov.py Scripted HF→OpenVINO export wrapper for the dummy LLM.
src/test/dummy_genai/create_vlm_model.py Constructs the dummy LLaVA-style VLM with deterministic text backbone.
src/test/dummy_genai/create_model.py Constructs the dummy GPT-2 LLM with deterministic next-token mapping.
src/test/dummy_genai/build.sh End-to-end builder for dummy LLM (venv → HF → OV → verify → zip).
src/test/dummy_genai/build_vlm.sh End-to-end builder for dummy VLM (venv → HF → OV → tokenizer → zip).
src/test/dummy_genai/.gitignore Keeps generated model outputs/venv artifacts out of git.
prepare_llm_models.sh Downloads dummy OV artifacts for Linux test setup and removes old VLM download.
Makefile Excludes dummy_genai path from style tooling traversal and adjusts cpplint invocation.
Review details

Suppressed comments (2)

src/test/llm/llmnode_test.cpp:1461

  • This structured-output test is parameterized but always targets the fixed model "lm_cb_with_tool_parser", so it runs redundantly for each parameter value. Add a skip guard (or move to a non-parameterized fixture) to avoid multiplying runtime.
TEST_P(LLMFlowHttpTestParameterized, unaryStructuredOutputBadSchema) {
    // Structured output needs real guided generation, not the dummy cyclic model.
    std::string requestBody = R"(

src/test/llm/llmnode_test.cpp:1499

  • This structured-output test is parameterized but always targets the fixed model "lm_cb_with_tool_parser", so it runs redundantly for each parameter value. Add a skip guard (or move to a non-parameterized fixture) to avoid multiplying runtime.
TEST_P(LLMFlowHttpTestParameterized, unaryStructuredOutputNonOpenAI) {
    // Structured output needs real guided generation, not the dummy cyclic model.
    std::string requestBody = R"(
  • Files reviewed: 23/23 changed files
  • Comments generated: 6
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread windows_prepare_llm_models.bat
Comment thread Makefile Outdated
Comment on lines +57 to +64
cmd = (
f"optimum-cli export openvino"
f" --model {hf_dir}"
f" --task text-generation-with-past"
f" --weight-format {weight_format}"
f" {extra}"
f" {ov_dir}"
)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add quotes.

Comment thread prepare_llm_models.sh Outdated
# if [ -f "$1/$DUMMY_VLM_MODEL/$TOKENIZER_FILE" ]; then
# echo "Model file $1/$DUMMY_VLM_MODEL/$TOKENIZER_FILE exists. Skipping downloading models."
# else
pip3 install --upgrade typer==0.25.1

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove duplicate.

Comment thread src/test/dummy_genai/create_vlm_model.py Outdated
Comment on lines 1410 to 1412
TEST_P(LLMFlowHttpTestParameterized, unaryStructuredOutput) {
auto params = GetParam();
// Structured output needs real guided generation, not the dummy cyclic model.
std::string requestBody = R"(
@mzegla
mzegla requested a review from dtrawins September 9, 2026 13:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants