Dummy LLM and VLM models for testing - #4524
Conversation
b83bc50 to
d9270e2
Compare
There was a problem hiding this comment.
🟡 Changes recommended
A few changes introduce concrete reliability issues in scripts/tests (e.g., Windows download helper swallowing failures and redundant parameterized structured-output tests), which should be fixed before merging.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR introduces deterministic “dummy cyclic” LLM/VLM model fixtures (hosted on HuggingFace as pre-converted OpenVINO IR) and migrates a large portion of OVMS LLM/VLM tests and model-prep scripts to use them, reducing reliance on heavyweight real models for integration testing.
Changes:
- Switch LLM/VLM test graphs and C++ tests to use
mzeglars/dummy-cyclic-gpt2-ovandmzeglars/dummy-cyclic-llava-ov. - Update Linux/Windows model preparation scripts to download the dummy model artifacts from HuggingFace.
- Add
src/test/dummy_genai/with scripts/docs to build, export, and verify the dummy models.
File summaries
| File | Description |
|---|---|
| windows_prepare_llm_models.bat | Downloads dummy HF-hosted OV artifacts for Windows test setup. |
| src/test/llm/visual_language_model/vlm_legacy_regular.pbtxt | Points VLM legacy test graph to dummy VLM model path. |
| src/test/llm/visual_language_model/vlm_cb_regular.pbtxt | Points VLM CB test graph to dummy VLM model path. |
| src/test/llm/visual_language_model/initialization_test.cpp | Updates initialization tests to use dummy VLM model paths. |
| src/test/llm/tokenize_endpoint_test.cpp | Migrates tokenize tests to GPT-2-based dummy tokenizer ids/pad token. |
| src/test/llm/lm_legacy_regular.pbtxt | Points LM legacy test graph to dummy LLM model path. |
| src/test/llm/lm_cb_regular.pbtxt | Points LM CB test graph to dummy LLM model path. |
| src/test/llm/lm_cb_regular_queue.pbtxt | Points queue-graph LM CB fixture to dummy LLM model path. |
| src/test/llm/llmnode_test.cpp | Migrates many HTTP tests to dummy model behavior/deterministic stop tokens. |
| src/test/dummy_genai/verify_model.py | Adds verification for HF + OpenVINO dummy model generation behavior. |
| src/test/dummy_genai/test.py | Minimal manual sanity script for LLMPipeline/VLMPipeline. |
| src/test/dummy_genai/requirements.txt | Local build/export deps for dummy model generation and verification. |
| src/test/dummy_genai/README.md | Documents purpose, behavior, and regeneration workflow for dummy models. |
| src/test/dummy_genai/model_card_vlm.md | Model card content for publishing the dummy VLM to HF. |
| src/test/dummy_genai/model_card_llm.md | Model card content for publishing the dummy LLM to HF. |
| src/test/dummy_genai/export_to_ov.py | Scripted HF→OpenVINO export wrapper for the dummy LLM. |
| src/test/dummy_genai/create_vlm_model.py | Constructs the dummy LLaVA-style VLM with deterministic text backbone. |
| src/test/dummy_genai/create_model.py | Constructs the dummy GPT-2 LLM with deterministic next-token mapping. |
| src/test/dummy_genai/build.sh | End-to-end builder for dummy LLM (venv → HF → OV → verify → zip). |
| src/test/dummy_genai/build_vlm.sh | End-to-end builder for dummy VLM (venv → HF → OV → tokenizer → zip). |
| src/test/dummy_genai/.gitignore | Keeps generated model outputs/venv artifacts out of git. |
| prepare_llm_models.sh | Downloads dummy OV artifacts for Linux test setup and removes old VLM download. |
| Makefile | Excludes dummy_genai path from style tooling traversal and adjusts cpplint invocation. |
Review details
Suppressed comments (2)
src/test/llm/llmnode_test.cpp:1461
- This structured-output test is parameterized but always targets the fixed model "lm_cb_with_tool_parser", so it runs redundantly for each parameter value. Add a skip guard (or move to a non-parameterized fixture) to avoid multiplying runtime.
TEST_P(LLMFlowHttpTestParameterized, unaryStructuredOutputBadSchema) {
// Structured output needs real guided generation, not the dummy cyclic model.
std::string requestBody = R"(
src/test/llm/llmnode_test.cpp:1499
- This structured-output test is parameterized but always targets the fixed model "lm_cb_with_tool_parser", so it runs redundantly for each parameter value. Add a skip guard (or move to a non-parameterized fixture) to avoid multiplying runtime.
TEST_P(LLMFlowHttpTestParameterized, unaryStructuredOutputNonOpenAI) {
// Structured output needs real guided generation, not the dummy cyclic model.
std::string requestBody = R"(
- Files reviewed: 23/23 changed files
- Comments generated: 6
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| cmd = ( | ||
| f"optimum-cli export openvino" | ||
| f" --model {hf_dir}" | ||
| f" --task text-generation-with-past" | ||
| f" --weight-format {weight_format}" | ||
| f" {extra}" | ||
| f" {ov_dir}" | ||
| ) |
| # if [ -f "$1/$DUMMY_VLM_MODEL/$TOKENIZER_FILE" ]; then | ||
| # echo "Model file $1/$DUMMY_VLM_MODEL/$TOKENIZER_FILE exists. Skipping downloading models." | ||
| # else | ||
| pip3 install --upgrade typer==0.25.1 |
| TEST_P(LLMFlowHttpTestParameterized, unaryStructuredOutput) { | ||
| auto params = GetParam(); | ||
| // Structured output needs real guided generation, not the dummy cyclic model. | ||
| std::string requestBody = R"( |
No description provided.