Multimodal Image Search on COCO

Text-to-image retrieval system comparing three multimodal encoders on COCO Captions:

• CLIP ViT-B/32 (openai/clip-vit-base-patch32)
• SigLIP 2 So400m/14-384 (google/siglip2-so400m-patch14-384)
• Qwen3-VL-Embedding-2B (Qwen/Qwen3-VL-Embedding-2B, INT8 via bitsandbytes)

Vector index: Qdrant (one collection per model, cosine similarity).

See REPORT.md for results.

Requirements

• Linux + NVIDIA GPU (8+ GB VRAM, tested on RTX 3070)
• CUDA 12.x
• Python 3.11
• Docker (for Qdrant)
• ~15 GB free disk for COCO + HF model weights

Quickstart

# 1. Start Qdrant
docker compose up -d

# 2. Python environment
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python -m spacy download en_core_web_sm

# 3. Download COCO Karpathy split
python scripts/01_download_data.py

# 4. Build indexes for all three models
for model in clip siglip2 qwen3vl; do
    python scripts/02_build_index.py --model $model
done

# 5. Run evaluation
for model in clip siglip2 qwen3vl; do
    python scripts/03_run_eval.py --model $model
    python scripts/04_analyze_subsets.py --model $model
done

# 6. Generate report figures and tables
python scripts/05_make_report.py

Results land in reports/figures/ and REPORT.md is updated with the metrics table.

Project layout

src/         # library code: data, models, indexer, search, eval, viz, config
scripts/     # numbered CLI entry points (01..05)
tests/       # smoke tests
reports/     # eval JSON + figures/ (PNG + CSV)
data/        # COCO images and captions (gitignored)