Skip to content

MiniMax H3 mmproj support for Qwen3 VL 32B GGUFs - #473

Open
Nif00 wants to merge 39 commits into
city96:mainfrom
Nif00:minimax-h3-mmproj
Open

MiniMax H3 mmproj support for Qwen3 VL 32B GGUFs#473
Nif00 wants to merge 39 commits into
city96:mainfrom
Nif00:minimax-h3-mmproj

Conversation

@Nif00

@Nif00 Nif00 commented Aug 5, 2026

Copy link
Copy Markdown

No description provided.

radicazz and others added 30 commits January 16, 2026 02:29
- fix tokenizer model_type assignment (was comparison)
- fix cast_bias_weight call signature in embedding forward
- add resolve_full_path helper to centralize path lookup
- add resolve_clip_path with smart fallback logic
- fix mutable default argument in update_folder_names_and_paths
- add FileNotFoundError with descriptive messages
- apply pep8 formatting for consistency
Add _dropdown() and _ensure_dropdown() helper functions to provide
consistent dropdown configuration format expected by ComfyUI frontend
in newer builds.

Update all GGUF loader classes (UnetLoaderGGUF, UnetLoaderGGUFAdvanced,
CLIPLoaderGGUF, DualCLIPLoaderGGUF, TripleCLIPLoaderGGUF,
QuadrupleCLIPLoaderGGUF) to use the new helpers, ensuring proper
frontend dropdown formatting with default value support.
Vibe code. To be reviewed.
If in dynamic mode, load GGUF as a QT.
Refactor this to support the new reconstructability protocol in the
comfy core. This is needed for DynamicVRAM (to support legacy
demotion for fallbacks). Add the logic for dynamic_vram construction.

This is also needed for worksplit multi-gpu branch where the model
is deep-cloned via reconstruction to put the model on two parallel
GPUs.
Refactor this to support the new reconstructability protocol in the
comfy core. This is needed for DynamicVRAM (to support legacy
demotion for fallbacks). Add the logic for dynamic_vram construction.

This is also needed for worksplit multi-gpu branch where the model
is deep-cloned via reconstruction to put the model on two parallel
GPUs.
Factor this out to a helper and implement the new core reconstruction
protocol. Consider the mmap_released flag 1:1 with the underlying model
such that it moves with the base model in model_override.
…XXS, IQ3_S

Implements native PyTorch dequantization functions for lower IQ quant types,
replacing the slow numpy fallback path for models quantized with these formats
(e.g. Unsloth UD quants used as text encoders).

All six new functions are verified against gguf.quants.dequantize() reference.
Co-authored-by: Cursor <cursoragent@cursor.com>
feat: add torch dequantization for IQ1_S, IQ1_M, IQ2_XXS, IQ2_S, IQ3_…
Add Ideogram 4 architecture support for GGUF loading
Handle WeightAdapters in ops.move_patch_to_device
m8rr added a commit to m8rr/ComfyUI-GGUF that referenced this pull request Aug 5, 2026
ChrisColeTech added a commit to ChrisColeTech/ComfyUI-GGUF-Loader that referenced this pull request Aug 12, 2026
Cherry-pick production fixes from city96/ComfyUI-GGUF open PRs:
- city96#472 dequant device-constant cache (major LTX/sampling speedup)
- city96#433 IQ1/IQ2/IQ3 torch dequant (extra TE quants)
- city96#470 QK-norm .scale→.weight (silent NaN/black Flux-compat)
- city96#467 dequant bare nn.Parameters (LTX learnable_registers)
- city96#392 lumina2/zimage pad token shape fix
- city96#456/city96#468 GGMLTensor dtype + dequantize() for core cast path
- city96#461 WeightAdapter-aware move_patch_to_device
- city96#469 force_patch on partial load/unload
- city96#440/city96#436 mistral3 TE, city96#438 qwen35, qwen2 allowlist
- city96#473 partial: Qwen3-VL deepstack mmproj map for MiniMax H3 TE

Skipped mega/draft rewrites (city96#459, full city96#473 LazyGGUFReader, city96#336 Triton).
See PR_BACKPORT.md for the full matrix.
@allanmeng

Copy link
Copy Markdown

I hit the same problem from the Krea2 side (#464) and traced it to the Qwen3-VL mmproj
mapping. Your branch is the only one that wires qwen3vl into gguf_mmproj_loader at
all — but as written the DeepStack keys can't land, so Krea2 still fails. Two defects:

  1. CLIP_VISION_QWEN3_MAP maps "v.deepstast.". The real tensors in
    Qwen3-VL-4B-Instruct-mmproj-bf16.gguf are v.deepstack.{5,11,17}.{norm,fc1,fc2}.{weight,bias}
    (18 tensors, verified with GGUFReader). "deepstast" never matches, so the DeepStack
    tensors pass through unrename­d.

  2. The layer-index remap (ck.5.st.0., ck.16.st.1., …) is bound to a ck. prefix
    that doesn't exist in the file. Even with (1) fixed, you'd emit
    model.visual.deepstack_merger_list.5/11/17.*, but comfy.sd.detect_te_model() only
    probes model.visual.deepstack_merger_list.0.norm.weight — so it still falls back to
    QWEN3_4B and Krea2 still raises "expects 12x2560=30720 features but got 2560".

I have a small fix (loader.py only, +53/-3) that derives the index by sorting the layer
ids present in the file instead of hardcoding 4B/8B, plus an offline check against
qwen3vl_4b_fp8_scaled.safetensors (315/315 keys, cos ≈ 1.000) and a B580 hardware run.
Happy to rebase it on top of your branch if you'd rather keep everything in this PR —
otherwise I'll open it separately as a focused fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants