Add Ideogram support and improve BF16 dequantization handling - #459
Open
molbal wants to merge 72 commits into
Open
Add Ideogram support and improve BF16 dequantization handling#459molbal wants to merge 72 commits into
molbal wants to merge 72 commits into
Conversation
|
Great! I successfully ran it, but my device doesn't support bf16; it gets converted to fp32 computation, which makes it very slow. Can you make it run on my device in fp16? [INFO] got prompt |
Author
|
Hi @yu234567 - try now. It should work better now, can you verify please? |
Thank you so much, it worked! |
|
classifies Qwen3-VL-8B-Instruct q4_0 quant as _k quant and errors out ? is this expected or what quantization are you running for the te |
- Updated IMG_ARCH_LIST to include 'krea2'. - Introduced ModelKrea2 class with architecture details and tensor handling. - Enhanced convert_file function to support quantization types. - Added tools/convert_krea2_gguf.py for batch conversion of Krea-2 models to multiple GGUF quant levels.
…ation support limitations
…tensors and skip 0-dim scalars during GGUF conversion
Author
Fixed that was my bad |
Feature/molbal dynamic gguf
Add DynamicVRAM-aware GGUF loading and node variants. loader.py: attach mmap-backed file slices, preserve custom GGUF quant configs, and switch to dynamic handling (including BF16/quant paths). nodes.py: add DynamicVRAM loaders and GGUFModelPatcherDynamic, plus helpers to load Unet/CLIP via DynamicVRAM. quant_ops.py: implement GGML quantized tensor layout for Comfy's QuantizedTensor system. tools/convert.py: preserve safetensors metadata, introduce keys_noquant, and avoid quantizing specified small tensors. Add documentation pages for the new Dynamic VRAM loader nodes.
Add Dynamic VRAM support for GGUF loaders
## `lcpp.patch` changes (llama.cpp)
All additions follow the existing pattern used for the other image-model architectures (Flux, SD3, Aura, LTXV, HyVid, Wan, HiDream, Cosmos, Lumina2) --five/six standard insertion points plus two architecture-specific refinements (one a required correctness fix, one an optional quality tuning knob).
### Standard architecture registration (5 places in `src/llama.cpp`)
1. **`enum llm_arch`**: add `LLM_ARCH_KREA2,`
2. **`LLM_ARCH_NAMES` map**: add `{ LLM_ARCH_KREA2, "krea2" },`
3. **`LLM_TENSOR_NAMES` map**: add `{ LLM_ARCH_KREA2, {}},`
4. **`llm_load_hparams`**, "disable LLM metadata for image models" switch: add
`case LLM_ARCH_KREA2:` alongside the other image archs (skips LLM-style
hparam parsing, since this isn't a language model).
5. **`llama_model_quantize_internal`**, "rules for image models" section: add
a `LLM_ARCH_KREA2` block excluding the small/non-repeating tensors from
quantization (kept at their original F32/F16 precision):
```cpp
if (model.arch == LLM_ARCH_KREA2) {
image_model = true;
quantize &= name.find("first.") == std::string::npos;
quantize &= name.find("last.") == std::string::npos;
quantize &= name.find("tproj.") == std::string::npos;
quantize &= name.find("tmlp.") == std::string::npos;
quantize &= name.find("txtmlp.") == std::string::npos;
quantize &= name.find("txtfusion.projector.") == std::string::npos;
}
```
### The one architecture-specific fix: `txtfusion.projector.weight`
**Symptom:** ComfyUI failed to load the *quantized* GGUF (worked fine for the
BF16 intermediate file) with:
```
File ".../comfy/model_detection.py", line 917, in detect_unet_config
dit_config["txtlayers"] = state_dict['{}txtfusion.projector.weight'.format(key_prefix)].shape[1]
IndexError: tuple index out of range
```
**Root cause:** `txtfusion.projector.weight` has shape `(1, 12)` in the original checkpoint. When `llama-quantize` loads this into an actual `ggml_tensor` and writes it back out via `gguf_add_tensor()`, the tensor's dimensionality is derived from `ggml_n_dims()`, which scans the `ne[]` array from the highest index downward and drops trailing 1s. Since GGML's `ne` order is the reverse of the original torch shape, `(1, 12)` becomes `ne = [12, 1, 1, 1]` -- and `ne[1] == 1` gets trimmed away, collapsing the
tensor to 1D. The BF16 GGUF written directly by `convert.py`'s Python `gguf` library does *not* have this problem (it just serializes the given shape faithfully); the collapse is specific to the C++ `llama-quantize` step.
This is the same class of bug already handled for other architectures (SD3's `pos_embed`, AuraFlow's `positional_encoding`/`register_tokens`, Wan's `.modulation`/`img_emb.emb_pos`), all of which use a helper added to GGML for exactly this purpose: `gguf_set_tensor_ndim()`.
**Fix**, added to the same tensor-writing loop in `llama_model_quantize_internal` where the other archs' fixes live (right after the `LLM_ARCH_WAN` block):
```cpp
// Krea2's txtfusion.projector.weight has shape (1, 12) -- the leading
// dim of 1 becomes a trailing `ne` entry and gets truncated by
// ggml_n_dims() unless corrected explicitly.
if (model.arch == LLM_ARCH_KREA2) {
const std::string name = ggml_get_name(tensor);
if (name == "txtfusion.projector.weight" && tensor->ne[1] == 1) {
const int n_dim = 2;
gguf_set_tensor_ndim(ctx_outs[i_split], "txtfusion.projector.weight", n_dim);
LLAMA_LOG_INFO("\n%s: Correcting txtfusion.projector.weight shape for Krea2: [key:%s]\n", __func__, tensor->name);
}
}
```
Note the difference from the SD3/Aura/Wan precedents: those check `tensor->ne[2] == 1` and restore `n_dim = 3` because their tensors were originally 3D (e.g. `(1, H, W)`). Krea2's `txtfusion.projector.weight` is only 2D (`(1, 12)`), so the check is `tensor->ne[1] == 1` and the restored dimension is `n_dim = 2`.
**No other Krea2 tensors needed this fix.** All other tensors are either purely 1D, or 2D with no leading-1 dimension (verified against actual shapes pulled from the checkpoint).
### Optional refinement: `to_v` attention quant-type rules
llama.cpp's `img_tensor_get_type()` has a block of special-case rules that give the attention value projection ("to_v") preferential K-quant subtype treatment at certain `ftype`s, already applied to the other image archs (Flux, SD3, etc. -- matched via names like `.to_v.weight`, `.attn.w1v.weight`, `.attn.w2v.weight`). Krea2's equivalent tensor, `.attn.wv.weight`, wasn't originally in that match list. Added:
```cpp
if ( // Rules for to_v attention
(name.find("attn_v.weight") != std::string::npos) ||
(name.find(".to_v.weight") != std::string::npos) ||
(name.find(".v.weight") != std::string::npos) ||
(name.find(".attn.w1v.weight") != std::string::npos) ||
(name.find(".attn.w2v.weight") != std::string::npos) ||
+ (name.find(".attn.wv.weight") != std::string::npos) ||
(name.find("_attn.v_proj.weight") != std::string::npos)
){
```
Unlike the `txtfusion.projector.weight` fix above, this isn't a correctness issue -- without it, Krea2 quantizes and runs fine, `.attn.wv.weight` just gets the generic type-selection path instead of the to_v-specific one. It's a size/quality tuning knob, initially left out to keep the first working version minimal, then added and verified once the base support was confirmed stable.
Updated read_tensors.py to support tensor comparison and improved shape checking.
Enhance tensor reading and comparison functionality
Update tensor handling for Krea2 architecture
Added key_matches function to match tensor names against patterns and validate key patterns for model architectures.
Implement key_matches function for tensor name validation
…ization defaults in conversion script
Add icon.png to the repository and update pyproject.toml's [tool.comfy] Icon entry to point to the raw GitHub URL for the icon (refs/heads/main). This enables ComfyUI to display the package icon.
Introduce TARGET_SIZE-based GGUF quantization and progress reporting. Added planning and conversion logic (tools/convert.py) to select per-tensor quant types to meet a target size, CLI --max-size-mb support, and unit tests for targeted quantization. Propagate progress callbacks into safetensors/pt readers and GGUF loaders (loader.py, nodes.py) and add GGUFLoadProgress to coordinate ComfyUI progress bars for multi-file loads. Make ops compatible with mixed Q8_CR/Q4_0 GGUFs by materializing GGML weights before mixed-precision loading (ops.py). Update README and register a TargetedQuantizationGGUF node; include node.zip and tests.
…mode
Fix GGMLTensor.dtype crash on inference-mode tensors ("Inference tensors do not track version counter")
|
Nice work @molbal! Any particular reason for |
Author
|
Thanks @joeblowma! nodes.zip is created by comfy-cli when I push the nodes to Comfy Registry, but I forgot to remove it. (I have since added a github action to do it, but it seems I forgot to remove the zip and accidentally committed it) |
Author
|
But anyways my fork grew out of proportions and for it to be merged back I would need to create a separate fork without all the changes I did (e.g. new node names to avoid conflict, changed readme, etc) |
Ensure force_patch_weights is set when partially loading/unloading mo…
Implement dequantization for IQ2 and IQ3 GGML quant types in dequant.py (new grid/ksigns helpers, IQ3_S, IQ3_XXS, IQ2_S, IQ2_XS, IQ2_XXS dequant functions and mapping entries). Add numpy dependency usage for grid extraction and handle sign-byte tables. Update README with instructions for pruned Qwen3-VL-32B MiniMax H3 text encoders and CLIP loader selection. Add unit tests to verify the pruned-32B CLIP loader marker injection and that the new IQ quant types dequantize to the correct shape/dtype and are present in dequantize_functions.
Add CLIP_VISION_QWEN3_MAP and mmproj loading/mapping for Qwen3-VL vision towers. gguf_mmproj_loader will map deepstack tensors into the ComfyUI visual naming; gguf_clip_loader now loads mmproj for qwen2vl/qwen3vl, handles MiniMax H3's differing 'model.visual.*' vs 'visual.*' prefixes, and prevents misclassification when the visual tower is missing. Tests updated/added to validate mapping and mmproj behavior. README updated with instructions to place *-mmproj-BF16.gguf beside the text encoder.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move the deepstack handling in gguf_mmproj_loader to run after 4D->5D stacking so temporal patch embeddings are concatenated before applying the CLIP_VISION_QWEN3_MAP. Add a unit test (test_qwen3vl_mmproj_stacks_temporal_patch_embeddings) that mocks gguf_sd_loader, returns two patch tensors and a deepstack key, and verifies the resulting stacked weight shape and contents. Changes in loader.py and tests/test_targeted_quantization.py.
Handle a known misspelling from older llama.cpp Qwen3-VL exporters by adding a CLIP_VISION_QWEN3_MAP entry mapping "v.deepstast." to "model.visual.deepstack_merger_list.", and update gguf_mmproj_loader to detect either "deepstack" or "deepstast" keys. Tests in tests/test_targeted_quantization.py were updated to cover the misspelling and ensure mappings and patch-embedding stacking still work as expected.
Text encoder + vision tower support. Image-to-text requires a matching mmproj file.
Add Qwen3.5 support
Implement support for the experimental Q4_CR quantization format with group-wise INT4 weights (W4A16), backed by comfy_kitchen's AWQ GEMV kernel. Update `loader.py`, `tools/convert.py`, and `ops.py` to handle `int4_cr` quantized weight layouts with associated metadata. Add dequantization logic, new tests in `test_targeted_quantization.py`, and README documentation for usage.
Introduce the new Q4_CR_W4A4 quantization format with support for fast INT4 weights backed by comfy_kitchen's ConvRot tensor-core MMA (W4A4). Update `loader.py`, `tools/convert.py`, and `ops.py` to process the `int4_cr_w4a4` layout, which includes pre-rotated weights and per-row scales. Retire the older W4A16-backed Q4_CR format, adding compatibility checks and migration guidance. Extend `test_targeted_quantization.py` with tests for quantization/dequantization and end-to-end workflows. Update requirements and HTML conversion interface to enable Q4_CR_W4A4.load and dequant fallback logic Enhance `test_targeted_quantization.py` to verify behavior of Q4_CR_W4A4 ops under offload and value-patching scenarios. Update `ops.py` to optimize weight relocation and dequantization paths by caching device-resident weights and reducing redundant transfers. Refactor README for clarity on INT4 quantization setup and performance tradeoffs.
… proper fallback to full-precision when active. Add new tests for patch application and behavior under dynamic VRAM offload. Update `ops.py` and README for clarity on patching logic and performance implications.
… proper fallback to full-precision when active. Add new tests for patch application and behavior under dynamic VRAM offload. Update `ops.py` and README for clarity on patching logic and performance implications.
…paration, and performance logging. Update tests, README, and ops for enhanced quantization workflows and diagnostic capabilities.
…rioritization Replace `modules` with `active_modules` to identify layers requiring fusion. Introduce device-aware fusion, using GPU when available for improved performance, with CPU as the fallback. Add tqdm progress for layer processing and safeguard against partial fusions by evicting incomplete caches on failure. Update tests, README, and ops for detailed behavior and diagnostics.
…Update fallback logic to avoid INT4 delta loss during re-quantization. Add tests for different LoRA behaviors, device-based cache eviction, and fused weight movement tracking. Update README and ops for clarity on patch handling.
# Conflicts: # README.md # ops.py # tests/test_targeted_quantization.py
…LoRA patching. Extend tests for `Q4_CR_W4A4` fallback and INT4 matrix behavior. Refactor README for clarity and streamline instructions.
…for INT4/INT8 GGUF support.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This adds support for Ideogram GGUF models.
What Changed
ideogramto the supported image GGUF architectures.Notes
Tested on Windows 11, Python version: 3.12.11 (main, Jul 23 2025, 00:32:20) [MSC v.1944 64 bit (AMD64)] [INFO] Total VRAM 8192 MB, total RAM 48394 MB
[INFO] pytorch version: 2.12.0+cu130
[INFO] Set vram state to: LOW_VRAM
[INFO] Device: cuda:0 NVIDIA GeForce RTX 3080 Laptop GPU
Tested with Q4_0 gguf from https://huggingface.co/leejet/ideogram-4-GGUF
Other GGUF quant types still use the existing dequant paths.