Repository navigation
Fix several quantizer config bugs (AutoRound override, ModelOpt JSON crash, single-file pre_quantized, config mutation, Nunchaku group_size, dataclass eq/repr) - #15121
Draft
OnePunchMonk wants to merge 8 commits into
Conversation
github-actions
Bot
added
quantization
tests
single-file
fixes-issue
size/L
OnePunchMonk
marked this pull request as ready for review
October 24, 2026 16:57
OnePunchMonk
force-pushed
the
fix/quantizer-config-bugs
branch
from
October 24, 2026 17:07
26b7ca7 to
ba77b98
Compare
merge_quantization_configs looped over every field in the user-passed AutoRoundConfig and overrode any field differing from the checkpoint's, not just backend. The documented AutoRoundConfig(backend="auto") call was silently rewriting a checkpoint's bits/group_size/sym to dataclass defaults. Only backend is a runtime-only override; the rest describe how the checkpoint was quantized. Towards huggingface#15120.
forward_loop is a documented NVIDIAModelOptConfig parameter, but to_dict() had no override and inherited the base deepcopy-of-__dict__ implementation, so the raw callable flowed into modeling_utils.py's json.dumps(...) telemetry call and raised TypeError on every from_pretrained/save_pretrained. Towards huggingface#15120.
…ackends DiffusersAutoQuantizer.from_config() defaults pre_quantized to True when not passed, and the single-file loader never passed it. GGUF and BFL set pre_quantized themselves from their config so were unaffected, but bitsandbytes (4-bit/8-bit), TorchAO, and Quanto need to quantize on load rather than assume already-quantized weights, so single-file loading broke or silently no-opped for them. No dedicated test: exercising this needs a real single-file checkpoint load, which isn't practical as a CPU-only unit test. Towards huggingface#15120.
bnb (4-bit + 8-bit), TorchAO, Quanto, and ModelOpt quantizers aliased modules_to_not_convert-style lists directly off the quantization_config object and then .extend()-ed them in place. Reusing the same config across two from_pretrained calls leaked state between loads (e.g. offloaded module names accumulating on the shared config and getting written into config.json). Copy the list before mutating it. Towards huggingface#15120.
Config validation for the Nunchaku AWQ path only checked precision == "int4"; the actual AWQ kernel call site hardcodes group_size 64. A checkpoint with group_size=128 loaded without error and would silently produce wrong output. The SVDQ path in the same config already validates its group size, so apply the same pattern here. Towards huggingface#15120.
…ng() Every quant config subclass is @dataclass-decorated but declares no dataclass fields beyond the inherited quant_method, so the generated __eq__/__repr__ only see that one field. Two GGUFQuantizationConfig instances with different compute_dtype compared equal, repr() hid every real parameter, and the configs were unhashable. Remove the decorator from subclasses with a custom __init__ and no declared fields. Separately, to_json_string() calls self.to_diff_dict(), which only BitsAndBytesConfig defined; every other config raised AttributeError. Add a base to_diff_dict (no comparable "default" instance exists for these configs in general, so it's just to_dict()). Removing @DataClass exposed GGUFQuantizationConfig/BFLQuantizationConfig's raw torch.dtype compute_dtype to the now-working diff path in to_json_string(), which would have been a new JSON-serialization crash, so both get a to_dict() override that stringifies it. Towards huggingface#15120.
_validate_init_kwargs_in_backends compared init_kwargs_transformers (None when transformers isn't involved for a given backend) against init_kwargs_diffusers (a set), which is always unequal, so the signature-mismatch error fired unconditionally. No dedicated test: this needs no existing test file (no coverage of PipelineQuantizationConfig anywhere in the repo) and the fix is a one-line guard with no side effects to assert on beyond "doesn't raise when it shouldn't", which overlaps the normal pipeline-quantization flow already covered elsewhere. Towards huggingface#15120.
- Two log lines were missing an f-string prefix, so {current_device}
and {target_dtype} printed literally instead of interpolating.
- check_quantized_param_shape's error message had expected and actual
shapes swapped.
- The SCB/weight_format key lookups used param_name.replace("weight",
"SCB"), which corrupts the key for any module name containing
"weight" elsewhere than the trailing ".weight" param (e.g.
"weight_proj.weight" -> "weight_proj.SCB" would actually become
"SCB_proj.weight"). Switch to suffix-safe slicing, matching the
pattern this file already uses elsewhere.
Towards huggingface#15120.
OnePunchMonk
force-pushed
the
fix/quantizer-config-bugs
branch
from
October 24, 2026 17:18
ba77b98 to
fee4c69
Compare
OnePunchMonk
marked this pull request as draft
October 24, 2026 17:21
Author
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes #15120, #15122, #15123, #15124, #15125, #15126.
Six independent correctness bugs in
src/diffusers/quantizers/, each its own commit:backendis now overridable from user-passed args when merging with a checkpoint's quantization config. (AutoRound: user-passed quantization_config silently overwrites checkpoint's quant settings #15120)NVIDIAModelOptConfig.to_dict()now drops theforward_loopcallable so telemetry doesn't crashfrom_pretrained/save_pretrained. (NVIDIAModelOptConfig.to_dict() crashes from_pretrained/save_pretrained when forward_loop is set #15122)from_single_filenow passespre_quantizedthrough from the config instead of defaulting toTrue, fixing bnb/TorchAO/Quanto single-file loading. (from_single_file forces pre_quantized=True, breaking bnb/TorchAO/Quanto single-file loading #15123)modules_to_not_convert-style lists before mutating them. (Quantizer configs are mutated in place, leaking modules_to_not_convert across loads #15124)group_size != 64, matching the kernel. (Nunchaku AWQ config accepts any group_size, but the kernel only supports 64 #15125)@dataclassfrom quant config subclasses with a custom__init__and no declared fields (was producing broken__eq__/__repr__/hashing); added a baseto_diff_dictsoto_json_string()works on every config. (@dataclass on quant configs breaks __eq__/__repr__/hashing, and to_json_string() raises for most configs #15126)PipelineQuantizationConfigvalidation bug that always raised when the transformers-side signature wasNone, an unsafe"weight"→"SCB"key lookup in bnb, two missing f-string prefixes, and a swapped expected/actual shape-mismatch message. (not filed separately — found and fixed alongside Quantizer configs are mutated in place, leaking modules_to_not_convert across loads #15124/@dataclass on quant configs breaks __eq__/__repr__/hashing, and to_json_string() raises for most configs #15126 in the same files)Test plan
Verified on Modal CPU (no GPU used), pre-diff vs. post-diff, for the config-only logic that doesn't need a backend installed:
pip install -e . --no-deps -q python -m pytest -v --no-header tests/quantization/test_quantization_config.pygroup_size128 != 32,forward_loopleaking intoto_dict(), GGUF configs comparing equal / empty repr,QuantoConfighas noto_diff_dict, Nunchaku acceptinggroup_size=128). The 7th is a valid-input case that correctly passes either way.The bnb (
tests/quantization/bnb/test_mixed_int8.py) and TorchAO (tests/quantization/torchao/test_torchao.py) mutation-fix tests are gated behind@require_torch_acceleratorby those existing test classes' own convention, so they weren't run here — see self-review notes below.Self-review notes (per the
self-reviewskill, run against this diff):review-rules.md's "ephemeral context" rule against my own added comments — trimmed two that referenced "the fix" or restated the test name instead of standing alone.@dataclassfrom the config classes breaks anything relying ondataclasses.fields()elsewhere in the codebase — nothing does.@require_torch_acceleratorby the existing test classes' own convention, so they haven't been executed here (no GPU used for this PR) — only verified by reading the code path. Happy to run them on GPU if a maintainer wants that confirmed before merge.tests/quantization/test_quantization_config.pyis a new file because AutoRound, ModelOpt, Nunchaku, and the base config/dataclass logic have no existing test coverage anywhere in the repo to extend instead.Before submitting
self-reviewskill on the diff?Who can review?
Anyone in the community is free to review. Tagging @sayakpaul and @DN6 per the template's guide for general functionalities / loading.