You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Nunchaku AWQ config accepts any group_size, but the kernel only supports 64 #15125
NunchakuLiteQuantizationConfig's validation for the AWQ path (src/diffusers/quantizers/quantization_config.py) only checks precision == "int4". It never checks group_size. The actual AWQ kernel call site (AWQW4A16Linear) always calls awq_gemm_w4a16_g64_int32 / passes a literal 64 to gemv, so it only ever supports group_size=64.
A checkpoint quantized with group_size=128 loads without any error and silently produces wrong output. The SVDQ path in the same config class does validate its group size against the expected value per precision, so this looks like an oversight in the AWQ branch rather than intentional.
Reproduction
fromdiffusers.quantizers.quantization_configimportNunchakuLiteQuantizationConfigconfig=NunchakuLiteQuantizationConfig(
awq_w4a16={"precision": "int4", "group_size": 128, "targets": ["final_linear"]}
)
# No error raised here, even though the AWQ kernel only supports group_size=64.print(config.awq_w4a16["group_size"]) # 128
System Info
diffusers @ main, commit aaf57d0
Python 3.12, Linux
Who can help?
(leaving blank — no specific quantization reviewer listed in the template's tag guide)
Activity