Repository navigation
set_adapters applies component weights only to the first adapter when broadcasting a dictionary #15127
Description
Describe the bug
When set_adapters() activates multiple LoRA adapters, a single dictionary of component weights only applies to the first adapter. The remaining adapters use the default weight of 1.0 without warning.
For example:
pipe.set_adapters(["style_a", "style_b"], {"unet": 0.5})This sets the weights to 0.5 and 1.0 instead of 0.5 for both adapters. Setting the weight to 0 can also leave the second adapter enabled. Passing a scalar weight or separate dictionaries for each adapter works correctly.
This affects setups where the same component weights should apply to multiple LoRA adapters. I reproduced the wrong scaling and its effect on generated latents using a tiny SDXL pipeline with synthetic LoRA adapters. This seems to happen because LoraBaseMixin.set_adapters repeats the same dictionary using list multiplication. Then weights.pop(component, None) removes the shared entry when processing the first adapter.
I'd be happy to fix this and add CPU regression tests if the scope looks appropriate. I'm also happy to be assigned the issue if that's possible. I used codex to help review the code.
Reproduction
Tested against Diffusers main at commit aaf57d03a3edfbd9de457d2872417a92f149a91d.
import torch
from diffusers import StableDiffusionXLPipeline
from peft import LoraConfig
from peft.tuners.tuners_utils import BaseTunerLayer
torch.set_num_threads(2)
torch.manual_seed(123)
pipe = StableDiffusionXLPipeline.from_pretrained(
"hf-internal-testing/tiny-stable-diffusion-xl-pipe",
revision="fa06b5c3d6d45fabd978486616c6dae068519e85",
)
pipe.set_progress_bar_config(disable=True)
names = ["style_a", "style_b"]
for name in names:
pipe.unet.add_adapter(
LoraConfig(
r=2,
lora_alpha=2,
target_modules=["to_q", "to_k", "to_v", "to_out.0"],
init_lora_weights=False,
),
adapter_name=name,
)
layer = next(
m for m in pipe.unet.modules()
if isinstance(m, BaseTunerLayer)
)
def generate(weights):
pipe.set_adapters(names, adapter_weights=weights)
print("scales:", {
name: layer.scaling[name] for name in names
})
return pipe(
"a dog",
height=64,
width=64,
num_inference_steps=2,
guidance_scale=1.0,
generator=torch.Generator().manual_seed(0),
output_type="latent",
).images
explicit = generate([{"unet": 0.5}, {"unet": 0.5}])
scalar = generate(0.5)
broadcast = generate({"unet": 0.5})
torch.testing.assert_close(scalar, explicit, rtol=0, atol=0)
print(
"scalar vs explicit max diff:",
(scalar - explicit).abs().max().item(),
)
print(
"dictionary vs explicit max diff:",
(broadcast - explicit).abs().max().item(),
)Logs
scales: {'style_a': 0.5, 'style_b': 0.5}
scales: {'style_a': 0.5, 'style_b': 0.5}
scales: {'style_a': 0.5, 'style_b': 1.0}
scalar vs explicit max diff: 0.0
dictionary vs explicit max diff: 0.1969885230064392
I also reproduced this with SDXL text-encoder adapters and a tiny Flux transformer. These tests used small model components and synthetic adapters, not production-model quality evaluations.System Info
- Diffusers:
0.42.0.dev0(main,aaf57d03) - Python:
3.12.1 - OS: Linux x86_64
- PyTorch:
2.14.1+cpu - PEFT:
0.21.2 - Transformers:
5.20.2 - Accelerate:
1.15.0 - Safetensors:
0.8.0 - GPU: No
Activity
sayakpaul commentedon Oct 25, 2026
Thanks for the detailed issue thread! Yes, please open a PR to contribute the fix, we'd be grateful! Please also tag @BenjaminBossan.