Repository navigation
fix(lora): reject diffusers LoRAs of other DiTs by Anima's attention widths - #10223
Pfannkuchensack wants to merge 3 commits into
Conversation
…widths Diffusers-named LoRAs of LTX-2, LTX-Video, CogView4, Mochi, CogVideoX, PixArt and Sana share Anima's `transformer_blocks.N.attn1/attn2` names and were identified as Anima by name alone, failing only at generation. The Anima probe now checks the input width of those attention projections: 2048 for self-attention and the cross-attention query/output, 1024 for the cross-attention key/value, the same in every Anima variant. Kohya and native layouts are unaffected. Closes invoke-ai#10081
github-actions
Bot
added
python
joshistoast
left a comment
joshistoast
left a comment
There was a problem hiding this comment.
No material runtime regression found. The approach is focused, checks shapes without tensor computation, and improves installation behavior by rejecting incompatible adapters earlier. Native/Kohya handling appears preserved.
| Diffusers LoRAs of other transformers that use the same layer names (for example LTX-2, LTX-Video, CogView4, | ||
| PixArt, Sana) are not recognized as Anima, because their attention widths do not match Anima's. |
There was a problem hiding this comment.
[P3] Documentation overstates protection.
Says LTX-Video LoRAs are not recognized as Anima. Self-attention-only LTX-Video LoRAs still pass because both use width 2048. Qualify this claim and explain the remaining ambiguity.
There was a problem hiding this comment.
Fixed in 2e0c5d1. LTX-Video is out of the example list. The page now says that its self-attention is as wide as Anima's, that a LoRA training the cross-attention is refused, and that one training only the self-attention still installs as Anima.
An LTX-Video LoRA that trains only self-attention has Anima's widths and still installs as Anima.
| self-attention is as wide as Anima's, so only its cross-attention tells them apart. An LTX-Video LoRA that trains | ||
| the cross-attention is refused; one that trains only the self-attention installs as Anima. |
There was a problem hiding this comment.
[P3] Cross-attention rejection claim is still too broad.
The text says LTX-Video adapters that train cross-attention are refused. Adapters targeting only cross-attention query or output layers still pass: their input width matches Anima’s. Only key/value layers expose the mismatch. Clarify that adapters without cross-attention key/value layers can still install as Anima. The detection limitation already existed; the misleading wording is new.
Separately, backend guidance requires updating 03.model-format-support.mdx when identification acceptance changes. That update remains missing.
There was a problem hiding this comment.
Both fixed in 20acceb.
- The family page now names the layers that differ: only the cross-attention key and value layers (
attn2.to_k,attn2.to_v) read a wider text embedding in LTX-Video. An LTX-Video LoRA with those layers is not recognized as Anima. One without them, including one that trains only the cross-attention query or output, still installs as Anima. 03.model-format-support.mdxgets a paragraph under the LoRA table. It says that Anima's Diffusers PEFT LoRAs are told apart by their attention widths, that other files install as unknown models, and it states the same LTX-Video limit with a link to the family page.02.model-families.mdxnames no Anima LoRA format, so it stays as is.
Only the cross-attention key/value layers differ in width; the format support page now notes the width check.
joshistoast
left a comment
joshistoast
left a comment
There was a problem hiding this comment.
Some suggested re-wording, otherwise looks good.
| Sana and others). A file whose attention widths are not Anima's is not recognized as Anima and installs as an | ||
| unknown model. LTX-Video LoRAs without cross-attention key and value layers have Anima's widths and still install as |
There was a problem hiding this comment.
[P3] Rejected by Anima does not always mean “unknown”.
The new wording says adapters with different attention widths install as unknown. Supported LTX-2 adapters containing video and audio layers still identify correctly as LTX-2. This could mislead users into thinking those adapters are unsupported.
Suggested wording: “These files are not recognized as Anima. If no other supported model family matches, they install as unknown.”
Summary
Diffusers-named LoRAs of other DiTs use the same module names as Anima's diffusers layout (
transformer_blocks.N.attn1/attn2), so the Anima probe accepted them by name alone. They installed as Anima and failed only at generation (Unsupported diffusers Anima LoRA layer, or skipped layers on shape mismatch).The Anima probe now also checks the input width of those attention projections, read from low-rank, LoHA, full and LoKR (factor product) layers:
attn1.to_q/k/v/out.0,attn2.to_q,attn2.to_out.0: 2048attn2.to_k/v(text context): 1024These widths hold for every Anima variant (base, 2.9B, 3.8B v1.1; checked against the checkpoints). A mismatch raises
NotAMatchErrornaming the layer and its width, so such files now end up as unknown instead of the wrong family. Kohya and nativeblocks.N.layouts are not affected. A layer that carries no width (a lone DoRA magnitude) does not count.The width reader used by the Qwen-Image-2.1 detection now takes the projection pattern as a parameter, so both probes share it. Qwen-Image behavior is unchanged.
Issue repro, before → after:
transformer.base_model.model.base_model.model.transformer.Related Issues / Discussions
Closes #10081
Follow-up to #10076. LTX-2's own gaps for these prefixes (e.g. video+audio under
base_model.model.still landing on unknown) are #10080.QA Instructions
CPU only, no models needed.
uv run --no-sync pytest tests/backend/model_manager/configs/ tests/model_identification/test_identification.py(LFS fixtures checked out): 1751 passed, 1 skipped.test_anima_diffusers_lora_config.py: realistic Anima fixture (withattn2.to_k= 1024) under four prefixes. Video-only LTX-2 and CogView4, Mochi, CogVideoX 2B/5B, PixArt-Σ, Sana 600M/1600M and LTX-Video, with widths from their diffusers configs, all come out unknown. SD 3.5 Medium stays SD 3. LoKR with matching and mismatching factor products. A width-less DoRA key. A Kohya regression case.test_ltx2_lora_probe_independence.py: the Anima probe rejects a video-only LTX-2 LoRA under every prefix it accepts.test_sd3_lora_probe_independence.py: the CogView4 test expected the diffusers spelling to be claimed by Anima, which it called a known misidentification. It now expects no probe to claim it.ruff@0.11.2 checkandformat --checkclean.Review
Review asked for the docs to state the remaining ambiguity precisely and for the format support page to note the change; both are done. Remaining limit: an LTX-Video LoRA without cross-attention key/value layers (
attn2.to_k,attn2.to_v) is still identified as Anima. Its other attention layers are 2048 wide like Anima's, so neither names nor widths separate them.Checklist
What's Newcopy (if doing a release after this PR)Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.