[Pfannkuchensack]

Summary

On the CPU (cpu_only, or an install without a GPU), T5 encoders from a Diffusers folder or a GGUF file now hold and compute in float32 instead of bf16, as the single-file T5 already did. Elsewhere they use choose_bfloat16_safe_dtype (before: the folder's stored dtype via "auto", and hard-coded bf16 for GGUF). CPU bf16 GEMM faults with 0xc000001d on part of the Windows fleet and is emulated slowly on CPUs without native bf16.

• One rule for the three T5 formats: t5_dtype_for_device (invokeai/backend/t5/t5_encoder.py).
• The dtype is part of the cache-key variant, so device caches never share weights of different dtypes.
• The Diffusers T5 reserves RAM at the dtype it is built in (float32 on the CPU is twice the bf16 folder).
• The FLUX and SD3 text-encoder invocations size the dequant transient with the same dtype.

Embedding error on the CPU (T5-XXL, 512 tokens) against a float32 GPU reference (TF32 off), relative L2:

T5	before (bf16)	after (float32)
GGUF Q6_K	6.3 %	5.6e-6
Diffusers, bf16 folder	5.7 %	5.3e-6

GPU bf16, the normal GPU path, is at 7.6 % on the same prompt.

Cost: the Diffusers T5-XXL holds ~18 GB instead of ~9 GB in RAM; GGUF grows by ~0.3 GB (its dequantized embeddings). Speed depends on the CPU. For the Linears of one T5-XXL block at 512 tokens, float32 is 2.2× slower than bf16 on a CPU with native bf16 GEMM (Ryzen 9 7900, AVX512-BF16) and 4.7× faster on the AVX2 path CPUs without native bf16 take (simulated on the same CPU: oneDNN off, ATen and MKL capped to AVX2). End to end on the Ryzen, a Diffusers T5-XXL encode goes from 4.7 s to 10.5 s.

Related Issues / Discussions

Closes #10144

Follow-up for the remaining T5/UMT5 paths (SDNQ, the SD3 pipeline's own T5 under cpu_only, Wan UMT5): #10215

QA Instructions

• CUDA_VISIBLE_DEVICES=-1 uv run --no-sync pytest tests/backend/model_manager/load/test_t5_single_file_loader.py tests/backend/model_manager/load/test_t5_diffusers_loader.py tests/backend/model_manager/load/test_t5_gguf_loader.py tests/backend/t5 tests/backend/model_manager/load/test_load_default_cpu_only.py tests/backend/model_manager/load/test_shared_weight_adoption.py tests/backend/model_manager/load/test_model_loader_registry_entries.py tests/backend/model_manager/load/test_side_channel_blind_loaders.py tests/backend/model_manager/load/test_clip_single_file_loader.py tests/backend/model_manager/configs/test_t5_encoder_checkpoint_config.py tests/app/invocations/test_sd3_text_encoder.py tests/app/invocations/test_text_encoders_with_packed_layers.py tests/app/services/session_processor/test_encoder_offload.py: 134 passed.
• New tests:
  • test_t5_diffusers_loader.py: float32 on the CPU with a forward against a float32 reference, cpu_only on a GPU box, GPU dtype and key, no sharing across dtypes through the shared CPU store (both load orders), and the cold-load RAM reservation covering the built model.
  • test_t5_gguf_loader.py: a tiny T5 v1.1 GGUF (Q8_0, F16, F32) checked for compute dtype, unchanged packed bytes and a forward against gguf-py's dequantizer, on the CPU, under cpu_only, and on a GPU box.
• Ruff check and format: clean.
• Real files, CPU only, before/after interleaved, three runs each: t5-v1_1-xxl-encoder-Q6_K.gguf and the bf16 Diffusers T5-XXL starter. Outputs are bit-identical across runs, plus the GPU float32 reference on an RTX 4090.
• Not checked: a machine that shows the 0xc000001d fault, MPS/XPU/ROCm, and a real CPU without native bf16 (simulated only).

Review

Resolved: the Diffusers T5 reserved RAM at the folder's bf16 size while building float32 on the CPU. It is now sized from the safetensors headers at the built dtype. The family pages are updated too.

Remaining: float32 costs about 2× RAM, and about 2× time on CPUs with native bf16 GEMM. On a CUDA device that rejected bf16, choose_bfloat16_safe_dtype would return float16 and a GGUF T5 would compute in fp16. That case is not reachable with current torch, and it was no safer before (hard-coded bf16).

Compatibility / Rollout

Cache keys of Diffusers and GGUF T5 encoders gain a :<dtype> suffix. They are in-memory only, and their consumers already match by prefix. Nothing persisted changes. On a GPU, a Diffusers T5 now loads in choose_bfloat16_safe_dtype's dtype instead of its stored dtype. That is unchanged for bf16 folders (the FLUX.1 starter); an fp16 or fp32 folder now loads as bf16.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Meaningful regression coverage added / updated where needed; obsolete tests/code removed
• Persisted-state and API changes include required migrations / compatibility validation
• Relevant performance/efficiency opportunities considered; material claims have evidence
• Material review findings resolved and relevant checks rerun
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

---

^{Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.}

[joshistoast]

One nonblocking improvement: add a sharded Diffusers fixture to test the index-based RAM-sizing branch; current coverage exercises an unsharded folder.

[Pfannkuchensack]

Thanks! Added in 3ef5e46: the cold-load RAM-sizing test now also runs on a sharded text_encoder_2/ (index plus several shards), and it fails if the index branch is skipped.