Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions .ai/references/modular.md
Original file line number Diff line number Diff line change
Expand Up @@ -324,20 +324,22 @@ ComponentSpec(

5. **Using `InputParam.template()` / `OutputParam.template()` when semantics don't match.** Templates carry predefined descriptions — e.g. the `"latents"` output template means "Denoised latents". Don't use it for initial noisy latents from a prepare-latents step. Use a plain `InputParam(...)` / `OutputParam(...)` with an accurate description instead.

6. **Test model paths pointing to contributor repos.** Tiny test models ultimately live under `hf-internal-testing/`, not personal repos like `username/tiny-model`. Developing against a personal repo is fine and not merge-blocking — a maintainer moves the model (before or after merge) and updates the path.
6. **`required=True` on an input that carries a default.** An input cannot be both required and have a default. When that happens, `InputParam` keeps the default, treats the input as optional and logs a warning. This is likely to happen when you reuse a template for its type hint and description but make it required: `InputParam.template("num_inference_steps", required=True)` stays an optional input that defaults to 50, plus a warning. Decide which one you mean: drop `required=True` to keep the template default without the warning, or pass `default=None` explicitly to make the input required.

7. **Respect the declared IO system.** Components in `expected_components`, fields in `inputs` / `intermediate_outputs` — once declared, the modular framework guarantees them. So:
7. **Test model paths pointing to contributor repos.** Tiny test models ultimately live under `hf-internal-testing/`, not personal repos like `username/tiny-model`. Developing against a personal repo is fine and not merge-blocking — a maintainer moves the model (before or after merge) and updates the path.

8. **Respect the declared IO system.** Components in `expected_components`, fields in `inputs` / `intermediate_outputs` — once declared, the modular framework guarantees them. So:
- **Don't read defensively.** Declared components are always set as attributes (possibly `None`); declared upstream outputs are always populated in `block_state` after the upstream block runs. `getattr(components, "vae", None)`, `hasattr(self, "vae")`, `getattr(block_state, "prompt_embeds", None)` are dead code that hides typos. Use `components.vae` / `block_state.prompt_embeds` directly. Check `is not None` only when nullability is meaningful (a component the user might not have loaded).
- **Don't write undeclared.** If a block sets `block_state.foo = ...`, declare `OutputParam("foo", ...)` in `intermediate_outputs`. The declarations are the public contract — undeclared writes can't be wired to downstream blocks.
- **Don't call `state.set()` directly inside a block.** Write to state only through declared `intermediate_outputs` via `self.get_block_state(state)` / `self.set_block_state(state, block_state)`. A direct `state.set("foo", value)` bypasses the block's interface entirely — the field never appears as a declared output, so downstream blocks can't see it through the normal wiring and the framework can't generate docs / validate types for it.

8. **No-op skip logic inside an optional block.** If a step is conditional (e.g. an optional prompt enhancer), don't have the block check a flag at the top of `__call__` and `return` early. Wrap it in an `AutoPipelineBlocks` with `block_trigger_inputs = ["use_xxx"]` so the block is only assembled into the pipeline when the trigger input is provided. The block's own `__call__` should always assume its components and inputs are present.
9. **No-op skip logic inside an optional block.** If a step is conditional (e.g. an optional prompt enhancer), don't have the block check a flag at the top of `__call__` and `return` early. Wrap it in an `AutoPipelineBlocks` with `block_trigger_inputs = ["use_xxx"]` so the block is only assembled into the pipeline when the trigger input is provided. The block's own `__call__` should always assume its components and inputs are present.

9. **Serving a checkpoint variant through a config flag in a shared block.** `ConfigSpec(name="is_distilled")` plus `if components.config.is_distilled:` bundles two checkpoints' behavior into one blockset — and it can't change the input surface at all (the distilled variant would still accept `negative_prompt`). Suggest a separate blockset for the variant instead (see Key pattern: Checkpoint variants).
10. **Serving a checkpoint variant through a config flag in a shared block.** `ConfigSpec(name="is_distilled")` plus `if components.config.is_distilled:` bundles two checkpoints' behavior into one blockset — and it can't change the input surface at all (the distilled variant would still accept `negative_prompt`). Suggest a separate blockset for the variant instead (see Key pattern: Checkpoint variants).

10. **Raw `torch.randn(device=...)` for noise.** Use `randn_tensor(...)` from `utils/torch_utils`: it draws on the generator's device and moves the result, so CPU generators (what the test mixins pass) work, and the CUDA-generator path is bit-identical to `torch.randn`.
11. **Raw `torch.randn(device=...)` for noise.** Use `randn_tensor(...)` from `utils/torch_utils`: it draws on the generator's device and moves the result, so CPU generators (what the test mixins pass) work, and the CUDA-generator path is bit-identical to `torch.randn`.

11. **Helpers taking `components`.** Prefer explicit args, for example, `get_xx_prompt_embeds(text_encoder, tokenizer, prompt)` instead of `get_xx_prompt_embeds(components, prompt)`. Helpers and block methods should take only what they use so dependencies are visible and they can run without an assembled pipeline. Leave the full `components` object to `__call__`, which unpacks what its helpers need. See `get_qwen_prompt_embeds` in `qwenimage/encoders.py`.
12. **Helpers taking `components`.** Prefer explicit args, for example, `get_xx_prompt_embeds(text_encoder, tokenizer, prompt)` instead of `get_xx_prompt_embeds(components, prompt)`. Helpers and block methods should take only what they use so dependencies are visible and they can run without an assembled pipeline. Leave the full `components` object to `__call__`, which unpacks what its helpers need. See `get_qwen_prompt_embeds` in `qwenimage/encoders.py`.

## Conversion checklist

Expand Down
1 change: 1 addition & 0 deletions .ai/references/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ Since [#14268](https://github.com/huggingface/diffusers/pull/14268), a standard

- Location: `tests/modular_pipelines/<model>/test_modular_pipeline_<model>.py` (one config class + set of test classes per blockset / pipeline variant).
- **Define one config class**, `<Pipeline>ModularPipelineTesterConfig`, subclassing `BaseModularPipelineTesterConfig` (from `..testing_utils`). Set `pipeline_class`, `pipeline_blocks_class`, `pretrained_model_name_or_path`, `params` / `batch_params`, and implement `get_dummy_inputs(seed=0)`. Set `expected_workflow_blocks` to pin the block name → class ordering per workflow. The config holds the whole testing contract and performs no assertions.
- **Set `expected_workflow_defaults`** to what each workflow is expected to have: every component with its class, every config with its default, and every input — the optional ones with their defaults under `inputs`, the required ones by name under `required_inputs`. The test fails when the code declares anything else, so this table is the pipeline's contract written out in one place, and writing it is also the moment to check that contract. Once it is filled in, read it against the same tables of the other families under `tests/modular_pipelines/`: for example, `prompt` is normally required, `num_inference_steps` is normally optional with a default, `latents` is normally accepted so a run can be reproduced, and so on. An entry that breaks the common pattern needs a reason. If there isn't one, something is wrong: find out what and fix it, rather than writing the odd entry into the table.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test fails when the code declares anything else, so this table is the pipeline's contract written out in one place, and writing it is also the moment to check that contract.

What table is being referred to here? These are Python properties right?

- **Then one test class per concern**, each composing the config with a tester mixin from `..testing_utils`. Keep them separate — pytest reads class-level markers off the whole MRO, so folding a marked mixin (`@is_memory`, ...) into the same class as the others would tag every test in it:
- `ModularPipelineTesterMixin` — call signature, batch consistency, float16, device placement, NaN-free output. Put pipeline-specific tests as methods on this class.
- `ModularLoadingTesterMixin` — `save_pretrained`/`from_pretrained` round-trips, `modular_model_index.json` contents, `load_components`/`unload_components`.
Expand Down
4 changes: 2 additions & 2 deletions src/diffusers/modular_pipelines/cosmos/before_denoise.py
Original file line number Diff line number Diff line change
Expand Up @@ -979,7 +979,7 @@ def expected_configs(self) -> list[ConfigSpec]:
@property
def inputs(self) -> list[InputParam]:
return [
InputParam.template("num_inference_steps", required=True),
InputParam.template("num_inference_steps"),
]

@property
Expand Down Expand Up @@ -1258,7 +1258,7 @@ def expected_components(self) -> list[ComponentSpec]:

@property
def inputs(self) -> list[InputParam]:
return [InputParam.template("num_inference_steps", required=True)]
return [InputParam.template("num_inference_steps")]

@property
def intermediate_outputs(self) -> list[OutputParam]:
Expand Down
2 changes: 1 addition & 1 deletion src/diffusers/modular_pipelines/cosmos/denoise.py
Original file line number Diff line number Diff line change
Expand Up @@ -500,7 +500,7 @@ def loop_expected_components(self) -> list[ComponentSpec]:
def loop_inputs(self) -> list[InputParam]:
return [
InputParam.template("timesteps", required=True),
InputParam.template("num_inference_steps", required=True),
InputParam.template("num_inference_steps", required=True, default=None),
InputParam(
name="num_warmup_steps", type_hint=int, required=True, description="Number of scheduler warmup steps."
),
Expand Down
12 changes: 6 additions & 6 deletions src/diffusers/modular_pipelines/cosmos/modular_blocks_cosmos3.py
Original file line number Diff line number Diff line change
Expand Up @@ -394,7 +394,7 @@ class Cosmos3VisionCoreDenoiseStep(SequentialPipelineBlocks):
Pre-generated noisy vision latents.
generator (`Generator`, *optional*):
Torch generator for deterministic generation.
num_inference_steps (`int`):
num_inference_steps (`int`, *optional*, defaults to 50):
The number of denoising steps.
mixed_precision_format (`str`, *optional*):
None follows the ModelOpt FP8 checkpoint schedule; 'none' keeps the native quantized forward; 'fp8' is
Expand Down Expand Up @@ -474,7 +474,7 @@ class Cosmos3VisionSoundCoreDenoiseStep(SequentialPipelineBlocks):
Pre-generated noisy vision latents.
generator (`Generator`, *optional*):
Torch generator for deterministic generation.
num_inference_steps (`int`):
num_inference_steps (`int`, *optional*, defaults to 50):
The number of denoising steps.
sound_latents (`Tensor`, *optional*):
Pre-generated noisy sound latents.
Expand Down Expand Up @@ -567,7 +567,7 @@ class Cosmos3VisionActionCoreDenoiseStep(SequentialPipelineBlocks):
Pre-generated noisy vision latents.
generator (`Generator`, *optional*):
Torch generator for deterministic generation.
num_inference_steps (`int`):
num_inference_steps (`int`, *optional*, defaults to 50):
The number of denoising steps.
action (`CosmosActionCondition`):
Action-conditioning metadata.
Expand Down Expand Up @@ -664,7 +664,7 @@ class Cosmos3VisionSoundActionCoreDenoiseStep(SequentialPipelineBlocks):
Pre-generated noisy vision latents.
generator (`Generator`, *optional*):
Torch generator for deterministic generation.
num_inference_steps (`int`):
num_inference_steps (`int`, *optional*, defaults to 50):
The number of denoising steps.
sound_latents (`Tensor`, *optional*):
Pre-generated noisy sound latents.
Expand Down Expand Up @@ -782,7 +782,7 @@ class Cosmos3TransferChunkDenoiseStep(SequentialPipelineBlocks):
Unconditional text segment.
fps (`float`, *optional*, defaults to 24.0):
Frame rate of the generated video.
num_inference_steps (`int`):
num_inference_steps (`int`, *optional*, defaults to 50):
The number of denoising steps.
mixed_precision_format (`str`, *optional*):
None follows the ModelOpt FP8 checkpoint schedule; 'none' keeps the native quantized forward; 'fp8' is
Expand Down Expand Up @@ -937,7 +937,7 @@ class Cosmos3TransferCoreDenoiseStep(SequentialPipelineBlocks):
Torch generator for deterministic generation.
fps (`float`, *optional*, defaults to 24.0):
Frame rate of the generated video.
num_inference_steps (`int`):
num_inference_steps (`int`, *optional*, defaults to 50):
The number of denoising steps.
mixed_precision_format (`str`, *optional*):
None follows the ModelOpt FP8 checkpoint schedule; 'none' keeps the native quantized forward; 'fp8' is
Expand Down
4 changes: 2 additions & 2 deletions src/diffusers/modular_pipelines/krea2/denoise.py
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ def expected_components(self) -> list[ComponentSpec]:
def inputs(self) -> list[InputParam]:
return [
InputParam(name="latents", required=True, type_hint=torch.Tensor, description="Packed image latents."),
InputParam.template("num_inference_steps", required=True),
InputParam.template("num_inference_steps", required=True, default=None),
InputParam(
name="prompt_embeds",
required=True,
Expand Down Expand Up @@ -264,7 +264,7 @@ def loop_inputs(self) -> list[InputParam]:
type_hint=torch.Tensor,
description="Denoising timesteps from set_timesteps.",
),
InputParam.template("num_inference_steps", required=True),
InputParam.template("num_inference_steps", required=True, default=None),
InputParam.template("attention_kwargs"),
]

Expand Down
4 changes: 2 additions & 2 deletions src/diffusers/modular_pipelines/ltx/before_denoise.py
Original file line number Diff line number Diff line change
Expand Up @@ -282,7 +282,7 @@ def inputs(self) -> list[InputParam]:
InputParam.template("latents"),
InputParam.template("num_images_per_prompt", name="num_videos_per_prompt"),
InputParam.template("generator"),
InputParam.template("batch_size", required=True),
InputParam.template("batch_size", required=True, default=None),
]

@property
Expand Down Expand Up @@ -348,7 +348,7 @@ def inputs(self) -> list[InputParam]:
InputParam.template("width", default=704),
InputParam("num_frames", type_hint=int, default=161),
InputParam.template("num_images_per_prompt", name="num_videos_per_prompt"),
InputParam.template("batch_size", required=True),
InputParam.template("batch_size", required=True, default=None),
]

@property
Expand Down
2 changes: 1 addition & 1 deletion src/diffusers/modular_pipelines/ltx/decoders.py
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ def inputs(self) -> list[tuple[str, Any]]:
InputParam("decode_noise_scale", default=None),
InputParam.template("generator"),
InputParam.template("batch_size"),
InputParam.template("dtype", required=True),
InputParam.template("dtype", required=True, default=None),
# Only read on the deprecated path that still accepts packed `[B, S, C]` latents.
InputParam.template("height", default=512),
InputParam.template("width", default=704),
Expand Down
10 changes: 5 additions & 5 deletions src/diffusers/modular_pipelines/ltx/denoise.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ def description(self) -> str:
def inputs(self) -> list[InputParam]:
return [
InputParam.template("latents", required=True),
InputParam.template("dtype", required=True),
InputParam.template("dtype", required=True, default=None),
]

@torch.no_grad()
Expand Down Expand Up @@ -97,7 +97,7 @@ def description(self) -> str:
def inputs(self) -> list[tuple[str, Any]]:
inputs = [
InputParam.template("attention_kwargs"),
InputParam.template("num_inference_steps", required=True),
InputParam.template("num_inference_steps", required=True, default=None),
InputParam("rope_interpolation_scale", type_hint=tuple),
InputParam.template("height"),
InputParam.template("width"),
Expand Down Expand Up @@ -211,7 +211,7 @@ def loop_expected_components(self) -> list[ComponentSpec]:
def loop_inputs(self) -> list[InputParam]:
return [
InputParam.template("timesteps", required=True),
InputParam.template("num_inference_steps", required=True),
InputParam.template("num_inference_steps", required=True, default=None),
]

@torch.no_grad()
Expand Down Expand Up @@ -277,7 +277,7 @@ def inputs(self) -> list[InputParam]:
return [
InputParam.template("latents", required=True),
InputParam("conditioning_mask", required=True, type_hint=torch.Tensor),
InputParam.template("dtype", required=True),
InputParam.template("dtype", required=True, default=None),
]

@torch.no_grad()
Expand Down Expand Up @@ -331,7 +331,7 @@ def description(self) -> str:
def inputs(self) -> list[tuple[str, Any]]:
inputs = [
InputParam.template("attention_kwargs"),
InputParam.template("num_inference_steps", required=True),
InputParam.template("num_inference_steps", required=True, default=None),
InputParam("rope_interpolation_scale", type_hint=tuple),
InputParam.template("height"),
InputParam.template("width"),
Expand Down
4 changes: 2 additions & 2 deletions src/diffusers/modular_pipelines/ltx2/decoders.py
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ def inputs(self) -> list[tuple[str, Any]]:
"num_frames", type_hint=int, default=121, description="The number of frames in the generated video."
),
InputParam.template("generator"),
InputParam.template("dtype", required=True),
InputParam.template("dtype", required=True, default=None),
]

@property
Expand Down Expand Up @@ -251,7 +251,7 @@ def inputs(self) -> list[tuple[str, Any]]:
),
InputParam.template("generator"),
InputParam.template("batch_size"),
InputParam.template("dtype", required=True),
InputParam.template("dtype", required=True, default=None),
]

@property
Expand Down
4 changes: 2 additions & 2 deletions src/diffusers/modular_pipelines/ltx2/denoise.py
Original file line number Diff line number Diff line change
Expand Up @@ -305,7 +305,7 @@ def inputs(self) -> list[InputParam]:
# `audio_num_frames`, `video_coords`, `audio_coords` arrive tagged `denoiser_input_fields` upstream and
# are collected from the tagged dict (filtered against the transformer signature) in `__call__`.
InputParam.template("denoiser_input_fields"),
InputParam.template("num_inference_steps", required=True),
InputParam.template("num_inference_steps", required=True, default=None),
InputParam.template("height", default=512),
InputParam.template("width", default=704),
InputParam(
Expand Down Expand Up @@ -654,7 +654,7 @@ def loop_expected_components(self) -> list[ComponentSpec]:
def loop_inputs(self) -> list[InputParam]:
return [
InputParam("timesteps", type_hint=torch.Tensor, required=True),
InputParam.template("num_inference_steps", required=True),
InputParam.template("num_inference_steps", required=True, default=None),
]

@torch.no_grad()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1146,7 +1146,7 @@ def expected_components(self) -> list[ComponentSpec]:
@property
def inputs(self) -> list[InputParam]:
return [
InputParam.template("num_inference_steps", required=True),
InputParam.template("num_inference_steps"),
InputParam(
name="video_indices",
type_hint=torch.Tensor,
Expand Down
Loading
Loading