Add Hy-Nomas support with a modular pipeline by jiaxiangc · Pull Request #15114 · huggingface/diffusers

jiaxiangc · 2026-10-23T12:05:00Z

What does this PR do?

Adds Hy-Nomas inference support to Diffusers through HyNomasModularPipeline, HyNomasTransformer3DModel, and PixelUMMDecoder.

I am a member of the Hy-Nomas team.

Closes #15113

The issue and this PR are being submitted together for feedback on the proposed scope and implementation. Maintainer acknowledgment is pending.

Changes

Support plain and structured prompts with segment-relative action timestamps.
Support optional camera conditioning and multi-turn continuation.
Add checkpoint conversion and standard component loading/saving.
Add optional reference attention backends.
Add model/pipeline tests, usage documentation, and a keyboard overlay example.

The workflow targets the distilled checkpoint with guidance scale 1.

Checkpoints

Preconverted checkpoints are available at tencent/Hy-Nomas-Diffusers.

The repository includes all components required for inference, including the text encoder and tokenizer. Users do not need to convert the original checkpoints or download a separate Qwen checkpoint. The bundled text encoder is distinct from the optional Nomas-PE prompt enhancement model.

Dependencies

The optional hy-nomas extra includes:

PyTorch >= 2.10.0
Transformers >= 5.20.2
Accelerate >= 1.13.0
imageio and imageio-ffmpeg

FlashAttention 2, FlashAttention 3, and MagiAttention are optional dependencies used only for reference-attention inference. They are not installed by the hy-nomas extra, and the default PyTorch attention path does not require them.

Local validation

make quality: passed.

The following CPU test run completed with 114 passed, 52 skipped, and 39 deselected:

CUDA_VISIBLE_DEVICES="" OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 PYTHONPATH=src pytest \
  tests/models/transformers/test_models_transformer_hy_nomas.py \
  tests/models/autoencoders/test_models_pixelumm.py \
  tests/modular_pipelines/hy_nomas/test_modular_pipeline_hy_nomas.py \
  tests/others/test_hy_nomas_keyboard_overlay.py \
  -m "not slow and not memory" -q -p no:cacheprovider

A real-checkpoint comparison against the original Hy-Nomas implementation used:

Two five-second segments: 240 frames at 24 FPS.
Four inference steps, seed 42, and BF16.
Deterministic reference attention and matching video export settings.

Both the raw RGB frames and decoded MP4 frames were pixel-identical, with maximum absolute difference 0.

This pixel-level equivalence result applies to the matched reference-attention configuration. It does not establish numerical equivalence for the default PyTorch attention path.

make repo-consistency remains blocked by an existing checker importing the missing diffusers.models.auto module. The checker and missing module are unchanged from the PR base; unrelated checker fixes are excluded.

AI-assisted development and self-review

Codex assisted with implementation and validation. The complete contribution was reviewed using the repository's self-review skill. No blocking findings or confirmed dead code were identified.

Two non-blocking points are left for maintainer review:

The tested model-level camera_attention_mask gate is retained, although the guidance-one modular pipeline does not currently pass it.
A small unrelated README URL cleanup removes a pre-existing credential-bearing query parameter. Maintainers may prefer this cleanup in a separate change.

Before submitting

cc @yiyixuxu @asomoza

Co-authored-by: Cursor <cursoragent@cursor.com>

github-actions · 2026-10-24T07:38:13Z

Hi @jiaxiangc, thanks for the PR! It does not appear to link an issue it fixes. If this PR addresses an existing issue, please add a closing keyword (e.g. Fixes #1234) to the PR description so the issue is linked. See the contribution guide for more details. If this PR intentionally does not fix a tracked issue, a maintainer can add the no-issue-needed label to silence this reminder.

Please note that PRs without a linked issue are likely to be automatically closed 10 days after this notice.

Once the PR links an issue (or gets the no-issue-needed label), you can ignore this message — it stays here as a comment, but it no longer applies.

yiyixuxu

thanks for the PR! I left some initial comments,

additionally, could you remove the streaming support from this PR for now? we are working on adding first class streaming support for modular pipelines #14159 so happy to work together on a follow up to make sure your use case works with that API

yiyixuxu · 2026-10-24T08:25:19Z

    "hf-doc-builder>=0.3.0",
    "huggingface-hub>=1.32.0,<3.0",
    "requests-mock==1.10.0",
+    "imageio",


can you remove the changes added to setup.py

yiyixuxu · 2026-10-24T08:29:49Z

+
+
+@dataclass(frozen=True)
+class PixelUMMArchitecture:


can you remove this

yiyixuxu · 2026-10-24T08:31:22Z

+    return torch.cat((-x[..., half:], x[..., :half]), dim=-1)
+
+
+def mrope_tables(arch: PixelUMMArchitecture, grid: tuple[int, int, int], device) -> tuple[torch.Tensor, torch.Tensor]:


Suggested change

def mrope_tables(arch: PixelUMMArchitecture, grid: tuple[int, int, int], device) -> tuple[torch.Tensor, torch.Tensor]:

def mrope_tables(rope_thera_t, rope_theta_hw, , rope_sections, grid: tuple[int, int, int], device) -> tuple[torch.Tensor, torch.Tensor]:

yiyixuxu · 2026-10-24T08:31:45Z

+def mrope_tables(arch: PixelUMMArchitecture, grid: tuple[int, int, int], device) -> tuple[torch.Tensor, torch.Tensor]:
+    """cos/sin [1, T*H*W, 1, head_dim] for a THW-ordered grid, laid out [T | H | W]."""
+    positions = torch.meshgrid(*(torch.arange(n, device=device) for n in grid), indexing="ij")
+    thetas = (arch.rope_theta_t, arch.rope_theta_hw, arch.rope_theta_hw)


Suggested change

thetas = (arch.rope_theta_t, arch.rope_theta_hw, arch.rope_theta_hw)

thetas = (rope_theta_t, rope_theta_hw, rope_theta_hw)

yiyixuxu · 2026-10-24T08:34:35Z

+    return permutation, torch.argsort(permutation), (neighborhoods, block_mask)
+
+
+def _pixelumm_flex_attention(query, key, value, block_mask, parallel_config):


can you inline this?

yiyixuxu · 2026-10-25T08:58:17Z

+class HyNomasMagiCameraAttnProcessor(HyNomasCameraAttnProcessor):
+    """Reference frame-local camera range attention."""
+
+    def _compute_attention(self, query, key, value, num_frames, tokens_per_frame):
+        if num_frames == 1:
+            return dispatch_attention_fn(query, key, value, backend="flash")
+        starts = torch.arange(num_frames, dtype=torch.int32, device=query.device)
+        frame_ranges = torch.stack((starts, starts + 1), dim=-1)
+        return dispatch_attention_fn(
+            query,
+            key,
+            value,
+            backend="magi",
+            attention_kwargs={
+                "query_ranges": frame_ranges * tokens_per_frame,
+                "key_ranges": frame_ranges,
+                "attn_type_map": torch.zeros(num_frames, dtype=torch.int32, device=query.device),
+            },
+        )


Suggested change

class HyNomasMagiCameraAttnProcessor(HyNomasCameraAttnProcessor):

"""Reference frame-local camera range attention."""

def _compute_attention(self, query, key, value, num_frames, tokens_per_frame):

if num_frames == 1:

return dispatch_attention_fn(query, key, value, backend="flash")

starts = torch.arange(num_frames, dtype=torch.int32, device=query.device)

frame_ranges = torch.stack((starts, starts + 1), dim=-1)

return dispatch_attention_fn(

query,

key,

value,

backend="magi",

attention_kwargs={

"query_ranges": frame_ranges * tokens_per_frame,

"key_ranges": frame_ranges,

"attn_type_map": torch.zeros(num_frames, dtype=torch.int32, device=query.device),

},

)

yiyixuxu · 2026-10-25T08:58:25Z

+
+class HyNomasCameraAttention(nn.Module, AttentionModuleMixin):
+    _default_processor_cls = HyNomasCameraAttnProcessor
+    _available_processors = [HyNomasCameraAttnProcessor, HyNomasMagiCameraAttnProcessor]


Suggested change

_available_processors = [HyNomasCameraAttnProcessor, HyNomasMagiCameraAttnProcessor]

_available_processors = [HyNomasCameraAttnProcessor]

yiyixuxu · 2026-10-25T09:02:45Z

+            raise ValueError("Action offsets require one entry per sample and action_frame_windows.")
+        if cache_action_start is not None and not 0 <= cache_action_start <= encoder_hidden_states.shape[1]:
+            raise ValueError("cache_action_start must be a valid text boundary or None.")
+        if kv_cache is not None:


can you give kv cache a class? we usually have a per-layer cache and a container class see example https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/transformers/transformer_qwenimage21.py#L56

yiyixuxu · 2026-10-25T09:03:07Z

    AITER_FA2_HUB = "aiter_fa2_hub"

+    # `magi_attention` rectangle-range attention
+    MAGI = "magi"


let's not add magi in this PR

yiyixuxu · 2026-10-25T09:39:41Z

+
+
+@dataclass
+class HyNomasGenerationState:


Can we try to remove HyNomasGenerationState and support continued generation by passing outputs back as inputs? A few things on top of my mind:

latents is both the initial-noise input and the denoised output; to work with this use case, you probably need to rename the latents input to something else, e.g. initial_noise/initial_latents

You can tag everything a continuation needs (kv_cache, frame_offset ...) with kwargs_type, e.g. OutputParam("kv_cache", ..., kwargs_type="world_state") on the loop's outputs and InputParam("kv_cache", ..., kwargs_type="world_state") on the prepare step's optional inputs. PipelineState groups them, so state.get_by_kwargs("world_state") returns them as a dict and the pipeline accepts that dict back as a single kwarg. A turn then looks like:

turn1 = pipe(prompt="walk forward", camera_extrinsics=cam1) turn2 = pipe(world_state=turn1.get_by_kwargs("world_state"), prompt="turn right", camera_extrinsics=cam2, output="videos")

see more on https://github.com/huggingface/diffusers/blob/main/.ai/references/modular.md#key-pattern-kwargs_type-inputs-denoiser_input_fields

let me know if you run into any other issues

jiaxiangc and others added 2 commits October 23, 2026 19:33

Add Hy-Nomas pipeline and PixelUMM decoder integration

01cf006

Co-authored-by: Cursor <cursoragent@cursor.com>

Update Hy-Nomas installation and inference examples

ec0886c

github-actions Bot added documentation Improvements or additions to documentation models tests modular-pipelines utils examples size/L PR with diff > 200 LOC labels Oct 23, 2026

jiaxiangc mentioned this pull request Oct 23, 2026

[Model integration] Add Hy-Nomas support to Diffusers #15113

Open

2 tasks done

docs: update Hy-Nomas Diffusers model ID

12722ef

yiyixuxu added the no-issue-needed for PRs that do not require link to an issue label Oct 24, 2026

docs: clarify Hy-Nomas reference attention setup

1a0aaa3

github-actions Bot added the fixes-issue label Oct 24, 2026

yiyixuxu reviewed Oct 25, 2026

View reviewed changes

Navigation Menu

Repository navigation

Add Hy-Nomas support with a modular pipeline - #15114

Add Hy-Nomas support with a modular pipeline#15114
jiaxiangc wants to merge 4 commits into
huggingface:mainfrom
jiaxiangc:hy-nomas

jiaxiangc commented Oct 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 24, 2026

Uh oh!

yiyixuxu left a comment

Uh oh!

yiyixuxu Oct 24, 2026

Uh oh!

yiyixuxu Oct 24, 2026

Uh oh!

yiyixuxu Oct 24, 2026

Uh oh!

yiyixuxu Oct 24, 2026

Uh oh!

yiyixuxu Oct 24, 2026

Uh oh!

yiyixuxu Oct 25, 2026

Uh oh!

yiyixuxu Oct 25, 2026

Uh oh!

yiyixuxu Oct 25, 2026

Uh oh!

yiyixuxu Oct 25, 2026

Uh oh!

yiyixuxu Oct 25, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		return torch.cat((-x[..., half:], x[..., :half]), dim=-1)


		def mrope_tables(arch: PixelUMMArchitecture, grid: tuple[int, int, int], device) -> tuple[torch.Tensor, torch.Tensor]:

	def mrope_tables(arch: PixelUMMArchitecture, grid: tuple[int, int, int], device) -> tuple[torch.Tensor, torch.Tensor]:
	def mrope_tables(rope_thera_t, rope_theta_hw, , rope_sections, grid: tuple[int, int, int], device) -> tuple[torch.Tensor, torch.Tensor]:

	thetas = (arch.rope_theta_t, arch.rope_theta_hw, arch.rope_theta_hw)
	thetas = (rope_theta_t, rope_theta_hw, rope_theta_hw)

		return permutation, torch.argsort(permutation), (neighborhoods, block_mask)


		def _pixelumm_flex_attention(query, key, value, block_mask, parallel_config):

	_available_processors = [HyNomasCameraAttnProcessor, HyNomasMagiCameraAttnProcessor]
	_available_processors = [HyNomasCameraAttnProcessor]

Conversation

jiaxiangc commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Changes

Checkpoints

Dependencies

Local validation

AI-assisted development and self-review

Before submitting

Uh oh!

github-actions Bot commented Oct 24, 2026

Uh oh!

yiyixuxu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yiyixuxu Oct 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jiaxiangc commented Oct 23, 2026 •

edited

Loading

yiyixuxu Oct 25, 2026 •

edited

Loading