Kandinsky 5 is finally in Diffusers!

[leffff]

What does this PR do?

This PR adds Kandinsky5T2VPipeline and Kandinsky5Transformer3DModel as well as several layer classes neede for Kandinsky 5.0 Lite T2V model

@sayakpaul Please review

[sayakpaul]

Could you please update the PR with test code and some example outputs?

[leffff]

Sure!

[leffff]

@sayakpaul

The example is here:
https://github.com/leffff/diffusers/blob/04efb19b1aeba3b41b7b1bd6d0353a1715c0f839/src/diffusers/pipelines/kandinsky5/pipeline_kandinsky.py#L51

Just like in Wan:

diffusers/src/diffusers/pipelines/wan/pipeline_wan.py

Line 46 in fa468c5

EXAMPLE_DOC_STRING = """

[leffff]

Dear @sayakpaul @yiyixuxu @DN6
How should the test code and example outputs look like?

[leffff]

import torch
from diffusers import Kandinsky5T2VPipeline
from diffusers.utils import export_to_video

pipe = Kandinsky5T2VPipeline.from_pretrained(
    "ai-forever/Kandinsky-5.0-T2V-Lite-sft-5s-Diffusers", 
    torch_dtype=torch.bfloat16
)
pipe = pipe.to("cuda")

negative_prompt = [
    "Static, 2D cartoon, cartoon, 2d animation, paintings, images, worst quality, low quality, ugly, deformed, walking backwards",
]
prompt = [
    "A cat and a dog baking a cake together in a kitchen.",
]

output = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    height=512,
    width=768,
    num_frames=121,
    num_inference_steps=50,
    guidance_scale=5.0,
    num_videos_per_prompt=1,
    generator=torch.Generator(42)
)

output.10.mp4

prompt = [
    "A monkey ridign a skateboard",
]

output.10.mp4

prompt = [
    "Several giant wooly mammoths threading through the meadow",
]

output.10.mp4

[sayakpaul]

Great, thanks for providing the examples! Does the model also do realistic generations? 👀

@linoytsaban @apolinario @asomoza in case you wanna test it?

[leffff]

Yes of course!

A stylish woman struts confidently down a rain-drenched Tokyo street, where vibrant neon signs flicker and pulse with electric color. She wears a sleek black leather jacket over a flowing red dress, paired with polished black boots and a matching black purse. Her sunglasses reflect the glowing cityscape as she moves with a calm, assured demeanor, red lipstick adding a bold contrast to her look. The wet pavement mirrors the dazzling lights, doubling the intensity of the urban glow around her. Pedestrians bustle along the sidewalks, their silhouettes blending into the dynamic, cinematic atmosphere of the neon-lit metropolis.

output.10.mp4

A cinematic movie trailer unfolds with a 30-year-old space man traversing a vast salt desert beneath a brilliant blue sky. He wears a uniquely styled red wool knitted motorcycle helmet, adding an eccentric yet rugged charm to his spacefaring look. As he rides a retro-futuristic vehicle across the shimmering white terrain, the wind kicks up clouds of glittering salt, creating a surreal atmosphere. The scene is captured in a vivid, cinematic style, shot on 35mm film to enhance the nostalgic and dramatic grain. Explosions of color and dynamic camera movements highlight the space man's daring escape from a collapsing alien base in the distance.

output.11.mp4

[asomoza]

thanks, looks cool! left some suggestions for unused imports

[cbensimon]

Suggested change

	_repeated_blocks = [
	"Kandinsky5TransformerEncoderBlock",
	"Kandinsky5TransformerDecoderBlock",
	]

Should we declare repeated blocks @sayakpaul?

[leffff]

@yiyixuxu

[yiyixuxu]

yes let's add that

[leffff]

@yiyixuxu
I've made lot's of corrections. Please review them. I have followed through the whole feedback, tackling every issue!

[yiyixuxu]

can we look into if we can refactor the attention to use dispatch_attention_fn instead, so that we can use different attention implemnentation out of box, flex or others

see this PR #11916
reference code: https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/transformers/transformer_flux.py#L118
doc: https://huggingface.co/docs/diffusers/main/en/optimization/attention_backends

[leffff]

We want to move on with the integration as soon as possible.
Can we contribute the code as is, but add support for 10 sec models later?

[leffff]

We indeed proposed a new attention algorithm.
So does that require implementing it as the new attention backend?

[yiyixuxu]

ok, so I think by default we still use SDPA, no? the flex stuff is only optional and user has to config the attention_type to be flex in order to use it - if that's the case, I think the fatest way to get this PR in is to remove all the flex related stuff for now, and add that in a follow-up PR using dispatch_attention_fn

[leffff]

Ok, I see!
You want to have a KandinskyAttnProcessor with several backends. Okay

[yiyixuxu]

yes, you should be able to structure KandinskyAttnProcessor to use dispatch_attention_fn so that it works with different backends out of box - instead of having to manually handle it like current code does
but if it will take to much time - we can just support the default one and make it work with dispatch_attention_fn in a follow-up PR

[yiyixuxu]

@leffff
i refactored kandinsky 5 attention here in this commit acabbc0
you can cherrypick that commit or just do something similar to what I did there

I tested the default SDPA backend, did not test flex but the code/logic should roughly look like that

[sayakpaul]

Just commenting to note that we support all kinds of different attention backends now. So, as long as we implement the attention class in this way, for example, swapping a backend from SDPA ("native" in our terminology) to "flex" should be very easy.

model.set_attention_backend("flex")

[leffff]

Just commenting to note that we support all kinds of different attention backends now. So, as long as we implement the attention class in this way, for example, swapping a backend from SDPA ("native" in our terminology) to "flex" should be very easy.

model.set_attention_backend("flex")

Great!
For this iteration, we are going to drop 5 sec models on regular SDPA.
In the next iteration we are going to add Nabla (on Flex)

[leffff]

@leffff i refactored kandinsky 5 attention here in this commit acabbc0 you can cherrypick that commit or just do something similar to what I did there

I tested the default SDPA backend, did not test flex but the code/logic should roughly look like that

Thank you very much! I am going to cherry pick the SDPA version!