[LoRA] Support original format loras for HunyuanVideo

[a-r-r-o-w]

Fixes #10106 (comment)

Some LoRAs for testing:

• https://huggingface.co/svjack/Genshin_Impact_XiangLing_Low_Res_HunyuanVideo_lora_early
• https://huggingface.co/Cseti/HunyuanVideo-LoRA-Arcane_Jinx-v1
• https://huggingface.co/trojblue/HunyuanVideo-lora-AnimeShots

import torch
from diffusers import HunyuanVideoPipeline, HunyuanVideoTransformer3DModel
from diffusers.utils import export_to_video

model_id = "hunyuanvideo-community/HunyuanVideo"
transformer = HunyuanVideoTransformer3DModel.from_pretrained(model_id, subfolder="transformer", torch_dtype=torch.bfloat16)
pipe = HunyuanVideoPipeline.from_pretrained(model_id, transformer=transformer, torch_dtype=torch.float16)
pipe.load_lora_weights("svjack/Genshin_Impact_XiangLing_Low_Res_HunyuanVideo_lora_early", weight_name="xiangling_ep1_lora.safetensors", adapter_name="hunyuan-lora")
pipe.set_adapters("hunyuan-lora", 0.8)
pipe.vae.enable_tiling()
pipe.to("cuda")

output = pipe(
    prompt=".....",
    height=320,
    width=512,
    num_frames=61,
    num_inference_steps=30,
    generator=torch.Generator().manual_seed(42),
).frames[0]
export_to_video(output, "output.mp4", fps=15)

cc @svjack Would you be able to give this a try?

[a-r-r-o-w]

@sayakpaul @DN6 Not comfortable adding integrations tests with any of the existing original-format loras on the Hub/CivitAI for HunyuanVideo. They are either Anime/NSFW/Person-related loras which we usually don't use for testing. I'll work on training a style lora over the next few days and convert it to original format for integration tests. Does that sound good?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[sayakpaul]

Perfect! Okay to merge for me after an integration test!

[sayakpaul]

We usually add a check on original_state_dict if it's been emptied for converters like this. Do we wanna do that here?

diffusers/src/diffusers/loaders/lora_conversion_utils.py

Line 969 in cd991d1

if len(original_state_dict) > 0:

[nitinmukesh]

Subscribing to this PR

[yardenfren1996]

Hi, does Diffusers have a training script available for training HunyuanVideo LoRAs?

[sayakpaul]

We do, thanks to @a-r-r-o-w 🐐
Check out https://github.com/a-r-r-o-w/finetrainers