Replies: 5 comments 5 replies
-
You can adapt our ControlNet training script and refer to the LoRA training scripts to do this. For example, after initializing he from peft import LoraConfig
lora_config = LoraConfig(
r=args.rank,
lora_alpha=args.rank,
init_lora_weights="gaussian",
target_modules=["to_k", "to_q", "to_v", "to_out.0"],
)
controlnet.add_adapter(lora_config) Does that help? |
Beta Was this translation helpful? Give feedback.
-
Hello @sayakpaul, I am trying to do a similar thing. I have managed to add the adapters with |
Beta Was this translation helpful? Give feedback.
-
I have tried to train a controlnet with lora, but after I merge the controlnet model and lora, the weight of controlnet seems not changed, and the output is sd, not controlnet |
Beta Was this translation helpful? Give feedback.
-
@sayakpaul I see that the SDXL Lora Script here https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth_lora_sdxl.py, it is plugging lora to the UNet of StableDiffusion, how to add the adapter to the ControlNet's down and mid blocks instead ? Can you refer me to any implementation ? When I do controlnet.add_adapter I get |
Beta Was this translation helpful? Give feedback.
-
Thanks for your requests and discussions. There are a couple of ways doing Control-LoRA training right now with the library. Option 1Use Here, you would initialize Rest of the training process should be similar. To export and use the trained LoRA parameters, you can refer to this notebook: Option 2Have the And then you can write your Hopefully, it makes sense now. |
Beta Was this translation helpful? Give feedback.
-
train full controlnet need much resource and time, so how to train controlnet with lora?
Beta Was this translation helpful? Give feedback.
All reactions