Adapting ControlNet to a New Finetuned Model

[crapthings]

I used the ControlNet training script from diffusers to obtain a model.
The model was trained based on jzli/majicMIX-realistic-7.
Then I fine-tuned this base model using the DreamBooth script from diffusers.
The ControlNet's effectiveness has decreased. Do I need to retrain ControlNet based on this new model?

i tested this script before, but the dreambooth train get a folder layout
lllyasviel/ControlNet#12

• How to convert a dreambooth trained model into safetensors that support this script?

Or

• Is there a way to adapt with diffusers like they do in a1111?

[asomoza]

Hi, first, you probably should train the controlnet with the base SD 1.5 model and not a fine tuned one, with that you can ensure it will work with all the SD 1.5 finetuned models.

That said, it should also work with some similar models, you said the effectiveness decreased that means, with that model works or not? Do you have a version of it that works good?

How to convert a dreambooth trained model into safetensors that support this script?

Safetensors is the file format, diffusers style model (what you call folder layout) are also safetensors but they're just with the correct separation in different directories with each one having a config file. The format you're probably referring as safetensors is the single file format, which is just that they group all the models in a single file container.

To convert from the diffusers format to the original single file format, you can use this script.

But to use a controlnet, you can use it with either the from_pretrained or the from_single_file, it doesn't matter.

Is there a way to adapt with diffusers like they do in a1111?

What do you mean with this? what is adapt in this case?

[crapthings]

ok, i will train it with base 1.5.
and another question is
when
base + controlnet + lora(face)
the likeness is degrade what is wrong?
but base + lora is okay
when lower controlnet strength it loose control, but increase likeness

[crapthings]

what if controlnet model doesn't have the weight same(same dataset?) as lora?
is lora work in this situation?
i mean the lora is train with new dataset, like face, background etc
i use controlnet strength .9 and condition end step .75

[asomoza]

I don't really understand this:

what if controlnet model doesn't have the weight same(same dataset?) as lora?

you mean you were training the controlnet and the lora with the same dataset before? what do you mean by weight? they're different things, usually we refer to weights as model weights which is the pretrained model.

I don't really know what are you training and on what, so it's hard to give advice here, but loras and controlnets are totally different things, controlnets are for conditioning the generation and loras are for adding/changing the knowledge of the model (finetuning).

i mean the lora is train with new dataset, like face, background etc

As I told you before, I use controlnets with loras all the time, but you have to make sure the controlnet is not interfering with the lora. For example, if the lora was trained with backgrounds of mountains, you can use a depth controlnet if you also use mountains but if the depth map is a beach with the ocean they will conflict with each other, there's also other factors in play like the prompt and the cfg too.

[crapthings]

is it possible to keep original color with controlnet trainning?
my controlnet model is doing something like inpainting(keep cloth original, fill rest), but the result cloth color changed.
its like shift hue, i don't know how to describe it properly
maybe the preprocess should use bgr map as conditional image?

[asomoza]

I don't know about SD 1.5, but probably the only controlnet for SDXL that doesn't have the color/brightness changing problem is the last controlnet union, the SD3 and Flux ones don't have that problem either. I haven't trained one so I don't really know what's the cause of the problem or how they fixed it, probably better to ask in their repos.

[crapthings]

okay

by the way the custom model is trained from scratch by using diffusers examples controlnet trainning script
i just want to keep original cloth
i don't know if its normal phenomenon

the conditional image is just like this with black blackground

without apply overlay

.image_processor.apply_overlay(mask_image_copy, input_image, output_image), apply original mask area

apply overlay

i will try train a sdxl from scratch to test if it have the same effect