how to finetune stable diffusion img2img model with only one input channel or gray scale image?

[sapkun]

Hi! experts, I want to finetune stable diffusion img2img model (instructpix2pix or controlnet) with only one input channel or gray scale image. is that possible to do that or is there any example code?
Thanks in advance.

[sapkun]

@sayakpaul Thanks for your reply.

[sapkun]

@patrickvonplaten @sayakpaul
Looking forward to your answer!
Thanks in advance!

[sayakpaul]

What happens if you just reduce the number of channels at initialization and prepare your dataset accordingly to meet the needs?

We don't have any example code snippet for this but I believe the existing code snippets should be useful references already.

[sapkun]

Thank you for your reply.
As for instruct_pix2pix, i found in here

diffusers/examples/instruct_pix2pix/train_instruct_pix2pix.py

Line 460 in 02338c9

in_channels = 8

i can change the in_channel, but as for controlnet or t2i_adapter, i didn't find anywhere to change. i wonder is that feasible to reduce the number of in_channel controlnet and t2i_adapter?