smaller image encoder for IP Adapter

[kmpartner]

Are there smaller image encoder for IP Adapter?

Official IP Adapter repository using laion/CLIP-ViT-H-14-laion2B-s32B-b79K, that is very large in size. Are there smaller and faster image encoder that work with IP Adapter?

[SahilCarterr]

You can try using this model laion/CLIP-ViT-B-16-laion2B-s34B-b88K as This is a B (Base) version with a smaller patch size (16x16), trained on 34B tokens and 88K steps.

[kmpartner]

When I tried to use laion/CLIP-ViT-B-16-laion2B-s34B-b88K, I got error. How can I solve this problem?

/usr/local/lib/python3.10/dist-packages/torch/nn/modules/linear.py in forward(self, input)
115
116 def forward(self, input: Tensor) -> Tensor:
--> 117 return F.linear(input, self.weight, self.bias)
118
119 def extra_repr(self) -> str:

RuntimeError: mat1 and mat2 shapes cannot be multiplied (50x768 and 1280x768)

[kmpartner]

After using IP Adapter's config settings, image can be generated. But it failed to get input image structure or colors. Some config changes did not help to improve image.

Since I am not familiar with image encoder and clip, I am not sure, but is it possible to change config to get better images? (sdxl ip-adapter seems to work with sd1.5 image encoder) Or Is it required to re-training using new image encoder?

[xiankgx]

Errrr, you shouldn't be changing the image encoder "simply", especially if you are just using the model for predictions (generations). IP adapter variants are trained with specific image encoders and you should only use the image encoder that was trained.

You could of course change the image encoder and then finetune it for your usecase.

[a-r-r-o-w]

Moving to discussion as this is not a diffusers-specific issue. Like @xiankgx said, we can't arbitrarily use any image encoder with any model for generation. It might work in some cases, but will/should not generally. You'd have to train your model for every image encoder that you'd like to use.