more ip-adapters

[lllyasviel]

This workload contains some more recent XL community control models.
This will mainly related to new ip-adapters and lllite controls.
For sd 1.5, a previous missing recoloring controlnet will be added then.
Other control models like controlnet XS is under investigation - if not added in this second round, then we will have a third round.
This workload will also include several bug fixes to sd-webui-controlnet 1.1.4

[lllyasviel]

If anyone know some bugs should be fixed with highest priority, please take a note here by replying this

[huchenlei]

Probably not related to the new models support, but #2148 is a known issue that is both reproducable and worth fixing.

[kft334]

If anyone know some bugs should be fixed with highest priority, please take a note here by replying this

Using any ip-adapter with only masked does not produce good results. Perform this experiment with both 'Whole image' and 'Only masked': Using image-to-image, mask a face, set ControlNet start to 25% and observe how the face changes at 25% of steps. You'll see an abrupt and dramatic change and very good likeness with 'Whole image' but a very weak change and poor likeness with 'Only masked'. I can reproduce this 100% of the time. This is with SD 1.5. I haven't tested with XL.

Here's an example for the issue. From left to right: Source, target, full res off, full res on. For both gens: no LORA, prompt = portrait of a man, strength = 0.75, start = 0.0, end = 1.0

[sdbds]

#2127
hope to support this preprocesser.

https://huggingface.co/bdsqlsz/qinglong_controlnet-lllite
add some new controlnet lllite

bdsqlsz_controlllite_xl_mlsd_V2.safetensors

bdsqlsz_controlllite_xl_normal.safetensors

bdsqlsz_controlllite_xl_t2i-adapter_color_shuffle.safetensors

bdsqlsz_controllllite_xl_segment_animeface_V2

bdsqlsz_controlllite_xl_lineart_anime_denoise

bdsqlsz_controlllite_xl_recolor_luminance

bdsqlsz_controlllite_xl_canny

[SpenserCai]

Hop to support this:
#2166

https://huggingface.co/h94/IP-Adapter/blob/main/sdxl_models/ip-adapter-plus_sdxl_vit-h.bin

[rabidcopy]

Would it be possible to support passing multiple inputs to IP-Adapter? (not a batch to be processed one by one) A very neat thing in ComfyUI right now is being able to pass multiple images of a subject or style to help the final result be more varied and less like an img2img pass. Or it can be used to mix styles and concepts simultaneously.
https://civitai.com/articles/2345/aloeveras-instant-lora-no-training-15-read-new-info
https://www.youtube.com/watch?v=HtmIC6fqsMQ
https://github.com/cubiq/ComfyUI_IPAdapter_plus

[huchenlei]

Would it be possible to support passing multiple inputs to IP-Adapter? (not a batch to be processed one by one) A very neat thing in ComfyUI right now is being able to pass multiple images of a subject or style to help the final result be more varied and less like an img2img pass. Or it can be used to mix styles and concepts simultaneously. https://civitai.com/articles/2345/aloeveras-instant-lora-no-training-15-read-new-info https://www.youtube.com/watch?v=HtmIC6fqsMQ https://github.com/cubiq/ComfyUI_IPAdapter_plus

What would be the difference between this feature and using multiple IP-Adapter ControlNet units?

[rabidcopy]

Would it be possible to support passing multiple inputs to IP-Adapter? (not a batch to be processed one by one) A very neat thing in ComfyUI right now is being able to pass multiple images of a subject or style to help the final result be more varied and less like an img2img pass. Or it can be used to mix styles and concepts simultaneously. https://civitai.com/articles/2345/aloeveras-instant-lora-no-training-15-read-new-info https://www.youtube.com/watch?v=HtmIC6fqsMQ https://github.com/cubiq/ComfyUI_IPAdapter_plus

What would be the difference between this feature and using multiple IP-Adapter ControlNet units?

Wouldn't the latter being loading the ControlNet multiple times and wasting resources and compute? Also when I tried doing that it error'd out. Edit: Correction, it appears to work that way. Though still a concern is raised about it being more cumbersome this way and performance/VRAM implications. Also the limit of ControlNet units is 10. While using the former method in ComfyUI, you are able to use many images exceeding just 10. Also passing multiple images through multiple IP-Adapter units doesn't seem to produce usable results.

[huchenlei]

Thanks for the feedback. I thought lvmin implemented the logics in e4e5807, but obviously it is not working properly. Can you file a separate issue to track this problem?

[sdbds]

#2113
Fannovel16/comfyui_controlnet_aux#71

i met same error for torch 2.1,numpy.int is another class.

i recommend change it for updating.

[George0726]

Could you mind support the ControlNet-XS models?
ControlNet-XS: https://github.com/vislearn/ControlNet-XS

[lllyasviel]

These models are all tested and now they all works.
The workload is a bit heavy and let us have other rounds for other issues.
However, these may be more important so lets just merge here first.

[lllyasviel]

help on some other tasks are also welcomed.

[sdbds]

help on some other tasks are also welcomed.

i had finished my tasks in this list, hope to review soon.

#2211 #2212