1.1.409 New model supports

[lllyasviel]

Below new models are supported

kohya_controllllite_xl_blur_anime_beta.safetensors
kohya_controllllite_xl_blur_anime.safetensors
kohya_controllllite_xl_blur.safetensors
kohya_controllllite_xl_canny.safetensors
kohya_controllllite_xl_depth.safetensors
t2i-adapter_diffusers_xl_canny.safetensors
t2i-adapter_diffusers_xl_depth_midas.safetensors
t2i-adapter_diffusers_xl_depth_zoe.safetensors
t2i-adapter_diffusers_xl_lineart.safetensors
t2i-adapter_diffusers_xl_openpose.safetensors
t2i-adapter_diffusers_xl_sketch.safetensors

All files are mirrored at https://huggingface.co/lllyasviel/sd_control_collection/tree/main (Use this link for downloading models)

You can download all files in the above link; the list of all original sources are here (do NOT use this link for downloading models). Feel free to let us know if you are original authors of some files and want to add/remove files from the list.

You can put models in stable-diffusion-webui\extensions\sd-webui-controlnet\models or stable-diffusion-webui\models\ControlNet.

UI change: "blur" preprocessor added to "tile" group

About VRAM

All methods have been tested with 8GB VRAM and 6GB VRAM. The 6GB VRAM tests are conducted with GPUs with float16 support.

For 8GB~16GB vram (including 8GB vram), the recommended cmd flag is "--medvram-sdxl". You may edit your "webui-user.bat" as

@echo off

set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS=

call webui.bat --medvram-sdxl --xformers

For vram less than 8GB (like 6GB or 4GB, excluding 8GB vram), the recommended cmd flag is "--lowvram". You may edit your "webui-user.bat" as

@echo off

set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS=

call webui.bat --lowvram --xformers

About Speed

See the speed collection here.

[Michoko92]

Absolutely awesome! Thank you so much for your amazing work! 🙏

[lllyasviel]

wrong control mode

[Michoko92]

Ok thank you. So you mean only balanced mode is supported? Is it by design? Because the balanced mode seems to follow the input image more loosely, and I know that with SD 1.5, "controlnet is more important" allowed the most precise results.

I personally use lineart to do render style transfer: for example I have a regular digital art image I'm happy with , then use a strongly styled model ( comic book oriented) to recreate the exact same image in a comic style. Hence the need to follow the lines precisely. Maybe the same workflow is possible now with other SDXL controlnets?

Hope I make sense 🙂

[Michoko92]

Here's the one by Tencent:

Here's the one by A1111:

[lllyasviel]

what the hell - the model is trained on 384 resolution???????

why they use that low resolution of lineart ???????

well you can do the same thing with

I really do not understand this decision - well sometimes things like this just happens

I will edit that post again

[lllyasviel]

well it works

you need to use the very low resolution 384 - sometimes i really cannot get the ideas of these people especially when some models are from diffusers

[Michoko92]

Thank you very much, it seems indeed that it improves things a lot. I'll make further tests later, and report possible issues, if any. Again, thank you for your great support and work. Cheers!

Edit: would that low resolution explain why those models are so small in size?

[kasimgu2000]

I have the exact same issue with either of the Lineart versions. Lowering the preprocessor resolution to 384 doesnt seem to work either....was there any successful repeatable fix to this yet?

[CCRcmcpe]

I found controllite models have no effect at all with no errors? But T2I adapters works anyway.

[CCRcmcpe]

#2092 issue raised.

[ivanoff13]

please show examples of how it works

[Jonseed]

What does the blur_gaussian preprocesser and blur model do? I'm guessing it does something similar to Tile since it is combined in that same group, so sharpens a blurry image? How does it compare to Tile? Are there examples of these somewhere?

[if-ai]

̶I̶t̶ ̶i̶s̶ ̶n̶o̶t̶ ̶t̶o̶o̶ ̶o̶b̶v̶i̶o̶u̶s̶ ̶b̶u̶t̶ ̶I̶ ̶w̶a̶n̶t̶ ̶t̶o̶ ̶u̶s̶e̶ ̶a̶ ̶b̶i̶n̶ ̶m̶o̶d̶e̶l̶ ̶I̶ ̶f̶o̶u̶n̶d̶ ̶o̶n̶ ̶t̶h̶e̶ ̶i̶n̶t̶e̶r̶n̶e̶t̶ ̶h̶o̶w̶ ̶I̶ ̶c̶o̶n̶v̶e̶r̶t̶ ̶i̶t̶ ̶t̶o̶ ̶s̶a̶f̶e̶ ̶t̶e̶n̶s̶o̶r̶ ̶o̶r̶ ̶u̶s̶e̶ ̶t̶h̶e̶ ̶b̶i̶n̶ ̶w̶i̶t̶h̶ ̶t̶h̶e̶ ̶C̶N̶ ̶e̶x̶t̶e̶n̶s̶i̶o̶n̶ ̶h̶e̶r̶e̶ ̶i̶s̶ ̶t̶h̶e̶ ̶m̶o̶d̶e̶l̶ ̶i̶n̶ ̶q̶u̶e̶s̶t̶i̶o̶n̶ ̶h̶t̶t̶p̶s̶:̶/̶/̶h̶u̶g̶g̶i̶n̶g̶f̶a̶c̶e̶.̶c̶o̶/̶h̶9̶4̶/̶I̶P̶-̶A̶d̶a̶p̶t̶e̶r̶/̶b̶l̶o̶b̶/̶m̶a̶i̶n̶/̶m̶o̶d̶e̶l̶s̶/̶i̶p̶-̶a̶d̶a̶p̶t̶e̶r̶-̶p̶l̶u̶s̶-̶f̶a̶c̶e̶_̶s̶d̶1̶5̶.̶b̶i̶n̶

no worries I found if I change the extension to pth or safetensors it works