support Inpaint models #511

stduhpf · 2024-12-04T22:42:43Z

Adds support for sd1.x inpaint, sd2.x inpaint, sdxl inpaint, and Flux Fill models, as well as "inpainting" (masked img2img) with normal diffusion models.

How to use

Download weights: (example: https://huggingface.co/webui/stable-diffusion-inpainting/blob/main/sd-v1-5-inpainting.safetensors)
run in img2img mode with the --mask argument to specify the path to the masking image
I recommend setting strength to 1

Examples:

Using Inpaint model:

sd.exe -M img2img --model ..\models\sd-v1-5-inpainting.safetensors -p "a lovely dog" --cfg-scale 16 --sampling-method euler -t 24 --color --steps 30 --vae-tiling --strength 1 -i '.\inpaint-base.png' --mask '.\inpaint-mask.png'

Input image	mask	result

Using normal model (not as consistent):

sd.exe -M img2img --model ..\models\unet\sd3.5_medium-q6_k.gguf --clip_g ..\models\clip\clip_g.q8_0.gguf --clip_l ..\models\clip\clip_l.q8_0.gguf --t5xxl ..\models\clip\t5xxl_q4_k.gguf -p 'a lovely dog sitting patiently on a park bench' --cfg-scale 4.5 --sampling-method euler --steps 30 -t 24 -W 512 -H 512 --seed 0 --color --vae-tiling --strength 1 -i '.\New folder\inpaint-base.png' --mask '.\New folder\inpaint-mask.png'

Input image	mask	result

Caveats

Implementing DDIM scheduler would probably help get better results, but this is already pretty good with the default one.
Mask needs to be the same size as the image, resizing doesn't work.
Since is based on img2img, the original image is always added to the random noise distribution, even at strength 1. This can be useful, for example to guide the generation by sketching what you want on the image, but it could also lead the model to regenerate what you want to remove from the image. If you want to avoid that, pre-fill the masked parts with 50% gray in the input image.
Parts of the image outside the inpainting area can be slightly modified, especially with heavily quantized models, which tend to add artefacts all over the image.

Fixes #105

SD1 inpaint models

Amin456789 · 2024-12-06T09:16:20Z

this is great, thank u so much, please add outpainting while u r at it, i think they are the same
with this and ltx video sd cpp will be the best sd out there

stduhpf · 2024-12-06T14:58:05Z

@Amin456789 You can set it up to outpaint already, you just need to manually prepare the image and mask (I recommend pre-filling parts to inpaint with 50% grey, since it's based on img2img)

sd.exe -M img2img --model ..\models\sd-v1-5-inpainting.safetensors -p "." --cfg-scale 0 --sampling-method euler -t 24 --color --steps 30 --vae-tiling --strength 1 -i '.\outpaint-base.png' --mask '.\outpaint-mask.png' -W 768

Input image	mask	result

Amin456789 · 2024-12-06T16:16:07Z

greaat. thank u
@fszontagh please add this outpainting feature to ur gui too to make it easier than manually doing it if possible

fszontagh · 2024-12-06T18:05:33Z

greaat. thank u @fszontagh please add this outpainting feature to ur gui too to make it easier than manually doing it if possible

If the pr is merged, i will add to it. This is already in my to-do 🥳

stduhpf · 2024-12-07T18:40:31Z

Input image	mask	result

with Q3_K diffusion model, --guidance set to 35 (Very high guidance seems to be required to get good results for some reason)

thxCode · 2024-12-25T03:14:42Z

model.cpp

+    bool is_xl   = false;
+    bool is_flux = false;
+
+#define found_family (is_xl || is_flux)


found_family seems to impact the detecting of sd3.

Could you elaborate? In my testing, sd3 seems to be detected reliably...

nvm, with the recommended safetensors loading, it works with sd3. however, if text_encoders.clip_g.* is converted to cond_stage_model.1.*(gguf mode), the following processing will be judged as sdxl, and sd3 will not be recognized.

@thxCode what about now?

thxCode · 2024-12-25T03:17:11Z

model.cpp

+    if (is_xl) {
+        if (is_inpaint) {
+            return VERSION_SDXL_INPAINT;
+        }
+        return VERSION_SDXL;
+    }


sdxl takes priority over flux.

Is this a problem? In theory it should be impossible for a model to be deteted as SDXL and Flux at the same time, no?

ditto, this should only be encountered after merging all FLUX components into one gguf file following cond_stage_model.* / first_stage_model.* / model.diffusion_model.* format.

thxCode · 2024-12-25T03:18:44Z

stable-diffusion.cpp

+
+        struct ggml_tensor* concat = is_inpaint ? ggml_new_tensor_4d(work_ctx, GGML_TYPE_F32, 8, 8, 5, 1) : NULL;
+        ggml_set_f32(timesteps, 0);
+


here seems to impact sd2

What kind of impact?

sorry, this might be a false positive from my testing(i remember testing on 4090 resulting in sd2 using v-prediction mode instead, but I can't reproduce it on my mac m1 max.).

btw, it looks strange here. timesteps changed from 999 to 0 without any judgment? or do you mean setting concat to 0?

btw, it looks strange here. timesteps changed from 999 to 0 without any judgment? or do you mean setting concat to 0?

Oh right, that's exactly what I meant to do.

leejet · 2024-12-28T05:04:57Z

Thank you for your contribution.

stduhpf added 7 commits December 4, 2024 22:25

Add support for loading

67919e6

SD1 inpaint models

inpaint: try things

5674601

Actually Support Inpaint models

048b8f7

unet: Fix c_concat

f1e6b0d

support image mask input

5a03915

Do not crash in txt2img with Inpaint model

a040094

Fix issue introduced by rebase

8ee3512

fszontagh mentioned this pull request Dec 5, 2024

[WIP] Inpaint in GUI fszontagh/sd.cpp.gui.wx#23

Open

stduhpf added 5 commits December 5, 2024 17:59

Add "inpaint" support for diffusion models in img2img via denoise mask

10c16da

inpaint: fix halo on blurry masks

a40f4c0

sd2 inpaint support

148731d

sdxl inpaint support

8085e48

remove some logging

a9e7ee6

stduhpf changed the title ~~SD1.x Inpaint support~~ support Unet Inpaint models Dec 6, 2024

Fix non inpaint sd2

ed41e75

Fix model_version _str

d741e2d

stduhpf force-pushed the inpaint branch from 22d1fd0 to d741e2d Compare December 6, 2024 18:08

stduhpf added 2 commits December 7, 2024 00:57

Detect Flux fill models

26fab5a

Flux fill load

29b6fd8

stduhpf force-pushed the inpaint branch from 7ecc88b to bacbccc Compare December 7, 2024 00:22

Flux Fill working!!

0683c03

stduhpf force-pushed the inpaint branch from bacbccc to 0683c03 Compare December 7, 2024 00:26

stduhpf changed the title ~~support Unet Inpaint models~~ support Inpaint models Dec 7, 2024

thxCode reviewed Dec 25, 2024

View reviewed changes

Stéphane du Hamel added 2 commits December 26, 2024 19:57

fix mistake in sd2 mode check

17b4fc5

more specific sdxl fingerprint

fe35689

leejet merged commit 8f4ab9a into leejet:master Dec 28, 2024
9 checks passed

stduhpf added a commit to stduhpf/stable-diffusion.cpp that referenced this pull request Dec 28, 2024

feat: support Inpaint models (leejet#511)

573252f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support Inpaint models #511

support Inpaint models #511

stduhpf commented Dec 4, 2024 •

edited

Loading

Amin456789 commented Dec 6, 2024

stduhpf commented Dec 6, 2024 •

edited

Loading

Amin456789 commented Dec 6, 2024

fszontagh commented Dec 6, 2024

stduhpf commented Dec 7, 2024 •

edited

Loading

thxCode Dec 25, 2024

stduhpf Dec 25, 2024

thxCode Dec 26, 2024

stduhpf Dec 26, 2024

thxCode Dec 27, 2024

thxCode Dec 25, 2024

stduhpf Dec 25, 2024

thxCode Dec 26, 2024

thxCode Dec 25, 2024

stduhpf Dec 25, 2024

thxCode Dec 26, 2024 •

edited

Loading

stduhpf Dec 26, 2024

leejet commented Dec 28, 2024


		struct ggml_tensor* concat = is_inpaint ? ggml_new_tensor_4d(work_ctx, GGML_TYPE_F32, 8, 8, 5, 1) : NULL;
		ggml_set_f32(timesteps, 0);

support Inpaint models #511

support Inpaint models #511

Conversation

stduhpf commented Dec 4, 2024 • edited Loading

How to use

Examples:

Using Inpaint model:

Using normal model (not as consistent):

Caveats

Amin456789 commented Dec 6, 2024

stduhpf commented Dec 6, 2024 • edited Loading

Amin456789 commented Dec 6, 2024

fszontagh commented Dec 6, 2024

stduhpf commented Dec 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thxCode Dec 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leejet commented Dec 28, 2024

stduhpf commented Dec 4, 2024 •

edited

Loading

stduhpf commented Dec 6, 2024 •

edited

Loading

stduhpf commented Dec 7, 2024 •

edited

Loading

thxCode Dec 26, 2024 •

edited

Loading