[1.1.222] Preprocessor: inpaint_only+lama

[lllyasviel]

1.1.222 added a new inpaint preprocessor: inpaint_only+lama

LaMa: Resolution-robust Large Mask Inpainting with Fourier Convolutions (Apache-2.0 license)
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, Victor Lempitsky
(Samsung Research and EPFL)

The basic idea of "inpaint_only+lama" is inspired by Automaic1111’s upscaler design: use some other neural networks (like super resolution GANs) to process images and then use Stable Diffusion to refine and generate the final image.

We bring the similar idea to inpaint. When you use the new inpaint_only+lama preprocessor, your image will be first processed with the model LAMA, and then the lama image will be encoded by your vae and blended to the initial noise of Stable Diffusion to guide the generating.

The results from inpaint_only+lama usually looks similar to inpaint_only but a bit “cleaner”: less complicated, more consistent, and fewer random objects. This makes inpaint_only+lama suitable for image outpainting or object removal.

Comparison

Input image (this image is from Stability's post about Clipdrop)

Configuration:

(The Resize Mode "Resize and Fill" enables outpaint)

Results:

inpaint_only
(9 non-cherry-picked raw outputs)

inpaint_only+lama
(9 non-cherry-picked raw outputs)

You can see that inpaint_only+lama is a bit "cleaner" than inpaint_only (and also a little bit more robust).

You need at least 1.1.222 to use this feature.

[AlespRless]

very nice in hand fix

[yemenyirannianqing]

nice job! could you talk about hand fix steps? maybe share the configuration? thanks a lot.

[Woisek]

Look good!
Are we still restricted in size when doing the outpainting? And will there be an option to define in which direction to outpaint to? Not every time a "zoom out" is wanted. Would be great!

[catboxanon]

Seems like a nice improvement!

As with the last update, I still seem to be getting nicer results when using this in combination with the reference preprocessor. When outpainting using only inpaint_only+lama it seems more inclined to produce more blobby/blurry shapes to fill in the remaining picture, rather than trying to make something more believeable. In my short testing so far it's moreso a problem for illustration/anime artwork (in contrast, your example of a photo works perfectly fine). It's easier to see if you a/b the examples below in a new tab.

All examples directly below use the same parameters above, with the exception of the input images, and these settings for the model:

reference preprocessor settings

original image

Original, highres was used from https://danbooru.donmai.us/posts/6196599. Lower res version below just to preview here.

inpaint_only

inpaint_only AND reference_adain+attn @ 0.5 strength, 0.5 style fidelity

inpaint_only+lama (better results)

inpaint_only+lama AND reference_adain+attn @ 0.5 strength, 0.5 style fidelity (much better results, imo)

I also tried using the same settings for the example in the OP but with the addition of the reference preprocessor, just to see if it helps at all. I verified I could reproduce the original results beforehand, so the only difference in this example below is the addition of the reference preprocessor.

OP example, with reference_adain+attn @ 0.5 strength, 0.5 style fidelity

In this case, it actually makes the results worse. I'm guessing this has something to do with the reference image becoming expanded and cropped to a small, wide area in the center (due to the Crop and Resize resize mode). That same crop behavior occurrs for the anime illustration example as well though, so I don't think it necessarily explains how it helps in one case but not another. Just Resize and Resize and Fill don't make much sense to use either since the former introduces stretching, and the latter introduces borders, which affect the generated image to then take those "artifacts" into account. Using the image w/o modifications seems like the obvious way the reference preprocessor should use the image but I'm guessing that's a limitation of the method (I'm not 100% familiar with how it functions, so if that actually could work, that should probably be fixed).

If I were to throw in my own two cents, I do think that ultimately no current publicly available models can fully handle illustrated artwork very well (not even Waifu Diffusion), so I don't think it's a fault of the system but the base models/data. I would love to be proven wrong though.

---

Edit: To make my above statement more concrete, these are non-cherry picked outputs directly from Clipdrop's Uncrop. These are objectively the best results out of all my examples. I don't think any model weights publicly available can quite compete yet (Uncrop is using SDXL, as the blog post mentions, which has many advantages including being trained with RLHF, and is not yet available). I would test with Adobe Firefly as well but I believe outpainting can only be done in the app, which I do not own. SDXL has the promise of eventually being released though which is a plus.

As an aside, giving an input prompt should always improve results (it does for me at least), but since the goal here is for promptless in/outpainting I think these comparisons are still important.

Clipdrop Uncrop

[lllyasviel]

Interesting results; it seems that this image does not need an anime model

v1-5-pruned-emaonly+vae840000, No Prompt, seed 12345

And

realisticVisionV20_v20+vae840000, No Prompt, seed 12345

(Above tests use same configs in this post)

[catboxanon]

That is interesting! I wonder what exactly causes models heavily finetuned towards anime to produce worse results. It's not always the case but it does seem to crop up often. I remember this same phenomenon occurring here too, where back when control model offsets were still a feature, the result looked worse when the weights from AnythingV3 were applied.

[tsound97]

I found that if I enable inpaint, reference, and IP2P at the same time can have a very good inpaint result.

Origin picture by Aitasai

Parameters I use

1 girl wear a bikini
Negative prompt: EasyNegativeV2
Steps: 20, Sampler: DPM++ SDE Karras, CFG scale: 4.5, Seed: 1822710821, Size: 512x768
(To speed up the testing process, I did not enable the highres fix or merge an inpainting model, both of which can produce better results)

Only use inpaint_only+lama (ControlNet is more important)

As you can see, the pose of the girl change

|
|
|

If add an openpose controlnet to control the pose?

the pose is even stranger

|
|
|

Use inpaint_only+lama (ControlNet is more important) + IP2P (ControlNet is more important)

The pose of the girl is much more similar to the origin picture, but it seems a part of the sleeves has been preserved. The result is bad.
(Reducing the weight of IP2P controlnet can mitigate this issue, but it also makes the pose go wrong again)

|
|
|

Use inpaint_only+lama (ControlNet is more important) +reference_adain+attn (Balanced, Style Fidelity:0.5) + IP2P (ControlNet is more important)

Here comes the result, bingo!

[2blackbar]

Great stuff, do You guys plan to add also zits model which is superior to default lama ? it does wireframe and has better looking results, might not be that much of improvement when sd is generating image anyway but its definitely my default model in lama cleaner

[CyberLykan]

Is ControlNet inpaint currently capable of inpainting transparent pixels (as in fixing a tilted photo) or accepting a custom inpaint mask? It seems to just stretch the border pixels unless I draw a mask myself.

[SLAPaper]

Consistently getting color&contrast difference when outpainting, any ideas?

original:

outpainting:

Steps: 40, Sampler: Euler a, CFG scale: 4, Seed: 12346, Size: 512x512, Model hash: a7e26193ce, Model: original_sd-v1-5-inpainting, Denoising strength: 0.4, Conditional mask weight: 1.0, RNG: CPU, ControlNet: "preprocessor: inpaint_only, model: control_v11p_sd15_inpaint [ebff9138], weight: 1, starting/ending: (0, 1), resize mode: Resize and Fill, pixel perfect: False, control mode: ControlNet is more important, preprocessor params: (64, 64, 64)", Hires upscale: 2, Hires steps: 10, Hires upscaler: R-ESRGAN 4x+ Anime6B, Version: v1.3.2-1-gc96567cc

Configs:

[lllyasviel]

it seems that you need "Steps: 50, Sampler: DPM++ 2M SDE Karras, CFG scale: 4". use this as a sanity check to see what is missing:

v1-5-pruned-emaonly + CN:

no prompt, Steps: 50, Sampler: DPM++ 2M SDE Karras, CFG scale: 4, Seed: 12345, Size: 512x512, Model hash: cc6cb27103, Model: v1-5-pruned-emaonly, ControlNet 0: "preprocessor: inpaint_only+lama, model: control_v11p_sd15_inpaint [ebff9138], weight: 1, starting/ending: (0, 1), resize mode: Resize and Fill, pixel perfect: True, control mode: ControlNet is more important, preprocessor params: (64, 64, 64)", Version: v1.3.2

sd-v1-5-inpainting + CN:

no prompt, Steps: 50, Sampler: DPM++ 2M SDE Karras, CFG scale: 4, Seed: 12345, Size: 512x512, Model hash: c6bbc15e32, Model: sd-v1-5-inpainting, Conditional mask weight: 1.0, ControlNet 0: "preprocessor: inpaint_only+lama, model: control_v11p_sd15_inpaint [ebff9138], weight: 1, starting/ending: (0, 1), resize mode: Resize and Fill, pixel perfect: True, control mode: ControlNet is more important, preprocessor params: (64, 64, 64)", Version: v1.3.2

[SLAPaper]

@lllyasviel Thanks for your examples! After some testing, here is my observations:

1. It seems that straight text-to-image works fine with correct model conf yaml
2. normal sd1.5 model tends to achieve slightly better color matching but a lot worse shape matching than sd1.5 inpainting model, at least in this outpainting scene
3. some issue with hires fix workflow as suggested in [1.1.202 Inpaint] Improvement: Everything Related to Adobe Firefly Generative Fill #1464 , some upscalers make color faded more, especially in low denoising strength

sd1.5 + CN

Steps: 50, Sampler: DPM++ 2M SDE Karras, CFG scale: 4, Seed: 12345, Size: 512x512, Model hash: cc6cb27103, Model: original_v1-5-pruned-emaonly, ControlNet 0: "preprocessor: inpaint_only+lama, model: control_v11p_sd15_inpaint [ebff9138], weight: 1, starting/ending: (0, 1), resize mode: Resize and Fill, pixel perfect: True, control mode: ControlNet is more important, preprocessor params: (64, 64, 64)", Version: v1.4.0-RC-1-g81966672

sd1.5 inpainting + CN

Steps: 50, Sampler: DPM++ 2M SDE Karras, CFG scale: 4, Seed: 12345, Size: 512x512, Model hash: c6bbc15e32, Model: original_sd-v1-5-inpainting, Conditional mask weight: 1.0, RNG: CPU, ControlNet 0: "preprocessor: inpaint_only+lama, model: control_v11p_sd15_inpaint [ebff9138], weight: 1, starting/ending: (0, 1), resize mode: Resize and Fill, pixel perfect: True, control mode: ControlNet is more important, preprocessor params: (64, 64, 64)", Version: v1.4.0-RC-1-g81966672

sd1.5 inpainting + CN + hires-fix 2x with different denoising strength & upscaler

Steps: 50, Sampler: DPM++ 2M SDE Karras, CFG scale: 4, Seed: 12345, Size: 512x512, Model hash: c6bbc15e32, Model: original_sd-v1-5-inpainting, Denoising strength: 0.01, Conditional mask weight: 1.0, RNG: CPU, ControlNet 0: "preprocessor: inpaint_only+lama, model: control_v11p_sd15_inpaint [ebff9138], weight: 1, starting/ending: (0, 1), resize mode: Resize and Fill, pixel perfect: True, control mode: ControlNet is more important, preprocessor params: (64, 64, 64)", Hires upscale: 2, Hires steps: 10, Hires upscaler: Lanczos, Script: X/Y/Z plot, X Type: Denoising, X Values: "0.01,0.5,1", Y Type: Hires upscaler, Y Values: "Lanczos,R-ESRGAN 4x+ Anime6B", Version: v1.4.0-RC-1-g81966672

[NeoAnthropocene]

This is a very helpful thread. Thanks for sharing with us 👍

[mlundstrom23]

How do you get ControlNet 1.1.222? I just updated mine but it's only on v1.1.208

Edit: nvm got it to work, my settings were just off.

[gigend]

where's nvm setting? how to update it?

[FrancisDacian]

I guess nvm means never mind bro

[dan4ik94]

Yea, choosing between lama models would be nice. I like zits too.

[matrix4767]

So is txt2img inpainting possible without resulting in a blurry and/or seamed mask? What settings do you actually need to avoid that?

[almfahd41]

thanks for the inpaint

[vonpheus]

HELP. When I select Inpaint+ llama the model controlV11p inpaint... is not showing so I just left it at none. SInce the tutorial I watched said it'll get downloaded the first time I use inpaint+ llama. But it doesnt seem so. How'd you guys get that model? thanks

[dan4ik94]

Get it from here:
https://huggingface.co/comfyanonymous/ControlNet-v1-1_fp16_safetensors/tree/main

[vonpheus]

Oh okay thank you so much!

[StableInfo]

Hello Illyas, I hope people are supporting you through some patreon or something. or just donations. Thanks again

[dzzhuipang]

I expanded the image according to the Auther's settings, tried N times, the result is that the expanded part is not completely transparent, please give me some advice, thanks.

[SLAPaper]

You are using "ip2p" CN model which means "Instruct Pix2Pix", while the inpainting model is called "inpaint"

[dzzhuipang]

I see, thanks

[2blackbar]

It looks like inpaint works well in txt2img and masking in small controlnet window but img2img or inpaint mode still gives result with lots of artifacts, How i can make img2img to work like txt2img ?
Especially the seams from inpainting are very visible in img2img or inpaint but in txt2img they are not visible at all.
Issue with inpaint in txt2img is that i have to guess my image size and with inpaint only masked i dont have to but only masked doesnt give as good results.
Maybe if in "only masked" mode it would try to send whole image to controlnet inpaint so it would better calculate only masked area

[huchenlei]

#1771

There are definitely some people misuse img2img inpaint. Can you describe what masking and input configuration are you using?

[2blackbar]

You know, it would be faster to let me know what actually works, i tried them all, padding pixels tto 256, only masked, its the one im interested with cause it preserves resolution, so i tried crop,resioze,resizenfill , no mater what it gives me low res result even with 680x800 res definitely much lower compared to actual 680x800 without using inpaint controlnet , i can lower padding pixels but then it would not know whats outside of the inpainted area... and would not do inpainting as seamless
Tried to give him armor, also preserve the pose if possible, most of the time i get bad result, once in a while theres something passable like 20 bad ones and 1 good.

man wearing armor
Negative prompt:
Steps: 20, Sampler: DPM2 a Karras, CFG scale: 5, Seed: 3493963238, Size: 608x640, Model hash: fc52756a74, Model: dreamlikePhotoreal20_, Denoising strength: 1, Mask blur: 14, ControlNet 0: "preprocessor: inpaint_only, model: control_v11p_sd15_inpaint_fp16 [be8bc0ed], weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: ControlNet is more important, preprocessor params: (-1, -1, -1)", Version: ## 1.4.0

This is rsult from adobe firefly which had like 8 results and 2 good ones where pose matched
https://firefly.adobe.com/generate/inpaint

[Tillerz]

Inpaint + Lama is currently broken for me in Vladmantic's a1111 branch.

Problem one: vladmandic/automatic#1603 - resize mode is greyed out
Problem two: vladmandic/automatic#1571 - RuntimeError: The size of tensor a (3) must match the size of tensor b (0) at non-singleton dimension 2

Anyone any ideas about those?

[huchenlei]

Problem one is expected behaviour. Hover on the greyed out radio will tell you why.

[Tillerz]

I have no idea how to get rid of the tensor error. So right now that extension is completely broken for me.

[huchenlei]

Seems like the same tensor error here? #1766 (comment)

Can you provide the specific configurations you use that produce the crash? (Preferable screenshot)

[Tillerz]

Here you go:

=========================
Checkpoint and VAE:

=========================
img2img / Controlnet Settings:

Steps: 20, Sampler: UniPC, CFG scale: 6, Seed: 3322998812, Size: 512x768,
Model hash: ed989d673d, Model: 2023-07_dreamshaper_7, VAE: vae-ft-mse-840000-ema-pruned,
Denoising strength: 0.75, Clip skip: 1, Version: 3bcca6f, Token merging ratio: 0.5,
Parser: Full parser, ControlNet 0: "preprocessor: inpaint_only+lama,
model: control_v11p_sd15_inpaint [ebff9138], weight: 1, starting/ending: (0, 1),
resize mode: Crop and Resize, pixel perfect: False,
control mode: ControlNet is more important, preprocessor params: (-1, -1, -1)"

=========================
SD Settings:

=========================
System Settings:

[Tillerz]

Still broken for me with the above message.

[mandal-rahul]

Feature Request: txt2img inpainting should also have a mask upload. Would be easier to inpaint through mask instead of drawing the mask every time with a brush

[wywywywy]

I keep getting visible seams when using this for outpainting.

Anyone got any tips on avoiding this please?

Original

Outpainted (see the visible seams!)

Preprocessor output

ControlNet settings

[Guillaume-Fgt]

Hi, I did some tests with your image. I had the same results with a marked seam. But I found a solution: if I draw a mask on the edges of the image in the ControlNet tab, the seams disappear:

Not cherry pick batch of 4:

I suppose the mask tells ControlNet to consider this area as part of the generation. But it didn't seem to be necessary in the example at the top of the post.

[wywywywy]

This works remarkably well. Thank you very much for the tip!

Drawing the 2 lines is a bit tedious. Hopefully a "feathering" feature will be added in the future!

[wywywywy]

Guess what! A feathering feature has been added! #1813

Thanks @catboxanon

[aulerius]

When I am outpainting via API, by using txt2img and manually providing the mask, I always get the seams and color mismatch.

From left to right: ControlNet input, mask, and output.

How can I make use of this feathering feature, or do whatever was done in that webui example? Providing a feathered mask via API seems to do nothing, as I assume the code converts it to binary anyway.

[dzzhuipang]

I have tried many times, the pic of your email, I have generated new pic of a cat, and also generated other type of pic, all of them have this problem, there is a clearly boundary between the extended part and the original pic, the brighter and simple patterns is easer to see. I just tested the function, not actually use it , the background of the pic I used for testing was dark, so I didn't notice this problem, and I don't know how to solve it.

[ZongWei-HUST]

Is it possible to reproduce the inpaint_only+lama preprocessor out of stable-diffusion-webui, as an API? I'm trying to integrate using gradio_inpaint.py from https://github.com/lllyasviel/ControlNet-v1-1-nightly and https://github.com/advimman/lama, but it's hard to achieve webui results.

[bigcornflake]

+1

[huchenlei]

I did it sucessfully with API in my photopea plugin:
https://github.com/huchenlei/stable-diffusion-ps-pea#features

It is nothing fancy, you just pass the necessary input image and mask to A1111.

[tiimgreen]

What does that mean, can you go into a bit more detail? the text2img endpoint doesn't let you pass a mask and I can't see how to pass one to control net unless I'm missing something.

[limingyu1993]

me too，lama preprocesser need a mask? I try controlnet inpaint+lama in comfyui，the resulting image will darken in color

[andupotorac]

Can this be used to remove an object from a photo?

[Guillaume-Fgt]

This makes inpaint_only+lama suitable for image outpainting or object removal.

Yes, first message.

For example, you want to remove the plant from this interior:

Using txt2img tab:

Result:

[StableInfo]

could you have had the same result without a positive prompt, and just a word in negative prompt?

[andupotorac]

No, I tried with negative prompt and positive too and didn't work.

[andupotorac]

Thanks!

[StableInfo]

Hello @lllyasviel , this method is still lacking in term of control, I want a more effiscient method, can you check my request here: lllyasviel/ControlNet#508
Best

[0sparsh2]

Would anyone be knowing on how to implement this in diffusers pipeline? (The Lama + Inpaint preprocessor majorly)

[2blackbar]

Can You guys add 3rd option to use just lama result? I can see that masked result from inpaint + lama is clean and it removes objects without issues while with 2 existing inpaint methods (only and +lama) you have to try a dozen times before it removes the object.
I think it would be very useful to have just lama result and even better would be having zits model from lama.

[lawsonxwl]

When using inpaint_only+lama pipeline，there is still random object present in the generated mask area! How can we solve this problem

[zengyh1900]

Hi all,

I want to share our recent model for image inpainting, PowerPaint.
It can be the most powerful inpainting model that enables text-guided object inpainting, text-free object removal, and image outpainting, achieving comparable performance as Adobe Firefly.

Importantly, it can effectively remove objects without the random object present.

youtube: https://youtube.com/watch?v=7QsiY1JQUfg
online demo: https://openxlab.org.cn/apps/detail/rangoliu/PowerPaint
arxiv: https://arxiv.org/abs/2312.03594
code: https://github.com/open-mmlab/mmagic/tree/main/projects/powerpaint
We are looking forward to any feedback!

[zengyh1900]

@lawsonxwl thank you for reporting. Would you mind sharing the results?

[lawsonxwl]

do you mind adding wechat? Or do you have a wechat group about your project? In company I can't upload image

[lawsonxwl]

cfg：15
ddim：30steps
prompt：pos：P_ctxt，neg：P_obj
seed：0

[zengyh1900]

@lawsonxwl thanks for providing this case. It's wired. I test it via online demo https://openxlab.org.cn/apps/detail/rangoliu/powerpaint
I cannot reproduce your results

Sure. My wechat id is the same as my GitHub name zengyh1900

[icsl-Jeon]

Hello. Thank you for your amazing work!
I am curious how did you optimize prompt and unet at the same time..?
You wrote that T2I unet is used (presumably, not sd1.5-inpainting model) but could not find mentions about finetune unet.

[ride5k]

sadly, no lama option for AMD users. at least not one i've come across yet... am I missing something?

[zhuang2002]

We have recently developed a new version of PowerPaint, drawing inspiration from Brushnet. The Online Demo has been updated accordingly. In the coming days, we intend to make the model weights available as open source.

The PowerPaint model possesses the ability to carry out diverse inpainting tasks, such as object insertion, object removal, shape-guided object insertion, and outpainting. Notably, we have retained the cross-attention layer that BrushNet had removed, which is essential for task prompt input.

	Object insertion	Object Removal	Shape-guided Object Insertion	Outpainting
Original Image
Output