I've made an Extension "Conditioning Highres. fix" for use with Inpaiting model!

[aleksusklim]

Here is the repo: https://github.com/klimaleksus/stable-diffusion-webui-conditioning-highres-fix

I have made a few experiments with lowering Inpainting conditioning mask strength (a new option for sd-v1-5-inpainting model, can be put in quicksettings by inpainting_mask_weight) which greatly improves img2img for the same prompt that was used for txt2img: it reduces duplication of subjects on picture.

As you know, high Denoising strength (larger than 0.5, insane after 0.8) destroys concepts when applied to large resolutions. Example: #1716

This means that for reliable highres-fix you'll have to use smaller values like 0.25−0.45, but they are often too blurry. You can double your CFG to get some sharpness, but it is still unusable on high denoisings.

But, new Inpainting model gives access to additional hyperparameter, "conditioning mask strength" which is 1.0 by default and can be lowered to 0.0 for paying more attention to existing image composition rather than new denoised content.

Cool news that it can be used not only for actual inpainting, but even for general img2img: https://www.reddit.com/r/StableDiffusion/comments/yki3hj/exploring_the_automatic1111_implementation_of/

Since it is another variable parameter, it needs some trial-and-errors to get the best value for each use-case. But there's a lot of other parameters already (to name a few: samplers, CFG, steps, resolution), not to mention Denoising strength itself!

Can we tie its value to something other and change at runtime? Is it possible to dynamically adjust Inpainting conditioning mask strength when we know the current sampler, steps, CFG, denoising strength and image resolution?

My naive testing showed that "the lower = the better", but over-saturates (like CFG does) on high denoisings but low conditioning: (you can view this grip cell-wise with the help of #3819)

Grid preview: 1 Mb, linked image 14 Mb

I came up with a simple formula: Conditioning = Denoising / 2

I believe it is very good for highres-fix: it fully prevents object duplication, it sharpens the image and adds details without oversaturating it.

This way, you will care about Denoising strength only! You might set it yourself each time, but you won't be able to draw a grid with recalculated Conditioning for individual cells.

That's why I implemented it as Extension, so it can be applied automatically behind the scene.

I suggest you to try my method for highres-fix, and if it is really useful indeed – I propose to add it to official extensions index! (If you don't want to install, you can at least set the Conditioning manually to half of current Denoising – and see how your Highres. fix improves).

If you can argue that my formula is incorrect, or not the best, or not applicable at all – then:

• Make sure that you are using highres-fix or img2img with correct generated image from txt2img (not a random picture).
• Ensure that images with C=D/2 are worse for ALL of denoising strength, not for a particular one.
• Give your prompt and parameters, proposing new formula that is better (for example it can be C=X+D/Y with new X and Y; or it can depend on CFG, or be true only for specific sampler, etc.)

[Rogal80]

Awesome!!!! It works perfectly not just only with shed-inpaint but with other models and samplers, for DDIM I am using DDIM discretize - quad this gives similar effects with high denoising

[aleksusklim]

I am using DDIM discretize - quad this gives similar effects with high denoising

I've quickly tried Quad, and I don't see magic here. It doesn't give sharpness at low levels!

It looks like it just lies about real denoising strength, shifting it up to "-1.5". So you will have sharpness only at 0.9−1.0, which corresponds to "0.7−0.8"-level coherency without Quad.

Image

[BoxJockey]

Is there a way to move the 'Denoising Strength' slider to the top of the Inpainting tab?

[aleksusklim]

My extension got added to official in-app Extensions index!
Thanks to @ClashSAN