Stable Diffusion XL Inpainting struggles with masked background areas, often failing to generate new content there

[ammarft-ai]

I am working on an Inpainting task and I am using Stable diffusion XL 1.0 Inpainting. It is working very well when I mask people, animals or objects, but when I give it a mask of background, it doesn't generate anything and give me the original input image. I have tried changing parameters as well but nothing could make it work. Below is the approach I am using:

model_id = "diffusers/stable-diffusion-xl-1.0-inpainting-0.1"

model = AutoPipelineForInpainting.from_pretrained(
     model_id, 
     torch_dtype = torch.float16,
     cache_dir = "model_cache",
)

model = pipeline.to("cuda")

if base64_img is not None:
     img_data = base64.b64decode(base64_img)
     img = Image.open(io.BytesIO(img_data))

     print("Recieved image of size: ", img.size)
        
if base64_mask is not None:
     mask_data = base64.b64decode(base64_mask)
     mask = Image.open(io.BytesIO(mask_data))

     print("Recieved mask of size: ",mask.size)

img = img.convert("RGB")
mask = mask.convert("L")
        
generator = torch.Generator(device="cuda").manual_seed(0)
        
image = model(
     prompt=prompt,
     negative_prompt=IMG_INPAINTING_NEG_PROMPT,
     image=img,
     mask_image=mask,
     guidance_scale=8.0,
     num_inference_steps=25,
     strength=strength,
     generator=generator,
).images[0]

I can't understand how to make this work as this is one of the primary requirements.

[ukaprch]

I don't find any problems with inpainting in SDXL. What you're essentially doing is a replace vs an insert. Normally for insert you mask the object of interest and fill it in with something new. For replace, you mask the areas you wish to retain and then do this with your mask if a 'PIL' image:

            mask_image = Image.fromarray(255 - mask),

So what you're essentially doing is creating a masked background (reversing the black / white mask areas) of what you want to replace.
As an aside, a new feature was added to inpainting, see the argument below which you include in your model above.
padding_mask_crop=padding_mask_crop,

padding_mask_crop is similar to dilation in image segmentation. It expands your mask area by 'X' number of pixels.

Hope this helps.

[ammarft-ai]

It kinda worked, like now it is adding the object in the prompt but also adds some weird borders around it. I am attaching the output

[crapthings]

maybe you can make mask larger and apply this

https://github.com/Photoroom/fast-foreground-estimation

[yiyixuxu]

cc @asomoza