"Local prompts"

[ETCore7]

Not sure if this is possible. Right now the text prompt affects the whole image and/or selections. It would be great to have a global image prompt plus optional "local prompts" for selections. For example, image prompt 'landscape' generates a landscape, while a selection A gets prompt 'tree' and another selection B gets prompt 'house', so you get a landscape with a tree and a house.

[jccluaviz]

I think this can be achived with inpainting.

First create a landscape with a prompt. Then select the area where you wanna draw a tree or a house and modify the prompt so you can ajust it to a tree or a house.

I hope this can help you.

[ETCore7]

Thank you for your reply.

I know this can be done with inpainting, but last year I saw an interface where areas were moved around, each with their own prompt. It was very easy to arrange compositions since the local prompts did not affect the rest of the image. In Krita this could be something like assigning prompts to layers or local selections, but I don't know whether this is possible with text.

Of course it already works with pure img2img. When you move a layer with i.e. a banana it will get inpainted at its new position, but the AI may interpret the banana as something else in relation to other objects or its perceived perspective. For example, text prompt "table with fruit" on a light gray canvas and a layer with an orange and a layer with a banana can turn the babana into a watermelon piece or a yellow adjuma pepper. Labels for the objects would prevent this. And of course the main prompt could be something like "Orange and banana on table", but local prompts would offer more flexibility. You may want to try a brown banana, or a peeled banana, or a banana with a sticker, or a different shape. A fixed img2img banana is kind of static.

Again, I don't know whether this is possible, I was just wondering. As for the demo I think I saw it on Youtube, channel TwoMinutePapers.

[Acly]

Similar ideas discussed in #386 #387

Something like this definitely makes sense, but it needs some time to try out approaches. Just enabling masked conditioning is not that useful in my experience, it works for simple examples like this, but I find inpainting more reliable. Maybe a mix of both.

[ETCore7]

Thank you, it was GLIGEN as mentioned in #387.

I saw it here. The OP made a GUI to be used with Comfy, it can be downloaded here: github/mut-ex/gligen-gui.

[Danamir]

As mentioned in the other tickets, I've started a PR #639 that may be of interest in this case.