[New Extension] Continuous Prompt Interpolation

[ljleb]

We built an extension that allows to interpolate continuously between different elements in a prompt. Specifically, the extension allows to interpolate between the embeddings of different prompts, and attention values.

For example, it is possible to sample linearly from cat to dog using [cat:dog: -1, 28] or simply [cat:dog: , ] assuming 30 steps.
It is also possible to interpolate attention linearly, i.e. start sampling (cat:1) and end with (cat:2) using (cat:1,2).

https://github.com/ljleb/prompt-fusion-extension

I'm pasting the readme.md here to facilitate access 😄

Examples

These examples were generated using 30 sampler steps.

1. Influencing the beginning of the sampling process

Interpolate linearly (the default) from lion (step 0) to bird (step 7) to girl (step 10) and stay at girl for the rest of the sampling steps:

[lion:bird:girl: , 6, 9]

2. Influencing the middle of the sampling process

Interpolate using an embeddings bezier curve from fireball monster (step 0) to dragon monster (step 30 * 0.4 = step 12), while using seawater monster as an intermediate control point to steer the curve away during interpolation and get creative results:

[fireball:seawater:dragon: , .1, .4:bezier] monster

Features

• Prompt interpolation using a curve function
• Weight interpolation aware of contextual prompt editing
• Complete backwards compatibility with prompt syntax from https://github.com/AUTOMATIC1111/stable-diffusion-webui

Docs

For more information on the syntax and other features of the repo, check out the wiki.

[ClashSAN]

What samplers do you feel work best with this?

Thank you, this is 🤯!

cc: @Ehplodor (you requested this feature a while ago)

[ljleb]

Hey I'm glad you like it!

I'm not an expert, but IIUC it is more usual to pair samplers with models rather than with prompt settings. Something I noticed however is that using a greater number of sampler steps will give better results if the interpolation points are far away from each other, as it increases the resolution of the interpolation curve. In a nutshell, this makes the consecutive interpolated conditionings closer to each other.

[ClashSAN]

@ljleb What would be a summary of this technique(s) that enough normal people will understand?

Enough people get prompt editing / alternating words.

This technique is a prompt travel, like https://github.com/Kahsolt/stable-diffusion-webui-prompt-travel but those videos only use the final finished images as frames. This is for inbetween the sampling process.
It can also do https://github.com/yownas/shift-attention. but again, those are videos made with finished images, so this is (new!)

It also serves as a useful syntax for just changing the prompt each step. (if you just want to use it for improved prompt editing syntax)

[ljleb]

That's a good way to put it indeed I think! This extension allows you to prompt travel and to shift attention during the sampling process. Interpolation is a tool that allows you to either refine a prompt or let the AI do weird and unexpected things.

It is worthy of note however that Fusion does not use the latent space to travel but instead uses embedding space. In that regard, Prompt Travel is not similar to prompt interpolation in the same way that Shift Attention is to attention interpolation, as the embedding space of SD does not have the same geometry as the latent space.

I agree with you that one way to use prompt interpolation is as an alternative to prompt editing: instead of say [blue:red: 10], you could write [blue:red: 9, 11] or [blue:red: 8, 12] for example to sample an average of the embeds of both colors between a range of sampler steps. It will give different results, but not necessarily better nor worst. It's up to you to judge and decide what looks better.

[ljleb]

I think I'll add a link to these repos in the readme to help clarify the purpose. Thanks for the idea!

[ghost]

Hey, i tested your extension for a bit, seem nice and has it's uses, especially if you need to nudge subject to specific look, not entirely, but helps for sure.

One thing though, it seem to break prompts with wildcards, which i use a lot.

[ljleb]

Not sure why it breaks this other extension. I'd think either we receive a prompt with wildcards replaced or we just pass along the wildcards to the next hijacked parsing function.

Please open an issue with a link to the extension you are using and an example prompt if you want me to take a look!

[ghost]

I sure will, when i'll have time to gather all needed screenshots

[ghost]

Damn, so, it seem i was too sleepy that day. The culprit was a VERY sneaky dot after one of weights of wildcard that didn't have a number after, which threw syntax of your extension off. That's it. Sorry, if it was on your mind those days @ljleb xD

[ghost]

I wish there was the possibility to alternate multiple keywords. Currently it is only possible with [oil painting|acrylic painting]. I wish I could do multiple keywords which would then alternate until the final step. I could use the first example syntax but its a bit of a hassle to enumerate everything myself

I was quite fascinated with this extension when I also used the script that lets you see intermediate steps. Im still trying to find the best setting of percentage / steps. Sometimes it doesnt work that well because the classifier free guidance has a mind of its own and decides what the end result will be. But really good work!

[ljleb]

I'm not sure what your intention is with the prompt alternation example. This extension should not interfere with the webui behaviour of any preexisting prompt syntax. IIUC In your example it should work to just use [oil|acrylic] painting instead I think? If you meant to use the syntax from the extension, a similar pattern can be used: instead of writing [a c:b c: , ] you can write [a:b: , ] c to simplify the prompt.

For more complex cases we thought it could come in handy to introduce variables declaration and substitution, and so we implemented this feature. But there are other extensions that already implement this like unprompted, which introduces many other powerful programming-like facilities.

I agree that controlling the output with embedding interpolation isn't that straightforward in all cases. We are considering additionally implementing latent space interpolation to see if it gives better results for that purpose.

[ghost]

My example was a bit unfortunate. I would like to use something like [stunning|impressive|amazing|wonderful] painting. And these keywords would be used at step 1, 2, 3 and 4 and on step 5 they would start all over again.

Thanks for the information!

[PladsElsker]

If I understand correctly, I believe this is already a feature of the webui!

If you write as a prompt [cat|dog|duck], it should be working as described, which means:
cat at step 1, then dog at step 2, then duck at step 3, then cat again at step 4... and so on.

The fusion extension does not modify the "alternating words" feature of the webui, it should work as described in the features.