Symmetry implementation custom script

[1ort]

Hello! I want to write a custom script that will potentially allow you to get images with a choice of horizontal, vertical or other symmetry.

I know that I can prompt this, but I want to get more predictable behavior. In addition, the neural network does not understand horizontal symmetry well, you don’t even have to think about angular and their combinations.

I am currently considering several options:

1. Somehow mirror the noise received from the seed.
2. Mirror the image every n steps. Perhaps the first n steps

Now I am actively studying the code, but many points are poorly documented or not documented.
Which way should I look?
Perhaps, Someone has already tried to implement this and it worked or failed. This information is also of interest to me.

upd. I settled on this code and it seems to work. Examples below. There is still a lot of work to be done. Currently does not support generating multiple images at a time and various bugs and crashes are possible
https://gist.github.com/1ort/2fe6214cf1abe4c07087aac8d91d0d8a

[ClashSAN]

is this related? horizontal/vertical tiling https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Custom-Scripts#asymmetric-tiling

[1ort]

Hi I have been looking into this code. I'm afraid this doesn't work for me because I want to get an image with a symmetrical composition
.

[1ort]

Apparently I need to overload one of the functions responsible for image processing each step and then manipulate the numpy array if I want to implement "force symmetry". I think cutting the picture in half and mirroring one half every n steps or at some specific step is a good idea. I will dwell on it.
It remains to figure out how to do all this with an numpy array

[victorca25]

From the linked script, this is the important part: https://github.com/tjm35/asymmetric-tiling-sd-webui/blob/76c8bfcf6475e9427c6621dcb682d2ace70fd218/asymmetric_tiling.py#L47 where it is converting the padding of the conv layer from constant to circular, forcing each axis to be more or less consistent at the edges.

It should be easy to follow your idea of copying and flipping the tensor, you can test this function offline (in a separate script or in Collab, etc) to visualize that the dimension flipping is done correctly. Combining the original and flipped tensors for the symmetry should also be easy, look for tensor slicing so you grab only half of the flipped tensor and replace that in the original.

[1ort]

Update. I was able to implement the symmetry of the generated noise and apparently this affects the composition of the image a little. This should work with all samplers. Below are two examples. The seed is the same, the prompt was "face". In the first example I applied noise symmetry by simply reflecting a copy of the tensor and adding the two halves.

If you are interested in research in this direction - I invite you to join me and share your experience. At the moment, I'm struggling with applying symmetry every n steps in the generation process.

I still could not figure out how to correctly decode and encode the latent, related question here: #2543

I will be glad to any suggestions.

upd. Also, it seems to me that applying such a technique to the generated noise is not the best option. I can't explain why, I just feel it. The images on which the neural network was trained are in most cases asymmetric, and the noise is random by definition.

[1ort]

Update!
I have implemented a script that applies symmetry to the image every n steps and sends the result further to img2img. The method of applying the symmetry will have to change because there is now a "double exposure" effect.
But the results make me happy now

[1ort]

Proof of concept script. There is still a lot of work to be done. Currently does not support generating multiple images at a time

https://gist.github.com/1ort/2fe6214cf1abe4c07087aac8d91d0d8a

[1ort]

Update.
https://gist.github.com/1ort/2fe6214cf1abe4c07087aac8d91d0d8a

Implemented new "apply_symmetry" method, which seems a little bit better

[ClashSAN]

@1ort trying your script alot of images I'm getting are hazy. The results you've gotten look nice, would you tell me what settings you've used? Maybe make a writeup and repo, so we can link this better.

[1ort]

Hello. Through testing, I came to the conclusion that it is better to set "skip last n steps" as much as we can. Think of this as the number of steps in img2img for the final post-processing.

For example, if the total number of steps is 50, I would put skip steps around 20-25.
In turn, apply every step at too low values ​​will not draw the details, so we set it around 5-10. Try these settings. So far, I have not been able to work on the script. I think it makes sense to set these default settings lately.

Also, if the total steps is 50, then it makes sense to set the apply every step to 1, and set the skip last steps to 49. Then we will get a more or less symmetrical composition.

Also, you shouldn't enable the alternative mode checkbox, because it turns on the old symmetry function, which simply reflects and superimposes the image on itself without cutting it.

[B34STW4RS]

This could be very useful for designing character reference sheets before finalizing the design with inpainting, good work implementing a proof of concept so quickly.

[dfaker]

For ancestral samplers there's another option of messing with the KDiffusionSampler.callback_state callback, what you have as mirrored blending, or just flipping the image on alternate steps both work, but only for ancestral samplers, and it eventually breaks denosing resulting in excessive blur.

    def callback_state(self, d):
        step = d['i']
        latent = d["denoised"]

        store_latent(latent)
        self.last_latent = latent

        if step < state.sampling_steps*opts.sampler_mirroring_fraction:
            if opts.sampler_mirroring_method == 'Alternate Sample Flip':
                d["x"][:, :, :, :] = torch.flip(d["x"], [3])
            elif opts.sampler_mirroring_method == 'Avergage Flipped Copy':
                d["x"][:, :, :, :] = (torch.flip(d["x"], [3]) + d["x"])/2

        if self.stop_at is not None and step > self.stop_at:
            raise InterruptedException

        state.sampling_step = step
        shared.total_tqdm.update()

Can be quite subtle if you keep the steps it ends at low:

[dfaker]

After a bit of experimentation, putting the flips in the forward pass of CFGDenoiser seems to play best with all k-diffusion samplers:

class CFGDenoiser(torch.nn.Module):
    def __init__(self, model):
        super().__init__()
        self.inner_model = model
        self.mask = None
        self.nmask = None
        self.init_latent = None
        self.step = 0

    def forward(self, x, sigma, uncond, cond, cond_scale, image_cond):
        if state.interrupted or state.skipped:
            raise InterruptedException

        conds_list, tensor = prompt_parser.reconstruct_multicond_batch(cond, self.step)
        uncond = prompt_parser.reconstruct_cond_batch(uncond, self.step)

        batch_size = len(conds_list)
        repeats = [len(conds_list[i]) for i in range(batch_size)]

        x_in = torch.cat([torch.stack([x[i] for _ in range(n)]) for i, n in enumerate(repeats)] + [x])
        image_cond_in = torch.cat([torch.stack([image_cond[i] for _ in range(n)]) for i, n in enumerate(repeats)] + [image_cond])
        sigma_in = torch.cat([torch.stack([sigma[i] for _ in range(n)]) for i, n in enumerate(repeats)] + [sigma])

        if state.sampling_step < state.sampling_steps*opts.sampler_mirroring_fraction:
            if opts.sampler_mirroring_method == 'Alternate Sample Flip':
                if opts.sampler_mirroring_mode == 'V-mirror':
                    x_in[:, :, :, :] = torch.flip(x_in, [3])
                elif opts.sampler_mirroring_mode == 'H-mirror':
                    x_in[:, :, :, :] = torch.flip(x_in, [2])
                elif opts.sampler_mirroring_mode == 'Rot-90':
                    x_in[:, :, :, :] = torch.rot90(x_in, dims=[2, 3])
                elif opts.sampler_mirroring_mode == 'Rot-180':
                    x_in[:, :, :, :] = torch.rot90(torch.rot90(x_in, dims=[2, 3]),dims=[2, 3])
            elif opts.sampler_mirroring_method == 'Avergage Flipped Copy':
                if opts.sampler_mirroring_mode == 'V-mirror':
                    x_in[:, :, :, :] = (torch.flip(x_in, [3]) + x_in)/2
                elif opts.sampler_mirroring_mode == 'H-mirror':
                    x_in[:, :, :, :] = (torch.flip(x_in, [2]) + x_in)/2
                elif opts.sampler_mirroring_mode == 'Rot-90':
                    x_in[:, :, :, :] = (torch.rot90(x_in, (2, 3)) + x_in)/2
                elif opts.sampler_mirroring_mode == 'Rot-180':
                    x_in[:, :, :, :] = (torch.rot90(torch.rot90(x_in, dims=[2, 3]),dims=[2, 3]) + x_in)/2
            ...

The 'Alternate Flip' method works much better than merging here, Producing some quite unexpectedly creative mirrored compositions at low sample cut-offs:

Horiz:

Vert:

180-rotation:

90-degree rotations tend to either make abstract dinner plates or loosely frame central subjects:

As well as the desired hard symmetry

Img2img also works, with the denoising levels one would expect for a pretty through transformation of the image:

[grexzen]

These are extraordinary gens,

[dfaker]

Small PR offered to open up a call-back into CFGDenoiser here: #4021

With a demo-script if that lands: https://gist.github.com/dfaker/ac031e87174a94d8a170d897caac9ff6

But much more hackery is in theory possible, like overlaying alternate or swapping totally different latent image representations or boosting contrast, hue and vignetting in image latent representations.

[1ort]

Looks very interesting and more elaborate than my original implementation. Would you like to create a separate repository?

[dfaker]

Would you like to create a separate repository?

If that PR is accepted the code to take advantage of it (excluding callback setup) is tiny, I'm not sure the specific mirroring part warrants a separate repo more than a few lines in an existing script.

[chromesun]

To @1ort and @dfaker
I thought this symmetry stuff sounded really interesting so I wanted to have a go with both your scripts. I dl’d force_symmetry.py and latent_mirroring_script.py to my scripts folder and started up the webui (d98eace from 2nd Nov).

I’m using a 2060/6GB, with COMMANDLINE_ARGS set to: --medvram --xformers --listen --ckpt "D:\stable-diffusion-webui-master\models\Stable-diffusion\v1-5-pruned-emaonly.ckpt"

Both “Force symmetry” and “Mirroring” show up in txt2img’s Script list.
Only “Mirroring” shows up in img2img’s Script list.
Looking at the python code I had expected both to show up only in the img2img Script box. (This is my first experience of Python so I’m not sure I’m reading it correctly.)

Decided to have a go with “Force symmetry” in txt2img anyway. I get an output pic with, erm :-), something done to it all right. I’ll need to play with parameters to get a feel for them. My output folder also gets a noised/blurred pic during the generation. I don’t have any of the “Save a copy of image before” boxes ticked in Settings.

Setting Script back to “None” and generating a new txt2img works fine.

Using the “Mirroring” script in txt2img I also get some fascinating output images, although no extra saved interim image this time.

Setting Script back to “None” and generating a new txt2img does not work correctly. I get the same image generated (I’m using fixed seed) as when “Mirroring” was active. In fact all txt2img generations with any prompt are now affected by “Mirroring”

If I now select “Force Symmetry” again, what I get is an output image that seems to have the effects of “Force Symmetry” and “Mirroring” combined!

The only way I’ve found to stop “Mirroring” taking effect is to ctrl-c out of the console and restart the webui.

I have a 2nd machine with a 1650s/4GB, same webui=d98eace, and the same thing happens.

Am I doing something stupid? It wouldn’t be the first time :-)

[dfaker]

There we go, guard to make sure the callback only runs when the script is loaded, updated version here: https://github.com/dfaker/SD-latent-mirroring

[chromesun]

Great, working properly :-) Thank you. By the way, your example images are excellent!

[dfaker]

Sounds like the callbacks aren't being removed when the script is switched away from, which I thought was a feature of the script runner, I'll take a look.

[pablonaj]

Hi, I tried the script and it's really good, love the results it's giving. Thanks for working on it!

I would like to combine it with the x/y plot script to explore some combinations, any idea how that could work? Or any plans to move it to a plugin so it can be used separately from other scripts?

[dfaker]

Glad you're enjoying it!

Since this requires a callback the easiest way I can see is to move that code into a custom x/y plot script.

[dfaker]

As for the extension conversion, the repo is converted to that format now, and defaults to 'None' mirroring.

[pablonaj]

Perfect, that's exactly what I needed, I can use it now with the x/y plot script! Thanks so much!

[pablonaj]

Hi, thanks for the extension, I've been using it a lot...

I found a bug today: when using high-res fix the mirroring is applied again when it starts generating the larger image, basically breaking the original image and defeating the purpose of the high res fix.

[stefanhuber1993]

Fascinating thread and interesting code examples!

All these approaches depend on "symmetrising" the latents, either by averaging, copying one half to the other side or transforming the latent space (e.g. flipping).

This seems to work well when symmetry is only imposed during early iterations, while for later iterations the images become abstract or smeared out.

I was wondering if you have ideas how the symmetry could be baked directly into the denoising process of Stable Diffusion, somehow forcing it to converge to symmetric solutions during the image generation process without disturbing the latent space representation.

[stefanhuber1993]

Also, here is a fork of @dfaker 's repo. I realised that only imposing symmetry during a few selectable iterations (provided by a list like "0.1, 0.2, 0.3") can give cleaner results, compared to imposing symmetry on every iteration until a cutoff.
https://github.com/stefanhuber1993/SD-latent-symImpose

This is the same approach that is available in DiscoDiffusion.

[andupotorac]

Wouldn't regional prompting solve this?

[mibri77]

Hi, I need to apply this exact use-case but not sure how to apply it to SDXL model directly without using a1111?