What‘s the different between torch.manual_seed() and torch.Generator(device="cuda").manual_seed() ?

[Joanne0720]

I'm using generator = torch.Generator(device="cuda").manual_seed(0) and generator = torch.manual_seed(0) in StableDiffusionXLPipeline, and get different results.

code:

import torch
from diffusers import StableDiffusionXLPipeline

pipe = StableDiffusionXLPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16)
pipe = pipe.to("cuda")

prompt = "a photo of an astronaut riding a horse on mars"

generator1= torch.manual_seed(1234)
image1 = pipe(prompt,generator=generator1).images[0]

generator2= torch.Generator(device="cuda").manual_seed(1234)
image2 = pipe(prompt,generator=generator2).images[0]

image1:

image2:

I know that generator = torch.manual_seed(1234) is as same as generator = torch.Generator(device="cpu").manual_seed(1234)

But i'm confused about:

1. Since I'm using GPU and moved model to cuda, is it necessary to use torch.Generator(device="cuda").manual_seed() to get the correct result ?
2. When i use torch.Generator(device="cpu").manual_seed(1234), why is the result also "reproducible" if the random tensors are being created in the GPU ?
3. How does the seed work during the generation process ?

[asomoza]

Diffusers has a nice article in the documentation about this:

https://huggingface.co/docs/diffusers/using-diffusers/reproducibility#control-randomness

For your question, even if you pass a CPU generator and the pipeline is in the GPU, the pipelines use a custom function from diffusers:

diffusers/src/diffusers/utils/torch_utils.py

Line 37 in 9112028

def randn_tensor(

where your answer is in the docstring:

A helper function to create random tensors on the desired `device` with the desired `dtype`. When 
passing a list of generators, you can seed each batch size individually. If CPU generators are passed, 
the tensor is always created on the CPU.

As to how the seed works in the generation, I have a general understanding of it but I'm no expert so I prefer that someone else answers that question.

[Joanne0720]

Wow thank you!
The article you shared was incredibly useful and addressed the my second question!

🧨 Diffusers has a randn_tensor() function for creating random noise on the CPU, and then moving the tensor to a GPU if necessary. The randn_tensor function is used everywhere inside the pipeline, allowing the user to always pass a CPU Generator even if the pipeline is run on a GPU.

So I guess that's the reason why I moved pipeline to GPU and passed a CPU Generator, and it still worked !
🫶🏻🫶🏻🫶🏻

[asomoza]

I forgot about the first question, there is no correct result in stable diffusion just different generations, you're not going to get the same result if you use the same seed in the CPU or in the GPU, even if you change the precision from float to bfloat the result would drastically change.

Since the GPUs are made to be fast, they're less deterministic than the CPU, so if you want to be able to reproduce the image again, you should stick with creating the generator in the CPU.

[sayakpaul]

There are some ways we still try to ensure full determinism:

• Such as disabling the non-deterministic CUDA kernels. Here's a very nice documentation: https://pytorch.org/docs/stable/notes/randomness.html#avoiding-nondeterministic-algorithms.
• Running everything in full-precision because lower precision usually leads to the launch of faster kernels provided the underlying GPU has specialized cores to leverage them.
• Running everything using the vanilla attention instead of using an efficient form of attention.

We have two relevant methods:

• enable_full_determinism()
• disable_full_determinism()

https://github.com/huggingface/diffusers/blob/main/tests/pipelines/stable_diffusion/test_stable_diffusion.py provides some nice examples.

[Joanne0720]

@sayakpaul
Thank you for your additional input! I'll give the method you mentioned a try.
So when I'm passing a GPU Generator, I should also enable_full_determinism()

[sayakpaul]

@Joanne0720 can you try to create the generator on CPU?

generator = torch.manual_seed(0)

[Joanne0720]

Yes sure! I'm currently comparing this two different generators.
While reproducibility is certainly important, I'm more concerned with the quality of the outcomes.

Could you please share your thoughts on which method tends to produce superior quality?

Thank you 🙌

[sayakpaul]

Improving quality is a bit of an art. There's no single answer to that.

We have a document here that provides some initial guidance. https://huggingface.co/docs/diffusers/main/en/stable_diffusion