red noise always appears when loading

[lymanzhao]

I use the following code to load the lora I trained and the lora downloaded from civitai. The calculated pictures always have noise, as shown in the figure：
pipeline.load_lora_weights("models\lora", weight_name=loadrola01)
pipeline.fuse_lora(lora_scale=1)

[asomoza]

To be able to help you you'll need to post a reproducible code snippet with the error and also provide a lora model that has this problem, also it would help even more if you also provide the diffusers version and which OS you're using.

[lymanzhao]

prompt = "fr3u,a woman,cinematic light"
negative_prompt="noise,long neck,text,watermaek,logo,disfigured, ugly, bad, immature,frame,worst quality, low quality, letterboxed, NSFW, Cleavage, Pubic Hair, Nudity, Naked, Au naturel, Watermark, Text, censored, deformed, bad anatomy, disfigured, poorly drawn face, mutated, extra limb, ugly, poorly drawn hands, missing limb, floating limbs, disconnected limbs, disconnected head, malformed hands, long neck, mutated hands and fingers, bad hands, missing fingers, cropped, worst quality, low quality, mutation, poorly drawn, huge calf, bad hands, fused hand, missing hand, disappearing arms, disappearing thigh, disappearing calf, disappearing legs, missing fingers, fused fingers, abnormal eye proportion, Abnormal hands, abnormal legs, abnormal feet, abnormal fingers"



def XLSampler():

    global pipeline

    pipeline = StableDiffusionXLPipeline.from_pretrained(

    "repo/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16, variant="fp16", use_safetensors=True

).to("cuda")

loadrola01 = "fr3u.safetensors"

def XLora():

    pipeline.load_lora_weights("models\lora", weight_name=loadrola01)


def Imagsave():

    import random

    seed = random.randint(0,999999999999999)

    steps=40

    guidance_scale=10

    image = pipeline(prompt=prompt,negative_prompt=negative_prompt,width=1280, 

​    height=720, 

​    num_inference_steps=steps,guidance_scale=guidance_scale,generator=torch.Generator("cuda").manual_seed(seed)).images[0]

    import time

    from PIL.PngImagePlugin import PngInfo

    metadata = PngInfo()

    metadata.add_text("prompt", prompt) 

    metadata.add_text("negative_prompt", negative_prompt) 

    metadata.add_text("seed", str(seed))

    metadata.add_text("parameter", "step:"+str(steps)+",guidance_scale:"+str(guidance_scale))

    filename_number = time.time()

    global imagefilename

    imagefilename = 'output/'+ str(filename_number) +'.png'

    outimage=image.save(imagefilename, pnginfo=metadata)

    print ("save image:"+ imagefilename)

XLSampler()
XLora()
Imagsave()

[lymanzhao]

lora download from：
https://civitai.com/models/125960/lucian-freud-sdxl-10-art-style-lora

[asomoza]

I see, there's no problem with the lora or your code, so the problem should be with the info you haven't provided like the version of diffusers or your OS.

I remember there was some old versions of diffusers where if you put a long prompt it messed up the generation, get rid of that long negative prompt, trust me, for SDXL you don't need it and it could make the generations worse, if you insist in using extremely long prompts you'll need to use some kind of strategy to be able to pass the tokens to the unet, there's a community pipeline that allows the use of long prompts.

Also in the code you provided you're just loading the lora but the first post you wrote that you used the fuse_lora, so I don't know which case is the one that you have the error or if it's in both of them.

Finally, diffusers recently uses PEFT for loading the loras, I use it so I don't know if it works without it, if you aren't using the latest diffusers version with PEFT I suggest you install them because there's a lot of performance improvements.

[lymanzhao]

Thank you very much, I use win11 operating system, Diffusers 0.26,The same problem will occur whether fuse_lora is used or not.
PEFT, I found it was installed, but I have not used it.

[lymanzhao]

To add, setting adapter_name still has the same problem.

[asomoza]

ok, to test the code without peft I used a google collab and it worked, so that's not the problem either, here you can see it:

https://colab.research.google.com/drive/1Ue4Mkwd27wCRjOvs3igto0fWO3lEmG87?usp=sharing

I only removed the part for image saving.

I didn't think that was the problem but maybe what's wrong it's the VAE or the model, you can try manually loading another VAE (https://huggingface.co/madebyollin/sdxl-vae-fp16-fix). You can also try downloading the base model again from hugging face (https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0)

[lymanzhao]

Thank you, I updated the latest nvidia driver and it solved the problem. It seems that the driver is too old and has not been updated for several months.

[sayakpaul]

Looks like we have an answer here, do you mind ticking one?

Thanks to @asomoza for your helpful discussions and guidance. We truly appreciate that ❤️