Clarification on noise scheduler mismatch in example train_controlnet.py

[vidyasiv]

Hello,

In examples/controlnet/train_controlnet.py, the noise scheduler class is DDPMScheduler but however the scheduler config loaded based on the one in the model's scheduler folder i.e PNDM Scheduler per https://huggingface.co/runwayml/stable-diffusion-v1-5/blob/main/scheduler/scheduler_config.json . Why don't they match?

In particular, I see output that follows model config and not the scheduler class: Loaded scheduler as PNDMScheduler from scheduler subfolder of runwayml/stable-diffusion-v1-5.

@patrickvonplaten

[vidyasiv]

I was likely confused by __class__ which reflects the python class we ascribe and matches the code override and the variable _class_name which seems to remain PNDMScheduler (from the config?) but correct me if I am wrong.

from diffusers import StableDiffusionPipeline
from diffusers import (
    DDPMScheduler,
    DDIMScheduler,
    PNDMScheduler,
    LMSDiscreteScheduler,
    EulerAncestralDiscreteScheduler,
    EulerDiscreteScheduler,
    DPMSolverMultistepScheduler,
)

repo_id = "runwayml/stable-diffusion-v1-5"

ddpm = DDPMScheduler.from_pretrained(repo_id, subfolder="scheduler")
ddim = DDIMScheduler.from_pretrained(repo_id, subfolder="scheduler")
pndm = PNDMScheduler.from_pretrained(repo_id, subfolder="scheduler")
lms = LMSDiscreteScheduler.from_pretrained(repo_id, subfolder="scheduler")
euler_anc = EulerAncestralDiscreteScheduler.from_pretrained(repo_id, subfolder="scheduler")
euler = EulerDiscreteScheduler.from_pretrained(repo_id, subfolder="scheduler")
dpm = DPMSolverMultistepScheduler.from_pretrained(repo_id, subfolder="scheduler")

for scheduler in [ddpm, ddim, pndm, lms, euler_anc, euler, dpm]:
    pipeline = StableDiffusionPipeline.from_pretrained(repo_id, scheduler=scheduler, use_safetensors=True)
    print(f" Python class: {pipeline.scheduler.__class__} and _class_name:{pipeline.scheduler._class_name}")

Output (cleaned up to only show prints) I see is:

Python class: <class 'diffusers.schedulers.scheduling_ddpm.DDPMScheduler'> and _class_name:PNDMScheduler
Python class: <class 'diffusers.schedulers.scheduling_ddim.DDIMScheduler'> and _class_name:PNDMScheduler
Python class: <class 'diffusers.schedulers.scheduling_pndm.PNDMScheduler'> and _class_name:PNDMScheduler
Python class: <class 'diffusers.schedulers.scheduling_lms_discrete.LMSDiscreteScheduler'> and _class_name:PNDMScheduler
Python class: <class 'diffusers.schedulers.scheduling_euler_ancestral_discrete.EulerAncestralDiscreteScheduler'> and _class_name:PNDMScheduler
Python class: <class 'diffusers.schedulers.scheduling_euler_discrete.EulerDiscreteScheduler'> and _class_name:PNDMScheduler
Python class: <class 'diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler'> and _class_name:PNDMScheduler

[patrickvonplaten]

Hey @vidyasiv,

Note that when loading a the pipeline for runwayml/stable-diffusion-v1-5, we're loading the model for inference, not for training.
For inference the checkpoint works with an array of schedulers (also see doc here).

The DDPM scheduler in its original form is very slow when used for inference as it requires 1000 inference steps (see paper here) - it's prefered to use faster schedulers such as PNDM for increase speed for inference.

Also note that the original implementation of runwayml/stable-diffusion-v1-5 uses PNDM/PLMS for inference (see here).

[vidyasiv]

Thanks for clarifying @patrickvonplaten . I too realized yesterday the training vs inference scheduler. I understand now.