(macOS) modified to use DDIM and PLMS, suggestions needed on how to improve speed. #3885

FreeBlues · 2022-10-28T19:48:47Z

FreeBlues
Oct 28, 2022

GPU: [eGPU] eGPU Radeon Rx 6600xt
VRAM: [8G]
CPU arch: [x86] macbookpro 2020, Intel Core i5, 32G ram
OS: [macOS] Monterey 12.6
Browsers: [Safari]
Python: [miniconda3]
Branch: [main]
Commit: []

When I use DDIM to generate an image, it broken and return error AssertionError: Torch not compiled with CUDA enabled, according to the error info, I open the file File "/Users/ppt/Github/stable-diffusion-webui/repositories/stable-diffusion/ldm/models/diffusion/ddim.py", line 23, in register_buffer.

the original code is:

def register_buffer(self, name, attr):
        if type(attr) == torch.Tensor:
            if attr.device != torch.device("cuda"):
                attr = attr.to(torch.device("cuda"))

I modified it to below:

def register_buffer(self, name, attr):
        if type(attr) == torch.Tensor:
            if attr.device != torch.device("cuda"):
                if torch.cuda.is_available():
                    attr = attr.to(torch.device("cuda"))
                elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
                    attr = attr.to(torch.device("mps"))
                else:
                    attr = attr.to(torch.device("cpu"))

Run it and get another error: TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead.

I modified again to below:

def register_buffer(self, name, attr):
        if type(attr) == torch.Tensor:
            if attr.device != torch.device("cuda"):
                if torch.cuda.is_available():
                    attr = attr.to(torch.device("cuda"))
                elif hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
                    attr = attr.to(torch.float32).to("mps")
                else:
                    attr = attr.to(torch.device("cpu"))

It works now, but it is very very slow, about 45s/it, just like it is running on the CPU only, but in fact the GPU usage becomes high:

Steps to reproduce the problem

Using a old MacBook Pro (Intel CPU) with eGPU
Run web-ui
Generate image using ddim or plms.

What should have happened?

It should work(after modify the code, it can work now).
The speed shouldn't so slow, it should more faster.

Commit where the problem happens

3e15f8e

What platforms do you use to access UI ?

MacOS

What browsers do you use to access the UI ?

Apple Safari

Command Line Arguments

start web with:

`python webui.py --precision=full --no-half`

prompt:

`'wide shot, horizon zero down, digital art, highly detailed, digital painting, symmetry, concept art, sharp focus, illustration, art by artgerm! greg rutkowski magali villeneuve wlop! ilya kuvshinov!!, octane render ', '', 'None', 'None', 20, 9, False, False, 1, 1, 7, -1.0, -1.0, 0, 0, 0, False, 512, 512, False, 0.7, 0, 0, 0, False, False, None, '', 1, '', 0, '', True, False, False`

Additional information, context and logs

1st error log AssertionError: Torch not compiled with CUDA enabled

(web-ui) ppt@pptdeMacBook-Pro stable-diffusion-webui %  python webui.py --precision=full --no-half
WARNING:root:Pytorch pre-release version 1.13.0.dev20221002 - assuming intent to test it
Warning: caught exception 'Torch not compiled with CUDA enabled', memory monitor disabled
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
Loading weights [7460a6fa] from /Users/ppt/Github/stable-diffusion-webui/models/Stable-diffusion/sd-v1-4.ckpt
Global Step: 470000
Applying cross attention optimization (InvokeAI).
Model loaded.
Loaded a total of 1 textual inversion embeddings.
Embeddings: learned_embeds
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Error completing request
Arguments: ('wide shot, horizon zero down, digital art, highly detailed, digital painting, symmetry, concept art, sharp focus, illustration, art by artgerm! greg rutkowski magali villeneuve wlop! ilya kuvshinov!!, octane render ', '', 'None', 'None', 20, 9, False, False, 1, 1, 7, -1.0, -1.0, 0, 0, 0, False, 512, 512, False, 0.7, 0, 0, 0, False, False, None, '', 1, '', 0, '', True, False, False) {}
Traceback (most recent call last):
  File "/Users/ppt/Github/stable-diffusion-webui/modules/sd_samplers.py", line 251, in sample
    samples_ddim = self.launch_sampling(steps, lambda: self.sampler.sample(S=steps, conditioning=conditioning, batch_size=int(x.shape[0]), shape=x[0].shape, verbose=False, unconditional_guidance_scale=p.cfg_scale, unconditional_conditioning=unconditional_conditioning, x_T=x, eta=self.eta)[0])
  File "/Users/ppt/Github/stable-diffusion-webui/modules/sd_samplers.py", line 139, in launch_sampling
    return func()
  File "/Users/ppt/Github/stable-diffusion-webui/modules/sd_samplers.py", line 251, in <lambda>
    samples_ddim = self.launch_sampling(steps, lambda: self.sampler.sample(S=steps, conditioning=conditioning, batch_size=int(x.shape[0]), shape=x[0].shape, verbose=False, unconditional_guidance_scale=p.cfg_scale, unconditional_conditioning=unconditional_conditioning, x_T=x, eta=self.eta)[0])
  File "/Users/ppt/miniconda/envs/web-ui/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/Users/ppt/Github/stable-diffusion-webui/modules/sd_hijack_inpainting.py", line 56, in sample_ddim
    self.make_schedule(ddim_num_steps=S, ddim_eta=eta, verbose=verbose)
  File "/Users/ppt/Github/stable-diffusion-webui/repositories/stable-diffusion/ldm/models/diffusion/ddim.py", line 38, in make_schedule
    self.register_buffer('betas', to_torch(self.model.betas))
  File "/Users/ppt/Github/stable-diffusion-webui/repositories/stable-diffusion/ldm/models/diffusion/ddim.py", line 23, in register_buffer
    attr = attr.to(torch.device("cuda"))
  File "/Users/ppt/miniconda/envs/web-ui/lib/python3.10/site-packages/torch/cuda/__init__.py", line 221, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ppt/Github/stable-diffusion-webui/modules/ui.py", line 221, in f
    res = list(func(*args, **kwargs))
  File "/Users/ppt/Github/stable-diffusion-webui/webui.py", line 63, in f
    res = func(*args, **kwargs)
  File "/Users/ppt/Github/stable-diffusion-webui/modules/txt2img.py", line 48, in txt2img
    processed = process_images(p)
  File "/Users/ppt/Github/stable-diffusion-webui/modules/processing.py", line 437, in process_images
    samples_ddim = p.sample(conditioning=c, unconditional_conditioning=uc, seeds=seeds, subseeds=subseeds, subseed_strength=p.subseed_strength)
  File "/Users/ppt/Github/stable-diffusion-webui/modules/processing.py", line 585, in sample
    samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.create_dummy_mask(x))
  File "/Users/ppt/Github/stable-diffusion-webui/modules/sd_samplers.py", line 253, in sample
    samples_ddim = self.launch_sampling(steps, lambda: self.sampler.sample(S=steps+1, conditioning=conditioning, batch_size=int(x.shape[0]), shape=x[0].shape, verbose=False, unconditional_guidance_scale=p.cfg_scale, unconditional_conditioning=unconditional_conditioning, x_T=x, eta=self.eta)[0])
  File "/Users/ppt/Github/stable-diffusion-webui/modules/sd_samplers.py", line 139, in launch_sampling
    return func()
  File "/Users/ppt/Github/stable-diffusion-webui/modules/sd_samplers.py", line 253, in <lambda>
    samples_ddim = self.launch_sampling(steps, lambda: self.sampler.sample(S=steps+1, conditioning=conditioning, batch_size=int(x.shape[0]), shape=x[0].shape, verbose=False, unconditional_guidance_scale=p.cfg_scale, unconditional_conditioning=unconditional_conditioning, x_T=x, eta=self.eta)[0])
  File "/Users/ppt/miniconda/envs/web-ui/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/Users/ppt/Github/stable-diffusion-webui/modules/sd_hijack_inpainting.py", line 56, in sample_ddim
    self.make_schedule(ddim_num_steps=S, ddim_eta=eta, verbose=verbose)
  File "/Users/ppt/Github/stable-diffusion-webui/repositories/stable-diffusion/ldm/models/diffusion/ddim.py", line 38, in make_schedule
    self.register_buffer('betas', to_torch(self.model.betas))
  File "/Users/ppt/Github/stable-diffusion-webui/repositories/stable-diffusion/ldm/models/diffusion/ddim.py", line 23, in register_buffer
    attr = attr.to(torch.device("cuda"))
  File "/Users/ppt/miniconda/envs/web-ui/lib/python3.10/site-packages/torch/cuda/__init__.py", line 221, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

2nd error log: TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead.

(web-ui) ppt@pptdeMacBook-Pro stable-diffusion-webui %  python webui.py --precision=full --no-half
WARNING:root:Pytorch pre-release version 1.13.0.dev20221002 - assuming intent to test it
Warning: caught exception 'Torch not compiled with CUDA enabled', memory monitor disabled
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
Loading weights [7460a6fa] from /Users/ppt/Github/stable-diffusion-webui/models/Stable-diffusion/sd-v1-4.ckpt
Global Step: 470000
Applying cross attention optimization (InvokeAI).
Model loaded.
Loaded a total of 1 textual inversion embeddings.
Embeddings: learned_embeds
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Error completing request
Arguments: ('wide shot, horizon zero down, digital art, highly detailed, digital painting, symmetry, concept art, sharp focus, illustration, art by artgerm! greg rutkowski magali villeneuve wlop! ilya kuvshinov!!, octane render ', '', 'None', 'None', 20, 9, False, False, 1, 1, 7, -1.0, -1.0, 0, 0, 0, False, 512, 512, False, 0.7, 0, 0, 0, False, False, None, '', 1, '', 0, '', True, False, False) {}
Traceback (most recent call last):
  File "/Users/ppt/Github/stable-diffusion-webui/modules/sd_samplers.py", line 251, in sample
    samples_ddim = self.launch_sampling(steps, lambda: self.sampler.sample(S=steps, conditioning=conditioning, batch_size=int(x.shape[0]), shape=x[0].shape, verbose=False, unconditional_guidance_scale=p.cfg_scale, unconditional_conditioning=unconditional_conditioning, x_T=x, eta=self.eta)[0])
  File "/Users/ppt/Github/stable-diffusion-webui/modules/sd_samplers.py", line 139, in launch_sampling
    return func()
  File "/Users/ppt/Github/stable-diffusion-webui/modules/sd_samplers.py", line 251, in <lambda>
    samples_ddim = self.launch_sampling(steps, lambda: self.sampler.sample(S=steps, conditioning=conditioning, batch_size=int(x.shape[0]), shape=x[0].shape, verbose=False, unconditional_guidance_scale=p.cfg_scale, unconditional_conditioning=unconditional_conditioning, x_T=x, eta=self.eta)[0])
  File "/Users/ppt/miniconda/envs/web-ui/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/Users/ppt/Github/stable-diffusion-webui/modules/sd_hijack_inpainting.py", line 56, in sample_ddim
    self.make_schedule(ddim_num_steps=S, ddim_eta=eta, verbose=verbose)
  File "/Users/ppt/Github/stable-diffusion-webui/repositories/stable-diffusion/ldm/models/diffusion/ddim.py", line 54, in make_schedule
    self.register_buffer('ddim_sigmas', ddim_sigmas)
  File "/Users/ppt/Github/stable-diffusion-webui/repositories/stable-diffusion/ldm/models/diffusion/ddim.py", line 26, in register_buffer
    attr = attr.to(torch.device("mps"))
TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ppt/Github/stable-diffusion-webui/modules/ui.py", line 221, in f
    res = list(func(*args, **kwargs))
  File "/Users/ppt/Github/stable-diffusion-webui/webui.py", line 63, in f
    res = func(*args, **kwargs)
  File "/Users/ppt/Github/stable-diffusion-webui/modules/txt2img.py", line 48, in txt2img
    processed = process_images(p)
  File "/Users/ppt/Github/stable-diffusion-webui/modules/processing.py", line 437, in process_images
    samples_ddim = p.sample(conditioning=c, unconditional_conditioning=uc, seeds=seeds, subseeds=subseeds, subseed_strength=p.subseed_strength)
  File "/Users/ppt/Github/stable-diffusion-webui/modules/processing.py", line 585, in sample
    samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.create_dummy_mask(x))
  File "/Users/ppt/Github/stable-diffusion-webui/modules/sd_samplers.py", line 253, in sample
    samples_ddim = self.launch_sampling(steps, lambda: self.sampler.sample(S=steps+1, conditioning=conditioning, batch_size=int(x.shape[0]), shape=x[0].shape, verbose=False, unconditional_guidance_scale=p.cfg_scale, unconditional_conditioning=unconditional_conditioning, x_T=x, eta=self.eta)[0])
  File "/Users/ppt/Github/stable-diffusion-webui/modules/sd_samplers.py", line 139, in launch_sampling
    return func()
  File "/Users/ppt/Github/stable-diffusion-webui/modules/sd_samplers.py", line 253, in <lambda>
    samples_ddim = self.launch_sampling(steps, lambda: self.sampler.sample(S=steps+1, conditioning=conditioning, batch_size=int(x.shape[0]), shape=x[0].shape, verbose=False, unconditional_guidance_scale=p.cfg_scale, unconditional_conditioning=unconditional_conditioning, x_T=x, eta=self.eta)[0])
  File "/Users/ppt/miniconda/envs/web-ui/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/Users/ppt/Github/stable-diffusion-webui/modules/sd_hijack_inpainting.py", line 56, in sample_ddim
    self.make_schedule(ddim_num_steps=S, ddim_eta=eta, verbose=verbose)
  File "/Users/ppt/Github/stable-diffusion-webui/repositories/stable-diffusion/ldm/models/diffusion/ddim.py", line 54, in make_schedule
    self.register_buffer('ddim_sigmas', ddim_sigmas)
  File "/Users/ppt/Github/stable-diffusion-webui/repositories/stable-diffusion/ldm/models/diffusion/ddim.py", line 26, in register_buffer
    attr = attr.to(torch.device("mps"))
TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead.

3rd work log: very slow

(web-ui) ppt@pptdeMacBook-Pro stable-diffusion-webui %  python webui.py --precision=full --no-half
WARNING:root:Pytorch pre-release version 1.13.0.dev20221002 - assuming intent to test it
Warning: caught exception 'Torch not compiled with CUDA enabled', memory monitor disabled
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
Loading weights [7460a6fa] from /Users/ppt/Github/stable-diffusion-webui/models/Stable-diffusion/sd-v1-4.ckpt
Global Step: 470000
Applying cross attention optimization (InvokeAI).
Model loaded.
Loaded a total of 1 textual inversion embeddings.
Embeddings: learned_embeds
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Data shape for DDIM sampling is (1, 4, 64, 64), eta 0.0
Running DDIM Sampling with 20 timesteps
DDIM Sampler: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [15:04<00:00, 45.22s/it]
Total progress: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [14:20<00:00, 43.02s/it]

ClashSAN · 2022-10-28T20:31:41Z

ClashSAN
Oct 28, 2022
Collaborator

I have no idea about this.

But the fact that you got it working is a good thing, as this repo is not intended for mac.
It is reported that ddim and plms don't work

EDIT: They all seem to be working now except for "DPM fast" (which returns random noise), and DDIM and PLMS (both of which fail immediately with the following error: "AssertionError: Torch not compiled with CUDA enabled"). According to a user on an M1 Max macbook pro with 32 gigs of ram #3694

See if you can submit a PR to here to share your findings.

0 replies

FreeBlues · 2022-10-29T03:19:36Z

FreeBlues
Oct 29, 2022
Author

@ClashSAN Yes, after modification, now all samplers can run on macOS, I will submit a PR, and I would like to get help on how to improve the speed, thank you

1 reply

jet3004 Nov 3, 2022

Thanks for your work on this! Would love to see full DDIM support on Mac.

thesved · 2022-11-03T20:53:16Z

thesved
Nov 3, 2022

Got it working at normal speed: #4233

2 replies

FreeBlues Nov 7, 2022
Author

@jet3004 I am glad to hear that it can help you.

@thesved Thanks a lot!
I got it, on my macOS, I should use the parameter opt-split-attention-v1, it speedup about 15x vs others 2. below is my command:

python webui.py  --precision full --no-half --opt-split-attention-v1

thesved Nov 7, 2022

Glad you made it work @FreeBlues.

For the record on M1 Max apple silicon --opt-split-attention-v1 made performance slightly worse for DDIM, 1.19it/s vs 1.47 it/s without it. So it makes sense to test it.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(macOS) modified to use DDIM and PLMS, suggestions needed on how to improve speed. #3885

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 3 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

(macOS) modified to use DDIM and PLMS, suggestions needed on how to improve speed. #3885

FreeBlues Oct 28, 2022

Steps to reproduce the problem

What should have happened?

Commit where the problem happens

What platforms do you use to access UI ?

What browsers do you use to access the UI ?

Command Line Arguments

Additional information, context and logs

Replies: 3 comments · 3 replies

ClashSAN Oct 28, 2022 Collaborator

FreeBlues Oct 29, 2022 Author

jet3004 Nov 3, 2022

thesved Nov 3, 2022

FreeBlues Nov 7, 2022 Author

thesved Nov 7, 2022

FreeBlues
Oct 28, 2022

Replies: 3 comments 3 replies

ClashSAN
Oct 28, 2022
Collaborator

FreeBlues
Oct 29, 2022
Author

thesved
Nov 3, 2022

FreeBlues Nov 7, 2022
Author