[Tester Needed] Improve SD performance by disabling Hardware GPU scheduling

[nProtect]

Hello,

I have found this few days ago and seem no one / wiki write about this.

By disabling Hardware-accelerated GPU scheduling on Windows Settings, This improve about 10-50%
of SD performance, Affect both training and image generation. (with xformer enabled on my settings)

I have tested on NVIDIA RTX 3090, Image generating went from 11it/s to 17-18it/s
Hypernetwork training 512x512 went from 3.1it/s to 3.9-4.1it/s

On another topic confirm this also improve performance on 3080 Ti
#2977 (reply in thread)

*PS: Disable this option require to restart PC, this may drop gaming performance abit but I not feel when playing games.

also, Tech explain needed why Hardware-accelerated GPU scheduling settings
affect the SD performance for more research. Thank you.

[People-11]

Found a strange problem, but first of all, switching the hardware acceleration scheduling does not have any acceleration effect on my computer. (3.39it/s to 3.43it/s）

My problem is: I run webui with Xformers, and after disabling hardware acceleration scheduling, I can not generate the same image using the same parameters, pnginfo send to txt2img too, even with re-enabling it, still generating the same image with disabled, not the same with before again.

Edit: My problem is solved, not related to both hardware acceleration scheduling or Xformers, it's my own problem

2nd edit: After my own problem solved, I tested once again: txt2img, 704x1152, batch 4, RTX 3080Ti Mobile, Xformers
Hardware-Acceleration Scheduing ON

Hardware-Acceleration Scheduing OFF

As I said before, this didn't have any acceleration effect on my computer.

[sgsdxzy]

Found a strange problem, but first of all, switching the hardware acceleration scheduling does not have any acceleration effect on my computer. (3.39it/s to 3.43it/s） My problem is: I run webui with Xformers, and after disabling hardware acceleration scheduling, I can not generate the same image using the same parameters, pnginfo send to txt2img too, even with re-enabling it, still generating the same image with disabled, not the same with before again.

I think that's related to xformers. You can try with both xformers disabled.

[C43H66N12O12S2]

I had tested this previously. IIRC moved from 22it/s to 28it/s on a 4090

[djkacevedo]

If you don't mind me asking, what else do you have in your PC? Now that I turned off GPU scheduling I'm getting about 19it/s; a far cry from your 28.

Other than the 4090 though, my computer's starting to get dated, I've got a z370 with pretty much everything as good as the board can take.

[C43H66N12O12S2]

Try channelslast. Also see #2449. Some people seem to have lower performance and I couldn't figure out why.

11700K
DDR4-3600

[djkacevedo]

No beans. channelslast didn't change anything. I also tried some of the other performance options, and other than your xformers, nothing has a positive impact on my it/s

Your memory is basically the same as mine, but your processor is a few generations newer, maybe that's it.

Also, I'm on PCIe 3, so that might be having an impact.

Looks like I might need to upgrade my system; I have a couple spare graphics cards lying around, so getting something with thunderbolt would be good anyway.

[aliencaocao]

Try channelslast. Also see #2449. Some people seem to have lower performance and I couldn't figure out why.

11700K
DDR4-3600

Hi i would like to ask what kind of difference there could be if I use channel last? Im on a RTX3080ti

[djkacevedo]

Moved my 4090 from 14.69it/s to 18.98it/s

Thanks for the suggestion! It's particularly noticeable at larger resolutions.

[mykeehu]

I have faster generation with it on (RTX 3060 12 GB), but not much difference

[USBhost]

Its me the VM user running a win11 guest with a passthrough 3080 TI on a Dell R720 with a vary old cpu (E5-2690v2).
I am not by all means a normal user. PS I'm running --xformers

Ran a few Batch counts with 512x512 with 200 sampling
With the setting ON

With the setting OFF

ON but with a batch of 8 with 28 sampling

OFF

ON but now 1024x1024 200 sampling

OFF

From what I can see having this setting off cant hurt but can only help. However by what amount it depends. In my case the more cpu bound I was the more it helped. Batch size and resolution, the larger it was the less CPU bound I became.

[nProtect]

Same result on me
Seem OFF increase it/s too much on lower resolution but less noticable on high resolution image.

I'm thinking, on lower resolution with setting OFF, CPU can push more jobs to GPU
However, on large resolution, it's limited by GPU performance itself so it's not increase.

[aliencaocao]

Almost doubled my speed at 768*768, using NAI animefinal pruned, rtx 3080ti

[USBhost]

What CPU?

[aliencaocao]

AMD 5800X

[aliencaocao]

A bit of tech explanation:
GPU scheduling is a mechanism usually ran on CPUs that allocates tasks to GPU, specifically, to GPU's frame buffer or VRAM, so that GPU can process data from its VRAM in the sequence that is needed by the program. When switching ON hardware GPU scheduling, this allocation process is ran on the GPU. This reduce latency for games because GPUs now do not need to wait for CPU to give it tasks - it can assign itself. This reduces IO bounded latency from GPU to CPU and back. However, this means a small part of GPU's processing power is taken up by this allocator thread. For games, this small performance loss is easily overtaken by the latency benefits, however for non games where frame latency does not matter, e.g. SD, using hardware scheduling will take up some memory bandwidth, which hurts the performance of VRAM-access intensive workloads, which is exactly what neural network inferencing is.

The theoretically expected observation is that, the stronger your CPU (esp. single threaded performance), the bigger of benefit you should see with SD speed if you turn hardware scheduling OFF, because if it is OFF, your CPU has to be fast enough to keep up with your GPU in allocating it tasks.

[USBhost]

Then how is it when I'm CPU bound with that setting on I get worse it/s? But when I turn it off which in your words moves that scheduling to the CPU which should make it worse but doesn't.

[USBhost]

Maybe there's a bug in Windows that makes that toggle the opposite. On is really off and off is really on?

[nProtect]

Thank you for explaination.

@USBhost I'm reading in NVIDIA document, seem latest driver (include old driver) have bug about GPU scheduling related and still doens't fixed until now. They write as known bugs and recommend to turn OFF GPU Scheduler, but I not sure if this bug related in SD performance or not.

[aliencaocao]

Then how is it when I'm CPU bound with that setting on I get worse it/s? But when I turn it off which in your words moves that scheduling to the CPU which should make it worse but doesn't.

It is normal, you are not CPU bounded enough for the loss in GPU performance to overcome it. The additional CPU bound for this task is very very light.

[papuSpartan]

This made a noticeable improvement to my generation times, even at just a glance watching it generate.

With hardware scheduling: (never went above ~6.4)

Without: frequently peaked at around 8 or even 9 it/s while averaging around 7

This is on Windows 11 22H2 25231.1000 running on bare metal. Both tests were run using --xformers with nothing else open and minimal background services.

[eur0pa]

Can confirm. Windows 10 21H2, i7-6700, 32gb ddr4, 3060ti FE.

Batch count: 4
Batch size: 4
Standard "hello asuka" rounds with -1 seed

--opt-split-attention --xformers --deepdanbooru --no-half-vae 

HAGS on, 9.6it/s:

HAGS off, 10.6it/s:

About a 10% improvement

edit: attached the wrong screenshot

[Jonseed]

I see about a 5% speed loss turning it off, but that is probably because my CPU is pretty weak/old (Intel i7-920 2.66 GHz). My graphics card, on the other hand, is a RTX3060 12GB, so probably better for me to keep hardware GPU scheduling turned on so my GPU can do all the work.

[USBhost]

Interesting. Thought I had the oldest one here.

[Panchovix]

I got about a 30-35% improvement, I have a Ryzen 7 5800X and a RTX 4090. Many thanks!

[Metal079]

Brought me from ~17it/s to ~22it/s Thanks!

[Fai-Nim]

Hi! I am new to this stuff i have a 1060 3Gb variant and was wondering if stable diffusion can run on it? I saw that at minimum you can run it on a 4gb gpu like a 1050ti. So i was wondering if i can run it too after some optimizations?

[merodeador]

Definitely, i've been running on 2GB vram (GTX 750Ti). Don't expect it to be fast, but it works. After a lot of optimizations i got it up to ~0.3 it/s with SD1.5 models at 512x512 (creating an image with 30 steps takes around a minute and a half)

[janwilmans]

I'm on a 1660 Super so it's slow, but it did give me a 19 second improvement over 3 minutes, so ~ +10%, pretty nice.

However, using

set COMMANDLINE_ARGS=--no-half --no-half-vae --xformers --autolaunch

(specifically --no-half ) gave my a much better improvement (however, at the cost of being able to render bigger then 512x512 images)
It runs out of memory when I try 512x768 for example.

Still for tweaking/experimenting it's great!

[ghost]

I used the systeminfo extension to benchmark, seemed like there was a slight decrease with this setting turned off. I am running on an RTX 3080 10GB and i7-10700K, xformers on, no --opt-channelslast (I also found that channelslast seemed to decrease it/s some).

[RaphielHS]

It has a improvement for me, from 10s/it (Yes, 10 seconds per iteration) to 4 to 9s/it, im running with NVidia MX330 2GB and making 512x1024

[littleboot]

Windows 11, i9-13900K + RTX4090. Disabling this option yielded ~ 4it/s (from ~31 to 35 it/s)
(SD.1.5pruned 100sampling steps batch 20)

[SansQuartier]

I'm on Windows 11 with an RTX 2070 Max-Q.

I did 2 gens (x768 on a 2.1 custom model) one with Hardware GPU Scheduling and one without.

No noticeable performance increase, both gens took 19s with about 2 it/s

[CaptainDoubleYou]

I have AMD 3900X CPU, 64 GB RAM, ASUS RTX 3070 running Windows 11 Version 10.0.22631 Build 22631

GPU Scheduler Disabled
5.2 / 6.23 / 7.67
4.92 / 6.31 / 7.66

**GPU Scheduler Enabled **
4.23 / 6.81 / 8.74
4.31 / 7.81 / 9.8

So definitely disabling it lead to drop in performance

[firecat6666]

Hello, is this still a thing? I was googling for stuff I could tweak to speed up generation and came across this but this post is quite old.

I tested this twice with 512x512 and 1080x1080 images and saw a... 1%? increase in speed in both cases. It was so small that it could be anything, I had to end a session that was several days old when I restarted the computer, so it could be just that.

My hardware is an 12400f, rtx3060 12GB and I'm on win11. Also, I installed A1111 just a couple days ago, using the automated-pls-do-everything-for-me installer for windows idiots who don't know anything about computers (even after using them for damn nearly 3 decades, what).

[OtakuOW]

Hi, I have the same problem. I have a 12Gb RTX 4070 OC but I can't create because it tells me I have no memory. Not being an expert in programming, could you explain to me very easily how to remedy this problem? Photos are also welcome

[Genzo9319]

I did a little testing

SD generation details:
SD arguments: --opt-sdp-attention --lowvram --xformers
SD Settings: No previews
CKPT-epicrealism
Txt2Img; prompt: cat
Sampler: DPM++ 2M Karras
Size 512x512
Steps: 20
Batch count: 2
CFG: 7

Results:
Without Hardware Acceleration: 1 image 48s, 2.42s/it, total time: 1:37s
With Hardware Acceleration: 1 image 45s, 2.30s/it, total time: 1:32s

Conclusion:
When Hardware Accelerated GPU Scheduling was turned on, image generation was faster.

Laptop:
OS - Windows 10 Pro N (19045) 22H2 Media Feature Pack for Windows 10/11 N (February 2023)
GPU - Nvidia GTX 1050 ti - 4GB VRAM, Performance settings
CPU - Intel Core i7 8750H @ 2.20GHz
SSD - XPG GAMMIX S70 BLADE (SSD) 1 TB
RAM - Samsung DDR4-2666 (1333 MHz) 16 Gb
Browser: Google chrome
Laptop was restarted before each generation batch. Upon restart, that was the first thing I did.