I only got 31 it/s on Windows with my 4090 and I am happy about that.

[aifartist]

Why? Because I finally got tired of all the noise about windows users not getting the 39.5 it/s or faster that I get on Linux.
Why am I happy it wasn't as fast? Because it only took me a few minutes to repro the problem so I could see the issue myself.
I had a dual boot system since I bought my high end box in Dec. but have been on Ubuntu ever since the first week having it.

Today I booted Windows.
I had to upgrade the python there to 3.10 from the 3.9 already installed.
I did a git clone of A1111.
Did the first time start up, for it to install everything, and saw the expected slow 13 it/s.
Upgraded to torch 2.0.0+cu118, and got about 31 it/s.
Bingo! 39+ it/s on Ubuntu and 31-32 on Windows.

Using Task Manager I can see quite large amounts of system CPU usage.
You have to click "show kernel times" and it is easier if you use the per logical processor view.
My estimate based on what I saw is that, in total, 2X cores are being used instead of the 1 expected.
The main python process is using 100% of the processor it is running on but one, or sometimes two other processors just get hammered with system processing.
Turned off GPU scheduling and got to 35 it/s.
I suspect with the SD 2.1 model it'd be closer to 37 it/s.
Ran on my slower 4.3 GHz cpus, to simulate users with slower processors, trying to push a 4090, and only got 22.5.
I was a bit surprised that turning off GPU scheduling made more of a difference on slower CPU.
On a slow CPU one would think the GPU offload to a fast GPU might be better.

But I don't intend to do a deep tech dive looking at the Windows interrupt handling system(possible source of high kernel time) and thread scheduling which seems to bounce processes back and forth between CPU's more than Linux does.
Windows 10 vs Windows 11?
Windows vs Windows pro?
Windows pro vs Windows Enterprise?
Windows version?
PCI capabilities, or BIOS settings, allowing or not allowing kernel bypass without needing interrupts.
To bad I wasn't playing with this stuff just before I retired from Microsoft. I could have asked internally.
Now I'd want to know where to send the bill to figure this out. :-)

I've satisfied my curiosity. I'm back on Ubuntu now.

@SamueleLorefice

[SamueleLorefice]

Out of all the people in that thread, you tagged me... I just was curious about why it is that way...

So, long story short: windows sucks, better use linux.
Also, better buy a new cpu. Gotcha!

Thanks for your insight!

[aifartist]

Out of all the people in that thread, you tagged me... I just was curious about why it is that way...

Because you were playing with process priority and cpu affinity/binding so I thought you'd find it interesting and would be more likely to appreciate the issues of kernel and interrupt processing overhead.

And two other techies...
@ItsLogic
@seggybop

[SamueleLorefice]

I would appreciate them if this was something I'd have to deal with at work, but this is just hobby stuff and a bit over my knowledge level when it comes to performance optimizations for me. (My lowest level is game engine optimizations/GPGPU computing, never went deeper into cache/kernel). So uhhh, all I can say is... good to know, I'll boot my linux partition when I want to play around with it. I hope someone can come up with some other throughput optimizations sometime, TensorRT would be awesome for example, but to put it bluntly, my python skills are too low to contribute in any possible way here, so... I find it nice, it has been interesting, but there isn't much I can do, aside providing data.

[ghost]

So to get the linux speedboost, would it be sufficient to run A1111 with --listen on a windows ubuntu wsl machine? And then access it from windows via browser and IP? Or does it has to be linux on boot?

There are still so many users out there who I tried to convince to get a better CPU for their 4090 because the card really needs it, but nope. "SD only uses GPU there is no CPU involved!" yea...no... and memory bandwith and PCIE version also absolutely dont matter... /sarcasm off.

[SamueleLorefice]

someone did a test with the memory bandwidth and turned out to not matter. WSL would not fix the issue because you're still going trough windows kernel I guess, my WSL test didn't go any faster anyways...

[aifartist]

@piglandscape and even the 13900K isn't fast enough once the 4090 is running a torch.compile'ed model. Today I discover that I was only getting the 5.5 GHz boost on my 13900 and not the TurboBoost 3.0 5.8 GHz. Fixing that got me from 50 it/s to 51 it/s with the torch.compiled model. If not using torch.compile the 13900 is fast enough even at 5.5 GHz.

[SamueleLorefice]

A 4090 is able to run at 90 it/s on TensorRT, at this point this sounds like a torch optimization problem.

[ItsLogic]

I can hit 40 on windows... just barely but I did get the mystical 40.00
It is slower than my linux install but the difference in peaks is ~1.5it/s which I could easily throw down to differences in cpu scheduling or kernels
SD1.5

SD2.1

no moving DLLS or weird windows specific patches.
I created a second venv and installed all of the dependencies only manually installing torch 2 cu118 and used the exact same launch args as I did on linux, as I was using the same webui files I still had the torch.backends.cudnn.benchmark = True line in my webui.py

[SamueleLorefice]

What CPU are you on?

[ItsLogic]

13900k

[aifartist]

Looks like @ItsLogic is getting the same speed as I with our 13900K's.
The 51 it/s require the torch.compile() source code change found elsewhere in other discussions.

[neurogen-dev]

With my Ryzen 9 5900X I get 30-33 it/s for batch size 1 and 49 it/s for bath size 4 on Ubuntu 22.04 WSL2 (Windows 11)
1.5

2023-03-20 10:51:51.475232	32.29 / 40.43 / 47.87	updated:2023-03-13 hash:eb960aa7 url:https://github.com/vladmandic/automatic.git/tree/master	arch:x86_64 cpu:x86_64 system:Linux release:5.15.90.1-microsoft-standard-WSL2 python:3.10.10	torch:2.1.0a0+gitbdd09e6 autocast half xformers:unavailable accelerate:0.18.0.dev0 transformers:4.27.1	device:NVIDIA GeForce RTX 4090 (1) (sm_89) (8, 9) cuda:12.1 cudnn:8801 24GB	sdp none	v1-5-pruned-emaonly.safetensors [6ce0161689]	wsl		cc7c92

2.1

2023-03-20 10:22:45.803644	30.37 / 41.83 / 49.39	updated:2023-03-13 hash:eb960aa7 url:https://github.com/vladmandic/automatic.git/tree/master	arch:x86_64 cpu:x86_64 system:Linux release:5.15.90.1-microsoft-standard-WSL2 python:3.10.10	torch:2.1.0a0+gitbdd09e6 autocast half xformers:unavailable accelerate:0.18.0.dev0 transformers:4.27.1	device:NVIDIA GeForce RTX 4090 (1) (sm_89) (8, 9) cuda:12.1 cudnn:8801 24GB	sdp none	v2-1_768-ema-pruned.safetensors [dcd690123c]	wsl		cc7c92

[FullBleed]

With my Ryzen 9 5900X I get 30-33 it/s for batch size 1 and 49 it/s for bath size 4 on Ubuntu 22.04 WSL2 (Windows 11)

I was thinking of running the webui via docker desktop on Ubuntu through WSL2 and wondered if it would negatively affect performance.

But I'm not all that familiar with WSL2 and wasn't sure if I'd be able to run the photoshop (on windows) extension this way.

[aifartist]

@Em1tSan What are these numbers you are showing?
30.37 / 41.83 / 49.39
Is this with @vladmandic torch.compile() change?

[neurogen-dev]

@Em1tSan What are these numbers you are showing? 30.37 / 41.83 / 49.39 Is this with @vladmandic torch.compile() change?

This is the webui version : https://github.com/vladmandic/automatic
It's are benchmark (from sd-extension-system-info) results for batch size 1, 2, 4

[iterhating]

I don't understand these numbers. I have 6 G of VRAM and I'm pretty happy with ~1.5 it/s. Are you able to generate one image a second or something? Is this some CPU-based metric which doesn't compare equivalently? Or have I been doing something wrong and you all have been generating real time video for weeks now?

[seggybop]

There are a couple issues specifically with the 4090 that prevent it from performing up to its theoretical potential (>40 it/s) without a sufficiently fast CPU and some extra setup. So, no, you are not doing anything wrong.

[iterhating]

Ah, thanks for explaining. Sometimes the rumblings from Mount Olympus can confuse and frighten us mortals living down here on the mundane plane.

[remghoost]

I found this discussion looking to see if WSL was faster than Windows.

My 1060 6GB and I are happy to see someone else down here on this mortal plane.
There are dozens of us, I tell you! Dozens of us!

[aifartist]

I don't understand these numbers. I have 6 G of VRAM and I'm pretty happy with ~1.5 it/s. Are you able to generate one image a second or something? Is this some CPU-based metric which doesn't compare equivalently? Or have I been doing something wrong and you all have been generating real time video for weeks now?

Yes, with a 4090, a good cpu, and the right software version a image takes about .5 seconds to generate.

[rycossette]

And here I am on an m1 max getting 3 it/s max. Which is after optimizations, it was 1.5 before pytorch 2.0.... Now I'm depressed, thank you.

[aifartist]

And here I am on an m1 max getting 3 it/s max. Which is after optimizations, it was 1.5 before pytorch 2.0.... Now I'm depressed, thank you.

But at least you saved money! :-)
40 it/s is probably something like at least an extra $2500.

[zejacky]

@aifartist
How you get 31 it/s? Which model/sampler and resolution do you use.
If I create a image with basic sd 1.5 and 512x512, I have max. 7-8 it/s.
Running on a 4090 and 12900k. I have torch: 1.13.1+cu117 and python 3.10.6 installed. Is this the problem?

How to upgrade to 2.0.0+cu118.
Thank you

[zejacky]

@aifartist How you get 31 it/s? Which model/sampler and resolution do you use. If I create a image with basic sd 1.5 and 512x512, I have max. 7-8 it/s. Running on a 4090 and 12900k. I have torch: 1.13.1+cu117 and python 3.10.6 installed. Is this the problem?

How to upgrade to 2.0.0+cu118. Thank you

I found following article, and my creation speed improved. However not 31 it/s:
https://medium.com/@j.night/fix-your-rtx-4090s-poor-performance-in-stable-diffusion-with-new-pytorch-2-0-and-cuda-11-8-d5cb689be841

Example:
Browser: Opera GX and Chrome
Model: illuminatiDiffusionV1_v11
Restore Face: No
Resolution: 768x768
Sampling method: DPM++ SDE Karras
Sampling steps: 30

torch: 1.13.1+cu117
1 pic = 12 sec / 2.39 it/s
Batch size 8 = 49 sec / 1.64 s/its

Now
python: 3.10.10  •  torch: 2.0.0+cu118
HW accelerated GPU scaling = off
COMMANDLINE_ARGS= --opt-sdp-attention
1 pic = 5 sec / 5.63 it/s
Batch size 8 = 30 sec / 5.5 it/s

I'm already happy :-)

[aifartist]

@aifartist How you get 31 it/s? Which model/sampler and resolution do you use. If I create a image with basic sd 1.5 and 512x512, I have max. 7-8 it/s. Running on a 4090 and 12900k. I have torch: 1.13.1+cu117 and python 3.10.6 installed. Is this the problem?

How to upgrade to 2.0.0+cu118. Thank you

The 31 it/s is my "bad" number. I got it by booting Windows instead of Ubuntu just to see if I saw the same slow ~30 it/s others see on Windows. euler_a, model=sd1.5, 512x512, prompt no longer than 10 or so words although it probably isn't a problem till you hit 75. As a pure guess your 12900K is probably about 10% slower than my 13900K. Windows is the biggest problem which might be solvable with a careful pinning/affinity of the python process to specific cpu's but I don't know because I have no reason to tweak my Windows performance. I'm a hard core Linux perf guy.

[zejacky]

@aifartist How you get 31 it/s? Which model/sampler and resolution do you use. If I create a image with basic sd 1.5 and 512x512, I have max. 7-8 it/s. Running on a 4090 and 12900k. I have torch: 1.13.1+cu117 and python 3.10.6 installed. Is this the problem?

How to upgrade to 2.0.0+cu118. Thank you

The 31 it/s is my "bad" number. I got it by booting Windows instead of Ubuntu just to see if I saw the same slow ~30 it/s others see on Windows. euler_a, model=sd1.5, 512x512, prompt no longer than 10 or so words although it probably isn't a problem till you hit 75. As a pure guess your 12900K is probably about 10% slower than my 13900K. Windows is the biggest problem which might be solvable with a careful pinning/affinity of the python process to specific cpu's but I don't know because I have no reason to tweak my Windows performance. I'm a hard core Linux perf guy.

Thank you for your Feedback.
As you can see in my previous post, I could already improve my speed. However, I will consider trying a1111 on a linux distro.

[bbecausereasonss]

I only get 21, 31 would be amazing. Win11 4090

[BasedAnon]

What about Messaged Signalling Interrupts?
I may have turned on, trying to squeeze gaming performance out of my 4090.
This all just proves that I need to seperate my work and gaming OSes: just boot into whatever I wanna do.
I could go the Promox route, but I was having issues with 4090 drivers and passthrough, when I tried it like 2 months ago.

[x-ili-x]