Running on Kepler

[zceemja]

So everywhere I've seen in discussion it is impossible to run stable diffusion in Kepler GPUs except K80 because it supports CUDA compute capability 3.7. Main limitation to run it on CUDA 3.5 is that started pytorch packages come compiled without support for it, you can check it with:

import torch
torch.cuda.get_arch_list()

But what if we compile torch with support for CUDA 3.5 support? Or even better, use precompiled torch from https://github.com/nelson-liu/pytorch-manylinux-binaries.

Tested on linux with:

• nvidia 470.199.02 driver,
• torch 1.13.1+cu117,
• stable-diffusion-webui commit c9c8485bc1e8720aba70f029d25cba1c4abf2b5c,
• Tested on 2 Titan Blacks and 1 Titan Z (all 6GB VRAM)
• no COMMANDLINE_ARGS,

I can confirm that stable diffusion works fine, no weird artefacts. 512x512 image with 20 sampling steps take about a minute to generate. One Titan Black was crashing once in a while but seems to be problem with that particular card.

If anyone interested I can run more tests.

[Pup-In-Cup]

I didn't even imagine that anyone would be interested in making AI run on Kepler chips! I have a Kepler card with 4GB memory, which should technically be enough to fit SD in with a --lowvram argument. But I don't have the knowledge to make it work!
If you're interested, can you make it run for WIndows? I don't have Linux, and I am too uneducated to try anything beyond plain-guessing.

[zceemja]

It seems like it would be possible to run on windows, but there is no precompiled pytorch with Kepler support. On linux its as simple as changing webui-user.sh and adding

TORCH_COMMAND="pip install torch==1.13.1+cu117 -f https://nelsonliu.me/files/pytorch/whl/torch_stable.html"

There is no repository as far as I can find with windows binaries so you'd have to compile pytorch yourself which is not that user friendly.

[Pup-In-Cup]

That's the thing - while I was "researching" the topic, I discovered that it might be possible to force the Kepler chip to run a neural network. I ended up with the foggy idea that Kepler can be made to compute an ANN, but because it's not compatible with the libraries, the neural network does inference incorrectly. Therefore, the only two answers that I found on the topic are either: "It can't be done," or "It can be done, but it'll output gibberish."
Not to mention that I didn't even know anything Python can be compiled. Advanced programing meets skill issue.

[zceemja]

Nah you definitely can compute ANN on Kepler, look at some old tensorflow v1 releases as they support cards as old as cuda 3.0. There is nothing special about regular ANN inference, just multiply and add 32bit floats. You should be able to run ANN even on older cards using compute shaders.

Its just nvidia dropped support for Kepler drivers in 2021 and no one bothers to include them in library support. I did a bit of browsing too and got same answers that it can't be done. That's why I wrote this discussion, especially as it quite easy to run (at least in linux), and it works well enough. I would image some newer NN architectures like transformers would also run, but be horribly slow to run due to lack of optimisation on cuda, but who knows.

[arya-coding]

Hello,
just for the record, if anyone is looking for something similar, I've been trying for fun to run Stable Diffusion Web UI with the following config:
Nvidia 425.31
Torch 1.8.1+cu101
GT 740M with 4GB VRAM
stable-diffusion-webui Windows release
no COMMANDLINE_ARGS (I mean vlowram etc., but it doesn't work anyway).
I had to downgrade Python to 3.9 (in stablediffusion), to use Torch 1.8.1.

However, PyTorch only supports CUDA >3.5 and not CUDA 3.0, which is the baseline for the GT7XXX.Some people mention you could build PyTorch yourself for your CUDA 3.0 instructions, but I just gave up at that point.

I found out that the only way to run models on those legacy hardware is through CLBlast NoAVX2. However, it's for textual models, not image ones.

[alighoosh]

I really need this in Windows...

[biocrypt]

@zceemja what steps are required to build Stable Diffusion for Titan Black GPU?

[zceemja]

check driver version, you should be able to update it up to 470.256.02

[biocrypt]

@zceemja I updated drivers to version 470.256. Result is the same: failure. Can you make your compilation once again? Please provide more details about your development environment.

[biocrypt]

@zceemja Have you seen my last questions? 🙂

[zceemja]

Can you provide more information than just a failure? Send the output or log.

[biocrypt]

@zceemja this is the file with logs:
logsSD2.txt