How to use demo in AMD GPU and driver ?

[waldolin]

How to use demo in AMD GPU and driver ?

i have download and run it failed.

PS D:\github\github\sd3.5> python sd3_infer.py --prompt "cute wallpaper art of a cat"
C:\Users\lin\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\amp\autocast_mode.py:265: UserWarning: User provided device_type of 'cuda', but CUDA is not available. Disabling
  warnings.warn(
Loading tokenizers...
You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
Loading OpenAI CLIP L...
Loading OpenCLIP bigG...
Loading Google T5-v1-XXL...
Skipping key 'shared.weight' in safetensors file as 'shared' does not exist in python model
Loading SD3 model sd3.5_large.safetensors...
Loading VAE model...
Models loaded.
Saving images to outputs\sd3.5_large\cute wallpaper art of a cat_2024-10-23T17-09-22
  0%|                                                                                            | 0/1 [00:01<?, ?it/s]
Traceback (most recent call last):
  File "D:\github\github\sd3.5\sd3_infer.py", line 436, in <module>
    fire.Fire(main)
  File "C:\Users\lin\AppData\Local\Programs\Python\Python310\lib\site-packages\fire\core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "C:\Users\lin\AppData\Local\Programs\Python\Python310\lib\site-packages\fire\core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "C:\Users\lin\AppData\Local\Programs\Python\Python310\lib\site-packages\fire\core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "C:\Users\lin\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\utils\_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "D:\github\github\sd3.5\sd3_infer.py", line 421, in main
    inferencer.gen_image(
  File "D:\github\github\sd3.5\sd3_infer.py", line 342, in gen_image
    sampled_latent = self.do_sampling(
  File "D:\github\github\sd3.5\sd3_infer.py", line 262, in do_sampling
    latent = latent.half().cuda()
  File "C:\Users\lin\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\cuda\__init__.py", line 314, in _lazy_init
    torch._C._cuda_init()
RuntimeError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx

I will appreciate it if you help me to use it!

[thinwoodsman]

You need to install ROCM; that will provide a CUDA compatibility layer for your AMD GPU. I don't use Windows so I cannot offer any specific instructions on this step. Make sure rocminfo is in the system path.

I had to install an rocm-compatible fork of bits and bytes. Depending on your AMD card, you may have to add a specific target e.g. "ROCM_TARGET=gfx1102".

Once those are going, install the latest torch and from there everything should just work.

To install the latest transformers/diffusers:

pip install git+https://github.com/huggingface/transformers
pip install git+https://github.com/huggingface/diffusers

Example of test code using diffusers:

import os
os.putenv("DRI_PRIME", "1")
os.putenv("HSA_OVERRIDE_GFX_VERSION", "11.0.0")
os.putenv("TF_CPP_MIN_LOG_LEVEL")
from diffusers import DiffusionPipeline

memspec={0: "6GiB", "cpu": "24GiB"}
pipe = DiffusionPipeline.from_pretrained( "stabilityai/stable-diffusion-3.5-large", max_memory=memspec,add_prefix_space=False,torch_dtype=torch.bfloat16)

The demo uses torch, not transformers, but it is a dependency of transformers/diffusers so should be working fine if the above works. I haven't actually run the demo script or checked out the repo as I tend to write my own code for these things.