A1111 SDXL CUDA - something is eating GPU memory since clean install

[alanm8]

hi all hope this is the right place...

After happily using 1.5 for a long time and SDXL for a few months on my 12G 3060, I decided to do a clean install (around 8/8/24) as some of the versions were very old.

Now when using simple txt2img, (nothing special really) its running out of memory after a while. This never used to happen and every generation seems to be eating more memory and every stop and change of prompt eats a bit more. It still runs nice and fast.

I've set batch to 2 (4 doesn't last to long!) and there is a definite pattern over an hour or so where python usage in nvidia-smi goes from 9030MiB to around 11700Mib then it gives the usual CUDA out of Memory error.

Seems there is a pattern - anyone else noticed this and any hints how I can manually free this while its running etc.etc?

I did take the torch up from something really old to latest, wondering if its that?

Yes I know medvram makes it last longer but that's taking away the preview which is really useful, presumably it would still eventually reserve all the memory...

GPU usage after each batch (batch size 2)
max during render | sudden peak at 100%
9030 | 111??
9054 | 11188?
9078 | 11168
9054 | 11188
9078 | 11142
9102 | 11168
9078 | 111??
9102 | 11166
9126 | 11190
9138 | 11212
9102 | 11166
9126 | 11190
9150 | 11240
9126 | 11262
9150 | 11286
9174 | 11238
9150 | 11214
anyways this continues up till it dies.
version: v1.10.1  •  python: 3.10.12  •  torch: 2.1.2+cu121  •  xformers: 0.0.23.post1  •  gradio: 3.41.2
Ubuntu 22.04
starting with
export COMMANDLINE_ARGS="--xformers --data-dir data "

cheers

Al

[Soulreaver90]

I’ve noticed this as well. I can leave the prompt as is and it will generate no issues. Changing the prompt or Lora’s seems to negatively affect memory usage, to the point where eventually generations will take forever to start and memory usage is maxed out. I have to constantly close and relaunch A1111.

[AlexKaleda]

This may be related: #16186
Change torch+cuda version combo.

Also #16394

[Soulreaver90]

I am running a AMD card with rocm and a beta torch version so not sure if relevant. I will probably need to either revert or clone the previous version to do some A+B testing.
I saw another post about a similar issue and reference to controlnet, which I do have installed but not active, so might test removing that as well.

[alanm8]

Thanks both. Yes I did try upgrading torch and xformers to a different newer version, it then broke xformers and ended up doing another clean install! Same. Have noticed there is another thread with a different version pair so will try that in a couple of days when I have some spare time and report back. I really think there is a "leak" issue generating in batch plus where it doesnt free it all on each gen another one when changing prompts which is more severe 🤔