Replies: 15 comments 23 replies
-
Beta Was this translation helpful? Give feedback.
-
|
On top of my head one thing you can try is to switch the base image from I remember deciding against this because these containers have python 3.7 or 3.8 per default, but most UIs need 3.10. Not sure what the state is now. |
Beta Was this translation helpful? Give feedback.
-
|
I also found this when installing the controlnet extension, I needed to add Regarding the PyTorch images, yes they are still using Python 3.8, we can track this here: |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, I agree that we need some shared dependencies already installed in the container. The question is, where do we stop? how much is enough? |
Beta Was this translation helpful? Give feedback.
-
|
Maybe we can have an "EXTRA_APT_PACKAGES" ARG in the dockerfile that we can specify in our own docker-compose files, with a list of extra packages to install as part of the dockerfile build. Cuda is a bit more complicated, as it requires downloading of a deb, installing it, then an |
Beta Was this translation helpful? Give feedback.
-
|
I'm not a fan of more configuration, since the main benefit of using docker is that things work out of the box. But I understand there is a concern of the image getting bloated. Here is the official list of extensions sorted by the number of stars, top 10: If you want to be qualitative, you can come up with some metric base on this. But I would say it's not excessive to support the top extension by default. |
Beta Was this translation helpful? Give feedback.
-
|
@chakflying sorry for late answer. I think supporting top extensions should be fine, do you know which It would be great if you could create a PR. |
Beta Was this translation helpful? Give feedback.
-
|
I agree that getting extensions dependencies is the biggest pain point right now. I am a relatively amateur programmer so please take my proposal with a grain of salt. Up until now I have been manually adding lines to the docker file similar to @kelvie method above. However, it's a tedious process. Is it possible to wrap the whole thing in 1 virtual environment? If this were the case, I believe that the launch import that is present in the install.py files of the extensions would be able to call the launch.py file in the automatic1111 repo. If this is possible, I think it might be a better solution than maintaining a separate hypothetical requirements.txt file for common packages. |
Beta Was this translation helpful? Give feedback.
-
|
@cbolich: |
Beta Was this translation helpful? Give feedback.
-
|
I still have to install many dependencies separately. For what it's worth here are my additions to the dockerfile: RUN apt-get update && apt-get install -y ENV NVIDIA_VISIBLE_DEVICES all RUN pip install vispy update: |
Beta Was this translation helpful? Give feedback.
-
|
Instead of changing Dockerfile you should do it in |
Beta Was this translation helpful? Give feedback.
-
You can call any scripts you want in the
Yeah, and unfortunately, I don't see that getting better anytime soon. Every extension has its own set of dependencies and they overlap and overwrite each other. I don't think this is a problem that we can solve in this repo, we may make it a bit easier by installing a "separate hypothetical requirements.txt file for common packages", but this is an ecosystem problem. |
Beta Was this translation helpful? Give feedback.
-
|
I've got a feeling that this subject pops up too often. Should we just add info about this in FAQ? |
Beta Was this translation helpful? Give feedback.
-
|
Sure! do you have a suggestion in mind? or should I just improvise 😅 |
Beta Was this translation helpful? Give feedback.
-
|
@AbdBarho It's already there. Most of it. https://github.com/AbdBarho/stable-diffusion-webui-docker/wiki/Usage#extensions I would aim for something like this (but You should rephrase it probably). |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, updated! |
Beta Was this translation helpful? Give feedback.
-
|
Ran into this tonight. I'm using this repo to setup ComfyUI and was having trouble installing the WAS Node Suite since it requires dlib, and requires a couple dependencies from apt-get to properly build/install the dlib python library/dependency.
I ended up just making a shell script that I run to update the ComfyUI Dockerfile (
I'm happy enough with that solution for now, but it would be nice to have dlib pre-installed for the ComfyUI setup, since it's one of the main recommended plugins. (Full script for reference)
|
Beta Was this translation helpful? Give feedback.
-
|
Please don't do that! My eyes are bleeding when I see it. |
Beta Was this translation helpful? Give feedback.
-
You mean build the first one, then use the FROM command to extend the new one? I was looking at doing that at first, but didn't want to depend on building the first image and then building another one. I find myself rebuilding the Comfy image enough that just making edits to the original Dockerfile seemed fine. The only issue is doubting that the apt-get line is going to get updated/touched. This way is kept outside of the folder and is safe between git pulls and I can blow away the folder without worrying about losing these scripts. The image itself gets rebuilt as part of the Is the |
Beta Was this translation helpful? Give feedback.
-
|
@dereekb: Yes, that's what I mean. There are several drawbacks when you edit Dockerfile. When using docker you want to benefit from all of it's pros and not to limit your flexibility with images. Right now your method is somewhat easy to do when you mastered all "change that" stuff. But later you might encounter some problem and using modified image almost all of the time will make debugging harder. You can emulate |
Beta Was this translation helpful? Give feedback.
-
|
as an aside I tried the advice from here https://github.com/AbdBarho/stable-diffusion-webui-docker/wiki/Usage#extensions And got the following error: /data/config/auto/startup.sh: line 2: shopt: nullglob |
Beta Was this translation helpful? Give feedback.
-
|
my bad. Windows EOL rather than Linux. Soz |
Beta Was this translation helpful? Give feedback.
-
|
this is okay for me atm |
Beta Was this translation helpful? Give feedback.
-
|
I have installed all the extensions from With the modifications from They all worked except for the following impacts: adieyal/sd-dynamic-prompt thygate/stable-diffusion-webui-depthmap-script |
Beta Was this translation helpful? Give feedback.
-
depthmap:apt-get install libglfw3-dev libgles2-mesa-dev deforum: controlnet:mediapipe dreambooth:accelerate~=0.18.0
dynamicprompts:send2trash==1.8.0 |
Beta Was this translation helpful? Give feedback.
-
|
@simonmcnair I'm right there with you and my list looks pretty similar. suggest you change: transforms3d==0.4.1 Have you been able to get the video generation to work in the depth extension? That's the final thing I'm having trouble getting to work. I've tried all the solutions I could find in those threads, but no luck so far. The rest of the extension runs fine, its just that last video piece. webui-docker-auto-1 | Traceback (most recent call last): |
Beta Was this translation helpful? Give feedback.
-
|
Sorry for the delay. I don't use it, I just installed it to try it. Finally got it it working with the following: aria2c https://developer.download.nvidia.com/compute/cuda/repos/debian11/x86_64/cuda-keyring_1.0-1_all.deb apt-get install libglfw3-dev libgles2-mesa-dev pip install ffmpeg diffusers discord_webhook trimesh rich vispy numexpr moviepy==1.0.2 onnx onnxruntime networkx==2.5 imageio imageio_ffmpeg transforms3d==0.4.1 importlib-metadata importlib-metadata matplotlib numba==0.57.0 rembg pip install -q --force-reinstall opencv-python-headless |
Beta Was this translation helpful? Give feedback.
-
|
I have a PR with some new dependencies, if you could also recommend some missing ones, at least for the controlnet extension. that would be great. |
Beta Was this translation helpful? Give feedback.
-
|
Might be worth having 2 options a1111 minimal and a1111 full. If people don't use extensions they could go minimal or deal with it themselves ? |
Beta Was this translation helpful? Give feedback.
-
|
maybe? I guess most people who use a1111 are there for controlnet or dreambooth. |
Beta Was this translation helpful? Give feedback.
-
|
My latest looks like this: |
Beta Was this translation helpful? Give feedback.
-
|
The underlying issue is that Auto1111 installs the required packages in the container upon installation of the extension, but the extension is persisted outside of the container. If the container is ever rebuilt, the extension is still present, but its dependencies have been wiped away. You either need a simple store in the Docker container to track what extensions have already had dependencies installed in that container or you just pip install all the requirements for all extant extensions on every startup.
The answer is to not hardcode requirements at all and just have something on startup check through all the existing extensions and make sure their dependencies are installed in the container. I did something like that here: 04620dc |
Beta Was this translation helpful? Give feedback.
-
|
this is all mentioned in the wiki |
Beta Was this translation helpful? Give feedback.
-
|
Thanks! That's great! I was unaware that you already had a place to put this. It makes a lot more sense now. What was your thought process behind not adding this default startup.sh for everyone? Just too much individual variation? Some extensions have an install.py that needs to be run. I suggest the startup script should look something more like: # install all packages for the extensions
shopt -s nullglob
list=(./extensions/*/requirements.txt)
for req in "${list[@]}"; do
pip install -q -r "$req"
done
list_of_install_scripts=(./extensions/*/install.py)
for install_script in "${list_of_install_scripts[@]}"; do
export PYTHONPATH="/stable-diffusion-webui"
python "$install_script"
done
#opencv-python-headless to not rely on opengl and drivers.
pip install -q --force-reinstall opencv-python-headless |
Beta Was this translation helpful? Give feedback.
-
|
As far as I can see half the requirements of one, break the requirements of another. The other point is that this is only required on container rebuild, not restart, which can add a lot of start up delays. As well as apt prospectively updating components out of band. |
Beta Was this translation helpful? Give feedback.
-
Bingo
The current solution with the |
Beta Was this translation helpful? Give feedback.
-
|
This is my latest since the image changes were implemented in https://github.com/AbdBarho/stable-diffusion-webui-docker/releases/tag/7.0.0: |
Beta Was this translation helpful? Give feedback.
-
|
I can confirm some of depth map will not work without cuda-libraries-dev-11-8 installed. that results in the EGL error |
Beta Was this translation helpful? Give feedback.
-
I just spent some time debugging why the dreambooth (in auto) doesn't work right anymore, and finally narrowed it down to the fact that one of the modules, bitsandbytes, requires a bunch of cuda libs: bitsandbytes-foundation/bitsandbytes#85
Being that nvidia and python are both atrocious for packaging, I totally get it, but perhaps this might not be the only other python library that expects to be able to find cuda libs. Should this docker image consider adding (back?) cuda-libraries-dev?
Otherwise I'm just going to be maintaining my own patches to the dockerfile on top of this one to let some extensions work, currently, this is what I add to my dockerfile to get my set of extensions working:
Is there a cleaner and more maintanable way to add packages I need to the dockerfile? It was mentioned that the
startup.shis the best way for doing the pip installs, but it takes a lot longer to do the apt-installs every time, and it seems harder to cache (the pip cache is persisted across docker compose runs)Beta Was this translation helpful? Give feedback.
All reactions