How Can I preload a model during the comfyui in container launch?

[solarpush]

Hello, is it possible to use either "prestartup_script.py" or a script higher up in a container to start loading a model as soon as the container is opened so that when the workflow is received by the comfyui server the node load model is already loaded in the GPU vram?

The aim, of course, is to shorten a pod's cold-start time.

I found in comfyui that models are loaded with safetensors.torch, is that right?

(for .safetensors only I guess)

Will comfyui not reload the model if it's not aware that the model is already in the vram? if so, how?

Thanks for your answer.

[ltdrdata]

Don't use prestartup_script.py for that it will break many things.

The prestartup_script.py is a script introduced for performing lightweight preloading tasks with minimized module loading, aimed at avoiding issues caused by file locks on packages during package installation in Windows environments.

What you desire cannot be achieved with basic nodes like CheckpointLoaderSimple. This is because it doesn't rely on cache.

If you utilize the Shared Checkpoint Loader from the Inspire Pack, once loaded, it can be reused in a cached state. However, there is still a constraint that necessitates the use of the Shared Checkpoint Loader in the workflow.

If you intend to use it for very personal purposes, you may override 'CheckpointLoaderSimple' in NODE_CLASS_MAPPINGS to a version that utilizes cache. Also, preloading vram can be done in the __init__.py file, which should suffice.

[solarpush]

I'll analyze the "shared" node to understand how it retrieves the model from the cache and try to adapt it to an efficientloader.
Thanks for the information.

[enjoylib-zfm]

Have you solved it

[solarpush]

No sorry , in my case i have split differents models in differents worker. But i thinks the comfyui manager have activate the unload button in menu. I am not sure.

[menahem121]

up

[ltdrdata]

While this isn't an answer to the method mentioned, if you want to reduce the model loading time, here's a tip for limited environments:
If you're using Linux and have sufficient free RAM, you can use vmtouch -t <model path> to pre-cache the model file in memory, which can significantly reduce the model loading time.

[maxIrvine]

@ltdrdata @solarpush I tried loading my models with vmtouch but saw no improvement in model load time. Curious if either of you found a solution?

[solarpush]

I'm sorry @maxIrvine I couldn't find any ready-made solutions, but @ltdrdata explained at the beginning that one approach was to take an existing neud whose code is used to share the cache and to recuperate the code to implement the loader logic directly in the container instance (whose checkpoint would be recuperated by the same node later) or to rewrite the checkpointLoaderSimple to make it do basically the same thing.

If I've understood correctly

Once again, we're talking about comfyui's execution time in an instance with a cold startup. The time we're trying to save is counted in seconds at the first demmargag, depending on cloud providers, machines, instance types, cuda versions, graphics cards, and lots of other things.

I stopped optimizing as soon as we had a rendering consistent with our internal needs, and I haven't put my hand to the grindstone for a few months now, and the changes have been very rapid.

in 0.2.2 there may be solutions

in my opinion, the best strategies are still to better control the number of instances opened using terraform or a gettaway which, depending on the requests it receives, can anticipate the opening of an instance with a special lost startup call without image feedback, which only serves to preheat the instance and load the model.

Nice day !

[maxIrvine]

@solarpush Thank you for the detailed response! I am going to keep digging and see if I can find a solution. I will let you know if I do :)