CLIP Interrogator extension!

[pharmapsychotic]

Hello friends!

I've created an extension so the full CLIP Interrogator can be used in the Web UI now. Give it an image and it will create a prompt to give similar results with Stable Diffusion v1 and v2. It can give you a nice starting point and ideas for your prompts.

https://github.com/pharmapsychotic/clip-interrogator-ext

It honors the low/med vram option of the web UI and does it own detection to switch into a low VRAM mode so should hopefully work across a wide range of GPUs. I've tested with nvidia 1070 8GB. Precomputed text embeddings are downloaded in safetensors format from huggingface and stored in models/clip-interrogator to speed up processing.

The first time you run it the progress doesn't display so check the console to see how it's doing.

It's my first attempt at an extension so let me know if there's anything I've done wrong or could improve!

[ClashSAN]

nice extension! do you have any info on which ones take the least vram?

Since the model has loaded into vram at startup, I unloaded it with https://github.com/hako-mikan/sd-webui-supermerger, an extension that has an "unload model" option. "fast" and "classic" mode seem to use the least.

[pharmapsychotic]

thanks! RN50/openai probably uses the least. ViTL/openai though should give best results with SD 1.* models and ViTH/laion best for SD 2.* models.

ooh thanks for the pointer, I see how they're unloading the generation model there - i'll give it a try!

[pharmapsychotic]

Updated now with batch mode so you can point at folder of images and it will create prompt for each and store in either txt file for each image, big txt file with all the prompts, or csv file.

I haven't sorted out how to display progress bar nicely. I've been looking at how it's done for the textual inversion training and tried to replicate but run into Gradio errors so people still have to watch the console to see how it is progressing. Does anyone know more info on how to display progress bars in your extension nicely?

[gsgoldma]

is there anyway to get the clip models to save into the stable diffusion folder instead of in my appdata or some other location?

[MeinDeutschkurs]

Updated now with batch mode so you can point at folder of images and it will create prompt for each and store in either txt file for each image, big txt file with all the prompts, or csv file.

I haven't sorted out how to display progress bar nicely. I've been looking at how it's done for the textual inversion training and tried to replicate but run into Gradio errors so people still have to watch the console to see how it is progressing. Does anyone know more info on how to display progress bars in your extension nicely?

It's great. I sent you some issues I ran into at your issue tracker.

BTW, with API-usage. Is there a way to submit the mode together with the JSON? I'd like to use 'fast' instead of 'best'.

[C00reNUT]

Hello,
may I ask you what would we be the option that would produce the same results as the ones produces by the 'default' Interrogate CLIP button in webui?

[Painter59]

This is VERY useful! There are error messages when offline that indicate it is possible to set up to work locally. As usual, they are very cryptic and go off talking about installing transformers, etc. So far I have not beemn able to set everything working. Is there a simpl;e guide to local automatic1111 use that even I can follow?? Thanks.

[rpillala]

I love this extension. Lately it seems to be running really slow. I see that it has switched to low VRAM because I have 10 GB of VRAM. How much VRAM would keep it from doing that?

[schoenid]

How to install additional models to this extension?
I've tried to copy a model to the \Stable Diffusion WebUI\models\clip-interrogator directory, but nothing happens.
I've also tried to copy the belonging json file to the same location or even to the \Stable Diffusion WebUI\venv\Lib\site-packages\open_clip\model_configs directory, but without success.
Is there any description available, how to do it?
I found also that there is a ViT-H-16.json located in the venv directory, but no model is available in the interrogator extension interface. Ho to get it?