update_extensions.py - EXTREMLY FAST UPDATE EVERYTHING - by ChatGPT

[Gustavsenay]

Here is the Repo

https://github.com/Gustavsenay/Super-Updater-by-ChatGPT

As an artist, I was using the Automatic1111 WebUi and i noticed one issue with the project is that the update process for its many extensions can be very slow and tedious. This is where the update_extensions.py script comes in!!

The script, written by ChatGPT, is a Python script that automates the update process for all of the extensions included in the Automatic1111 WebUI project. The script uses the git version control system to check the status of each extension, and then pulls the latest changes from the remote repository if necessary, but is EXTREMLY faster (I think is because the subprocess, but I'm not an expert) and it is able to check a 50+ etensions folder in less than 3 seconds.

EDIT - UPDATE 2:

Using also multiprocessing the time to check everything is even lower, plus It now is a little nicer to watch, the following is the updated script:

this is the script:

import os
import time
import multiprocessing
import subprocess
from multiprocessing import Lock

# ANSI escape codes for colors
RED = "\033[31m"
GREEN = "\033[32m"
YELLOW = "\033[33m"
BLUE = "\033[34m"
RESET = "\033[0m"

def update_repository(repo_path):
    try:
        subprocess.run(["git", "fetch"], cwd=repo_path, check=True)
        return "SUCCESS", repo_path
    except subprocess.CalledProcessError as e:
        return "ERROR", repo_path, str(e)
    except Exception as e:
        return "EXCEPTION", repo_path, str(e)

if __name__ == "__main__":
    extensions_folder = "extensions"

    # Find all subdirectories in extensions folder that contain a .git directory
    git_dirs = [os.path.join(extensions_folder, d) for d in os.listdir(extensions_folder) if os.path.isdir(os.path.join(extensions_folder, d, ".git"))]

    # Set the number of processes to use and divide the repositories into groups
    num_processes = multiprocessing.cpu_count()
    repo_groups = [git_dirs[i:i+num_processes] for i in range(0, len(git_dirs), num_processes)]

    # Create a lock to ensure that only one process updates a repository at a time
    lock = Lock()

    # Update each repository in parallel using multiprocessing
    print(f"{YELLOW}Updating Git repositories in parallel...{RESET}")
    start_time = time.time()
    with multiprocessing.Pool(processes=num_processes) as pool:
        results = []
        for i, group in enumerate(repo_groups):
            for repo in group:
                results.append(pool.apply_async(update_repository, (repo,)))
                print(f"\033[KUpdating repository {i*num_processes+1} to {(i+1)*num_processes}...\r",)
                 
                
                
            while not all(result.ready() for result in results):
                time.sleep(0.1)
            print("\033[K\033[1A\033[K", end="")
    end_time = time.time()

    # Print summary
    # Print summary
    print(f"\033[KUpdated {len(git_dirs)} Git repositories in {end_time - start_time:.2f} seconds.")
    num_success = 0
    num_skipped = 0
    successes = []
    errors = []
    exceptions = []
    skipped = []
    for result in results:
        status, repo_path, *message = result.get()
        if status == "SUCCESS":
            num_success += 1
            successes.append(os.path.basename(repo_path))
        elif status == "ERROR":
            errors.append(os.path.basename(repo_path))
            print(f"  \033[31mERROR\033[0m: {os.path.basename(repo_path)} - {message[0]}")
        elif status == "EXCEPTION":
            exceptions.append(os.path.basename(repo_path))
            print(f"  \033[33mEXCEPTION\033[0m: {os.path.basename(repo_path)} - {message[0]}")
        elif status == "SKIPPED":
            num_skipped += 1
            skipped.append(os.path.basename(repo_path))
            print(f"  \033[35mSKIPPED\033[0m: {os.path.basename(repo_path)}")

    # Print detailed report of successful updates
    print("Fetch report:")
    for success in successes:
        print(f"  \033[32mSUCCESS\033[0m: {success}")
    for skip in skipped:
        print(f"  \033[35mSKIPPED\033[0m: {skip}")
    for error in errors:
        print(f"  \033[31mERROR\033[0m: {error}")
    for exception in exceptions:
        print(f"  \033[33mEXCEPTION\033[0m: {exception}")

    # Print summary of updates and skipped folders
    if num_skipped > 0 and len(errors) > 0:
        print(f"\033[K{num_success} repositories updated successfully; {num_skipped} folders skipped; {len(errors)} errors.")
    elif num_skipped > 0:
        print(f"\033[K{num_success} repositories updated successfully; {num_skipped} folders skipped.")
    elif len(errors) > 0:
        print(f"\033[K{num_success} repositories updated successfully; {len(errors)} errors.")
    else:
        print(f"\033[K{num_success} repositories updated successfully.")

It must be saved in the main WebUI folder or in any folder that contains the extensions folder.

To use the script, simply create a update_extensions.py file in the same directory as the WebUi and use the following command in a run_update.bat file to run the script:

python
Copy code
@echo off
python "%~dp0update_extensions.py"

Or, to use it at every run of the webui-user.bat call it before the webui.bat like this:

@echo off

set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS=

call update_extensions.py

call webui.bat

The update_extensions.py script also provides detailed output to the user, including the names of extensions that have been updated and the commit messages associated with those updates. In the event of errors or issues with the update process, the script will also provide detailed error messages to help the user diagnose and resolve any issues.

In summary, the update_extensions.py script is a powerful tool that makes it easy to keep the extensions for the Automatic1111 WebUI up to date. By automating the update process, the script saves time and ensures that the extensions are always ready to use.

This is what it looks like in the terminal (all the white text makes it less quick to read and it prints really fast)

update_extensions.zip

To check 49 extensions it takes 1s average vs the 20s average that the WebUI needs to check for updates.

It can vary on each machine.

(I'll update the screenshot as soon as I'll have something that have been update, multiple runs to check and fix the script caused my extensions to all be very update when I finally come to a decent result to me)

Here i add the files that i personally use:

[midihex]

nice, I had to add os.system("") to the start to get the ansi colours behaving on windows

[Gustavsenay]

I would investigate it but really I don't have the knowledge to, I use it on windows 11 and I asked ChatGPT to write it in a way that it should work for mac pc and linux.
The only thing that came to my mind is that maybe you mis-copied it and don't have the import os at the beginning.

[Gustavsenay]

the code was not reliable, just updated it

[midihex]

No need to Investigate, I was simply fixing the code. I wrote this for others benefit if they run into the same issue in future,
os.system("") makes the ANSI escape sequence work correctly on Windows. Just add it near the top of the script somewhere after import os

[vladmandic]

  ls extensions/ | while read LINE; do
    pushd extensions/$LINE >/dev/null
    git pull --quiet
    VER=$(git log -1 --pretty=format:"%h %ad")
    URL=$(git remote get-url origin)
    popd >/dev/null
    echo "- $VER $URL"
  done

output:

Extensions:
- 017757d Sun Feb 5 15:08:07 2023 +0800 https://github.com/guaneec/custom-diffusion-webui.git
- a43d11c Wed Feb 22 13:35:54 2023 -0300 https://github.com/zero01101/openOutpaint-webUI-extension
- e5b773a Sat Feb 11 19:38:18 2023 +0500 https://github.com/klimaleksus/stable-diffusion-webui-embedding-merge
- 0f3f699 Fri Dec 9 11:50:47 2022 +0800 https://github.com/yfszzx/stable-diffusion-webui-inspiration

[Gustavsenay]

sorry i'm too ignorant to understand this without an explanation..

[vladmandic]

its just a short shell script that does the same thing and below is the output of the script.

[Gustavsenay]

Is as slow as the webUI for me.
This is supposed to do the same in shell but quicker, I haven't the time to really try btw:

ls extensions/ | parallel --will-cite --line-buffer -j0 'pushd extensions/{} >/dev/null && git pull --quiet && VER=$(git log -1 --pretty=format:"%h %ad") && URL=$(git remote get-url origin) && echo "- $VER $URL" && popd >/dev/null' | while read LINE; do
    echo "$LINE"
    read -n1 -r -s -p "Press any key to continue"
done

[sashaok123]

update_extensions_and_auto1111.zip

[Gustavsenay]

I've just updated the script using multiprocessing, the results under 1s are incredible

[sashaok123]

Listen, but you can do the same thing, but to install dependencies from requirements and requirements_versions, think about it.

[Gustavsenay]

is exactly what I'm trying to do right now my friend, isn't easy because is about modifying the launch.py and webui.py and those are pretty long scripts that makes chatgpt to be a bit buggy or forget things while printing, I'll reply here as soon as I'll have a reliable solution.

[Gustavsenay]

meanwhile check the updated script

[AugmentedRealityCat]

Should this be made into a Pull Request ?

[Gustavsenay]

Whad do you mean? remember I'm almost illiterate here

[AugmentedRealityCat]

I'm almost illiterate as well, so take this with a grain of salt as maybe I'm completely wrong but it is my understanding that, once ready and 100% functional, we could make this code into an official Pull Request over here, and basically ask Automatic1111 to officially merge it in the next release.

[Gustavsenay]

I'll investigate that, in the meanwhile I saved everything as a repo (working on readme right now), then I'll investigate the pull request which ui seems unfriendly to be honest.

[Gustavsenay]

here is the repo btw:

https://github.com/Gustavsenay/Super-Updater-by-ChatGPT

[Gustavsenay]

just updated, a bit slower (4 secs average) but safe and reliable

[jayp0]

with Pool(processes=os.cpu_count()) as pool:

Wouldn't it be better to replace os.cpu_count() with a more sensible number?
It will slow things somewhat, but you shouldn't run 8~32 git pulls in parallel.

[Gustavsenay]

why is that something I should avoid?

[Gustavsenay]

I've investigated, I'm fixing the script to work with git fetch, plus I'm implementing a script that check the extension folder and updates the submodules list before the fetch runs. Thanks to have pointed out something so important!

[Gustavsenay]

I'v just updated the script taking also that in account

[riperbot]

I wrote a script with Gemini 1.5 Pro support to update extensions in a multithreaded manner:
multithreaded_extensions_update_script.zip

This archive contains:
update-extensions_only.bat
webui-user_with_update-extensions.bat
update_extensions.py
status_description.txt