Add a button to prune unused hash #16700
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Haoming02
requested review from
AUTOMATIC1111,
w-e-w and
catboxanon
as code owners
w-e-w
requested changes
Dec 17, 2024
Comment on lines
+146
to
+168
| for db in ('hashes', 'hashes-addnet', 'safetensors-metadata'): | ||
| existing_cache = cache(db) | ||
| total_count = len(existing_cache) | ||
| with tqdm.tqdm(total=total_count, desc=f'pruning {db}') as progress: | ||
| for name in existing_cache: | ||
| if '/' not in name: | ||
| progress.update(1) | ||
| continue | ||
|
|
||
| category, filename = name.split('/', 1) | ||
| if category.lower() == 'lora': | ||
| exists = file_exists(os.path.join(models_path, 'Lora'), filename) | ||
| elif category.lower() == 'checkpoint': | ||
| exists = file_exists(os.path.join(models_path, 'Stable-diffusion'), filename) | ||
| elif category.lower() == 'textual_inversion': | ||
| exists = file_exists(cmd_opts.embeddings_dir, filename) | ||
| else: | ||
| progress.update(1) | ||
| continue | ||
|
|
||
| if not exists: | ||
| del existing_cache[name] | ||
| progress.update(1) |
There was a problem hiding this comment.
this is wrong
-
completely forgot about
hypernet -
you are assuming that the user is using the default directories which may not be the case, that the path of each model directory is a complicated mess, some models such as Lora can come from multiple directories
-
extensions could also add own cache entries, I don't think there's a way that you can avoid not accidentally removing those entries
as you said
which can reduce file size by a tiny amount, and improve load time by an even tinier amount...
so personally I wouldn't bother makeing it work, not worth the effort
|
But yeah, not really worth it to deal with the mess that is the current model system... |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Simple Description: When you download/train a model to experiment, the Webui caches the hash for it. Overtime, you will have hundreds if not thousands of hash for models that no longer exist. Therefore, I added a button to clean this up, which can reduce file size by a tiny amount, and improve load time by an even tinier amount...
Summary of Changes:
Prune all unused hashbutton in Settings/Actions, which when clicked calls:prune_unused_hashfunction incache.pyChecklist
Performance
Checking hundreds of entries took less than a second~