Textual Inversion loss logging suggestion

[Heathen]

As the tile says, could we get a couple CSV files with the loss logged during training?

File 1: Just the loss for each step
File 2: The loss for each image each time they are processed

Mainly so we can see if the training is converging and if there are any problematic images that are increasing loss.

Edit for a second idea that seemed useful to me:
I just edited my own textual_inversion.py to do the following;
Every N steps, when generating the image, generate one with a random seed and one with a fixed seed.

Random seed helps with visualizing overall understanding of the concept and fixed seed with smaller details of the concept, so you know if your learning rate is too high (Images start bouncing back and forth) or too low (barely any changes) and when to stop training.

3rd edit:
For people wondering about learning rate, You can be really aggressive with it with 0.1to 0.5 for 500 steps, then drop back down to 0.005 which is already a bit strong for Adam. Default for the algorithm is 0.001. You can go lower than that if you're already getting good results but it might be missing details. In any case, the way it works, it should always converge or bounce closer to convergence, it's just a matter of when.

[telemole]

Could you share with us those changes? this would really help me know if i"m 'heading in the right direction' or wasting cycles - thanks!

[Heathen]

Sure, you want to go to /modules/textual_inversion)/textual_inversion.py and change

        if embedding.step > 0 and images_dir is not None and embedding.step % create_image_every == 0:
            last_saved_image = os.path.join(images_dir, f'{embedding_name}-{embedding.step}.png')

            p = processing.StableDiffusionProcessingTxt2Img(
                sd_model=shared.sd_model,
                prompt=text,
                steps=20,
                do_not_save_grid=True,
                do_not_save_samples=True,
            )

            processed = processing.process_images(p)
            image = processed.images[0]

            shared.state.current_image = image
            image.save(last_saved_image)

            last_saved_image += f", prompt: {text}"

to

        if embedding.step > 0 and images_dir is not None and embedding.step % create_image_every == 0:
            last_saved_image = os.path.join(images_dir, f'Seed4242-{embedding_name}-{embedding.step}.png')

            p = processing.StableDiffusionProcessingTxt2Img(
                sd_model=shared.sd_model,
                prompt=text,
                seed=4242,
                steps=20,
                do_not_save_grid=True,
                do_not_save_samples=True,
            )

            processed = processing.process_images(p)
            image = processed.images[0]

            shared.state.current_image = image
            image.save(last_saved_image)

            last_saved_image = os.path.join(images_dir, f'{embedding_name}-{embedding.step}.png')

            p = processing.StableDiffusionProcessingTxt2Img(
                sd_model=shared.sd_model,
                prompt=text,
                steps=20,
                do_not_save_grid=True,
                do_not_save_samples=True,
            )

            processed = processing.process_images(p)
            image = processed.images[0]

            shared.state.current_image = image
            image.save(last_saved_image)

            last_saved_image += f", prompt: {text}"

Be forewarned I did this in the most jank possible way and assuming no seed in the argument means random seed.

I've been asking it to create images after a certain number of epochs to avoid wasting time creating images. For example, if you have 24 images in your dataset, ask for it to generate every 240 steps. That's 10 epochs, 10 times it goes through every single image.

Also if you want to change the seed there, change 4242 to something else. I don't really want to learn how to mess around with Gradio to make an UI for it. So I left it as a suggestion for people with more knowledge and patience.

[telemole]

this is amazing thanks! I'll report back as I have some trainings I want to do tonight :)

[Heathen]

On a separate note, I'm testing changing
optimizer = torch.optim.AdamW([embedding.vec], lr=learn_rate)
to
optimizer = torch.optim.AdamW([embedding.vec], lr=learn_rate, amsgrad=True)

I'm on a 1070ti, so testing this is a little bit slow, 1.09s/it with the card limited to 60c. My plan is to do a 5000 step training with amsgrad on and off to see if it indeed converges better. Amsgrad option is from this paper.

[CodeExplode]

Just wondering if you saw any benefit using amsgrad?

[Heathen]

Not so far. I kind of gave up for now until my 3080 arrives, finally found a new 12gb Gigabyte one for cheap in my region.

I had none that gave the visual results I was hoping for.

This is magic ring ken after 1650 steps without amsgrad. 5000 steps currently takes about one hour and a half for me and I had to stop to use the computer for work.

From reading the paper, it should converge faster at least. It maximizes the correction vector where it should and avoids non-convergence. The thing is, without logging losses, you can't really objectively compare the two of them and I have to find some time to add a simple logging to the training. It's sunday and I'm about to go out deal with some RL stuff, so it's tough.