Please add to the colab...

[DarkAlchy]

I would love it if you would tell us in the colab that you have found the NAME.ckpt and are resuming it (if we check the box) before it says resuming just as a way to verify it really did work as I left my chkpt as the old name of _300 but the new subject was _600 (another 300 steps) so it started up with no error. I stopped it and renamed the original .ckpt to _600 then restarted to be sure it really was working.

This was a few hours after the previous one was created.

edit: I see after I did not leave the session I did another 300 training and it said "Resuming Training.." so apparently I can't close it and come back later?

[TheLastBen]

it displays "resuming" only if you're continuing the traning during a current session.

If you want to resume a previous ckpt training, just enter its path in the "download model" cell in "path_to_trained_model"

[DarkAlchy]

That must not be in my notebook version because I just ran there to check for "path_to_trained_model" and nothing was found.

[TheLastBen]

I will add the option to resume previous session and display whether it found a previous model or not

[DarkAlchy]

Thank you.

[TheLastBen]

Now you can resume a previous session simply by entering the session name, make sure the box "resume training" is checked

[TheLastBen]

yes I implemented that, once you enter a session name, if it exists in your gdrive, it will use all the previous files, instances images and model

[TheLastBen]

and you can resume the training straightaway

[DarkAlchy]

Oh? Kick ass. Damn colab has shrunk me now to 2h per 24h-36h and just a month ago I was at 12h per day then 6 then 4 then 2.5 now down to 2h.

[joanorsky]

That is why kaggle might be a good option since they do have persistent drives and 37 hours per week of free GPU

[DarkAlchy]

That is why kaggle might be a good option since they do have persistent drives and 37 hours per week of free GPU

Never heard of them. As it is I use Colab only to train with.