Progressive training?

[immartian]

Hi @karpathy , thanks for sharing this interesting series. They are not only educational but also inspiring.

While I enjoy the tutorial together with my own copycat of your code, I wonder if you can advise whether it's possible to incrementally train a model, say adding Leo Tolstoy onto Shakespere, rather than retraining a data set combined time by time?

I'm asking because I wonder any new methods different from GPTs that can learn progressively like the human brain. If so, we can probably realize some level of baby intelligence without directly going to LLM.

[immartian]

I also found some similar arguments https://www.linkedin.com/feed/update/urn:li:activity:7233335447110696960?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A7233335447110696960%2C7233471273022959616%29&dashCommentUrn=urn%3Ali%3Afsd_comment%3A%287233471273022959616%2Curn%3Ali%3Aactivity%3A7233335447110696960%29 and feel it's an imminent challenge.

[WeileiZeng]

Surely, you can do it. You can change the data source in any epoch, and the model will learn from it.

[immartian]

I guess what I'm asking is when you've pretrained a model, can you add incremental states upon it?

[chris-aeviator]

@immartian that's exactly what @WeileiZeng said - an epoch is the incremental next state that you are referring to and you can take a pre trained model and add more epochs to it, also with different data as the previous. You can find some useful info on what to train in which order in the Llama 3 paper (94 pages).