Arxiv Data

[ZhengTang1120]

Hi,

You mentioned that the model is trained on scientific papers(29B arxiv data) as a part of math component. I am wondering if you included the full articles or just math contents?

Thank you,
Zheng Tang

[billxbf]

+1

[hills-code]

I use the arxiv dataset as a subset of the proof-pile-2 dataset (https://huggingface.co/datasets/EleutherAI/proof-pile-2)