Is the MetaMathQA dataset suitable for pre-training?

[JingyiWang3]

Hi，thank you for your excellent work!
I would like to know if augmented datasets like MetaMathQA are suitable for pre-training?

[yulonghui]

Hi，Jingyi, Many thanks for your kind words.
I think MetaMathQA is suitable for pre-training. In fact, OpenChat-3.5 utilized our data and indeed achieved impressive performance. https://huggingface.co/openchat/openchat_3.5

However, one thing I believe needs attention is that you should publicly disclose the type of data you used, such as MetaMathQA. This way, others can assess whether your model has encountered GSM8K-like data.