About Vocabulary extension

[alielfilali01]

I'am sorry but this more like a question to the team behind this impressive paper, rather than an issue

First thing first, thank you so much for these efforts 🙏🏻, I would love to know how you guys managed to extend the vocabulary of the original tokenizer without training the tokenizer from scratch again ?

[wabyking]

We are working on this and will release it as soon as possible.

Thanks for your comment.