arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.The original BERT uses a subword-level tokenization with the vocabulary size of 30K which is learned after input preprocessing and using several heuristics. RoBERTa uses bytes instead o