3 Tokenizer 2/2 | Build an LLM from Scratch with Pytorch | Ep.3

Now we train the tokenizer itself. We use Byte Pair Encoding (BPE) with a vocabulary of 32,000 tokens. It learns from a sample of Arabic and English documents. We add special tokens for padding, start and end of text, language, and chat roles. Then we test the tokenizer on real Arabic and English sentences. Finally, we tokenize the whole corpus once and save it as a binary file. Each document gets [BOS] and [EOS] markers, so the model knows where one document ends and the next one starts. Source code: https://github.com/bakrianoo/jabarti-...