Distributed training in PyTorch on 4 GPUs with DDP | Minimum Viable GPT #12

For this episode, we rent four RTX 4090s on Vast.ai and pretrain a model about six times bigger than our laptop's, in GPT-2 small's shape, with PyTorch's DistributedDataParallel and torchrun. We try it on the laptop first, then measure the training speed on one, two and four GPUs, and train for thirty minutes on all of TinyStories. And we compare the result with the laptop's model, in bits per character and in its stories, and save it for Hugging Face. The series: https://mvgpt.nenuadrian.com The code: https://github.com/nenuadrian/mvGPT Run it in Colab: https://colab.research.google.com/git... The playlist:    • Minimum Viable Generative Pre-trained Tran...   Previous episode:    • A byte-level BPE tokenizer for a GPT in C+...   Next episode:    • Flash attention from scratch in PyTorch an...   Chapters: 0:00 Intro 2:21 Data parallelism 4:16 Pretraining with DDP 10:50 Tried on the laptop 11:33 All of TinyStories 14:46 Renting four GPUs 16:47 One, two and four GPUs 18:25 Thirty minutes on four GPUs 19:53 What thirty minutes bought 20:57 On Hugging Face 27:36 Wrap-up To train on GPUs you rent by the hour, cheaply, I recommend Vast.ai (referral link): https://cloud.vast.ai/?ref_id=746308 Music: John Tramp (https://pixabay.com/users/tramp963-35...) #llm #pytorch #machinelearning