GPT-2 as a Spam Classifier: Classification Fine-Tuning Explained

Can a GPT model classify text? Yes. In this video, I turn pre-trained GPT-2 into a spam classifier using classification fine-tuning in PyTorch. You do not need a separate ML model for classification. You keep the GPT-2 body, change the output head, and train a small part of the model. What you will learn: The difference between instruction fine-tuning and classification fine-tuning Why we replace the 768 → 50,257 output head with 768 → 2 (spam / not spam) How to prepare the SMS Spam Collection dataset: labels, a 70/10/20 split, and padding to 120 tokens with token 50256 Freeze, replace, unfreeze: why only about 7.09M of 124M parameters train Why we classify from the last token (causal attention) Why we train on cross-entropy loss but report accuracy Chapters 0:00 Intro: LLMs can classify text 0:34 What is fine-tuning? Instruction vs classification 1:35 How GPT-2 becomes a classifier 3:41 The full pipeline: 10 steps in 3 stages 4:21 Dataset: download, label and split 5:12 Why tokenize if the weights are frozen? 5:46 Make all messages the same length (truncate vs pad) 6:17 Padding with |endoftext| (50256) 6:55 Data loaders and batch shapes 7:21 Freeze, replace, unfreeze (code) 8:13 What is trainable now 8:48 How the output shape changes 9:06 Why we use the last token 9:53 Accuracy 10:21 Cross-entropy loss 10:47 Training loop and hyperparameters 11:12 Results and wrap-up Setup used: GPT-2 small (124M), AdamW (lr 5e-5, weight decay 0.1), 5 epochs, batch size 8. If this helped, subscribe for more videos on building and fine-tuning LLMs from scratch. #GPT2 #FineTuning #LLM #PyTorch #MachineLearning