Direct Preference Optimization: Your LLM Is Secretly a Reward Model

RLHF needs a separate reward model and a fragile reinforcement learning loop just to align a language model with human preferences. DPO shows you can skip both — the optimal reward model is already hiding inside the language model itself. This video derives DPO from scratch: starting from the same KL-constrained objective as RLHF, it shows algebraically why the "impossible" partition-function term cancels out, leaving a single classification-style loss you can train directly on preference pairs. We cover: • Why you can't score a single response, only compare two • The standard RLHF recipe, and why the RL step is fragile • Deriving DPO's closed-form optimal policy • The term that makes the reward "unusable" — and why it cancels anyway • The full DPO loss, in one line • What one gradient step actually does to the model • Does skipping RL cost you anything? Generality, judge reliability, real preference data 📄 Paper: "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" — Rafailov, Sharma, Mitchell, Ermon, Manning & Finn (Stanford) 🔗 https://arxiv.org/abs/2305.18290 ⏱️ Chapters 0:00 What you're actually asking for 0:39 You can't score it, but you can compare it 1:30 The standard recipe: RLHF 2:15 Why step three hurts 3:04 A world with three answers 3:49 The exact answer, run on real numbers 4:51 Why that exact answer is unusable 5:40 Read the equation backwards 6:31 The impossible term cancels 7:18 The whole method is one loss 8:12 One step of training 9:00 What the update actually does 9:53 Does the trick cost you generality? 10:50 Does it optimise better? 11:39 Does it hold up on real preferences? 12:30 Can you trust the judge? 13:18 What it costs, and what it doesn't settle #DPO #RLHF #LLM #MachineLearning #AIAlignment