What did Andrej Karpathy's PhD teach AI to do? The Karpathy thesis, explained

Andrej Karpathy was a founding member of OpenAI and later the director of AI at Tesla. But before that, he was a PhD student at Stanford, working on a simple-sounding question: can a computer look at a picture and tell you, in plain English, what is going on? This time we make an exception: instead of a paper, we explain a full doctoral thesis, "Connecting Images and Natural Language" by Andrej Karpathy (Stanford University, 2016), advised by Fei-Fei Li: • Matching images and sentences: each image was split into 20 regions and each word was matched to the region that fit it best. The model learned these connections without being told which word goes with which part of the image. On Flickr30K, the right description was in the top 10 results 61.4% of the time, against 41.1% for a previous method. • Writing new captions: an image network summarises the photo, and a recurrent neural network writes the caption one word at a time. On MS COCO it scored 0.66 CIDEr, against 0.38 for copying a caption of the most similar training image. • DenseCap: one network that finds the interesting regions of an image and describes each one, in about 240 milliseconds per image. We also cover the limits, many of them measured by Karpathy himself. With his best settings, 46% of generated captions could be found word for word in the training data. Only 30% of the model's captions were unique (95.7% for humans), and the model used only 1,079 different words. In the conclusion he calls his own model "a texture-based sequence memorizer with some limited generalization", and explains what is still missing. Chapters 0:00 Who is Andrej Karpathy? 0:30 An exception: a PhD thesis 0:53 Turing's idea 1:11 What a computer sees 1:39 Words instead of labels 2:03 Three levels of seeing 2:41 Matching images and sentences 3:32 Did it work? 3:54 The limits of matching 4:26 Writing new captions 5:05 Glued pieces and a volleyball net 5:51 How original were the captions? 6:43 DenseCap 7:37 Fast, but not perfect 8:08 The conclusion 8:22 The scale photo 9:07 What is missing 9:47 So, what did his PhD teach AI? Thesis Karpathy, A. (2016) Connecting Images and Natural Language. PhD dissertation, Department of Computer Science, Stanford University. #AI #MachineLearning #ComputerVision #Karpathy #Science