LLMs Explained: Tokens, Embeddings, Transformers and More

In this video CJ explains chat bots, neural networks, tokenization, embeddings, transformers and more. He shows real working code for everything and gives historical context along the way. View the code here: https://github.com/w3cj/how-llms-work 00:00 - intro 01:54 - history of chatbots 04:21 - chat bot code 06:49 - black box thinking 07:41 - what are neural networks? 08:33 - history of neural networks 10:27 - XOR neural network code 15:35 - what is tokenization? 16:32 - history of tokenization 17:55 - tokenizer code 21:37 - what are embeddings? 22:04 - history of embeddings 23:33 - what is a vector? 24:39 - embedding code 30:48 - history of transformers 32:33 - what is a transformer? 33:11 - what is self attention? 34:25 - what is a feed forward network? 34:37 - what are stacked transformer blocks? 41:37 - what are softmax, temperature and top-p? 42:54 - the autoregressive loop 43:28 - the context window 44:45 - the full picture 45:26 - what is fine-tuning 46:39 - RLHF 47:24 - tool calling 48:36 - the AI summit 49:51 - final thoughts Claude Shannon | https://en.wikipedia.org/wiki/Claude_... Betty Shannon | https://en.wikipedia.org/wiki/Betty_S... Prediction and Entropy of Printed English (1951) | https://www.princeton.edu/~wbialek/ro... Alan Turing | https://en.wikipedia.org/wiki/Alan_Tu... Turing Test | https://en.wikipedia.org/wiki/Computi... Joseph Weizenbaum | https://en.wikipedia.org/wiki/Joseph_... ELIZA | https://en.wikipedia.org/wiki/ELIZA ELIZA - Weizenbaum (1966) | https://dl.acm.org/doi/10.1145/365153... ELIZA Archaeology Project | https://sites.google.com/view/elizaar... Computer Power and Human Reason | https://archive.org/details/computerp... Kenneth Colby | https://en.wikipedia.org/wiki/Kenneth... PARRY | https://en.wikipedia.org/wiki/PARRY Artificial Paranoia (1971) (PDF) | https://courses.cs.umbc.edu/graduate/... ELIZA connected to PARRY - RFC 439 | https://datatracker.ietf.org/doc/html... ALICE | https://en.wikipedia.org/wiki/Artific... SmarterChild | https://en.wikipedia.org/wiki/Smarter... McCulloch & Pitts (1943) | https://en.wikipedia.org/wiki/A_Logic... Frank Rosenblatt | https://en.wikipedia.org/wiki/Frank_R... Perceptron | https://en.wikipedia.org/wiki/Perceptron Rosenblatt's Perceptron - Cornell | https://news.cornell.edu/stories/2019... New Navy Device Learns by Doing - NYT (1958) | https://timesmachine.nytimes.com/time... Perceptrons | https://en.wikipedia.org/wiki/Percept...) Backpropagation - Rumelhart, Hinton & Williams (1986) | https://www.nature.com/articles/323533a0 BPE - Philip Gage (1994) (PDF) | https://www.derczynski.com/papers/arc... Byte Pair Encoding | https://en.wikipedia.org/wiki/Byte-pa... BPE for NMT - Sennrich, Haddow & Birch (2015) | https://arxiv.org/abs/1508.07909 Gottlob Frege | https://en.wikipedia.org/wiki/Gottlob... Foundations of Arithmetic - Frege (1884) | https://archive.org/details/foundatio... Context Principle | https://en.wikipedia.org/wiki/Context... J.R. Firth | https://en.wikipedia.org/wiki/John_Ru... Studies in Linguistic Analysis - Firth (1957) | https://archive.org/details/studiesin... Word2Vec - Mikolov et al. (2013) | https://arxiv.org/abs/1301.3781 Word2Vec Negative Sampling - Mikolov et al. (2013) | https://arxiv.org/abs/1310.4546 Attention Mechanism - Bahdanau, Cho & Bengio (2014) | https://arxiv.org/abs/1409.0473 GNMT - Wu et al. (2016) | https://arxiv.org/abs/1609.08144 Xavier Initialization - Glorot & Bengio (2010) | https://proceedings.mlr.press/v9/glor... Attention Is All You Need - Vaswani et al. (2017) | https://arxiv.org/abs/1706.03762 Transformer | https://en.wikipedia.org/wiki/Transfo...) Common Crawl | https://en.wikipedia.org/wiki/Common_... RLHF | https://en.wikipedia.org/wiki/Reinfor... Dartmouth Workshop (1956) | https://en.wikipedia.org/wiki/Dartmou... Dartmouth AI Proposal (1955) | https://www-formal.stanford.edu/jmc/h... Mamba - Gu & Dao (2023) | https://arxiv.org/abs/2312.00752 xLSTM - Beck et al. (2024) | https://arxiv.org/abs/2405.04517 Jamba - Lieber et al. (2024) | https://arxiv.org/abs/2403.19887 JEPA - Yann LeCun (2022) | https://openreview.net/pdf?id=BZ5a1r-... ---------------------------------------------------------------------------- http://www.syntax.fm Brought to you by https://sentry.io/syntax #LLM #machinelearning #programming #explained #javascript #typescript #syntax #syntaxfm