Inside a GPU, and Why Tokens Per Second Is a Memory Number | Inference Engineering Ep 11

A datacenter GPU's cores run at under half the clock speed of a laptop CPU's, yet it does the matrix math of a forward pass about a hundred times faster. This episode opens up the chip: Streaming Multiprocessors, Tensor Cores and the Matrix Multiply and Accumulate instruction, FLOPS and the sparsity asterisk on spec sheets. Then the memory side: VRAM, HBM, SRAM caches, capacity and out-of-memory errors, and why the H200 generates more tokens per second than the H100 with identical compute. Based on the book Inference Engineering by Philip Kiely (Baseten Books, 2026): https://philipkiely.com/books Credits (the people and work behind the ideas in this episode): Streaming Multiprocessor: E. Lindholm, J. Nickolls, S. Oberman, J. Montrym, "NVIDIA Tesla: A Unified Graphics and Computing Architecture", IEEE Micro, 2008. https://doi.org/10.1109/MM.2008.31 CUDA: J. Nickolls, I. Buck, M. Garland, K. Skadron, "Scalable Parallel Programming with CUDA", ACM Queue, 2008. https://queue.acm.org/detail.cfm?id=1... Tensor Cores: NVIDIA, "NVIDIA Tesla V100 GPU Architecture" (Volta whitepaper), 2017. https://images.nvidia.com/content/vol... 2:4 structured sparsity: A. Mishra, J. Albericio Latorre, J. Pool, D. Stosic, D. Stosic, G. Venkatesh, C. Yu, P. Micikevicius, "Accelerating Sparse Deep Neural Networks", 2021. https://arxiv.org/abs/2104.08378 High Bandwidth Memory (HBM): developed by AMD and SK hynix; JEDEC standard JESD235 (2013; current revision JESD235A). https://www.jedec.org/standards-docum... DRAM: Robert H. Dennard, IBM Research, the one-transistor DRAM cell, 1966 (patented 1968). https://www.ibm.com/history/dram Chapters: 00:00 Which has the faster cores? 00:16 Slower clocks, about 100x the math 00:36 One brilliant chef vs a stadium of line cooks 01:20 From the rack to the SMs 01:35 CUDA Cores, Tensor Cores, Special Function Units 01:55 Matrix Multiply and Accumulate 02:24 A hundred thousand slow computers 02:45 FLOPS and the spec sheet 03:05 The sparsity asterisk 03:33 FLOPS double as precision halves 03:49 When FLOPS decide speed 04:07 H100 vs H200: same FLOPS, more tokens 04:43 Decode waits on memory 05:03 VRAM and HBM 05:25 DRAM, SRAM and the caches 06:03 The memory pipes 06:24 Capacity and out-of-memory errors 07:10 Bandwidth sets tokens per second 07:41 Tokens per second is a memory number 07:59 The rest of the lineup Inference Engineering, episode 11. Animated in Manim; narration voice: Cartesia Sonic (Casper). Company and GPU names are plain-text labels; no logos.