GPU Programming | Threads, Warps, HBM, FlashAttention | Triton Lecture 1

What's in this video why GPUs have thousands of lanes, how threads, warps, blocks and grids work, why memory (not math) is usually the bottleneck, and why Triton lets you think in tiles. We finish with FlashAttention. No CUDA background needed; real numbers from the NVIDIA H100. What you'll learn • The GPU in your computer: PCIe, NVLink, clusters • Inside an H100: SMs, warp scheduler, tensor cores, shared memory • Threads → warps → blocks → grid, SIMT • Memory: SRAM, DRAM, HBM, caches, coalescing, fusion • CUDA vs Triton, roofline, memory- vs compute-bound • FlashAttention: tiles + online softmax Next: our first Triton kernel, vector add. GO DEEPER (my blogs) • The Transformers: a complete visual breakdown of the architecture, plus coding one from scratch https://www.vizuaranewsletter.com/p/t... • Speculative Decoding: theory and implementation in vLLM https://vizuara.substack.com/p/specul... • All my blogs (transformers, audio, inference): https://blogs.mayankpratapsingh.in/ ABOUT ME I'm Mayank Pratap Singh: deep learning, speech/audio and inference optimization. • Website / portfolio: https://www.mayankpratapsingh.in/ • LinkedIn:   / mayankpratapsingh022   • X (Twitter): https://x.com/Mayank_022 • GitHub: https://github.com/Mayankpratapsingh022 • YouTube:    / @mayankpratapsingh022   More Triton lectures and an audio deep learning playlist are coming. Subscribe so you don't miss them. REFERENCES • Triton documentation and tutorials: https://triton-lang.org/ • Introducing Triton (OpenAI): https://openai.com/index/triton/ • FlashAttention paper (Dao et al., 2022): https://arxiv.org/abs/2205.14135 CHAPTERS 00:00 Intro: you already use the GPU 01:32 What this lecture covers, and about me 01:53 One task: add two lists of numbers 02:20 Two ways to spend a chip's area 03:13 What a processor actually does 04:47 Every program is this loop 05:48 Latency vs throughput: a car and a bus 06:55 The same idea on chips: one fast core vs many slower lanes 08:02 Parallel work must be independent 09:01 Thousands of workers only help with independent work 09:55 The GPU inside your computer 11:57 What is PCIe? The road between the CPU and the GPU 12:58 When one GPU is not enough: NVLink, servers and clusters 15:32 Inside a real GPU: the NVIDIA H100 16:11 Zoom into one SM 17:00 Inside the SM, part 1: FP32 lanes, the calculators 17:55 Inside the SM, part 2: the warp scheduler, the conductor 19:22 Inside the SM, part 3: the tensor core, a matrix machine 20:20 Inside the SM, part 4: shared memory and the L1 cache 21:44 Thread, warp, block, grid by zooming out 22:42 A thread: one worker, one element 24:01 A warp: 32 threads that move in step 27:12 SIMT: single instruction, multiple threads 29:40 A block: a team of warps that can wait for each other 31:43 The grid: all the blocks of one launch 34:14 Blocks are dealt to SMs in waves 36:31 Each thread finds its element 39:07 Memory: close and tiny, or far and huge 40:43 Latency and bandwidth: think of a pipe 41:11 Two ways to store a bit: SRAM and DRAM 42:45 Three kinds of DRAM: DDR, GDDR and HBM 44:08 Why a GPU needs HBM: thousands of hungry lanes 45:44 HBM up close: memory stacked like a building 47:25 How much each level holds on an H100 47:45 What is a cache? L1 and L2 48:21 Hiding the wait: many warps take turns 49:32 Coalescing: read neighbours together 50:52 Fusion: don't send intermediates back to memory 52:12 Why Triton: the compiler takes over the hard parts 52:47 Thinking in threads vs thinking in tiles 53:25 Where Triton sits 53:47 Two ways a kernel can be slow 55:01 Which one is it? Count the math per byte 56:24 Count the work: math per byte 57:10 The roofline: which limit do you hit first? 58:05 Attention needs a score for every pair of tokens 59:38 How the N × N matrix fills GPU memory 1:00:55 Standard attention: the big matrix keeps leaving the chip 1:02:34 FlashAttention, idea 1: work in tiles that fit on-chip 1:03:34 FlashAttention, idea 2: softmax without seeing the whole row 1:05:03 FlashAttention: one fused kernel, every SM at once 1:05:52 The result: memory that grows like N, not N² 1:06:41 Go deeper: transformer and inference blogs 1:07:22 What's next: your first Triton kernel #GPUProgramming #Triton #CUDA