How Data Moves in AI Training: Four Tiers from Archive to GPU Memory

Data for large AI training flows through four tiers, from archive storage up to the memory on the accelerator, and a slowdown in any one tier leaves accelerators idle. This video follows that path in order: Tier 0 (CPU memory and accelerator memory), Tier 1 (local NVMe SSDs, and CXL), Tier 2 (shared cluster storage) and Tier 3 (capacity and archive). It says what each tier is for and what is changing in it. The main point is balance between cost and performance. Chapters 00:00 Why look at AI infrastructure through data flow 00:35 Tier 0: CPU memory (DRAM) and accelerator memory (HBM) 02:02 Tier 1: local NVMe SSDs, and CXL 03:48 Tier 2: shared storage for the cluster 05:28 Tier 3: capacity and archive 06:46 Balancing cost and performance 07:14 What to cover next Next in the series: KV Cache 101: What It Is, How It Works, Where to Keep It    • KV Cache 101: What It Is, How It Works, Wh...   Start the full series here: Memory Wall playlist    • Memory Wall