How is this Chinese AI so efficient? Kimi K3 technical report explained.

How does a 2.8-trillion-parameter open model read a million tokens? This lesson rebuilds the three architectural ideas at the heart of the Kimi K3 technical report, from scratch and with worked examples, following four lines of code (x = 3 … x = 7 … print(x)) through the model. • Along the sequence: why a fixed-size memory that only adds can't overwrite a stale value, how the delta rule fixes it (predict, compare, write the error — one gradient step), and what Kimi Delta Attention adds: a forgetting factor for every channel, a lower bound that keeps the chunked GPU computation in range, and a few global attention layers with no position encoding. • Through depth: Attention Residuals replace the residual stream's uniform sum with attention over earlier layers. • Across width: 896 experts with 16 active, routed in a half-width latent space, kept stable by normalization, a bounded gate (SiTU-GLU) and Quantile Balancing. • Plus: multi-effort RL and multi-teacher on-policy distillation, and what the report's evidence does and doesn't show. Toy numbers are constructed teaching examples, labelled on screen; reported numbers come from the papers below. *Chapters* 00:00 Three directions 01:56 A memory that does not grow 04:44 The delta rule 08:28 Kimi Delta Attention 12:13 Keeping some full attention 15:11 Attention across depth 18:59 Many experts, few active 21:18 Keeping a thousand experts stable 25:30 One model, three effort levels 27:33 What the report shows *Sources* • Kimi Team, "Kimi K3: Open Frontier Intelligence" (technical report), arXiv:2607.24653 — https://arxiv.org/abs/2607.24653 • Kimi Team, "Kimi Linear: An Expressive, Efficient Attention Architecture", arXiv:2510.26692 — https://arxiv.org/abs/2510.26692 • Kimi Team, "Attention Residuals", arXiv:2603.15031 — https://arxiv.org/abs/2603.15031 • Elango et al., "LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts", arXiv:2601.18089 — https://arxiv.org/abs/2601.18089 • Model weights: https://huggingface.co/moonshotai/Kimi-K3 *Hashtags* #KimiK3 #LLM #MachineLearning #DeepLearning #Transformers #LinearAttention #DeltaRule #MixtureOfExperts #AttentionResiduals #AIResearch