LoRA Explained (1/6): Low-Rank Updates, Real Memory Savings, and the Math

Part 1/6 of Parameter-Efficient & Memory-Efficient LLM Training. Series playlist:    • Parameter-Efficient & Memory-Efficient LLM...   Order: LoRA, QLoRA, ESFT, GaLore, BAdam, LISA. Next episode: QLoRA; follow the playlist rather than an unverified future video URL. Across the series: which weights train, and which memory term shrinks? Parameter-efficient adaptation is distinct from memory-efficient full-parameter training. Exact citation: Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685v2, 16 October 2021. https://arxiv.org/abs/2106.09685v2 All 26 pages, including appendices A-H, reviewed. Existing PDF SHA-256 e9a0d3128767db616085dc0f4e6e455e672e89af823e8ed1282793682787395a matches a fresh version-two download byte for byte. This explains v2, not an unmarked later revision. Mechanism: freeze W0 with d outputs and k inputs; train B of shape d by r and A of shape r by k. The effective layer is W0 plus (alpha/r)BA. Factor count is r(d+k), versus dk for a dense correction. Gaussian A and zero B preserve the initial prediction: the first loss gradient reaches B, while A's initially vanishes. Later both learn. Merge the scaled product for ordinary dense inference. A merged adapter is one task choice; mixed-task batching is less straightforward. Checked illustrative trace: a two-output/three-input frozen matrix, rank-one adapter, explicit squared loss, first gradients, gradient-descent step, merged multiplication, and second-pass A gradient. Loss 2.5 to0.025 is toy arithmetic, NOT a benchmark result. A separate 4096-wide rank-eight count gives256x fewer entries for one matrix, NOT256x less total GPU memory. Headline denominators: GPT-3 175B, rank4 on query and value over96 layers. Paper Section4.2 rounds full task checkpoint350GB versus adapter35MB to10,000x. Exact architecture count is18,874,368 adapter entries; FP16 is37.75 decimal MB, and full-to-adapter parameter ratio about9,285x. The350GB base still must be hosted. Reported training memory is1.2TB to350GB, about3.43x from rounded values, described as roughly3x, NOT10,000x. Main savings remove original-weight gradients and Adam states; activations, frozen weights and buffers remain. Precision, sequence length, batch size, sharding and implementation matter. Throughput footnote:32.5 to43.1 tokens/s per V100 with equal model-parallel shard count means32.6% higher throughput, or24.6% less time/token. The paper's25% speedup wording needs that distinction. Results: Table4 WikiSQL73.4% small LoRA versus73.8% full tuning; reported fluctuation around0.5 percentage points. Table15 disambiguates4.7M budget: value-onlyrank2 gives MNLI91.7%, q/vrank1 gives91.3%, versus full89.5%. Table6 low-rank sufficiency is empirical, not universal. Appendix Table18 GPT2 E2E BLEU68.72 at rank1 versus70.38 at rank4; hyperparameters partly tuned at rank4. Public limitations/ambiguities: low rank constrains accumulated BA, not merely each independent step, and does not force W0+BA low rank. Useful full-tuning changes need not fit a small fixed adapter. Subspace heatmaps/amplification are geometric evidence, not identified semantic concepts or a proof about all full-tuning solutions. Main study freezes MLP; other modules and rank selection remain open. Some GLUE tasks initialize adapters from MNLI unless controlled dagger setup; imported baselines and checkpoint selection matter. Appendix D.4 maps all-four projectionsrank2 to37.7M, inconsistent with formula/Table15's18.8M. We use explicit counts and disclose the discrepancy. Table18 rounded loss ties do not identify a unique optimal rank. Equation3 omits the alpha/r scale in notation; prose specifies it and our merge includes it. Independent explanation, no author endorsement. Authors' measurements; no experiments reproduced. AI-generated narration: local Kokoro af_bella, en-us, speed1.0.