QLoRA Explained: What Actually Trains in 4 Bits? | Part 2/6

Episode 2/6 of Parameter-Efficient & Memory-Efficient LLM Training. Series playlist:    • Parameter-Efficient & Memory-Efficient LLM...   Order: LoRA, QLoRA, ESFT, GaLore, BAdam, LISA. Previously: LoRA reduces trainable parameters. Next: ESFT. No future episode URLs are assumed. Exact citation: Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314v1, 23 May 2023. https://arxiv.org/abs/2305.14314v1 All 26 PDF pages and appendices A-G reviewed, including visual equations/tables. Local PDF SHA-256 matches a fresh explicit-v1 download: 33a4e757c19d6ea8d3cca8958fce8fe405e290272e135c99ee66065c775d5bd1. The central distinction: LoRA changes which parameters train; QLoRA also compresses the frozen base. Base weights are stored as NF4 codes plus metadata, reconstructed for usually BF16 computation. Higher-precision adapters receive parameter updates. Gradients still propagate through frozen layers; this is not full-model four-bit gradient training. Activations, workspaces, adapter parameters/gradients, and optimizer state still cost memory. NF4 uses normal-distribution quantiles rather than naive linear spacing, with an asymmetric exact-zero construction and 16 real representatives. Optimality language is distribution-specific, not a universal minimum-error guarantee after block normalization. Appendix F finds departures from normality. Our five-level, three-bit illustrative toy is NOT NF4: block [-2,-0.4,0.37,2], scale2, reconstruction [-2,-0.5,0.5,2], signed errors [0,-0.1,0.13,0], MSE0.006725. A separate real Appendix-E NF4 lookup maps0.37/2=0.185 to0.1609302014, reconstructing0.3218604028, error-0.0481395972. These are checked arithmetic, not model experiments. Double quantization compresses scales. FP32 scale per64 weights costs32/64=0.5 bits/weight. FP8 scale codes and a FP32 scale per256 scale codes cost8/64+32/(64*256)=0.126953125. Total idealized payload plus scales:4.126953125 bits/weight. Savings0.373046875 bits/weight, about3.031 decimal GB at65B. Other metadata, mean offsets, padding, unquantized tensors and training allocations are not included in this idealized arithmetic. Paged optimizers move optimizer-state pages between CPU RAM and GPU memory using unified memory. They do not erase states, eliminate activation memory, or guarantee no slowdown. Page6 gives a scoped comparable-speed observation but leaves systematic slowdown characterization open. Host capacity and bandwidth matter. 65B on48GB is an authors' workload-specific result, not a promise for all contexts/batches. AppendixG Figure6 uses batch1, sequence512 and checkpointing; its input-gradient estimate excludes attention. Table6 deployment memory41GB is not full training peak. All-linear-layer adapter coverage mattered in the tested comparisons. Full16bit full-weight equivalence at33B/65B was explicitly NOT established; large-scale comparisons use16bit LoRA. Table4 mean MMLU53.1% NF4+DQ vs53.0% BF16 is scoped evidence, not guaranteed equality. Guanaco uses supervised cross-entropy, not RLHF; AppendixB selects9,209 OASST1 examples. The famous99.3% is a historical GPT4-judged relative score against then-current ChatGPT over80Vicuna prompts: order scores96.7%/101.9%, mean99.3%, reported95%CI ±4.4points. It is NOT general ability equivalence, today's ChatGPT comparison, or safety certification. Human/automated judgments and benchmarks disagree. Public failures include confident factual mistakes, arithmetic contradictions and unreliable instruction following; bias evaluation is limited. Adapter merging/requantization is a deployment caution, not a paper experiment. Independent explanation; no author endorsement; no experiments reproduced. AI-generated narration: local Kokoro af_bella, en-us, speed1.0. Suggest a paper: https://forms.gle/QLFzswLemfpFx5CXA