llama.cpp FP8 vs Q8: Which Quant Should You Run?

For the 6.74B-parameter Llama model in llama.cpp's Metal benchmark, F16 uses 12.55 GiB and Q8_0 uses 6.67 GiB. OCP FP8 E4M3 lands in roughly the same storage class as Q8_0, so it is not automatically a better fit or a faster quant. This video compares FP8, Q8_0, and F16 across storage, execution, and output quality. It explains the current paths for NVIDIA H100, AMD CDNA4 and RDNA4, CPUs, and Apple Silicon, including conversion overhead, architecture limits, disabled fusions, and what to benchmark before switching formats. Topics covered: • FP8 E4M3 storage versus Q8_0 and F16 • Why FP8 and Q8_0 occupy the same broad capacity tier • Native FP8 requirements on NVIDIA H100 • OCP E4M3 support on AMD CDNA4 and RDNA4 • FNUZ encoding considerations on AMD MI300 • Why Metal blocks FP8 on Apple Silicon • M1 Pro and M2 Ultra Q8_0 benchmark results • Scaling, conversion, dequantization, and disabled fusion overhead • FP8 quality evidence and its limits for llama.cpp • Hardware-specific quant selection rules For those of you using Apple Silicon or an FP8-capable GPU, would you choose proven Q8_0 kernels or test FP8 for future backend gains? Include your hardware and generation rate. Every number in this video comes from the sources linked below. Corrections and your own measurements are welcome in the comments. Subscribe to Tensor Garage for practical local AI, GPU and open-model breakdowns without the marketing fluff. Research and pricing were checked in October 2026. Hardware prices, availability and runtime support change quickly, so verify current details before purchasing. Narration is AI-generated using an authorized voice reference. Chapters: 00:00 - What Eight Bits Save 01:33 - Which Hardware Accelerates It 02:50 - Backend Beats Bit Width 03:51 - Where Speed Disappears 04:44 - Quality Needs Your Workload 06:07 - What Works First Sources: [1] Release b12000 · ggml-org/llama.cpp — https://github.com/ggml-org/llama.cpp... [2] Quant: OCP FP8 E4M3 support by ORippler · Pull Request #27512 · ggml-org/llama.cpp — https://github.com/ggml-org/llama.cpp... [3] 15.18. __nv_fp8_e4m3 — https://docs.nvidia.com/cuda/cuda-mat... [4] NVIDIA Hopper Architecture In-Depth | NVIDIA Technical Blog — https://developer.nvidia.com/blog/nvi... [5] Data types and precision support — https://rocm.docs.amd.com/en/latest/r... [6] Performance of llama.cpp on Apple Silicon M-series · ggml-org llama.cpp · Discussion #4167 — https://github.com/ggml-org/llama.cpp... [7] FP8 Formats for Deep Learning — https://arxiv.org/abs/2209.05433 #llamacpp #FP8 #GGUF #LocalLLM #Quantization