Qwen 3.8 27B at 67 tok/s on Two $78 Nvidia Tesla P100s

What if two Nvidia Tesla P100 GPUs from 2016, $78 each, could run Qwen 3.8 27B (Q6) locally at 67 tokens per second? ChatGPT and Claude both predicted 17-19. This is the whole story: the black screen, the power, the heat, and the exact recipe so you can copy it. If you're looking for the cheapest way to run Qwen 3.8 27B locally, or a cheap GPU for local AI that isn't an RTX 3090: this is a budget local LLM setup with two 16 GB GPUs (32 GB VRAM total), dual GPU with llama.cpp, Q6 quant, 128K context. Numbers from this video (2x Tesla P100 16 GB, Qwen3.8 27B Q6_K, speculative decoding on): 49.2 tok/s on normal writing, 66.7 tok/s on a reasoning puzzle (Q4_K_M: 67) 29.7 tok/s with speculative decoding off ~400 tok/s prompt reading, 128K context, 37 tok/s with the context full Gemma 4 26B-A4B (IQ4_XS) on ONE card: 79 tok/s RTX 3090 with the same speculative decoding: 66.4 tok/s How I measured: the 66.7 / 67 tok/s puzzle runs used reasoning ON at "Low" effort (single run each, both answers correct). 49.2 tok/s = a ~4,000-token document prompt, median of 3 runs. The 128K-context test ran with thinking off. Higher reasoning effort is slower (it thinks longer and the speculative guesses land less often). Since this video was made I've pushed it to 74 tok/s; that's the next video. THE RECIPE (exact settings) Hardware Dell Precision T5810 workstation (any board with two full-size PCIe x16 slots and room for two long cards) 2x NVIDIA Tesla P100 PCIe 16 GB (HBM2), $78 each, used A power supply strong enough for both cards (mine wasn't, so I added a second one just for card 2) The right power adapter cables for the cards (check what your power supply has before you buy) A ducted fan shroud over both cards (they have no fans; an unducted fan hit 79 °C in 35 s) Making the cards boot (older BIOS can't map 16 GB BAR1) Back up each card's firmware first: nvflash --save p100_cardN_backup.rom Switch to graphics mode: nvflash -i N --gpumode graphics (BAR1 16 GB → 256 MB). The first reboot can power-cycle three times before it shows anything. Don't touch it. The flash turns ECC off. Turn it back on: nvidia-smi -i N -e 1, then reboot. Power cap 150 W per card: nvidia-smi -pm 1 then nvidia-smi -i N -pl 150. More power made it less than 1% faster. System Ubuntu 24.04, NVIDIA driver 580 (server branch, the last one that supports Pascal), CUDA 12.9 Kernel boot line: iommu=pt. Needed if you turn on P2P (GGML_CUDA_P2P=1); without it P2P crashed both cards. Without iommu=pt, leave P2P off: it's only worth a few percent. Software: Kmic-68's P100 fork of llama.cpp https://github.com/Kmic-68/llama.cpp (branch p100-optimizations, tested at commit d90ec36a3) Built for sm_60 only, with CUDA 12.9: -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=60 -DGGML_CUDA_NCCL=OFF -DGGML_CUDA_FA_QUANTS=all (adding a second architecture switches off the Pascal tuning). See the repo's p100-docs/BUILD.md. Model Qwen3.8-27B, Q6_K GGUF that includes the MTP (next-token prediction) head. The file I ran is Qwen3.8-27B-Uncensored-Q6_K.gguf (a community finetune of the same model, same size and speed). Launch (llama-server) GGML_CUDA_P2P=1 GGML_CUDA_GRAPHS_PRE_VOLTA=3 LLAMA_SPEC_SAMPLE_TEMP=1.0 LLAMA_SPEC_DRAFT_TOPK=20 \ llama-server -m Qwen3.8-27B-Uncensored-Q6_K.gguf -ngl 99 -sm tensor -fa 1 -ctk q4_0 -ctv q4_0 \ -c 131072 -b 32768 -ub 2048 -np 1 \ --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.2 -ngld 99 -ubd 64 -ctkd q4_0 -ctvd q4_0 \ --jinja --temp 1.0 --top-k 20 --top-p 0.95 --min-p 0.0 (Drop GGML_CUDA_P2P=1 if you don't have iommu=pt. Two drafts vs four: about the same. One card at PCIe x8: ~2% slower.) What these cards are NOT for: video generation, gaming (no video outputs). CREDITS: the people who keep Pascal alive. Go give them a star. Kmic: P100 optimizations for llama.cpp, the build used here: https://github.com/Kmic-68/llama.cpp fallentree: GP100 llama.cpp fork: https://github.com/thefallentree/llama.cpp... qr0n: P100 patches: https://github.com/key-sh/p100-llama-cpp llama.cpp: https://github.com/ggml-org/llama.cpp The animation in "Made by Qwen" was written by Qwen3.8 27B itself, on these two cards. CHAPTERS 0:00 Intro: 67 tok/s on $78 GPUs 0:36 The hype: 16 of 24 benchmarks? 1:02 The problem: 8 GB of VRAM 1:23 The goal: Q6 at 30 tokens/sec 1:57 YouTube says: buy a 3090 2:23 The find: Tesla P100 for $78 2:59 The warning: "17 to 19 tokens/sec" 3:25 The rabbit hole: the P100 community 4:13 The nightmare: black screen, power, heat 6:01 Alive: the ducted shroud 6:24 The first number: speculative decoding 7:07 The logic puzzle: Q4 vs Q6 7:45 Why it's fast, 128K context, Gemma 4 8:24 What worked and what didn't 9:21 Made by Qwen 9:44 The recipe 10:49 Conclusion #LocalLLM #Qwen #llamacpp