Перейти к содержимому

Every Way to Get 256 GB of VRAM for Local AI at Full Context

Kai

0:00 / 0:00

Every Way to Get 256 GB of VRAM for Local AI at Full Context

14 411 просмотров · 7 ч назад
Kai
63,6 тыс. подписчиков
14 411 просмотров · 7 ч назад
256GB of VRAM runs DeepSeek V4.1 Flash for local AI, but at about 3 bits, not the full model. I priced every way to get there: Mac Studio M5 Ultra, DGX Spark vs Mac Studio, Strix Halo, an RTX 3090 + DDR4 server, RTX 5090 / RTX PRO 6000 piles, and old V100 and MI50 servers. The number on the box tells you what fits. The bits tell you what you're actually running. DeepSeek V4.1 Flash is 552B parameters plus a 196B Engram lookup table (about 510GB to download), and the Engram part can stay on your SSD. What's left needs roughly 165–200GB at 2-bit, 234–262GB at 3-bit and about 298GB at 4-bit. That makes a 256GB machine the 3-bit tier. Using Unsloth's measurements, a 2-bit build picks a different next token about once every 11 tokens, 3-bit once every 20 and 4-bit once every 67. So this video prices the fourth bit on every route. It runs from $798 (two more DDR4 sticks) to $64,000 (four RTX PRO 6000s), for the same model and the same bit. What's covered: • Mac Studio M5 Ultra 256GB: $9,499 (64-core GPU) or $10,799 (80-core). macOS gives the GPU only about 192GB by default, which is a 2-bit DeepSeek. One terminal command raises it to 248GB for 3-bit, and it resets on reboot. Ship times run 13–17 weeks. • Two NVIDIA DGX Sparks: $13,900 for about 260GB usable. Prompt reading runs around 2,100 tok/s, and writing is 31 tok/s, or 87–131 with DeepSeek's draft head. • Two Strix Halo mini PCs (Framework Desktop, Minisforum MS-S1 MAX, GMKtec EVO-X2): the cheapest new pair. On Windows they land on 2-bit, and USB4 measured about 9.2 Gbit/s in practice. Plus the 192GB Gorgon Halo. • One RTX 3090 + 256GB of DDR4: DeepSeek V4.1 Flash at 26 tok/s with a 262K context, for about $6,000. DDR4 server RAM is $6.23/GB against $29.37/GB for DDR5. • Eight 32GB cards (R9700, Arc Pro B70, RTX 5090) and why the pile has no upgrade path. • V100 and MI50 servers: the cheapest fast memory, as CUDA 13 and vLLM 0.20 drop support. • Full context at 256GB: the KV cache is about 890 bytes per token, but a 1M-token prompt still takes 10–29 minutes to read. • Local LLM vs the API: when the DeepSeek API ($0.60 per million output tokens off-peak) wins, and the one buyer it doesn't fit. Speeds are individual owners' and developers' measurements, not averages. Prices were checked October 28–29, 2026, and several of them move weekly, so check again before you buy. CHAPTERS 00:00 256GB of VRAM for local AI 02:04 Why 256GB? The $22,299 Mac Studio math 03:52 Bits per weight: 2-bit vs 3-bit vs 4-bit 06:13 Mac Studio M5 Ultra 256GB 08:49 DGX Spark x2 (DGX Spark vs Mac Studio) 10:47 Strix Halo mini PCs + Gorgon Halo 13:27 RTX 3090 + 256GB DDR4 (the cheap route) 16:33 Eight-GPU pile: R9700, B70, RTX 5090 18:13 V100 and MI50 servers from eBay 20:05 Full context: what a 1M-token prompt costs 21:56 Local vs the DeepSeek API 23:32 The price of the fourth bit 24:48 Which 256GB route to buy SOURCES • Mac Studio M5 Ultra 512GB pricing (MacRumors): https://www.macrumors.com/2026/10/28/... • Mac Studio tech specs: https://www.apple.com/mac-studio/specs/ • Unsloth DeepSeek V4.1 Flash quant table: https://unsloth.ai/docs/models/deepse... • DwarfStar V4 vs V4.1 Flash hardware guide: https://dwarfstar.sh/blog/deepseek-v4... • TensorFold, DeepSeek V4.1 Flash on one 256GB M5 Ultra: https://github.com/BobClawblaw/deepse... • mlx-serve M5 Ultra measurements: https://github.com/ddalcu/mlx-serve/i... • MiaAI Lab, DeepSeek V4.1 Flash on two DGX Sparks: https://github.com/MiaAI-Lab/DeepSeek... • DeepSeek V4.1 Flash on two Strix Halo nodes: https://github.com/hyukudan/strix-hal... • Server RAM price tracker: https://datacenterdisk.com/news/memor... • CUDA 13 release notes (Volta dropped): https://docs.nvidia.com/cuda/archive/... • DeepSeek API pricing: https://api-docs.deepseek.com/quick_s... • r/DGX_Spark, two Sparks EXL3 2.9-bit:   / 1xha3qo   • r/StrixHalo, 2x Strix vs Strix + R9700:   / 1xfp73u   • r/LocalLLaMA, ISTA 2-bit "too good to be true?":   / 1xes5x0   Photo credits (Wikimedia Commons): DGX Spark by Daniel Lu (CC BY-SA 4.0); RTX 3090 and RTX 5090 (CC BY 4.0); Arc Pro B70 by Biggo Finance (CC BY-SA 4.0); DDR4 RDIMM by Sebastiaan Broekhoven (CC BY-SA 4.0). Mac Studio images: Apple Newsroom. If you've built one of these, especially a DDR4 build pushed to 4-bit, drop your numbers in the comments. #LocalAI #DeepSeek #MacStudio