I ran a 125B AI Model on a 32 GB M6 Mac mini!
Suhas Rao
0:00 / 0:00
I ran a 125B AI Model on a 32 GB M6 Mac mini!
30 745 просмотров · 2 дн. назад
Suhas Rao
290 подписчиков
30 745 просмотров · 2 дн. назад
A 125B AI model, on a Mac mini with 32 GB of memory. It shouldn't fit, but it runs. Qwen3.8 Flash Next (125B parameters) streams its experts from an external SSD, and on Qwen's own benchmarks it sits level with Claude Opus 4.6. So is it actually smarter than the 27B model that fits in memory? I tested it.
In this video:
• How a mixture-of-experts model can run from an SSD (only about 6B of its 125B parameters work on each token)
• Why I use a 1 TB external USB4 SSD instead of the Mac's 256 GB internal drive
• SlotStream vs oMLX for streaming the model
• The same benchmark questions for both models, with the same settings: a 10-question quiz, MMLU-Pro and GPQA Diamond
• The trick question both models got wrong, and why web search didn't save them
• Confident is not the same as correct
• Bonus: local image generation with FLUX 3 Image Klein on the same Mac
📂 Scripts, settings and results: https://github.com/suhasaraos/mac-mini-ssd... Local LLM setup from my first video: https://github.com/suhasaraos/mac-mini-loc...
▶️ My first video, a 27B model running locally on this Mac: • I ran a 27B AI Model on a Mac mini M6… Her... ▶️ My last video, speculative decoding and photo to 3D: • I ran a 27B AI Model on a Mac mini M6… Her...
Tools used: SlotStream, oMLX, Apple MLX, Inspect (UK AI Security Institute), Qwen3.8 Flash Next, Qwen3.8 27B, FLUX 3 Image Klein, Real-ESRGAN.
Your turn: have you run a model bigger than your memory? What speed did you get, and on which drive? Tell me in the comments.
#LocalAI #MacMini #LLM #Qwen #Engineering