Как AI-модель на 74 млрд параметров помещается на RTX 3090 (24 ГБ)

NVIDIA's new 74B AI model runs on a 6-year-old RTX 3090 with just 24GB of VRAM. One independent test reported 86.6 tokens/sec, and even managed to load a 256K context configuration onto the same GPU. But there's a catch. NVIDIA's own recommended accelerated setups need more memory. So how does a 74-billion-parameter model fit into 24GB? And is it actually better than smaller local LLMs like Qwen 3.8 27B? In this video, I investigate NVIDIA Nemotron 3.5 Bolt 74B-A9B, its extreme quantization, RTX 3090 community benchmarks, long-context performance, and whether it makes sense for local coding agents. *RTX 3090 BENCHMARK RESULTS* Nemotron 3.5 Bolt: • 74B total parameters / 9B active • Native 2-bit quantization • Approximately 22GB main GGUF model file • Up to 256K configured context • 21.8GB reported peak VRAM at 256K context Reported generation speeds: • Empty context: 86.6 tok/s • 64K context: 65.2 tok/s • 128K context: 52.2 tok/s In the same tester's comparison, Qwen 3.8 27B reached 42.3, 30.0, and 23.3 tok/s at those respective context depths. These are community measurements using specific model files, quantization, cache settings, and llama.cpp configurations—not universal performance guarantees. *WHAT'S THE CATCH?* NVIDIA's 24GB model uses a hybrid Mamba-Transformer architecture, Mixture of Experts, and aggressive quantization to reduce memory requirements. But the complete accelerated deployment can require additional memory for the vision component, speculative decoding, runtime buffers, and context. There's also a difference between fitting a 256K context configuration into VRAM and accurately reasoning across 256K tokens. And for coding agents, generation speed isn't everything. Prompt processing, tool use, and total task completion time matter too. We look at NVIDIA's own agent benchmarks and the trade-offs you should understand before replacing your current local model. *CHAPTERS* 00:00 A 74B AI model on one RTX 3090 01:20 How 74B fits inside 24GB 02:32 Mixture of Experts and 2-bit quantization 04:35 The 256K context test 06:02 Nemotron Bolt vs Qwen 27B 06:56 Why the KV cache changes everything 08:43 Generation speed at longer context 09:43 Prefill vs decoding 10:50 The catch in NVIDIA's requirements 12:29 Why inference software matters 13:22 Coding and agent benchmarks 14:18 Faster tokens vs faster agent tasks 16:11 Should you run Bolt on an RTX 3090? 17:56 Final verdict *SOURCES* NVIDIA's official technical report: https://research.nvidia.com/labs/nemo... Official Nemotron 3.5 Bolt model card: https://huggingface.co/nvidia/NVIDIA-... Official GGUF download and deployment details: https://huggingface.co/nvidia/NVIDIA-... Independent RTX 3090 benchmark and 256K context comparison:   / round_2_nemotron_35_bolt_fits_its_full_256k   NVIDIA speculative decoding component: https://huggingface.co/nvidia/NVIDIA-... llama.cpp inference engine: https://github.com/ggml-org/llama.cpp Benchmarks are from NVIDIA and independent community reports. Cian Mag has not personally tested these GPUs. The 256K result demonstrates configuration and memory fit, not validated full-context reasoning. Would you try Nemotron Bolt on your RTX 3090, or stick with a smaller model? Subscribe for local AI hardware comparisons, benchmarks, and ways to run powerful AI without overspending. #LocalAI #RTX3090 #Nemotron