How to Run Local AI on Linux with llama cpp | Deep Learning From Scratch

Run a real language model on your own Linux machine: install llama.cpp, serve an OpenAI-compatible API, and secure it, with the math of quantization explained. Welcome to Keduka AI School’s Deep Learning From Scratch. This lesson runs a real, trained language model on your own Linux machine with llama.cpp: no account, no bill per message, and no data leaving the room. You install the tool with one command, chat with a GGUF model in the terminal, look inside the file to see how quantization stores each weight in about four and a half bits, serve the model over HTTP with an OpenAI-compatible API, call it from curl and Python, tune the context, the KV cache and the GPU, send it images, lock it down with an API key and TLS, and deploy it for production. Every command shown is real, and every number is checked. Continue learning: https://keduka.com/ Watch the full playlist:    • Deep Learning From Scratch   IN THIS LESSON • Install llama.cpp with one command, check it, and chat with a GGUF model in the terminal (-hf fetches it by name, -m loads a local file) • Inside a GGUF file: Q4_0 cuts the weights into blocks of 32, keeps one scale d per block and a 4-bit code q per weight, and reads each one back as w ≈ d · (q − 8) • Bits per weight: (32 · 4 + 16) ÷ 32 = 4.5 for Q4_0 and 8.5 for Q8_0, so 8 billion weights take 16 GB at F16 and about 4.5 GB at Q4_0 • llama serve: the OpenAI-compatible HTTP server, its port, 127.0.0.1 versus 0.0.0.0, and the health check (503 while loading, 200 when ready) • Chat requests with curl: system and user messages, streaming token by token, and the native /completion endpoint • The same server from Python with the OpenAI library: only the base URL changes • Context and memory: -c, the KV cache that grows with the context, and GPU offload with -ngl (default auto) • Locking the door: an API key, a key file instead of the command line, and TLS in front • Vision models: the mmproj projector, image patches as tokens, and why a photo goes in a file (131,072 × ¾ = 98,304 bytes) • Production: a pinned model, systemd or a container on 127.0.0.1, nginx with TLS and rate limits, and -c 32768 -np 4 giving 8,192 tokens per slot • Two exercises, and the common mistakes with the reason each one fails CHAPTERS 00:00 Your own AI, on your own machine 01:16 Install and first run 02:31 Inside a GGUF file 05:17 Serve it over HTTP 06:35 Who can reach your server? 07:55 Talking to the model 09:47 From Python 10:43 Context, memory and the GPU 13:12 Locking the door 14:11 Teaching the model to see 16:10 A practical server, and troubleshooting 17:07 Running it in production 20:09 Try it yourself 21:05 Common mistakes 22:06 Conclusion TRY IT YOURSELF (pause and work it out first) 1. A model has 7 billion weights, quantized as Q4_0 at 4.5 bits per weight. Roughly how many GB does it need? 2. You start a server with -c 24576 -np 3. How many tokens does each conversation (each slot) get? Answers: 1. 7 × 10⁹ × 4.5 ÷ 8 = 3.9375 × 10⁹ bytes ≈ 3.9 GB. 2. 24,576 ÷ 3 = 8,192 tokens per slot. COMMANDS IN THIS LESSON 1. Install curl -LsSf https://llama.app/install.sh | sh 2. Check installation llama cli --version 3. Test a model interactively llama cli -hf ggml-org/gemma-4-E4B-it-GGUF:Q4_0 4. Serve it on port 9000 llama serve \ -hf ggml-org/gemma-4-E4B-it-GGUF:Q4_0 \ --port 9000 5. In another terminal, check the server curl http://localhost:9000/health 6. Send an OpenAI-compatible request curl http://localhost:9000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "messages": [ { "role": "user", "content": "Hello from Linux!" } ] }' Subscribe to Keduka AI School for visual lessons that build your understanding of artificial intelligence, one idea at a time. #LlamaCpp #LocalAI #Linux #GGUF #KedukaAISchool #AI #MachineLearning #AIEducation