How I Built a Truly Private AI: The Full Software Setup

This is the second and final video on my 3x RTX 3090 beastly local AI server. I promised you a complete walkthrough of the AI software stack, start to finish, and friends, I'm here to deliver! In Part 2 of my local AI server build, I break down the entire self-hosted AI stack running on this box: Ollama, Open WebUI, Qdrant, Apache Tika, SearXNG, Infinity, and Open Terminal. I cover what each one does, why I picked it, and provide example scenarios of how they all work together in one Docker Compose file. I also cover the most fundamental lesson I learned while building this system: AI isn't magic; it's just software. The "it just knows" experience you get from ChatGPT or Claude is really a model calling a bunch of tools behind the scenes. So I walk through three real scenarios step by step: searching the web for current info, chatting with your own documents using RAG with hybrid search and reranking, and having the model write and run its own script in a sandboxed terminal. Then we go through the full compose file block by block, so you can deploy the same stack on your own hardware. The result is seven containers and one compose file on a single box. It can search the web, read your documents, and write and run its own code, and none of your prompts go to a cloud model. 🔗 Get the Docker Compose file: https://2guystek.tv/homelab/ai/infras... 🔗 Part 1, the hardware build:    • Three RTX 3090s in One Server - Homelab AI...   Root-level Patreon and YouTube members get access to this system. Come build with us: Patreon:   / 2guystek   YouTube Membership:    / @2guystek   *GET SOCIAL AND MORE WITH US HERE!* Get help with your Homelab, ask questions, and chat with us! 🎮   / discord   *Check out HomeLab Gear for all your homelab needs!* 💿 https://homelabgear.shop/ Subscribe and follow us on all the socials, would ya? 📸   / 2guystek   💻   / 2guystek   Find all things 2GT on our website! 🌍 https://2guystek.tv/ More of a podcast kinda person? Check out our Podcast here: 🎙️ https://www.buzzsprout.com/1852562 ** Timestamps ** 00:00 | Intro 01:52 | AI isn't magic, it's tools 03:47 | Foundation: Linux and Docker 05:13 | Model runner: Ollama 06:13 | User interface: Open WebUI 07:44 | Vector database: Qdrant 08:50 | Document extraction: Apache Tika 09:34 | Web search: SearXNG 10:31 | Reranking: Infinity 11:27 | Sandboxed CLI: Open Terminal 12:43 | How it all connects 13:15 | Three scenarios to show how it works 18:06 | Why sandboxing matters 18:48 | Docker Compose walkthrough 19:23 | Open WebUI config 21:05 | Qdrant config 21:31 | Tika config 21:45 | SearXNG config 22:20 | Ollama config 23:21 | Open Terminal config 23:50 | Infinity config 24:30 | Placeholders and spinning it up 25:04 | Wrap up: Digital sovereignty