Project 3: What Happens When Something Breaks at 3am? Full SRE Observability on Kubernetes

Two builds in, this cluster was proven to deploy and proven to autoscale under load. Neither one answers what happens when something breaks and nobody's watching. This build adds the other half — on local minikube, no AWS account required. Promtail tails every pod's stdout straight off disk and ships it to Loki. Every service sends OpenTelemetry traces to a collector, which forwards the full request waterfall to Tempo — and derives real request-rate, error-rate, and duration metrics directly from those same spans, because this application has never exposed that data natively. Prometheus evaluates 4 alert rules against that data and fires into Alertmanager. Then I force-delete a pod live and watch the alert actually fire in Alertmanager's own UI — not simulated, not inferred. A real bug shipped on the first install attempt too: Promtail discovered every pod correctly but never shipped a single log line, silently, because of one missing relabel config. It's on camera, not edited out. What's inside: ✅ Centralized logging with Loki + Promtail — no per-service logging agent, no app code changed ✅ Distributed tracing with Grafana Tempo — a real multi-service trace, frontend → checkoutservice → paymentservice ✅ RED metrics (rate/errors/duration) derived from trace spans, not a native app metric ✅ 4 Prometheus alert rules: HighRequestLatency, HighErrorRate, PodCrashLooping, PodNotReady ✅ A real pod force-deleted on camera, alert confirmed firing in Alertmanager's UI ✅ The exact Promtail bug that shipped broken — and the one-line fix Same 11 services, same Helm chart, not one line of it touched — only the target changes, from AWS EKS to local minikube, because this layer doesn't need a cloud account to prove. 📌 Third build in this series — same repo evolves across every video. Code at THIS video's exact state (tag: v3.0-sre-observability): https://github.com/ThinkWithOps/think... 🔗 RESOURCES: GitHub Code (this video's state): https://github.com/ThinkWithOps/think... GitHub Code (latest/main): https://github.com/ThinkWithOps/think... 🛠️ Tech Stack: Loki + Promtail — centralized logging, DaemonSet, no per-service agent Grafana Tempo — distributed tracing across services OpenTelemetry Collector + spanmetrics connector — real RED metrics derived from spans Prometheus + Alertmanager — 4 alert rules, severity-based routing Grafana — Prometheus + Loki + Tempo in one pane of glass Minikube — local cluster, zero AWS cost for this layer #Kubernetes #Observability #SRE #Grafana #Prometheus #devops 📺 Full Series — ThinkWithOps: Online Boutique DevOps V1 — EKS Deployment (Terraform, ECR, GitHub Actions OIDC CI/CD) ▶️    • Project 1: I Deployed Google's Online Bout...   🏷️ https://github.com/ThinkWithOps/think... V2 — Load Testing + Autoscaling (HPA, Karpenter, k6) ▶️    • Project 2: Can This Handle 100,000 Users? ...   🏷️ https://github.com/ThinkWithOps/think... V3 — SRE / Observability + Incident Response (Loki, Tempo, Prometheus, Alertmanager) ▶️    • Project 3: What Happens When Something Bre...   🏷️ https://github.com/ThinkWithOps/think... V4 — GitOps with ArgoCD ▶️    • Project 4:  I Removed the Last Manual Depl...   🏷️ https://github.com/ThinkWithOps/think... 🔗 Full repo: https://github.com/ThinkWithOps/think...