I Built a Kubernetes Monitoring Stack — And Breaking It Was the Best Part

I didn't build this project to add a line to my resume. I built it because I kept reading about Prometheus and Grafana, nodding along like I understood it, and then freezing when someone asked me "so how does Prometheus actually discover your pods?" I didn't know. Not really. So I decided to stop reading and start breaking things. What I built A complete observability pipeline from scratch: A Python Flask app with custom Prometheus metrics Deployed on Kubernetes with 3 replicas Scraped by Prometheus via ServiceMonitor Visualized in Grafana with PromQL dashboards Load tested with real traffic using hey The repo is here if you want to follow along: 👉 github.com/adil-khan-723/k8s-observability-stack But the code isn't the interesting part. What I learned by watching it fail is. Mistake #1 — I used raw counters in Grafana and wondered why nothing made sense First dashboard. I added http_requests_total as a panel. The number just kept climbing. 1000. 5000. 23000. I stared at it thinking "ok