I Built a $400/mo LLM Cost Monitoring System (Here's What I Learned)

Six months ago I got a $3,000 LLM bill. I had no idea where it came from. Now I have a monitoring system that tracks every call. Here's what I built. What Happened I was running a small SaaS with GPT-4. The bill came in at $3,127 for the month. I had maybe 500 paying users. That's $6/user in LLM costs. My product was $20/month. I was losing money on every power user. I had no idea what was happening because I wasn't tracking: Cost per user Cost per feature Cost per model Request volume The System I Built 1. Per-Call Logging def llm_call ( messages , model = " gpt-4o " ): start = time . time () response = openai . ChatCompletion . create ( model = model , messages = messages ) # Log everything cost = calculate_cost ( model , response . usage ) log ({ " model " : model , " prompt_tokens " : response . usage . prompt_tokens , " completion_tokens " : response . usage . completion_tokens , " cost " : cost , " latency_ms " : ( time . time () - start ) * 1000 , " user_id " : get_current_user