Production Optimization: Inference Cost and Performance Control

1. Introduction: The Dual Pain Points of Inference Cost and Performance in Customer Service This is Part 7 of the series 8 Weeks from Zero to One: Full-Stack Engineering Practice for a Production-Grade LLM Customer Service System . In the first six parts, we completed the full-pipeline closure of the system's core capabilities. However, in enterprise-grade production deployments, runaway costs and performance instability are more operationally fatal than incomplete features. Our real production logs and load-test data from the e-commerce customer service system revealed the following: Over 70% of user queries are repetitive or semantically similar (e.g., "What is the return process?", "How do I return an item?", "What steps do I need to follow to return something?"). Calling the LLM indiscriminately for every request wastes significant resources. Before optimization, all requests were routed uniformly to the DeepSeek-R1:14B private deployment. Monthly inference costs (calculated across