The model is only 20% of the work — building Gmail Smart Compose taught me this

Spent a few weeks reverse-engineering Gmail Smart Compose and building a working version. Biggest surprise: the hard engineering has nothing to do with ML. - A 200ms debounce saves 98% of model costs - Last-5-words cache key gets 50-70% Redis hit rate - Kafka is the only real answer to the 9AM traffic spike - Acceptance rate (did user press Tab?) matters more than perplexity Built across 3 tiers in Java + Spring AI, from laptop to Kubernetes. Repo and Explainatelry video details in comments submitted by /u/Money_Cow4556 [link] [comments]