Building a Multi-Language Voice AI Agent: Automatic Language Detection for Restaurant Phone Systems

At RingFoods , we build AI voice agents that answer restaurant phone calls. One of the hardest engineering challenges we faced was making the system work seamlessly across multiple languages without requiring the caller to press 1 for English or 2 for Spanish. Here is how we approached automatic language detection in a real-time voice pipeline, and what we learned along the way. Why Language Matters for Restaurant Phone Systems Restaurants in cities like Miami, Los Angeles, Toronto, and New York serve communities where English is not always the first language. A Thai restaurant in LA might get calls in Thai, Mandarin, Spanish, and English on the same afternoon. A pho shop in Montreal fields calls in French, Vietnamese, and English. Traditional IVR menus that ask callers to select a language add friction. Callers hang up. The whole point of an AI phone agent is to reduce friction, not add it. The Detection Pipeline Our approach uses a three-stage detection system: Stage 1: First Utteran