Building a Real-Time Voice AI Agent for Restaurants with OpenAI and Twilio

I've been building voice AI systems for small businesses, and I wanted to share the architecture behind a real-time voice agent designed specifically for restaurants. This post walks through how we connected OpenAI's Realtime API with Twilio to create an AI that answers phone calls, handles reservations, and takes orders — all without human intervention. The Problem Restaurants miss 15-25% of incoming calls. During peak hours, that number can hit 40%. Every missed call is a lost reservation, a missed takeout order, or a frustrated customer who calls your competitor instead. Hiring a dedicated phone person costs $2,500-4,000/month and only covers one shift. Answering services cost $500-1,500/month and mostly just take messages. We wanted to build something that could actually handle the calls — book reservations, take orders, answer questions — 24/7. Architecture Overview The system uses a triage agent pattern with specialized sub-agents: Incoming Call (Twilio) → WebSocket Connection →