Building a Real-Time Multimodal AI Communication Coach

Most AI tools on the market today are fundamentally text-based. Even when they "listen" to audio, they act on static transcripts after the fact. But human communication doesn't happen after the fact. It happens in the moment—in our tone of voice, our pacing, our posture, and our eye contact. “Vision agents don’t just see pixels — they interpret intent, context, and meaning in motion.” I wanted to build an AI that could actually coach people on how they communicate, rather than just summarizing what they said. I wanted an AI that could see if you were slouching, hear if you were speaking too fast, and interrupt you politely with actionable advice. The result is Visions, an open-source real-time AI Communication Coach. "Words are only part of the message. Your posture, your pace, your pauses — they say everything the words don't." In this post, I want to pull back the curtain on the architecture behind Visions. We'll look at how we orchestrated Gemini Realtime API , GetStream , Deepgram