Building an AI Synesthesia Engine with Gemini Live API and ADK

How we built MUSE, a real-time multimodal agent that translates between senses using Gemini 2.5 Flash Native Audio, ADK multi-agent orchestration, and some surprisingly tricky WebSocket plumbing. The Idea: Synesthesia as an AI Paradigm Synesthesia is a neurological condition where stimulation of one sense automatically triggers another. A synesthete might hear colors, see sounds, or taste shapes. For most people it's involuntary, poetic, and hard to explain. For an AI that processes multiple modalities simultaneously, it should be native. That realization was the seed of MUSE, the Multimodal Synesthetic Experience Engine. The premise: instead of asking an AI to describe a painting, ask it to hear the painting. Instead of transcribing a melody, ask it to see the melody. MUSE does not just process inputs and produce outputs. It performs cross-modal translation as its core function. Every visual input becomes a sonic description. Every audio input becomes a visual one. And throughout, it