How I Built a Voice-Controlled Local AI Agent from Scratch

Introduction When I first read the assignment brief — "build a voice-controlled AI agent that runs locally" — it sounded simple. Record audio, transcribe it, do something with it. But as I started building, I realized there were a dozen small problems hiding inside that one big one. This article walks through the architecture I chose, the models I used, and the real challenges I faced along the way. What the System Does The agent accepts voice input (microphone or uploaded audio file), converts it to text, classifies the user's intent using an LLM, and then executes the right action on your local machine — creating files, generating code, summarizing text, or having a general conversation. The entire pipeline is displayed in a clean Streamlit UI. Architecture Overview The system has four layers: Audio Input — Streamlit's built-in st.audio_input() handles browser microphone recording. File upload supports .wav, .mp3, and .m4a. Speech-to-Text (STT) — I used Groq's hosted Whisper API (whi