Building "Vexa": The Journey of Crafting a Voice-Controlled Local AI Agent from Scratch

In today's world of massive cloud-API dependencies, I decided to challenge myself: Could I build a fully functional, intelligent, voice-controlled AI agent entirely natively, running on my localized hardware without relying on paid OpenAI or Anthropic endpoints? The answer is yes! . Meet Vexa , an industry-grade program file builder that listens to your voice, understands your intent, and physically writes code on your machine while keeping your data 100% private. Here is a breakdown of how the architecture came together, my model tiering choices, and the immense engineering hurdles overcome along the way. The System Architecture The architecture is split into a robust FastAPI backend and a visually stunning React frontend. The Ears (Speech-to-Text): When a user speaks into the microphone (or uploads an audio file), the React frontend bundles the blob and shoots it to the FastAPI backend. Here, a locally hosted HuggingFace Whisper (openai/whisper-tiny) pipeline kicks in. It processes t