Building a Voice-Controlled Local AI Agent: A Journey into Speech-to-Text and Tool-Use

In the era of Large Language Models (LLMs), the gap between "chatting with an AI" and "controlling your computer" is rapidly closing. Recently, I embarked on a project to build a Voice-Controlled Local AI Agent that allows users to manage their filesystem, generate code, and summarize text—all through natural speech. In this article, I'll walk you through the architecture, the high-performance models I chose, and the unique challenges I faced along the way. The Vision The goal was simple but ambitious: create a specialized agent that accepts audio input (via mic or file upload), understands the user's intent, and executes the appropriate local tool (like creating a file or writing a Python script). The Architecture The agent is built on a "Three-Step Pipeline" designed for speed and reliability: Speech-to-Text (STT) : Converting raw audio into clean, actionable text. Intent Classification : Using an LLM to "parse" the text into a structured JSON object (Intent + Arguments). Tool Execut