Building a Sovereign AI Stack: From Zero to POC

In an era where data privacy is paramount, relying on cloud-based AI providers isn't always an option. Whether for compliance, security, or just peace of mind, running a Sovereign AI Stack —a completely local, self-controlled AI infrastructure—is the ultimate goal for many organizations. Today, we built a Proof of Concept (POC) for such a stack, leveraging open-source tools to create a private, observable, and searchable AI environment. Here is our journey. The Architecture Our stack consists of three core components, orchestrated by a Node.js application: AI Server : A local LLM running on llama.cpp (serving OpenAI-compatible API). This provides the intelligence without data leaving the network. Search Engine : Manticore Search (running in Docker). We chose Manticore for its lightweight footprint and powerful full-text search capabilities, essential for RAG (Retrieval-Augmented Generation). Observability : AI Observer (running in Docker). You can't manage what you can't measure. This