Self-Hosted AI on Linux: A DevOps Home Lab Guide

Audience: Intermediate DevOps/Systems Engineers | Series: Part 1 of 4 Fun Part:- Chat with you own LLM without worrying about token expiration. Section 1 — Introduction 1.1 The 5 Layers of AI Ecosystem Layer Role Dev / Home Lab Production 5 Applications Simple chatbot scripts RAG pipelines, Agents, Chatbots 4 Frameworks LangChain, LlamaIndex LangChain, LlamaIndex, LiteLLM 3 Model Serving Ollama vLLM, TGI, Triton 2 Models phi3:mini, gemma:2b Mistral 7B, Llama 3 70B 1 Infrastructure VirtualBox VM,Mac Mini M-series, Local hardware AWS/GCP/Azure, GPU servers This post covers Layers 1, 2 and 3. Layers 4 and 5 will be covered in posts ahead. 1.2 What This Post Covers Setting up Ubuntu Server VM on VirtualBox. The server running the LLM. Installing and configuring Ollama as a systemd service. Ollama is a program which helps to manage LLM model. LLM model being used is phi which is superlight for homelab setup. It is similar to sonnet or gemini but on much smaller scale. Automating the entire