How to Build a Self-Healing AI Agent System: Lessons from 70+ Production Bugs

Running one AI agent is straightforward. Running three or more agents in a persistent pipeline — with an orchestrator dispatching tasks, workers executing them, and a bridge syncing state — is a different animal entirely. Processes crash silently. Infinite restart loops burn through your PM2 logs. Tasks get stuck in "doing" status forever. AI-generated scripts contain eval() calls that could nuke your filesystem. This is the story of how we built AI System Guardian — a self-healing monitoring daemon — after surviving an 11-round debugging marathon that uncovered 70+ production bugs in a multi-agent AI system. Every pattern in the scanner and every health check exists because something actually broke in production. The Architecture That Kept Breaking Our system has four core components running as PM2 processes on a single Windows machine: Orchestrator — reads a backlog of tasks and dispatches them to workers Logic Executor — picks up task files and runs AI-generated Python scripts Bridg