Building a Self-Healing Web Scraping Pipeline with n8n and Gemini AI

Web scraping breaks. Pages redesign, HTML structures change, and your regex stops matching. I built a pipeline that uses AI to handle format changes automatically. The Architecture I track data from 33+ websites that update their content frequently. The goal: detect changes within 6 hours and update my database automatically. n8n (scheduler) → Firecrawl (scraper) → Gemini AI (parser) → SQLite (storage) ↓ ↓ Error handler Telegram notification Why n8n Over Custom Code I initially built this as a Node.js cron job. It worked, but: Debugging required reading logs line by line Error handling was ad-hoc ( try/catch everywhere) Adding a new source meant modifying code and redeploying n8n gives you: Visual workflow editor (see errors at a glance) Built-in retry logic per node Webhook triggers for manual re-runs Credential management (no API keys in code) Step 1: Firecrawl for JS-Heavy Sites Many modern websites load content via client-side JavaScript. A simple fetch() returns an empty shell. Fi