How I Built a RAG-Powered AI Chatbot Builder with Next.js, Supabase, and pgvector

Every website has docs, FAQs, and knowledge buried across dozens of pages. I wanted to turn all of that into an AI chatbot that actually knows the website — not a generic GPT wrapper, but something grounded in real content. So I built a RAG (Retrieval-Augmented Generation) pipeline: scrape a website, chunk the text, generate vector embeddings, store them in Postgres with pgvector, and retrieve relevant context at query time. Here's how the whole thing works under the hood. The Architecture The stack: Next.js 15 (App Router) — frontend + API routes Supabase — Postgres database with pgvector extension OpenAI — embeddings ( text-embedding-3-small ) and chat completions ( gpt-4o-mini ) Cheerio — server-side HTML parsing for the scraper The flow looks like this: User enters URL → Scrape website → Chunk text → Generate embeddings → Store in pgvector ↓ Visitor asks question → Embed query → Vector similarity search → Build context → LLM generates answer Step 1: The Web Scraper The scraper star