Web Scraping Pipeline for RAG: Clean Data for LLMs

Web Scraping Pipeline for RAG: Feed Clean Data into Your LLM Without Token Waste Raw HTML is poison for RAG. A typical news article page is 45,000 characters—roughly 11,000 tokens. The actual article is 800 words, or about 1,100 tokens. You are paying 10× to embed navigation menus, cookie banners, footer links, and inline scripts that actively dilute your embeddings and degrade retrieval quality. The fix is a five-stage pipeline: reliable fetch → content extraction → normalization → semantic chunking → embed and index. Each stage has a single responsibility. Each failure is isolated and debuggable. This post walks through a production implementation in Python. Pipeline Architecture Stage 1: Reliable Fetching The hardest part of scraping at scale is not parsing—it is getting the HTML. Bot detection blocks requests . JavaScript-rendered SPAs return skeleton HTML to static fetches. IP ranges accumulate blocks. AlterLab's scraping API handles this in a single POST: rotating residential pro