Web Scraping Pipelines for AI Agents: Cut Token Waste

Build the Pipeline First, Optimize the Prompt Second Fetch rendered HTML, strip the noise, convert to markdown or typed JSON, then pass clean content to your agent. Done in that order, this pipeline cuts per-page token costs by 10–50x and eliminates hallucinations caused by LLMs trying to parse navigation menus. The architecture is straightforward. The implementation details—waiting for DOM mutations, isolating content zones, handling JS-heavy SPAs at scale—are where most pipelines break down. This post covers each step with working Python code. The Token Math A typical SaaS pricing page: Raw HTML (scripts, styles, nav, footer included): ~110KB → ~27,000 tokens Article body as clean markdown: ~4KB → ~1,000 tokens At GPT-4o pricing, scraping 50,000 pages daily costs: Raw HTML pipeline: ~$3,375/day Clean extraction pipeline: ~$125/day That delta compounds. At 50,000 pages/day, you're paying for 1.35 billion unnecessary tokens. That's not a prompt engineering problem—it's a data pipeline