I Built an AI-Ready Content Crawler for RAG Pipelines (Open Source)

If you've built a RAG pipeline, you know the pain: you need clean text from websites, but what you get is a soup of HTML tags, navigation menus, cookie banners, and ads. I got tired of writing the same extraction + chunking logic for every project, so I built AI Content Crawler — an open-source tool that turns any website into clean markdown with smart chunking, ready for embeddings and vector databases. GitHub: kai-agent-free/ai-content-crawler The Problem Typical web scraping gives you this: <div class= "nav" > ... </div> <div class= "sidebar" > ... </div> <article> <p> The actual content you want... </p> </article> <footer> ... </footer> <script> tracking (); </script> For RAG, you need just the article content , converted to clean text, split into chunks with overlap for retrieval. Most tools make you handle extraction and chunking separately. This crawler does both in one step. The Solution AI Content Crawler is built on Crawlee and uses Mozilla's Readability (the same algorithm b