5 robots.txt Mistakes That Make Your Site Invisible to AI Search

AI search engines like ChatGPT, Perplexity, and Claude now answer millions of queries daily. But here's the thing — they can only cite your site if their crawlers can actually reach it. I've scanned thousands of sites and these are the 5 most common robots.txt mistakes I see: 1. Blanket Disallow for unknown user agents Many sites default to blocking all unrecognized bots. Problem: GPTBot, ClaudeBot, and PerplexityBot are relatively new. If your robots.txt has a catch-all block, you're invisible to AI search. 2. Blocking /api/ paths that contain public content Some sites serve blog content through API routes. Blocking /api/ might seem like good security practice, but if your content lives there, AI crawlers can't see it. 3. No explicit Allow for AI crawlers Even if you don't block them, explicitly allowing AI bots signals that you welcome their indexing. Add these to your robots.txt: User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent