How to Set Up llms.txt and robots.txt for AI Crawlers on WordPress (2026 Guide)

If AI crawlers cannot reach your WordPress site, your content will never appear in ChatGPT or Perplexity answers. Here is the exact setup, file by file. I manage WordPress sites for EdTech brands in India. A few months back I noticed something that made no sense: pages ranking well on Google were getting zero citations on Perplexity. Same content, same domain, completely invisible to AI search. The problem was not the content. It was three technical things I had never thought to check. Here is exactly what I fixed, with the actual code. ## Step 1: Fix your robots.txt for AI crawlers Most WordPress robots.txt files either block AI crawlers explicitly or do not mention them at all. Neither is ideal. Open yourdomain.com/robots.txt in your browser and check what is there. If you see any of these, AI crawlers are blocked: User - agent : GPTBot Disallow : / User - agent : PerplexityBot Disallow : / User - agent : ClaudeBot Disallow : / To fix this, you need to edit your robots.txt. On WordPr