The Publisher's Dilemma: Block, Tolerate, or Cooperate

If you run a website with meaningful traffic, you have already encountered this problem, whether you know it or not. AI agents are crawling your pages. They are consuming your content. And you are getting almost nothing back. The numbers are stark. Cloudflare reported in August 2025 that Anthropic's crawlers fetch approximately 38,000 pages for every single page visit they refer back to publishers. OpenAI's ratio is better but still lopsided. Perplexity's ratio actually got worse over the course of 2025, with crawling volume increasing while referral traffic decreased. Meanwhile, Google referrals to news sites have been declining since February 2025, coinciding with the expansion of AI Overviews. The Pew Research Center found that users are less likely to click through to a source when Google displays an AI summary at the top of search results. The traffic pipeline that has sustained the web publishing model for two decades is weakening. And the new consumers that are replacing human v