
The Internet Had a "Payment Required" Button for 29 Years. Cloudflare Just Turned It On.
HTTP status code 402 has existed since 1997. The original HTTP/1.1 specification reserved it "for future use." For nearly three decades, no one found a use for it. Last week, Cloudflare and Stack Overflow changed that. The system is called pay-per-crawl. When an AI crawler requests a page from a participating website, the server responds with HTTP 402 — Payment Required — along with a price header. The crawler either pays or leaves. No negotiation. No "we'll give you exposure." Cash or nothing. Cloudflare handles roughly one in five websites on the internet. That's not a niche experiment. That's infrastructure-level monetization of AI training data, deployed at the same scale the crawlers themselves operate at. How the Tollbooth Works The technical implementation is straightforward. Publishers set a flat per-request price for their domain. Cloudflare's Web Application Firewall identifies AI crawlers, categorizes them, and serves the 402 response instead of the content. Crawlers authent
Continue reading on Dev.to Webdev
Opens in a new tab




