Cloudflare will block AI bots from crawling websites by default for new customers, and broker pay-per-crawl deals between its customers and bot operators.
Cloudflare will block AI crawlers from accessing new customers’ websites without permission starting July 1 and is testing a way to make AI pay for the data it gathers.
Furthermore, website owners can now decide who crawls their sites, and for what purpose, and AI companies can reveal via Cloudflare whether the data they gather will be used for training, inference, or search, to help owners decide whether to allow the crawl.
The company began enabling its customers to choose to block AI crawlers in July 2024. Since then, it said, over one million customers have opted in.
“For decades, the Internet has operated on a simple exchange: search engines index content and direct users back to original websites, generating traffic and ad revenue for websites of all sizes. This cycle rewards creators that produce quality content with money and a following, while helping users discover new and interesting information,” Cloudflare said in its announcement. “That model is now broken. AI crawlers collect content like text, articles, and images to generate answers, without sending visitors to the original source — depriving content creators of revenue, and the satisfaction of knowing someone is reading their content. If the incentive to create original, quality content disappears, society ends up losing, and the future of the Internet is at risk.”
Hellbound and Down 