Market & Regulation
Cloudflare Sets New AI Crawler Policy for 2026
TechCrunch AIhigh confidence
Why it matters
- →Cloudflare's policy could reshape how AI companies access web content for training.
- →It introduces a new revenue model for publishers through 'Pay Per Use.'
- →The move addresses concerns about intellectual property and AI's impact on digital content.
Cloudflare has announced a new policy requiring AI companies to separate their web crawlers used for search from those used for AI training by September 15, 2026. This change will block mixed-use crawlers from accessing ad-hosting pages by default, unless site owners opt out. The policy is designed to protect publishers' content from being used without compensation by AI services. Cloudflare is also introducing 'Pay Per Use,' allowing publishers to charge AI companies when their content is used, not just fetched. This move could significantly impact how AI models access and use web content.
Read originalThe story around this
Earlier coverage that leads up to this article, and what followed. Lines connect each piece to the closest one after it, converging here.
What happened next
- 1Patreon Blocks AI Bots from Scraping Content
TechCrunch AI · July 17, 2026 · Related
- 2Cloudflare unveils Kitesurf browser for AI agents
TechCrunch AI · August 7, 2026 · Related
More from TechCrunch AI
© TechCrunch AIMarket & Regulationbusiness
Crusoe cancels $1.25B Boom turbine deal
The collapse of Crusoe’s $1.25 billion order for Boom Supersonic’s stationary turbines exposes the fragility of AI infrastructure financing. While Crusoe raised $3.9 billion, it pivoted away from on-site gas generation, opting instead for grid power and diverse energy mixes. This signals that even well-funded data center operators are prioritizing flexibility over massive, long-term capital commitments to specialized hardware. Boom’s pivot to sell jet engines as power plants was a bold bet on AI energy needs, but losing its anchor customer suggests the market is more cautious than anticipated.
TechCrunch AI
© TechCrunch AIMore in Market & Regulation
© Lev SelectorMarket & Regulationbusiness
OpenAI Eyes $1.5 Trillion Valuation
Reports indicate OpenAI is targeting a valuation of $1.5 trillion in its next funding round, reflecting massive investor confidence.
Lev Selector
© Lev SelectorMarket & Regulationbusiness
