What it is
In 2025 Cloudflare, which sits in front of a large share of websites, launched a way for site owners to charge AI crawlers per page. A crawler can pay, or get blocked. Cloudflare also began blocking AI crawlers by default on new sites unless the owner allows them.
Why it exists
Publishers argue AI companies use their writing to answer questions, so fewer people click through to them. Charging for crawls, or blocking, is a way to get paid or protect their work.
What it means for brands
For a business that wants AI to recommend it, blocking AI crawlers usually works against you. You want AI search bots to read your pricing and product pages.
Check your settings. If your site is on Cloudflare or another platform that blocks AI bots by default, you may be blocking them without knowing.
Example
A new startup launches on Cloudflare and never touches the bot settings. AI crawlers are blocked by default. ChatGPT can't read its site and describes it from other pages only.
After allowing AI search crawlers, its own pages start showing up as sources.
Frequently asked questions
Should I charge AI companies to crawl my site?
If you're a publisher whose content is the product, maybe. If you sell something else, you want AI to read you for free.
Do AI companies pay?
Some have signed licensing deals with publishers. Pay per crawl is newer and still being adopted.
How do I check if Cloudflare blocks AI bots on my site?
In the Cloudflare dashboard, look at the AI crawl or bot settings for your domain.