Pay per crawl

Pay per crawl lets website owners charge AI companies for crawling their pages. Cloudflare launched it in 2025.

TechnicalUpdated 1 min read

Also called pay-per-crawl · AI crawl licensing

What it is

In 2025 Cloudflare, which sits in front of a large share of websites, launched a way for site owners to charge AI crawlers per page. A crawler can pay, or get blocked. Cloudflare also began blocking AI crawlers by default on new sites unless the owner allows them.

Why it exists

Publishers argue AI companies use their writing to answer questions, so fewer people click through to them. Charging for crawls, or blocking, is a way to get paid or protect their work.

What it means for brands

For a business that wants AI to recommend it, blocking AI crawlers usually works against you. You want AI search bots to read your pricing and product pages.

Check your settings. If your site is on Cloudflare or another platform that blocks AI bots by default, you may be blocking them without knowing.

Example

A new startup launches on Cloudflare and never touches the bot settings. AI crawlers are blocked by default. ChatGPT can't read its site and describes it from other pages only.

After allowing AI search crawlers, its own pages start showing up as sources.

Free tool: AI Crawler CheckerSee which AI crawlers your robots.txt and firewall let in, and the exact line that blocks each one.Try it →

Frequently asked questions

Should I charge AI companies to crawl my site?

If you're a publisher whose content is the product, maybe. If you sell something else, you want AI to read you for free.

Do AI companies pay?

Some have signed licensing deals with publishers. Pay per crawl is newer and still being adopted.

How do I check if Cloudflare blocks AI bots on my site?

In the Cloudflare dashboard, look at the AI crawl or bot settings for your domain.

Go deeper

All posts →

Make AI search your next revenue channel

Start free trial