What they are
A crawler is a program that visits web pages and reads them. Search engines have used crawlers for decades. AI companies now run their own, and each one identifies itself with a name (its user agent).
The three kinds
AI crawlers do different jobs, and it matters which is which:
- Training crawlers collect pages to train future models. Blocking them keeps your content out of future training, but doesn't change today's answers.
- Search crawlers build the index an AI tool searches when it answers. Blocking them can keep you out of AI answers.
- User fetchers load a page at the moment a user's question needs it.
The main ones
| Company | Training | Search | User fetch |
|---|---|---|---|
| OpenAI (ChatGPT) | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic (Claude) | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | PerplexityBot | Perplexity-User | |
| Google-Extended (a setting, not a separate bot) | Googlebot |
Google is a special case. AI Overviews and AI Mode use the normal Googlebot index. Google-Extended only controls whether your content is used for Gemini models. Blocking it doesn't remove you from AI Overviews.
Should you block them?
If you want AI tools to recommend you, don't block the search crawlers. A tool can't recommend what it can't read. Check your robots.txt, and check your firewall too: some security settings, including Cloudflare's bot protection, block AI bots by default.
Whether to block training crawlers is a separate choice about your content, and it doesn't change today's answers.
How to see them
Your server logs show every visit with its user agent. Looking at them tells you which AI tools read your site, how often, and which pages they read.
Example
A web shop's server logs show visits from "GPTBot", "OAI-SearchBot" and "ChatGPT-User" in the same week.
GPTBot was collecting pages that may be used to train future models. OAI-SearchBot was indexing the shop for ChatGPT search. ChatGPT-User came when a person asked ChatGPT something and it opened the shop's page live to answer. Three bots, three different jobs.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT?
Not from ChatGPT search. GPTBot collects training data. ChatGPT's search uses OAI-SearchBot, and live page loads use ChatGPT-User. To stay in ChatGPT's answers, allow those two.
Can a site that blocks bots still be recommended?
Yes, if other sites talk about it. The AI reads the lists and reviews that mention you, not only your own site. But it can't check your own pages for prices or details.
How do I block an AI crawler?
Add a rule for its name in your robots.txt file. Well-behaved crawlers follow it. Block training bots and search bots separately, because blocking a search bot keeps you out of that engine's live answers.
Do AI crawlers slow down my site?
Most sites won't notice. Very large sites sometimes see heavy crawling and limit it with robots.txt or their firewall.