What it is
robots.txt is a plain text file at the root of your site, like yoursite.com/robots.txt. It has rules for crawlers: which bots can visit which parts of the site.
Well-behaved crawlers, including the big AI companies' bots, read it before crawling. It's a request, not a lock. Bad bots can ignore it.
How a rule looks
Each rule names a bot, then what it may visit. Here's a file that blocks OpenAI's training bot but allows its search bot:
| Line | What it means |
|---|---|
User-agent: GPTBot | This rule is for OpenAI's training bot |
Disallow: / | It may not visit any page |
User-agent: OAI-SearchBot | This rule is for ChatGPT's search bot |
Allow: / | It may visit every page |
The AI bots that matter
- GPTBot: OpenAI's bot for training.
- OAI-SearchBot: indexes pages for ChatGPT search.
- ChatGPT-User: fetches a page when a ChatGPT user's question needs it.
- Google-Extended: controls whether Google can use your pages to train Gemini. It doesn't affect Google Search or AI Overviews.
- PerplexityBot: indexes pages for Perplexity.
- ClaudeBot: Anthropic's bot.
- CCBot: Common Crawl, a public dataset used by many AI companies.
What to block
For most businesses that want to be recommended, the answer is: don't block the search bots. If OAI-SearchBot or PerplexityBot can't read your site, those engines can't cite it.
Blocking training bots is a separate choice. It keeps your content out of future training, but also means future models learn less about you from your own site.
Example
A software company copied a robots.txt template that blocks "all AI bots". Months later it notices ChatGPT never cites its pricing page.
It checks the file, sees OAI-SearchBot blocked, and allows it. A few weeks later its pages start showing up in ChatGPT's sources.
Frequently asked questions
Does blocking Googlebot remove me from AI Overviews?
Yes, but it also removes you from Google search. There's no robots.txt rule that keeps you in Google search but out of AI Overviews.
Is robots.txt the same as llms.txt?
No. robots.txt tells bots what they can visit. llms.txt is an optional guide to your most useful pages.
How do I check my robots.txt?
Open yoursite.com/robots.txt in a browser. Look for any rule that names an AI bot, or a "User-agent: *" rule with "Disallow: /".