robots.txt

robots.txt is a small text file on your website that tells crawlers, including AI bots, which pages they may and may not visit.

TechnicalUpdated 2 min read

Also called robots file · robots exclusion

What it is

robots.txt is a plain text file at the root of your site, like yoursite.com/robots.txt. It has rules for crawlers: which bots can visit which parts of the site.

Well-behaved crawlers, including the big AI companies' bots, read it before crawling. It's a request, not a lock. Bad bots can ignore it.

How a rule looks

Each rule names a bot, then what it may visit. Here's a file that blocks OpenAI's training bot but allows its search bot:

LineWhat it means
User-agent: GPTBotThis rule is for OpenAI's training bot
Disallow: /It may not visit any page
User-agent: OAI-SearchBotThis rule is for ChatGPT's search bot
Allow: /It may visit every page

The AI bots that matter

  • GPTBot: OpenAI's bot for training.
  • OAI-SearchBot: indexes pages for ChatGPT search.
  • ChatGPT-User: fetches a page when a ChatGPT user's question needs it.
  • Google-Extended: controls whether Google can use your pages to train Gemini. It doesn't affect Google Search or AI Overviews.
  • PerplexityBot: indexes pages for Perplexity.
  • ClaudeBot: Anthropic's bot.
  • CCBot: Common Crawl, a public dataset used by many AI companies.

What to block

For most businesses that want to be recommended, the answer is: don't block the search bots. If OAI-SearchBot or PerplexityBot can't read your site, those engines can't cite it.

Blocking training bots is a separate choice. It keeps your content out of future training, but also means future models learn less about you from your own site.

Example

A software company copied a robots.txt template that blocks "all AI bots". Months later it notices ChatGPT never cites its pricing page.

It checks the file, sees OAI-SearchBot blocked, and allows it. A few weeks later its pages start showing up in ChatGPT's sources.

Free tool: Robots.txt Generator for AIChoose which AI crawlers may read your site and get a robots.txt that keeps your existing rules.Try it →

Frequently asked questions

Does blocking Googlebot remove me from AI Overviews?

Yes, but it also removes you from Google search. There's no robots.txt rule that keeps you in Google search but out of AI Overviews.

Is robots.txt the same as llms.txt?

No. robots.txt tells bots what they can visit. llms.txt is an optional guide to your most useful pages.

How do I check my robots.txt?

Open yoursite.com/robots.txt in a browser. Look for any rule that names an AI bot, or a "User-agent: *" rule with "Disallow: /".

Go deeper

All posts →
Research

AI product recommendations are mostly a summary of listicles

We fetched every 'best X' page ChatGPT, Gemini and Google AI cited in 839 answers. 90% of the brands they recommended were on one of those lists. Brands on none were recommended 2.5% of the time. Here's how it works and what to do about it.

Make AI search your next revenue channel

Start free trial