Free tool · No signup
Robots.txt generator for AI crawlers
Decide which AI crawlers may read your site. Load your current robots.txt to keep your own rules, pick a policy, and copy the result.
1 Start from your current robots.txt (optional)
2 Choose a policy
Crawlers that put you in AI answers
Crawlers that train future models
Googlebot and Bingbot aren't listed: blocking them removes you from Google and Bing, and from the AI answers built on them.
3 Extras
4 Your robots.txt
# AI crawler rules made with supercite.ai/tools/robots-txt-generator User-agent: * Allow: / # AI crawlers you let in User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Google-Extended User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: Applebot User-agent: DuckAssistBot User-agent: GPTBot User-agent: ClaudeBot User-agent: Applebot-Extended User-agent: Meta-ExternalAgent User-agent: CCBot User-agent: Bytespider User-agent: Amazonbot User-agent: cohere-ai Allow: /
Upload it to the root of your site so it opens at yoursite.com/robots.txt, then check it with the AI crawler checker.
Search crawlers and training crawlers are different
AI companies run several crawlers, and they do different jobs. Search crawlers like OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages so their AI search can cite them. Live fetchers like ChatGPT-User open a page the moment someone asks about it. Block these and you can't be recommended.
Training crawlers like GPTBot, ClaudeBot, Google-Extended and CCBot collect text to train future models. They don't send visitors, so blocking them is a fair choice if you don't want your writing in training data. It won't take you out of ChatGPT search.
Which policy should you pick?
Most businesses: allow everything. Being known to future models helps them describe and recommend you, and most of what they learn about you comes from other sites anyway.
Publishers and anyone selling their content: “Be found, block training”. You stay citable in AI search and keep your text out of training sets. Only pick “Block all” if you really want to disappear from AI answers.
Why it keeps your existing rules
Most generators hand you a new file and quietly drop the rules you already had. This one keeps every group that isn't about the AI crawlers it manages, and copies your private paths (like /admin or /cart) into the AI crawler group. That matters: a crawler named in its own group ignores the rules for all crawlers (“User-agent: *”), so without this step you'd open your private pages to AI. More in the glossary: robots.txt.
Questions about the Robots.txt Generator for AI
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot only collects training data. ChatGPT search uses OAI-SearchBot to index pages and ChatGPT-User to open them live. Allow those two and you can still be cited.
Does Google-Extended affect AI Overviews?
No. Google-Extended controls whether Google uses your content to train Gemini and to ground answers in the Gemini app. AI Overviews and AI Mode use Googlebot, the normal search crawler, so the only way out of them is to leave Google Search too.
Do AI crawlers respect robots.txt?
The major ones from OpenAI, Anthropic, Google, Microsoft, Apple and Perplexity say they do. Some smaller crawlers don't, and robots.txt can't stop them. For those you need firewall rules.
Where do I put robots.txt?
At the root of your domain, so it opens at yoursite.com/robots.txt. Each subdomain needs its own. Changes take effect the next time a crawler reads the file, usually within a day.
Is this tool free?
Yes, with no signup. Everything happens in your browser, except loading your current robots.txt.