Free tool · No signup

AI crawler checker

See which AI crawlers your robots.txt lets in, the exact line that blocks each one, and whether your firewall turns them away anyway.

Try:

What it checks

19 crawlers from OpenAI, Google, Microsoft, Anthropic, Perplexity, Apple, Meta and others, sorted by what they do for you: the ones that put you in AI search, the ones that open your page when someone asks, and the ones that only train future models.

For each one we apply your robots.txt the way crawlers do (the most specific group wins, then the longest rule) and show the line that decides it. Then we request the page as a browser and as OpenAI's search crawler, to catch firewalls that block AI even when robots.txt allows it.

What to allow

If you want AI to recommend you, allow the search and live crawlers: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, and of course Googlebot and Bingbot. Blocking them keeps your pages out of those engines' answers.

Training crawlers like GPTBot and ClaudeBot are a separate choice. Blocking them doesn't remove you from today's answers, but future models learn less about you from your own site. See AI crawlers for the full picture.

The most common problem we see

A robots.txt copied from a template that blocks “all AI bots”, search crawlers included. Or a CDN setting like Cloudflare's “Block AI bots”, turned on by default for many new sites since 2025, that blocks them at the firewall where robots.txt can't show it. This tool catches both.

Questions about the AI Crawler Checker

Does blocking GPTBot keep me out of ChatGPT?

No. GPTBot only collects pages for training. ChatGPT search uses OAI-SearchBot, and ChatGPT-User opens pages live when someone asks. Block GPTBot and allow those two, and ChatGPT can still read and cite you.

Can I block Google's AI but stay in Google Search?

Only partly. Google-Extended stops Google using your pages to train Gemini, but AI Overviews and AI Mode use Googlebot, the same crawler as Search. The only way out of them is a nosnippet tag, which also limits your normal search results.

Why does it say my firewall blocks AI crawlers?

Your page answered a normal browser but refused a request that identified itself as OpenAI's search crawler. That's usually a bot-protection setting in Cloudflare or another firewall. Since our request doesn't come from OpenAI's own servers, a few firewalls block it while letting the real crawler in: check your firewall's settings to be sure.

Can I check one page instead of the whole site?

Yes. Enter the full URL, like yoursite.com/pricing, and we apply your robots.txt rules to that path. Rules often block folders like /blog/ or /docs/ without touching the homepage.

Is this tool free?

Yes, with no signup. You can run it as often as you like within a fair limit of 30 checks every 10 minutes.

The terms behind it

AI search glossary →

More free tools

All tools →

Make AI search your next revenue channel

Start free trial