Common Crawl

Common Crawl is a free, public archive of billions of web pages, collected by a non-profit and used to train many AI models.

TechnicalUpdated 1 min read

Also called CCBot

What it is

Common Crawl is a non-profit that has crawled the web since 2008 and publishes the results for free. Its archive holds billions of pages, with a new crawl every month or two.

Because it's free and huge, it's a core part of the training data for many AI models, from OpenAI's early GPT models to many open-weight models.

Its crawler

Common Crawl's bot is called CCBot. It follows robots.txt, so you can block it.

Should you allow it?

If you want AI models to learn about your brand, allowing CCBot is one of the widest ways to get into training data. Blocking it keeps your pages out of future crawls, though not out of past ones, and not out of other sites that write about you.

Many news publishers block it to protect their content. Most businesses that want to be recommended have little reason to.

Example

A software company finds its robots.txt blocks CCBot, copied from a template. It allows it again. Its pages are included in the next crawls, and in the training data of models built from them later.

Free tool: Robots.txt Generator for AIChoose which AI crawlers may read your site and get a robots.txt that keeps your existing rules.Try it →

Frequently asked questions

Does Common Crawl affect AI answers today?

Not directly. It affects what future models learn. Today's answers that search the web read live pages.

Can I remove my pages from Common Crawl?

You can block future crawls. Past archives stay public.

Is Common Crawl used by ChatGPT?

OpenAI has used it in training in the past. Companies don't always publish their full training sources.

Go deeper

All posts →

Make AI search your next revenue channel

Start free trial