What it is
Common Crawl is a non-profit that has crawled the web since 2008 and publishes the results for free. Its archive holds billions of pages, with a new crawl every month or two.
Because it's free and huge, it's a core part of the training data for many AI models, from OpenAI's early GPT models to many open-weight models.
Its crawler
Common Crawl's bot is called CCBot. It follows robots.txt, so you can block it.
Should you allow it?
If you want AI models to learn about your brand, allowing CCBot is one of the widest ways to get into training data. Blocking it keeps your pages out of future crawls, though not out of past ones, and not out of other sites that write about you.
Many news publishers block it to protect their content. Most businesses that want to be recommended have little reason to.
Example
A software company finds its robots.txt blocks CCBot, copied from a template. It allows it again. Its pages are included in the next crawls, and in the training data of models built from them later.
Frequently asked questions
Does Common Crawl affect AI answers today?
Not directly. It affects what future models learn. Today's answers that search the web read live pages.
Can I remove my pages from Common Crawl?
You can block future crawls. Past archives stay public.
Is Common Crawl used by ChatGPT?
OpenAI has used it in training in the past. Companies don't always publish their full training sources.