AI bots · Training
CCBot
Builds the open Common Crawl web archive, which is widely used as AI training data.
- Run by
- Common Crawl
- Type
- Training bot
- robots.txt token
CCBot- User-agent string
CCBot/2.0 (https://commoncrawl.org/faq/)- Honours robots.txt
- Yes. Common Crawl documents Disallow for CCBot as the opt-out.
- Official docs
- commoncrawl.org
- Published IP list
- index.commoncrawl.org/ccbot.json
Block it
User-agent: CCBot Disallow: /
Allow it
User-agent: CCBot Allow: /
A named group replaces the * group for this bot, so repeat any private-path Disallow lines in it. Firewalls (Cloudflare AI Crawl Control, Vercel AI Bots ruleset) can block it even when robots.txt allows it. See fixes per stack.
Other training bots
- GPTBot (OpenAI)
- ClaudeBot (Anthropic)
- Google-Extended (Google)
- Applebot-Extended (Apple)
- Meta-ExternalAgent (Meta)
- Amazonbot (Amazon)
- MistralAI-Training (Mistral AI)
- Bytespider (ByteDance)