AI bots · Training
ClaudeBot
Collects web content that could contribute to training Anthropic's models.
- Run by
- Anthropic
- Type
- Training bot
- robots.txt token
ClaudeBot- User-agent string
Contains ClaudeBot (Anthropic does not publish a full example string on its help page)- Honours robots.txt
- Yes. Anthropic says its bots honour robots.txt and the non-standard Crawl-delay.
- Official docs
- support.claude.com
- Published IP list
- claude.com/crawling/bots.json
Block it
User-agent: ClaudeBot Disallow: /
Allow it
User-agent: ClaudeBot Allow: /
A named group replaces the * group for this bot, so repeat any private-path Disallow lines in it. Firewalls (Cloudflare AI Crawl Control, Vercel AI Bots ruleset) can block it even when robots.txt allows it. See fixes per stack.
Other Anthropic bots
- Claude-SearchBot (search)
- Claude-User (user fetch)
Other training bots
- GPTBot (OpenAI)
- Google-Extended (Google)
- Applebot-Extended (Apple)
- CCBot (Common Crawl)
- Meta-ExternalAgent (Meta)
- Amazonbot (Amazon)
- MistralAI-Training (Mistral AI)
- Bytespider (ByteDance)