AI bots · Search
Claude-SearchBot
Indexes content to improve the quality of search results in Claude.
- Run by
- Anthropic
- Type
- Search bot
- robots.txt token
Claude-SearchBot- User-agent string
Contains Claude-SearchBot- Honours robots.txt
- Yes. Anthropic says blocking it may reduce your visibility in Claude search results.
- Official docs
- support.claude.com
- Published IP list
- claude.com/crawling/bots.json
Block it
User-agent: Claude-SearchBot Disallow: /
Allow it
User-agent: Claude-SearchBot Allow: /
A named group replaces the * group for this bot, so repeat any private-path Disallow lines in it. Firewalls (Cloudflare AI Crawl Control, Vercel AI Bots ruleset) can block it even when robots.txt allows it. See fixes per stack.
Other Anthropic bots
- ClaudeBot (training)
- Claude-User (user fetch)
Other search bots
- OAI-SearchBot (OpenAI)
- PerplexityBot (Perplexity)
- Applebot (Apple)
- Amzn-SearchBot (Amazon)
- MistralAI-Index (Mistral AI)
- DuckAssistBot (DuckDuckGo)
- Meta-WebIndexer (Meta)