AI bots · User fetch
Claude-User
Fetches pages when a Claude user asks a question that needs them.
- Run by
- Anthropic
- Type
- User fetch bot
- robots.txt token
Claude-User- User-agent string
Contains Claude-User- Honours robots.txt
- Yes. Anthropic says disabling Claude-User prevents Claude from retrieving your content for user queries.
- Official docs
- support.claude.com
- Published IP list
- claude.com/crawling/bots.json
Block it
User-agent: Claude-User Disallow: /
Allow it
User-agent: Claude-User Allow: /
A named group replaces the * group for this bot, so repeat any private-path Disallow lines in it. Firewalls (Cloudflare AI Crawl Control, Vercel AI Bots ruleset) can block it even when robots.txt allows it. See fixes per stack.
Other Anthropic bots
- ClaudeBot (training)
- Claude-SearchBot (search)
Other user fetch bots
- ChatGPT-User (OpenAI)
- Perplexity-User (Perplexity)
- Meta-ExternalFetcher (Meta)
- Amzn-User (Amazon)
- MistralAI-User (Mistral AI)