AI bots · User fetch
Meta-ExternalFetcher
Fetches individual links at a user's request, including for agentic tasks in Meta AI.
- Run by
- Meta
- Type
- User fetch bot
- robots.txt token
Meta-ExternalFetcher- User-agent string
meta-externalfetcher/1.1- Honours robots.txt
- Not always. Meta says it may bypass robots.txt because the fetches are user-requested. Meta says crawlers may cache robots.txt for up to 24 hours (docs updated May 21, 2026).
- Official docs
- developers.facebook.com
Block it
User-agent: Meta-ExternalFetcher Disallow: /
Allow it
User-agent: Meta-ExternalFetcher Allow: /
A named group replaces the * group for this bot, so repeat any private-path Disallow lines in it. Firewalls (Cloudflare AI Crawl Control, Vercel AI Bots ruleset) can block it even when robots.txt allows it. See fixes per stack.
Other Meta bots
- Meta-ExternalAgent (training)
- Meta-WebIndexer (search)
Other user fetch bots
- ChatGPT-User (OpenAI)
- Claude-User (Anthropic)
- Perplexity-User (Perplexity)
- Amzn-User (Amazon)
- MistralAI-User (Mistral AI)