AI bots · Search
Meta-WebIndexer
Indexes the web to improve Meta AI search results, so Meta AI can cite and link to your content.
- Run by
- Meta
- Type
- Search bot
- robots.txt token
Meta-WebIndexer- User-agent string
meta-webindexer/1.1- Honours robots.txt
- Yes. Meta documents robots.txt rules for its crawlers. Meta says crawlers may cache robots.txt for up to 24 hours (docs updated May 21, 2026).
- Official docs
- developers.facebook.com
Block it
User-agent: Meta-WebIndexer Disallow: /
Allow it
User-agent: Meta-WebIndexer Allow: /
A named group replaces the * group for this bot, so repeat any private-path Disallow lines in it. Firewalls (Cloudflare AI Crawl Control, Vercel AI Bots ruleset) can block it even when robots.txt allows it. See fixes per stack.
Other Meta bots
- Meta-ExternalAgent (training)
- Meta-ExternalFetcher (user fetch)
Other search bots
- OAI-SearchBot (OpenAI)
- Claude-SearchBot (Anthropic)
- PerplexityBot (Perplexity)
- Applebot (Apple)
- Amzn-SearchBot (Amazon)
- MistralAI-Index (Mistral AI)
- DuckAssistBot (DuckDuckGo)