AI bots · Training
GPTBot
Crawls content that may be used to train OpenAI's generative AI foundation models.
- Run by
- OpenAI
- Type
- Training bot
- robots.txt token
GPTBot- User-agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot- Honours robots.txt
- Yes. OpenAI says disallowing GPTBot indicates your content should not be used for training.
- Official docs
- platform.openai.com
- Published IP list
- openai.com/gptbot.json
Block it
User-agent: GPTBot Disallow: /
Allow it
User-agent: GPTBot Allow: /
A named group replaces the * group for this bot, so repeat any private-path Disallow lines in it. Firewalls (Cloudflare AI Crawl Control, Vercel AI Bots ruleset) can block it even when robots.txt allows it. See fixes per stack.
Other OpenAI bots
- OAI-SearchBot (search)
- ChatGPT-User (user fetch)
Other training bots
- ClaudeBot (Anthropic)
- Google-Extended (Google)
- Applebot-Extended (Apple)
- CCBot (Common Crawl)
- Meta-ExternalAgent (Meta)
- Amazonbot (Amazon)
- MistralAI-Training (Mistral AI)
- Bytespider (ByteDance)