AI bots · Training
Bytespider
ByteDance's crawler, widely reported to collect data for AI models. ByteDance does not publish an official page for it that we could find.
- Run by
- ByteDance
- Type
- Training bot
- robots.txt token
Bytespider- User-agent string
Contains Bytespider- Honours robots.txt
- Not documented by the vendor. The ai.robots.txt project lists it as not respecting robots.txt, so use a firewall rule if you want it gone.
- Official docs
- github.com
Block it
User-agent: Bytespider Disallow: /
Allow it
User-agent: Bytespider Allow: /
A named group replaces the * group for this bot, so repeat any private-path Disallow lines in it. Firewalls (Cloudflare AI Crawl Control, Vercel AI Bots ruleset) can block it even when robots.txt allows it. See fixes per stack.
Other training bots
- GPTBot (OpenAI)
- ClaudeBot (Anthropic)
- Google-Extended (Google)
- Applebot-Extended (Apple)
- CCBot (Common Crawl)
- Meta-ExternalAgent (Meta)
- Amazonbot (Amazon)
- MistralAI-Training (Mistral AI)