ShipSafe · AI bots
Is your AI-built app invisible to ChatGPT?
Enter your URL. We show which of 23 AI crawlers your robots.txt allows or blocks, and the exact line that decided it. Free, no signup.
Recommended preset
Let AI search and user-fetch bots in so assistants can find and cite you. Training bots are your call; this preset blocks them. A named group replaces the * group for that bot, so if you disallow private paths such as /admin under *, copy those Disallow lines into the first group too.
# AI search and user-fetch bots: allowed User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Applebot User-agent: Amzn-SearchBot User-agent: MistralAI-Index User-agent: DuckAssistBot User-agent: Meta-WebIndexer User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: Meta-ExternalFetcher User-agent: Amzn-User User-agent: MistralAI-User Allow: / # AI training: your choice. Delete this group to allow training. User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Meta-ExternalAgent User-agent: Amazonbot User-agent: MistralAI-Training User-agent: Bytespider Disallow: / # Everyone else User-agent: * Allow: / Sitemap: https://yourdomain.com/sitemap.xml
Where to put it for your stack: copy-paste fixes for Next.js, Vercel, Cloudflare, Shopify, WordPress, Lovable and Bolt.
How the check works
- We read your robots.txt the way RFC 9309 describes: the group naming the bot wins, otherwise the
*group; within a group the longest matching rule wins and Allow wins a tie. - We test the path you entered (the home page if you gave none). Robots.txt that returns 404 means everything is allowed; a 5xx means crawlers should treat the site as blocked.
- Applebot follows your Googlebot rules when Applebot is not named, as Apple documents. Other fallbacks are not applied.
- robots.txt is a request, not a lock. Some user-fetch bots say they may ignore it, and firewalls (Cloudflare, Vercel) can block bots that robots.txt allows. We only see robots.txt and headers.
- We also report an
X-Robots-Tagheader on your page and whether/llms.txtexists. llms.txt is a proposal; no major crawler documents reading it.
AI bot directory
Who runs each bot, what it is for and whether it honours robots.txt, from vendor docs.
Search bots
Index your pages so AI search answers can cite and link to you.
- OAI-SearchBot OpenAI
- Claude-SearchBot Anthropic
- PerplexityBot Perplexity
- Applebot Apple
- Amzn-SearchBot Amazon
- MistralAI-Index Mistral AI
- DuckAssistBot DuckDuckGo
- Meta-WebIndexer Meta
User fetch bots
Fetch a page when a person asks an assistant about it.
- ChatGPT-User OpenAI
- Claude-User Anthropic
- Perplexity-User Perplexity
- Meta-ExternalFetcher Meta
- Amzn-User Amazon
- MistralAI-User Mistral AI
Training bots
Collect content for training AI models, or control whether crawled content is used for training.
- GPTBot OpenAI
- ClaudeBot Anthropic
- Google-Extended Google
- Applebot-Extended Apple
- CCBot Common Crawl
- Meta-ExternalAgent Meta
- Amazonbot Amazon
- MistralAI-Training Mistral AI
- Bytespider ByteDance
Fixes per stack · Bot protection for forms