AI bots · Fixes

Fixes per stack

The recommended preset lets AI search and user-fetch bots in and blocks training bots. Delete the training group if you are happy to be used for training. Then put it where your stack serves robots.txt.

The preset

A named group replaces the * group for that bot, so if you disallow private paths such as /admin under *, copy those Disallow lines into the first group too.

# AI search and user-fetch bots: allowed
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Applebot
User-agent: Amzn-SearchBot
User-agent: MistralAI-Index
User-agent: DuckAssistBot
User-agent: Meta-WebIndexer
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: Meta-ExternalFetcher
User-agent: Amzn-User
User-agent: MistralAI-User
Allow: /

# AI training: your choice. Delete this group to allow training.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: Bytespider
Disallow: /

# Everyone else
User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Next.js (App Router) · Static robots.txt (any host) · Vercel · Cloudflare · Shopify · WordPress · Lovable and Bolt

Next.js (App Router): app/robots.ts

Next.js generates /robots.txt from app/robots.ts. If you also have public/robots.txt, delete one so you know which is served.

import type { MetadataRoute } from 'next'

const search = ['OAI-SearchBot', 'ChatGPT-User', 'Claude-SearchBot', 'Claude-User', 'PerplexityBot', 'Perplexity-User', 'Applebot', 'DuckAssistBot']
const training = ['GPTBot', 'ClaudeBot', 'Google-Extended', 'Applebot-Extended', 'CCBot', 'Meta-ExternalAgent']

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: search, allow: '/', disallow: ['/api/', '/admin/'] },
      { userAgent: training, disallow: '/' }, // remove to allow training
      { userAgent: '*', allow: '/', disallow: ['/api/', '/admin/'] },
    ],
    sitemap: 'https://yourdomain.com/sitemap.xml',
  }
}

• Next.js robots.txt file convention

Static robots.txt (any host)

Serve a plain text file at the root of each hostname, for example public/robots.txt in most frameworks. Subdomains need their own file.

• RFC 9309: Robots Exclusion Protocol
• Google: create a robots.txt file

Vercel

robots.txt is not the only gate. In your project go to Firewall, then Rules, then Bot Management. The AI Bots managed ruleset is off (Allow) by default; set to Deny it blocks known AI bots at the edge, including search and user-fetch bots, whatever robots.txt says. The Bot Protection ruleset challenges non-browser traffic but lets verified bots through. Changes apply after you publish them. Preview deployments behind Deployment Protection are not reachable by any crawler, which is usually what you want.

• Vercel WAF managed rulesets
• Vercel bot management

Cloudflare

Cloudflare can change both your robots.txt and who gets through. AI Crawl Control (formerly AI Audit) lets you allow or block each AI crawler; blocks are enforced with a WAF custom rule. The managed robots.txt setting, with Bot Preference Sync on, prepends Disallow lines for AI crawlers to your robots.txt and can serve one if you have none. Open your live /robots.txt after changing either, and run this checker again.

• Cloudflare AI Crawl Control
• Cloudflare managed robots.txt

Shopify: templates/robots.txt.liquid

Shopify generates robots.txt for you. To add AI bot rules, create templates/robots.txt.liquid in the theme code editor, keep the default groups and add your own groups after them. Save a copy of your current /robots.txt first and compare the output.

{% for group in robots.default_groups %}
  {{- group.user_agent }}
  {%- for rule in group.rules -%}
    {{ rule }}
  {%- endfor -%}
  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /

• Shopify: customize robots.txt

WordPress

Three common causes of blocked AI bots: Settings, Reading, "Discourage search engines from indexing this site" (adds a site-wide block); an SEO plugin that edits the virtual robots.txt; and security plugins or host firewalls that block bot user agents outright, which robots.txt cannot override. Check the bot or firewall settings of whichever security plugin you use, and ask your host whether they block AI crawlers at the server.

• WordPress robots_txt filter

Lovable and Bolt

Lovable creates a robots.txt with every new project and its SEO and AI review flags one that blocks crawlers; ask it in chat to apply the preset above, publish, then open yourdomain.com/robots.txt. Bolt and other Vite projects serve files in public/ from the site root, so put the preset in public/robots.txt. Workspace-only or unpublished Lovable URLs never appear in search.

• Lovable: SEO and AI search
• Vite: the public directory

About llms.txt

llms.txt is a proposal for a Markdown summary of your site for language models. It is cheap to add, but none of the vendors above document reading it, and it does not replace robots.txt. Our checker reports whether you have one without counting it for or against you.

Check it worked

Changes can take about a day to reach the crawlers (OpenAI and Perplexity say up to 24 hours, DuckDuckGo 72). Run the AI bot checker against your live domain after you deploy.