Which AI Crawlers to Allow in robots.txt (and What Blocking Them Costs You)
Somewhere in 2024 it became fashionable to block AI crawlers. Hosting companies and CDNs started shipping one-click "block AI bots" toggles, and a lot of business owners flipped them on without reading what they did. Then, a year later, they asked us why ChatGPT recommends every competitor but them.
Blocking is a legitimate choice for a publisher whose product is the content itself. For a service business whose product is a plumber showing up on time, it's self-sabotage. Here's how to make the decision on purpose.
Three Kinds of Bot, Not One
The major AI companies now run separate agents for separate jobs, and each one checks robots.txt under its own name.
Training crawlers collect pages to build future models. OpenAI's is GPTBot, Anthropic's is ClaudeBot, Meta's is meta-externalagent, and Common Crawl's CCBot feeds many others. Blocking these means your content won't be in the next generation of models' baked-in knowledge.
Search crawlers index pages so the engine can cite them in live answers. OpenAI's is OAI-SearchBot, Anthropic's is Claude-SearchBot, and Perplexity's is PerplexityBot. Blocking these removes you from the answer today. This is the one that hurts a local business.
User-triggered fetchers retrieve a specific page because a person asked about it in a chat. ChatGPT-User, Claude-User, and Perplexity-User fall here. Blocking them means when a prospect pastes your URL and asks "is this company any good," the engine can't read your site to answer.
Google is the odd one out. Google-Extended isn't a crawler at all. It's a control token that Googlebot checks to decide whether your pages can be used for Gemini training. Blocking it does not remove you from Google Search, AI Overviews, or AI Mode. Those follow the normal Googlebot rules.
What We Recommend for a Service Business
Allow everything. The upside of being in the training data and in the live index is that engines know you and can quote you. The downside is that your service descriptions, which you published so people would read them, get read. For most businesses that's the whole point.
If you want a middle position, allow search and user-triggered agents and block training only. That keeps you citable today while opting out of future model training. Understand that it's a partial opt-out; models trained on other sites' descriptions of you will still know you exist.
A robots.txt You Can Copy
User-agent: *
Allow: /
# Search and user-triggered AI agents: allow, these are how you get cited
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /
# Training crawlers: allow (recommended) or change to Disallow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Allow: /
Sitemap: https://www.example.com/sitemap.xmlCheck Before You Assume
Load yourdomain.com/robots.txt in a browser. Then check your CDN and hosting control panel for an AI bot toggle, because those act above robots.txt and can block agents your file explicitly allows. Cloudflare and several hosts default new accounts to blocking. Finally, grep your server access log for OAI-SearchBot and PerplexityBot. If you never see them and you're not blocking them, they haven't found a reason to visit yet, which brings us back to legibility and speed.