How robots.txt works for AI crawlers
robots.txt is a plain-text file at the root of a host that tells automated clients which paths they may fetch. It is a request, not a lock: it has no authentication and no enforcement, and it works only because the large operators choose to honour it. OpenAI, Anthropic, Google, Apple, Microsoft, Amazon, Meta, Perplexity, DuckDuckGo and Common Crawl all publish the tokens they answer to, and all state that their crawlers obey the directives you write for them. Their user-initiated fetchers are a separate case, and the next section is about exactly that.
The mechanic that trips people up is group matching. A crawler obeys the single most specific group that names it and ignores every other group, including User-agent: *. So if your * group blocks /admin and you then add a group for GPTBot containing only Allow: /, you have just opened /admin to GPTBot. That is why this generator repeats your private paths inside every allowed agent group rather than trusting the wildcard to cascade.
Training vs. search vs. user fetch — the distinction that decides everything
An AI crawler is not one thing. Vendors run up to three separate agents with three separate jobs, and blocking the wrong one is how companies delete themselves from AI answers by accident.
Training crawlers — GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent — collect text that may end up in a model’s weights. Blocking them is a content-licensing decision with a slow, diffuse effect: over successive model generations you fade out of the model’s background knowledge of your category. Search crawlers — OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot, Applebot, DuckAssistBot — build the retrieval indexes that answer engines consult before they reply. Blocking one of these has an immediate, total effect: OpenAI states outright that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. User fetchers — ChatGPT-User, Perplexity-User, Claude-User — retrieve a page because a human asked about it. Here the vendors diverge: Anthropic says all its bots honour robots.txt, OpenAI says robots.txt rules may not apply to user-initiated actions, and Perplexity says its user fetcher generally ignores robots.txt outright.
Two entries in the list are not crawlers at all. Google-Extended and Applebot-Extended fetch nothing; they are permission tokens that govern how data already collected by Googlebot and Applebot may be reused. Both vendors say so explicitly, and both say blocking them leaves search inclusion untouched.
What blocking actually costs a B2B company
For a media business whose archive is the product, blocking training crawlers is a rational negotiating position. For B2B SaaS it usually is not. Your content is a sales asset, not inventory: you publish comparison pages and docs precisely so that someone evaluating your category finds you. When a buyer asks an assistant “best tool for X”, the assistant answers from what it retrieves and what it knows — and a competitor who stayed open occupies both layers while you occupy neither.
The asymmetry is what makes this decision easy to get wrong. Blocking a training crawler costs you slowly and invisibly. Blocking a search crawler costs you immediately and completely, and nothing in your analytics will tell you: there is no impression count for an answer you were left out of. That is the gap Perciva exists to close — it runs your buyers’ questions through each engine and shows you the verbatim answer, including who got recommended instead of you. Our full guide to AI crawler access walks through auditing an existing file, and how to get cited by ChatGPT covers what to do once the crawlers are through the door.
Common mistakes
- Blocking GPTBot and assuming ChatGPT search is covered. They are independent settings. Most people who wanted to opt out of training have also left OAI-SearchBot untouched, which is correct — but plenty do the reverse and never notice.
- Blocking Google-Extended to escape AI Overviews. It does nothing for AI Overviews or AI Mode, which are part of Search and follow Googlebot plus
nosnippet/max-snippet. Google-Extended only governs Gemini and Vertex AI training and grounding. - Blocking Applebot when you meant Applebot-Extended. Applebot is Apple Search. Applebot-Extended is the training token, and blocking it leaves you in Spotlight, Siri and Safari suggestions.
- Treating robots.txt as a paywall. Bytespider is widely reported to ignore it, Perplexity documents that its user fetcher does, and anything already trained or quoted elsewhere stays reachable regardless. If you need enforcement, that is a WAF or bot-management job.
- Fixing the marketing site and forgetting the subdomains. robots.txt is per-host. Docs, help centres and blogs on separate hosts each need their own file — and docs are usually the pages AI most wants to cite.
- Publishing an llms.txt that points at blocked pages. Audit both files together, or the map leads somewhere the crawler is not allowed to go.