AI Crawlers and robots.txt: What Blocking GPTBot, ClaudeBot and the Others Costs
GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended: which ones train models, which ones decide whether you appear in AI answers, and how to write a robots.txt that blocks only what you mean to block.
"Block the AI bots" sounds like one decision. It is at least two: whether your content may be used to train models, and whether your site may appear in AI search answers. Most vendors use separate crawlers or tokens for those jobs, and a robots.txt that treats them as one group usually blocks more than its author intended. This article lists what each vendor documents, then the rules of the file itself. The vendor statements link to their sources, all read on November 11, 2026.
Three Jobs, Three Kinds of Crawler
- Training crawlers collect pages that may be used to train models. Blocking one is an opt-out from training. OpenAI, Perplexity, Google and Apple each document it as separate from appearing in their search product.
- Search crawlers build the index an assistant searches before it answers. Blocking one takes you out of the answers, or at least lowers your visibility in them.
- User-triggered fetchers load a page because a person asked the assistant about it. They are not crawling on their own, and two of the three vendors below say robots.txt may not apply to them.
What Each Vendor Documents
OpenAI
GPTBot(training): blocking it signals that the content should not be used for training.OAI-SearchBot(search): the site is not shown in ChatGPT search answers; it can still appear as a navigational link.ChatGPT-User(user fetch): robots.txt rules "may not apply"; it is not used to decide what appears in search.
Anthropic
ClaudeBot(training): blocking it signals that the site's future content should be left out of training data.Claude-SearchBot(search): the content is not indexed for search, which may reduce visibility in results.Claude-User(user fetch): Claude cannot retrieve the page to answer a user's question.
Perplexity
PerplexityBot(search): no cost is stated; Perplexity recommends allowing it to appear in its results.Perplexity-User(user fetch): it "generally ignores robots.txt rules", since a user requested the fetch.
Google-Extended(training and grounding control): the content is not used to train future Gemini models or for grounding in Gemini Apps and Vertex AI.
Sources: OpenAI, Anthropic, Perplexity, Google.
Four details in those pages are easy to miss:
- Perplexity's search crawler does not train. Perplexity says
PerplexityBot"is not used to crawl content for AI foundation models". Blocking it protects nothing on the training side. Google-Extendedis not a crawler. Google says it "doesn't have a separate HTTP request user agent string": it is a token you put in robots.txt, and it "does not impact a site's inclusion in Google Search". It will never show up in your server logs, and blocking it does not take you out of AI Overviews: Google's page on AI features names Googlebot's robots.txt rules as the control, withnosnippet,data-nosnippet,max-snippetandnoindexto limit what is shown.- OpenAI also runs
OAI-AdsBot, which only visits pages submitted as ads on ChatGPT and whose data is not used for training. - Anthropic words its training opt-out for the future: blocking
ClaudeBotsignals that the site's "future materials" should be excluded.
Two more tokens come up often. Apple's Applebot-Extended works like
Google's: it "does not crawl webpages", it only tells Apple not to use what Applebot crawled to train its models,
and pages that disallow it "can still be included in search results". CCBot is
Common Crawl's crawler, which builds an open repository of web crawl data.
How a Crawler Picks Its Rules
This is where most files go wrong. Google's
robots.txt reference states that
"only one group is valid for a particular crawler": the one with the most specific matching user agent, and that
named groups and the * group "are not combined". The standard itself,
RFC 9309, makes the * group the fallback for crawlers that match
no other group.
So a crawler that has its own group stops reading the * group. Take this file:
User-agent: *
Disallow: /admin/
Disallow: /checkout/
User-agent: GPTBot
Allow: /
A crawler following the standard reads this as: GPTBot may fetch everything, /admin/ and /checkout/ included.
The author meant "GPTBot is welcome", and wrote "GPTBot is exempt from my other rules". If a named bot should keep
the general restrictions, repeat them inside its group.
Other rules worth knowing, all from Google's reference:
- The order of groups does not matter, and user-agent names are case-insensitive. Paths are case-sensitive.
- A robots.txt applies only to the host, protocol and port it is served from.
www.example.comandshop.example.comeach need their own file. - If the file answers with a 4xx error (other than 429), Google treats it as if there were no file: no restrictions.
- Google does not support
Crawl-delay. Anthropic says it does.
Changes are not instant. OpenAI says it can take about 24 hours for its search systems to adjust after a robots.txt update, Perplexity says up to 24 hours, and Google generally caches the file for up to 24 hours. Allow a day before testing.
A File That Blocks Training and Keeps Search
OpenAI and Perplexity both say each setting works independently, and OpenAI's own example allows OAI-SearchBot
while disallowing GPTBot. A file that opts out of training on the four training tokens above, keeps Common
Crawl's open repository out as well, and leaves search and user fetches under the general rules:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
User-agent: *
Disallow: /admin/
Sitemap: https://www.example.com/sitemap.xml
OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot have no group of their own here, so
they follow the * group. Whether to opt out of training at all is a business decision, not a technical one. What
I advise against is the copied "block all AI" list that includes the search crawlers, on a site that also wants to
be cited in AI answers.
What robots.txt Cannot Do
- It is a request, not a lock. RFC 9309 says the rules "are not a form of access authorization", and Google says robots.txt "cannot enforce crawler behavior": it is up to each crawler to obey. Anthropic says its bots honor robots.txt directives. A scraper that ignores the file is not stopped by it.
- It does not cover every user-triggered fetch. See
ChatGPT-UserandPerplexity-Userin the list above. - It does not remove a page from Google. Google says a disallowed page "can still be indexed if linked to from
other sites". To keep a page out of results, use
noindexand let it be crawled. - It is public. The RFC notes that listing paths in the file makes them discoverable. Do not use it to hide a private URL.
Check the File That Is Actually Served
The file in your CMS is not always the file crawlers receive, and a crawler allowed in robots.txt can still be stopped before it reaches the page.
- Your CDN may edit the file. On Cloudflare, when Bot Preference Sync is on and an AI policy is set to block or
disallow, the documentation
says Cloudflare "prepends managed directives" to your robots.txt. Its sample output disallows
GPTBot,ClaudeBot,Google-Extended,Applebot-Extended,CCBotand three others. Openyoursite.com/robots.txtin a browser and read what is really there. - Your CDN may block at the edge. Cloudflare's AI crawler policies set separate actions for search, agent and training traffic and, per that page (updated November 2, 2026), a training action also applies to crawlers used for both search and training. New domains default to allow.
- Your firewall may block a bot you allowed. Perplexity's documentation has a whole section on it: with a web application firewall, "you may need to explicitly allow Perplexity's bots", and it notes that Cloudflare's Bot Fight Mode cannot be skipped with a custom rule.
- A user agent can be faked. OpenAI, Anthropic, Perplexity and Common Crawl each publish the IP ranges their crawlers use, and Common Crawl warns about crawlers "falsely identifying themselves as CCBot". Check the IP before concluding from a log line that a vendor ignored your rules.
Cloudflare's AI Crawl Control, available on all plans, shows which AI services request your pages and which ones follow your robots.txt.
What I Do on My Own Site
My robots.txt is one line, the sitemap. Nothing is disallowed, so every crawler in this article may read the site. I want the pages cited, and I have nothing on it that I would keep from training. A publisher or a site with paid content can reasonably choose otherwise for the training tokens. The search tokens are the ones to think twice about.
Check Your Site
My free SEO audit starts with an instant check that reads your robots.txt for nine crawler tokens. It flags a block on the six that decide search visibility and notes a block on the training ones without counting it against you. For the wider picture of what the engines document, see AI SEO: What Google, OpenAI, Anthropic and Perplexity Actually Document.
See What Your Site Is Missing
Start with a free audit of your site, checked by me. Or book thirty minutes with your project on screen: you leave with a scoped plan and a number, whether or not we work together.
