← All Posts
November 11, 2026
Technical SEO
7

AI Crawlers and robots.txt: What Blocking GPTBot, ClaudeBot and the Others Costs

GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended: which ones train models, which ones decide whether you appear in AI answers, and how to write a robots.txt that blocks only what you mean to block.

"Block the AI bots" sounds like one decision. It is at least two: whether your content may be used to train models, and whether your site may appear in AI search answers. Most vendors use separate crawlers or tokens for those jobs, and a robots.txt that treats them as one group usually blocks more than its author intended. This article lists what each vendor documents, then the rules of the file itself. The vendor statements link to their sources, all read on November 11, 2026.

Three Jobs, Three Kinds of Crawler

  • Training crawlers collect pages that may be used to train models. Blocking one is an opt-out from training. OpenAI, Perplexity, Google and Apple each document it as separate from appearing in their search product.
  • Search crawlers build the index an assistant searches before it answers. Blocking one takes you out of the answers, or at least lowers your visibility in them.
  • User-triggered fetchers load a page because a person asked the assistant about it. They are not crawling on their own, and two of the three vendors below say robots.txt may not apply to them.

What Each Vendor Documents

OpenAI

  • GPTBot (training): blocking it signals that the content should not be used for training.
  • OAI-SearchBot (search): the site is not shown in ChatGPT search answers; it can still appear as a navigational link.
  • ChatGPT-User (user fetch): robots.txt rules "may not apply"; it is not used to decide what appears in search.

Anthropic

  • ClaudeBot (training): blocking it signals that the site's future content should be left out of training data.
  • Claude-SearchBot (search): the content is not indexed for search, which may reduce visibility in results.
  • Claude-User (user fetch): Claude cannot retrieve the page to answer a user's question.

Perplexity

  • PerplexityBot (search): no cost is stated; Perplexity recommends allowing it to appear in its results.
  • Perplexity-User (user fetch): it "generally ignores robots.txt rules", since a user requested the fetch.

Google

  • Google-Extended (training and grounding control): the content is not used to train future Gemini models or for grounding in Gemini Apps and Vertex AI.

Sources: OpenAI, Anthropic, Perplexity, Google.

Four details in those pages are easy to miss:

  • Perplexity's search crawler does not train. Perplexity says PerplexityBot "is not used to crawl content for AI foundation models". Blocking it protects nothing on the training side.
  • Google-Extended is not a crawler. Google says it "doesn't have a separate HTTP request user agent string": it is a token you put in robots.txt, and it "does not impact a site's inclusion in Google Search". It will never show up in your server logs, and blocking it does not take you out of AI Overviews: Google's page on AI features names Googlebot's robots.txt rules as the control, with nosnippet, data-nosnippet, max-snippet and noindex to limit what is shown.
  • OpenAI also runs OAI-AdsBot, which only visits pages submitted as ads on ChatGPT and whose data is not used for training.
  • Anthropic words its training opt-out for the future: blocking ClaudeBot signals that the site's "future materials" should be excluded.

Two more tokens come up often. Apple's Applebot-Extended works like Google's: it "does not crawl webpages", it only tells Apple not to use what Applebot crawled to train its models, and pages that disallow it "can still be included in search results". CCBot is Common Crawl's crawler, which builds an open repository of web crawl data.

How a Crawler Picks Its Rules

This is where most files go wrong. Google's robots.txt reference states that "only one group is valid for a particular crawler": the one with the most specific matching user agent, and that named groups and the * group "are not combined". The standard itself, RFC 9309, makes the * group the fallback for crawlers that match no other group.

So a crawler that has its own group stops reading the * group. Take this file:

User-agent: *
Disallow: /admin/
Disallow: /checkout/

User-agent: GPTBot
Allow: /

A crawler following the standard reads this as: GPTBot may fetch everything, /admin/ and /checkout/ included. The author meant "GPTBot is welcome", and wrote "GPTBot is exempt from my other rules". If a named bot should keep the general restrictions, repeat them inside its group.

Other rules worth knowing, all from Google's reference:

  • The order of groups does not matter, and user-agent names are case-insensitive. Paths are case-sensitive.
  • A robots.txt applies only to the host, protocol and port it is served from. www.example.com and shop.example.com each need their own file.
  • If the file answers with a 4xx error (other than 429), Google treats it as if there were no file: no restrictions.
  • Google does not support Crawl-delay. Anthropic says it does.

Changes are not instant. OpenAI says it can take about 24 hours for its search systems to adjust after a robots.txt update, Perplexity says up to 24 hours, and Google generally caches the file for up to 24 hours. Allow a day before testing.

A File That Blocks Training and Keeps Search

OpenAI and Perplexity both say each setting works independently, and OpenAI's own example allows OAI-SearchBot while disallowing GPTBot. A file that opts out of training on the four training tokens above, keeps Common Crawl's open repository out as well, and leaves search and user fetches under the general rules:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot have no group of their own here, so they follow the * group. Whether to opt out of training at all is a business decision, not a technical one. What I advise against is the copied "block all AI" list that includes the search crawlers, on a site that also wants to be cited in AI answers.

What robots.txt Cannot Do

  • It is a request, not a lock. RFC 9309 says the rules "are not a form of access authorization", and Google says robots.txt "cannot enforce crawler behavior": it is up to each crawler to obey. Anthropic says its bots honor robots.txt directives. A scraper that ignores the file is not stopped by it.
  • It does not cover every user-triggered fetch. See ChatGPT-User and Perplexity-User in the list above.
  • It does not remove a page from Google. Google says a disallowed page "can still be indexed if linked to from other sites". To keep a page out of results, use noindex and let it be crawled.
  • It is public. The RFC notes that listing paths in the file makes them discoverable. Do not use it to hide a private URL.

Check the File That Is Actually Served

The file in your CMS is not always the file crawlers receive, and a crawler allowed in robots.txt can still be stopped before it reaches the page.

  • Your CDN may edit the file. On Cloudflare, when Bot Preference Sync is on and an AI policy is set to block or disallow, the documentation says Cloudflare "prepends managed directives" to your robots.txt. Its sample output disallows GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and three others. Open yoursite.com/robots.txt in a browser and read what is really there.
  • Your CDN may block at the edge. Cloudflare's AI crawler policies set separate actions for search, agent and training traffic and, per that page (updated November 2, 2026), a training action also applies to crawlers used for both search and training. New domains default to allow.
  • Your firewall may block a bot you allowed. Perplexity's documentation has a whole section on it: with a web application firewall, "you may need to explicitly allow Perplexity's bots", and it notes that Cloudflare's Bot Fight Mode cannot be skipped with a custom rule.
  • A user agent can be faked. OpenAI, Anthropic, Perplexity and Common Crawl each publish the IP ranges their crawlers use, and Common Crawl warns about crawlers "falsely identifying themselves as CCBot". Check the IP before concluding from a log line that a vendor ignored your rules.

Cloudflare's AI Crawl Control, available on all plans, shows which AI services request your pages and which ones follow your robots.txt.

What I Do on My Own Site

My robots.txt is one line, the sitemap. Nothing is disallowed, so every crawler in this article may read the site. I want the pages cited, and I have nothing on it that I would keep from training. A publisher or a site with paid content can reasonably choose otherwise for the training tokens. The search tokens are the ones to think twice about.

Check Your Site

My free SEO audit starts with an instant check that reads your robots.txt for nine crawler tokens. It flags a block on the six that decide search visibility and notes a block on the training ones without counting it against you. For the wider picture of what the engines document, see AI SEO: What Google, OpenAI, Anthropic and Perplexity Actually Document.

Post
Long Form
Topic
Technical SEO
Published
November 11, 2026
Read
7
Written by
Gregoire Geoffroy
Contents
In This Post

See What Your Site Is Missing

Start with a free audit of your site, checked by me. Or book thirty minutes with your project on screen: you leave with a scoped plan and a number, whether or not we work together.

Grégoire Geoffroy
Grégoire Geoffroy
Webflow Partner
Free audit
Report link within 3 business days
Call
30 min · Google Meet
Time zone
Europe/Paris
Reply
Within one business day