AI SEO: What Google, OpenAI, Anthropic and Perplexity Actually Document
Getting cited by AI Overviews, ChatGPT, Perplexity or Claude: what the official documentation says about crawlers, robots.txt, structured data and llms.txt, and what I measured on client sites.
AI SEO, also sold as GEO or AEO, means being found and cited by answer engines: Google's AI Overviews and AI Mode, ChatGPT search, Perplexity, Claude. This article sticks to what the platforms document themselves, with a link to each source, and to what I measured on client sites, dated.
What Google Says
Google's page on AI features (AI features and your website, updated December 10, 2025) says there are "no additional requirements to appear in AI Overviews or AI Mode". To show as a source link, a page has to be indexed and eligible to appear in Google with a snippet, and meeting that guarantees nothing. The same page explains that these answers may use "query fan-out": several related searches across subtopics to build one answer.
In May 2026 Google published an AI optimization guide (last updated July 10, 2026). It names AEO and GEO, says many of the "hacks" sold under those names are not effective for Google Search, and lists what you can ignore: llms.txt and other AI text files, "chunking" content into small pieces, rewriting pages just for AI, chasing inauthentic mentions. It also says structured data is not required for generative AI search and that there is no special schema.org markup to add, while it still helps with rich-result eligibility.
The guide adds one condition the 2025 page did not have: a site must be included in Search generative AI features in Search Console. That is the Search generative AI control, rolled out to all sites by August 31, 2026, under Settings in Search Console. Include is the default; choosing exclude keeps the site's links and content out of AI Overviews, AI Mode and the generative AI features in Discover. Worth a look on any site you manage.
So on Google's side, AI SEO is SEO plus one Search Console setting: the same basics, applied to pages that have to work as a source. What changes, in my view, is how many crawlers you manage and how you measure.
What OpenAI Says About ChatGPT Search
OpenAI's guidance for site owners fits on one page, its crawler documentation. For ChatGPT search it comes down to four points:
- Let the search crawler in. "To help ensure your site appears in search results, we recommend allowing OAI-SearchBot in your site's robots.txt file and allowing requests from our published IP ranges." The second half matters as much as the first: a firewall or CDN rule that blocks those addresses undoes the robots.txt line.
- Opting out removes you from the answers, not from every link. Sites that opt out of
OAI-SearchBot"will not be shown in ChatGPT search answers, though can still appear as navigational links". - Search and training are separate settings. OpenAI's own example allows
OAI-SearchBotto appear in search while disallowingGPTBot, the crawler for training data. When a site allows both, OpenAI says it may use one crawl for both purposes. - Changes take about a day. "It can take ~24 hours from a site's robots.txt update for our systems to adjust."
What the page does not mention is just as useful to know: it says nothing about structured data, sitemaps or llms.txt, and nothing about how ChatGPT search ranks one source above another. As documented, SEO for OpenAI is about access. The rest, in my view, is the same work that makes a page a good source anywhere: clear text, a fast page, and content worth citing.
The Crawlers, One by One
Each platform separates its crawlers by purpose: training, search, or a visit on a user's request.
OpenAI (crawler documentation)
OAI-SearchBotsurfaces sites in ChatGPT search. A site that blocks it will not be shown in ChatGPT's search answers, though it can still appear as a navigational link.GPTBotcrawls content that may be used to train models. Blocking it says your content should not be used for training.ChatGPT-Uservisits a page when a user asks a question. OpenAI says robots.txt rules "may not apply" to these visits, and that this agent does not decide inclusion in search.
Anthropic (help page on crawlers)
ClaudeBotcollects content that may contribute to training; blocking it signals that your future content should be excluded from training data.Claude-SearchBotimproves the quality of search results. Blocking it prevents your content from being indexed for search, which may reduce your visibility.Claude-Usermay fetch a page when a Claude user asks a question. Blocking it stops Claude from retrieving your content to answer.
Anthropic says it respects robots.txt directives.
Perplexity (crawler documentation)
PerplexityBotsurfaces and links sites in Perplexity's results and is not used to train models. Perplexity recommends allowing it to appear in its results.Perplexity-Useracts on a user's request and, according to Perplexity, generally ignores robots.txt.
Google (list of Google crawlers)
Google-Extended is not a separate crawler: it is a robots.txt token that controls whether your content is used to
train future Gemini models and to ground answers in some products (Gemini Apps, Vertex AI). Google says it "does
not impact a site's inclusion in Google Search". For AI Overviews, Googlebot is what counts, and the display
controls are nosnippet, data-nosnippet, max-snippet and noindex.
robots.txt: Choose Knowingly
Blocking training (GPTBot, ClaudeBot, Google-Extended) and allowing search (OAI-SearchBot,
Claude-SearchBot, PerplexityBot, Googlebot) are two separate decisions: OpenAI and Perplexity document their
settings as independent, and Anthropic and Google describe crawlers with distinct effects. A blanket block on "all
AI bots" therefore blocks the search crawlers too: with OpenAI, the site no longer appears in ChatGPT's search
answers, and with Anthropic, visibility in Claude's results may drop. Google also reminds site owners that crawling
has to be allowed in robots.txt and by your CDN or host, and Perplexity gives a section of its documentation to
web application firewall (WAF) settings.
The Technical List That Actually Applies
Google's technical requirements are to be indexed, eligible to show with a snippet, and included in Search generative AI features in Search Console. The practices it lists for AI features are SEO practices:
- Crawling: allowed in robots.txt and by your CDN or host.
- Internal links: content that is easy to find through the site's links.
- Page experience: Google's page experience page opens with a question: do your pages have good Core Web Vitals?
- Text content: important information available as text, supported by good images and video where useful.
- Structured data: Google recommends checking that it matches the visible text, and says there is no special schema.org markup to add for AI.
llms.txt: What It Is, and What It Is Not
llms.txt is a proposal, published by Jeremy Howard on September 3, 2024 (version 2 page modified August 10, 2026): a Markdown file, usually at the site root, that gives agents an organized overview with links to useful content. The project site calls itself "a proposal to standardise on using an /llms.txt file".
None of the crawler documentation cited here says a crawler reads llms.txt to choose its sources. Google is explicit: on June 15, 2026 it added a note to its AI optimization guide saying Google Search does not use these files, and that keeping one for other services "will neither harm nor help" your visibility or rankings in Google Search. I publish one on my site and on several client sites because I think it costs little to maintain, not because any engine has committed to reading it.
Measure: Rates, Not One Snapshot
For Google, traffic from AI Overviews and AI Mode is counted in Search Console's Performance report, within the "Web" search type, with the rest of search. The AI optimization guide also points to a Generative AI performance report in Search Console for this. Presence in AI answers is measured by repeated snapshots, because it moves.
On Pioneer Golf, I tracked AI Overview citations across 19 target queries. Between two snapshots eleven days apart, 6 of the 19 queries had their AI Overview appear or disappear. The site's citation on its main commercial query ("scotland golf trips", 1,600 searches a month in the US) appeared in July 2026, then was gone at the next snapshot. A single snapshot is never a settled result.
What I Observed on Client Sites
These figures are dated snapshots, not guarantees or proof of cause:
- Carda Health: at the July 14, 2026 baseline, cardahealth.com was cited in 363 Google AI answers, against 29, 6 and 14 for its three closest commercial competitors.
- ProfitPad: the program asked four engines (ChatGPT, Claude, Gemini, Perplexity) the target questions. At the August 4, 2026 snapshot, all four recommended ProfitPad for the Eclipse ERP integration question. On the best-partner question, coverage went from one engine in four (August 4) to two in four (August 27), with Claude and Perplexity citing an article from the site as a source.
- Pioneer Golf: ChatGPT cited the site's Old Course page for the client's main product query, in an observation from May 2026.
Check Your Site
My free SEO audit starts with an instant, automatic check: your robots.txt rules for nine crawlers (it flags the six search crawlers and notes the training ones), whether an llms.txt exists (noted, not counted), your JSON-LD structured data, indexability and Core Web Vitals on mobile. Citations in AI answers are part of the free report, which I check myself.
See What Your Site Is Missing
Start with a free audit of your site, checked by me. Or book thirty minutes with your project on screen: you leave with a scoped plan and a number, whether or not we work together.
