Guide · Getting started
What AI search companies tell website owners
Compare official website guidance from Google, Microsoft, OpenAI, Anthropic and Perplexity, including where their controls differ.
Start from the companies' own pages
Google, Microsoft, OpenAI, Anthropic and Perplexity each publish guidance for website owners. Their pages are free to read.
Read the page itself before you act on a summary, including this one. Note the date you read it, because these pages change and a rule you remember may have moved.
Where they agree
Google and Microsoft start from the same basics. Pages must be reachable by their crawlers, readable as text and worth reading for the people who land on them.
Neither promises that a page will be chosen. Google says meeting its requirements doesn't mean it will index or show a page. Microsoft says neither search work nor generative engine optimisation promises rankings, traffic or citations.
Google treats it as ordinary search work
Google says optimising for its generative AI features is optimising for the search experience. It says there are no additional requirements to appear in AI Overviews.
Its newer guide adds one condition. A site must be included in Search generative AI features in Search Console to be eligible. Check that setting in your own Search Console account.
Microsoft asks for answers that stand alone
Microsoft says Bing and Copilot rely on the same crawling, indexing and ranking foundation as traditional search. So a page Bing can't index is unlikely to reach Copilot.
It asks for content that stands on its own, with facts stated plainly and key information near the top. It also asks site owners to tell Bing about new, changed and removed pages through IndexNow.
Different points of emphasis
Google says there is no requirement to break content into tiny pieces for AI. Microsoft says pages focused on one topic are more likely to be chosen for grounding.
Google says no special schema markup is needed. Microsoft says structured data may support clearer grounding but promises nothing. A clear page on one topic, in sections people can follow, fits both.
Which crawler does what
OpenAI and Anthropic each separate the crawler that builds their search index from the one that collects training data. OpenAI uses OAI-SearchBot for search and GPTBot for training. Anthropic uses Claude-SearchBot and ClaudeBot.
Perplexity says PerplexityBot surfaces sites in its search and isn't used to train foundation models. Google uses a robots.txt token, Google-Extended, for its other AI systems rather than a separate crawler. OpenAI says a site can block GPTBot and still allow OAI-SearchBot for search.
Visits made for a person
Some requests come because a person asked a question. OpenAI says robots.txt rules may not apply to ChatGPT-User. Perplexity says Perplexity-User generally ignores robots.txt rules.
So robots.txt controls the crawlers, not every visit from an assistant. Check your firewall as well, because a security setting can block a crawler that your robots.txt allows.
What you can leave alone
Google says you don't need AI text files such as llms.txt for its search, and that Google Search ignores them. It says keeping one does no harm there.
Google also warns against making pages for every variation of a question, and against seeking inauthentic mentions. Microsoft names prompt injection and fake social promotion as abuse. Spend the time on pages people would find useful instead.
A worked example
A plumber reads a blog post about AI text files.
| Detail | Observation | Review or next step |
|---|---|---|
| The claim | A blog says an llms.txt file gets a site into Google's AI answers. | Google says its search ignores llms.txt files. |
| The check | The plumber's robots.txt blocks OAI-SearchBot and Claude-SearchBot. | Allow the search crawlers for the AI search tools the plumber wants to appear in. |
The file made no difference to Google. The blocked search crawlers were the thing to change.
Common questions
Is AEO different from ordinary search work?
For Google, follow its Search guidance. For another system, read that provider's documentation rather than assuming the same requirements apply.
Should I block AI training crawlers?
That is your choice. OpenAI says a site can allow OAI-SearchBot for its search while blocking GPTBot for training. Google says Google-Extended doesn't affect Google Search, but it also covers grounding in Gemini Apps.
Is AEO different from ordinary search work?
For Google, follow its Search guidance. For another system, read that provider's documentation rather than assuming the same requirements apply.
Should I block AI training crawlers?
That is your choice. OpenAI says a site can allow OAI-SearchBot for its search while blocking GPTBot for training. Google says Google-Extended doesn't affect Google Search, but it also covers grounding in Gemini Apps.
Check your understanding
Which crawler should you allow if you want to appear in ChatGPT search?
Read the answer and explanation
- GPTBot
- OAI-SearchBot
- ChatGPT-User
Answer. OAI-SearchBot
OpenAI says OAI-SearchBot is used to surface websites in ChatGPT search. GPTBot collects training data.
Answer and explanation
OAI-SearchBot
OpenAI says OAI-SearchBot is used to surface websites in ChatGPT search. GPTBot collects training data.
Your next steps
Sources and method
A Carqo review method. Provider statements link to the relevant documentation.
- Google, optimizing your website for generative AI features on Google Search Link checked 4 November 2026
- Google guidance on AI features Link checked 4 November 2026
- Microsoft, Bing Webmaster Guidelines Link checked 4 November 2026
- Microsoft Bing Webmaster Tools, Why IndexNow Link checked 4 November 2026
- OpenAI crawler documentation Link checked 4 November 2026
- Anthropic, how Anthropic crawls the web and how site owners can block it Link checked 4 November 2026
- Perplexity, Perplexity crawlers Link checked 4 November 2026
- llmstxt.org, The llms.txt file proposal Link checked 29 October 2026
- Google crawling documentation, Google's common crawlers Link checked 4 November 2026