Guide · Website checks
Check whether crawlers can reach a page
Distinguish a browser visit, a crawler permission and a successful fetch when checking access to a public page.
Start with the page itself
Open the public page while signed out. Note whether it loads and whether the main information is readable.
Ask your website operator to check the response and any firewall or hosting rules. Opening a page in your browser doesn't show what a crawler gets.
Crawler access affects what a search system can read. Google can still list a blocked URL when it finds links to it elsewhere.
Test the address a customer would actually reach, including the version with or without www. Note every redirect on the way, because each one is another response a crawler has to follow.
Keep the crawler response evidence apart from your browser check.
Run the check from outside your own network where you can. Some hosting rules treat an office or home connection differently from an unknown visitor.
Check the crawler you mean to allow
OpenAI documents OAI-SearchBot for search and GPTBot for training. Their controls serve different purposes.
Ask the website owner to review the rules for the crawler they intend to allow.
Keep a short list of the crawlers the owner has decided to allow, with the purpose and the date of each decision. It makes the next review quicker and helps stop a choice being reversed by accident.
Don't remove a broad rule just to make a scan pass. Work out which pages should be public and which should stay restricted.
Robots.txt asks crawlers to behave. It isn't a password. Keep private content behind real access controls.
Write down what each rule is for before anyone changes it. A rule added long ago to stop a scraper may now block a crawler the owner wants.
Sort crawlers by what they do
AI companies publish separate names for crawlers with different jobs. Some build a search index, some gather text for model training and some fetch a page when a person asks a question.
OpenAI lists OAI-SearchBot for search, GPTBot for training and ChatGPT-User for actions a person starts. It says robots.txt rules may not apply to those user actions.
Anthropic describes ClaudeBot for training, Claude-SearchBot for search quality and Claude-User for requests a person makes. It explains how to allow or limit each one in robots.txt.
Perplexity names PerplexityBot for its search and says it isn't used to collect content for AI foundation models. It says Perplexity-User generally ignores robots.txt, because a person requested the fetch.
So decide by purpose, not by company. An owner might allow search crawlers and refuse training crawlers, and each choice is a separate group in the file.
Read each company's own page before you write a rule. Names and purposes change, and a rule written for an old name does nothing.
Know how Google's controls differ
Google says robots.txt rules for Googlebot are the control for how a site is crawled for Search. It treats its AI features in Search as part of Search, under the same control.
Google-Extended is a product token, not a separate crawler. Google says it governs use of content for training and grounding Gemini models, and doesn't affect inclusion in Google Search.
To limit what Google shows from a page in Search, it points to snippet controls and noindex. Those only work if Googlebot can fetch the page and read them.
That creates a trap. Blocking a page in robots.txt and adding noindex means Google never reads the noindex. Its documentation says the page can then remain in results if other pages link to it.
Treat these as Google's own controls. Other companies document their own, and one company's token means nothing to another company's crawler.
Know what robots.txt does
A robots.txt file gives crawlers instructions. It's not a lock for private information.
Protect private pages with proper access controls, and ask your operator to check both the crawler rules and how the host behaves.
Know how crawlers read the file
The Robots Exclusion Protocol standard says the file sits at the top level of the site, named robots.txt in lower case. Google says a file applies only to its own host, protocol and port.
So a shop or booking system on a subdomain needs its own file. Check every host that serves public pages, not only the main address.
A crawler follows the group that names it. The standard says it falls back to the group for all crawlers only when no group matches its name. A named group therefore replaces the general rules for that crawler, so repeat any general rule you still want it to follow.
Within a group, the most specific matching path wins. Where an allow rule and a disallow rule match equally, the standard says the allow rule should be used.
Changes aren't read at once. Google says it generally caches the file for up to a day. OpenAI and Perplexity each say their systems can take about a day to reflect a change.
You may also hear about llms.txt. Its own site calls it a proposal for giving language models a guide to a site, with a different purpose from robots.txt. Google says no AI text files are needed for its AI features in Search.
Check the file's own response
The response for robots.txt matters as much as what the file says. The standard treats a missing file and a failing server very differently.
If the server reports the file as unavailable, with a client error such as not found, a crawler may access anything. If the server fails with an error of its own, the crawler must assume everything is disallowed.
Google describes its own handling in detail. It treats most client errors as if no file existed. A server error makes it pause crawling, then fall back to the last copy it could read.
A firewall rule that blocks or challenges crawlers on the robots.txt address can therefore change access to the whole site. Ask your administrator to check that address as well as the pages.
Anthropic says blocking its crawlers by IP address may not work reliably, partly because it stops them reading robots.txt. It asks site owners to use the file instead.
Keep observations separate
Repeat the same request after the owner changes a setting. Keep that result separate from indexing and from later AI answer observations.
Where you have logs, keep the timestamp and how you verified the visitor. Anyone can imitate a user-agent string.
Logs often hold more than you need. Keep the fields that answer the access question, such as time, path, response and user agent, and leave the rest out.
Check that private pages and account routes are still restricted. A targeted fix shouldn't open up unrelated content.
A crawler request shows a visit to your server. A citation, a recommendation and a customer visit each need their own evidence.
Allow for delay before you judge the result. A crawler may keep using an earlier copy of your robots.txt for a while, so repeat the check the next day.
Check who the visitor really is
A user-agent string is a label the visitor chooses. Treat a log line that names a crawler as a claim until you check where the request came from.
Google describes two ways to verify its crawlers. One is a reverse DNS lookup on the address, checked against Google's domains and then confirmed with a forward lookup.
The other is to compare the address with the IP ranges Google publishes. OpenAI and Perplexity publish address lists for each of their crawlers, and Anthropic publishes a list too.
Record which method you used beside each count. A verified visit and a claimed visit belong in separate columns, because the claimed column can include imitators.
Check what the response contains
A successful response can still hold a challenge page or an empty shell. Check the title and the main business text in what came back.
If essential text only appears after a script runs, note that dependency and ask your administrator to check how the page is retrieved.
Keep your conclusion narrow. Say which page and response you checked. One successful retrieval says nothing about every assistant.
Check the status code as well as the body. A not found message sent with a success code, or a real page sent with an error code, can mislead an automated client.
Watch for pages that need scripts
Some pages send a near-empty shell and fill in the text with JavaScript. A person sees the full page, but a client that doesn't run scripts only receives the shell.
Google says it renders pages in a headless browser after crawling them, and that this can take time. It also says not all bots can run JavaScript, so server-side rendering or pre-rendering is still worth doing.
Blocked files cause a similar gap. Google says it won't render JavaScript from blocked files, so a disallowed script folder can leave its view of the page incomplete.
Google's URL Inspection tool in Search Console shows the HTML Googlebot received. A live test adds a screenshot of the rendered page.
That shows one crawler's view only. For other crawlers, ask your administrator to fetch the page without running scripts and check that the business text is there.
Keep essential facts, such as the service, the area and the contact route, in the HTML the server sends. Google's guidance on AI features also asks for important content in textual form.
A worked example
A service page opens in a browser.
| Detail | Observation | Review or next step |
|---|---|---|
| Browser check | The page loads for a person. | Record that browser access was observed. |
| Crawler check | A permitted retrieval returns a challenge page. | Ask the administrator to inspect that rule. |
The two responses are separate facts, and neither says whether an assistant recommended the business.
Common questions
Does a robots.txt rule that allows a crawler prove the page was read?
No, it only describes a permission. Check the actual response separately, then look at answers to see if there's a citation or mention.
Can I use this checklist to open private pages?
No. Review only pages the owner means to make public. Account pages and private documents need access controls. Ask the administrator for a targeted change and don't remove unrelated restrictions.
Does a robots.txt rule that allows a crawler prove the page was read?
No, it only describes a permission. Check the actual response separately, then look at answers to see if there's a citation or mention.
Can I use this checklist to open private pages?
No. Review only pages the owner means to make public. Account pages and private documents need access controls. Ask the administrator for a targeted change and don't remove unrelated restrictions.
Check your understanding
A crawler requested your page. What does that show?
Read the answer and explanation
- A request was recorded under the stated logging method.
- An assistant recommended your business.
- A customer visited your shop.
Answer. A request was recorded under the stated logging method.
A request was logged under your logging method. Answers and customer visits need their own evidence.
Answer and explanation
A request was recorded under the stated logging method.
A request was logged under your logging method. Answers and customer visits need their own evidence.
Your next steps
Sources and method
A Carqo review method. Provider statements link to the relevant documentation.
- OpenAI crawler documentation Link checked 4 November 2026
- Google introduction to robots.txt Link checked 13 October 2026
- Anthropic, how Anthropic crawls the web and how site owners can block it Link checked 4 November 2026
- Perplexity, Perplexity crawlers Link checked 4 November 2026
- Google guidance on AI features Link checked 4 November 2026
- Google crawling documentation, Google's common crawlers Link checked 4 November 2026
- Google guidance on blocking indexing with noindex Link checked 29 October 2026
- IETF, the Robots Exclusion Protocol standard Link checked 4 November 2026
- Google guidance on how it reads robots.txt Link checked 4 November 2026
- llmstxt.org, The llms.txt file proposal Link checked 29 October 2026
- Google guidance on verifying its crawlers Link checked 4 November 2026
- Google guide to JavaScript and Search Link checked 29 October 2026
- Search Console Help, URL Inspection tool Link checked 4 November 2026