AI crawlers
See which AI crawlers request your pages, and why.
Read AI crawler requests from your own server logs, and check what your robots.txt asks of each crawler. Every figure says where it came from and what it cannot prove.
Follow the answer back to its sources.
A citation records a link, not a visit or a sale.
Your workflow
- 01Prove the website is yours
- 02Add a log
- 03Read your robots.txt
- 04Explore and decide
01 / AI crawlers
Why AI crawlers matter
A machine usually has to fetch a page before an AI assistant can use it. AI crawler analytics sorts each request in the server logs you add by crawler and purpose.
02 / AI crawlers
Add your server logs
Upload a day or a month of access logs from your web server or CDN. Apache and Nginx logs are accepted, as are Cloudflare JSON logs and CSV files with a header row.
03 / AI crawlers
Sort requests by crawler and purpose
Each request is matched to a named AI crawler by its user agent, and every crawler has a purpose. A user agent is a claim, and anyone can send one.
Requests stay labelled as claimed unless a verification check confirms them, and verified requests are counted on their own.
04 / AI crawlers
Explore requests
Choose the last 30 or 90 complete days, then filter by crawler, purpose, page response or folder. A pages table shows each path with its requests, errors, refusals and crawlers.
05 / AI crawlers
Errors, refusals and robots.txt
An error is your website failing to serve a page. A refusal is your website choosing not to, which may be what you want. You can set pages crawled beside pages that AI answers cited. A request does not prove an AI answer used the page, and the screen says so.
For a website you have proved is yours, you can read your robots.txt on demand. Every decision is allowed, blocked or not known, and not known is never treated as allowed.
How it works
- Prove the website is yours Add your website and complete the ownership proof.
- Add a log Upload server or CDN logs and say where they came from.
- Read your robots.txt See what it asks of each named AI crawler.
- Explore and decide Filter requests by crawler, purpose and page, then decide what to change.
Who it is for
Service businesses
See whether your main service pages were requested and whether your site answered with an error.
Practices
Errors and refusals are listed by page, so you can take specific addresses to your supplier.
Shops
Check that your robots.txt does what you intended for each purpose.
How Carqo works
You connect your website, review what AI search says and receive drafts for your next steps.
Carqo checks the facts
Facts are checked against their source where Carqo holds one. Conflicts and out-of-date facts are flagged to you.
Carqo shows where AI search mentions you
It follows your customer prompts and shows what the answers said about your business.
You decide what to do with each draft
A person on your side reviews each draft and decides whether to use it.
Common questions
Do I need to install anything on my website?
No. You upload log files your web server or CDN already writes. Robots.txt checks read the public file at the root of your website.
Can I trust the crawler name in a log?
Not on its own, because any client can send any user agent. Verified requests are counted separately.
Should I block AI crawlers?
That is your decision, and it depends on purpose. Robots.txt is a request, and a crawler can ignore it.
Does a crawler request mean an AI answer used my page?
No. A request shows that a fetch happened. Pages crawled and pages cited are shown side by side.
Do I need to install anything on my website?
No. You upload log files your web server or CDN already writes. Robots.txt checks read the public file at the root of your website.
Can I trust the crawler name in a log?
Not on its own, because any client can send any user agent. Verified requests are counted separately.
Should I block AI crawlers?
That is your decision, and it depends on purpose. Robots.txt is a request, and a crawler can ignore it.
Does a crawler request mean an AI answer used my page?
No. A request shows that a fetch happened. Pages crawled and pages cited are shown side by side.
See the answers.
Then choose your next step.
Start with a free AI visibility report.