AI crawler & citation checker: is your site in the evidence pool?

A grounded AI answer is built from pages the engine could fetch. If AI crawlers can't reach your product pages — or can, and never cite them — you are invisible in answers for reasons content work won't fix. Both conditions are checkable directly, and the checks take minutes, not a migration.

The user-agents that matter

Each vendor runs distinct crawlers for distinct purposes, and blocking one is not blocking the others. The strings to look for in logs and robots.txt:

Check 1 — your logs: are they visiting?

If you have server or CDN logs, one search answers it. Grep the last 30 days of access logs for the strings above — for example: grep -iE "gptbot|oai-searchbot|claudebot|perplexitybot|ccbot" access.log — and look at which paths they hit. Product pages being fetched by OAI-SearchBot or PerplexityBot is direct evidence you're reachable; a total absence across 30 days means either robots.txt blocks them or nothing points the crawlers at you. CDN dashboards (Cloudflare and others expose bot-traffic views) show the same thing without shell access.

Check 2 — robots.txt: are you blocking them?

Open yourdomain.com/robots.txt and read it against the table. Three findings matter. A blanket Disallow for a vendor's search or user-fetch agents (OAI-SearchBot, ChatGPT-User, PerplexityBot) removes you from those engines' grounded answers — that's usually an accident inherited from a "block AI" decision aimed at training crawlers. Blocking only training agents (GPTBot, ClaudeBot, Google-Extended, CCBot) is a legitimate policy choice, but know what it trades: absence from future training data also means weaker brand association in future models. And an empty robots.txt blocks nothing — absence of a rule is permission.

Check 3 — citations: reachable is not cited

The harder question: when engines answer your category's questions, do they ever cite your domain? Crawlability is a prerequisite, not a result — a reachable site with no citable pages still loses every answer. The only way to check is to ask the engines your buyers' questions with live search on and read the sources they return. Our audit does exactly this and reports a source map: every domain cited across all stored answers, ranked — your domain either appears in it or it doesn't.

We run this check on ourselves and publish the result: in our own 2026-08-23 baseline audit, no answer cited aeosearch.io — 4 of 5 prerequisites passed, and that one failed. The free scorecard runs the same readiness checks against your domain — crawler access, schema.org, llms.txt, and citation presence — on one condition: the run finished with answers and not one unprompted answer named you. A brand the engines already name gets its visibility numbers without the checklist; the question for it is why it loses, not whether the engines can reach it.

While you're in there: llms.txt and schema

Two low-cost additions ride the same deploy. An llms.txt at your root is optional, and we say so plainly: Google has said its Search systems do not read the file, and Ahrefs studied 137,000 domains and found 97% of the files were never requested. It costs an afternoon, so we keep one — ours is at aeosearch.io/llms.txt. Schema.org structured data (Organization, Product, FAQPage on pages with real Q&A) gives extraction-driven engines typed facts instead of prose to parse. It does not guarantee a citation; it removes a reason to skip you.

Your Questions