AI crawler & citation checker: is your site in the evidence pool?

Each vendor runs distinct crawlers for distinct purposes, and blocking one is not blocking the others. The strings to look for in logs and robots.txt:

If you have server or CDN logs, one search answers it. Grep the last 30 days of access logs for the strings above — for example: grep -iE "gptbot|oai-searchbot|claudebot|perplexitybot|ccbot" access.log — and look at which paths they hit. Product pages being fetched by OAI-SearchBot or PerplexityBot is direct evidence you're reachable; a total absence across 30 days means either robots.txt blocks them or nothing points the crawlers at you. CDN dashboards (Cloudflare and others expose bot-traffic views) show the same thing without shell access.

Open yourdomain.com/robots.txt and read it against the table. Three findings matter. A blanket Disallow for a vendor's search or user-fetch agents (OAI-SearchBot, ChatGPT-User, PerplexityBot) removes you from those engines' grounded answers — that's usually an accident inherited from a "block AI" decision aimed at training crawlers. Blocking only training agents (GPTBot, ClaudeBot, Google-Extended, CCBot) is a legitimate policy choice, but know what it trades: absence from future training data also means weaker brand association in future models. And an empty robots.txt blocks nothing — absence of a rule is permission.

The harder question: when engines answer your category's questions, do they ever cite your domain? Crawlability is a prerequisite, not a result — a reachable site with no citable pages still loses every answer. The only way to check is to ask the engines your buyers' questions with live search on and read the sources they return. Our audit does exactly this and reports a source map: every domain cited across all stored answers, ranked — your domain either appears in it or it doesn't.

We run this check on ourselves and publish the result: in our own 2026-08-23 baseline audit, no answer cited aeosearch.io — 4 of 5 prerequisites passed, and that one failed. The free scorecard runs the same readiness checks against your domain: crawler access, schema.org, llms.txt, and citation presence, alongside the visibility numbers themselves.

Two low-cost additions ride the same deploy. An llms.txt at your root gives answer engines a curated, plain-language map of what your site is and where the substance lives — ours is at aeosearch.io/llms.txt as a working example. And schema.org structured data (Organization, Product, FAQPage on pages with real Q&A) gives extraction-driven engines typed facts instead of prose to parse. Neither guarantees a citation; both remove a reason to skip you.