Free · no account · nothing stored
Can AI crawlers read your site?
Type your address and you'll see your robots.txt read the way 18 crawlers from OpenAI, Anthropic, Google, Perplexity, Apple and others read it, with the exact line that opens or closes the door for each one. We also knock as GPTBot, ClaudeBot and PerplexityBot, because a firewall can turn them away before robots.txt is ever read.
What the check reads
- robots.txt
- Read per RFC 9309, the standard the crawlers follow: the most specific group wins, and the longest path decides.
- Your firewall
- Your homepage asked for as a browser, then as three AI crawlers. A refusal only they get is a bot setting at your host or CDN.
- llms.txt and sitemap
- Whether assistants find a short guide to your site, and whether crawlers find a list of your pages.