AI crawler robots.txt checker

What this checks

The tool fetches https://yourdomain/robots.txt and evaluates, for each crawler below, whether the site root (/) is allowed using the standard longest-match rule. It reports AI crawlers and search crawlers separately, and flags whether a Sitemap: directive is present. AI crawlers checked include: ClaudeBot, Claude-User, Claude-SearchBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Amazonbot, DuckAssistBot, meta-externalagent, CCBot, plus Googlebot, Bingbot, and Applebot.

Why it matters

AI answer engines only quote pages their crawlers are allowed to fetch. A single stray Disallow: /, a blocked GPTBot, or a robots.txt that returns a 5xx can keep your content out of ChatGPT, Claude, Perplexity, and Google's AI overviews entirely. This check surfaces those gaps in one request.