AI Crawler Checker

See which AI crawlers a website allows or blocks. We read the site's robots.txt and evaluate the rules for GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot and other AI bots, plus Googlebot and Bingbot, so you can confirm search engines still get in.

How to use the AI Crawler Checker

1Enter a website, for example example.com.
2Press Check robots.txt and we fetch https://example.com/robots.txt.
3Read the table: each crawler shows Allowed, Partly blocked or Blocked, and whether it has its own rules.
4Open “Show robots.txt” to see the exact file, then adjust it with the robots.txt generator.

Training crawlers vs AI search bots

Not every AI bot does the same job. Training crawlers such as GPTBot, ClaudeBot and CCBot collect pages to train models. Search and answer bots such as OAI-SearchBot and PerplexityBot fetch pages to answer questions, often with a link back to you. User-triggered fetchers such as ChatGPT-User visit a page because a person asked about it.

Many sites block training crawlers while allowing AI search bots, so they stay visible in AI answers without contributing content to training data. The checker groups bots by purpose so you can see which approach a site takes.

How the rules are evaluated

We follow the robots.txt standard (RFC 9309): a crawler uses the group that names it; if none does, it falls back to the User-agent: * group. Within a group, the longest matching rule wins and Allow wins a tie.

If robots.txt returns a 4xx error, crawlers may treat the whole site as allowed. If the server returns a 5xx error, well-behaved crawlers assume everything is disallowed until it recovers.

robots.txt is a request, not a lock

Reputable AI companies document that their crawlers respect robots.txt, but the file cannot physically stop a bot that ignores it or disguises its user agent. Enforcement needs a firewall that verifies crawlers and blocks or challenges the rest.

AI search visibility

Want ChatGPT, Gemini and Perplexity to cite your business? Devma tunes your robots.txt, llms.txt, schema and content for AI search.

Learn more

Questions? Answers.

Will blocking AI crawlers hurt my Google ranking?

Blocking AI-specific tokens such as GPTBot or Google-Extended does not block Googlebot, so normal search indexing continues. Check that Googlebot and Bingbot still show Allowed in the results.

What does “Partly blocked” mean?

The crawler may fetch the home page, but some paths are disallowed for it, for example /private/ or /wp-admin/.

Is Google-Extended a real crawler?

No. Google-Extended is a robots.txt token that controls whether content crawled by Googlebot may be used for Gemini model training and grounding. It does not affect Google Search.

Why is my robots.txt not found?

The site may not have one, or it may be served from a different hostname (for example only on www). Without a robots.txt, every crawler is allowed by default.

More free tools

All free tools →

Abstract blue swirling shapes with light reflections

Ready to grow?

Let’s build a solution that’s built for your business. Whether you’re starting fresh or scaling fast, we’re here to help you go further, faster.

Get a free proposal WhatsApp us