Free SEO tool

Can ChatGPT, Claude and Perplexity read your site?

The AI Crawler Checker is a free tool from Ergora for site owners that reads your robots.txt and shows which of 19 AI crawlers, such as GPTBot, ClaudeBot and PerplexityBot, can reach your pages.

Enter your website and we read its robots.txt the way each AI crawler does, then show which assistants can reach your pages and which ones you have shut out, on purpose or not.

Free, no sign-up. We fetch public pages only and store nothing.

What this checks

  • Your robots.txt, read with the same rules the crawlers follow (RFC 9309)
  • 11 AI search and answer crawlers, including OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot
  • 8 AI training crawlers, including GPTBot, ClaudeBot, Google-Extended and CCBot
  • The exact rule that allows or blocks each one, so you know what to change
  • Whether robots.txt points crawlers to your sitemap

Answers are the new search results

People now ask ChatGPT, Claude and Perplexity for recommendations. An assistant can only cite pages its crawler is allowed to read, and one line in robots.txt, often added by a plugin or a cautious developer, can quietly remove you from all of them.

Training and answers are separate switches

Most AI companies run one crawler to train models and another to answer questions. You can block training (GPTBot, ClaudeBot, Google-Extended) and still appear in answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot). This check shows the two groups apart so the choice is a deliberate one.

Questions

Should I block AI crawlers?

Block the training crawlers if you do not want your content used to train models; that has no effect on being cited in answers. Blocking the search and answer crawlers is rarely a good idea for a business, because it removes you from the places people increasingly ask for recommendations.

Does blocking Google-Extended affect my Google rankings?

No. Google-Extended only controls whether Google may use your content for Gemini training and grounding. Google Search, including AI Overviews, uses Googlebot.

My robots.txt allows everything, but AI tools still can't see my site. Why?

robots.txt is a request, not a lock. Firewalls and CDNs can block crawlers outright; Cloudflare, for example, has a setting that blocks AI crawlers on every site that turns it on. If this check says you are open but assistants never mention you, look there next.

How do I unblock a crawler?

Edit robots.txt at the root of your site: remove the Disallow: / line under that crawler's User-agent, or add a group for it with Allow: /. Each crawler picks up the change the next time it fetches the file, usually within a day.

What is the difference between OAI-SearchBot and ChatGPT-User?

OAI-SearchBot builds the index behind ChatGPT search results. ChatGPT-User fetches a page when a person asks ChatGPT to look at it. Blocking either one limits how ChatGPT can use your pages.

Free, no account

Want more than one check?

Get a free read of your whole website by email: an agent looks at your brand, the site itself and what matters for your kind of business, and tells you what to fix first.

No account, no card. A read of your brand, site and trade by email, then two short follow-ups you can stop with one click.

Content pack

Want this done for you, continuously?

This is a one-off check. In Ergora the SEO specialist in the Content pack runs checks like this across every page, connects Search Console, and turns what it finds into fixes, briefs and posts that help you rank, week after week.

Explore the Content pack

Guides that go deeper

More free tools