Free AI Crawler Checker

Find out in seconds whether ChatGPT, Claude, Perplexity, Gemini and other AI assistants are allowed to read your website.

What this checker does

It reads your robots.txt and tells you, for 18 AI crawlers and assistants, whether they may access your homepage. It also checks if you publish an llms.txt file. Nothing is stored and no account is needed.

Training bots vs. answer bots

Not all AI bots are equal. Training bots such as GPTBot, ClaudeBot or CCBot collect content to train models. Search and answer bots such as OAI-SearchBot, PerplexityBot, Claude-SearchBot and ChatGPT-User fetch pages to answer a user's question right now – and link back to you. Many sites block training but allow answer bots, so they keep getting cited and receiving AI traffic.

How to allow or block a bot

Add a group to your robots.txt. For example, to block training but stay visible in ChatGPT search:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Keep in mind that robots.txt is a request, not a technical barrier. Reputable AI companies follow it.

Curious how big sites handle this? See our report on which websites block AI crawlers, with data for 800+ top domains like nytimes.com, amazon.com and github.com.

FAQ

Should I block AI crawlers?

It depends on your goals. Blocking training bots protects content from model training, but blocking search and answer bots like OAI-SearchBot or PerplexityBot removes you from AI answers and the traffic they send.

What is llms.txt?

llms.txt is a proposed file at your site root that gives AI models a short, structured overview of your site and its key pages. It is optional and not yet an official standard.

Does blocking Google-Extended remove me from Google Search?

No. Google-Extended only controls use of your content for Gemini. Googlebot, which powers Search and AI Overviews, is a separate token.

Is this tool free?

Yes, completely free and without signup.

Related guides