AI crawler checker
Type your domain. The checker reads your robots.txt the way AI crawlers are told to, and shows which of 20 crawlers it lets in and which it blocks.
I built this after checking my own site by hand. It fetches two files from your domain, /robots.txt and /llms.txt, and nothing else. The answers are for your home page, the path /.
It follows RFC 9309, the robots.txt standard. A group that names a crawler beats the * group. The longest matching rule wins. On a tie, Allow wins. It also tells you whether robots.txt names a sitemap, and whether you serve an llms.txt.
Results
- robots.txt
- Sitemap in robots.txt
- llms.txt
Your robots.txt is over 128 KB. Only the first 128 KB was read.
| Crawler | What it is for | Your site |
|---|---|---|
| OpenAI | ||
GPTBot | Collects pages to train OpenAI's models. | |
OAI-SearchBot | Builds the index behind OpenAI's search answers. | |
ChatGPT-User | Fetches a page when someone asks OpenAI's assistant about it. | |
| Anthropic | ||
ClaudeBot | Collects pages to train Anthropic's models. | |
Claude-SearchBot | Indexes pages for Anthropic's search answers. | |
Claude-User | Fetches a page when someone asks Anthropic's assistant about it. | |
| Perplexity | ||
PerplexityBot | Indexes pages so Perplexity can show and link them in its answers. | |
Perplexity-User | Fetches a page when someone asks Perplexity about it. | |
Googlebot | Google's search crawler. Its index also feeds Google's AI answers in search. | |
Google-Extended | Not a crawler. A robots.txt token that says whether Google may use your pages for its AI models. | |
| Microsoft | ||
Bingbot | Microsoft's search crawler. Its index feeds Bing and the AI answers built on it. | |
| Apple | ||
Applebot | Apple's crawler for Siri and Spotlight suggestions. | |
Applebot-Extended | Not a crawler. A robots.txt token that says whether Apple may use your pages to train its AI models. | |
| Meta | ||
Meta-ExternalAgent | Collects pages to train Meta's AI models. | |
Meta-ExternalFetcher | Fetches a page when someone asks Meta's AI about it. | |
| Amazon | ||
Amazonbot | Amazon's crawler. Amazon says what it collects may be used to train its AI models. | |
| ByteDance | ||
Bytespider | ByteDance's crawler. It collects pages for training AI models. | |
| Common Crawl | ||
CCBot | Builds Common Crawl, the open web archive many AI models are trained on. | |
| DuckDuckGo | ||
DuckAssistBot | Fetches pages for DuckDuckGo's AI answers. | |
| Mistral | ||
MistralAI-User | Fetches a page when someone asks Mistral's assistant about it. | |
What this does not check
- Your firewall or CDN. robots.txt can say yes while Cloudflare's AI bot setting, a firewall rule or a bot filter answers the crawler with a 403. This checker only reads what the file says. Fetchers that act for a user may not read robots.txt at all.
- Your server logs. It cannot tell you which crawlers actually came, or how often.
- Whether an AI cites you. Being readable is the first step. Being quoted depends on what is on the page and who else covers the topic.
To see what a crawler really gets back, request a page with its name. How I checked whether AI crawlers can read my own site has the command and the five checks I now run.
On llms.txt, I counted for 30 days: no AI crawler read my llms.txt, while they fetched my robots.txt 605 times.
FAQ
How do I check which AI bots can access my website?
Start with robots.txt: type your domain above and this checker reads it for 20 AI crawlers. Then look at your firewall or CDN, because a block there overrides a yes in robots.txt. To be sure, request a page with each crawler's user agent and read the status code. A 200 means it got in.
Does blocking GPTBot stop ChatGPT search?
No. GPTBot collects pages for training. OpenAI's search uses OAI-SearchBot, and a page someone asks about is fetched by ChatGPT-User. Block GPTBot and allow OAI-SearchBot if you want to stay out of training but still show up in its search answers.
Is llms.txt needed?
No. It is a proposal, not a standard. OpenAI, Anthropic and Perplexity point site owners to robots.txt for their crawlers and do not say those crawlers read llms.txt. On my site, in 30 days, AI crawlers fetched it zero times. I keep mine because it costs nothing, but robots.txt, a sitemap and plain HTML pages matter more.