Which AI bots visit my website? I counted for 30 days

Key takeaways

• In the 30 days to 20 October 2026, the AI crawlers Cloudflare recognises made about 4,040 requests to this site, a personal blog with eight posts.
• About 1,720 of the roughly 3,540 successful requests were for training. Only 26 were fetches for a person asking an assistant.
• The sitemaps were read 687 times. No single post was read more than 66 times.
• Being crawled is not being cited. OpenAI's crawlers made 508 requests in the month; ChatGPT cited this site in none of 12 answers.
• If your site is on Cloudflare, even on the free plan, you can see all of this under AI Crawl Control in a few minutes.

Who visited: ten companies

In the 30 days to 20 October 2026, ten companies sent most of the AI crawler traffic to this site. Cloudflare groups each company's crawlers together, so OpenAI's row covers GPTBot and two more of its bots.

Googlebot is the crawler behind Google Search, and Google's AI answers draw on the same index, which is why Cloudflare lists it here. Anthropic came second, ahead of OpenAI. Perplexity, whose answers did cite this site on 17 October, is not in the top ten at all.

A week earlier, on 13 October, the same 30-day view showed about 3,000 requests: Googlebot 602, ClaudeBot 464, GPTBot 356, Applebot 221. On 20 October it showed Googlebot 889, ClaudeBot 642, GPTBot 373, Applebot 377. The two windows share 23 days, so this is not a clean month on month figure. The direction is still plain: more crawling, most of the growth from Google, Anthropic and Apple.

What they came for

Cloudflare sorts every AI crawler by purpose, and on this site training came first by a wide margin. Of about 3,540 successful requests in the 30 days to 20 October 2026:

That last row is the one I care about most. Twenty six times in a month, someone asked an AI something and it came to read a page here. Everything else was machines filling their stores in case they need it later.

What they read

The most read file on this site is not a post. It is the list of posts. In the 30 days to 20 October 2026:

The four sitemap files took 687 requests between them. The busiest post took 66. Crawlers spend most of their visits finding out what exists, and much less reading it. 271 different paths were requested in all. In bytes, the whole month came to about 74 MB, Googlebot 18.63 MB and GPTBot 11.24 MB of it. On Cloudflare's free plan that cost me nothing.

So a correct, current sitemap matters more than any special file made for AI. In the 30 days to 13 October, the same crawlers fetched my llms.txt zero times and my robots.txt 605 times.

The scanners that borrow a crawler's name

Not every request under an AI crawler's name came from an AI company. Among the ten missing paths requested most often in the 30 days to 20 October 2026, eight were files that hold passwords and keys:

• /web.config and /credentials.json, 4 requests each
• /.env.local, /.env.bak, /terraform.tfvars, /.git/config, /application.yml and /.git/HEAD, 3 each

A real training crawler follows links and sitemaps. It has no reason to ask for a .env file that nothing links to. These were scanners using a crawler's name in their user agent, and on the free plan Cloudflare recognises crawlers by that name, so they were counted as AI crawlers. Every one got a 404, because none of those files exist here. Borrowing a name is that easy: in my first check of this site I did it myself, from my own computer, to see what each crawler gets back.

If someone else built your website, ask them to check this today. Open yoursite.com/.env and yoursite.com/.git/config in a browser. Both should give an error page, never a file.

Being crawled is not being cited

All that reading has not yet turned into AI answers that name this site. The same month, side by side:

• OpenAI's crawlers made 508 allowed requests in the 30 days to 20 October. On 17 October I asked ChatGPT twelve questions this site answers, from my name to the topic of each post. It cited this site in 0 of 12.
• Microsoft's BingBot made 309. Bing Webmaster Tools' AI Performance report showed 0 citations in three months.
• Google sent 961. In Search Console's 28 days to 17 October, this site had 14 impressions in Google's AI features.
• Perplexity is not in the top ten senders, yet it cited this site in 4 of 8 answers on 17 October.

A crawler visit means my page is in the pile. Whether an answer quotes it depends on what the page says, who else says it, and how far the engine trusts the site. The number of visits does not predict the number of citations.

I also cannot tell you whether any of those answers sent me a visitor. Cloudflare's Referral Traffic column showed a dash on every path. Its documentation says referral counts are for paid plans, so on the free plan that dash means "not counted", not zero.

How to see which AI bots visit your own site

There are three ways to check, and the first takes a few minutes if your site already runs on Cloudflare, free plan included.

1. Cloudflare AI Crawl Control. Log in to the Cloudflare dashboard, pick your account and your domain, and open AI Crawl Control.

• Overview shows the total and the companies.
• Crawlers lists each crawler, its company, its purpose, how many requests were allowed or failed, and whether it is allowed or blocked.
• Metrics has the charts. Set the range to "Last 30 days", the longest it offers. Group by Category to see the purposes, read "Most crawled paths", or filter by a path to see who read one page.
• robots.txt is not in Metrics. Cloudflare counts it on the Signals page.

Two limits: on the free plan a crawler is recognised by the name it gives, so a scanner wearing GPTBot's name counts as GPTBot; and there is nothing older than 30 days.

2. Your server log. If your site is not on Cloudflare, ask your host for the raw access log; on cPanel hosting it is under "Raw Access". Then count the crawler names:

grep -oiE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|Googlebot|Bingbot|Applebot|PetalBot|Meta-ExternalAgent|CCBot|Amazonbot|Bytespider" access.log | sort | uniq -c | sort -rn

A name is only a name, so check the address of any request that matters: Google and OpenAI publish the address ranges their crawlers use. Google Analytics will not show any of this. It leaves known bots out, and most of these crawlers never run its script.

3. Your robots.txt. Logs tell you who came. robots.txt tells you who is allowed. My free AI crawler checker reads your robots.txt for every major AI crawler at once and shows which ones it lets in. To see what a crawler really gets back, request a page with its name, as I did in my first check.

Why I let them all in

My robots.txt lets every one of these crawlers in, training crawlers included. I write here so that people, search engines and AI answers can find what I think, and an answer cannot quote a page its crawler never read. A month of data has not changed that. It has only shown me that letting them in is the start, not the result.

FAQ

How do I check which AI bots visit my website?

If your site is on Cloudflare, open AI Crawl Control for your domain: it lists every AI crawler it recognises, its company, its purpose and the pages it read, for up to 30 days, on the free plan too. Otherwise, search your server's access log for crawler names such as GPTBot, ClaudeBot and PerplexityBot. To see which bots your robots.txt allows, use an AI crawler checker.

Which AI crawlers visit websites the most?

On this site, in the 30 days to 20 October 2026, Google sent the most allowed requests (961), then Anthropic (680), OpenAI (508), Apple (377) and Microsoft (309). By single crawler, Googlebot led with 889 requests, then ClaudeBot 642, Applebot 377 and GPTBot 373. Other sites will differ, but these names are the ones to look for first.

What is the difference between an AI training crawler and an AI search crawler?

A training crawler, such as GPTBot or ClaudeBot, collects pages to train AI models. An AI search crawler, such as OAI-SearchBot, builds the index an AI searches when it answers. An AI assistant fetch, such as ChatGPT-User or Claude-User, happens when a person asks a question and the assistant reads a page for them. On this site the third kind was 26 requests in 30 days, out of about 3,540.

Can a bot pretend to be an AI crawler?

Yes. Any program can put GPTBot or ClaudeBot in its user agent. On this site, in the 30 days to 20 October 2026, requests under AI crawler names asked for .env, .git and credentials.json files, which no real crawler needs. Check the address against the ranges the company publishes, and keep no secret files on your web server.

If AI crawlers visit my site, will ChatGPT cite it?

Not necessarily. OpenAI's crawlers made 508 requests to this site in the 30 days to 20 October 2026, and ChatGPT cited it in 0 of 12 answers on 17 October. Being crawled puts your page in the pile; being cited depends on what the page says and how far the engine trusts your site.