Does llms.txt work? In 30 days, no AI crawler read mine

Key takeaways

• In 30 days, no AI crawler that Cloudflare recognises fetched /llms.txt or /llms-full.txt on this site, not once.
• In the same 30 days those crawlers fetched robots.txt 605 times, the post sitemap 192 times and the home page 207 times.
• If you want AI crawlers to find your pages, robots.txt, a sitemap and plain HTML pages are what they read. An llms.txt does not replace any of them.
• The count covers only crawlers Cloudflare recognises, over 30 days. It cannot see a person, or an AI tool that fetches the file as an ordinary browser.

What my llms.txt says

My llms.txt, 6,406 bytes on 13 October 2026, is Markdown that the Cloudflare Worker running this site writes fresh on every request, from the same pages and posts the site shows. A first version went up on 1 August 2026; since 28 September it has been generated on each request. It opens like this:

# Yap Pat Yih

> General Manager and co-founder of BlackRevo, founder of Habitus Mind, builder of Lumen. Based in Kuala Lumpur, Malaysia. Works on systems where people and AI work together.

After that come a short paragraph about me and two lists, "Pages (English)" and "Writing", with one line per page: its title, its address and what it is about.

Its companion, llms-full.txt, has been there since 29 September 2026. It is the full text of every post in one file, 122,659 bytes on 13 October 2026. Both are served as plain text.

How I counted

Cloudflare's AI Crawl Control counts the requests that AI crawlers it recognises make to a site, path by path, and it works on this site's free plan. On 13 October 2026 I filtered it for every path containing llms over its longest range, the last 30 days, which began on 14 September. The answer was "No requests data found". I read it again the next morning, for 7 days and for 30: still zero.

A zero only means something if the filter could have seen the file. So I checked three things:

1. The same path filter found 97 requests for /sitemap-posts.xml in 7 days. The filter works.
2. "Path contains .txt" found one request in 30 days: BingBot fetching the small key file this site uses for IndexNow. Text files are counted.
3. The llms files are built on each request by the same Worker that builds the post pages, and the post pages are counted: 57 requests for one post in the same 30 days.

The reading was done by the Hand, one of the two AI sessions I run this site with, in my Cloudflare account, read only.

What the AI crawlers read instead

In the same 30 days the AI crawlers Cloudflare recognises made about 3,000 requests to this site. These are the ones that matter here:

ClaudeBot made 128 of the 192 sitemap fetches, two thirds of them. Cloudflare counts robots.txt on a separate page, Signals, which does not split it by crawler; all 605 fetches succeeded.

The crawlers came, and often. They read what crawlers have read for years: robots.txt to learn the rules, the sitemap to find the pages, and the pages themselves.

What the people behind llms.txt and the AI companies say

Jeremy Howard proposed llms.txt on 3 September 2024 at llmstxt.org, and the proposal itself does not put crawlers first. It describes a file read on demand, when an AI tool needs to know about a topic while it helps someone. In its own words: "Our expectation was that llms.txt would mainly be useful for inference rather than training". My measurement sees crawlers only, so it tests a use the proposal does not lead with.

Google's Search Central page on AI features and your website says you do not need new machine readable files or AI text files to appear in Google's AI features. It does not name llms.txt.

OpenAI, Anthropic and Perplexity each document their crawlers, and each points site owners to robots.txt to control them. None of those crawler pages says the crawler reads a site's llms.txt. All three do publish an llms.txt for their own developer documentation, which fits the proposal's aim: a tool reading docs when asked, not a crawler sweeping the web.

What this measurement cannot see

Cloudflare's AI crawler view counts only the crawlers it recognises, so three kinds of visit are invisible here: a person who opens /llms.txt in a browser, a bot Cloudflare does not recognise, and an AI tool that fetches the file while helping someone but presents itself as an ordinary browser. The last one is exactly the use the proposal describes, and I cannot rule it out.

I also cannot look back past 14 September. Thirty days is the longest range the dashboard offers, the free plan's other analytics cannot filter by path, and the Worker keeps no logs.

So the honest claim is narrow. In these 30 days, no AI crawler Cloudflare recognises asked for the file.

Why I keep it anyway

I keep both files, for two reasons. They cost nothing: the Worker builds them from the same posts the site already shows, so there is nothing to maintain by hand. And they give a person, or an AI tool someone points at this site, one clean text file to read instead of a page full of layout.

If nobody ever asks for them, I have lost nothing. What I will not do is count them as a way to be found.

If you want AI to read your site

The numbers above show what AI crawlers actually read on this site: robots.txt, the sitemap and the pages. That is where the work belongs.

1. Let the crawlers in. Check that your robots.txt does not block them and that your host or firewall does not either. This is how I checked mine.
2. Keep a sitemap that lists every page, and name it in robots.txt.
3. Make each page plain HTML that carries its words without waiting for a script.
4. If you add an llms.txt, add it as a courtesy to readers and tools, not as a step to be found, and measure whether anything asks for it.

FAQ

What is llms.txt?

llms.txt is a Markdown file at the root of a website, at /llms.txt, that names the site, sums it up in a quoted line and lists its main pages with a short note on each. Jeremy Howard proposed it on 3 September 2024 at llmstxt.org. It is meant to help an AI tool understand a site quickly when someone asks it to.

Does Google use llms.txt?

Google's Search Central page on AI features says you do not need to create new machine readable files or AI text files to appear in Google's AI features. The page does not mention llms.txt by name. So adding one is not a step toward Google's AI answers.

Do AI crawlers read llms.txt?

Not on this site. From 14 September to 13 October 2026, the AI crawlers Cloudflare recognises fetched /llms.txt and /llms-full.txt zero times, while they fetched robots.txt 605 times. OpenAI, Anthropic and Perplexity document robots.txt for their crawlers and do not say those crawlers read llms.txt.

Should I add llms.txt to my website?

Add it if it is cheap for you to keep accurate, as a courtesy to people and AI tools that look for it. Do not expect crawler visits or search traffic from it: on this site it had none in 30 days. Spend your effort first on robots.txt, a sitemap and plain HTML pages.

What is the difference between llms.txt and robots.txt?

robots.txt tells crawlers which paths they may fetch, and OpenAI, Anthropic and Perplexity all document it as the way to control their crawlers. llms.txt is a summary of the site and a list of its pages, meant for an AI tool to read when someone asks about the site. On this site, AI crawlers fetched robots.txt 605 times in 30 days and llms.txt zero times.