Does llms.txt Work? 14 Days of Our Server Logs
OpenAI, Anthropic, Perplexity and Google crawlers made 126,310 requests to our sites in 14 days and never asked for llms.txt. Who did, and when to bother.
By Tyler Phan7 min readLast reviewed
Part of our guide to AI search and GEO
On this page
Between October 6 and 19, crawlers from OpenAI, Anthropic, Perplexity and Google made 126,310 requests to the server that runs elmrank.com and seven other sites of ours. Seven of the eight sites publish an llms.txt, three of them since April or May. Not one of those 126,310 requests asked for it.
The file was requested 293 times, most often by one SEO tool and by browsers and agents we couldn't identify. A file almost nobody reads is normal: in the largest study we know of, Ahrefs found that 97% of the llms.txt files on 38,000 domains got no requests at all in May 2026. Below is what the file is, what Google and Chrome say about it, what our log shows, and when we think it's worth having.
What llms.txt is
The proposal comes from Jeremy Howard, published on September 3, 2024 and revised this August. Its own summary: "A proposal to standardise on using an /llms.txt file to provide information to help agents use a website." The file is Markdown, with a title, a short summary and lists of links to the pages that matter. The proposal points out that "The AI labs themselves publish llms.txt files for their own developer docs: OpenAI, Anthropic, and Gemini." Publishing one for your docs is a different thing from their crawlers reading yours.
Most of the files out there weren't written by hand. Common Crawl analyzed 584,107 llms.txt files from its July 2026 crawl and found that 68.27% came from templates. Wix alone made 41.34% of the corpus, and SEO plugins such as All in One SEO and Yoast were among the other generators it identified. If your site runs on one of those, you may already have one.
What Google and Chrome say
Google's guide to generative AI search lists llms.txt under things you can ignore: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." The same section goes on: "It's completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files. Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."
Chrome's Lighthouse team sees a different use. Its agentic browsing audit checks for the file and says "Without this file, agents may spend more time crawling the site to understand its high-level structure and primary content." A missing file doesn't fail the audit; it's marked not applicable, "as providing the file is optional at the moment." We read that as a file for agents that browse a site on someone's behalf, not for the crawlers that decide what shows up in search or AI answers.
What our server log shows
One server runs elmrank.com, the three sites on our use cases page and four other sites of ours, plus one API. They share one access log. We grouped every request from October 6 to 19 (UTC) by the user agent it sent and counted the requests for /robots.txt and /llms.txt.
| Company and crawlers | Requests | robots.txt | llms.txt |
|---|---|---|---|
| OpenAI: OAI-SearchBot, ChatGPT-User, GPTBot | 65,197 | 1,057 | 0 |
| Google: Googlebot | 42,231 | 6,881 | 0 |
| Anthropic: ClaudeBot, Claude-SearchBot, Claude-User | 16,139 | 2,731 | 0 |
| Perplexity: PerplexityBot, Perplexity-User | 2,743 | 534 | 0 |
OAI-SearchBot alone made 50,155 of OpenAI's requests, ChatGPT-User 11,669 and GPTBot 3,373.
Every one of those crawlers was active on our sites, and all four companies' crawlers read robots.txt hundreds or thousands of times. None asked for llms.txt, and none asked for /llms-full.txt either. Two caveats apply. We grouped by the user agent as sent, without checking IP addresses, so a few requests may be from impostors; that would only add to the count, not explain a zero. And none of our sites links to its llms.txt from its home page or from robots.txt, so a crawler would have had to go looking for it.
That second point matters. Ahrefs found that AI bots don't go looking for files that aren't there: "Zero requests came from AI bots for llms.txt files that don't exist. They never go looking." Its conclusion was that AI tools "fetch llms.txt when a link, an index, or a user instruction tells them it exists." Our files were there, at the standard path, on three of the sites since spring, and in these 14 days none of the four companies' crawlers asked for them.
Other people's logs show our zero isn't the whole picture. EZY, which sells an AI visibility tool, watched 83 small-business sites with llms.txt from April 27 to July 19, 2026. OpenAI's crawlers fetched the file 7 times, Anthropic's 9, PerplexityBot never and Googlebot 67, while each of them fetched robots.txt between 775 and 5,125 times. Meta's crawler fetched it 193 times there; on our server, Meta's crawlers made 185,213 requests, 165,852 of them from its two AI crawlers, Meta-WebIndexer and Meta-ExternalAgent, and none for llms.txt. DEJAN's log of its own site counted 4 fetches from Google, Anthropic and OpenAI crawlers in 20 days (3 from Googlebot, 1 from ClaudeBot), against 1,150 for robots.txt. So the big crawlers do ask for the file now and then on some sites, far less often than for robots.txt.
Who did ask for it
The 293 requests for /llms.txt came from SEOJuice, an SEO tool (116), browsers and agents we couldn't place (84), headless Chrome (31), other SEO and GEO tools (25), Bingbot (13), scripts such as curl (12, some of them our own checks), Andi, an AI search engine (10), and YandexBot (2). Redirects from a bare domain to its www version count as requests, so the number of files actually delivered is lower. The only search engines that named themselves were Bing and Yandex, and the only AI product that did was Andi.
Ahrefs' sample looked similar in shape: 96% of requests to llms.txt files came from bots, and "77% of the bots fetching llms.txt aren't AI tools at all." Where we differ is GPTBot. In Ahrefs' data it was the single largest AI fetcher of the file; on our server it made 3,373 requests and none for llms.txt. Ahrefs measured the 3% of files that got any traffic across roughly 1,100 domains, and we measured eight sites, so both can be true.
Should you add one?
- 1Your platform made oneKeep it, read it once, fix what's wrong
- 2Agents use your docs or productWrite one and link it from a page they read
- 3You want AI citationsWork on crawler access and answer pages instead
If your platform makes one for you, keep it and read it once, because whatever it says is what an agent gets told about your business. Common Crawl found prompt injections in a few of these files, and Ahrefs reports people already probing llms.txt for prompt injection. If you're writing one by hand, it's a short Markdown file, and we wouldn't pay anyone to write it as an AI search tactic.
Where it can earn its keep is when agents act on your site: developer docs, or a product that AI coding tools or shopping agents use. In Ahrefs' data, AI agents and agent tooling such as Claude-Code fetched llms.txt more often than AI search bots did, and its advice was to "Link the file from your HTML, reference it in your docs, or mention it anywhere agents receive instructions about your site." For a small business that wants to be cited in ChatGPT or Perplexity, we'd spend the time on things those crawlers do read: robots.txt rules, the CDN or firewall settings that can block them, and pages that answer a buyer's question in plain text.
We publish one on elmrank.com, live since our launch on October 14. It lists our service, prices and pages. In the five and a half days it was up during our log window, no crawler from the four companies asked for it.
- Search your access log for "llms.txt" and see who, if anyone, requested it
- If your CMS or SEO plugin generated one, read it and fix anything wrong or out of date
- Don't buy llms.txt as an AI search service, and don't expect citations from it
- If agents use your docs or product, link the file from a page they read
- Check that robots.txt and your CDN let OAI-SearchBot, PerplexityBot and ClaudeBot in, if you want to be cited
How we handle llms.txt for a client
On our AI search optimization page we offer an llms.txt only if you want one, and we tell you first that Google Search doesn't use it. The work we do count on for AI answers is crawler access, schema, tracking and pages written to answer questions, set up in the first month of our plan ($500 with an SEO and GEO audit, site design and a content plan). After that, $300 a month covers up to 30 articles, ongoing fixes, hosting and a monthly report with your Search Console clicks and impressions; our sample report is filled in with one of our own sites. We don't promise citations; the one result our terms promise is about Google clicks. Our posts on getting cited by Perplexity and getting mentioned in ChatGPT cover the access side.
How we know this
Written by Tyler Phan. Every source below was read on the date shown; the page was last reviewed on Oct 21, 2026.
- llmstxt.org, The /llms.txt file, v2 (Jeremy Howard; published Sept 3, 2024, modified Aug 10, 2026), accessed Oct 20, 2026
- Google Search Central, Optimizing for generative AI features in Google Search (last updated July 10, 2026), accessed Oct 20, 2026
- Chrome for Developers, Lighthouse agentic browsing audits: llms.txt (last updated May 5, 2026), accessed Oct 20, 2026
- Ahrefs, We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read (Louise Linehan, Xibeijia Guan; June 15, 2026, modified Aug 27, 2026), accessed Oct 20, 2026
- Common Crawl, A Content Analysis of llms.txt Files from the July 2026 Crawl Archive (Malte Ostendorff, Aug 31, 2026), accessed Oct 20, 2026
- EZY Research, We Put llms.txt on 83 Websites. OpenAI Read It 7 Times. (July 27, 2026), accessed Oct 20, 2026
- DEJAN (Dan Petrovic), Google, Anthropic and OpenAI crawlers fetched dejan.ai's robots.txt 1,150 times and its llms.txt 4 times in 30 days (Aug 15, 2026), accessed Oct 20, 2026
- Elmrank, requests by user agent in the access log of the server that runs elmrank.com and seven other sites we run, Oct 6 to 19, 2026 (UTC), accessed Oct 20, 2026
Send us your site.
We reply within 12 hours with what we'd fix first. Then we run it for you, technical SEO and content: $500 for the first month, then $300 a month for up to 30 articles. You don't need any technical knowledge, and you can cancel any time.
