We added an llms.txt file to this site a few weeks ago. Ahrefs just published the largest study of llms.txt files to date, and the honest read of their data is that it probably won't do what most people building one think it will.

Ahrefs analyzed server logs and live traffic across 137,210 domains in its Web Analytics tool. 28% of them (38,360 sites) publish a valid llms.txt file. Of those, 97% received zero requests of any kind in May 2026. No bots. No humans. Nothing.

The 3% that do get read aren't who you'd expect

Of the requests that did reach an llms.txt file, 96% came from bots and 4% from humans, many of those apparently people sharing the link in Slack, where the app's own link-preview bot fetches it. Slackbot alone out-fetched PerplexityBot, one of the AI search engines the file was supposedly built to help.

Breaking the AI-bot share down further makes the picture worse for anyone hoping this drives citations: AI agents and agentic infrastructure accounted for 10.5% of requests, AI training crawlers 5.3%, AI assistants 2.5%, and AI retrieval bots, the category that actually includes OAI-SearchBot and PerplexityBot, just 1.1%. The bots capable of turning a page into a citation are the smallest slice of an already tiny 3%.

Google said the quiet part in its own documentation

Google's own guide to optimizing for generative AI features has a section literally titled "mythbusting," stating plainly that machine-readable files like llms.txt aren't needed to appear in generative AI search. Days after Google published that, Chrome's Lighthouse tool shipped an experimental audit that flags a missing llms.txt file, an apparent contradiction that John Mueller addressed directly when pressed: he called llms.txt "not done for search," a "temporary crutch, perhaps to save some tokens" for AI coding tools parsing developer documentation, not something most sites need to worry about.

Even the file's own intended audience barely shows up. Zero requests came from AI bots for llms.txt files that don't exist. AI tools never went looking for a file that wasn't there, which kills the idea that a missing llms.txt is some kind of gap an AI system will notice and penalize you for.

Why we built one anyway

Not because we think it'll move our citations in ChatGPT or AI Overviews. The data above doesn't support that, and we're not going to write content elsewhere on this site about evidence-based GEO and then ignore it the moment it's inconvenient for our own site.

We built it because the one part of this study that does land somewhere real is the 10.5% share going to AI agents and agentic infrastructure, tools like Claude Code that act on a user's behalf and, per Ahrefs, out-fetch every AI retrieval bot, assistant, and training crawler combined. That's a narrower, more defensible bet than "AI search visibility": if someone's coding agent or an agentic browser ever needs a clean, structured summary of what GrowthFollo does, the file is there. It cost about half a day to write, costs nothing to maintain since it's generated automatically from the same data that builds the rest of the site, and Ahrefs' own verdict is that this is roughly the right way to think about it.

What it isn't is a substitute for the things that actually influence whether AI systems cite a brand, covered in more depth in how brands can get cited in AI search and how to build authority so AI models cite your brand. If a page isn't crawlable, isn't specific, and has nothing external backing its claims, an llms.txt file sitting at the root of the domain won't fix any of that.