Two SaaS companies launched llms.txt the same month last year. One saw its AI Mode citations shift within days. The other checked its server logs three months later and found exactly zero requests for the file. Same standard, same effort, wildly different outcomes.
That gap is the real story behind llms.txt, and it’s why the “should you ship it” debate keeps going in circles. The people saying yes and the people saying no are usually talking about different kinds of websites.
What llms.txt Actually Promises to Do
llms.txt is a plain-text Markdown file you place at your site’s root, something like example.com/llms.txt. It gives large language models a curated map of your content instead of forcing them to parse a full HTML page just to find your value proposition.
It’s not a replacement for robots.txt. Robots.txt controls access. llms.txt is closer to a briefing document, one that tells an agent what your site is, who it’s for, and which pages matter most.
That distinction matters because the two files serve completely different jobs, and confusing them is where a lot of the hype started. Search engines like Google use robots.txt to decide what to crawl at all. llms.txt only helps once something is already reading your site, which is a much narrower promise than most marketing posts about it suggest.
The Real Question Isn’t “Should I?” It’s “Does My Site Even Need Guiding?”
Most of the debate skips a more basic question: does an AI system actually struggle to understand your site without help?
A ten-page marketing site with a clear homepage doesn’t need a curated map. An agent can read the whole thing in seconds. A 400-page documentation set with nested API references and versioned guides is a different story. That’s a genuine navigation problem, and it’s exactly the kind of problem llms.txt was built to solve.
That’s the filter worth applying before anything else: content volume, structural complexity, and whether AI agents are already interacting with your site in a way that depends on navigation, not just crawling.
Here’s the part most guides bury: llms.txt fixes a discovery problem, not a content quality problem. If your product pages are thin or your docs are outdated, a tidy index just helps an agent find the weak content faster.
The Verdict, by Site Type
| Site Type | Verdict | Why |
|---|---|---|
| Documentation & developer platforms | Ship it | Coding agents like Cursor, GitHub Copilot, and Claude Code actively fetch llms.txt from docs sites during real sessions |
| SaaS with heavy technical docs | Ship it | Same agent-routing benefit as pure docs sites, plus it’s cheap to maintain alongside existing documentation workflows |
| Small marketing sites and blogs (under 1,000 pages) | Skip or defer | The homepage and nav already summarize the site well enough that a curated map adds little |
| Large ecommerce (10,000+ pages) | Skip | Maintenance cost of keeping the file accurate outpaces the upside; product data and structured markup do more of the real work |
| Small ecommerce (under 1,000 pages) | Optional experiment | Cheap enough to test, but treat it as a minor bet, not a strategy |
| News and publisher sites | Skip for now | No major consumer AI search engine, including ChatGPT search, Perplexity, or Google AI Overviews, has confirmed it reads llms.txt for answering user queries |
Documentation sites are the one category where the evidence is unambiguous. Anthropic, Stripe, Vercel, Cloudflare, and Supabase all ship llms.txt on their developer docs, largely because Mintlify’s late-2024 rollout across hosted docs sites put thousands of platforms on the standard overnight. Coding agents fetch these files as a matter of routine, not as a hopeful bet on future adoption.
Everyone else is placing a smaller, cheaper bet on a standard that hasn’t been confirmed by the platforms that matter most for organic visibility.
Where Most Teams Get llms.txt Wrong
The biggest mistake is treating llms.txt as an AI visibility strategy instead of a small technical convenience. It isn’t a ranking signal, and Google has said so directly.
Google’s own search advocates have been unusually blunt about this. Gary Illyes confirmed Google doesn’t support llms.txt and has no plans to, and John Mueller went further, saying flat out that “for non-developer sites, I don’t think this makes much sense.” That’s the same Google whose Chrome Lighthouse tool has started auditing for llms.txt presence, which tells you the confusion isn’t just coming from marketers.
The numbers back up the skepticism. A study of 300,000 domains found llms.txt adoption sitting at 10.13% after roughly eighteen months of industry conversation, and a separate June 2026 sample of the top 1,000 sites put confirmed adoption at 8.7%. Adoption isn’t accelerating the way early advocates predicted.
Crawler behavior tells the sharper story. An analysis of 137,000 domains found that 97% of llms.txt files received zero crawler hits at all, and of the hits that did land, only 1% came from AI-related bots. A separate 90-day monitoring window across 500 million AI bot visits found roughly 408 requests actually targeting llms.txt files, close to 0.1% of total AI bot traffic.
That’s a single-sentence gut check worth sitting with: the file most teams built for AI crawlers isn’t the thing AI crawlers are reading.
A 90-day before-and-after study across ten sites in finance, B2B SaaS, ecommerce, insurance, and pet care found eight of the ten saw no measurable change in AI traffic after implementation, and one site actually declined by 19.7%. The two sites that did see gains had unrelated changes running in parallel, like PR campaigns and restructured comparison pages, so llms.txt wasn’t the cause.
None of this means the file is worthless everywhere. It means the sites seeing zero return are usually the ones that never needed it in the first place.
How to Know If It’s Actually Working
Here’s the honest gap in almost every llms.txt guide: they tell you how to build the file, then stop. Nobody tells you how to check whether it changed anything.
The right question after shipping llms.txt isn’t “is it live.” It’s whether AI platforms are actually citing your domain more often, and whether the specific pages you flagged as priority are the ones showing up in AI answers. That’s a citation-tracking problem, not a file-formatting problem.
This is exactly where Topify‘s Source Analysis comes in. It tracks the exact domains and URLs that AI platforms cite across ChatGPT, Perplexity, Gemini, and Google AI Overviews, which means you can see whether your llms.txt-linked pages are actually showing up as sources or whether the file is just sitting unread at your root. Pair that with AI Volume Analytics to check whether the topics your llms.txt prioritizes are even the ones generating meaningful AI search demand in the first place.
For ecommerce brands weighing the maintenance cost, that visibility matters even more. Shopify reported AI-driven traffic to its stores grew 8x year over year, with AI-powered search orders up nearly 13x. That’s real upside, but it’s upside you can only capture if you’re measuring whether your AI visibility work, llms.txt included, is actually moving the needle instead of guessing.
Conclusion
There’s no universal answer to whether you should ship llms.txt, and anyone giving you one is skipping the part where site type changes everything. Documentation-heavy and developer-facing sites have a real, demonstrated case: coding agents use these files today, not hypothetically. Everyone else is looking at a low-cost, low-evidence bet that Google’s own search team has publicly called into question.
Before you spend an afternoon on it, run through three checks: does your site have enough structural complexity that an agent would actually benefit from a map, is agent traffic a real part of your growth plan, and do you have the bandwidth to keep the file accurate as your site changes. If two of those three are no, your time is better spent on content structure and citation tracking than on a file most crawlers still aren’t reading.
FAQ
Q: What is an llms.txt file, exactly?
A: It’s a plain-text Markdown file placed at a site’s root, typically at /llms.txt, that gives AI systems a curated index of the site’s most important pages instead of asking them to parse full HTML.
Q: Is llms.txt the same as robots.txt?
A: No. Robots.txt tells crawlers what they’re allowed to access at all. llms.txt only helps an AI agent navigate content it can already reach, which makes it a convenience layer, not an access control.
Q: Does llms.txt actually work for AI search visibility?
A: For consumer AI search like ChatGPT search, Perplexity, or Google AI Overviews, the evidence so far shows little to no measurable effect. For AI coding agents reading documentation sites, it demonstrably works, since tools like Cursor and Claude Code fetch these files during real coding sessions.
Q: Do I need llms.txt for my blog or small marketing site?
A: Usually not as a priority. If your homepage and navigation already summarize the site clearly, a curated map adds little. That time is typically better spent on content structure and technical SEO fixes.
