Your team added an llms.txt file to the root of your domain three weeks ago. You expected ChatGPT and Perplexity to start citing your product pages more often. Nothing changed, and that gap between what the file promises and what models actually do with it is the whole problem.
llms.txt Was Never a Rule Models Have to Follow
The confusion starts with what llms.txt actually is. It’s a plain-text convention proposed in 2024, and it comes with no backing from any recognized standards body and no enforcement mechanism. AI providers can read it or skip it entirely, and there’s no penalty either way.
That’s a different situation than robots.txt. Search engines built compliance into their crawlers over two decades, under pressure from a legal and reputational ecosystem that doesn’t exist yet for LLMs. llms.txt has no equivalent history, and no equivalent pressure.
The data backs this up. A study of nearly 300,000 domains found that only 10.13% had an llms.txt file in place, a fraction of the adoption robots.txt or sitemaps reached. Adoption alone doesn’t prove impact, but it tells you the file hasn’t become infrastructure. It’s still an optional add-on that most sites skip.
A separate audit went further and checked whether AI crawlers even request the file. Thirty days of CDN logs across 1,000 domains showed zero requests from GPTBot, ClaudeBot, or PerplexityBot. Google’s crawler accounted for 95% of hits, and it was mostly there for search indexing, not llms.txt specifically. If the bots that generate AI answers aren’t fetching the file, no amount of careful formatting inside it changes what the model does at inference time.
Why Models Choose Sources That Never Declared Anything
If llms.txt isn’t the deciding factor, something else is. Models select sources based on signals they can verify independently: heading structure, consistent entity references, and how often other credible sites point back to the same content. None of that requires a file that says “trust me.”
The same 300,000-domain study found no measurable correlation between having an llms.txt file and citation frequency. In fact, the model performed slightly better on sites without one, which suggests the file isn’t compensating for weak content. It’s just sitting next to it.
The doesn’t file build trust. The content does.
Research into what actually predicts citation backs this up with harder numbers. An analysis of pages cited by ChatGPT found 68.7% follow logical heading hierarchies, and pages using three or more schema types show a 13% higher citation likelihood. Those are structural signals a model can check against the page itself. A declaration in a separate file isn’t something it can check against anything.
The Gap Between Claiming Trust and Earning It
This is where llms.txt and robots.txt diverge in a way worth naming directly. robots.txt tells a crawler what it’s allowed to do. llms.txt tries to tell a model what to believe about your site, and belief isn’t something a directive file can grant.
Authority accumulates from external signals: consistent facts about your brand across multiple sources, clear entity identity, and content that other sites reference on their own. Organization schema plays a bigger role here than most teams expect, since it’s often the first thing AI systems use to evaluate whether a source is reliable, before the model ever gets to your product pages.
Google has been the most direct of any major provider about where llms.txt fits on this. Its own guidance states the file has no effect on Search rankings or AI results, and staff have compared it to the long-abandoned keywords meta tag. As of the latest checks, none of OpenAI, Google, Anthropic, Meta, or Mistral has publicly committed to reading llms.txt in production answer systems. That’s not a rumor about one provider. It’s the absence of commitment across every major one.
How to Know If Your llms.txt Is Actually Working
Here’s the part most teams skip: verifying it. Deploying the file and waiting to see if citations change is a guess, not a measurement, because AI answers vary run to run and attribution is inconsistent even for well-established sources.
A more direct approach is comparing what you declared in llms.txt against what AI platforms actually cite. This is exactly what Topify’s Source Analysis does: it tracks the domains and URLs that ChatGPT, Perplexity, Gemini, and other platforms pull from when answering prompts in your category, so you can see whether your claimed pages show up at all.
If the pages you listed in llms.txt never appear in the citation data, that’s a clear signal the problem sits with content authority, not file syntax. If unrelated pages on your domain are getting cited instead, that tells you something too: the model already found a path to trust certain content, just not the path you tried to point it toward.
What to Do When the Model Ignores What You Wrote
The practical shift here is moving effort away from file maintenance and toward the signals that actually move citation. That means structured content with clear heading hierarchies, consistent entity data across pages, and schema markup that gives models something concrete to verify rather than take on faith.
Visibility Tracking rounds this out by measuring whether that work is paying off over time, not just in a single snapshot. It’s often useful to pair with prompt-level monitoring, since a brand can be well-cited on one query type and invisible on a closely related one, and averaging the two hides the pattern.
Once you can see where citations are landing and where they’re not, the next step is usually operational rather than analytical: adjusting which pages get restructured first, which entities need clearer markup, and which competitor is quietly winning the citations you expected to get. That’s less about writing a better file and more about running content decisions off real data instead of assumption.
Conclusion
llms.txt isn’t a switch that turns on AI trust, and treating it that way is where most of the frustration comes from. The file can still be worth deploying as a low-effort hedge for future standards, but it won’t compensate for content that lacks structure or authority today. The teams making progress aren’t the ones with the cleanest llms.txt file. They’re the ones tracking what AI platforms actually cite and adjusting based on that data instead of a declaration nobody’s required to read.
FAQ
Q: Does llms.txt actually work?
A: Current evidence says no, at least not as a direct driver of citations. A 300,000-domain study found no correlation between having the file and citation frequency, and no major AI provider has confirmed reading it in production.
Q: What’s the difference between llms.txt and robots.txt?
A: robots.txt gives crawlers enforceable instructions about what they can access, backed by decades of compliance norms. llms.txt is a voluntary suggestion with no enforcement mechanism and no equivalent adoption history.
Q: How do I know if AI is reading my llms.txt file?
A: Server log audits are one option, filtering for user agents like GPTBot or ClaudeBot, though most audits find little to no activity from these bots on llms.txt specifically. A more reliable approach is tracking whether the pages you listed actually show up in AI citations, which is what source-level monitoring tools are built for.
Q: If llms.txt doesn’t help, what should I focus on instead?
A: Structural clarity and verifiable authority signals, things like consistent heading hierarchies, complete schema markup, and entity consistency across your site. These are the signals models can check directly, unlike a file that simply asserts what your important pages are.
