A content team ships an llms.txt file, checks the box, and moves on. Three months later, ChatGPT still can’t accurately summarize the product page, and server logs show zero requests to the file they spent an afternoon writing.
That’s not a bug. It’s the current state of llms.txt in practice.
What llms.txt Actually Controls, and What It Doesn’t
llms.txt is a Markdown file at the root of a domain that gives AI systems a curated map of a site’s most useful content. It’s a navigation aid, not a gate.
The data on how AI systems actually treat it is blunt. A study across 300,000 domains found adoption sitting around 10%, and among the fifty most AI-cited domains, only one had the file at all. Monitoring across a 90-day window turned up only a handful of hundred requests to /llms.txt out of over 500 million AI bot events, with GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and Google-Extended overwhelmingly crawling HTML pages directly instead.
Google has been explicit about where it stands. Google’s Gary Illyes confirmed the company doesn’t support llms.txt and has no plans to, and John Mueller compared it to the discredited keywords meta tag. Separate testing found that eight out of nine sites saw no measurable traffic change after adding the file, and Mueller noted server logs show AI crawlers don’t even check for it.
None of that means llms.txt is worthless. It costs almost nothing to publish and gives agentic tools a cleaner entry point if adoption grows. But it does mean llms.txt sits in a specific, narrow slot: a declaration of what a site would like AI systems to prioritize, with zero enforcement power behind it.
The Five-Layer Permission Stack Behind Every AI Crawler Visit
llms.txt is one layer in a stack that runs from soft declarations to hard technical enforcement. Understanding the full stack matters more than optimizing any single file.
Layer 1: robots.txt. Standardized as RFC 9309, robots.txt tells crawlers what they’re asked not to fetch. It carries no legal force and doesn’t authenticate anything. Compliance depends entirely on whether a given bot chooses to honor it, and well-behaved crawlers generally do while others historically haven’t.
Layer 2: llms.txt. As covered above, this is a content curation layer, not a permission layer. It suggests what to read first. It restricts nothing.
Layer 3: CDN and WAF enforcement. This is where declarations turn into actual blocking. Cloudflare’s shift illustrates the pace of change here. In September 2026, Cloudflare will start blocking “mixed-use” crawlers, ones that blend search, agent, and training traffic, by default on any page carrying ads, unless the site owner overrides it. That follows a year of escalating economics: Cloudflare’s own data showed Anthropic’s crawler fetching roughly 38,000 pages for every referral visit it sent back, and OpenAI’s ratio landing around 1,091 crawls per referral. By June 2026, training-related crawlers made up 50.6% of all bot traffic on Cloudflare’s network, with search-related bots down to just 10.7%.
Layer 4: Bot identity verification. Declaring rules is one thing. Knowing who’s actually knocking is another. Server logs and User-Agent verification catch crawlers that spoof legitimate identities or ignore declared rules entirely, and they’re the only way to confirm whether Layer 1 and Layer 2 are having any real effect.
Layer 5: Licensing and legal terms. Terms of service, TDM opt-out clauses, and active litigation now form the outer boundary. Courts have kept public, logged-out scraping legal in cases like hiQ and Meta v. Bright Data, while training-specific disputes like Reddit v. Perplexity are actively testing where those lines sit. This is the layer where “allowed” gets defined in ways no text file can settle on its own.
Declaring intent isn’t the same as enforcing it.
Where Most Teams Get the Stack Wrong
The most common mistake is treating Layer 2 as if it were Layer 3. A team writes a careful llms.txt, feels covered, and never checks whether their CDN is already blocking the same crawlers by default.
That gap is widening fast. Analysis across Cloudflare’s network found GPTBot is now the most blocked AI crawler by robots.txt directive, and close to 90% of all AI crawler traffic serves training or mixed purposes rather than pure search. Separately, roughly 2.5 million sites now disallow AI training outright, and GPTBot alone is blocked by an estimated 19% of sites.
Layer conflicts are common and mostly unresolved. If a CDN already blocks GPTBot at the network edge, an llms.txt file that welcomes it does nothing. The technical layer wins by default because it executes; the declaration layer only requests.
There’s also a data-quality problem inside Layer 2 itself. One estimate put the share of llms.txt files that amount to little more than generic plugin stubs at nearly 40%, which suggests a lot of teams are checking a box rather than building something a machine-reading system could actually use.
Getting the Permission Layer Right Doesn’t Guarantee AI Visibility
Here’s the part that trips up even careful teams. Every layer in this stack governs access. None of them govern outcome.
A site can configure robots.txt correctly, publish a genuinely useful llms.txt, keep its CDN rules aligned, and verify bot identities in its logs, and still never get mentioned when someone asks an AI assistant for a recommendation in its category. Permission is the entry ticket. It says nothing about whether the AI system finds the content worth citing once it’s inside.
What actually drives citation is a separate set of factors: content structure, topical authority, and how often a brand’s name shows up across the sources an AI model actually pulls from when it forms an answer. That’s a visibility problem, not a permissions problem, and it needs its own monitoring layer.
This is where Topify fits into the stack, not as a sixth permission layer, but as the measurement layer sitting on top of it. Once the technical access questions are settled, the open question becomes whether ChatGPT, Perplexity, or Google AI Overviews are actually citing the site, how often, and against which competitors. Topify’s Source Analysis tracks the exact domains and URLs AI platforms cite, which is the only reliable way to tell whether a permission configuration is translating into real mentions rather than just theoretical access.
How to Audit Your Own Permission Stack in Practice
A working audit runs through all five layers, in order, rather than stopping at whichever one is easiest to configure.
Start with robots.txt. Confirm it explicitly addresses the AI user-agents that matter for the goal, whether that’s allowing search-oriented bots like OAI-SearchBot and PerplexityBot for citation eligibility, or blocking training-oriented bots like GPTBot and Google-Extended to keep content out of model training.
Check llms.txt only after that, and only if there’s a genuine use case for agent-driven navigation. Skip generating a full Markdown mirror of every page. Indexable duplicate mirrors dilute crawl budget and can actively suppress the original pages in search results.
Verify the CDN and WAF layer independently of what robots.txt claims. A rule declared in one place can be silently overridden or duplicated at the network edge, and the only way to know is to check both configurations side by side.
Pull server logs and filter by known AI crawler user-agents to see what’s actually happening, not what the configuration implies should be happening. A honeypot link inside llms.txt that only an automated reader would follow is a simple way to confirm whether anything is reading the file at all.
Finally, track outcomes, not just access. Set up ongoing monitoring for whether the brand shows up in AI answers, which sources get cited instead, and how that shifts as the permission layers change. This is the step most audits skip, and it’s the one that actually connects configuration work to business results.
Conclusion
llms.txt is real, cheap to publish, and worth having if a site already has its content fundamentals in order. What it isn’t is a permission system. It sits at the declaration end of a five-layer stack that runs through robots.txt, CDN and WAF enforcement, bot identity verification, and licensing terms, with real access control concentrated in the middle three layers, not the file getting most of the attention.
Getting that stack configured correctly answers one question: can AI systems reach the content at all. It doesn’t answer the more important one: once they can, do they actually recommend the brand. That second question needs its own audit trail, separate from anything a text file at the root of a domain can provide.
FAQ
What is llms.txt used for?
It’s a Markdown file that gives AI systems a curated list of a site’s most relevant content, meant to help agentic tools navigate faster. It doesn’t restrict access or function as a security control.
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers what they may not access and is broadly, though not universally, respected. llms.txt does the opposite: it suggests what to read first and carries no restrictive power at all.
Does Google support llms.txt?
No. Google has stated on record that it doesn’t support the format and has no plans to, comparing it to the deprecated keywords meta tag.
How do I check if AI crawlers are reading my llms.txt file?
Filter server access logs for requests to /llms.txt by known AI user-agents, or embed a unique link inside the file that only an automated reader would follow and monitor for traffic to that link.
