Your SEO lead pings you asking whether the site needs an llms.txt file. You check Google’s own AI optimization guide and it says skip it. Then you run a Lighthouse report that same afternoon and see an audit flagging the exact same file. Same company, two contradicting answers, eight days apart. The deadline to make a call on this is this sprint, not next quarter.
Two Google Teams, Two Opposite Instructions on llms.txt
Here’s the timeline that started the confusion. On May 7, 2026, Chrome’s Lighthouse 13.3 promoted a new Agentic Browsing category from experimental to default, and one of its checks looks for an llms.txt file at the root of the site.
Eight days later, on May 15, Google Search Central published its first consolidated AI optimization guide. Buried in a mythbusting section, the guide states plainly that site owners don’t need new machine-readable files, AI text files, markup, or Markdown to appear in generative AI search.
That’s not a policy reversal. It’s two product teams publishing their positions in the same month for the first time. One team is telling you to skip a file. The other team just started grading you on whether you have it.
Why Search Says Skip It
Google’s Search Central guide groups llms.txt with content chunking, AI-specific rewriting, and inflated structured data as tactics that don’t move the needle for AI Overviews or AI Mode. The reasoning is straightforward. Googlebot already renders and reads your actual HTML, so a separate summary file adds nothing it doesn’t already have.
This isn’t a new stance dressed up as news. Search Advocate John Mueller has compared llms.txt to the keywords meta tag, a tag search engines stopped trusting over a decade ago because anyone can write anything into it with zero verification. Back in July 2025, he was already suggesting sites noindex the file so it doesn’t accidentally get indexed and confuse users. Gary Illyes went further at Search Central Live APAC, confirming Google doesn’t support llms.txt for ranking and has no plans to start.
None of that changed on May 15. It just became official documentation instead of a forum reply.
Why Chrome Audits It Anyway
The Lighthouse Agentic Browsing category isn’t measuring what Search measures. It checks four things: WebMCP integration, agent accessibility, layout stability, and llms.txt. None of these produce a weighted 0-100 score the way Performance or SEO categories do, because Google says the standards for the agentic web are still forming.
The llms.txt check specifically looks at whether an AI browsing agent, the kind that fills out forms or compares products on a user’s behalf, can find a quick summary of your site’s structure without having to crawl every page first. Chrome’s own documentation frames the absence of the file as a minor tax: agents “may spend more time crawling the site to understand its structure” without one.
That’s a real but narrow use case. It has nothing to do with whether ChatGPT or Google’s AI Overviews decide to cite your brand.
The Real Signal Google Is Betting On: WebMCP, Not llms.txt
Most coverage of this split gets stuck arguing about a text file and misses the more telling detail sitting right next to it in the same audit category. WebMCP, the Web Model Context Protocol, is the second item Lighthouse checks, and it’s where Google appears to be putting its actual weight.
WebMCP lets a site declare structured “tool contracts” directly in HTML attributes or JavaScript, so an agent can execute an action, book a slot, add to cart, submit a form, inside a live session instead of scraping the DOM or driving the UI through screenshots. It showed up in Chrome Canary in February 2026, got featured at Google I/O the same year, and is now running an origin trial in Chrome 149.
A static file describing your site is a much smaller bet than a protocol that lets agents act on it. Chrome’s roadmap makes that priority obvious even if nobody’s saying it outright.
Does llms.txt Actually Get Used? The Bot Traffic Says No
Adoption numbers make the practical stakes clear. Roughly 10% of sites have already published an llms.txt file, but AI bots request it in only about 0.1% of cases. That gap between creation and actual use is close to the widest you’ll see for any SEO tactic in recent memory.
A broader Ahrefs study of over 137,000 domains found that 28% had published a valid llms.txt, yet 97% of those files got zero bot requests in a full month of tracking. Of the small slice that did see traffic, 96% came from bots generally, and under a fifth came from named AI tools, mostly coding agents like GPTBot and Claude-Code pulling developer docs.
A file nobody requests can’t be a citation signal. That’s the whole argument, reduced to one sentence.
A Decision Framework: When llms.txt Is Actually Worth It
Strip away the noise and the decision splits cleanly along one line: who’s actually consuming your site.
For most consumer-facing brands, e-commerce stores, SaaS marketing pages, local business sites, the file does nothing measurable. Google Search doesn’t read it for ranking, AI Overviews, or AI Mode, and the traffic data confirms almost nobody is fetching it anyway. Your time is better spent on crawlable pages, non-commodity content, and the standard SEO fundamentals the Search Central guide reaffirms.
For developer documentation, API references, or products where coding agents are a meaningful referral source, the calculus flips. Creating the file costs almost nothing, and Mueller himself acknowledged it can act as a token-saving shortcut for AI systems that already read your HTML fine but benefit from a simplified map.
Here’s the part that matters more than the file format either way. Whether or not you publish llms.txt, you still need to know what AI models are actually saying about your brand and which sources they’re pulling that from. That’s a data problem, not a file problem, and it’s the layer most teams have zero visibility into.
This is exactly the gap Topify is built to close. Its Source Analysis feature reverse-engineers the exact domains and URLs AI platforms cite when they answer questions in your category, so instead of guessing whether a machine-readable file helped, you see which pages actually earned the citation and which competitor’s content is filling the gap yours left open. Layered on top, Comprehensive GEO Analytics tracks visibility, sentiment, position, volume, mentions, intent, and CVR across ChatGPT, Gemini, and Perplexity in one view, which is the same foundational SEO signal set Google’s own guide points back to.
Conclusion
Google Search and Chrome aren’t contradicting each other so much as optimizing for different consumers of your site: one ranks crawlable HTML, the other checks readiness for browsing agents. For nearly every brand outside developer tooling, the answer is to skip the file and put that energy into content and crawlability that actually move AI citations. If you want to know whether that effort is working, get started with Topify and track the citations directly instead of betting on a file format nobody’s asking for yet.
FAQ
Q: Does llms.txt help with Google Search rankings or AI Overviews?
A: No. Google’s May 2026 AI optimization guide states directly that no special machine-readable files, AI text files, or Markdown are needed to appear in generative AI search features.
Q: Should developer documentation sites still create an llms.txt file?
A: It’s a reasonable low-cost option there. Coding agents like GPTBot and Claude-Code make up most of the small amount of real llms.txt traffic that exists, and Mueller has acknowledged it can help AI systems parse developer references more efficiently.
Q: Will Lighthouse’s llms.txt audit lower my SEO score if I don’t have the file?
A: No. The Agentic Browsing category doesn’t produce a weighted score like Performance or SEO categories do, and a missing file with a normal 404 response is marked Not Applicable rather than flagged as an error.
Q: What’s the difference between llms.txt and robots.txt?
A: robots.txt tells crawlers what they’re allowed to access and is respected across search engines. llms.txt is a self-declared summary of a site’s content with no verification mechanism, and no major AI vendor has committed to treating it as a ranking or citation signal.
