Google has added support for detecting llms.txt files in its Lighthouse auditing tool, signaling growing emphasis on making websites more interpretable by AI agents and large language models.
Announced via social media and covered in developer documentation, the update appears in Lighthouse version 13.3 under the new experimental Agentic Browsing category. This category evaluates how well sites support machine-driven interactions, such as those performed by autonomous AI agents.
What is llms.txt?
llms.txt is a plain-text file placed at the root of a domain (e.g., https://example.com/llms.txt). It provides a concise, machine-readable summary of a website’s business, products, services, and key pages—often in Markdown format with links. Proponents compare it to robots.txt (which guides crawlers on what to access) but positioned as a helpful guide for LLMs to understand context quickly and reduce hallucinations or inefficient scraping.
Without it, AI agents may need to crawl and parse full HTML pages, increasing computational overhead.
Lighthouse’s New Audit
The llms.txt audit falls under Lighthouse’s Agentic Browsing scoring, alongside checks for:
More Read
- WebMCP integration (for exposing site logic and forms to agents)
- Accessibility tree integrity
- Layout stability (Cumulative Layout Shift)
It flags the presence of the file at the domain root and notes server errors if retrieval fails. The category remains experimental and based on proposed standards, meaning scores in this area do not yet impact core web vitals or traditional search rankings.
Context and Caveats from Google
Google’s Search team has explicitly stated that llms.txt (and similar special files or markup) is not required for visibility in generative AI search features like AI Overviews. Core SEO practices—high-quality content, proper crawling/indexing, and structured data—remain the priority.
The Lighthouse addition focuses on agentic readiness (e.g., future browser-based AI assistants performing tasks on behalf of users) rather than ranking signals. Critics in discussions around the announcement noted it as low-priority or optional, with some accusing promoters of overhyping its immediate impact.
More Read
Practical Implications for Site Owners
Creating an llms.txt file takes minutes and could benefit AI tools that respect the emerging convention (e.g., certain agents or chat interfaces). Examples and generators are available online, and some companies have already implemented them for better AI citations.
However, experts advise against treating it as a replacement for strong fundamentals. As one SEO commenter put it in replies to the original post, good content and community presence still rank higher on the to-do list.
This development reflects the broader industry shift toward “AI Engine Optimization” (AEO) or Generative Engine Optimization (GEO), where websites increasingly optimize not just for traditional search crawlers but for LLM-powered agents. Whether llms.txt becomes as standard as robots.txt or remains a niche signal will depend on adoption by major AI providers like OpenAI, Anthropic, and Perplexity.
For developers and site owners, running a Lighthouse audit now offers an easy way to check compliance with this new experimental category. The full documentation is available on Chrome for Developers.
This story is based on Google’s official Lighthouse updates and contemporaneous reporting as of May 2026.
