What it is
llms.txt is a machine-readable text file that sits at the root of a domain, typically at example.com/llms.txt. The format was proposed in 2024 as a way for site owners to guide large language model crawlers toward the content that best represents their organization. Unlike robots.txt, which only expresses crawl permission, llms.txt can express intent. It can list the most important pages, describe the site's purpose, and indicate which sections contain authoritative answers for common queries. The file uses a simple markdown-inspired syntax with sections for a site summary, key links, and optional notes for model trainers.
The proposal emerged because AI crawlers behave differently from search crawlers. A search indexer wants breadth. An answer engine wants the single best passage. Without guidance, a model crawler may waste budget on paginated archives, login walls, or duplicate content, then miss the practice-area page that should be cited. llms.txt acts as a curated table of contents for the model, not a directive the crawler must obey.
Why it matters for AI visibility
For law firms and healthcare practices, citation-worthiness depends on the model finding the right page at the right time. When a prospective client asks ChatGPT for a medical malpractice attorney in Chicago, the model retrieves from its index or from a live crawl. If the firm's site buries the relevant page under three layers of navigation, the crawler may never reach it before the token budget expires. An llms.txt file that surfaces the practice-area page, the attorney bios, and the case-results page increases the odds that the model quotes the firm instead of a directory aggregator.
The file also supports entity disambiguation. A firm named "Smith & Associates" competes with dozens of similar names. An llms.txt entry that includes the firm's legal name, bar numbers, and office addresses helps the model resolve the entity to the correct knowledge-graph node. This connects to schema-org markup on the page level. The two signals reinforce each other: structured data describes the page, llms.txt describes the site.
Crawler behavior varies. GPTBot, ClaudeBot, and PerplexityBot each interpret the file differently. Some treat it as a hint for live retrieval. Others may use it to prioritize URLs during training-data collection. The training-vs-retrieval-crawlers distinction matters here. A retrieval crawler fetching a page for a live answer benefits from a current llms.txt. A training crawler building a snapshot benefits from a stable llms.txt that reflects the site's enduring structure. Maintaining both signals is part of a complete LLMO strategy.
Where people get it wrong
The most common mistake is treating llms.txt as a replacement for robots.txt or sitemap.xml. It is neither. Robots.txt governs permission. Sitemaps govern discovery. llms.txt governs priority and context. A site that blocks GPTBot in robots.txt but lists pages in llms.txt sends a contradictory signal. The crawler will follow robots.txt first.
Another error is listing every URL. A file with five hundred links defeats the purpose. The crawler has a token budget. The file should contain the twenty to fifty pages that define the practice: core services, key attorneys, landmark results, and the contact page. Pages behind authentication, paginated blog archives, and thin category tags should be omitted.
Some firms generate the file once and forget it. When a new partner joins or a practice area launches, the llms.txt file must update. An outdated file points the model to stale bios or retired services, which reduces citation-worthiness and can trigger hallucination when the model fills gaps with plausible but incorrect details.
A third mistake is assuming the file guarantees citation. It does not. The model still evaluates content quality, authority signals, and relevance to the prompt. llms.txt improves the odds that the model sees the best evidence. It cannot compensate for thin content, missing schema-org, or a site that blocks AI crawlers entirely.
What to do about it
Start by auditing whether your site serves an llms.txt file. The AI Crawler Access Check (/tools/crawler-access) reports whether the file exists, whether it is valid, and whether major crawlers can fetch it. If the file is missing, use the llms.txt Generator (/tools/llms-txt-generator) to create a draft. The generator asks for your core pages, your firm or practice summary, and any special instructions for crawlers. It outputs a file you can review, edit, and deploy to the root.
Next, align the file with your on-page signals. Each URL listed in llms.txt should carry schema-org markup that matches the page type: LegalService, MedicalBusiness, Person, or Article. The Answer Visibility Meter (/tools/answer-visibility) can test whether those pages are being cited for your target queries. If a listed page never appears in answers, investigate content depth, canonical-url configuration, and whether the page is inadvertently blocked by robots.txt or meta tags.
Treat the file as a living document. When you publish a new practice-area page, add it. When an attorney leaves, remove the bio. When you restructure navigation, update the hierarchy in the file. The LLMO Readiness Score (/tools/llmo-score) includes an llms.txt check as part of its weekly sweep, so drift is caught automatically.
Finally, coordinate llms.txt with indexnow and your XML sitemap. IndexNow pushes URL changes to Bing and partner engines in real time. The sitemap gives search crawlers a complete map. llms.txt gives answer-engine crawlers a curated map. The three files work together. A change to the site structure should trigger updates to all three. This discipline keeps the signals consistent across search and answer engines, which is the foundation of durable AI visibility for regulated practices.