All blogs

SEO & GEO

llms.txt: what it is, whether AI search uses it, and how to add one

llms.txt is a proposed file that gives AI tools a clean, Markdown map of your website. This guide explains the format, what Google and the AI companies actually say about it, what server log studies show, and when it is worth the few minutes it takes to add one.

· 8 min read

Isometric glowing lime document file with heading and link lines, scanned by small crawler drones and linked to a browser window and server, with OpenAI, Claude and Google badges

llms.txt is a plain Markdown file at the root of a website, such as example.com/llms.txt, that gives AI tools a short summary of the site and a curated list of links to its most useful pages. It was proposed by Jeremy Howard in September 2024 and is not an official standard. Today it does very little for AI search visibility: Google says you do not need special AI text files to appear in AI Overviews or AI Mode, OpenAI and Anthropic control their crawlers through robots.txt, and an Ahrefs study of 137,210 sites found that 97% of llms.txt files received no requests at all in May 2026. It is still cheap to add and useful for documentation and coding agents, but the work that gets a business cited by ChatGPT, Claude, Perplexity and Google is crawler access, clear answers and trustworthy content.

What llms.txt is

Web pages are built for people, with navigation, scripts, pop-ups and layout around the actual content. Converting that into clean text is slow and error-prone for an AI system working within a limited context window. The llms.txt proposal offers a shortcut: one small Markdown file that says what the site is and points to the pages that matter, ideally in clean Markdown versions too.

It is often compared to robots.txt and sitemap.xml, but it does a different job. robots.txt tells crawlers what they may access, and a sitemap lists every URL for search engines to discover. llms.txt is an editorial summary: a short, curated reading list written for a language model.

The llms.txt format

The proposal at llmstxt.org defines a simple structure in Markdown. Only the first element is required, and the sections appear in this order.

  • An H1 heading with the name of the site or project, which is the only required part
  • A blockquote with a short summary of what the site is and the key facts needed to understand the rest
  • Optional paragraphs or lists with more detail, without any further headings
  • H2 sections, each holding a list of links in the form [page name](url), optionally followed by a colon and a short note
  • An H2 section called Optional for secondary links an AI tool can skip when it needs less context

What Google, OpenAI and Anthropic say

Google is the clearest. Its Search Central page on AI features says there are no additional requirements to appear in AI Overviews or AI Mode, and that you do not need to create new machine-readable files, AI text files or markup to appear in them. Google's Gary Illyes has also said, as reported by Search Engine Land, that Google does not support llms.txt and is not planning to.

OpenAI's crawler documentation explains how OAI-SearchBot (for ChatGPT search), GPTBot (for training) and ChatGPT-User (for pages a user asks about) work, and points site owners to robots.txt to control them. It does not describe llms.txt as something its crawlers use. Anthropic's guidance for site owners does the same for ClaudeBot, Claude-SearchBot and Claude-User. Anthropic does publish llms.txt files for its own documentation, which shows the format is useful for developer docs, not that its crawlers read other sites' files.

What crawler data shows

Server logs tell the same story. Ahrefs analysed 137,210 domains in May 2026 and found that 28% published an llms.txt file, but 97% of those files received no requests at all during the month. Of the requests that did arrive, AI retrieval bots, the kind that fetch pages to answer a question, made about 1.1%. SEO audit tools, general crawlers and tools studying llms.txt itself made far more. AI tools also never requested llms.txt on sites that did not have one, which suggests they are not looking for it.

The one group that does show interest is AI coding agents, which made the largest share of AI requests in the Ahrefs data. That fits the format's origin: it works well when a developer points an assistant at a documentation site and asks it to read the API reference.

Should your business add one?

For most business websites, llms.txt is a low-cost extra rather than a priority. It takes minutes to write, it does no harm, and it may help an AI agent or assistant that a person points at your site. It will not improve Google rankings or make ChatGPT cite you on its own, so it should never come before the work that does.

  • Worth adding: software products, APIs and documentation sites, where developers use coding assistants on your docs
  • Fine to add: service businesses that want a clean summary of services, approach and contact details in one place
  • Not a fix for: low AI visibility, missing citations or weak rankings, which come from access, content and authority

How to create an llms.txt file

Write it by hand rather than dumping every URL into it. A good file reads like a briefing note for a new colleague: what the business does, who it serves, and the ten to thirty pages that best explain it.

  • 1. Start with an H1 of your business name and a one or two sentence blockquote saying what you do and for whom
  • 2. Add H2 sections such as Services, Guides and Company, each with links to the key pages and a short note on what each covers
  • 3. Put lower-priority pages, such as older posts, under an Optional section
  • 4. Serve it as plain text at yoursite.com/llms.txt, and generate it from your content if your site changes often
  • 5. Optionally add llms-full.txt with the full text of key pages in Markdown, a common extension used by documentation platforms
  • 6. Check it loads without a login, redirect or cookie banner, and keep it in step with the live site

What actually gets you cited by AI search

AI search tools find and cite pages the same way they find anything else: by crawling them, then choosing passages that answer the question clearly and come from a source they trust. That means the essentials are allowing the search and user-fetch bots, such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and PerplexityBot, in robots.txt, and making sure your hosting or CDN does not block them. Then write pages that open with a direct answer, state facts with sources, and use structured data that matches what is on the page.

Our guide to getting cited by ChatGPT and Google AI Overviews covers that work in detail, and you can track AI search traffic in GA4 to see whether it is paying off. If you want an assistant to act in your systems rather than just read your site, that is a job for MCP, not llms.txt. For help with the whole picture, our websites and funnels service builds sites that are ready for both Google and AI search.

Frequently asked questions

What is llms.txt?

llms.txt is a Markdown file placed at the root of a website that gives AI tools a short summary of the site and a curated list of links to its most important pages. It was proposed by Jeremy Howard in September 2024 and is a community proposal, not an official web standard.

Does Google use llms.txt?

No. Google's Search Central documentation says you do not need to create new machine-readable files, AI text files or markup to appear in AI Overviews or AI Mode, and Google's Gary Illyes has said Google does not support it. Normal SEO practices are what count.

Do ChatGPT and Claude read llms.txt?

Neither OpenAI nor Anthropic says its crawlers use llms.txt, and both point site owners to robots.txt to control access. Server log studies show AI retrieval bots rarely request the file, although AI coding agents sometimes do when reading documentation.

Does llms.txt improve SEO or AI visibility?

There is no evidence that it does. An Ahrefs study of 137,210 sites found that 97% of llms.txt files received no requests in May 2026. Crawler access, clear answer-first content, structured data and authority have far more effect.

What is the difference between llms.txt and robots.txt?

robots.txt tells crawlers which parts of a site they may access and is respected by all major search and AI crawlers. llms.txt is an optional summary and reading list for language models and does not control access at all.

Is it worth adding llms.txt to my website?

It is worth a few minutes if you have documentation or want a clean summary of your business available to AI tools, and it does no harm. It should not take priority over allowing AI search bots in robots.txt and writing pages that answer questions clearly.

Sources

  1. The /llms.txt file, llmstxt.org
  2. AI Features and Your Website, Google Search Central
  3. Google says normal SEO works for ranking in AI Overviews and LLMS.txt won't be used, Search Engine Land
  4. We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read, Ahrefs
  5. Overview of OpenAI Crawlers, OpenAI
  6. Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic

Next step

Want your site cited by ChatGPT and Google AI Overviews?

Book a free 30-minute call and we will check your crawler access, content and structured data, and show you what would make AI search tools cite your business.