GUIDE

What is llms.txt and do you need one

llms.txt is a 2024 Markdown index proposal, but Ahrefs found in May 2026 that 97% of valid files got 0 requests in a month.

Updated October 30, 20265 min read

97%Valid llms.txt files with 0 requests in a month (Ahrefs, May 2026)
2MAI agent requests analyzed with 0 asking for llms.txt (Readable, 2026)
3.3xOdds of ChatGPT naming plumbers on cited pages (FoundByChat, 2026)

In short

Jeremy Howard (Answer.AI, fast.ai) proposed llms.txt on September 3, 2024, as a Markdown site summary at /llms.txt for language models.
Ahrefs studied 137,210 domains (May 2026 data): 28% had a valid llms.txt, and 97% of those received zero requests that month.
OpenAI, Anthropic, and Perplexity have not stated they read third-party llms.txt files, and Google's 2026 docs say Google Search ignores them.
For ChatGPT search, allow OAI-SearchBot in robots.txt, keep facts in plain HTML, and earn mentions on pages ChatGPT already cites.

Studies found few or no requests for unannounced llms.txt files in the samples they examined.

What is llms.txt, who proposed it, and how it is formatted

llms.txt is a proposed standard placing a Markdown directory of a site's pages at /llms.txt. Jeremy Howard (Answer.AI, fast.ai) proposed it on September 3, 2024. Under the v2 specification at llmstxt.org (which also allows subpath files like /docs/llms.txt), the file uses:

An H1 heading with the site name.
A blockquote site summary.
Optional body details.
H2 sections with Markdown links to key pages, plus a skippable ## Optional section.

Generators and a minimal llms.txt example

Per llmstxt.org, generators creating llms.txt automatically include Yoast SEO, AIOSEO, Mintlify, and GitBook. To write it yourself, save it at /llms.txt (or a subpath like /docs/llms.txt under v2) with this structure:

# Example Site Documentation

> Guides on OpenAI crawlers, JavaScript rendering, and generative engine optimization for ChatGPT search.

Optional details about how pages are organized.

## Guides
- [OpenAI Crawlers](/guides/openai-crawlers): OAI-SearchBot, GPTBot, and ChatGPT-User roles.
- [JavaScript SEO](/guides/javascript-seo): Keeping text in initial HTML.

## Optional
- [AI Crawlers List](/guides/ai-crawlers-list): Search and training bots.

What the data says about llms.txt adoption and crawler requests

In Ahrefs' May 2026 data, 28% of the domains studied had a valid llms.txt, and 97% of those files received no requests that month. Neither OpenAI, Anthropic, nor Perplexity has stated in their docs (as of June 5, 2026) that their web search systems read third-party llms.txt files.

Source and date	Sample size	Key finding on llms.txt
Ahrefs (May 2026 data, published Jun 15, 2026)	137,210 domains	28% had a valid file, 97% of those got zero requests that month, and Slackbot fetched it more than PerplexityBot.
Readable (Feb 20, 2026)	2 million AI agent requests	Zero requests asked for `llms.txt`.
Zenodo preprint (Oct 3, 2026)	7 websites	An unannounced `/llms.txt` was requested twice; the same file announced in `robots.txt` was fetched 1,549 times (1,131 by ClaudeBot).

In the Ahrefs study (updated August 27, 2026), no crawler requested non-existent llms.txt files, in the domains studied. In the October 3, 2026 Zenodo preprint, the author called the 1,549 fetches of the file announced in robots.txt "discovery, not adoption".

Google's 2026 AI optimization guide states you do not need llms.txt or other AI text files for Google Search, noting "Google Search itself doesn't use them" and "Google Search ignores them". On April 17, 2025, Google's John Mueller stated: "AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag". Google's Gary Illyes made the same comparison to the 1990s keywords meta tag on August 1, 2025.

What to fix instead for ChatGPT search

To show up in ChatGPT search, focus on how OpenAI crawlers fetch and index your regular web pages. OpenAI's publisher FAQ (October 2026) notes that any public website can appear in ChatGPT search if you do not block its search crawler.

Allow OAI-SearchBot in robots.txt. OpenAI documents (October 2026) that OAI-SearchBot (OAI-SearchBot/1.4; +https://openai.com/searchbot) surfaces websites in ChatGPT search. Opted-out sites are excluded from search answers, though they can still appear as navigational links. After a robots.txt change, ChatGPT search can take about 24 hours to adjust.
Separate search access from training controls. GPTBot (GPTBot/1.4; +https://openai.com/gptbot) crawls content that may be used to train generative AI foundation models, while ChatGPT-User (ChatGPT-User/1.0; +https://openai.com/bot) handles user actions in ChatGPT and Custom GPTs and does not decide Search inclusion. Settings are independent, though if you allow both OAI-SearchBot and GPTBot, OpenAI may reuse one crawl for both. Per OpenAI's October 19, 2026 guidance, ChatGPT advertisers must allow OAI-AdsBot (OAI-AdsBot/1.0).
Put facts in plain HTML. Keep your page title, H1, opening sentence, and facts up front in the HTML without JavaScript. Keep menu or cookie text out of the ~200-character snippet ChatGPT's search keeps.
Get listed on pages ChatGPT already cites. In FoundByChat's October 2026 local tests, businesses listed on pages ChatGPT cited had 3.3 times the odds of being named in the answer for plumbers and 2.1 times for dentists (for agencies, the 1.6 times lift was not statistically significant). This is core to what GEO is.

Common mistakes with llms.txt and AI crawlers

Mistaking OpenAI's own docs for crawler support. OpenAI, Anthropic, and Perplexity publish llms.txt for their own developer docs, which does not mean their search crawlers read yours.
Blocking OAI-SearchBot when trying to block training. Opting out of OAI-SearchBot excludes a site from ChatGPT search answers, though it can still appear as a navigational link. Disallow GPTBot to tell OpenAI the content should not be used for training while keeping OAI-SearchBot allowed.
Relying on robots.txt to hide a URL completely. OpenAI's October 2026 publisher FAQ notes a URL blocked in robots.txt but learned from a third-party search provider may still show as a bare link and page title; a crawler-readable noindex prevents that.

How to check your pages

Run a free GEO audit on your pages (up to 5 audits/day, no account needed). It fetches your page as OAI-SearchBot, rebuilds the title and ~200-character snippet ChatGPT keeps, looks the page up in ChatGPT, and scores your setup out of 100 across robots.txt rules (OAI-SearchBot, GPTBot, ChatGPT-User), non-JS HTML text, JSON-LD, and llms.txt.

To track pages over time, add your site and customer questions to FoundByChat (free to start). It runs those questions in ChatGPT with web search on every 2 hours (US English or Netherlands Dutch), lists your "Pages in ChatGPT's index" with the stored title and snippet, and emails you when a page enters the index or its snippet changes. Also check analytics for utm_source=chatgpt.com referrals.

FOUNDBYCHAT

Know the moment ChatGPT picks up your pages

We ask ChatGPT your questions every 2 hours and mail you what its index picked up or rewrote. Free to start: sign in with Google or an email link.

Start free Or run the free GEO audit

Questions

Does ChatGPT use llms.txt?

OpenAI has not stated that ChatGPT or OAI-SearchBot reads third-party llms.txt files. In Ahrefs' study of 137,210 domains using May 2026 data, 97% of valid llms.txt files received zero requests during the month.

Does Google Search use llms.txt?

No, Google's 2026 AI optimization documentation states that Google Search ignores llms.txt and other AI text files. Google's John Mueller (April 2025) and Gary Illyes (August 2025) both compared llms.txt to the keywords meta tag.

Should I delete my llms.txt file?

You do not need to delete an existing llms.txt file, especially if a generator like Yoast SEO, AIOSEO, Mintlify, or GitBook creates it automatically. However, it does not replace allowing OAI-SearchBot on your HTML pages.

Why did ClaudeBot fetch llms.txt in the Zenodo study?

In the October 3, 2026 Zenodo preprint across 7 sites, /llms.txt was requested twice when unannounced, and 1,549 times (1,131 by ClaudeBot) when announced in robots.txt. The author calls this "discovery, not adoption".