llms.txt: What It Is, How to Deploy It, What to Expect
llms.txt without the hype: the spec, a WordPress how-to, ready-made templates. And honestly: 97% of llms.txt files never get read by AI crawlers.
Ship it in an hour, expect nothing from it
llms.txt is a Markdown file at your domain root that gives AI models a curated map of your content. Deploying it takes an hour and breaks nothing. But honestly: according to Ahrefs (06/2026, 137,000 domains), 97% of llms.txt files receive zero requests from AI crawlers. Ship it, expect nothing from it, and put your energy into content.
of llms.txt files got zero requests from AI crawlers (Ahrefs, 06/2026)
monthly searches for “llms txt” in EN markets
year-over-year growth in search demand
is all it takes to deploy, review included
What is llms.txt?
llms.txt is a plain Markdown file served at yourdomain.com/llms.txt. It contains your site name, a short summary, and a curated list of your most important pages with descriptions. The goal is to give language models a clean overview of your content without the navigation menus, cookie banners, and JavaScript that make regular pages hard to parse.
The standard was proposed in September 2024 by Jeremy Howard, co-founder of Answer.AI and fast.ai. The logic: AI models have limited context windows and modern web pages are full of noise, so a curated Markdown summary saves them work.
Interest in the topic is exploding:
| Keyword | Market | Searches/mo | YoY growth |
|---|---|---|---|
| llms txt | EN (DACH+NL) | 3,600 | +266% |
| generative engine optimization | EN (DACH+NL) | 2,400 | +268% |
| answer engine optimization | EN (DACH+NL) | 390 | +143% |
Source: Google Keyword Planner, 07/2026. llms.txt out-searches the term generative engine optimization itself.
And that is exactly where the problem starts: demand for llms.txt is growing by hundreds of percent, while the evidence that the file actually does anything is thin. This article is a deployment guide and a reality check in one.
What does the llms.txt specification look like?
The spec is deliberately simple: one H1 with the project name, a blockquote summary, an optional paragraph of context, and H2 sections containing bullet lists of links. Each link carries a short description after a colon. A section named "Optional" marks content a model can skip when context is tight.
The structure per llmstxt.org:
- H1: the site or project name (the only required element)
- Blockquote: a one-to-two sentence summary of the site
- Free text: additional context without a heading
- H2 sections: thematic link groups in the format [title](url): description
- "Optional" section: lower-priority links
Alongside llms.txt there is llms-full.txt: one large Markdown file with the complete content of the site. It makes sense mainly for technical documentation (Anthropic, Stripe, and Vercel publish one); for a typical company site or online store, the llms.txt index is enough.
How is llms.txt different from robots.txt and sitemap.xml?
Each of the three files does a different job: robots.txt tells crawlers where they may go, sitemap.xml enumerates all URLs for indexing, and llms.txt offers a curated, readable summary for language models. llms.txt replaces neither of the other two.
| robots.txt | sitemap.xml | llms.txt | |
|---|---|---|---|
| Purpose | crawler access control | URL list for indexing | curated summary for AI |
| Format | directives (plain text) | XML | Markdown |
| Audience | all crawlers | search engines | language models |
| Status | de facto standard, respected | officially supported | proposal without official support |
| Impact on AI citations | critical (blocking = invisibility) | indirect | unproven |
The practical takeaway: if you care about visibility in AI answers, robots.txt matters a hundred times more than llms.txt. A blocked GPTBot means you do not exist for ChatGPT. A missing llms.txt means nothing.
Do AI crawlers actually read llms.txt?
Mostly no. In June 2026, Ahrefs analyzed 137,000 domains with an llms.txt file and found that 97% of them received not a single request for the file from AI crawlers. No major AI operator (OpenAI, Google, Anthropic) has officially confirmed using llms.txt for search or citations.
97% of llms.txt files received zero requests from AI crawlers (Ahrefs, 06/2026, sample of 137,000 domains).
Additional sobering signals:
- Google's John Mueller compared llms.txt to the keywords meta tag: a site making claims about itself that systems have no reason to trust (public statement, 2025).
- Google confirmed that neither AI Overviews nor AI Mode uses llms.txt.
- What demonstrably drives AI citations is different: content with statistics and sources (+30 to 41% visibility, Princeton KDD 2024), answer-first structure, and presence in Bing's index.
We run llms.txt on mairateam.com ourselves and watch the crawl logs. For context, here are the crawlers you should track in your logs. Their behavior on your regular pages (not on llms.txt) is what decides your visibility:
| Crawler | User-agent | Operator | Purpose |
|---|---|---|---|
| GPTBot | GPTBot | OpenAI | model training |
| OAI-SearchBot | OAI-SearchBot | OpenAI | ChatGPT search |
| ClaudeBot | ClaudeBot | Anthropic | Claude training and answers |
| PerplexityBot | PerplexityBot | Perplexity | Perplexity search index |
| Google-Extended | Google-Extended | Gemini training (a directive, not a bot) | |
| Bingbot | bingbot | Microsoft | Bing index, the backbone of ChatGPT search |
Why deploy llms.txt anyway?
Three reasons: it costs an hour, it carries zero risk, and the situation can change. Standard adoption is unpredictable; sitemap.xml also started as one company's proposal. If any major player switches support on, you are already done while others scramble.
- Zero risk: llms.txt does not affect SEO, site speed, or indexing in any way. There is nothing to lose.
- Low cost: for a site under a hundred pages, it is an hour of work including review.
- A future option: smaller AI tools and agents already read llms.txt today, and the big players can add support at any time.
- A useful side effect: writing llms.txt forces you to name your most important pages and how to describe them. That pays off beyond AI.
The fair summary: llms.txt is a cheap bet with a small chance of paying off, not an investment with a guaranteed return. Anyone selling you llms.txt as the key to AI visibility is selling hype.
How do you deploy llms.txt on WordPress?
The fastest path: create a text file named llms.txt, upload it via FTP or your file manager to the web root, and verify it loads at yourdomain.com/llms.txt. Alternatively, use a plugin that generates the file from your content automatically.
- Write the content using the template below (name, summary, link sections).
- Save as llms.txt in UTF-8 encoding.
- Upload to the web root (the same place robots.txt lives).
- Verify in a browser that the URL returns plain text, not a 404 or a redirect.
- Update it with every major content change, at least quarterly.
WordPress plugins can automate this (Yoast SEO generates llms.txt since version 26, AIOSEO does too). A plugin makes sense for large sites with frequently changing content. For a smaller site we recommend a hand-curated file: it beats an auto-generated dump of every URL.
What does a ready-to-use llms.txt template look like?
Below are templates for an online store and for an agency. Replace the bracketed text, delete the sections you do not need, and upload. Follow one rule: fewer links with better descriptions beat a complete export of all URLs.
Template for an online store:
# [Store name]
> [One sentence: what you sell, for whom, in which markets. E.g.: Outdoor gear store shipping to Germany, Austria, and the Netherlands.]
[Optionally 2-3 sentences of context: since when, what makes you different, key brands.]
## Categories
- [Category name](https://www.example.com/category-1/): [what the customer finds there]
- [Category name](https://www.example.com/category-2/): [what the customer finds there]
## Buying guides
- [Guide title](https://www.example.com/blog/guide/): [what question it answers]
## Service and policies
- [Shipping and payment](https://www.example.com/shipping/): costs, carriers, delivery times
- [Returns](https://www.example.com/returns/): deadlines and process
## Optional
- [About us](https://www.example.com/about/): history and teamTemplate for an agency or B2B services:
# [Company name]
> [One sentence: what you do, for whom, with what results. E.g.: Performance marketing agency managing 52M+ EUR in annual budgets across 30+ markets.]
## Services
- [Service name](https://www.example.com/service-1/): [for whom and with what outcome]
- [Service name](https://www.example.com/service-2/): [for whom and with what outcome]
## Case studies
- [Client + result](https://www.example.com/case-study-1/): [metric, timeframe]
## Know-how
- [Key article](https://www.example.com/blog/article-1/): [what question it answers]
## Optional
- [Careers](https://www.example.com/careers/): open positions
- [Contact](https://www.example.com/contact/): inquiries and company detailsFAQ
Is llms.txt an official standard?
No. It is a community proposal by Jeremy Howard (Answer.AI) from September 2024. None of the major AI operators (OpenAI, Google, Anthropic) has officially confirmed using it. The format is stable and documented at llmstxt.org, so deploying it is safe.
Will llms.txt improve my Google rankings?
No. Google confirmed it does not use llms.txt for search, AI Overviews, or AI Mode. The file has no effect on SEO, positive or negative.
What is the difference between llms.txt and llms-full.txt?
llms.txt is the index: a curated list of your most important pages with descriptions. llms-full.txt is the complete site content in one Markdown file. Documentation sites benefit from both; an online store or company site only needs llms.txt.
Should I update robots.txt for AI instead?
Yes, and it matters more than llms.txt. Check that robots.txt does not block GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, or Google-Extended. A blocked crawler means invisibility in that assistant regardless of how good your content is.
How do I know if AI crawlers read my llms.txt?
Only from server logs: filter requests to /llms.txt by user-agent (GPTBot, ClaudeBot, PerplexityBot). Be prepared for reality: per Ahrefs (06/2026), 97% of llms.txt files get zero requests. Track crawler activity on your regular pages instead; that is what decides.
Is a paid llms.txt generator worth it?
No. For a smaller site, a hand-curated file written in an hour beats any automated export. On WordPress, Yoast or AIOSEO generate the file for free. Paying for a dedicated llms.txt tool makes no sense for a file that 97% of crawlers never read.
Why is everyone talking about llms.txt if crawlers ignore it?
Because demand ran ahead of evidence. Searches for llms txt grew 266% year over year to 3,600 per month (Google Keyword Planner, 07/2026). The topic is new, sounds technical, and promises a simple shortcut to AI visibility. Shortcuts do not work: citations are driven by content, structure, and mentions, not by a file.
What actually gets a site cited by ChatGPT or Perplexity?
Verifiable statistics with sources (+30 to 41% visibility lift, Princeton KDD 2024), direct answers in the first 30% of the page (where 44.2% of citations come from), data tables, and presence in Bing's index (~87% of SearchGPT citations match Bing top results).
Conclusion
Deploy llms.txt: an hour of work, zero risk, templates above. Then move your energy to where AI visibility is actually won: sourced statistics, answer-first structure, Bing and schema hygiene, mentions on third-party sites. Want to know how visible your site is in AI answers? Grab a free audit and we will walk through it, no strings attached.
Jan Křelina
PPC Specialist
At Proficio I handle PPC and META advertising for our clients across Europe. I love it when I see positive results of my work and when I manage to take my client’s online marketing one step further. I love learning new things and platforms and I’m really enjoying the combination of PPC advertising and AI agents. In my spare time I like to travel, watch soap operas or hole up in the spa.
