Skip to content
BEACON Guide

llms.txt: what it is and whether you need one.

llms.txt is a proposed Markdown file at your site root that gives language models a curated, machine-readable map of your most important content. It’s an emerging convention — not yet a universal standard — so here’s an honest look at what it does, what it doesn’t, and how to write one.

Updated · By the Beacon team

A curated indexA Markdown file at /llms.txt that lists your key pages and a short description of each, so models can find the good stuff fast.
Not robots.txtrobots.txt controls who may crawl; llms.txt suggests what matters and how to read it. They solve different problems.
Markdown, not XMLUnlike a sitemap, llms.txt is human-readable Markdown — a title, a blockquote summary, then linked sections.
Adoption is earlySupport varies by engine and isn’t guaranteed. Treat it as a low-cost bet, not a silver bullet.
Cheap to addIt takes minutes and can’t hurt — a clear map of your best content is good hygiene regardless.
Check if you have oneBeacon’s free scan reports whether /llms.txt exists and whether AI crawlers can reach your content.

What goes in llms.txt

The convention is simple Markdown: an H1 with your site name, a short blockquote summary, then H2 sections grouping links to your most important pages with one-line descriptions.

  • # Your brand — one-line H1
  • > A blockquote summarising what you do
  • ## Docs / Products / Guides sections
  • Bulleted links: [Page title](url): short description
  • Optionally an /llms-full.txt with the full text of key pages

llms.txt vs robots.txt vs sitemap.xml

robots.txt grants or denies crawl access (allow GPTBot, PerplexityBot, ClaudeBot and Google-Extended there). sitemap.xml lists every URL for completeness. llms.txt is the editorial layer — a short, prioritised, human-readable guide to what actually matters. Use all three; they don’t compete.

Is it worth it in 2026?

Honestly: it’s low-effort and low-risk, but adoption across engines is still uneven, so don’t expect dramatic results on its own. Prioritise the fundamentals first — unblock AI crawlers, fix the render gap, add structured data — then add llms.txt as a finishing touch.

FAQ

What is llms.txt?

A proposed Markdown file at your site root (/llms.txt) that gives AI models a curated, prioritised map of your most important content and how to read it.

Is llms.txt the same as robots.txt?

No. robots.txt controls crawl permissions; llms.txt is an editorial guide to your best content. They’re complementary.

Do AI engines actually use llms.txt?

Support is emerging and varies by engine — it isn’t a guaranteed, universal standard yet. It’s a cheap, sensible addition, but fix crawlability, rendering and schema first.

How do I check if my site has llms.txt?

Run Beacon’s free AI Visibility check — it reports whether /llms.txt exists alongside your AI-crawler access and on-page signals.

Keep reading

How to allow AI crawlers in robots.txt. Otterly.ai alternative: Beacon vs Otterly. Free SEO Checker — a real on-page audit in seconds. Keyword research grounded in your real site — with measured data.

Check your AI-readiness for free

See your llms.txt status, crawler access and schema in seconds.