Is llms.txt still worth publishing?

Yes, if you treat it as a low-cost source map for your best public pages. It can help tools and future crawlers understand what matters, but it does not replace robots.txt, sitemap.xml, crawlable HTML, or indexing fundamentals.

Does llms.txt control crawler access?

No. llms.txt is not an access-control file. Use robots.txt, authentication, and CDN or application rules to control crawler access. Use llms.txt only to summarize and point to useful public source pages.

What if a third-party site calls llms.txt a standard?

Read that as a vendor or community specification unless it points to a ratified W3C Recommendation, IETF RFC, or a crawler-provider commitment. A third-party stable spec can be useful, but it is not the same as a formal web standard.

Question

Is llms.txt an official standard?

Q: Is llms.txt an official standard?

No. llms.txt is a community proposal and emerging convention. It is not a W3C or IETF standard, and it should not be treated as a guaranteed ranking or crawler-control mechanism.

Updated July 25, 2026

No. llms.txt is best described as a community proposal and emerging convention, not a ratified web standard. The original proposal was published by Jeremy Howard / Answer.AI, but it has not become a W3C standard, an IETF RFC, or a guaranteed protocol that AI providers must follow.

That distinction matters. You can publish /llms.txt because it is cheap and useful as a curated source map, but you should not sell it internally as a direct ranking lever or a binding crawler instruction.

Status snapshot

Question	Current answer	Operational meaning
Formal W3C or IETF standard?	No.	There is no standards-body requirement for crawlers to consume it.
Community proposal?	Yes.	It is a documented convention that site owners and tools can adopt voluntarily.
Access-control file?	No.	Use `robots.txt`, authentication, CDN rules, and application controls for access decisions.
Replacement for sitemap.xml?	No.	Keep sitemap.xml for URL discovery and indexing workflows.
Worth publishing?	Often yes.	Use it as a small, curated map of public pages you want answer engines to understand.

What that means for implementation

Publish it at /llms.txt only if you can keep it accurate.
List your strongest canonical pages, not every URL on the site.
Use short descriptions that explain why each page is useful.
Keep robots.txt and Cloudflare/WAF policy separate from llms.txt.
Measure real crawler and referral behavior instead of assuming adoption.

How to judge standard claims

You may see third-party pages describe llms.txt as a specification, stable format, or proposed standard. That language can still be useful, but it is not the same as formal standardization. Before making a product or compliance claim, check what kind of authority is behind it.

Claim type	How to interpret it	Risk if overstated
Answer.AI proposal	The original public proposal and reference implementation work.	Useful for implementation, but not binding on crawlers.
Third-party stable spec	A vendor or community can define a stable profile for its own tools.	May be mistaken for W3C/IETF standard status.
W3C Recommendation or IETF RFC	Formal standards-track output from a recognized standards body.	As of this update, `llms.txt` should not be described this way.
Major crawler/provider commitment	A provider documents that its crawler consumes or honors the file.	Without that commitment, adoption should be measured in logs, not assumed.

The safest external wording is: llms.txt is a community proposal for a voluntary source map that some tools may use. Avoid saying it is an official SEO requirement, a crawler directive, or a standards-body requirement.

How to explain it to a team

A practical internal description is: llms.txt is a voluntary source map for AI and agent tools. It helps us present our best public pages clearly, but it does not force any crawler to visit, index, quote, or rank those pages.

That framing keeps expectations realistic. It also avoids the common mistake of treating llms.txt like robots.txt. One is a policy/access file; the other is a curated context file.

When it is worth doing

You already have crawlable, useful pages worth summarizing.
Your sitemap and canonical tags are clean.
You want a maintainable source list for documentation, guides, FAQs, products, or policy pages.
You can review the file after major URL, product, or content changes.

When to skip or delay it

Your public pages are thin, duplicated, or mostly navigation.
You cannot decide which pages should represent the site.
You expect the file to solve indexing, ranking, or paywall policy problems by itself.
Your private/public content boundaries are not clear yet.

Verification checklist

curl -I https://example.com/llms.txt
curl -s https://example.com/llms.txt | head -40

# Check that listed URLs resolve cleanly.
curl -s https://example.com/llms.txt \
  | grep -Eo 'https?://[^ )]+' \
  | sort -u \
  | while read url; do
      curl -L -sS -o /dev/null -w "%{http_code} %{url_effective}\n" "$url"
    done

Useful references: the Answer.AI llms.txt proposal, the W3C standards and drafts index, how to test llms.txt, how many links to include, and whether ChatGPT uses llms.txt.

Generate your llms.txt