How do I test llms.txt?

Confirm the file is available at /llms.txt, returns HTTP 200, contains stable canonical URLs, avoids thin or private pages, and does not point crawlers to URLs blocked by robots.txt or CDN security rules.

What is the most common llms.txt testing mistake?

The most common mistake is testing only that the file exists. You also need to test every listed URL, canonical behavior, robots.txt policy, redirects, and whether AI crawlers can fetch the pages in server logs.

Should every URL in llms.txt be indexed by Google?

No. llms.txt should point to useful source pages for AI and search systems, not every URL on the site. A URL can be valid for llms.txt even if you would not request indexing for it, but thin, duplicate, private, or unstable URLs should be removed.

Question

How to test llms.txt

Updated July 18, 2026

Test llms.txt as a source map, not just as a text file. A useful test checks whether the file is reachable, whether its links are canonical and crawlable, and whether those pages are good enough to quote in AI or search answers.

Fast pass: file availability

The file should live at the domain root and return a clean 200 without login, WAF challenge, or redirect chain.

curl -I https://example.com/llms.txt
curl -s https://example.com/llms.txt | head -40

Good signs: 200, readable text, stable cache headers, and links that point to public canonical pages. Bad signs: 403, 404, JavaScript challenge pages, or a file that only works on one host but not the canonical host.

Link checks

Extract every URL from the file and test it. Do not include URLs that redirect repeatedly, return errors, require JavaScript, or represent duplicate tracking variants.

curl -s https://example.com/llms.txt \
  | grep -Eo 'https?://[^ )]+' \
  | sort -u \
  | while read url; do
      printf '%s ' "$url"
      curl -s -o /dev/null -w '%{http_code} %{url_effective}\n' "$url"
    done

Audit script for a small production site

For a site with fewer than a few hundred listed URLs, this shell check is usually enough for a release gate. It finds missing files, redirects, non-HTML responses, and URLs that no longer match the canonical host.

site="https://example.com"
curl -fsSL "$site/llms.txt" -o /tmp/llms.txt || exit 1

grep -Eo 'https?://[^ )]+' /tmp/llms.txt | sort -u | while read url; do
  code=$(curl -L -sS -o /tmp/page.html -w '%{http_code}' --max-time 20 "$url")
  final=$(curl -L -sS -o /dev/null -w '%{url_effective}' --max-time 20 "$url")
  ctype=$(curl -L -sSI --max-time 20 "$url" | awk -F': ' 'tolower($1)=="content-type"{print $2}' | tail -1)
  canon=$(grep -oiE '<link[^>]+rel=["'\'']canonical["'\''][^>]+' /tmp/page.html | head -1)
  printf '%s\t%s\t%s\t%s\t%s\n' "$code" "$final" "$ctype" "$canon" "$url"
done

Review the output manually. A 200 result is not enough if the final URL is on the wrong host, the content type is a PDF you did not intend to list, or the canonical tag points somewhere else.

How to interpret failures

Finding	What it usually means	Fix
`/llms.txt` returns 404	The file was not deployed at the root path	Publish it at `https://example.com/llms.txt`, not under docs or assets
Listed URL redirects twice or more	Old URL, mixed apex/www, or HTTP-to-HTTPS chain	Replace it with the final canonical HTTPS URL
Canonical tag points elsewhere	The listed page is a duplicate or variant	List the canonical page instead
403 for normal curl	WAF, CDN challenge, login wall, or bot rule is blocking access	Allow public source pages without weakening private paths
Page is thin or mostly navigation	The URL is not a useful source for an answer engine	Remove it or replace it with a focused guide, FAQ, or product page

Canonical and duplicate checks

Each listed URL should represent one stable source page. If the page canonicalizes elsewhere, list the canonical version instead.

Prefer final canonical URLs over redirecting URLs.
Avoid campaign parameters, sort parameters, previews, tag archives, and search result pages.
Use the same host pattern as the rest of the site, such as apex only or www only.

Robots and CDN policy checks

A listed page is weak if the relevant crawlers cannot fetch it. Check robots.txt and the CDN layer together.

curl -s https://example.com/robots.txt | grep -Ei 'OAI-SearchBot|GPTBot|Claude|Perplexity|User-agent'

# Server log check after publication
grep -Ei 'OAI-SearchBot|GPTBot|ClaudeBot|PerplexityBot|ChatGPT-User' /var/log/nginx/access.log \
  | awk '{print $7, $9, $0}' \
  | tail -50

If you use Cloudflare, also confirm that Bot Fight Mode, WAF managed rules, and rate limits are not challenging the exact pages in llms.txt. A robots.txt allow rule does not override a CDN challenge.

Content quality review

Do not list every page. List pages that help a model or search crawler understand the site. Strong entries usually include one clear topic, visible text, recent facts where dates matter, and direct answers. Weak entries include thin category pages, duplicate location pages, pages with no stable content, or pages that are primarily ads.

Release checklist

File returns 200 at /llms.txt.
Every listed URL returns 200 or a single intentional redirect to the canonical URL.
No private, admin, staging, search, tag, preview, or tracking URLs are included.
Listed pages are not blocked by robots.txt or CDN security rules.
The file is linked from your sitemap, footer, docs, or AI policy page if it is part of your public workflow.

Maintenance cadence

Re-check after major content updates, URL migrations, robots policy changes, CDN/WAF changes, and template changes that affect canonical tags. For stable sites, a monthly review is enough; for active documentation or product sites, use a weekly check.

Related next steps: decide how many links to keep in llms.txt, confirm where to publish the file, and use Search Console submission for the source pages that deserve indexing checks.

Run validator now