Question
Where to put llms.txt
Publish the file at your root path so assistants and crawlers can find it reliably. The expected URL is simple: https://yourdomain.com/llms.txt. If the file exists only inside docs, assets, or a framework route, many tools and crawlers will miss it.
Correct URL format
Your file should be available at https://yourdomain.com/llms.txt. Avoid deep paths like /docs/llms.txt unless you also maintain a root file.
Canonical deployment pattern by stack
- Static hosting: commit
llms.txtto the public root directory. - Nginx/Caddy: serve as a plain text/markdown file from site root.
- Framework apps: place in public assets so build/deploy preserves root path.
Correct location by hosting platform
| Platform | Put the file here | Verify this URL |
|---|---|---|
| Next.js | public/llms.txt | https://example.com/llms.txt |
| Astro, Vite, or SvelteKit static output | public/llms.txt or the static asset directory copied to the output root. | Root URL after the production build, not the dev server only. |
| WordPress | Document root beside robots.txt, or a plugin that serves the exact root path. | No login wall, no HTML theme wrapper, no redirect to an attachment page. |
| Caddy or Nginx on a VM | The configured site root, usually beside index.html. | Content-Type is text-like and status is 200. |
| Cloudflare Pages, Netlify, or Vercel | The deployed output root or public directory. | No Worker, Function, or redirect rule rewrites /llms.txt. |
Stack-specific examples
- Next.js: put the file in
/public/llms.txt, then deploy and check the root URL. - Vite/Astro: put it in the public directory, not inside source routes.
- WordPress: upload to the web root or use a plugin only if it serves the exact root path.
- Cloudflare Pages: include it in the output directory, then verify that no Worker route rewrites it.
Host consistency example
Pick one canonical host and keep llms.txt behavior identical there. A safe default is: www -> apex redirect, then serve /llms.txt from apex with 200.
# Check apex and www behavior together
curl -I https://yourdomain.com/llms.txt
curl -I https://www.yourdomain.com/llms.txt
Deployment checklist
- File exists at root path.
- Returns HTTP 200.
- Listed in robots.txt as a reference page.
- Contains only canonical URLs.
Common misdeploys to avoid
- Publishing only at
/docs/llms.txtwithout a root file. - Serving different content between apex and www.
- Returning
403from WAF challenge rules for crawler user agents. - Serving stale content from a CDN cache after the file was updated.
- Listing URLs that redirect several times or resolve to non-canonical hosts.
Should robots.txt mention llms.txt?
It is not required, but adding a reference can help humans and automated tools discover your policy files together. Keep robots permissions and llms.txt guidance separate: robots controls access; llms.txt maps the pages you want understood.
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
# Optional reference for humans and tooling:
# llms.txt: https://yourdomain.com/llms.txt
Fast verification commands
curl -I https://yourdomain.com/llms.txt
curl -s https://yourdomain.com/llms.txt | sed -n '1,80p'
curl -I https://www.yourdomain.com/llms.txt
Production checks that catch most mistakes
Run these checks against the public hostname after deployment, not against localhost. The most common failures are a root URL that redirects to HTML, a CDN rule serving an old file, or a framework route that works locally but is missing from the production build.
# Status, content type, final URL, and cache age
curl -L -sS -D - https://yourdomain.com/llms.txt -o /tmp/llms.txt \
| sed -n '1,20p'
# Confirm the file is text, not an HTML fallback page
file /tmp/llms.txt
sed -n '1,40p' /tmp/llms.txt
# Compare apex and www
curl -L -sS https://yourdomain.com/llms.txt | shasum -a 256
curl -L -sS https://www.yourdomain.com/llms.txt | shasum -a 256
| Bad result | Likely cause | Fix |
|---|---|---|
200 with full HTML | The app fallback route is catching /llms.txt. | Move the file to the public/static root or add an explicit file route before the fallback. |
301 to a deep docs page | The root discovery path is only a redirect. | Prefer serving the root file directly; link from it to deep docs if needed. |
Apex and www differ | Two hosts deploy from different roots or caches. | Choose one canonical host and make the other redirect or serve identical content. |
| Old content after deploy | CDN cache retained the previous file. | Purge /llms.txt or version the referenced source pages inside the file. |
If apex and www return different files, unify behavior before further SEO/GEO changes. Then confirm crawler hits with OAI-SearchBot log checks.
Next step: test llms.txt quality and set update cadence.