Troubleshooting hub

Cloudflare AI crawler troubleshooting hub

Use this hub when AI crawlers cannot reach your site reliably. Start with the matching failure pattern, then move to logs and rule tuning.

Step 1: Identify the failure class

• 5xx error page shown by Cloudflare: start with Cloudflare 520-526 differences.
• Unsure whether the issue is malformed response or timeout: use 520 vs 524 quick split.
• robots.txt allows crawl but logs show 403: start with OAI-SearchBot 403 behind Cloudflare.
• Zero Claude-SearchBot hits in 72h: use Claude-SearchBot zero-hit checklist.
• Need safe allow rules without opening sensitive paths: use allow OAI-SearchBot in Cloudflare WAF.

Example: choose by log symptom

If your logs show repeated Googlebot, OAI-SearchBot, or ClaudeBot requests with 200/304 on public pages, the crawler path is probably open. The next question is indexing, internal links, and source-page quality; use the Search Console submission workflow when deciding which source pages to inspect first. For Gemini policy decisions, separate crawl access from the Google-Extended robots token, then use the Google-Extended allow/block decision guide before changing rules.

If your logs show crawler requests only for /robots.txt and never for your question, guide, or product pages, check sitemap freshness, whether your important URLs are linked from crawlable HTML, and whether your llms.txt link list is focused enough. If requests appear but return 403, start with WAF events before editing robots.txt.

If most noise is scanner paths such as /.env, /wp-login.php, or phpunit, do not let those hits drive content work. We document this filtering approach in the LLMs File traffic baseline case study.

Fast triage table

Use this table before changing robots.txt or Cloudflare rules. It separates crawler discovery problems from origin reliability problems, which prevents broad allow rules from being used to fix unrelated TLS or timeout failures.

Minimal command sequence

When debugging a live site, collect these three views in the same time window. They show whether the failure is at Cloudflare, the origin, or the content-discovery layer.

# Public edge status for discovery files and one deep page.
for p in /robots.txt /sitemap.xml /llms.txt /questions/important-page/; do
  curl -L -sS -o /dev/null -w "%{http_code} %{url_effective}\n" "https://example.com$p"
done

# Origin log status mix for answer-engine crawlers.
grep -Ei 'OAI-SearchBot|GPTBot|ChatGPT-User|Claude|PerplexityBot' /var/log/nginx/access.log \
  | awk '{print $7, $9}' | sort | uniq -c | sort -nr | head -30

# Cloudflare 52x split, if an error page appears.
curl -IL https://example.com/questions/important-page/

If the public edge check fails, fix Cloudflare or origin first. If public status is healthy but crawler logs are empty, shift to sitemap, internal links, Search Console/Bing submission, and community distribution. If the discussion turns to llms.txt, set expectations with the llms.txt official-standard status page before treating it as a crawler guarantee.

Step 2: 520-526 fix pages

Step 3: Verify crawler access after fixes

1. Check security events for reduced blocks on public crawl paths.
2. Check origin logs for crawler status mix (200/304 should dominate).
3. Track /robots.txt, /sitemap.xml, /llms.txt, and key question pages for stable access over multiple days.

Use these pages as your post-fix verification path: how to check OAI-SearchBot visits, how to check GPTBot visits, how to check ChatGPT-User visits, how to test llms.txt after CDN changes, and PerplexityBot rollout decision guide.

When to stop editing Cloudflare

Stop editing Cloudflare rules once public pages return stable 200/304, crawler allow/deny behavior matches the intended policy, and origin logs show no crawler-specific failures. At that point, more WAF tuning is unlikely to create traffic. Move to submitting the priority URLs in Search Console and Bing Webmaster Tools, then publish the community snippets for the top demand pages.

Generate a Cloudflare command checklist