LLMs File crawler insights report
Date: 25/Jul/2026

Crawler totals
130 Bingbot
29 ClaudeBot
14 ChatGPT-User
9 Googlebot
5 OAI-SearchBot
2 PerplexityBot
2 GPTBot

Top paths by crawler
9 ClaudeBot || /sitemap.xml
9 ClaudeBot || /robots.txt
9 ClaudeBot || /links-map.txt
4 OAI-SearchBot || /robots.txt
3 ChatGPT-User || /guides/oai-searchbot-gptbot-chatgpt-user/
3 Bingbot || /questions/how-to-fix-cloudflare-525-with-caddy/
2 Googlebot || /robots.txt
2 ChatGPT-User || /guides/perplexitybot-robots-txt/
2 ChatGPT-User || /guides/claude-robots-txt/
2 ChatGPT-User || /api/config
2 Bingbot || /questions/what-is-geo-generative-engine-optimization/
2 Bingbot || /questions/should-i-block-bytespider/
2 Bingbot || /questions/should-i-allow-meta-externalagent/
2 Bingbot || /questions/should-i-allow-google-extended/
2 Bingbot || /questions/how-to-verify-ai-crawler-authenticity/
2 Bingbot || /questions/how-to-track-ai-search-referrals/
2 Bingbot || /questions/how-to-get-cited-by-chatgpt/
2 Bingbot || /questions/how-to-fix-cloudflare-521-web-server-down/
2 Bingbot || /questions/does-llms-txt-help-seo/
2 Bingbot || /questions/can-robots-txt-block-chatgpt-user/
2 Bingbot || /questions/can-llms-txt-replace-sitemap-xml/
2 Bingbot || /policies/how-to-audit-ai-crawler-policy/
2 Bingbot || /policies/allow-perplexitybot-template/
2 Bingbot || /platforms/shopify-llms-txt/
2 Bingbot || /industries/local-services-llms-txt/
2 Bingbot || /comparisons/llms-txt-vs-llms-full-txt/
2 Bingbot || /comparisons/geo-vs-seo/
2 Bingbot || /about/
2 Bingbot || /
1 PerplexityBot || /robots.txt
1 PerplexityBot || /platforms/nextjs-llms-txt/
1 OAI-SearchBot || /
1 GPTBot || /sitemap.xml
1 GPTBot || /links-map.txt
1 Googlebot || /zh/
1 Googlebot || /use-cases/llms-txt-generator-for-b2b-lead-generation-site/
1 Googlebot || /_rNd9xZ7kL3
1 Googlebot || /guides/oai-searchbot-gptbot-chatgpt-user/
1 Googlebot || /.env.local
1 Googlebot || /.dev.vars
1 Googlebot || /
1 ClaudeBot || /questions/what-is-applebot-extended/
1 ClaudeBot || /
1 ChatGPT-User || /secrets.json
1 ChatGPT-User || /.pypirc
1 ChatGPT-User || /.npmrc
1 ChatGPT-User || /.env.test
1 ChatGPT-User || /config.yml
1 Bingbot || /zh/questions/ruhe-xiufu-cloudflare-525-nginx/
1 Bingbot || /zh/hubs/cloudflare-ai-pachong-paicha/
1 Bingbot || /sitemap.xml
1 Bingbot || /robots.txt
1 Bingbot || /questions/what-is-google-extended/
1 Bingbot || /questions/what-is-claudebot/
1 Bingbot || /questions/what-is-claudebot
1 Bingbot || /questions/what-is-ccbot/
1 Bingbot || /questions/what-is-aeo-answer-engine-optimization/
1 Bingbot || /questions/should-i-block-amazonbot/
1 Bingbot || /questions/should-i-allow-perplexitybot/
1 Bingbot || /questions/should-i-allow-gptbot/

Status code mix by crawler
88 Bingbot 200
24 Bingbot 304
21 ClaudeBot 200
18 Bingbot 308
8 ClaudeBot 304
7 ChatGPT-User 404
7 ChatGPT-User 200
4 OAI-SearchBot 304
3 Googlebot 404
3 Googlebot 304
3 Googlebot 200
2 GPTBot 200
1 PerplexityBot 304
1 PerplexityBot 200
1 OAI-SearchBot 200