Question

Should I allow PerplexityBot?

Updated July 20, 2026

For visibility in Perplexity-driven answers, allowing PerplexityBot is commonly the right direction.

When allowing is the better default

• You publish explainers, docs, or comparisons: allow on public pages so answer engines can fetch source context.
• You want traffic from question-style queries: allow on guides and troubleshooting pages first.
• You have mixed public/private content: allow only public sections and keep account/API paths blocked.

Decision matrix

• Need visibility, low policy risk: allow PerplexityBot on all public content.
• Need visibility, high policy risk: allow only selected path groups and review logs weekly.
• No visibility goal: block explicitly and document the tradeoff for the team.

Quick policy checklist

1. Add an explicit User-agent: PerplexityBot section.
2. Avoid accidental global blocking in the wildcard group.
3. Keep sitemap URLs accessible and current.
4. Validate live behavior with logs and status checks.

Safe rollout sequence

1. Start with docs/guides/policy pages only.
2. Watch status mix for 3-7 days (200/304 should dominate).
3. Expand to broader public content after stable logs.

Practical rollout examples

• Local services site: allow /services/, /faq/, and /pricing/; keep account and checkout paths blocked.
• Documentation site: allow /docs/ and /changelog/; monitor if recently published docs start appearing in AI-answer referrals.
• Publisher/blog: allow category pages and evergreen explainers first; add breaking-news URLs after confirming normal crawler status mix.

Concrete robots.txt examples

For a visibility-first site, keep PerplexityBot open on the pages that explain your product, policies, examples, and use cases. Do not let the exception drift into authenticated or transactional areas.

User-agent: PerplexityBot
Allow: /robots.txt
Allow: /sitemap.xml
Allow: /llms.txt
Allow: /questions/
Allow: /guides/
Allow: /docs/
Allow: /case-studies/
Allow: /use-cases/
Disallow: /account/
Disallow: /admin/
Disallow: /checkout/
Disallow: /api/
Disallow: /internal-search/
Disallow: /staging/

For a publisher or paid course site, use a stricter split. Let public landing pages and free explainers be crawlable, but keep member content and licensed archives out of the policy.

User-agent: PerplexityBot
Allow: /free-guides/
Allow: /topics/
Allow: /course/
Disallow: /course/lessons/
Disallow: /members/
Disallow: /paid-reports/
Disallow: /licensed-content/

When not to allow it yet

Delay allowing PerplexityBot if public/private boundaries are unclear, if the site relies on paywalled content, or if Cloudflare/WAF rules cannot distinguish crawler fetches from spoofed user agents. Fix the boundary first, then allow a narrow path set and monitor logs.

Minimum safe allow list

If you are unsure how broad the rule should be, start with a small set that contains only discovery files and high-quality public source pages. This gives Perplexity a way to understand the site without granting crawler access to every URL pattern.

Path group	Default	Reason
/robots.txt, /sitemap.xml, /llms.txt	Allow	Discovery and source-map files should be reachable.
/guides/, /questions/, /docs/	Allow	These are usually the pages answer engines can quote usefully.
/case-studies/, /use-cases/, public product pages	Allow after review	Good for context, but remove thin or duplicate variants first.
/account/, /admin/, /checkout/, /api/	Block	These paths create security or privacy risk and rarely help answer visibility.
Paid lessons, member reports, licensed archives	Block by default	Do not expose commercial content unless the business has approved it.

After the first week, widen the rule only for path groups that produce successful fetches and relevant referral landings. Do not expand simply because the crawler can fetch more pages.

What to track in week one

1. PerplexityBot status ratio (200/304 vs 403/429).
2. Top fetched paths (are they the pages you intended to expose?).
3. Any referral uptick from answer engines to those same paths.

Audit for spoofed user agents

A robots.txt rule is public, and user-agent strings can be copied. Treat PerplexityBot log lines as candidates until the source looks consistent with a real crawler and the paths match your allow list.

# Group PerplexityBot-looking requests by IP, status, and path.
grep -i 'PerplexityBot' /var/log/nginx/access.log \
  | awk '{print $1, $7, $9}' \
  | sort | uniq -c | sort -nr | head -50

# Red flags:
# - same user agent probing /wp-admin, /.env, /api, or login paths
# - high 404/403 volume from one address
# - no requests to discovery files or public content paths

If the traffic looks suspicious, narrow the allow list instead of blocking all public pages. The goal is a controlled visibility path, not an all-or-nothing crawler policy.

Verification commands

# PerplexityBot path distribution
grep -i 'PerplexityBot' /var/lib/caddy/logs/llmsfile-access.log | awk '{print $7}' | sort | uniq -c | sort -nr | head -n 20

# PerplexityBot status mix
grep -i 'PerplexityBot' /var/lib/caddy/logs/llmsfile-access.log | awk '{print $9}' | sort | uniq -c | sort -nr

# Core crawl-entry checks
for p in /robots.txt /sitemap.xml /llms.txt; do curl -s -o /dev/null -w "%{http_code} %url_effective\n" "https://yourdomain.com$p"; done

Common failure pattern

A frequent issue is allowing PerplexityBot in robots.txt while Cloudflare edge rules still challenge non-browser requests. If you see repeated 403, verify edge rules before changing content files.

Example rollout that avoids policy drift

1. Week 1: allow only /guides/, /questions/, and policy pages; keep account and checkout paths blocked.
2. Week 2: if status mix stays mostly 200/304, expand to selected use-case pages.
3. Week 3: compare referral landings with allowed path groups; keep only pages that show intent traffic.

If your logs stay empty, run this diagnosis path first: zero-hit crawler checklist (same edge/policy principles apply).

If Cloudflare sits in front of origin, continue with Bot Fight Mode crawler impact checks.

For service businesses, pair this policy with a focused source map: local services template.

Use PerplexityBot template