How do I allow OAI-SearchBot in Cloudflare without weakening security?

Keep global bot protection in place, then add a narrowly scoped exception for OAI-SearchBot on public crawl paths only. Exclude admin, login, private, API, checkout, and account routes, and verify results in Security Events and origin logs.

Should I allow OAI-SearchBot by user agent only?

No. A user-agent-only allow rule is too broad because user agents can be spoofed. Combine the crawler condition with public path constraints, managed bot signals where available, rate limits, and post-change log review.

Which paths should be allowed first?

Start with robots.txt, sitemap.xml, llms.txt, and public source-page sections such as guides, questions, docs, tools, case studies, and product pages. Keep admin, account, checkout, API, staging, and private routes excluded.

Should the Cloudflare action be Allow or Skip?

Prefer the narrowest action that solves the block. If a specific WAF rule or managed challenge blocks public crawl paths, skip only that control for the scoped crawler paths. Avoid a broad Allow action that bypasses unrelated security checks across the site.

Question

How to allow OAI-SearchBot in Cloudflare WAF

Updated July 20, 2026

You do not need to disable Cloudflare security globally to make ChatGPT search discovery work. The safer approach is to keep strict protection in place, then add a narrow exception for OAI-SearchBot on public crawl paths only.

When this rule is needed

Add a Cloudflare exception only after you see evidence that public pages are blocked. Typical signals are 403 responses for OAI-SearchBot, Security Events showing managed challenges for the crawler, or origin logs showing no successful fetches even though robots.txt allows the bot.

Safe rule strategy

Keep your default managed challenge or bot score policy.
Add an exception only for the crawler token and public crawl paths.
Do not allow admin or private paths in the exception.
Use Cloudflare managed bot signals where available instead of trusting user-agent text alone.
Review Security Events and origin access logs after every change.

Cloudflare expression pattern (example)

This example is intentionally path-limited. Adjust the public paths to match your site structure.

(http.user_agent contains "OAI-SearchBot")
and (http.request.uri.path matches "^/(robots\\.txt|sitemap\\.xml|llms\\.txt|guides/|questions/|tools/)")
and not (http.request.uri.path matches "^/(admin|wp-admin|login|api/private)")

Example policy for a content and tools site

For a site that wants ChatGPT search visibility, start by allowing the crawler to fetch discovery files and stable public source sections. Keep forms, admin routes, and private APIs outside the exception even if they return harmless pages today.

(http.user_agent contains "OAI-SearchBot")
and (http.request.uri.path matches "^/(robots\\.txt|sitemap\\.xml|llms\\.txt|questions/|guides/|tools/|hubs/|case-studies/|use-cases/)")
and not (http.request.uri.path matches "^/(admin|wp-admin|login|account|checkout|api|internal|staging|preview)")

After publishing the rule, watch a 24-hour window. You want to see 200 or 304 on the public paths above, not a sudden rise in admin, API, or exploit-probe hits being allowed.

Stronger pattern if verified bot fields are available

If your Cloudflare plan exposes verified bot or bot management fields, prefer them. The exact expression depends on the account features, but the logic should be:

verified_bot_is_true
and user_agent_or_bot_name_matches_oai_searchbot
and path_is_public_content
and path_is_not_sensitive

If you only have user-agent text available, keep the exception narrow and monitor it. A user agent can be spoofed; a user-agent-only allow rule across the whole site is not safe.

Choose the narrowest Cloudflare action

Do not jump straight to a broad Allow rule. In many accounts, the safer fix is to skip only the specific managed challenge or WAF rule that blocks public crawler paths, while leaving rate limits, exploit protection, and sensitive route rules active.

Scenario	Preferred action	Why
One managed rule challenges public article URLs	Skip that rule for scoped public paths	Other security checks remain active.
Bot score policy blocks all non-browser requests	Exception only for discovery and source-page paths	Prevents account, API, and checkout exposure.
Requests hit login, admin, API, or uploads	Keep blocking	Those paths do not need answer-engine crawl access.
Only robots.txt or sitemap is blocked	Allow or skip for those exact files	Smallest change, easiest rollback.

Name the rule with the date and scope, for example 2026-07 OAI-SearchBot public source pages. That makes later reviews easier when someone asks why the exception exists.

Paths that should not be allowed

/admin, /wp-admin, /login, /account
/cart, /checkout, payment routes, and customer portals
private APIs, staging previews, exports, logs, and upload directories
any URL that returns personal, paid, unpublished, or tenant-specific content

Validation checklist

Security Events should show fewer 403 events for OAI-SearchBot paths.
Server logs should show 200/304 on allowed pages.
No growth in exploit probes from the same policy change window.
Public pages still return normal status for Googlebot, Bingbot, and human visitors.

24-hour review table

After changing the rule, review a full day before widening the scope. The goal is successful crawler access on public source pages, not a larger number of allowed requests.

Metric	Healthy result	What to do if unhealthy
OAI-SearchBot public path status	Mostly `200` or `304`	Check the exact Cloudflare rule still firing.
Sensitive path hits	Zero allowed requests	Narrow the path expression immediately.
Discovery files	`/robots.txt`, `/sitemap.xml`, and `/llms.txt` reachable	Add exact-file exceptions before broad directories.
Other bots and humans	No status regression	Roll back and split the rule into smaller scopes.

Origin log check

grep -Ei 'OAI-SearchBot|ChatGPT-User' /var/log/nginx/access.log \
  | awk '{print $1, $7, $9, $0}' \
  | tail -50

Look for successful requests on pages you want surfaced in ChatGPT search. If the log still shows 403, check Cloudflare Security Events first. If Cloudflare shows allowed traffic but the origin still blocks it, inspect Nginx, app middleware, or IP allowlists.

Rollback rule

Keep the change reversible. If you see exploit probes hitting allowed paths, unexpected admin hits, or a spike in blocked traffic with spoofed crawler names, disable the rule and narrow the path scope before trying again.

Common mistake

Teams often allow by user-agent only and accidentally open sensitive routes. Always combine the crawler condition with path constraints, and treat the first 24 hours after the change as a monitoring window.

If you still see blocked traffic, continue with why OAI-SearchBot gets 403 behind Cloudflare.

Check crawler behavior in logs