GPTBot blocked in robots.txt? Check every AI crawler and CDN
Which AI crawlers do what (GPTBot, OAI-SearchBot, Google-Extended, PerplexityBot, ClaudeBot and more), how CDNs block them silently, and how to check your logs.
- AI crawlers do three different jobs: training, search indexing and user-requested fetches. Blocking one does not block the others.
- Blocking GPTBot only opts out of OpenAI training; blocking OAI-SearchBot removes you from ChatGPT search answers.
- CDNs and WAFs can block AI bots by default or by challenge, so check edge settings and server logs, not just robots.txt.
Blocking GPTBot in robots.txt only tells OpenAI not to use your content for training; it does not remove you from ChatGPT search, which uses a separate crawler, OAI-SearchBot. Each AI company runs different bots for training, search indexing and user-requested fetches. To know whether AI engines can reach you, check robots.txt per bot, your CDN and WAF bot settings, and server logs.
This guide lists the major AI crawlers, what each one does according to its vendor, how edge security can block them without anyone in marketing knowing, and how to audit it. It is the technical companion to why ChatGPT doesn't mention your brand.
01What do the main AI crawlers actually do?
Vendors now separate their bots by purpose. The distinction that matters for visibility is training versus search versus user fetch.
| User agent | Vendor | Purpose (per vendor docs) | Robots.txt |
|---|---|---|---|
| GPTBot | OpenAI | Crawls content that may be used to train foundation models[1] | Respected |
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT search; opted-out sites won't appear in search answers[1] | Respected; changes take about 24 hours[1] |
| ChatGPT-User | OpenAI | User-initiated actions in ChatGPT and custom GPTs[1] | May not apply, since user-initiated[1] |
| Google-Extended | Control token for Gemini training and Gemini Apps grounding; does not affect Search[2] | Token only, no separate crawler[2] | |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity search; not for foundation-model training[3] | Respected[3] |
| Perplexity-User | Perplexity | Fetches pages when a user asks a question[3] | Generally ignored[3] |
| ClaudeBot | Anthropic | Collects content that may contribute to model training[4] | Respected; supports Crawl-delay[4] |
| Claude-SearchBot | Anthropic | Indexes content to improve search results for Claude users[4] | Respected[4] |
| Claude-User | Anthropic | Retrieves pages in response to a user's query[4] | Respected[4] |
| Applebot-Extended | Apple | Opt-out from training Apple's foundation models; does not crawl[5] | Token only[5] |
| Bingbot | Microsoft | Bing's search crawler; indexed content can appear in Bing's chat answers unless tagged NOARCHIVE[6] | Respected |
For Microsoft, the AI controls are page-level: Bing said in 2023 that content tagged NOARCHIVE will not be included in its chat answers, while NOCACHE allows limited use.[6]
Treating "AI bots" as one group. A blanket block on every AI user agent opts you out of training and out of AI search citations at the same time. Most enterprises want a deliberate, per-bot decision instead.
02Which bots should we allow if we want to appear in AI answers?
That is a business and legal decision, but the visibility logic is clear. Search and user-fetch bots are the ones that put you in answers today; training bots shape what future models know.
Google-Extended deserves a closer look. Google says it does not impact inclusion in Google Search, but it does control use of content for grounding in Gemini Apps.[2] Our guide on brands missing from Gemini covers that trade-off.
03Should we block AI training crawlers at all?
There are reasonable arguments both ways. Blocking training bots protects content you license or sell, and signals a position on how your work is used. Allowing them means future models are more likely to know your brand, products and terminology without needing to search. Publishers with paywalled content often block; brands whose goal is to be understood and recommended often allow.
Whatever you decide, make it a decision rather than a default. Write down which bots you allow and why, who approved it, and when it will be reviewed. Blocks added quickly when AI crawlers first made headlines often have no recorded owner or rationale, which makes them hard to revisit later.
It also helps to separate pages. You can allow search bots everywhere while restricting training bots on premium or gated sections, using path-level rules in robots.txt. Granular rules are usually better than a single site-wide switch.
04How can a CDN or WAF block AI crawlers without anyone noticing?
Robots.txt is only one layer. In large companies, the edge is owned by security, and its defaults change.
In July 2025 Cloudflare said every new domain would be asked whether to allow AI crawlers, with blocking as the starting default.[7] Its documentation now says that from September 15, 2026, new domains get updated defaults in which bots classified as Training or Agent are blocked on pages that display ads, while Search crawlers remain allowed.[8] Cloudflare defines Agent as real-time activity on a person's behalf, such as chat fetch bots, which covers user agents like ChatGPT-User.[8]
Cloudflare is the best documented example, not the only one. Other CDNs, WAFs and bot-management vendors offer AI-bot categories too, so ask whoever runs your edge which categories are blocked or challenged, on which hostnames, and since when.
Cloudflare also offers a managed robots.txt that prepends disallow rules for named AI crawlers, including GPTBot, ClaudeBot, Google-Extended and Applebot-Extended; it is not on by default.[9]
05How do I check whether AI crawlers can reach my site?
Do it per hostname, from the outside in. What matters is what the bot receives, not what your CMS says.
- 1Fetch robots.txt from the edgeDay 1Request /robots.txt on every hostname, including www, regional, docs and app subdomains. Compare with the CMS version.
- 2Review rules per user agentDay 1List which of the bots above are allowed or disallowed, and whether a catch-all rule applies.
- 3Pull CDN and WAF settingsDay 1–2Ask security for AI-bot, bot-management and challenge settings, plus any custom rules on user agents or ASNs.
- 4Search server or CDN logsDay 2Filter 30 days of logs for each user-agent string. Record request counts and status codes per bot.
- 5Verify IPsDay 2Match requests to vendors' published IP lists, for example OpenAI's searchbot.json and Perplexity's perplexitybot.json.
- 6Test a known answerDay 3+Ask each engine a question only your page answers. No citation after access is fixed points to content, not crawling.
In the logs, status codes tell the story. A healthy pattern is mostly 200s and some 304s. Clusters of 403, 429 or challenge responses mean the edge is blocking; 5xx errors mean the origin is struggling. Remember that Google-Extended and Applebot-Extended never appear in logs, because they are control tokens rather than crawlers.[2][5]
User-agent strings are easy to fake. Perplexity recommends combining user-agent matching with IP verification against its published ranges, refreshed automatically.[3] Apply the same to every vendor that publishes IP lists. In 2025 Cloudflare reported Perplexity using undeclared crawlers when blocked, a reminder that bot identity needs checking.[10]
If you do not have direct log access, ask for a simple export: user agent, status code, URL and timestamp for the bots in the table. A one-page summary per bot per hostname is enough for marketing to see whether the edge matches the policy. Analytics tools will not help here, because crawlers do not run JavaScript tags.
06What does a sensible robots.txt policy look like?
There is no single right answer, but there should be a written decision per bot, signed off by marketing, legal and security. An example policy shape:
A financial services firm decides to allow OAI-SearchBot, PerplexityBot, Claude-SearchBot and Bingbot so it can be cited; to disallow GPTBot and ClaudeBot for training; and to allow Google-Extended after weighing Gemini visibility. It documents the rationale, asks security to mirror the policy at the CDN, and re-checks quarterly.
Two operational rules matter as much as the policy. First, put the policy in your change process, so a CDN migration or new security vendor cannot silently reverse it. Second, re-audit after every platform change. OpenAI notes robots.txt changes take about 24 hours to affect search results, and Google generally caches robots.txt for up to 24 hours too.[1][11]
07Who should own AI crawler access?
Today, often no one. Security owns the CDN, web owns robots.txt, SEO owns indexing, and marketing owns visibility. A single owner who can see across all four is what stops the silent block from happening twice. OwnlyReal works that way inside marketing teams: measuring share of answer, tracing gaps to causes like crawler access, and shipping fixes through your existing approval chains.
Questions people also ask
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, and OpenAI says sites that opt out of OAI-SearchBot won't appear in ChatGPT search answers.
Does blocking Google-Extended affect my Google rankings?
Google says Google-Extended does not affect Search inclusion or ranking. It does control use of content for Gemini training and grounding in Gemini Apps.
Is Cloudflare blocking AI crawlers on my site?
It may be. Cloudflare changed defaults for new domains in 2025 and again from September 15, 2026. Check AI-bot settings, managed robots.txt and challenge rules in your dashboard.
How do I see AI crawlers in my server logs?
Filter logs by user-agent strings such as GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot, check status codes, and verify IPs against each vendor's published list.
Do user-triggered bots like ChatGPT-User follow robots.txt?
OpenAI says robots.txt rules may not apply to ChatGPT-User because actions are user-initiated, and Perplexity says Perplexity-User generally ignores it. Anthropic says its bots respect robots.txt.
