AI crawler (GPTBot, ClaudeBot)
AI crawlers are the bots that language model providers use to collect website content: GPTBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot, Amazonbot and others; for Google it is the Google-Extended token in robots.txt. When they cannot reach your site, your brand stops existing for that AI: the model has nothing to learn about you and your place in the answers goes to a competitor whose site the crawler could read.
In short
| What they are | Robots collecting content for the training and answers of language models |
| Two kinds | The indexing crawler fills the knowledge base. The answering crawler fetches a page only during a specific answer |
| Where they get blocked | Deliberately in robots.txt, but more often unknowingly at the firewall or the host's WAF |
| How to verify | By requesting the site directly with the bot's user agent. Checking robots.txt is not enough |
How it works
Three roles, three different robots
Every large language model provider runs several robots with different jobs, told apart by user agent name: robots.txt and firewall rules are written against that name.
| Model training | GPTBot, ClaudeBot, Amazonbot, Bytespider, CCBot. They gather training data; the effect shows only in the next model generation. |
| Index for the assistant's search | OAI-SearchBot, Claude-SearchBot, PerplexityBot. They fill the index the assistant draws its cited sources from. |
| Fetching on a user's request | ChatGPT-User, Claude-User, Perplexity-User. They fetch one page when the user pastes it or the assistant needs it for the answer being written. |
| Control token without its own robot | Google-Extended, Applebot-Extended. No user agent of their own; they only tell Googlebot and Applebot whether fetched content may train and ground Gemini or Apple Intelligence. |
Practical consequence: disallowing Google-Extended changes nothing for Google Search or AI Overviews; both run on Googlebot. Allowing ChatGPT-User without OAI-SearchBot means the assistant reads your site only when it comes across your URL elsewhere.
The request path and where it breaks
A rule-abiding robot first downloads robots.txt, finds the group with its name and follows its Allow and Disallow lines, then requests URLs over HTTP. Each takes the same path as a visitor: CDN and firewall (WAF, bot management), server rules (.htaccess, security plugin), and only then the application returning the HTML. A block at any layer before the application returns a 403, a browser challenge or an empty page; robots.txt plays no role there. Since July 2025 Cloudflare also asks newly added domains whether to admit AI crawlers, blocking by default, so a CDN account alone often shuts them out. Second limitation: AI crawlers as a rule do not run JavaScript; prices, descriptions and reviews rendered only in the browser may not exist for them.
Worked example: an e-shop with 5,000 URLs limits AI robots to 1 request per second. The whole site is read in 5,000 seconds, roughly 83 minutes, at negligible server load. A limit is therefore more sensible than a block.
Verification: two questions, two procedures
First question: does the robot reach the site? Only an outside request with the bot's user agent to several page types (homepage, category, product, article) answers it; WAF rules are often tiered by path. Check the body, not just the status code: a 200 carrying a browser verification challenge blocks a robot as surely as a 403. Second question: is the log visitor calling itself GPTBot really GPTBot? Anyone can set the user agent string. Providers therefore publish their robots' IP ranges (OpenAI, for example, as a public JSON file); a match against that list is the only reliable proof. Without it, AI robot visit counts and rules built on them mean nothing.
Two crawlers, two different kinds of damage
Model providers run separate robots. The indexing crawler, such as ClaudeBot or GPTBot, fills the model's knowledge base: without it the model never discovers you on its own. The answering crawler, such as Claude-User or ChatGPT-User, fetches a page only when it comes up inside a specific answer. Blocking each has a different consequence, and either can happen independently.
From our own practice: blocked bots nobody knew about
In audits we test language model crawlers with direct requests. Some of them routinely receive HTTP 403 even though robots.txt explicitly allows them. They are blocked by the hosting firewall or the CDN, a layer nobody routinely looks into. One telling detail: the answering crawler of the same provider passes; only the indexing one is blocked.
Deliberately blocking AI crawlers is a competitive disadvantage, not caution. And whoever blocks them usually does not even know it: the block comes from the firewall or the CDN, not from a decision.
Common mistakes
- Checking only robots.txt. Real availability is decided by the server. A firewall can silently override robots.txt.
- Blocking everything out of caution. Protecting content from training is a legitimate choice, but it has to be deliberate, not a side effect of a security rule.
- Verifying once and never again. Hosts change rules without notice. A monthly test belongs in the routine.
Related terms
See also RAG, GEO, llms.txt, AI visibility and indexing.
Frequently asked questions
Which crawlers should I allow?
At minimum GPTBot, ClaudeBot, PerplexityBot, Google-Extended and OAI-SearchBot. If you want AI visibility, allow them all. If you want to protect content from training, decide bot by bot.
How do I verify availability?
Request your own site with the bot's user agent and check the response code. The expected result is 200; anything else means a problem somewhere on the path.
Will crawlers slow the site down?
Normally no. If they do, the right answer is a rate limit, not a block: the bot gets the data more slowly, but it gets it.
How we can help
An AI crawler availability audit is the first step of our AI visibility agency service.