Reference
AI crawlers list
Last updated
AI crawlers are the programs AI companies use to fetch web pages, either to answer a question, to build a search index, or to gather training data. Each one announces itself with a name, called a user agent, that you can allow or block in your robots.txt file. This page lists the 17 names the CitableReport audit tests for, from 10 companies.
A user agent is the name a program gives when it asks a website for a page. Robots.txt is a plain text file at the root of a site, such as example.com/robots.txt,
that tells each user agent which pages it may fetch. For example, a rule addressed to GPTBot applies to OpenAI's training crawler and to nothing else.
Which AI crawlers are there?
| Name in robots.txt | Run by | Kind | What it does |
|---|---|---|---|
OAI-SearchBot |
OpenAI | Search index | Builds the index ChatGPT search draws its answers and links from. |
ChatGPT-User |
OpenAI | On a person's request | Fetches a page when a person asks ChatGPT about it or ChatGPT looks something up for them. |
GPTBot |
OpenAI | Training data | Collects pages that may be used to train OpenAI's models. |
Claude-SearchBot |
Anthropic | Search index | Indexes pages to improve the quality of Claude's search results. |
Claude-User |
Anthropic | On a person's request | Fetches a page when a person asks Claude a question that needs it. |
ClaudeBot |
Anthropic | Training data | Collects pages that may be used to train Anthropic's models. |
PerplexityBot |
Perplexity | Search index | Builds the index Perplexity cites in its answers. |
Perplexity-User |
Perplexity | On a person's request | Fetches a page when a person's question leads Perplexity to it. |
Googlebot |
Search index | Google's search crawler. Its index also feeds AI Overviews and AI Mode. | |
Google-Extended |
Control name | Not a crawler but a robots.txt name: it controls whether pages Google has crawled may be used for Gemini. | |
Bingbot |
Microsoft | Search index | Bing's search crawler. Its index feeds Copilot and is used by ChatGPT search. |
Applebot |
Apple | Search index | Powers search in Siri and Spotlight. |
Applebot-Extended |
Apple | Control name | Not a crawler but a robots.txt name: it controls whether pages Applebot has crawled may be used to train Apple's models. |
Amazonbot |
Amazon | Training data | Crawls for Amazon's services, including answers from Alexa. |
Meta-ExternalAgent |
Meta | Training data | Collects pages for training Meta's AI models and for its products. |
CCBot |
Common Crawl | Training data | Builds an open archive of the web that many AI models have been trained on. |
Bytespider |
ByteDance | Training data | Collects pages for ByteDance's products and models. |
What is the difference between a search crawler and a training crawler?
A search crawler fetches pages so an assistant can quote and link to them in an answer; a training crawler collects pages to train future models. The difference matters because blocking them has different effects.
- Search index crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, Applebot) decide whether you can appear in AI answers at all. Block these and you will not be cited.
- On-request fetchers (ChatGPT-User, Claude-User, Perplexity-User) visit a page when a person asks about it. Block these and the assistant cannot read a page even when someone pastes your link.
- Training crawlers (GPTBot, ClaudeBot, Amazonbot, Meta-ExternalAgent, CCBot, Bytespider) gather text for training. Blocking them does not remove you from AI search answers.
- Control names (Google-Extended, Applebot-Extended) are not crawlers. They are robots.txt names that tell a company how it may use pages its ordinary crawler already fetched.
How do I allow AI crawlers in robots.txt?
You allow AI crawlers by not disallowing them. A robots.txt that allows everything looks like this:
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlIf your robots.txt blocks some crawlers and you want to be sure the AI search ones are let in, name them:
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /How do I block training but stay in AI search?
To stay in AI search answers while opting out of training, block only the training crawlers and control names:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
User-agent: CCBot
User-agent: Bytespider
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /In the CitableReport audit, a site that does this gets a partial pass on the crawler check, not a fail.
Why might AI crawlers be blocked even though my robots.txt allows them?
AI crawlers can be blocked by a firewall or CDN even when robots.txt allows them. Several providers now block AI crawlers by default, or challenge them with a check that only a browser can pass. The crawler receives an error page instead of your content, and nothing in robots.txt shows it. The audit tests for this separately by requesting your front page as an AI crawler would and comparing the answer with what a browser gets.
How current is this list?
This list reflects the names the AI companies document publicly. They add and rename crawlers from time to time, so check each company's own documentation before relying on a rule. Robots.txt is also a request, not a lock: well-behaved crawlers obey it, and the ones listed here say they do.