AI Crawler User Agents: The List, Checked in Our Logs
AI crawler user agents from OpenAI, Anthropic, Perplexity, Meta, Google, Apple and Amazon: what each is for, if it reads robots.txt, and what our log saw.
By Tyler Phan9 min readLast reviewed
Part of our guide to AI search and GEO
On this page
An AI crawler user agent is the name a bot from an AI company puts in its requests, and the token you use in robots.txt to allow or block it. Most AI companies now run several: one that collects training data, one that builds a search index, and one that fetches a page while a user waits for an answer. They follow different rules. This list covers the main names OpenAI, Anthropic, Perplexity, Meta, Google, Apple, Amazon and Mistral document, with what each company says the bot is for and what each did in the log of the server that runs our eight sites over 13 days in October 2026.
The list
Each row uses the company's own wording where it has one. "IP list" means the company publishes the addresses its bot uses, so you can check a request against them.
| User agent | What it's for | robots.txt | IP list |
|---|---|---|---|
| GPTBot (OpenAI) | Content "that may be used in training our generative AI foundation models" | Followed: a Disallow means content "should not be used in training" | openai.com/gptbot.json |
| OAI-SearchBot (OpenAI) | Search results in ChatGPT | Followed: blocked sites "will not be shown in ChatGPT search answers, though can still appear as navigational links" | openai.com/searchbot.json |
| ChatGPT-User (OpenAI) | Visits a page when a ChatGPT user's question needs it | "robots.txt rules may not apply" | openai.com/chatgpt-user.json |
| ClaudeBot (Anthropic) | Web content that "could potentially contribute to their training" | Followed, and it supports Crawl-delay | claude.com/crawling/bots.json (one list for all three) |
| Claude-SearchBot (Anthropic) | Claude's search results | Followed; blocking it "may reduce your site's visibility and accuracy in user search results" | Same list |
| Claude-User (Anthropic) | Fetches pages when a Claude user asks | Followed; Anthropic says its bots honor robots.txt | Same list |
| PerplexityBot | Perplexity search results; "not used to crawl content for AI foundation models" | Perplexity recommends allowing it in robots.txt | perplexity.com/perplexitybot.json |
| Perplexity-User | Fetches pages when a Perplexity user asks | "generally ignores robots.txt rules" | perplexity.com/perplexity-user.json |
| meta-externalagent (Meta) | "training foundation AI models or improving products by indexing content directly" | To block it, Meta says to "add a disallow for the relevant crawler" | None on the page |
| meta-webindexer (Meta) | Meta AI search results | Allowing it "helps us cite and link to your content in Meta AI's responses" | None on the page |
| Amazonbot | Amazon products and services; "may be used to train Amazon AI models" | Followed, from a robots.txt copy up to 30 days old; no Crawl-delay | developer.amazon.com/amazonbot/ip-addresses |
| Applebot | Search in Spotlight, Siri and Safari; its data may also train Apple's models | Followed; with no Applebot group, it follows your Googlebot rules | search.developer.apple.com/applebot.json, or reverse DNS in applebot.apple.com |
| Applebot-Extended | A robots.txt token only: opts content out of Apple model training | "does not crawl webpages" | Not needed |
| Google-Extended | A robots.txt token only: Gemini training and grounding | No user agent of its own; doesn't affect Google Search | Not needed |
| GoogleOther | Google's generic crawler, for example "one-off crawls for internal research and development" | Google's common crawlers "always respect robots.txt rules for automatic crawls" | Google's common crawlers list |
| MistralAI-User | Fetches for user questions in Mistral's Vibe; not used for training | Its robots.txt token "governs which sites these user requests can be made to" | Published on Mistral's robots page |
| OAI-AdsBot (OpenAI) | Checks pages "submitted as ads on ChatGPT"; its data "is not used to train generative AI foundation models" | OpenAI documents no robots.txt token for it; it "only visits pages submitted as ads" | openai.com/adsbot.json |
| Meta-ExternalAds (Meta) | "improving advertising and other business-related products and services" | Same Disallow advice as Meta's other crawlers | None on the page |
Meta also lists meta-externalfetcher, which "may bypass robots.txt rules," and Amazon says the same of its user fetcher, Amzn-User: it "may not follow all robots.txt directives." A user agent named meta-externalfetcher made 522 requests to our sites between October 10 and 21, and only 3 got a page; the rest got redirects or errors. None of the 289 we could check came from 2a03:2880::/29, Facebook's IPv6 block, where 91,872 of the 107,674 meta-webindexer requests came from. They came from 16 cloud-server addresses, and the same 16 also sent requests as OAI-AdsBot: 484 of those over the same days, most of them for files like .env, and none of the 245 we could check from OpenAI's adsbot.json as we read it on October 22. We saw no Meta-ExternalAds. Amazon runs Amzn-SearchBot and Amzn-User next to Amazonbot, and Mistral runs MistralAI-Index; from October 8 to 22 we saw none of the three.
Google and Apple never send a request as Google-Extended or Applebot-Extended, because those aren't crawlers. Google says Google-Extended "doesn't have a separate HTTP request user agent string" and "does not impact a site's inclusion in Google Search." Apple says Applebot-Extended "is only used to determine how to use the data crawled by the Applebot user agent." So a request under either name is someone else's. Our log had 81 as Google-Extended on October 8 and 9, most of them for files like /.env and /.ssh/id_rsa, and none of the 39 we could check came from Google's addresses. Applebot-Extended never showed up.
What each one did on our server
Our eight sites share one server and one access log. We grouped every request from October 6 to 18, 2026 (UTC) by the name in its user agent, counted requests for robots.txt, and checked each source address against the company's published list. Sites behind Cloudflare log Cloudflare's address, so those requests can't be checked; the last column counts only the ones we could. Our llms.txt post read the same log through October 19, so its totals are a day larger.
| User agent | Requests | robots.txt fetches | From the company's list |
|---|---|---|---|
| meta-webindexer | 107,674 | 0 | No list |
| OAI-SearchBot | 49,047 | 990 | 99% (37,451 of 37,742) |
| meta-externalagent | 48,401 | 0 | No list |
| Amazonbot | 19,022 | 6 | Not checked |
| ChatGPT-User | 10,573 | 0 | 96% (7,752 of 8,060) |
| Applebot | 10,078 | 328 | 97% (8,055 of 8,341) |
| ClaudeBot | 9,369 | 1,389 | 96% (7,653 of 7,947) |
| Claude-User | 5,042 | 1,140 | 67% (2,332 of 3,488) |
| GPTBot | 3,272 | 0 | 86% (1,652 of 1,925) |
| PerplexityBot | 1,874 | 493 | 57% (399 of 705) |
| Perplexity-User | 596 | 0 | 0 of 296 |
| Claude-SearchBot | 524 | 0 | 0 of 250 |
GPTBot was small. It made 3,272 requests against OAI-SearchBot's 49,047 and never asked for robots.txt under its own name. OpenAI's page says that "if your site has allowed both bots, we may use the results from just one crawl for both use cases," and our robots.txt files allow both. OAI-SearchBot fetched robots.txt 990 times, 566 of them with the "robots.txt" marker OpenAI describes for those requests.
The bots that say robots.txt may not apply didn't fetch it. ChatGPT-User and Perplexity-User made no robots.txt requests at all, which matches what OpenAI and Perplexity write about them. Neither Meta name asked for robots.txt either, while facebookexternalhit, Meta's link-preview fetcher, asked 994 times; Meta's page doesn't say which of its agents reads the file for the others. Amazonbot asked 6 times in 13 days, which fits Amazon's note that it uses "a cached copy from the last 30 days."
Two names didn't come from their companies. None of the checkable Claude-SearchBot or Perplexity-User requests came from Anthropic's or Perplexity's lists, and most of them got a 4xx answer: 312 of 524 and 335 of 596. We think someone else is using those names. Our post on whether to block AI crawlers found 6,466 requests for secret files such as .env under 21 crawler names in the same log.
Claude-User covers Claude Code. Of its 5,042 requests, 1,224 carried the string "Claude-User (claude-code/…)", from Anthropic's coding tool, next to the usual browser-style string. Anthropic's changelog for Claude Code 2.1.83 (March 25, 2026) says its web fetch now identifies as Claude-User "so site operators can recognize and allowlist Claude Code traffic via robots.txt." Claude Code runs on its user's own computer, so those requests come from that computer's address, and that explains most of Claude-User's low share in the table: 832 of the 1,156 checkable requests from outside Anthropic's list were Claude Code. Without them, 88% came from the list (2,332 of 2,656). We can't explain PerplexityBot's 43% from outside Perplexity's list yet.
The exact strings
These are the most common full strings per name in our log. The version numbers change: we saw GPTBot 1.2 to 1.4 and OAI-SearchBot 1.0, 1.3 and 1.4. OpenAI notes that "the version number may change", and Google tells you to "use wildcards for the version number" when filtering its crawlers. Match on the name, not the whole string.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +claude-user@anthropic.com)
Claude-User (claude-code/2.1.295; +https://support.anthropic.com/)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)
Mozilla/5.0 (compatible; MistralAI-User/1.0; +https://mistral.ai/)
Three more names showed up that this list doesn't document: Bytespider (16,076 requests), DuckAssistBot (2,187) and CCBot (558). We left them out of the table because we haven't checked their makers' documentation for this post.
How to verify an AI crawler
- 1Match the nameThe token in the user agent, not the full string
- 2Check the addressAgainst the company's published IP list, or reverse DNS where documented
- 3Watch what it asks forReal crawlers want pages; requests for .env or .git are not a crawler
- 4Decide per jobTraining, search and user fetches have separate tokens
If your site sits behind Cloudflare or another proxy, your server log may show the proxy's address instead of the crawler's, as ours does for the proxied sites. Log the original address the proxy passes along, or check in the proxy's own dashboard.
Blocking training while staying in AI search
If you want your pages kept out of model training but still found by AI search, the vendors' descriptions point to this robots.txt. It's our reading of their pages, not a guarantee from them, and one line in it has a cost:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
The costly line is Google-Extended. Google says the token covers Gemini training and also "grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy)" in Gemini Apps, so blocking it can cost you citations in Gemini answers. Leave that group out if you want to stay in them.
Leave OAI-SearchBot, Claude-SearchBot, PerplexityBot and Applebot allowed. Meta and Amazon don't split training from everything else as cleanly: meta-externalagent also covers "improving products by indexing content directly," and Amazonbot serves Amazon's products as well as training. Our post on blocking lists both with the training crawlers; going by the vendors' wording, a block reaches further than training, so we left them out here. Blocking an IP range isn't a substitute, in Anthropic's words, because it "impedes our ability to read your robots.txt file." OpenAI says changes take "~24 hours" to reach its search systems. For whether to block at all, our post on blocking AI crawlers sets what each company's bots took against the visits they sent back.
Where this sits in our plan
Our AI search optimization service sets up a robots.txt that lets the OpenAI, Anthropic and Perplexity crawlers in, unless you want them out, as part of the first month's audit and setup. The plan is $500 for the first month and $300 a month after. To see what we'd fix first on your site, send it to us: the list is free, and we reply within 12 hours.
Questions
Which AI crawler should I block to stop AI training but stay in AI search?
By the vendors' own descriptions: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), Applebot-Extended (Apple) and meta-externalagent (Meta) are the training controls, while OAI-SearchBot, Claude-SearchBot and PerplexityBot feed AI search. Google-Extended also covers grounding in Gemini, so blocking it can cost you citations there. Meta says meta-externalagent is also used for "improving products by indexing content directly", and Amazon says Amazonbot "may be used to train Amazon AI models" alongside its other uses, so blocking those two reaches further than training.
Is Google-Extended a crawler?
No. Google says it "doesn't have a separate HTTP request user agent string": Google's other crawlers fetch the pages, and the Google-Extended line in robots.txt only decides whether that content may be used for Gemini training and grounding. Google never sends a request under that name, so any log line that carries it is someone else's. Our log had 81, most of them hunting for files like .env and SSH keys, and none of those we could check came from Google's addresses. Applebot-Extended works the same way for Apple.
How do I know a request really comes from an AI company?
Match the source IP against the list the company publishes (OpenAI, Anthropic, Perplexity, Apple, Amazon and Google all publish one), or use reverse DNS where the company documents it, as Apple does for applebot.apple.com. The user agent alone proves nothing: in our log, none of the Claude-SearchBot or Perplexity-User requests we could check came from those companies' lists.
How we know this
Written by Tyler Phan. Every source below was read on the date shown; the page was last reviewed on Oct 25, 2026.
- OpenAI, Overview of OpenAI Crawlers, accessed Oct 22, 2026
- Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?, accessed Oct 22, 2026
- Perplexity, Perplexity Crawlers, accessed Oct 22, 2026
- Meta for Developers, Meta Web Crawlers, accessed Oct 22, 2026
- Google Crawling Infrastructure, Google's common crawlers, accessed Oct 22, 2026
- Apple Support, About Applebot, accessed Oct 22, 2026
- Amazon Developer, Amazonbot, accessed Oct 22, 2026
- Mistral AI, Robots, accessed Oct 22, 2026
- Claude Code changelog, version 2.1.83 (March 25, 2026): WebFetch identifies as Claude-User, accessed Oct 22, 2026
- Elmrank, our VPS access log (eight sites), October 6 to 18, 2026: requests, robots.txt fetches, source addresses and user-agent strings per crawler, accessed Oct 22, 2026
Send us your site.
We reply within 12 hours with what we'd fix first. Then we run it for you, technical SEO and content: $500 for the first month, then $300 a month for up to 30 articles. You don't need any technical knowledge, and you can cancel any time.
