Should You Block AI Crawlers? What They Took and Sent Back

From October 6 to 18, requests under the names of Meta's two AI crawlers reached the server that runs elmrank.com and seven other sites of ours 156,075 times. Over the same 13 days, Google Analytics counted 4 visits to those sites from Meta AI. Requests under OpenAI's crawler names came to 62,892, and ChatGPT sent 315 visits. Googlebot's name was on 39,534 requests, 31,704 of them from Google's own published addresses, and Google Search sent 17,849 visits. We count by the name each request sent; further down we show how many of those names were borrowed.

Our robots.txt files keep none of these AI crawlers off our pages, and all of them reached our server. For most of them we'd keep it that way. On our numbers, AI crawlers take far more than they give back. What's worth deciding is which AI crawlers you can block without losing the few visits and citations they do send, and that depends on what each one is for. The companies spell it out.

Three kinds of AI crawler

OpenAI, Anthropic, Meta and Amazon each run separate crawlers for training, for search and for fetching a page while a user waits; Perplexity runs the last two. The table uses the companies' own descriptions.

Two lines in that table are easy to misread. Blocking Google-Extended doesn't affect "a site's inclusion in Google Search," in Google's words, but Google says the token also controls "grounding" in Gemini Apps, so it can cost you citations there. And Meta describes its training crawler as also "improving products by indexing content directly," so a block there may reach further than training.

Our view: the split that matters for a business is training against everything else. A training crawler takes your text into a model and sends nobody back. A search crawler or fetcher is how an assistant can name and link you. If you're going to block something, block the first kind.

What they cost us and what they sent back

One server runs elmrank.com, the three sites on our use cases page, four other sites of ours and one API, with one shared access log. We grouped every request from October 6 to 18 (UTC) by the crawler name it sent, added up the bytes served, and matched each company with the visits Google Analytics credited to its assistant on the same eight sites (October 6 to 18, the property's time zone).

Together, the six AI companies in that table made 271,470 requests, 10.2% of the 2,665,612 our server answered, and took 5.92 GB, 13.9% of the 42.55 GB it sent. Googlebot took 2.6% of the data. Visits are GA4 sessions, which miss visitors who block the tag or decline cookies where consent holds it back. ChatGPT is undercounted too: from about October 9 to 14 most of its links arrived with no tag or referrer, as our post on tracking ChatGPT in GA4 explains. On the seven days outside that gap, OpenAI made 32,173 requests and ChatGPT sent 279 visits, about 115 requests per visit.

Cloudflare measures the same thing across its network: requests for HTML pages from a platform's crawlers, divided by HTML requests whose referrer names that platform. In early April 2026, Business Insider reported its figures as 8,800 to 1 for Anthropic and 993 to 1 for OpenAI. Ours, counted from all requests and GA4 visits on eight sites, came to 14,935 to 1 for Anthropic and about 200 to 1 for OpenAI. Both point the same way: Google and Bing send readers back, and most AI crawlers send almost none.

Look inside each company before you decide, though. Only 3,272 of OpenAI's 62,892 requests came under GPTBot's name, its training crawler; most came as OAI-SearchBot, the one that decides whether ChatGPT search can show you. Blocking GPTBot would have cut about 5% of OpenAI's load and, by OpenAI's description of the two settings, none of the 315 visits. At Anthropic the training crawler, ClaudeBot, accounted for 9,369 of the 14,935 requests. At Meta, Meta-WebIndexer, the search one, accounted for 107,674 of the 156,075.

Not every "AI crawler" is one

A crawler's name is a line of text that anyone can send. In our 13 days, 6,466 requests under the names of 21 well-known crawlers asked for files that hold passwords and keys on a badly set-up server, such as /.env, .git folders, WordPress config files and cloud credentials; 1,558 of them used Googlebot's name. Of the 524 requests named Claude-SearchBot, 314 were of that kind, and none came from an address on the list Anthropic publishes, as we fetched it on October 20 (274 came through Cloudflare, which hides the address). Of the 596 named Perplexity-User, 331 were of that kind, and none came from Perplexity's list. Googlebot's name was on 39,534 requests in all; 31,704 came from addresses on Google's list, 2,300 came through Cloudflare and 5,530 from elsewhere.

A robots.txt block only stops crawlers that choose to obey it. The scanners above ignore it, and the user fetchers may too, by their companies' own description. A firewall rule that trusts a name can let the wrong traffic in. OpenAI, Anthropic, Perplexity, Google, Bing, Apple and Amazon publish the IP addresses their crawlers use; check a request against those before you allow it or block it by name. Anthropic adds a warning for the other direction: blocking its IP addresses instead of using robots.txt "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file."

Our call, by kind of site

1. 1Your pages sell a serviceAllow every crawler; being read is the point
2. 2Your content is the productBlock training crawlers, keep search and fetchers
3. 3Crawlers strain your serverThrottle by verified IP before you block outright

If your site exists to win customers, such as a plumber, a law firm, a SaaS product or an agency like ours, we'd allow every crawler in the table. Your pages are marketing, and an assistant that has read them can describe you correctly. The robots.txt files of our eight sites, elmrank.com included, let all of them read our pages; only admin, API and test paths are closed, to every crawler alike.

If your content is what you sell, such as recipes, original research, reviews or paid reporting, block the training crawlers and the two tokens, and keep the search crawlers and fetchers. That's a choice about licensing more than traffic: OpenAI says its training and search settings are independent, and the visits in our table came through search and assistants. Write each crawler its own group in robots.txt, and copy your existing Disallow lines into every group you add, because a group named for one crawler replaces the general rules for it (our Perplexity post shows how that caught us).

If crawlers slow your server down, look at your host's bill and your error rate first. Anthropic supports Crawl-delay; Amazon and Apple say they don't. Rate limits on verified addresses keep the visits and cut the load.

Check your CDN too. In April, u/PrincipleTop4437 told r/SEO that Cloudflare's "AI Scrapers and Crawlers" toggle, once switched on, had been putting its own block in front of their robots.txt, with "User-agent: GPTBot", "User-agent: ClaudeBot" and "User-agent: Google-Extended" each set to "Disallow: /", which they hadn't written. The post had 47 points when archived. The top reply, from u/SEOPub at 25 points, said "It wasn't quiet. Cloudflare announced they were doing this months ago." u/AbleInvestment2866, at 8, said they had checked "a bit over 1,000 websites" and found the setting off on all of them. Cloudflare's changelog says that from September 15, 2026, new domains get Training and Agent bots "blocked on pages that display ads, while Search remains allowed." A reply from u/dpaanlka, at 2 points, puts the dilemma in one sentence: "I'm personally not contented about AI scraping my website, and I definitely don't want to NOT appear in citations." Blocking only the training crawlers comes closest to both: it tells those companies not to train on your pages, and the search crawlers can still cite you.

• Open yoursite.com/robots.txt and list every AI crawler it names; look for a Cloudflare managed block you didn't write
• Decide on training crawlers separately from search crawlers and fetchers
• If you block, give each crawler its own group and copy your existing Disallow lines into it
• Leave Googlebot and Bingbot alone; block Google-Extended only if you accept losing Gemini grounding
• Check your CDN or firewall settings for AI bot rules, and match allow or block rules to published IP ranges, not names
• Search your access log for the crawler names and see how many requests each made and what they asked for
• Count AI visits in GA4 before and after any change, and allow about 24 hours for robots.txt changes to register

How we handle AI crawlers for a client

Crawler access is part of the technical work in the first month of our plan ($500, with an SEO and GEO audit, site design and a content plan): a robots.txt that lets the OpenAI, Anthropic and Perplexity crawlers in unless you want them out, and a check that they can reach your pages. After that, $300 a month covers up to 30 articles, ongoing fixes, hosting and a monthly report with your Search Console clicks and impressions; our sample report is filled in with one of our own sites. We don't promise citations or AI visits; the one result our terms promise is about Google clicks. The AI search optimization page lists what we set up, and our posts on why ChatGPT isn't citing a site and whether llms.txt works cover the other checks.

How we know this

Written by Tyler Phan. Every source below was read on the date shown; the page was last reviewed on Oct 22, 2026.

1. OpenAI, Overview of OpenAI Crawlers, accessed Oct 20, 2026
2. Anthropic (Claude Help Center), Does Anthropic crawl data from the web, and how can site owners block the crawler?, accessed Oct 20, 2026
3. Perplexity, Perplexity Crawlers, accessed Oct 20, 2026
4. Meta for Developers, Meta Web Crawlers, accessed Oct 20, 2026
5. Google Crawling Infrastructure, Google's common crawlers (Google-Extended; last updated Oct 15, 2026), accessed Oct 20, 2026
6. Apple Support, About Applebot (published Sept 4, 2026), accessed Oct 20, 2026
7. Amazon Developer, About Amazonbot, accessed Oct 20, 2026
8. Cloudflare changelog, New options to manage AI traffic (July 1, 2026), accessed Oct 20, 2026
9. Cloudflare blog, The crawl before the fall… of referrals: understanding AI's impact on content providers (how crawl-to-refer ratios are calculated), accessed Oct 20, 2026
10. Business Insider (Alistair Barr), The most 'ethical' AI company might also be the web's biggest freeloader (Cloudflare crawl-to-refer ratios, early April 2026; Apr 12, 2026), accessed Oct 20, 2026
11. r/SEO, Cloudflare has been quietly blocking GPTBot and PerplexityBot on my site for months (post by u/PrincipleTop4437; comments by u/SEOPub, u/AbleInvestment2866, u/dpaanlka), accessed Oct 20, 2026
12. Elmrank, requests by crawler in the access log of the server that runs elmrank.com and seven other sites we run (Oct 6 to 18, 2026, UTC), and GA4 sessions by source for the same eight sites (Oct 6 to 18), accessed Oct 20, 2026

Send us your site.

We reply within 12 hours with what we'd fix first. Then we run it for you, technical SEO and content: $500 for the first month, then $300 a month for up to 30 articles. You don't need any technical knowledge, and you can cancel any time.

Get my free first-fix listSee what's included