AI-Crawler Census 2026

Measured 2026-08-30 · 2,776 sites with a robots.txt, from the Tranco top 5,000 · 20 crawlers · reproducible

Plenty of reports count how many sites block GPTBot. Almost none measure what that blocking costs — because AI crawlers do three different jobs, and blocking the wrong one quietly removes you from AI search results.

44.4%

of sites that block GPTBot also block OAI-SearchBot — cutting themselves out of ChatGPT's search citations and the referral traffic that follows. GPTBot governs training; OAI-SearchBot governs being cited. They are not the same decision, and they are frequently made as if they were.

The three jobs (and why the distinction is the whole story)

Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) decide whether your content trains models. Search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) decide whether you can be cited in AI answers. User-action fetchers (ChatGPT-User, Perplexity-User) fetch your page live when a person asks an assistant about it. Blocking the first is a content-licensing choice. Blocking the second two is a traffic choice — and it is usually made by accident.

Headline numbers

MeasureSitesShare
Block at least one training crawler73926.6%
Block at least one AI-search crawler39114.1%
Block at least one user-action fetcher38113.7%
Block Googlebot (baseline)963.5%

The baseline matters: sites block AI training crawlers roughly 6× more often than they block Googlebot. Almost nobody is opting out of search — they are opting out of training, and sometimes taking search down with it.

Training crawlers

AgentVendorBlockedShare
ccbotCommon Crawl61222.1%
bytespiderByteDance57920.9%
gptbotOpenAI57720.8%
claudebotAnthropic56020.2%
meta-externalagentMeta52218.8%
google-extendedGoogle50218.1%
amazonbotAmazon46916.9%
applebot-extendedApple45316.3%
omgiliWebz.io40814.7%
diffbotDiffbot39414.2%
cohere-aiCohere39314.2%
timpibotTimpi32911.8%

AI-search / citation crawlers

AgentVendorBlockedShare
perplexitybotPerplexity37813.6%
claude-searchbotAnthropic2679.6%
oai-searchbotOpenAI2609.4%

User-action fetchers

AgentVendorBlockedShare
chatgpt-userOpenAI31911.5%
perplexity-userPerplexity2729.8%
claude-userAnthropic2689.7%

The accidental opt-out

301 sites (10.8%) block an AI-search crawler while still allowing Googlebot. These sites demonstrably want search traffic — so cutting off AI citation is very unlikely to be the intent. Among the largest sites doing this:

Verify any of these yourself — robots.txt is public. Every figure on this page comes from a live fetch, not an estimate.

Check your own site

Our free checker reads your live robots.txt and reports every major AI crawler split by job, flagging exactly this misconfiguration: Can AI Read This? → No signup, no data stored.

Method (reproducible)

The Tranco top 5,000 domains were fetched over HTTPS (falling back to HTTP), with HTML error pages served at /robots.txt discarded rather than parsed as policy. 2,776 returned a genuine robots.txt. Parsing follows the standard grouping rule — consecutive User-agent lines share one rule block — and applies longest-match precedence with Allow winning ties, evaluated against path /. A site counts as blocking an agent only when the most specific matching group disallows it. Scanner: tools/aicrawl_census.py. Raw data: aicrawl-census.json.

Published by Last Minute Deals HQ LLC. Free to cite and reuse with attribution (CC BY 4.0).