Home/Research/AI crawlers

Data · Updated 20 November 2026

Which websites block AI crawlers?

Every week we read the robots.txt of the 10,000 most popular websites and record how each one treats 21 crawlers, from GPTBot and ClaudeBot to Googlebot. Look up any site, compare crawlers and types of site, or download the full data.

At a glance

The six numbers worth remembering.

20.7%of top websites block at least one AI training crawler1,868 of 9,043 sites
14.8%block GPTBot, OpenAI's training crawler1,334 sites
65%of sites that block GPTBot still allow ChatGPT searchBlocking training while staying findable
5.3%block OAI-SearchBot, which powers ChatGPT search481 sites
18.3%of the top 1,000 websites block GPTBotThe biggest sites block the most
16%of GPTBot blocks come from Cloudflare's managed robots.txtA one-click setting

AI companies read the web in different ways. Some crawlers collect pages to train AI models, some index pages so AI assistants can cite them, and some fetch a page only when a person asks about it. A website's robots.txt file can allow or block each one separately.

This page shows what the 10,000 most popular websites have chosen, using our own weekly scan. The full results for every site are in the lookup below and in the CSV, free to use with a link to this page.

Look up

Check any of the 10,000 websites

Type a domain to see how its robots.txt treats every crawler in this scan. For any other site, our robots.txt checker reads it live.

Crawlers

How many sites block each crawler

Training crawlers are blocked about twice as often as AI search crawlers. Search engines like Googlebot are almost never blocked.
Search enginesIndex pages for classic search results.
CrawlerBlocked byBy namePartly blocked
GooglebotGoogle · Crawls for Google Search, including AI Overviews and AI Mode.1.0%7%6.1%
BingbotMicrosoft · Crawls for Bing search and Microsoft Copilot answers.1.7%35%4.5%
ApplebotApple · Crawls for Spotlight, Siri and Safari search features.3.2%54%1.8%
DuckDuckBotDuckDuckGo · Crawls to improve DuckDuckGo search results.1.5%4%1.1%
AI searchIndex pages so AI assistants can find and cite them in answers.
CrawlerBlocked byBy namePartly blocked
OAI-SearchBotOpenAI · Indexes pages for ChatGPT search. Blocking it keeps you out of ChatGPT search answers.5.3%70%3.7%
Claude-SearchBotAnthropic · Indexes pages to improve search results in Claude.5.3%67%2.3%
PerplexityBotPerplexity · Indexes pages for Perplexity search results. Perplexity says it is not used for model training.8.1%80%3.4%
User-triggered fetchesVisit a page when someone asks an AI assistant about it.
CrawlerBlocked byBy namePartly blocked
ChatGPT-UserOpenAI · Visits a page when a ChatGPT user or custom GPT asks about it.7.1%78%3.7%
Claude-UserAnthropic · Visits a page when a Claude user asks a question that needs it.5.5%67%2.1%
Perplexity-UserPerplexity · Visits a page to answer a Perplexity user's question.5.7%67%1.8%
Meta-ExternalFetcherMeta · Fetches individual links at a user's request in Meta AI products.6.0%67%0.6%
AI trainingCollect content that may be used to train AI models.
CrawlerBlocked byBy namePartly blocked
GPTBotOpenAI · Collects content that may be used to train OpenAI's models.14.8%89%4.0%
ClaudeBotAnthropic · Collects content that may be used to train Anthropic's models.13.8%87%3.2%
Google-ExtendedGoogle · Not a separate crawler. Controls whether Google uses your content for Gemini training and grounding. It has no effect on Google Search.11.8%86%2.9%
Applebot-ExtendedApple · Controls whether Apple uses content Applebot crawled to train its AI models.11.8%88%2.6%
Meta-ExternalAgentMeta · Crawls for uses including training Meta's AI models.12.5%85%2.0%
CCBotCommon Crawl · Builds the open Common Crawl dataset, widely used to train AI models.15.6%88%2.2%
BytespiderByteDance · ByteDance's crawler, widely reported to collect AI training data.15.4%87%1.4%
AmazonbotAmazon · Crawls for Amazon services, including Alexa answers and AI models.11.7%84%1.9%
SEO toolsCrawl for backlink and keyword databases.
CrawlerBlocked byBy namePartly blocked
AhrefsBotAhrefs · Crawls for the Ahrefs backlink and keyword index.6.9%71%1.2%
SemrushBotSemrush · Crawls for the Semrush backlink and site audit data.6.0%68%0.9%

Blocked means the robots.txt stops the crawler reading the homepage. By name is the share of those blocks that name the crawler; the rest come from a rule for all bots. Partly blocked means a rule names the crawler and keeps it out of some sections only.

By size

The biggest sites block the most

Blocking is most common among the 1,000 most popular websites.
Share of sites blocking each crawler, by popularity
SitesGPTBotClaudeBotGoogle-ExtendedCCBotOAI-SearchBotPerplexityBot
Top 1,00018.3%18.1%14.9%20.2%7.6%13.1%
1,001 to 5,00014.6%13.6%12.4%15.9%5.8%8.8%
5,001 to 10,00014.2%13.0%10.7%14.4%4.5%6.4%

By type of site

Which kinds of sites block AI crawlers

Each site is grouped by what it is, from news to shopping to government. Sorted by the share blocking at least one training crawler.
AI crawler blocking by type of site
Type of siteSitesAny training crawlerGPTBotClaudeBotOAI-SearchBot
News and media1,24949.7%38.9%38.0%18.3%
Lifestyle17546.9%14.9%27.4%4.6%
Sport19740.6%23.4%16.2%5.1%
Video, music and entertainment38730.2%22.7%20.4%4.1%
Social and community24929.3%27.3%22.1%11.6%
Games32823.2%18.6%18.9%4.3%
Health15622.4%16.0%16.7%6.4%
Travel and transport17319.7%11.6%9.8%0.0%
Shopping85119.6%12.8%12.0%3.6%
Jobs and property12419.4%12.1%12.1%4.8%
Reference and education1,01218.5%13.3%10.4%3.0%
Search and portals9917.2%15.2%14.1%8.1%
Business and professional services40612.1%6.7%5.2%2.2%
Finance39810.1%6.5%7.3%2.3%
Government and public sector5108.6%5.9%4.7%1.8%
Technology and software2,5427.2%4.6%4.1%2.1%

Types with fewer than 50 sites are left out of this table but included in the lookup and the CSV.

Top 100

What the 100 biggest websites allow

The six crawlers people ask about most, for the 100 most popular websites in the scan. Look up any other site above.
AI crawler rules on the 100 most popular websites
WebsiteGPTBotClaudeBotGoogle-ExtendedCCBotOAI-SearchBotPerplexityBot
google.comPartlyPartlyPartlyPartlyAllowedAllowed
cloudflare.comAllowedAllowedAllowedAllowedAllowedAllowed
facebook.comBlockedBlockedBlockedBlockedBlockedBlocked
microsoft.comAllowedAllowedAllowedAllowedAllowedAllowed
youtube.comAllowedAllowedAllowedAllowedAllowedAllowed
apple.comAllowedAllowedAllowedAllowedAllowedAllowed
instagram.comBlockedBlockedBlockedBlockedBlockedBlocked
twitter.comBlockedBlockedBlockedBlockedBlockedBlocked
azure.comAllowedAllowedAllowedAllowedAllowedAllowed
mail.ruAllowedAllowedAllowedAllowedAllowedAllowed
office.comAllowedAllowedAllowedAllowedAllowedAllowed
amazon.comBlockedBlockedBlockedBlockedBlockedBlocked
live.comAllowedAllowedAllowedAllowedAllowedAllowed
github.comPartlyPartlyAllowedAllowedPartlyPartly
dzen.ruAllowedAllowedAllowedAllowedAllowedAllowed
wikipedia.orgAllowedAllowedAllowedAllowedAllowedAllowed
netflix.comPartlyPartlyPartlyBlockedPartlyPartly
bing.comAllowedAllowedAllowedAllowedAllowedAllowed
pinterest.comBlockedBlockedBlockedBlockedBlockedBlocked
icloud.comAllowedAllowedAllowedAllowedAllowedAllowed
zoom.usAllowedAllowedAllowedAllowedAllowedAllowed
adobe.comAllowedAllowedAllowedAllowedAllowedAllowed
tiktok.comBlockedBlockedBlockedBlockedBlockedBlocked
skype.comAllowedAllowedAllowedAllowedAllowedAllowed
gandi.netAllowedAllowedAllowedAllowedAllowedAllowed
digicert.comAllowedAllowedAllowedAllowedAllowedAllowed
wordpress.orgAllowedAllowedAllowedAllowedAllowedAllowed
yahoo.comBlockedBlockedBlockedBlockedPartlyBlocked
spotify.comAllowedAllowedAllowedAllowedAllowedAllowed
amazonvideo.comAllowedAllowedAllowedBlockedAllowedBlocked
opera.comAllowedAllowedAllowedAllowedAllowedAllowed
roblox.comAllowedAllowedAllowedAllowedAllowedAllowed
vimeo.comPartlyPartlyPartlyBlockedAllowedAllowed
ntp.orgAllowedAllowedAllowedAllowedAllowedAllowed
qq.comAllowedAllowedAllowedAllowedAllowedAllowed
baidu.comBlockedBlockedBlockedBlockedBlockedBlocked
blogspot.comAllowedAllowedAllowedAllowedAllowedAllowed
mozilla.orgAllowedAllowedAllowedAllowedAllowedAllowed
intuit.comAllowedAllowedAllowedAllowedAllowedAllowed
msn.comBlockedBlockedBlockedBlockedBlockedBlocked
nginx.orgAllowedAllowedAllowedAllowedAllowedAllowed
wordpress.comAllowedAllowedAllowedAllowedAllowedAllowed
f5.comAllowedAllowedAllowedAllowedAllowedAllowed
t-online.deBlockedBlockedPartlyBlockedAllowedBlocked
t.meAllowedAllowedAllowedAllowedAllowedAllowed
sentry.ioAllowedAllowedAllowedAllowedAllowedAllowed
ui.comAllowedAllowedAllowedAllowedAllowedAllowed
chatgpt.comAllowedAllowedBlockedBlockedAllowedBlocked
europa.euAllowedAllowedAllowedAllowedAllowedAllowed
reddit.comBlockedBlockedBlockedBlockedBlockedBlocked
samsung.comPartlyAllowedPartlyAllowedPartlyPartly
openai.comAllowedAllowedAllowedAllowedAllowedAllowed
dropbox.comAllowedAllowedAllowedAllowedAllowedAllowed
android.comPartlyPartlyPartlyPartlyPartlyPartly
apache.orgAllowedAllowedAllowedAllowedAllowedAllowed
archive.orgAllowedAllowedAllowedAllowedAllowedAllowed
ipify.orgAllowedAllowedAllowedAllowedAllowedAllowed
tumblr.comAllowedBlockedBlockedBlockedAllowedAllowed
discord.ggAllowedAllowedAllowedAllowedAllowedAllowed
nytimes.comBlockedBlockedBlockedBlockedBlockedBlocked
ubuntu.comPartlyPartlyPartlyAllowedPartlyPartly
snapchat.comBlockedBlockedBlockedBlockedPartlyPartly
shopify.comAllowedAllowedAllowedAllowedAllowedAllowed
vk.comAllowedAllowedAllowedAllowedAllowedAllowed
gravatar.comBlockedAllowedBlockedBlockedAllowedAllowed
telekom.netAllowedAllowedAllowedAllowedAllowedAllowed
reg.ruAllowedAllowedAllowedAllowedAllowedAllowed
creativecommons.orgAllowedAllowedAllowedAllowedAllowedAllowed
paypal.comAllowedAllowedAllowedAllowedAllowedAllowed
cnn.comBlockedBlockedAllowedBlockedBlockedBlocked
w3.orgAllowedAllowedAllowedAllowedAllowedAllowed
flickr.comBlockedBlockedBlockedBlockedPartlyPartly
forter.comAllowedAllowedAllowedBlockedAllowedAllowed
vk.ruAllowedAllowedAllowedAllowedAllowedAllowed
kaspersky.comAllowedAllowedAllowedAllowedAllowedAllowed
soundcloud.comPartlyPartlyPartlyPartlyPartlyPartly
brave.comAllowedAllowedAllowedAllowedAllowedAllowed
ebay.comBlockedBlockedAllowedBlockedPartlyBlocked
theguardian.comAllowedBlockedAllowedBlockedAllowedBlocked
doi.orgAllowedAllowedAllowedAllowedAllowedAllowed
unity3d.comAllowedAllowedAllowedAllowedAllowedAllowed
medium.comBlockedBlockedAllowedAllowedAllowedAllowed
hcaptcha.comAllowedAllowedAllowedAllowedAllowedAllowed
roku.comAllowedAllowedAllowedAllowedAllowedAllowed
mts.ruAllowedAllowedAllowedAllowedAllowedAllowed
wikimedia.orgAllowedAllowedAllowedAllowedAllowedAllowed
stripe.comAllowedAllowedAllowedAllowedAllowedAllowed
bbc.comBlockedBlockedBlockedBlockedBlockedBlocked
criteo.comAllowedAllowedAllowedAllowedAllowedAllowed
amplitude.comPartlyPartlyPartlyPartlyPartlyPartly
afternic.comAllowedAllowedAllowedAllowedAllowedAllowed
centinel-analytica.comPartlyPartlyPartlyAllowedPartlyPartly
forbes.comBlockedBlockedAllowedBlockedAllowedBlocked
namecheap.comAllowedAllowedAllowedAllowedAllowedAllowed
weather.comAllowedAllowedAllowedBlockedAllowedAllowed
twitch.tvAllowedAllowedAllowedAllowedAllowedAllowed
astrumu.ioAllowedAllowedAllowedAllowedAllowedAllowed
mit.eduAllowedAllowedAllowedAllowedAllowedAllowed
nist.govAllowedAllowedAllowedAllowedAllowedAllowed
researchgate.netAllowedAllowedAllowedAllowedAllowedAllowed

How sites block

Named rules, catch-all rules and one-click settings

Sites block AI crawlers in three ways: a rule that names the crawler, a rule that blocks all bots, or a setting from their hosting or security provider.
89%of GPTBot blocks come from a rule that names GPTBot
221sites use Cloudflare's managed robots.txt, which blocks AI training crawlers
473sites publish a Content-Signal line saying how their content may be used, 358 of them with ai-train=no

How to

How to block or allow AI crawlers

Add these groups to your robots.txt, then check the result with our robots.txt checker.

Block AI training, stay in AI search

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

Block every major AI crawler

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: Meta-ExternalFetcher
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
User-agent: Bytespider
User-agent: Amazonbot
Disallow: /

Search engines such as Googlebot and Bingbot are left out of both, so the site stays in Google and Bing. Google-Extended and Applebot-Extended aren't separate crawlers; they tell Google and Apple whether they may use content for AI training.

Questions

Quick answers

What are AI crawlers?+

AI crawlers are bots that AI companies send to read websites. Training crawlers such as GPTBot, ClaudeBot and CCBot collect pages that may be used to train AI models. AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages so AI assistants can find and cite them. User-triggered fetchers such as ChatGPT-User visit a page when someone asks an assistant about it.

How many websites block AI crawlers?+

In our scan on 20 November 2026, 20.7% of the 10,000 most popular websites blocked at least one AI training crawler in their robots.txt, and 10.2% blocked at least one AI search or user-triggered crawler.

How many websites block GPTBot?+

14.8% of the top 10,000 websites block GPTBot, rising to 18.3% among the top 1,000.

Does blocking GPTBot remove a site from ChatGPT?+

No. GPTBot only collects training data. ChatGPT search uses OAI-SearchBot, and OpenAI says each setting is independent. 65% of the sites that block GPTBot in our scan still allow OAI-SearchBot.

Which AI crawler is blocked most?+

CCBot is blocked by the most sites in our scan, 15.6% of the top 10,000.

Is ChatGPT a web crawler?+

ChatGPT itself isn't, but OpenAI runs three bots: GPTBot for training, OAI-SearchBot for ChatGPT search and ChatGPT-User for pages people ask ChatGPT to read.

How do I block AI crawlers?+

Add a group to your robots.txt for each crawler you want to block, with User-agent set to the crawler's name and Disallow: /. To keep appearing in AI answers, block only the training crawlers and leave the search crawlers allowed. Our robots.txt checker shows how any site treats each crawler.

Does Reddit block AI crawlers?+

Yes. As of 20 November 2026, the robots.txt on reddit.com has one rule asking every crawler to stay out of the whole site, so all 15 AI crawlers we track are blocked, along with search engines such as Googlebot.

Does the New York Times block AI crawlers?+

Mostly. As of 20 November 2026, the robots.txt on nytimes.com blocks 14 of the 15 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and OAI-SearchBot. Blocking OAI-SearchBot means the New York Times opts out of ChatGPT search as well as AI training. It keeps Amazonbot out of some sections only.

Does Amazon block AI crawlers?+

Mostly. As of 20 November 2026, the robots.txt on amazon.com blocks 13 of the 15 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and OAI-SearchBot. Blocking OAI-SearchBot means Amazon opts out of ChatGPT search as well as AI training. It still allows Applebot-Extended and Amazonbot.

Does Google block AI crawlers in its robots.txt?+

Partly. As of 20 November 2026, the robots.txt on google.com doesn't fully block any of the 15 AI crawlers we track. It keeps GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider and Amazonbot out of some sections only. OAI-SearchBot is still allowed, so Google can appear in ChatGPT search.

Does the BBC block AI crawlers?+

Mostly. As of 20 November 2026, the robots.txt on bbc.co.uk blocks 13 of the 15 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and OAI-SearchBot. Blocking OAI-SearchBot means the BBC opts out of ChatGPT search as well as AI training. It still allows Claude-User and Meta-ExternalFetcher.

How often is this data updated?+

Every week. Each scan reads the robots.txt of the same panel of 10,000 websites, so changes from one week to the next show sites changing their rules.

Methodology

How we built this page

We start from a research ranking of the most popular domains on the web, which combines several sources of traffic data, and take the top 25,000. We keep domains that serve a website, leave out adult, malware and gambling sites and background services such as content delivery networks, and count a domain that redirects to a site already listed only once. The first 10,000 that remain form the panel, and the same panel is scanned every week.

For each site we read /robots.txt and apply its rules the way RFC 9309 and Google describe: the most specific user agent group wins and the longest matching rule wins. A crawler counts as blocked when it may not read the homepage. A site with no robots.txt allows every crawler. Percentages are of the 9,043 sites whose robots.txt we could read; 888 sites didn't share it with our scanner and are shown as such in the lookup and CSV.

Each site's type comes from its homepage title and description. Our scanner identifies itself as RibbiaBot. The data is free to use under CC BY 4.0 with a link to this page. Questions or corrections: research@ribbia.com.

Cite this page: Ribbia Research (2026). Which websites block AI crawlers?. ribbia.com/ai-crawlers. Updated 20 November 2026.

Sources

AI crawlers sources

  1. RFC 9309: Robots Exclusion Protocol

    rfc-editor.org · Used for how robots.txt rules are read

    Open ↗
  2. Cloudflare: managed robots.txt

    developers.cloudflare.com · Used for identifying Cloudflare's managed robots.txt

    Open ↗
  3. Google crawler documentation

    developers.google.com · Used for crawler names and what each one does

    Open ↗
  4. Microsoft crawler documentation

    bing.com · Used for crawler names and what each one does

    Open ↗
  5. Apple crawler documentation

    support.apple.com · Used for crawler names and what each one does

    Open ↗
  6. DuckDuckGo crawler documentation

    duckduckgo.com · Used for crawler names and what each one does

    Open ↗
  7. OpenAI crawler documentation

    developers.openai.com · Used for crawler names and what each one does

    Open ↗
  8. Anthropic crawler documentation

    support.claude.com · Used for crawler names and what each one does

    Open ↗
  9. Perplexity crawler documentation

    docs.perplexity.ai · Used for crawler names and what each one does

    Open ↗
  10. Meta crawler documentation

    developers.facebook.com · Used for crawler names and what each one does

    Open ↗
  11. Common Crawl crawler documentation

    commoncrawl.org · Used for crawler names and what each one does

    Open ↗
  12. Amazon crawler documentation

    developer.amazon.com · Used for crawler names and what each one does

    Open ↗
  13. Ahrefs crawler documentation

    ahrefs.com · Used for crawler names and what each one does

    Open ↗
  14. Semrush crawler documentation

    semrush.com · Used for crawler names and what each one does

    Open ↗

Ribbia

Want pages like this
for your brand?

Talk to us