Data · Updated 20 November 2026
Which websites block AI crawlers?
Every week we read the robots.txt of the 10,000 most popular websites and record how each one treats 21 crawlers, from GPTBot and ClaudeBot to Googlebot. Look up any site, compare crawlers and types of site, or download the full data.
At a glance
The six numbers worth remembering.
AI companies read the web in different ways. Some crawlers collect pages to train AI models, some index pages so AI assistants can cite them, and some fetch a page only when a person asks about it. A website's robots.txt file can allow or block each one separately.
This page shows what the 10,000 most popular websites have chosen, using our own weekly scan. The full results for every site are in the lookup below and in the CSV, free to use with a link to this page.
Look up
Check any of the 10,000 websites
Type a domain to see how its robots.txt treats every crawler in this scan. For any other site, our robots.txt checker reads it live.Crawlers
How many sites block each crawler
Training crawlers are blocked about twice as often as AI search crawlers. Search engines like Googlebot are almost never blocked.| Crawler | Blocked by | By name | Partly blocked |
|---|---|---|---|
| GooglebotGoogle · Crawls for Google Search, including AI Overviews and AI Mode. | 1.0% | 7% | 6.1% |
| BingbotMicrosoft · Crawls for Bing search and Microsoft Copilot answers. | 1.7% | 35% | 4.5% |
| ApplebotApple · Crawls for Spotlight, Siri and Safari search features. | 3.2% | 54% | 1.8% |
| DuckDuckBotDuckDuckGo · Crawls to improve DuckDuckGo search results. | 1.5% | 4% | 1.1% |
| Crawler | Blocked by | By name | Partly blocked |
|---|---|---|---|
| OAI-SearchBotOpenAI · Indexes pages for ChatGPT search. Blocking it keeps you out of ChatGPT search answers. | 5.3% | 70% | 3.7% |
| Claude-SearchBotAnthropic · Indexes pages to improve search results in Claude. | 5.3% | 67% | 2.3% |
| PerplexityBotPerplexity · Indexes pages for Perplexity search results. Perplexity says it is not used for model training. | 8.1% | 80% | 3.4% |
| Crawler | Blocked by | By name | Partly blocked |
|---|---|---|---|
| ChatGPT-UserOpenAI · Visits a page when a ChatGPT user or custom GPT asks about it. | 7.1% | 78% | 3.7% |
| Claude-UserAnthropic · Visits a page when a Claude user asks a question that needs it. | 5.5% | 67% | 2.1% |
| Perplexity-UserPerplexity · Visits a page to answer a Perplexity user's question. | 5.7% | 67% | 1.8% |
| Meta-ExternalFetcherMeta · Fetches individual links at a user's request in Meta AI products. | 6.0% | 67% | 0.6% |
| Crawler | Blocked by | By name | Partly blocked |
|---|---|---|---|
| GPTBotOpenAI · Collects content that may be used to train OpenAI's models. | 14.8% | 89% | 4.0% |
| ClaudeBotAnthropic · Collects content that may be used to train Anthropic's models. | 13.8% | 87% | 3.2% |
| Google-ExtendedGoogle · Not a separate crawler. Controls whether Google uses your content for Gemini training and grounding. It has no effect on Google Search. | 11.8% | 86% | 2.9% |
| Applebot-ExtendedApple · Controls whether Apple uses content Applebot crawled to train its AI models. | 11.8% | 88% | 2.6% |
| Meta-ExternalAgentMeta · Crawls for uses including training Meta's AI models. | 12.5% | 85% | 2.0% |
| CCBotCommon Crawl · Builds the open Common Crawl dataset, widely used to train AI models. | 15.6% | 88% | 2.2% |
| BytespiderByteDance · ByteDance's crawler, widely reported to collect AI training data. | 15.4% | 87% | 1.4% |
| AmazonbotAmazon · Crawls for Amazon services, including Alexa answers and AI models. | 11.7% | 84% | 1.9% |
| Crawler | Blocked by | By name | Partly blocked |
|---|---|---|---|
| AhrefsBotAhrefs · Crawls for the Ahrefs backlink and keyword index. | 6.9% | 71% | 1.2% |
| SemrushBotSemrush · Crawls for the Semrush backlink and site audit data. | 6.0% | 68% | 0.9% |
Blocked means the robots.txt stops the crawler reading the homepage. By name is the share of those blocks that name the crawler; the rest come from a rule for all bots. Partly blocked means a rule names the crawler and keeps it out of some sections only.
Training or search
Most sites block training, not search
Sites are choosing what to share. Most that keep AI companies from training on their content still let AI assistants find and cite them.By size
The biggest sites block the most
Blocking is most common among the 1,000 most popular websites.| Sites | GPTBot | ClaudeBot | Google-Extended | CCBot | OAI-SearchBot | PerplexityBot |
|---|---|---|---|---|---|---|
| Top 1,000 | 18.3% | 18.1% | 14.9% | 20.2% | 7.6% | 13.1% |
| 1,001 to 5,000 | 14.6% | 13.6% | 12.4% | 15.9% | 5.8% | 8.8% |
| 5,001 to 10,000 | 14.2% | 13.0% | 10.7% | 14.4% | 4.5% | 6.4% |
By type of site
Which kinds of sites block AI crawlers
Each site is grouped by what it is, from news to shopping to government. Sorted by the share blocking at least one training crawler.| Type of site | Sites | Any training crawler | GPTBot | ClaudeBot | OAI-SearchBot |
|---|---|---|---|---|---|
| News and media | 1,249 | 49.7% | 38.9% | 38.0% | 18.3% |
| Lifestyle | 175 | 46.9% | 14.9% | 27.4% | 4.6% |
| Sport | 197 | 40.6% | 23.4% | 16.2% | 5.1% |
| Video, music and entertainment | 387 | 30.2% | 22.7% | 20.4% | 4.1% |
| Social and community | 249 | 29.3% | 27.3% | 22.1% | 11.6% |
| Games | 328 | 23.2% | 18.6% | 18.9% | 4.3% |
| Health | 156 | 22.4% | 16.0% | 16.7% | 6.4% |
| Travel and transport | 173 | 19.7% | 11.6% | 9.8% | 0.0% |
| Shopping | 851 | 19.6% | 12.8% | 12.0% | 3.6% |
| Jobs and property | 124 | 19.4% | 12.1% | 12.1% | 4.8% |
| Reference and education | 1,012 | 18.5% | 13.3% | 10.4% | 3.0% |
| Search and portals | 99 | 17.2% | 15.2% | 14.1% | 8.1% |
| Business and professional services | 406 | 12.1% | 6.7% | 5.2% | 2.2% |
| Finance | 398 | 10.1% | 6.5% | 7.3% | 2.3% |
| Government and public sector | 510 | 8.6% | 5.9% | 4.7% | 1.8% |
| Technology and software | 2,542 | 7.2% | 4.6% | 4.1% | 2.1% |
Types with fewer than 50 sites are left out of this table but included in the lookup and the CSV.
Top 100
What the 100 biggest websites allow
The six crawlers people ask about most, for the 100 most popular websites in the scan. Look up any other site above.| Website | GPTBot | ClaudeBot | Google-Extended | CCBot | OAI-SearchBot | PerplexityBot |
|---|---|---|---|---|---|---|
| google.com | Partly | Partly | Partly | Partly | Allowed | Allowed |
| cloudflare.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| facebook.com | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked |
| microsoft.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| youtube.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| apple.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| instagram.com | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked |
| twitter.com | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked |
| azure.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| mail.ru | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| office.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| amazon.com | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked |
| live.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| github.com | Partly | Partly | Allowed | Allowed | Partly | Partly |
| dzen.ru | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| wikipedia.org | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| netflix.com | Partly | Partly | Partly | Blocked | Partly | Partly |
| bing.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| pinterest.com | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked |
| icloud.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| zoom.us | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| adobe.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| tiktok.com | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked |
| skype.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| gandi.net | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| digicert.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| wordpress.org | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| yahoo.com | Blocked | Blocked | Blocked | Blocked | Partly | Blocked |
| spotify.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| amazonvideo.com | Allowed | Allowed | Allowed | Blocked | Allowed | Blocked |
| opera.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| roblox.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| vimeo.com | Partly | Partly | Partly | Blocked | Allowed | Allowed |
| ntp.org | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| qq.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| baidu.com | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked |
| blogspot.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| mozilla.org | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| intuit.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| msn.com | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked |
| nginx.org | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| wordpress.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| f5.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| t-online.de | Blocked | Blocked | Partly | Blocked | Allowed | Blocked |
| t.me | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| sentry.io | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| ui.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| chatgpt.com | Allowed | Allowed | Blocked | Blocked | Allowed | Blocked |
| europa.eu | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| reddit.com | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked |
| samsung.com | Partly | Allowed | Partly | Allowed | Partly | Partly |
| openai.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| dropbox.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| android.com | Partly | Partly | Partly | Partly | Partly | Partly |
| apache.org | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| archive.org | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| ipify.org | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| tumblr.com | Allowed | Blocked | Blocked | Blocked | Allowed | Allowed |
| discord.gg | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| nytimes.com | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked |
| ubuntu.com | Partly | Partly | Partly | Allowed | Partly | Partly |
| snapchat.com | Blocked | Blocked | Blocked | Blocked | Partly | Partly |
| shopify.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| vk.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| gravatar.com | Blocked | Allowed | Blocked | Blocked | Allowed | Allowed |
| telekom.net | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| reg.ru | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| creativecommons.org | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| paypal.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| cnn.com | Blocked | Blocked | Allowed | Blocked | Blocked | Blocked |
| w3.org | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| flickr.com | Blocked | Blocked | Blocked | Blocked | Partly | Partly |
| forter.com | Allowed | Allowed | Allowed | Blocked | Allowed | Allowed |
| vk.ru | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| kaspersky.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| soundcloud.com | Partly | Partly | Partly | Partly | Partly | Partly |
| brave.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| ebay.com | Blocked | Blocked | Allowed | Blocked | Partly | Blocked |
| theguardian.com | Allowed | Blocked | Allowed | Blocked | Allowed | Blocked |
| doi.org | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| unity3d.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| medium.com | Blocked | Blocked | Allowed | Allowed | Allowed | Allowed |
| hcaptcha.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| roku.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| mts.ru | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| wikimedia.org | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| stripe.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| bbc.com | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked |
| criteo.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| amplitude.com | Partly | Partly | Partly | Partly | Partly | Partly |
| afternic.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| centinel-analytica.com | Partly | Partly | Partly | Allowed | Partly | Partly |
| forbes.com | Blocked | Blocked | Allowed | Blocked | Allowed | Blocked |
| namecheap.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| weather.com | Allowed | Allowed | Allowed | Blocked | Allowed | Allowed |
| twitch.tv | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| astrumu.io | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| mit.edu | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| nist.gov | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| researchgate.net | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
How sites block
Named rules, catch-all rules and one-click settings
Sites block AI crawlers in three ways: a rule that names the crawler, a rule that blocks all bots, or a setting from their hosting or security provider.How to
How to block or allow AI crawlers
Add these groups to your robots.txt, then check the result with our robots.txt checker.Block AI training, stay in AI search
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: Meta-ExternalAgent Disallow: /
Block every major AI crawler
User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: Meta-ExternalFetcher User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Meta-ExternalAgent User-agent: CCBot User-agent: Bytespider User-agent: Amazonbot Disallow: /
Search engines such as Googlebot and Bingbot are left out of both, so the site stays in Google and Bing. Google-Extended and Applebot-Extended aren't separate crawlers; they tell Google and Apple whether they may use content for AI training.
Questions
Quick answers
What are AI crawlers?+
AI crawlers are bots that AI companies send to read websites. Training crawlers such as GPTBot, ClaudeBot and CCBot collect pages that may be used to train AI models. AI search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages so AI assistants can find and cite them. User-triggered fetchers such as ChatGPT-User visit a page when someone asks an assistant about it.
How many websites block AI crawlers?+
In our scan on 20 November 2026, 20.7% of the 10,000 most popular websites blocked at least one AI training crawler in their robots.txt, and 10.2% blocked at least one AI search or user-triggered crawler.
How many websites block GPTBot?+
14.8% of the top 10,000 websites block GPTBot, rising to 18.3% among the top 1,000.
Does blocking GPTBot remove a site from ChatGPT?+
No. GPTBot only collects training data. ChatGPT search uses OAI-SearchBot, and OpenAI says each setting is independent. 65% of the sites that block GPTBot in our scan still allow OAI-SearchBot.
Which AI crawler is blocked most?+
CCBot is blocked by the most sites in our scan, 15.6% of the top 10,000.
Is ChatGPT a web crawler?+
ChatGPT itself isn't, but OpenAI runs three bots: GPTBot for training, OAI-SearchBot for ChatGPT search and ChatGPT-User for pages people ask ChatGPT to read.
How do I block AI crawlers?+
Add a group to your robots.txt for each crawler you want to block, with User-agent set to the crawler's name and Disallow: /. To keep appearing in AI answers, block only the training crawlers and leave the search crawlers allowed. Our robots.txt checker shows how any site treats each crawler.
Does Reddit block AI crawlers?+
Yes. As of 20 November 2026, the robots.txt on reddit.com has one rule asking every crawler to stay out of the whole site, so all 15 AI crawlers we track are blocked, along with search engines such as Googlebot.
Does the New York Times block AI crawlers?+
Mostly. As of 20 November 2026, the robots.txt on nytimes.com blocks 14 of the 15 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and OAI-SearchBot. Blocking OAI-SearchBot means the New York Times opts out of ChatGPT search as well as AI training. It keeps Amazonbot out of some sections only.
Does Amazon block AI crawlers?+
Mostly. As of 20 November 2026, the robots.txt on amazon.com blocks 13 of the 15 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and OAI-SearchBot. Blocking OAI-SearchBot means Amazon opts out of ChatGPT search as well as AI training. It still allows Applebot-Extended and Amazonbot.
Does Google block AI crawlers in its robots.txt?+
Partly. As of 20 November 2026, the robots.txt on google.com doesn't fully block any of the 15 AI crawlers we track. It keeps GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider and Amazonbot out of some sections only. OAI-SearchBot is still allowed, so Google can appear in ChatGPT search.
Does the BBC block AI crawlers?+
Mostly. As of 20 November 2026, the robots.txt on bbc.co.uk blocks 13 of the 15 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and OAI-SearchBot. Blocking OAI-SearchBot means the BBC opts out of ChatGPT search as well as AI training. It still allows Claude-User and Meta-ExternalFetcher.
How often is this data updated?+
Every week. Each scan reads the robots.txt of the same panel of 10,000 websites, so changes from one week to the next show sites changing their rules.
Methodology
How we built this page
We start from a research ranking of the most popular domains on the web, which combines several sources of traffic data, and take the top 25,000. We keep domains that serve a website, leave out adult, malware and gambling sites and background services such as content delivery networks, and count a domain that redirects to a site already listed only once. The first 10,000 that remain form the panel, and the same panel is scanned every week.
For each site we read /robots.txt and apply its rules the way RFC 9309 and Google describe: the most specific user agent group wins and the longest matching rule wins. A crawler counts as blocked when it may not read the homepage. A site with no robots.txt allows every crawler. Percentages are of the 9,043 sites whose robots.txt we could read; 888 sites didn't share it with our scanner and are shown as such in the lookup and CSV.
Each site's type comes from its homepage title and description. Our scanner identifies itself as RibbiaBot. The data is free to use under CC BY 4.0 with a link to this page. Questions or corrections: research@ribbia.com.
Cite this page: Ribbia Research (2026). Which websites block AI crawlers?. ribbia.com/ai-crawlers. Updated 20 November 2026.
Sources
AI crawlers sources
- RFC 9309: Robots Exclusion ProtocolOpen ↗
rfc-editor.org · Used for how robots.txt rules are read
- Cloudflare: managed robots.txtOpen ↗
developers.cloudflare.com · Used for identifying Cloudflare's managed robots.txt
- Google crawler documentationOpen ↗
developers.google.com · Used for crawler names and what each one does
- Microsoft crawler documentationOpen ↗
bing.com · Used for crawler names and what each one does
- Apple crawler documentationOpen ↗
support.apple.com · Used for crawler names and what each one does
- DuckDuckGo crawler documentationOpen ↗
duckduckgo.com · Used for crawler names and what each one does
- OpenAI crawler documentationOpen ↗
developers.openai.com · Used for crawler names and what each one does
- Anthropic crawler documentationOpen ↗
support.claude.com · Used for crawler names and what each one does
- Perplexity crawler documentationOpen ↗
docs.perplexity.ai · Used for crawler names and what each one does
- Meta crawler documentationOpen ↗
developers.facebook.com · Used for crawler names and what each one does
- Common Crawl crawler documentationOpen ↗
commoncrawl.org · Used for crawler names and what each one does
- Amazon crawler documentationOpen ↗
developer.amazon.com · Used for crawler names and what each one does
- Ahrefs crawler documentationOpen ↗
ahrefs.com · Used for crawler names and what each one does
- Semrush crawler documentationOpen ↗
semrush.com · Used for crawler names and what each one does