How to read your results
Each crawler shows Allowed or Blocked for the path you're testing, and the line in your robots.txt that made the decision. If no line matches, the crawler is allowed. Change the path to test another page, or edit the file and the results update as you type, so you can try a fix before it goes live.
Crawlers are grouped by what they do. That matters more than ever, because blocking the wrong AI crawler can take you out of AI answers without stopping your content being used for training, or the other way round.
- Search engines build the classic search index. Googlebot also feeds Google's AI Overviews and AI Mode, so blocking it removes you from both.
- AI search crawlers index pages so assistants can find and cite them. Block OAI-SearchBot and you won't appear in ChatGPT search answers.
- User-triggered fetches happen when someone asks an assistant about a specific page. They act on a person's request rather than crawling the web.
- AI training tokens control whether your content can be used to train models. Blocking them doesn't affect search.
Should you block AI crawlers?
For most businesses, the answer is to stay visible in AI answers and decide separately about training. Buyers increasingly ask ChatGPT, Claude, Perplexity and Google's AI features for recommendations, and those tools can only cite pages they can read.
| Your goal | Allow | Block |
|---|---|---|
| Be found everywhere | Everything | Nothing |
| Be cited by AI, but not used for training | Search engines, AI search, user-triggered fetches | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider |
| Keep paid or private content out of AI | Search engines for public pages | All AI crawlers on the paths you want to protect |
| Keep a staging site out of everything | Nothing | Everything, plus password protection |
The generator below writes any of these for you.
Robots.txt disallow all and allow all
These are the rules people look up most. Paste any of them into the checker above to see exactly which crawlers they block.
Disallow all
Blocks every crawler that follows robots.txt from every page:
User-agent: * Disallow: /
Allow all
Any of these lets every crawler in. They all do the same thing:
User-agent: * Disallow:
User-agent: * Allow: /
An empty robots.txt, or none at all, also allows everything. Look closely at the first two examples: Disallow: allows everything and Disallow: / blocks everything. One character separates them.
Disallow all except one folder or page
User-agent: * Disallow: / Allow: /public/
For the homepage only, use Allow: /$. The $ stops it matching every URL. Line order doesn't matter to Google or Bing. The longer matching rule wins, and Allow wins a tie (see how the checker decides). Bear in mind that an allowlist blocks every new page you publish until you add it, and that an allowed path stays allowed with any query string on the end.
Block one crawler, or allow only one
User-agent: GPTBot Disallow: /
User-agent: Googlebot Disallow: User-agent: * Disallow: /
The first blocks only GPTBot. The second lets Googlebot in and blocks everything else, which also removes you from Bing, ChatGPT search and every other AI answer engine. To block AI training but stay in search and AI answers, use the generator's “block AI training” preset rather than disallow all.
Disallow all won't remove pages from Google
Robots.txt controls crawling, not indexing. Pages Google has already indexed can stay in results, often with no description. Search Console reports them as “Indexed, though blocked by robots.txt”. To take a site out of search:
- Leave the pages crawlable and add a
noindexmeta tag orX-Robots-Tag: noindexheader. - For anything urgent, use the Removals tool in Search Console. It hides URLs for about six months.
- Wait until the pages drop out of the index, then add disallow all if you also want crawling to stop.
If you block first, Google can't crawl the pages to see the noindex, so they stay indexed.
Staging sites
Disallow all is a sensible extra on a staging or development site, but don't rely on it alone. The file is public, and crawlers that ignore it can still read everything. Put staging behind a password. Then check on launch day, because a staging robots.txt copied to the live site is one of the most common ways sites lose their traffic overnight. Audit tools won't necessarily warn you. Lighthouse, for example, can still give a blocked site a respectable SEO score. The checker above is the quickest way to confirm the live file lets crawlers in.
How the checker decides
The checker follows the Robots Exclusion Protocol (RFC 9309) and Google's documented behaviour, which most major crawlers share.
- Pick the group. A crawler obeys the group with the most specific User-agent line that matches its name, and ignores every other group. Googlebot-News, for example, uses a
googlebotgroup if there's nogooglebot-newsgroup. Only if nothing matches does it fall back toUser-agent: *. This catches out a lot of sites: aUser-agent: GPTBotgroup with one Disallow line means GPTBot ignores every rule in your*group. - Pick the rule. Within that group, the rule with the longest matching path wins. If an Allow and a Disallow match with the same length, Allow wins.
- Wildcards.
*matches any run of characters and$marks the end of the URL, soDisallow: /*.pdf$blocks PDFs but not/file.pdf?download=1. - No match means allowed. And robots.txt itself can always be fetched.
Common robots.txt mistakes
- Using Disallow to remove pages from Google. It stops crawling, so Google can't see a noindex tag and may keep the URL indexed anyway.
- Blocking CSS and JavaScript. Google renders pages like a browser. If it can't load your scripts and styles, it may not understand the page.
- Noindex or Crawl-delay lines. Google stopped supporting noindex in robots.txt in 2019 and never supported Crawl-delay. The checker flags both.
- A server error on robots.txt. A 5xx response can stop Google crawling your whole site. Some security tools and CDNs block crawlers from robots.txt by accident.
- Copying a staging file to production.
Disallow: /underUser-agent: *blocks every crawler from the whole site. - Relative sitemap URLs. Sitemap lines must be full URLs, starting with https://. Our guide to adding a sitemap to robots.txt covers the rest.
The crawlers we check
The user agent token is the name to put after User-agent:. Each link goes to the company's own documentation.
| Token | Company | What it does |
|---|---|---|
| Googlebot | Crawls for Google Search, including AI Overviews and AI Mode. | |
| Bingbot | Microsoft | Crawls for Bing search and Microsoft Copilot answers. |
| Applebot | Apple | Crawls for Spotlight, Siri and Safari search features. |
| DuckDuckBot | DuckDuckGo | Crawls to improve DuckDuckGo search results. |
| Token | Company | What it does |
|---|---|---|
| OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search. Blocking it keeps you out of ChatGPT search answers. |
| Claude-SearchBot | Anthropic | Indexes pages to improve search results in Claude. |
| PerplexityBot | Perplexity | Indexes pages for Perplexity search results. Perplexity says it is not used for model training. |
| Token | Company | What it does |
|---|---|---|
| ChatGPT-User | OpenAI | Visits a page when a ChatGPT user or custom GPT asks about it. |
| Claude-User | Anthropic | Visits a page when a Claude user asks a question that needs it. |
| Perplexity-User | Perplexity | Visits a page to answer a Perplexity user's question. |
| Meta-ExternalFetcher | Meta | Fetches individual links at a user's request in Meta AI products. |
| Token | Company | What it does |
|---|---|---|
| GPTBot | OpenAI | Collects content that may be used to train OpenAI's models. |
| ClaudeBot | Anthropic | Collects content that may be used to train Anthropic's models. |
| Google-Extended | Not a separate crawler. Controls whether Google uses your content for Gemini training and grounding. It has no effect on Google Search. | |
| Applebot-Extended | Apple | Controls whether Apple uses content Applebot crawled to train its AI models. |
| Meta-ExternalAgent | Meta | Crawls for uses including training Meta's AI models. |
| CCBot | Common Crawl | Builds the open Common Crawl dataset, widely used to train AI models. |
| Bytespider | ByteDance | ByteDance's crawler, widely reported to collect AI training data. |
| Amazonbot | Amazon | Crawls for Amazon services, including Alexa answers and AI models. |
| Token | Company | What it does |
|---|---|---|
| AhrefsBot | Ahrefs | Crawls for the Ahrefs backlink and keyword index. |
| SemrushBot | Semrush | Crawls for the Semrush backlink and site audit data. |
Robots.txt generator
Pick a starting point, add any paths you want crawlers to stay out of, and copy or download the file.
User-agent: * Disallow: /admin/ Disallow: /cart Disallow: /checkout User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Amazonbot Disallow: / Sitemap: https://example.com/sitemap.xml
FAQ
Does Google still have a robots.txt tester?+
Google retired the old robots.txt Tester in Search Console in 2023. Search Console now has a robots.txt report that shows which robots.txt files Google found and any errors, but it doesn't let you test a URL against a specific crawler. This checker does both, for Google and the main AI crawlers.
Will blocking GPTBot remove my site from ChatGPT?+
No. GPTBot controls whether your content can be used to train OpenAI's models. ChatGPT search uses OAI-SearchBot, and pages users ask about are fetched by ChatGPT-User. OpenAI says each setting is independent, so you can block GPTBot and still appear in ChatGPT search.
Does robots.txt stop a page being indexed?+
No. Robots.txt stops crawling, not indexing. Google can still index a blocked URL if other pages link to it, usually showing it without a description. To keep a page out of search results, allow crawling and add a noindex meta tag or X-Robots-Tag header.
How long does a robots.txt change take to work?+
Google generally caches robots.txt for up to 24 hours. OpenAI and Perplexity both say their systems can take about 24 hours to reflect a change.
Does robots.txt cover my subdomains?+
No. Each host needs its own file. Rules at example.com/robots.txt don't apply to shop.example.com or blog.example.com.
What happens if my robots.txt returns an error?+
If it returns a 4xx error other than 429, Google acts as if there is no robots.txt and crawls everything. If it returns a 5xx error, Google stops crawling the whole site for up to 12 hours, then uses its last good copy for up to 30 days.
Is there a size limit?+
Google reads the first 500 KiB of a robots.txt file and ignores everything after it.
Do all AI crawlers obey robots.txt?+
The major ones from OpenAI, Anthropic, Google, Perplexity, Apple and Meta document that they do. Robots.txt is a request, not a lock, so crawlers that ignore it need blocking at your server or firewall.
Sources
Rather not do this yourself?
Ribbia fixes crawling, indexing and AI visibility for growing businesses, and shows you the results in your own Search Console.
Get a free audit