Home/Tools/Sitemap in robots.txt

Guide and free checker

How to add a sitemap to robots.txt

Add one line to your robots.txt and every crawler that reads it can find your sitemap. Here's the line, the rules Google and Bing follow, and how to do it on WordPress, Shopify, Wix, Squarespace and Next.js. Check your own site first:

The line to add

Add a Sitemap: line with the full URL of your sitemap:

Sitemap: https://www.example.com/sitemap.xml

A complete robots.txt with a sitemap usually looks like this:

User-agent: *
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

Save the file as plain text, encoded in UTF-8, at the root of your site so it loads at https://www.example.com/robots.txt. Then open that address in a private browser window to confirm the live file has the line, and run it through our robots.txt checker to make sure the rest of the file isn't blocking anything it shouldn't.

The rules that matter

  • Use the full URL. Sitemap: /sitemap.xml is ignored. Include https:// and the exact host, because Google won't guess the http or www version for you.
  • It can go anywhere in the file. The Sitemap line isn't part of any User-agent group, so it applies to every crawler wherever you put it. The end of the file is the tidiest place.
  • The URL is case sensitive. sitemap: and Sitemap: both work, but /Sitemap.xml and /sitemap.xml are different files.
  • No wildcards. Sitemap is the one robots.txt line where * doesn't work. List each file.
  • Don't block the sitemap. If a Disallow rule covers the sitemap's URL, crawlers that obey it can't read the sitemap. The checker above tests for this.
  • It must return 200. A sitemap that redirects or returns an error is no use to crawlers. The checker tests this too.

The placement mistake that changes your rules

The line can go anywhere, with one exception that none of the guides we read mention. Don't put it between two User-agent lines. Google ignores the Sitemap line when it works out groups, so this:

User-agent: GPTBot
Sitemap: https://www.example.com/sitemap.xml
User-agent: ClaudeBot
Disallow: /

is read as one group for both crawlers. GPTBot is blocked from the whole site, even though it looks like it has no rules. Keep Sitemap lines at the top or bottom of the file, away from your groups. You can paste a file into our robots.txt checker to see how each crawler reads it.

Multiple sitemaps and sitemap indexes

Each sitemap can hold up to 50,000 URLs and 50 MB uncompressed. Larger sites split their URLs across several files. You can list each one:

Sitemap: https://www.example.com/sitemap-pages.xml
Sitemap: https://www.example.com/sitemap-products.xml
Sitemap: https://www.example.com/sitemap-blog.xml

Or list a single sitemap index, a file that points to your other sitemaps:

Sitemap: https://www.example.com/sitemap_index.xml

Both work. Listing just the index is enough, and it means you don't have to edit robots.txt when you add a sitemap. An index can list up to 50,000 sitemaps, all on the same site as the index.

Sitemaps on another domain, subdomain or CDN

The sitemap doesn't have to be on the same host as robots.txt. You can point to one on a CDN or a storage bucket:

Sitemap: https://cdn.example.com/sitemaps/www-sitemap.xml

Listing it in your robots.txt is what proves to search engines that you own both, so this is often the easiest way to use a sitemap hosted elsewhere. The sitemap may only list URLs from the site whose robots.txt points to it.

Every host needs its own robots.txt. Rules at www.example.com/robots.txt don't apply to shop.example.com or blog.example.com, so each subdomain should list its own sitemap.

How to add it on your platform

WordPress

WordPress 5.5 and later adds its built-in sitemap, /wp-sitemap.xml, to the robots.txt it generates. SEO plugins such as Yoast SEO and Rank Math replace that sitemap with their own, usually /sitemap_index.xml. Check that your live robots.txt lists the plugin's sitemap. If it doesn't, edit robots.txt from the plugin: Yoast SEO, then Tools, then File editor, or Rank Math, then General Settings, then Edit robots.txt. A physical robots.txt file uploaded to your server overrides the generated one, so if your edits don't show up, look for one.

Shopify

Shopify generates robots.txt for you, and it already includes Sitemap: https://your-store.com/sitemap.xml. You don't need to do anything. To change the file, add a robots.txt.liquid template to your theme.

Wix and Squarespace

Both generate robots.txt and a sitemap automatically and link them for you. Wix lets you edit robots.txt in its SEO settings. Squarespace doesn't let you edit it.

Next.js

With the App Router, add app/robots.ts and set sitemap. This is the file ribbia.com uses:

import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: { userAgent: "*", allow: "/" },
    sitemap: "https://ribbia.com/sitemap.xml",
  };
}

sitemap also accepts an array if you have several.

Any other site

Edit the robots.txt file in your site's root folder, over FTP, your host's file manager or your code repository, and add the line. If you don't have a robots.txt yet, our robots.txt generator writes one with your sitemap included.

Robots.txt, Search Console, and the end of sitemap pings

There are now two ways to tell Google about a sitemap: the robots.txt line and Search Console. Google retired its sitemap ping URL in 2023, and pings now return an error. Bing removed anonymous sitemap submission in 2022. Old plugins and scripts that still ping are doing nothing.

Use both remaining ways. The robots.txt line works for every crawler that supports it, including Bing and many smaller engines, with no account. Search Console and Bing Webmaster Tools are where you see whether the sitemap was read, how many URLs were found and what went wrong. Bing also supports IndexNow for telling it about changes straight away.

Don't list URLs you block

Your sitemap should only list pages you want crawled and indexed. If it lists URLs that robots.txt disallows, you're sending opposite signals, and Search Console reports those URLs as blocked by robots.txt. Blocked URLs can still end up indexed without a description, which is rarely what you want. Remove them from the sitemap, or unblock them if they belong in search.

How to check it worked

  1. Run your site through the checker at the top of this page. It reads your live robots.txt, checks every Sitemap line, and confirms each sitemap loads.
  2. In Search Console, open the robots.txt report (under Settings) to see the version Google last fetched.
  3. In the Sitemaps report, submit the same URL and check its status says Success.

FAQ

Does the Sitemap line have to go at the bottom of robots.txt?+

No. Google and the sitemaps protocol both say it can go anywhere in the file. The bottom is a convention that keeps it readable. Just don't put it between two User-agent lines, because Google then treats both crawlers as one group.

Can I list more than one sitemap?+

Yes. Add one Sitemap line per file. Google accepts any number of them. If you have many sitemaps, you can list a single sitemap index instead.

Do I still need to submit my sitemap in Search Console?+

It's worth doing. The robots.txt line lets any crawler find your sitemap, but only Search Console shows whether Google could read it, how many URLs it found and any errors. Bing Webmaster Tools does the same for Bing.

How quickly will Google find a new Sitemap line?+

The next time it fetches your robots.txt. Google generally caches the file for up to 24 hours.

Does Bing read the Sitemap line?+

Yes. Bing lists the robots.txt Sitemap line as one of its supported ways to find a sitemap, alongside Bing Webmaster Tools and IndexNow.

Do AI crawlers use the Sitemap line?+

Some do. Sitemap discovery through robots.txt is part of the open sitemaps protocol, so any crawler can use it. Most AI companies don't document how they discover pages, so treat it as a bonus rather than a guarantee. It costs nothing to include.

How do I find a site's robots.txt?+

Add /robots.txt to the root of the domain, for example example.com/robots.txt. It's always at that address, and each subdomain has its own.

Is robots.txt still used?+

Yes. It became an internet standard (RFC 9309) in 2022, and it matters more now than it did a few years ago, because it's how sites decide which AI crawlers can read their pages.

Sources

Rather not do this yourself?

Ribbia fixes crawling, indexing and AI visibility for growing businesses, and shows you the results in your own Search Console.

Get a free audit