XML sitemaps and robots.txt explained

XML sitemap

A sitemap is an XML file listing URLs, optionally with the date of last modification. It helps search engines discover pages, especially new ones and those with few internal links.

Good practice:

• include only canonical URLs that return status 200 and are meant to be indexed;
• exclude redirected, noindex, duplicate and parameter URLs;
• generate it automatically, so new pages appear and removed pages disappear;
• use lastmod only with real modification dates, not the build date;
• split very large sitemaps and reference them from a sitemap index;
• submit it in Google Search Console and Bing Webmaster Tools, and reference it in robots.txt.

Sitemaps for multilingual sites

For each URL, the sitemap can list its language alternates with xhtml:link elements, including a self-reference and x-default. This is an alternative or supplement to hreflang tags in the page head. See multilingual URL structure.

robots.txt

robots.txt is a text file at the root of the domain with rules for crawlers:

• User-agent — which crawler the rules apply to (* for all);
• Disallow — paths not to crawl;
• Allow — exceptions;
• Sitemap — the sitemap location.

Typical blocked paths: admin areas, internal search results, carts and checkout, filter combinations that create endless URLs.

Crawling is not indexing

A page blocked in robots.txt can still appear in search results if other sites link to it — without a description, because the crawler could not read it. To keep a page out of results, allow crawling and use a noindex meta tag or header, or protect it with a password. For private areas such as admin panels, authentication is the real protection; robots.txt is public and only a request.

Do not block rendering resources

Search engines render pages like a browser. If CSS or JavaScript files are blocked, they may see a broken page. Allow crawling of resources needed for rendering.

AI crawlers

AI companies operate crawlers for search features and for training. Examples include OpenAI's OAI-SearchBot and GPTBot, Anthropic's ClaudeBot, PerplexityBot and Google-Extended (a control token for Gemini training and grounding, separate from Google Search indexing). Each respects its own user-agent rules in robots.txt. Decide per crawler: blocking search-focused crawlers can reduce your visibility in AI answers; blocking training crawlers affects whether your content is used to train models. See AI search visibility.

Common mistakes

• Disallow: / left from a staging site after launch.
• Sitemap listing redirected or noindex pages.
• Sitemap on HTTP when the site is HTTPS, or on the wrong domain.
• Blocking pages with robots.txt and expecting them to disappear from search.
• Blocking CSS and JavaScript.
• Separate rules for a specific crawler that accidentally remove the general restrictions — a crawler follows only the most specific group that matches it.