Learn the different jobs of robots.txt and XML sitemaps and how they work together.

Robots.txt controls crawler access

Robots.txt is mainly a crawl-management file. It can tell compliant crawlers which paths they should or should not request.

Sitemaps list important URLs

An XML sitemap helps search engines discover canonical URLs and understand when content was updated.

They are not replacements for each other

A sitemap does not override robots.txt, and robots.txt is not a reliable method for removing already indexed URLs from search results.