Understanding XML Sitemaps: Recommendations and Examples

An XML sitemap is a document that informs search engines about which pages on a website are available for indexing. This file, formatted in XML, contains a list of URLs from the site and associated metadata, such as the last modification date, frequency of updates, and each page's priority within the site. This information helps search engines optimize the indexing process.

While the primary function of the robots.txt file is to restrict certain pages from being indexed, the sitemap.xml file serves the opposite purpose. It focuses on ensuring faster and more comprehensive indexing of the website.

Essentially, a sitemap acts as a roadmap for search engine crawlers, guiding them through the different sections of your website. Sitemaps are crucial for ensuring that all important pages are discovered and indexed, particularly for larger websites with complex structures or newly launched websites.

Sitemaps are vital for SEO as they help search engines find and crawl your website more effectively. This is particularly important if your site has pages that are not easily discoverable through regular crawling, such as pages buried deep within your site's structure or without many internal links. A well-structured sitemap ensures that all relevant content is indexed, increasing the likelihood of your pages appearing in search engine results.

There are two primary types of sitemaps: XML and HTML.

• XML Sitemap: An XML sitemap is specifically designed for search engines. It lists a website's URLs and provides additional information, like when a page was last updated, how often it changes, and its importance relative to other pages on the site. XML sitemaps are crucial for SEO and help search engines index your site more effectively.

• HTML Sitemap: An HTML sitemap is created for users rather than search engines. It typically appears as a page on your website that lists the links to your site's pages, organized in a hierarchical format. HTML sitemaps enhance user experience by making it easier for visitors to navigate your site.

An XML sitemap is structured as a simple text file written in XML format. There are different types of sitemaps, each serving a specific purpose. Below are examples of various sitemap structures that cater to different needs.

Basic XML sitemap example:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
 <url>
<loc>https://www.example.com/</loc>
   <lastmod>2024-01-01</lastmod>
<changefreq>monthly</changefreq>
   <priority>1.0</priority>
 </url>
 <url>
<loc>https://www.example.com/about</loc>
   <lastmod>2024-01-01</lastmod>
   <changefreq>yearly</changefreq>
   <priority>0.8</priority>
 </url>
</urlset>

This example highlights a basic structure where each URL is listed with relevant metadata.

HTML Sitemap

An HTML sitemap is user-facing and designed to help visitors easily navigate your site. It usually appears as a web page listing your site's main sections and pages in a hierarchical format. Here's how it might look:

<ul>
 <li><a href="https://www.example.com/">Home</a></li>
 <li><a href="https://www.example.com/about">About Us</a></li>
 <li><a href="https://www.example.com/services">Services</a>
   <ul>
     <li><a href="https://www.example.com/services/web-development">Web Development</a></li>
     <li><a href="https://www.example.com/services/seo">SEO</a></li>
   </ul>
 </li>
 <li><a href="https://www.example.com/contact">Contact</a></li>
</ul>

This example shows how HTML sitemaps facilitate user navigation by displaying links in a structured, readable format.

Sitemap Index File

A single sitemap might exceed the size limits for large websites with thousands of pages. In such cases, a sitemap index file can group multiple sitemaps. Here's an example:

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
 <sitemap>
<loc>https://www.example.com/sitemap-products.xml</loc>
   <lastmod>2024-01-01</lastmod>
 </sitemap>
 <sitemap>
<loc>https://www.example.com/sitemap-blog.xml</loc>
   <lastmod>2024-01-01</lastmod>
 </sitemap>
</sitemapindex>

This structure allows you to organize your sitemaps efficiently, ensuring that each is focused on a specific section of your site.

Video Sitemap

If your website hosts video content, a video sitemap can help search engines understand the context and details of your videos. Here’s a basic example:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:video="http://www.google.com/schemas/sitemap-video/1.1">
 <url>
<loc>https://www.example.com/video/cat-video</loc>
   <video:video>
     <video:thumbnail_loc>https://www.example.com/thumbnails/cat.jpg</video:thumbnail_loc>
     <video:title>Funny Cat Video</video:title>
     <video:description>A funny video of a cat playing.</video:description>
<video:content_loc>https://www.example.com/videos/cat.mp4</video:content_loc>
<video:duration>120</video:duration>
   </video:video>
 </url>
</urlset>

This sitemap example shows how to provide search engines with detailed information about each video, helping them index and display your video content more effectively.

Image Sitemap

An image sitemap is used to give search engines additional information about the images on your site. Here’s an example:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1">
 <url>
<loc>https://www.example.com/gallery/landscape</loc>
   <image:image>
<image:loc>https://www.example.com/images/landscape1.jpg</image:loc>
     <image:caption>Beautiful Landscape</image:caption>
   </image:image>
   <image:image>
<image:loc>https://www.example.com/images/landscape2.jpg</image:loc>
     <image:caption>Sunset over Mountains</image:caption>
   </image:image>
 </url>
</urlset>

This sitemap example helps search engines understand what images are on your site and how they relate to your content.

To find the sitemap of a website, you can typically look for it at the following URLs:

• https://www.example.com/sitemap.xml
• https://www.example.com/sitemap_index.xml

If these URLs do not work, you can check the robots.txt file located at https://www.example.com/robots.txt, which often contains a link to the sitemap.

Creating a sitemap can be done using various tools and methods:

1. Manually: You can manually create a sitemap using XML by listing all your site’s URLs in the correct format. However, this method is time-consuming and prone to errors, especially for larger websites.
2. Using Tools: There are numerous tools available that can automatically generate a sitemap for you. Popular options include:

• WordPress Plugins: WordPress offers numerous plugins specifically designed to create XML sitemaps. For example, Yoast SEO automatically generates an XML sitemap for your site. Another plugin that creates a sitemap for WordPress sites is Google XML Sitemaps.
• Screaming Frog: A powerful tool can crawl your website and generate a comprehensive XML sitemap.
• SEMrush: Provides a sitemap generation tool as part of its suite of SEO tools.

The sitemap should only include links to essential site pages containing unique content.

The sitemap.xml file can also suggest how often pages should be reindexed to search engines. This feature is precious for sites with frequently updated content, such as news portals.

It's crucial to ensure that all links in the sitemap return a 200 server response code, making them accessible to both robots and users. If a link in the sitemap returns a 404 (Not Found) error, search engines like Google or Bing will repeatedly crawl the page and add it to their error reports.

Additionally, it's best to avoid including pages with redirects (e.g., 301 redirects) in the sitemap. Instead, you should list the final destination page to which the redirect points. However, in cases where a mass update of page URLs occurs, webmasters may temporarily keep "old" links in the sitemap. This allows search engines to crawl the old pages, discover the new ones, and index them more quickly. Once this process is complete, you should replace the old 301 links in the sitemap with the new, direct URLs that return a 200 response code.

• Use Consistent URLs: Ensure all URLs in your sitemap are complete and consistent. Google will crawl the exact URLs you provide. For instance, if your site is https://www.example.com/, do not include variations like https://example.com/ (without www) or relative URLs like ./mypage.html.
• Place Sitemap at the Root Level: While you can place a sitemap anywhere on your site, it will only affect directories below its parent location. To ensure your sitemap covers the entire site, it is best to place it at the root level.
• Avoid Session IDs in URLs: Including session IDs in your sitemap can lead to unnecessary crawling of duplicate pages.
• Use hreflang for Language Versions: If you have different language versions of a page, mark them with the hreflang attribute to help search engines understand the content’s regional targeting.
• Ensure Proper Encoding: Sitemaps should be encoded in UTF-8, and all URLs should be properly escaped to avoid errors during crawling.
• Divide Large Sitemaps: If your sitemap exceeds 50,000 URLs or 50 MB in size (uncompressed), break it into smaller files and submit a sitemap index to Google instead of individual files.
• Include Only Canonical URLs: If multiple page versions exist, only include the canonical version in your sitemap. For sites with and without "www," choose one as the primary version, place the sitemap there, and use rel=canonical or redirects on the other.
• Mobile and Desktop URLs: If a page has different URLs for mobile and desktop versions, ideally include just one. If both must be included, annotate them to indicate the specific versions.
• Use Extended Sitemap Syntax: Use extended syntax in your sitemap to specify additional content types like images, videos, or news.
• Indicate Language/Region Variants: Use the hreflang attribute within the sitemap file or HTML tags for pages with different language or regional versions.
• Crawling Expectations: Google doesn't guarantee every URL will crawl in your sitemap, but sitemaps help indicate which pages you consider most important.
• <priority> and <changefreq> Tags: Google generally ignores the values in the <priority> and <changefreq> tags, so focus on accurate data elsewhere. You can use and, in this case, do it wisely: The <priority> tag in an XML sitemap allows you to indicate the importance of certain pages relative to others. Use this tag carefully, ensuring that your most important pages are given higher priority.
• <lastmod> Tag Accuracy: Google may use the <lastmod> tag value if it’s consistently accurate, for example, by verifying it against the page’s actual last modified date.
• Compress Your Sitemap: Use gzip to compress your sitemap file, reducing the load time for search engine crawlers.
• Link to Your Sitemap in robots.txt: Including a link to your sitemap in the robots.txt file helps search engines find it easily.
• Keep It Clean: Remove any outdated or broken links from your sitemap to maintain its effectiveness.
• URL Order in Sitemap: The order of URLs in a sitemap does not matter. Google will crawl the URLs regardless of their position in the file.
• Update Regularly: Keep your sitemap updated to reflect any changes to your website. Regular updates ensure that new pages are quickly indexed and search engines recognize any changes to existing pages.

By creating a sitemap, you guide search engines on which URLs to prioritize for display in search results. These selected URLs are known as canonical URLs. If your content is available on multiple URLs, identify the primary version of the page and include only that version in your sitemap.

Avoid including the following in your sitemap:

• Duplicate Content: Only include original pages that you want to be indexed.
• Low-Value Pages: Exclude pages like “Thank You”, Check Out, Payment pages, or other content that doesn't contribute to your SEO goals.
• Non-Indexable Pages: Don't include pages blocked by robots.txt or have a noindex tag.

Ways to Structure Sitemaps:

•         Simple Sitemap: A simple XML file listing all the URLs is sufficient for small websites with straightforward navigation.
•         Complex Sitemap: Larger websites may need multiple sitemaps, each covering different sections of the site. For example, one sitemap for blog posts, another for product pages, etc.

There are several methods to provide Google with access to your Sitemap:

1. Submit via Google Search Console: Use the Sitemap submission feature within the Google Search Console to submit your Sitemap file. To do this:

• Go to Google Search Console: Log in to your account and select the property for the website you want to manage.
• Navigate to Sitemaps: On the left-hand menu, click "Sitemaps."
• Submit Your Sitemap URL: Enter the URL of your sitemap (e.g., https://www.example.com/sitemap.xml) and click "Submit."

2. Ping Google: In your browser or command line, send a GET request
to the address below, specifying the full URL of the Sitemap file and verifying that the file is accessible: https://www.google.com/ping?sitemap=FULL_URL_OF_SITEMAP

 Example: https://www.google.com/ping?sitemap=https://example.com/sitemap.xml

3. Add to robots.txt: Insert a line specifying the path to your Sitemap anywhere within your robots.txt file. Google will detect this the next time it crawls the robots.txt file.

Example: Sitemap: https://example.com/my_sitemap.xml

The instructions in sitemap.xml and robots.txt should work together effectively. Here are three critical rules for their interaction:

• Consistency: sitemap.xml and robots.txt should not contain conflicting directives.
• Exclusion: Any pages blocked in robots.txt should also be excluded from sitemap.xml.
• Inclusion: All crucial pages permitted for indexing in robots.txt should be included in sitemap.xml.

If your Sitemap file exceeds 50MB, you must split it into smaller files. Multiple Sitemaps can be submitted together using a Sitemap index file. This XML file organizes and links to the individual Sitemap files in a format similar to a standard Sitemap.

Important: Google does not support multiple nested XML index maps.

You can find detailed instructions on building and submitting a sitemap in Google's documentation.

You should update your sitemap whenever you add new content, delete pages, or make significant changes to existing pages. Regular updates ensure that search engines have the most current view of your site. It's advisable to update the sitemap whenever changes are made, such as adding new pages or removing outdated ones. Most plugins for popular CMS platforms, like WordPress, are designed to automatically refresh the sitemap regularly to reflect these changes. If your site is undergoing frequent updates, it's recommended that you update the sitemap daily.

Sitemaps are crucial for ensuring that search engines crawl and index your website effectively. However, several common errors can undermine their effectiveness. Understanding these issues is key to maintaining a healthy and efficient sitemap.

1. Incorrect URL Formatting

One of the most frequent errors is incorrect URL formatting. All URLs in the sitemap should be absolute, starting with "http://" or "https://". Relative URLs can cause issues with indexing, leading to pages being missed by search engines.

2. Including Non-Canonical URLs

Sitemaps should only include canonical URLs—the versions of the pages you want search engines to index. Including duplicate or non-canonical URLs can confuse search engines and dilute your site's SEO effectiveness.

3. Exceeding Size Limits

A sitemap can contain up to 50,000 URLs and should not exceed 50MB when uncompressed. Exceeding these limits requires splitting the sitemap into multiple files and creating a sitemap index, which many overlook, leading to incomplete indexing.

4. Outdated Sitemaps and Page Inconsistencies

An inconsistency between the sitemap.xml and the actual site pages, often due to an outdated sitemap, is a common issue. This problem arises when the sitemap isn’t dynamically generated but is instead updated sporadically, perhaps through a CMS service or even a third-party tool. As a result, many dead pages may remain in the sitemap, leading to 404 errors when these pages have been deleted or moved with URL changes. Additionally, new pages are indexed much more slowly because they are not promptly included in the sitemap.xml.

5. Incorrect structure in the sitemap.xml

Another common error is an incorrect structure in the sitemap.xml file. This issue often arises in custom-built CMS platforms or when using poorly designed plugins with popular CMSs.

6. Inaccurate lastmod Dates

A related issue to point 5 involves improper handling of the <lastmod> field, which indicates the last modification date of a page. It's crucial to either accurately update this field to reflect the last significant change—such as when a price is updated, a product goes out of stock, or a description changes—or not use it at all (though this is less ideal). Setting the same modification date for all pages and inaccurate or outdated lastmod dates can mislead search engines about the freshness of your content, potentially affecting how often your site is crawled.This can lead to penalties, such as a reduced crawling quota, which may delay the indexing of new pages by weeks.

7. Inconsistent Directives Between Sitemap and robots.txt

The sitemap and robots.txt files must complement each other. A common mistake is including pages in the sitemap blocked by robots.txt. This inconsistency can prevent pages from being indexed, reducing their visibility in search results.

8. Broken Links in the Sitemap

If your sitemap contains broken links, it can negatively impact your SEO. Search engines encountering these links may flag your site as unreliable, leading to decreased crawl efficiency.

9. Failing to Update the Sitemap

As your site evolves, so should your sitemap. A common error is failing to update the sitemap with new or removed pages, leading to outdated information being provided to search engines.

10. Not Submitting the Sitemap to Search Engines

After creating or updating your sitemap, it's essential to submit it to search engines like Google through platforms like Google Search Console. Failing to do so can result in search engines not wholly indexing your site.

By being aware of these common sitemap errors, you can take proactive steps to avoid them, ensuring that your website remains fully accessible and effectively indexed by search engines.

Errors can occur when creating or submitting a sitemap. Common issues include incorrect URLs, unsupported formats, and oversized files. Use tools like Google Search Console to monitor your sitemap and receive notifications about any problems. Screaming Frog and other SEO tools can also scan your sitemap for errors, such as broken links or incorrect URLs.

Google Search Console is a valuable tool for monitoring and troubleshooting sitemap issues that may arise during indexing. Here's a concise guide on how to address common problems:

1. Identifying Errors

When you submit a sitemap in Google Search Console, you must check for any reported errors or warnings. These issues are typically listed under the "Sitemaps" section. Common errors include incorrect URL formats, unsupported file types, or large file sizes.

2. Fixing URL Issues

One of the most common problems is incorrect or inaccessible URLs in the sitemap. Ensure all URLs are correctly formatted and lead to accessible pages. URLs should be absolute, beginning with "http://" or "https://," and not include relative paths.

3. Handling Unsupported File Types

If your sitemap contains unsupported file types, Google will flag this in Search Console. Remove or update URLs pointing to file types incompatible with Google's crawling process.

4. Managing Large Sitemaps

Google Search Console limits individual sitemap files to 50,000 URLs or 50MB (uncompressed). If your sitemap exceeds these limits, you'll need to split it into smaller sitemaps and manage them using a sitemap index file.

5. Resolving Submission Errors

Sometimes, the submission process encounters issues like server errors or incorrect sitemap paths. Double-check the path of your sitemap file and ensure it is correctly linked in your robots.txt file. After making corrections, resubmit the sitemap.

6. Monitoring for Crawling and Indexing

Even after submitting a corrected sitemap, monitor Google Search Console for any ongoing issues. Check the "Indexing" report to see how many pages have been indexed and whether any are excluded. This can help you identify further problems, such as blocked URLs or pages marked as "noindex."

• Verify Sitemap Availability: It's crucial to ensure your sitemap is accessible and returns a 200 server response code. With Atomseo, you can set up automatic checks for sitemap availability at regular intervals (daily, weekly, etc.).

Add the Sitemap’s address to the main page. Press ‘Check Site’. Atomseo will extract all the links from the sitemap and check their statuses. You will get the report soon.

You will receive an email notification if any issues with the sitemap are detected.

A well-structured sitemap is essential for ensuring that search engines can effectively crawl and index your website. You can significantly improve your site's SEO performance by understanding the different types of sitemaps, how to create them, and best practices for maintaining them. Regular updates and careful monitoring will help ensure that your sitemap continues to support your website's visibility and ranking in search results.

You can easily verify your sitemap's availability using Atomseo Broken Link Checker by following the steps outlined above. Additionally, this tool also lets you check up to 1,500 links per day at no cost, helping you keep your site free of errors and broken links.