XML Sitemaps in SEO – Part 1
Many people consider XML sitemaps as an optional extra. But the fact is, XML sitemaps help to get your pages indexed by providing a full list of URLs on your website.
They also provide other useful information to search engines about your site. For example, by submitting “/” as your root domain instead of “default.aspx” through the XML sitemap, search engines like Google might just figure out “/” is the main URL. This may help with the canonical issues.
Other benefits of XML sitemaps include:
- Help improve the crawl rate and indexation on a site
- Help the crawling of newer content
- Help spot problems or raise trouble flags (In the case of Google, problems are reported in Webmaster Tools)
In XML Sitemaps, we generally include all the URLs of our site to help search engines discover most of our URLs. We can also create XML Sitemaps for other content like videos and mobile web pages to make them easily discoverable by search engines. In this post, I will cover most of the XML Sitemaps and will talk about videos and mobile sitemaps in a future post.
Location of XML Sitemap
By default, the location of XML Sitemap can be assumed as your-site/sitemap.xml. If we have multiple sitemaps or our CMS generates files with some unique names, then all we have to do is mention it by adding the following line in the robots.txt file:
Always remember, Google is not the only search engine out there. For Google and Bing, XML sitemaps can be easily submitted via Google Search Console and Bing Webmaster Tools. For all other search engines, the only way they can figure out the location of your XML sitemap is through your robots.txt file. (For more details, see our complete robots exclusion guide.)
When is XML Sitemap helpful?
XML sitemaps are only there to help search engines discover URLs for your website.
It is not guaranteed that all the URLs submitted using an XML sitemap will be crawled and indexed. For example, if we have a 500-page site and search engine spiders can only crawl the first 200 of the pages, then it is not a good idea to force the other 300 pages via a sitemap. The solution would rather be fixing the crawlability problems like making sure every page has an appropriate number of links and avoiding deep links. Also, if there is an indexation problem, we should focus on the other factors like the quality of the content, link building strategies, and social interactions.
Creating an XML Sitemap
It is highly recommended to use Sitemap 0.9 protocol as defined by sitemaps.org. All the major search engines accept this standard protocol. There are also many free tools available that can be used to create an XML Sitemap. Xenu is one tool that crawls websites and also can be used to create an XML Sitemap.
Optimizing an XML Sitemap
Sometimes, we might end up having thousands of URLs in a sitemap file. Google recommends having no more than 50,000 URLs and keeping the file size less than 50MB compressed.
If our sitemap exceeds this limit, then it is important to split the large file into multiple, smaller files. You then create a sitemap index file that lists all the sitemap files on your site. You only need to submit the sitemap index file to the search engines. To optimize the sitemap files, we can also compress the file using gzip.
It is important to consider using an XML sitemap especially for huge sites to make sure all the URLs are discoverable by search engines. There are results we have seen where a small change nearly tripled the indexation percentage.
Stay tuned and keep reading our blogs for more SEO information.