XML Sitemaps are an easy way for webmasters to tell search engines about pages on their site that may be crawled by their robots.

A typical XML Sitemap file lists each URL, together with information about when it was last updated, how often it normally changes, and how important it is, relative to other pages in the site). This helps search engines to more intelligently crawl your site.

In November 2006, Google, Yahoo! and MSN joined forces to support a new industry standard for sitemaps: Sitemaps 0.90. As long as webmasters follow the protocol, they can ensure their sites are fully and consistently indexed across all the major search engines (a real step forward). This article is important for all those with missing or poorly ranked pages.

The official site for the joint venture is at sitemaps.org and contains a lot of info about the new standard and it's syntax. What the site singularly fails to do is explain correctly how to submit your sitemap to the big three! The format suggested on the site of:

does not currently work at any of the three sites! Until it does, this short article provides instructions for how to (a) create your sitemap and (b) how to submit to each of the three main search engines...

Creating your Sitemap

Some hosting providers (for example 1and1) provide utilities via their web control panel, to create your sitemap, so you should always check with your provider first. If this service is not available, then make a visit to xml-sitemaps.com and enter your site URL into the generator box. Copy-and-paste the resulting sitemap into notepad, then save-and-upload to your site with the file name: sitemap.xml

If you want to validate the XML prior to uploading to the search engines (useful if you have made any manual amendments), look at the XML validator (at the same site) where you can put in the URL of your sitemap and check it against the standard.

Submit sitemap to MSN

MSN have yet to implement a formal interface for Sitemap submission (as at July 2007). To monitor the situation, please visit (from time to time) the MSN Official Livesearch Blog (where where future announcements are likely to be found).

Whilst MSN have yet to implement a front door, there is a recognised back door for submitting your sitemap to the MSN Search index; namely moreover.com! You should use the following syntax directly in your browser URL box:

Since February 2005, moreover.com have been the official provider of RSS feeds to the myMSN portal (see press release) and reliable evidence suggests that submission to Moreover will result in MSN spidering your pages within 2-3 weeks.

Note that, whilst MSN still do not support direct submission, they do suggest on their blog that you add a reference to your Sitemap into your robots.txt file (something now supported by sitemaps.org). For example:

This would tell MSN (and all other engines) to crawl your sitemap file but not to crawl your cgi-bin directory. For more info on how to implement a robots.txt file (in the root of your site webserver) please visit: http://www.robotstxt.org

Submit sitemap to Google

Google originally developed the XML schema for sitemaps and have developed a dedicated portal for webmasters, from where you can submit your sitemap:

First, you need to tell Google all the sites you own, then verfiy that you indeed own them. The verifiaction is achieved by adding a metatag between the head tags on your site homepage. The syntax for the tag is as follows:

There are full instructions on how to do this on the Google site.

Submit sitemap to Yahoo

Yahoo follows a similar approach to Google. Again, there is a dedicated sevice for webmasters (Yahoo! Site Explorer) and a procedure for verifying your ownsership of the site. First go to:

Add a site, then click on the verification button. You can then download a verfication key html file - which you will need to upload to the root directory of your webserver. Then you can return to Site Explorer and tell Yahoo to start authentication. This will take up to 24 hours. At the same time you can also add your sitemap by clicking on the manage button and then adding the sitemap as a feed.

Submit sitemap to Ask

Ask follows a simpler approach to the other three. To submit you sitemap, you simply enter a ping URL, followed by the full URL of where your sitemap is located:

After clicking return, you will get a reassuring message from Ask that they have received your submission. Very neat!