Online Marketing Blog

Dave Davis Dave attended DIT (Computer science degree) where he learned almost all the facts of life. Dave loves to travel and is a fully licensed skydiver. He loves to combine skydiving and travel. He also thinks that he can dance.

What Are Search Engine (XML) Sitemaps & Why Use Them?

By: Dave on Twitter You should follow Dave on Twitter
Date April 11th, 2007 Category Posted in: Search Engine Optimisation Comments 19 Comments

Vanessa Fox announced today on the Google webmaster central blog some pretty exciting news about sitemaps and the sitemaps protocol. First of all, what exactly are XML sitemaps? As apposed to a “Normal” sitemap you see on many sites that are simply a “page of links” to every page on your website, an XML sitemap is a document containing links to pages in your site that ALL MAJOR Search Engines can read and process.

The advantage of Search Engine Sitemaps (or XML sitemaps) over a normal “page of links” sitemap is that you can:

1) Specify the priority of pages to be crawled and/or indexed.

2) Exclude lower priority pages.

3) Insure that Search Engines know about every page on your website.

Until today, only Google, Yahoo and MSN supported this protocol. Today however, there is a new member to the family. Ask.com. On this blog, I mainly cover a lot of Google Search Engine Optimisation and Search Marketing but you may not be aware that my favorite Search Engine is actually Ask. Although it has only minimal market share, over the past year Ask has consistently converted better than all other Search Engines combined.

That’s not the only good news. Until today, it had been quite a daunting task to create a Search Engine Sitemap and submit it correctly to the major Search Engines. In many cases, a webmaster would have to carefully craft a Sitemap and submit it to each Search Engine manually through a special webmaster interface that can be somewhat complicated to the average site owner. No longer! Vanessa announced that all Search Engines have now agreed to accept Sitemap submissions through the robots.txt file on your server.

The robots.txt file is a Search Engine industry standard file that is the VERY FIRST file a LEGIT Search Engine will view when it first comes to your website. Now, you can simply add your sitemap URL to this file in the form of:

Sitemap: http://www.mysite.com/sitemap.xml

Simply create a sitemap with the free Search Engine Sitemap Generator and upload it to your server. Then open the robots.txt file on your server and add the address as above. This makes it as simple as ever to insure that all Search Engines know about your site, know about what pages are in your site and know what pages of your site to list in the search results.

Finally, it should be noted that a Search Engine Sitemap is NO SUBSTITUTION for proper Search Engine Optimisation and link building and is no guarantee of being listed in the Search Engines. Vanessa also makes a very good point that if possible, you should still submit your sitemaps to the dedicated webmaster sections with each search engine as they will be able to provide additional statistics and alert you to any errors with your Sitemap.

Related Articles:

Comments
  1. Elaine Burke Says:
    April 11th, 2007 at 7:32 pm

    Thanks for the update Dave.The only problem now is recreating my sitemap everytime I update my site with a new page or area. What do you suggest for this?

  2. Dave Davis Says:
    April 11th, 2007 at 7:53 pm

    Hi Elaine, I reccomend the softplus tool linked above. Once you set it up it is relatively simple to run it again and ping the SEs. Alternatively, for your wordpress blog, you may want to take a look at this great plugin:

    http://www.arnebrachhold.de/2005/06/05/google-sitemaps-generator-v2-final

    Let me know if you need a hand setting it up.

  3. Elaine Burke Says:
    April 13th, 2007 at 6:56 am

    Thanks Dave, I have installed that pugin and it works great! It even pings google sitemaps when updated But I guess it doesn’t matter now)

    avatar
    Dave Davis Reply:April 18, 2007 at 1:11 pm

    Everything looks set up as it should be. You’re welcome. This is in my opinion a wonderful step forward for making things that little bit easier for webmasters.

  4. Dave Davis Says:
    April 18th, 2007 at 2:04 pm

    I use http://www.xml-sitemaps.com/ and it works exactly the same as GSITECRAWLER but makes a yahoo urllist too. The only difference is that it does not upload the sitemap to the server.

  5. SEO Facility Says:
    April 20th, 2007 at 8:42 am

    Informative and useful post again Dave, thanks for keeping us updated. The benefits are that you don’t have to submit site map info to every single search engines unless you want to keep an eye on the statistics. Adding the site map tag on the robots.txt is really simple and already done. Guess sitemap.xml will be a standard similar to the robots.txt in near future.

    Tony

  6. Dave Davis Says:
    April 24th, 2007 at 1:07 am

    Thanks Tony. Yea, it’s great to see support from all the major search engines. Ask as I mentioned is one of my little gems and has always been a trouble maker for working out it’s basic organic ranking factors.

  7. Safras Says:
    April 25th, 2007 at 5:57 am

    Hello,
    I am from maldives and our website is atollparadise.com Last few days i am traying to take my search result in google in a diferent way but I dont know how to get it. for example if you search for “Sony” in google the result will come to there home page, about us page, contact us page, like there are 5 links will come together.
    I also want to do somthing like that, i mean if you search “Atoll Paradise” (We are comming first) but can any body tell me how to get listed like above i said for “sony”.
    Please help me.
    Thank You.
    Safras

  8. Dave Davis Says:
    April 25th, 2007 at 6:18 am

    Hi Safras,
    What you want is called “Sitelinks”. Unfortunately, you cannot request or add them.

    According to the Official Google Webmaster Central Blog here: http://googlewebmastercentral.blogspot.com/2006/09/information-about-sitelinks.html

    “Our process for generating Sitelinks is completely automated. We show them when we think they’ll be most useful to searchers, saving them time from hunting through web pages to find the information they are looking for”.

    In my experience, sitemaps awarded to high quality sites with a lot of deep links to the “Sitelinked” pages.

  9. SEO Facility Says:
    April 28th, 2007 at 10:12 pm

    Hello again – (ref your post april 24) Don’t now if it is a coincidence or if the new sitemap standard is put to work There has been a lot of spider activity since adding the sitemap tag to reobots.txt. I had to re-validate some of my sites as well and sadly admit that for a couple of sites the PR dropped.

    Concerning ASK I have never really paid much attention to it. Mainly because Ask kind of always been around but very seldom generate any traffic. So I don’t know much about ASK or its ranking criteria. Could you give me some hints, got a bit interested and might have a closer look since you mentioned it…

    Tony

  10. Dave Davis Says:
    May 2nd, 2007 at 5:38 pm

    I’d suspect the robots.txt standard addition does not only apply to the big three and that other engines have also adapted the protocol.

    Regarding the ASK ranking criteria, I will be publishing a post in the coming weeks about that. Stay tuned.

  11. Percy Says:
    June 26th, 2007 at 7:26 pm

    “Although it has only minimal market share, over the past year Ask has consistently converted better than all other Search Engines combined.”

    Converted? Customers for your services personally, or aggreaged for customers whose campaigns you manage?

  12. Dave Davis Says:
    June 28th, 2007 at 4:39 pm

    Percy, both. I mentioned before that ASK is a search engine that not so many “technical” people use, hence it is wonderful for attracting traffic and consumers for non technical offers like gardening and really niche topics.

  13. mlankton Says:
    September 8th, 2007 at 4:26 pm

    I use the wp-plugin as well. I don’t know that a sitemap helps you out any with google, but it may help you with the other search engines.

  14. Ecommerce Website Design Says:
    July 16th, 2008 at 9:37 pm

    I do think it helps you with Google – it gives them a quick glimpse into your site that they can’t get anywhere but an XML sitemap. Good stuff here!

  15. chetan Says:
    August 6th, 2008 at 1:34 pm

    As a wordpress user I use Google XML Sitemap plugin.
    http://www.arnebrachhold.de/redir/sitemap-home/

    Its awesome.

  16. xHTML Coding Says:
    September 15th, 2008 at 2:23 pm

    I’ve come to believe that sitemap’s are old school and Google knows “exactly” where to find new pages on your site.

    Just like pinging had a honorable death, it may be the same with sitemaps with Google.

    But, sitemaps are still great for Yahoo and MSN.

  17. Jonny Says:
    January 16th, 2010 at 4:20 am

    Hi Dave, are there any dangers using sitemaps. I’ve read on one or two forums that sitemaps can damage websites ranking. For instance I believe sitemaps have priority settings, if I set all pages to the same priority what effect will this have.

  18. Mario Moraca Says:
    July 7th, 2011 at 10:30 pm

    Thanks Dave! Just want to share my experience with sitemap creators. I run a database driven website and tried many sitemap applications, even the one you recommend (xml-sitemaps.com), but I found GSITECRAWLER the best application available today. Most of the sitemap creators didn’t indexed all my dynamic pages, but with GsiteCrawler I didn’t have any problems at all.