About Sitemaps and their Submission to Search Engines

January 30th, 2008 by Thibaut.

Sitemaps are a (recently) standardized way of telling search engines which pages they should crawl.

How useful are they ? My own experience is that for content oriented websites and blogs, sitemaps are an efficient way to get known by search engines.

For each post I write here, I tend to welcome readers through Google search about 10 days after the posting date. As a consequence I believe sitemaps are an interesting asset to give your future readers a better user experience while trying to reach you.

If Google’s marketshare is enough for you and if you have an RSS feed, you’ll want to check out Google BlogSearch (see the end of this article) for even faster indexing (under an hour).

I want to learn more about sitemaps, what should I do ?

To get detailed information on sitemaps, you can refer to the Sitemaps website or alternatively have a look at the Google Webmaster Tools Sitemap documentation and its FAQ.

Here’s how a sitemap looks like so you get the idea (basically an XML file with URLs in it):


<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>http://blog.logeek.fr/2008/1/19/a-beginner-s-guide-to-datawarehouse</loc>
    <lastmod>2008-01-21T03:23:05+00:00</lastmod>
    <changefreq>daily</changefreq>
    <priority>0.5</priority>
  </url>
  .... more urls ....
</urlset>

If you manage or create a website, you can generate the sitemap yourself or use an existing tool like Google’s sitemap generator to do the job.

If you use a blogging engine, most of them provide a way to generate sitemaps for you either natively (like Subtext) or through a plugin of some sort (like WordPress or Mephisto).

For LoGeek’s blog and The Evolving Worker, I’m using Mephisto: I have choosen to use Mika Tuupola’s Mephisto Sitemap plugin which took me a couple of minutes to install and works great (thanks Mika!).

How to use the sitemap once you have one ?

Like explained here there are a couple of ways to tell search engines that you are using a sitemap.

One of the options is to declare it in your robots.txt file.

Another option is to push the sitemap location to the search engines: if you have a new website I believe this option will help you spread the word.

Where to submit the sitemap location ? The nice thing is that unlike a couple of months or years ago, now most of the big search engines are committed to use Sitemaps. Here’s where you can submit your sitemap today (learned from various sources on the web):

What if I only want to focus on Google on an RSS enabled website ?

A while after I wrote this article PocketSEO advised me to have a look at Google blogsearch to get indexed very quickly (less than one hour typically).

Expect the short reaction time for articles posted after your first ping. Your older articles in the RSS feeds usually get indexed a few days later.

One last tip: have a look at how your pages look like once indexed using the site:mysite.com google query. What you get should be easy to read and understand to help people looking for content find what they need.

The comments system is brand new - don't be afraid to comment!

All the posts are listed below and on the homepage.