How Search engine bots work

An amazingly insightful article analyzing the way bots from the 3 major search engines – Google, Yahoo and MSN – work [via Google Blogoscoped]. The researchers created 2 billion webpages for this test and monitored the traffic of the 3 major search bots – Googlebot, Yahoo Slurp and msnbot – over each of these pages for one year.

Some of the findings confirm what sharp webmasters might have found out themselves: Yahoo Slurp is by far the most active bot, indexing over 100,000 pages as compared to 554 for Googlebot and just one for the msnbot. This confirms what I had found by my informal checks for my blog on Yahoo (23 results) and Google (1 result). Yahoo now leads Google in terms of volume and speed of indexing. This still gives rise to these 2 questions:

  • In search, does quality matter or quantity?
  • Despite indexing fewer pages, does Google actually send more referrals because of its popularity among users?

Another interesting finding from the research again confirms what acute webmasters would have found regarding comment spam – pharma-related spam accounts by far for the greatest comment spam. Luckily for us at WordPress, Akismet does an awesome gatekeeping job to keep the spammers away without bothering genuine commenters too much.

Fantastic article – check it out!

One thought on “How Search engine bots work

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s