Search Engine Musical Chairs

By Sean Burns
Posted Saturday, July 24, 2004

News broke this week that Yahoo has purchased the Inktomi search engine for around US$235 million. This is an interesting development in the search engine industry that may impact greatly on exactly where sites get their traffic from. Another new development in the past few weeks is the change to the HotBot service but first, I'd like to look at the ramifications of the Yahoo/Inktomi deal.

Here's a bit of background information on how the Yahoo and Inktomi search engines work. I'm sure everyone knows Yahoo but not everyone may be completely aware of how their search engine works. Yahoo is a directory. This means that it is a categorised list of sites that are listed by human editors. Getting listed in the Yahoo directory requires a yearly payment for commercial sites. You can get listed for free if you have a non-commercial site but it is very hard to do and can take months. Yahoo provides additional results when you use their search function that come from Google. In the past, Yahoo has used other search engines to provide these results (including Inktomi) but has used Google for the last couple of years.

They initially used results from other search engines to just compliment their own. So, if someone searched for something that wasn't in their directory, they could still get results. The way it worked was that they would provide results from their own directory first and then a user would click on "web pages" to get more results. Strangely, Yahoo actually changed the way this worked during the year so that the results from a search were a mix of listings from their own directory and Google with the Google results being the more prominent of the two. This has essentially made the payment of the US$299 per year for a directory listing an unnecessary expense. Many people still use the directory to browse for sites but, in my experience, more people use the search function.

Inktomi is a full crawler based search engine that provides results for other search engines. They do have a web site but no-one uses it to actually search - it's more of an informational site. Inktomi's results formerly appeared in many search engines but in the last few years, their popularity has declined. The main site that currently uses Inktomi is MSN but Inktomi only provides the fifth level of results. MSN provides sponsored listings from Overture, their own Microsoft sites, their own human edited directory results, LookSmart listings and then Inktomi. So, Inktomi does drive traffic to sites but far less than a search engine like Google.

So, what does all this mean to us? It's hard to say at the moment but there are three options. Either nothing will change (which is unlikely), Yahoo will use Inktomi together with Google on it's site or it will dump Google and use Inktomi for it's search results. Yahoo actually owns part of Google, maybe 5%, so there is a chance that they will keep using their results for this reason. Also, Google provide far better results for searchers than Inktomi so Yahoo would be making a mistake to replace Google. But, apparently Yahoo is annoyed that Google has become competition to them through Google's own site and that Google's news search service is also providing competition.

Without a crystal ball, it's very difficult to know what is going to happen but it's worth making sure that you are prepared for all occurrences. There is really very little that you need to do because it is only the popularity of Inktomi that may change. Even without the searches from Yahoo, Google will still be the biggest search engine in the world due to the fact that it's own site is very popular and it's results are also used by AOL (amongst others). So, what can you do to get the most from Inktomi?

Inktomi is the only major search engine that uses meta tags. This is actually the reason that it is not as popular as before because meta tags allow webmasters to "trick" Inktomi into thinking a page has more relevant content than there really is. Therefore, Inktomi's search results are not very good. However, as there is a good chance that Inktomi will now become popular again, you need to make sure that you have your meta keyword and meta description tags in place. Make sure they are relevant to the page that they are on - you don't want to be caught "spamming" their search engine. The other thing that will probably make a difference is to get other sites to link to you. Link popularity makes a difference on all search engines except possibly AllTheWeb.

I'll keep you updated here with any news on what Yahoo decides to do and any changes that I find out about the way the Inktomi generates results.

As I mentioned above, the other major search engine news in the last month is that HotBot has finally updated their site. HotBot is owned by Lycos and was quite a popular search engine a few years ago. It has become far less popular lately - since it was purchased by Lycos and was just left to "die". It's results largely came from a mix of the ODP, DirectHit and Inktomi. DirectHit has since become Teoma which is owned by AskJeeves but HotBot went for months after DirectHit ceased to exist saying that it's results were coming from them.

So, HotBot has essentially become another meta search engine - like DogPile or Excite. It now works pretty much like Netscape does. A searcher can choose which search engine they would like their results to come from. The search engines that you can choose from are Fast (AllTheWeb), Google, Inktomi and Teoma. This may mean that HotBot starts to get some of it's market share back but as webmasters, there is nothing that we can do to target HotBot. All we can do is target the search engines which provide it's results.

About The Author
Sean Burns is the author of the WebmastersReference.com Newsletter - (http://www.webmastersreference.com/newsletter). More than five years of experience in site design, marketing, income generation, search engine optimisation and more is passed on to subscribers - hype free. Sign up today to get real information of real value to webmasters.
seanb@webmastersreference.com