Yahoo! Introduces Its Own Search Engine

By Jim Hedger
Posted Saturday, September 18, 2004

Yahoo unleashed the biggest surprise of the year today with the formal introduction of their own search engine. While the vast majority of search engine analysts were expecting Yahoo to simply adopt results from the powerful Inktomi database, Yahoo has been quietly and apparently quite effectively developing their own spider, database and ranking algorithms. Last month, Yahoo CEO Terry Semel announced that Yahoo would be moving away from results licensed from Google's database by the end of this quarter. The move to produce in-house search results is part of the growing competition in the search engine industry. The sector is currently dominated by three big players, Yahoo, MSN and Google. Until recently, Google was considered the most important search engine to be ranked on, however the changes made by Yahoo and the anticipated arrival of MSN's own, proprietary search engine have changed the perceptions of many webmasters and search engine analysts. Where there was once only one, there are now three, opening the doors to more competition, better pricing and hopefully better results from all three.

We are very excited about the "new" Yahoo. The listings returned look highly relevant to keywords entered and we don't see much SPAM in the listings, based on the limited number of keyword phrases we have tested. We may still see Google results filtering in and out of Yahoo for a few weeks yet as the techs at Yahoo work out any bugs or problems associated with their new search engine. There are a few features of the "new" Yahoo webmasters and SEOs should be aware of.

* Yahoo seems to be looking at similar page elements as Inktomi, though the results displayed are somewhat different from those shown on HotBot which runs nearly pure Inktomi results. Elements such as keyword densities, keyword enriched titles and keyword arrangement seem to be important factors in Yahoo's current algorithm. We have also seen Google results popping up from time to time. This is very likely a temporary measure to ensure search-continuity as Yahoo engineers continue to develop their algorithm.
* The description meta tag weighs heavily in Yahoo's algorithm. Webmasters are strongly recommended to display similar incidents of keywords in titles, descriptions and body text for each page in their site. (Be certain the incidents are page and topic specific, of course)
* Yahoo has a new spider known as YahooSlurp. YahooSlurp will work a lot like GoogleBot in that it will follow every HREF link on found a website. YahooSlurp will NOT follow SRC links such as images or FRAMES. When optimizing a site that uses FRAMES, correct use of the <noframes> tag will be necessary. Yahoo's spider can follow dynamic links but Yahoo is advising webmasters to post static pages with HREF (text) links directed to specific sections featuring dynamic content. This tells us that the use of a sitemap will continue to be an SEO Best Practice technique.
* Yahoo will respect common ROBOTS.TXT statements. For more information on ROBOTS.TXT files, please visit: (http://www.robotstxt.org/wc/norobots.html)
* Paying to get into the Inktomi database should attract more attention from YahooSlurp. I am pretty sure that YahooSlurp will find your site if you don't pay however, paying will buy you much greater frequency of visits from the Slurp spider.
* Inktomi is very old-school when it comes to Keyword densities, keyword arrangement and simple-site structure. Optimizing for Inktomi is a lot like optimizing for AltaVista was a few years back (without the leader-pages and gateways that were once so popular). While Yahoo is clearly not running on pure Inktomi results, the fact they share a spider and likely a database leads us to believe that the two sections of the same firm are working together and will continue to share several common algorithmic attributes.

There are several other factors we are examining in regards to the new Yahoo. Hopefully we will have some answers to these questions in the near future.

* What is the relationship between Directory and Algorithmic listings?
* What is the average "turn-around" time for sites from submission to Top10 or Top20 placement?
* Will the expected absence of Lycos effect Yahoo or Inktomi's database in any way?
* What is the role of AlltheWeb and AltaVista? Both are owned by Yahoo which acquired the two companies when it purchased Overture. Speculation had Yahoo buying AV and ATW for their patents but what about their technologies and databases?

This move, in my mind, marks the second phase of the current war of dominance between the big three. Last year was marked by the many mergers and acquisitions that redrew the landscape. This year we've seen former big players such as LookSmart and Lycos fall below the radar screen as the first major casualties. While it is becoming more difficult to predict the next victims, clearly the big winner this month is Terry Semel and the team at Yahoo.

For search engine users, marketers and advertisers, the increased competition between the three major search tools will be, for the most part, beneficial. It will be easier to find relevant information and easier to achieve strong placements on one or more of the major search entities. In the best and most idyllic circumstances, competition breeds better products, fosters stronger customer relations, and pushes competitors to work harder and smarter. We will see more innovations, new product and service offerings, and a race to "lock-in" users by offering more personalized information resources such as Email, dating services and one-click lookups of common information (such as airplane schedules). Hype it or hope for it, the future looks very search friendly.

About the Author
None