The Search Engine Soap Opera

By Kalena Jordan
Posted Monday, September 20, 2004

The history of search engines is a bit like the plot of a soap opera. You know - Bo finds Hope, Bo loses Hope, Bo finds Hope again only to discover it's actually Hope's long lost evil twin Princess Gina and so on.

Just like the TV soaps, the search industry has a strange and illogical history. We started with a particular cast of search engines, new ones soon rose up and tried to usurp market share from the originals, some engines jumped into bed with each other, some of the well known characters died or were killed off by the newcomers, "good" engines decide to turn "evil" in the grab for market share, new industry darlings were born and so on.

Those of us who have been watching this particular soap opera for the past few years are quite addicted to all the plot twists and turns. The thing is, search engines seem to have finally come full circle. Most started up originally with a simple premise: to provide a useful service to persons surfing the Internet; a way to search the millions of web sites and find specific, relevant information, 24 hours a day.

However once a few key players became heavily trafficked, search engines became viable advertising vehicles, attracting mega bucks from companies willing to pay them for the privilege of displaying banner ads to the significant number of eyeballs viewing their sites on a daily basis. Soon everyone wanted in on the act. New search engines developed overnight, driven mainly by profiteers, hungry for their piece of the dot com boom. The "Who's Got the Biggest Index" game began and the searching public began to demand more relevancy and fresher results. Under pressure from over-inflated company valuations, the Dot Com bubble soon burst and everyone was left covered with the sticky mess of financial accountability.

Meanwhile, savvy webmasters had begun to study how search engines worked in order to understand how to structure their web site code to improve their ranking for target search queries. A whole new industry developed from this activity: search engine optimization. Webmasters who didn't have the time or inclination to learn search engine optimization techniques simply paid others who did. Popular directories such as Yahoo! and LookSmart took advantage of consumer demand for listings by introducing the first paid submission services. Industry players took note of the developments and introduced commercial search engines where web site owners could simply pay their way to the top of the rankings rather than rely on ranking algorithms - voila! - the first pay per click search engines were born.

It wasn't long before smaller search engines and directories followed the lead set by the larger directories and introduced services to assist webmasters to ensure a place for their sites in the search listings - either via a third party partnership with pay per click search engines, or by introducing a new guaranteed indexing service which became widely referred to as Paid Inclusion. Soon it seemed everyone was partnering with everyone else in order to get their cut of the deals being done. Some search engines were cannibalized by others or bought out by inexperienced companies and sacrificed at the altar of mis-management. Search veterans left cash poor by the dot com bust, or unable to cope with the competition, fell by the wayside.

At this point, you could say that the search industry was almost exclusively driven by profit and share price. At many of the majors, the needs of the searcher were temporarily replaced by (or mistaken for) the needs of the shareholders.

But in the background a relatively small search engine had been slowly building their database and gaining market share since 1998. Increasing numbers of searchers, disappointed with the irrelevant or outdated results they were receiving at other sites, began to flock to this newcomer with the curious name: Google. Despite still being in BETA mode, the search engine began to get a reputation for the quality of sites in its database, the lightening fast results it produced and the simple, no-nonsense site design. Media attention arrived, as did more market share.

The major search engines and directories now had no choice but to sit up and take notice. Almost too late, they realized what they had been doing wrong for the past few years and why they were losing market share so easily to this young upstart firm. Searchers have always wanted fast, relevant, up to date results from their chosen search engine. The fact was that very few directories and engines were offering this any longer. Their sites had become portals packed with advertising and third party information sources; the original search function seemingly lost in the forest of information. But Google had made a point of ALWAYS offering searchers what they wanted, hence their success. The penny dropped and the majors scrambled to get back to basics. Yahoo! took things one step further and embraced Google as their new third party results provider, taking a small investment stake in the company and dumping industry veteran Inktomi in the process.

So where are we now in the plot of "Days of our Search Engines"? Over the past 12 months, some search engines and directories lost their way completely, yielding to the pressure in the boardroom to become more profitable and in so doing, losing forever the trust of their market. Others simply slipped a long way off the radar and are desperately trying to claw their way back up the mountain. But with the KISS example set by Google and the glaring evidence that you CAN be profitable by listening to users rather than cash registers, the search industry storyline is finally getting back on track.

Yahoo! has recently combined Google results with their main search listings, new technically advanced engines such as FAST and Teoma are making an impact on the market and AltaVista appears to be making solid efforts to improve their index refresh rate and quality of results. The end result is a richer user experience for searchers and a more promising future for the search industry where content is, once again, king.

About the Author
Article by Kalena Jordan, CEO of Web Rank. Kalena was one of the first search engine optimization experts in Australasia and is well known and respected in her field. For more of her articles on search engine ranking and online marketing, please visit High Search Engine Ranking.