Where Are Search Engines Going? Paid Inclusion Trend Emerges

By Mike Banks Valentine
Posted Friday, October 22, 2004

The Search Engine Strategies conference and show, sponsored by AltaVista, Search Engine Watch and Internet.com on August 16-17 provided a glimpse of several emerging search trends, the biggest trend is toward "Paid Inclusion." The show, held in San Francisco at the Fairmont Hotel on Nob Hill, provided a look at where search is headed.

Paid inclusion is the latest of several revenue producing models considered by search engines as the cost of indexing a much larger and more complex web increases and advertising revenue declines. Paid inclusion was started quietly by several previously free search engines as a revenue producing move recently.

Virtually everyone scrambled for new business models when banner advertising effectiveness declined steeply in 2001. Search engines had relied heavily on banner advertising to generate the only income they were seeing in sparse times. When the tech economy took a nosedive in 2001, the justification given by advertising companies for continued banner advertising was now as a "Branding" strategy rather than a sales technique.

To satisfy investor demands for such absurd things as "profit" the search engines began probing for new cash infusions and realized that they could no longer rely on advertising to support free submissions. Since nobody was looking at banner ads any more (or at least not clicking through) support for free search listing nearly disappeared.

Who benefits from search engines most? Those who receive the traffic from searches. Simple. Who should pay for that traffic? Those who gain that traffic. Simple. How do you charge them? Ah! Now things get more complex.

Goto.com was the first to introduce the novel idea of charging businesses for traffic generation directly with "Keyword Bids" starting at a penny per click-through. Now companies began to bid for top positions knowing that being at the top of the list was worth more traffic than being further down the page.

Reaction to the CPC or cost per click model in 1998 was nearly unanimous from the web community, especially while banner advertising revenues were still a viable business model for search engines. It was, "You can't charge for search listings! It'll never fly!" "Searchers won't trust the listings they receive on a pay-per-click basis because they are now 'tainted' with commercial results!" At that time, in 1998, nobody believed first, that people would pay for traffic and second, that searchers would trust paid listings.

It very quickly became apparent after the decline in revenue from banner advertising in 2000 and the dismal performance of banners in delivering traffic that pay-per-click was actually going to work for GoTo. At that point several new pay for performance models were adopted by start-up PPC search engines, and being first to market with the idea, GoTo dominated the crowd.

It was then that the 900 pound gorilla, Yahoo stunned the web with a $199 charge for a review of a commercial web site. Wait, not $199 to be listed, but $199 to be LOOKED AT by a reviewer. It was still not a guarantee of inclusion in the directory! Search engines started thinking about that and realized that Yahoo had the clout to demand money to be reviewed, not listed, but reviewed.

While the major search engines stewed on the dramatic move by Yahoo, they quickly realized that demanding money to be reviewed would not work for them because they actively sought out sites by sending "spiders" out to "crawl" the web and index pages. Search engines wanted to be exhaustive in their coverage of the web and catalog the entire thing, not just the good sites, but ALL of them!

When Google surpassed the 1 billion page mark they tooted loudly that they had indexed more of the web than anyone else. This was seen as a milestone in search history and became the goal of many of the top search engines. Let's index the entire web! That prospect becomes very expensive for search companies and someone has to pay for it. But how? We can't all become pay-per-click engines if we want to index ALL the web, because only a limited number will pay to be listed. So they all stewed about it some more.

Meanwhile, several other directories followed the lead of Yahoo and began to charge to be listed. It became accepted at multiple directory sites, notably at LookSmart and NBCi. Several special interest and topical directories had been charging for listings longer, but directories that listed everyone had a harder time justifying those charges. Yahoo can do it becuase the entire world knows of Yahoo and wants to be seen there. Vertical portals can charge because they draw a very targeted searcher seeking specific businesses. It's worth paying for that targeted traffic.

When the goal is to index the entire web and results are far less relevant and traffic less likely to result in sales to the listed sites, how do you charge sites to be listed? Hmmmm. The search engines said, "We've been getting complaints about a couple of things from sites seeking listings, one that they submit and never get listed, and two that when they are listed it is only months later that they begin to benefit from delivered search engine traffic." AHA! An opportunity presents itself to charge money to someone for listings!

Bingo! Paid inclusion is launched at Inktomi and becomes an instant success with those who have been frustrated by slow listing times and especially with those that can't seem to be listed at all at important engines. Here is a way to get listed and get listed sooner, I'll pay for that! But how much? And are there any benefits above the free submission to entice the reluctant web business into paying for what, until now has been free. We'll guarantee, not only that you'll be listed, but quickly, and to sweeten the pot, we'll recrawl your site on a weekly basis to guarantee fresh listings.

Now it's time for everyone to leap into the new model and begin more frequent crawls for preferred listees. Certainly this will lead even further toward the commercialization of the web, which angers many but is the only solution to supporting the increasingly expensive proposistion of providing complex search to millions and indexing a dramatically larger web.

Time to innovate if you expect to attract those paid inclusion listings. At Fast AllTheWeb, they've come up with an intra- search facility that site owners with under a specified number of pages can install as a branded, hosted search for their own sites as an additional benefit. A hosted and branded site search will likely cost them very little more to implement for paid inclusion customers, while increasing revenue for Fast. As Fast is a partner to the Lycos network, this paid inclusion not only comes with intrasearch for your site and regular site recrawls to keep search results fresh, but also gets listings into the Lycos network as well.

A further enhancement in the race to paid inclusion will be extra cost "partner" or "sponsor" links that turn up at the top of the search results pages. This may affect some search partners differently than others as some now use GoTo paid listings as those "sponsor or partner" links. Certainly GoTo will fight to maintain current partners by making themselves more attractive to those search engines considering paid inclusion programs.

Inktomi has the larger partner list to support their version of paid inclusion and is more attractive for distribution and visibility, but offers little in the way of "perks" for the value driven crowd. Altavista offers their own version of paid inclusion, dividing the program into two separate categories, one for small businesses and sites under 500 pages and the other for the big boys of ebusiness with more than 500 pages to index.

Pricing is just now being determined by many search engines making the paid inclusion move, but it ranges anywhere from $39 per page to be listed up to yearly pricing models which come with the additional benefits, such as the intra-search option from Fast, AllTheWeb.com.

I recommend keeping an eye out for short term sign-up bonuses and perks as the paid inclusion model gains acceptance over the next 6 months. Just as the pay-per-click model of search took some time to gain traction, so too will paid inclusion.

Watch also for partnering changes in the near term as larger networks battle to gain or maintain market share. Some will be more agressive than others so bargains and benefits will no doubt emerge within 6 to 8 months to build customer and brand loyalty early in the race to paid inclusion dominance.

About the Author
Mike Banks Valentine
WebSite101 "Reading List" Weekly Netrepreneur Tip Sheet Weekly Ezine emphasizing small business on the Internet (http://website101.com/arch/) e-tutorial online at: (http://website101.com/shortcourse.html) By week's end you're ready expand your business to the web!