Trends in search engine and index submission

Posted Saturday, January 4, 2003

In the olds days (1999!), submission was a numbers game. The more sites you could properly submit to, the more traffic you'd get. But as the internet matures, this is rapidly changing.

12/12/01: Bruce Clay has published a great PDF chart of the relationships between all the search engines.
07/10/01: Altavista is now offering paid inclusion; see below for details.
05/15/01: The Mamma Metasearch Engine is now offering paid inclusion; see below for details.

The big difference is that these days, it costs to get a listing in the major indexes (Yahoo & Open Directory), and the search engines are also getting into the act -- Inktomi, Altavista and Mamma now have a paid-submission option, and the other major search engines are all considering following along.

There has been a lot of blathering on various discussion forums lately about the "death of free submission," but quite frankly I think that it's a good thing. I know that sounds weird, but bear with me for a minute.

From the standpoint of a commercial website, the problem with the net is that it is drowning in other websites. The number of webpages is growing rapidly, which makes it harder for people to find your website in the sea of other, similar websites.

When people want to find a website, they typically go to one of the major indexes or search engines. From your standpoint as a commercial website operator, would you like their search to list 10,000 websites (submitted for free), or 100 websites (each of which has paid $199 to get listed)? Obviously, the latter!

At the same time, as long as free submissions are available for non-commercial sites (and all the majors still permit this), non-commercial sites are not penalized, because they are not competing with commercial sites. So if you are not trying to make money on the web, fear not -- nothing has changed for you.

But if you are serious about doing business on the web, then paid submissions are a good thing. They cut out all the chaff. Or to put it more bluntly, "Money talks, B*llsh*t walks!"

The Economics of Paid Submission

If you make a profit of $10 when someone buys something on your site, and 1% of your visitors buy something, then each new visitor is worth 10 cents to you. If you can get visitors for 9 cents each, you make money; if they cost you 11 cents, you lose money. It's that simple.

There are three main sources for paid clicks that you should consider; paid submission to indexes (like Yahoo), paid inclusion in a search engine (currently only Inktomi, Altavista and Mamma provide this), and sponsored listings in a pay-per-click search engine (the biggest being http//Overture.com). All can be excellent sources of cost-effective traffic.

Paid Submission to Indexes

Three major indexes are now offering paid submission, and some of them require it in their business and shopping categories. It is important to understand that paid submission does not guarantee a listing -- if your site isn't good enough, they'll reject it and you're out the money (though you can appeal). I have an extensive tutorial on how to submit to Yahoo and the major indexes that should minimize your chances of a rejection and maximize your chances of a great listing that generates real traffic. I consider it the most important page on this site.

I paid $199 for my Yahoo listing (note: now they are charging $299!), and last year Yahoo sent me over 25,000 visitors. $199 / 30,000 = .8 cents a visitor. That's dirt cheap, and this year I get them for nothing, since a Yahoo registration is a 1-time thing. If you're selling on the web, that $299 is the best investment you'll ever make (other than the money you send to me, of course!). However, as of Dec 28, 2001, the Yahoo fee is now a yearly affair, so you need to carefully consider whether or not it will be cost effective.

Yahoo has just started offering Sponsored Listings for between $25 to $300 a month, depending on category. 5 sponsored listings are displayed at the top of category pages (if more than 5 people buy sponsored listings, they rotate randomly). In order to get a sponsored listing, you must first get a normal listing in Yahoo, then you can apply for a sponsored listing in the category your listing is in. You can't use this to change your listing title or description, by the way; it just gets you "up top."

Is it worth it? My experience is that Yahoo Sponsored Listings totally suck! Not only are the clickthroughs dismal, but when I applied for a sponsored listing, Yahoo edited my current Yahoo listing and totally trashed it. Talk about adding insult to injury! Yahoo was totally unresponsive to my inquiries regarding this action.

Note: Please don't confuse Yahoo's Sponsored Listings that appear in the directory listings with the Overture Sponsored Links that can appear in Yahoo search results. The latter can work just fine, and I have written a tutorial on how to use them properly.

Yahoo has also introduced "Most Popular" listings underneath the sponsored listings; so far it is unclear how a site becomes a Most Popular site.

Looksmart has (spring 2002) treacherously "upgraded" all their paid listings to a pay-per-click model. Don't give them a dime if you can avoid it. I no longer consider them to be a major index.

There are other, 2nd-tier indexes that offer paid submissions. None of them are worth it.

Paid Inclusion on Search Engines

There are now three Paid Inclusion services to consider, Inktomi, Altavista and Mamma.

Inktomi offers a paid inclusion service through several resellers. For $39 a year, this service will guarantee your URL stays in the Inktomi database, and check it every couple of days for changes. The advantages of coughing up the cash are: your listing is always fresh, containing your latest content. If you play with your keywords, you get to see how your changes affect your ranking in a couple of days instead of a couple of months.

Inktomi provides search engine results for many other search engines (such as HotBot and AOL), so you get broad placement.

Inktomi currently penalizes pages that are submitted through their free Add URL service; pages submitted through the paid inclusion service or stumbled upon by the Inktomi spider (which independently wanders around the web) do not suffer this penalty. Note however, that if you've submitted a URL through the free service and Inktomi later finds it on its own, the penalty is removed. This ranking penalty is Inktomi's way of dealing with "gateway" pages and spam pages; they claim that the vast majority of page submitted via the free Add URL service are spam.

In addition, according to Inktomi, paid inclusion is the only way to get "dynamic urls" (ie: ones with ? in them) into Inktomi, as the regular spider will not spider them.

I have tested the Inktomi paid inclusion service and it works as advertised. I recommend that commercial sites pay to list their homepage using the service. While you can spend extra and list other URLs on your site (at a discount) via paid inclusion, in most cases this isn't necessary. It appears to be the case that paid inclusion URLs are often used as starting points by the Inktomi spider, and so your other URLs tend to be quickly found and indexed with no penalty. Also, once your page has been found by the spider, resubmitting it via the free Add URL does not cause it to be penalized! So my advice is pay for your homepage, submit all your other pages via the free Add URL, and in a few weeks the spider will have found most of your pages and removed the penalty. I should note that Inktomi says that their paid inclusion service won't cause other URLs to be spidered, but my experience is otherwise. It may be that it doesn't cause your related URLs to be preferentially spidered. The jury is still out on this one.

As with every rule, there are exceptions, and if there are important pages on your site that are constantly changing, paying a little extra to have Inktomi refresh its listing more often may be a good investment.

Note: I get a referral payment if you sign up for Inktomi Paid Inclusion. As always, these payments are donated to the Salvation Army.

Altavista offers a similar service through a company called InfoSpider. Altavista also offers "listing enhancements" such as icons in the search results for an additional fee.

The big problem with Altavista is that their indexing in recent years has gotten slower and slower, and their relevance, to be kind, sucks. This means that their search results have been getting less and less useful. As a result, not surprisingly, they've been losing traffic to more useful sites like Google. Because it is so hard to get a top listing in Altavista other than by blind chance, my advice is that if you don't already have a good listing there -- and are generating significant traffic -- then stick with the regular free submit. If you do have a good listing, then paid inclusion will be a good way of keeping the listing fresh, and if you're on page 1 for a good keyword, then the listing enhancements like an icon may get you more attention.

I am investigating obtaining group rates for http//SelfPromotion.com users for these services.

Mamma, a meta-search engine, now offers paid inclusion into their Mamma's Collection database. Mamma's Collection is one of the sources Mamma uses when generating Mamma's search results (in other words, they combine it with results from Altavista, Yahoo, etc). The price is $60 (for 2 day turnaround on your request) or $30 (for 8 week turnaround), with a $20 annual subscription after the first year. You get to select your keywords, title, and description, subject to the editing of the Mamma staff.

My very preliminary results are encouraging; being in Mamma's Collection has raised my traffic from Mamma from a click every day or two to about 15 clicks a day. At that rate, the cost per click for the first year (assuming you pay the $60 to get in fast) is about a penny. However, as more competing sites get into Mamma's Collection I expect the traffic won't be as high, but even so, it seems like a decent deal for commercial sites.

Sponsored Listings on a Pay-Per-Click Search Engine

If the average visitor to your site earns you 10 cents in profit, and you can get one by paying 5 cents, then you're in a position to make money. That's the idea behind pay-per-click. Search engine results are ranked not by keywords, but by how much you are willing to pay for a click. You bid for each keyword you want to sponsor.

The two leaders in the field are http//Overture.com (by far the big kahuna) and FindWhat. The other sites aren't worth much, in my opinion. But the top two can make you money if you use them correctly, and I strongly recommend that commercial sites consider using them. I've written a detailed tutorial devoted to techniques for efficiently bidding on these sites, as well as Secret Net Tools (available to contributors) that help you find appropriate keywords to bid on and manage your bids.

Future Trends

Looking into my crystal ball, I think it's pretty clear that the days of effective free submission to the major search engines are numbered. This is a classic example of "the tragedy of the commons". If a resource is freely available to everyone, then nobody has any incentive to conserve it, and everyone will try and exploit it as much as they can before the other guy ruins it. The result, in a village common, is overgrazing; on search engines, it is spam.

If on the other hand, you charge people a small amount for listings, they get a lot more selective about what they submit. The quality of listings goes up. Spam goes down. This is a good thing. Furthermore, let's be honest, it's in our interests for the search engines to make money, because a bankrupt search engine is no help to anyone.

Since at the present time, most of the search engines (except for Yahoo) are losing money hand over fist, I think that all the major ones will adopt paid inclusion plans similar to Inktomi. I think they'd be stupid not to.

Going out further on a limb, I will make a stunning prediction. I predict that Yahoo will announce that they will start charging commercial sites a yearly maintenance fee for listings, in addition to their listing fee. And by the way, Yahoo guys, if you end up doing this because I suggested it, you owe me - small, unmarked, nonsequentially numbered bills, please.