Preparing your pages for search engines

Posted Saturday, August 30, 2003

Before you submit your pages to the search engines, it is crucial that you make sure they are search-engine friendly. Here are some basic tips on what to do.

Step 0 - Make sure your site is useful!

I'm always amazed by how many people miss this simple concept. They spend a huge amount of time trying to get good search engine rankings and lots of visitors, and then what do the visitors find? A poorly designed, badly written website. If there's one bit of advice I wish I could drum into everyone who visits this site, it is this:

"It isn't how many 'hits' you get, but how many 'sales' you make."

Making your pages "people-friendly" is as important as making them "search-engine" friendly. It's usually much easier to double the effectiveness of your website than it is to double your traffic, and the bottom line result is the same. At the bottom of this article, I'll give you some advice on how to do that.

But for now, let's talk about what you can do to help out the search engines.

Step 1 - Determine your Key Phrases

People get obsessive about their keywords. This is wrong. It is difficult if not impossible to get high rankings based on keywords. Instead, you need to think about keyphrases.

The easiest way to do this is ask yourself "what would someone trying to find me type in when they search?" Make a list of these. Try them out on the search engines -- pretend to be someone looking for your product or service.

If your business is geographically restricted, then your keyphrases should reflect this. For example, if you are a real-estate broker in Wilmington, North Carolina, then the key phrase "buying real estate" is a waste of time; instead, the more specific phrase "buying real estate in wilmington north carolina" is what you want to be thinking about.

Think about variations on the key phrases and write them down. Continuing with our example:

real estate in wilmington north carolina
real estate brokers in wilmington north carolina
buying real estate in wilmington north carolina
selling real estate in wilmington north carolina
renting real estate in wilmington north carolina
home buying in wilmington north carolina
house buying in wilmington north carolina
selling a house in wilmington north carolina
renting a house in wilmington north carolina
renting an apartment in wilmington north carolina
apartment renting in wilmington north carolina
real estate brokers in wilmington nc
buying real estate in wilmington nc
selling real estate in wilmington nc
renting real estate in wilmington nc
home buying in wilmington nc
house buying in wilmington nc
selling a house in wilmington nc
renting a house in wilmington nc
renting an apartment in wilmington nc
apartment renting in wilmington nc
real estate brokers in new hanover county
buying real estate in new hanover county
selling real estate in new hanover county
renting real estate in new hanover county
home buying in new hanover county
house buying in new hanover county
selling a house in new hanover county
renting a house in new hanover county
renting an apartment in new hanover county
apartment renting in new hanover county

The above is only a partial list, but you get the idea. You can also get a good idea of what keyphrases and page design techniques work well by looking at other pages that do well in the searches you've tried. Note that this sample list is just a list of possible keyphrases -- we're not going to use all of them because we won't have room.

One of your fellow users, Stephen Sherman, pointed out an interesting subtlety about keyphrase selection. Let's assume you find a keyphrase that you think people will type in a lot. Try it, and look at the results. If the results seem to be "on topic", then people are likely to drill down several pages to find a listing that is just right for them. This kind of keyphrase is one you want to target, but if you don't get on page one, you'll still get traffic. If, on the other hand, the results are mostly irrelevant (or full of spammed listings), then people will rarely look at page 2, or even more than the first few listings. These keyphrases are thus not as valuable. This doesn't automatically mean you shouldn't try to target it -- none of the criteria are absolutes -- but it does mean that it will be more difficult to get a useful listing with that keyphrase.

Two great resources for finding out what keyphrases are the most effective are the http//Overture.com Search Suggestions Page and WordTracker. On Overture's search suggestion page, you just type in a very general keyphrase (like "real estate") and it tells you all of the more specific keyphrases that relate to that keyphrase and how many hits they got.

Using our example of selling real estate in Wilmington, NC and entering "real estate" into the Overture tool, I found that in October 1999, the broad keyphrase "real estate" was searched for 67016 times (on Overture). However, you'd never want to target that keyphrase, because you don't care if someone in Kalamazoo wants real estate; you're only interested in people who want houses in Wilmington. Looking down the list of results, I found that "north carolina real estate" got 489 searches. That's a possibility. Doing a search suggestion on "wilmington" revealed several hundred hits on "wilmington north carolina," "wilmington nc" and related topics. Zooming in even further, getting a suggestion on "wilmington real" found 36 hits on specific queries related to real estate in wilmington, NC. These queries obviously don't get a huge amount of traffic, but because they're so focused on what you're trying to do, they're often the best ones to target.

Side note: I'm a big fan of pay-per click services like Overture. The top position on those nice, juicy specific queries on Overture could be had for a nickel each. Such a deal! Yeah, you might only get 10 visits a month by sponsoring "wilmington real" but each one would be looking for exactly what you're selling - at a cost of five cents per pre-qualified visitor! Recent changes at Overture have made it a lot less valuable for highly targeted advertisers, however.

Those of you who have contributed and gained access to the Secret Net Tools will want to check out the Keyword Susser tool, which will do multiple queries using the http//Overture.com Search Suggestions system and merge the results. It's a really great way to find out what your keyphrases actually are.

WordTracker goes a bit further. It helps you develop lists of relevant keyphrases, ranked by their popularity. It then queries the major search engines to determine which keyphrases are the least competitive. It's usually not much use targeting a popular keyphrase (lots of searches) if there are millions of other pages that contain that keyphrase. On the other hand, a relevant keyphrase that only gets a few searches a day but which has only a few pages competing for it is a good candidate, because it will be much easier to get a high ranking. WordTracker has a free trial that will give you a lot of information, and additional services available by subscription - including some great tools for working with Overture.

My advice is to use the Overture tool to get a rough idea what your keyphrases should be (and find ones you might not have thought about), and then use WordTracker to determine which ones you really should be targeting - and (this is key!) to rank them in order of importance.

OK. At this point, you know what your best keyphrases are. You've got your list. You've checked it twice. Now it's time to use it!

Step 2 - Crafting your TITLE tag

Most people make the mistake of using a page title that's good for people but lousy for the search engines. Big mistake. A title like "Bill Phillips - Real Estate Broker" is a disaster! The golden rule is this: All your most important keyphrases should be in the TITLE tag. So what you do is look at your keyphrases, make a list of all the important words, and create a title tag that uses them. Also, keep in mind that browsers only display the first few words of a title tag (whatever fits into the title bar of the window). So while the first sentence of your title tag should be "human readable", the rest can be just a list of keyphrases.

There is some debate as to whether a very long title tag is a good thing or a bad thing when it comes to search engines. Some people are concerned that a very long title tag might result in the search engines deciding the page is a "spam page." I'm waffling on this issue right now. Based on the available (scanty!) evidence, my advice is to keep the title between 15 and 20 words. But you might want to try longer title tags on some of your pages, just to see what happens! So Bill Phillips might have a title that looks like this:

"Real Estate in Wilmington, North Carolina - New Hanover County - Buying Selling Renting Houses Homes Apartments Commercial Property Office"

The reason for this is that the three most important places to have keyphrases and phrases are your title tag, your meta tags, and your first paragraph. You want them to all contain the same important words; this increases your keyphrase density and improves your rankings.

Step 3 - The Meta Tags

The fabled Meta tags are important to getting good rankings, and on many search engines, the page title (often truncated) and the Meta Description tag are what gets displayed.

Meta tags go in the HEAD section of the HTML page (the same section as the TITLE tag). The Meta Description tag should contain a short description of the web-page. Guess what? You've already written one for the TITLE tag! So just edit that to make it totally human readable (and perhaps a little shorter), and you're done.

My advice on the length of this description is keep it between 100 and 200 characters. Remember: the description tag should be written for humans to read. It should not be a list of keywords!

The other Meta tag is the Meta Keywords tag. What you do is take your keyphrases, and enter them in the order you think is most appropriate, separated by commas. Don't repeat a keyphrase, and don't repeat any individual word more than 5 times or so. This may mean that you can't use some of your better keyphrases.

The reason why you don't want to repeat any particular word more than 5 times is that some search engines may penalize you for doing this. Search engines aren't as sensitive to keyword repeating as they used to be (most of them ignore extra repeats), but play it safe. The exception is common "noise" words like "the", "in", "a", "and" and so on. Most search engines ignore them. Leave them in, but don't worry if you have more than 5 of any of them.

If you've got a lot of keyphrases that really are relevant to your site, the best thing to do is build "theme" pages devoted to a particular keyphrase or set of keyphrases. This is good for you, good for your visitors, and appreciated by the search engines. Use the most important keyphrases on your homepage.

Some people get confused about whether to use commas between phrases, and whether to capitalize keywords. The truth is, some search engines pay attention to the commas, some don't. But the ones who don't treat them as "white-space". So just use commas as appropriate, but don't waste a character putting a space after the comma. Similarly, just capitalize words as you might expect people to normally use them. Most search engines will ignore the capitalization, but it can't hurt to help out those that make note of it.

If you want to get really fancy, play the cunning comma trick. The search engines that don't pay attention to commas sometimes pay attention to sequences of words. So if you can put two keyphrases together with a comma between them, and the last words of the first keyphrase coupled with the first words of the next keyphrase make up one of your keyphrases, then you've gotten 3 keyphrases for the price of two! Normally, however, this is difficult, so don't waste too much time over it.

Keep your keywords meta-tag length between 200-400 characters. Unfortunately, this means you may not be able to include all of your key phrases in your meta keywords tag even if you don't repeat a word too often. The theme pages concept deals with this also. After pruning away, our sample keywords tag might look like this:

"real estate in wilmington north carolina,buying real estate in wilmington north carolina,selling real estate in wilmington north carolina,renting real estate in wilmington north carolina,real estate broker in wilmington north carolina,new hanover county,south-east north carolina,house broker,apartment broker,home sales,apartment rental"

Step 4 - The first paragraph

The first paragraph of your page should recapitulate and expand upon everything in your title and meta tags. You need to have all those keyphrases in it. However, since this is going to be read by people, it needs to be written with them in mind. This is where you introduce yourself to your visitors, so you want to make a good impression.

Try to put this first paragraph as close to the BODY tag as possible. Avoid putting graphics or other HTML in front of your first paragraph as much as you can. I don't have a banner ad on my homepage for this reason. Also, use the H1 or H2 tag to emphasize your opening sentence (but make sure it looks tasteful!).

Step 5 - Don't Go Overboard - and whatever you do, don't put up spam pages!

You clearly want to have your important keyphrases on your page more than once, because this is what gives the search engines a clue as to what your page is really about. But you don't want your keyphrases to appear too many times, because that might make the search engines think your page is a spam page trying to rank highly for a particular phrase.

The question then becomes, how much is too much? And the answer is, nobody knows for sure, and it's going to be different from search engine to search engine. Rumor has it that Google likes pages with less than 13 repeats of a keyphrase, for example.

My advice is to try and keep the number of repeats of important phrases down to 10 or less; this means all instances, in title, meta tags, and the text of the page. Sometimes this simply isn't possible, because the phrase is so integral to your topic, so don't get paranoid about this. Just keep it in mind.

There are certain classes of sites and pages that the big guys consider spam, and either won't list, or will penalize. The major indexes consider the following kinds sites to be spam and will not list them:

- Affiliate sites with same or similar content but a different site designs.
- Mirror sites. Submitting mirror URLs to different categories is also considered spam.
- Multi-lingual sites are acceptable as long as the URL resolves to the appropriate language.
- Sites that use redirects or any type of bait-and-switch practice.
- Using frames to hide a real URL, commonly referred to as "poor man's cloaking," is also considered spam.
- Sites whose sole purpose is to drive traffic to affiliate links or sites that contain these types of links.
- Sites without original content.
- Sites that are repeatedly resubmitted (over 5 times) without being accepted.

In addition, the major search engines are actively penalizing/banning sites that employ the following techniques:

- Web pages that are built primarily for the search engines and not your target audience, especially machine-generated pages.
- Pages that contain hidden text and hidden links.
- "Great quantity and little value" pages.
- Link farming and link spamming, particularly free-for-all (FFA) links.
- Cloaking, a practice in which the search engine and the end user do not view the same page.
- Sites with numerous, unnecessary host names (i.e. http//poker.abc.com, http//blackjack.abc.com, etc.).
- Excessively cross-linking sites to artificially inflate a site's apparent popularity.
- Affiliate spam.

People who repeatedly submit spam sites to the big guys have not only been blacklisted, but in some cases, their previously submitted (and legitimate) sites have been removed. So be nice to the Indexes, and they'll be nice to you. And credit where credit is due: Chris Sherman's SearchDay Newsletter is the place to find out what works -- and what doesn't -- with the search engines. Do yourself a favor and check it out.

Extra Credit!

Now you've got a decent webpage, with good content and meta tags. You're ready to submit to the search engines. Or are you?

What you've done so far are the basics. There's a lot more you can do, time and money permitting, to improve your odds of ranking highly. Here are some tips on what to do (and not to do), to get "extra credit" from the search engines.

Get your own domain name

This is the best investment you'll ever make. A domain name costs $20-30 a year, it's dirt cheap.

There are three big reasons for getting your own domain name:

- Some search engines won't list you unless you do.
- People are more likely to buy if you have your own domain name. What looks better to you, "http//www.fredjones.com/" or "http//members.aol.com/fredjones123/"?
- You can change your web hosting service without messing up all of your search engine listings.

I get emails every week from people saying "I'm moving from aol to msn, what will happen to my rankings?" If they had their own domain name, nothing, it's not a problem.

I personally use Internet Names WorldWide for my registrations, they are an Australian company with good service and a great user agreement (you own your domain name, not them), though not the cheapest.

This is an absolute no-brainer. Get your own domain name!

Don't get hyped about long domain names

I've been getting a lot of questions about the new, longer domain names that are available. There is a lot of misinformation being passed around about them. The big lie is that if you have a domain name with lots of keywords in it (eg: http//add-url-register-website-promote-site-selfpromotion.com) you will get a higher ranking in the search engines.

This is flat out not true. NONE of the major search engines will significantly boost your rankings based on keywords in your url. Not one. This is what they said when Danny Sullivan, editor of the highly recommended Search Engine Watch Newsletter asked them, and I've confirmed it by experiment. If the search engines look at them at all, they simply add the url text to the rest of the page, so the added benefit of keywords in the URL is totally insignificant. Don't waste your money.

My advice is to try and go for a short, memorable domain name, either 1 word or 2 words combined, or with an i, e, i- or e- prefix. Make it easy to type, and easy to remember.

If you insist on trying the keywords in URL, do it either using subdirectories (eg: http//selfpromotion.com/add-url/register-website/promote-site.html) or subdomains (eg: http//add-url-register-website-promote-site.selfpromotion.com/). You'll still be wasting your time, but at least you won't be wasting money!

Avoid Search Engine Tricks

Some "experts" advise trying to trick search engines by putting keyphrases in comments, putting them in text that is the same color as your background, and so on. I strongly advise that you not try these tricks. Bluntly, most of them don't work -- and the ones that do may stop working at any minute, as the search engines are constantly trying to detect and defeat them.

My philosophy is that you should try and help the search engines by making it as easy as possible to get a good idea of what your page is about. That way, as search engines get better and better at rating the contents of sites, your rankings will get better over time, with no effort from you.

I know this is sort of repeating what I just said a couple of screens higher on this page, but it bears repeating. If you try and fool the search engines, in the long run, you'll be the fool.

Got Links?

Once you have your pages up and running, and chock full of useful content, it's a very good idea to try and get other people to link to them. It's not enough to just get them in the search engines. There are three very good reasons for doing this: First, many search engines are now using link popularity (how many other pages link to your page) as a ranking criteria -- they figure that if other sites link to your page, it might be useful. Second, it's recently been revealed that Inktomi applies a ranking penalty to any url submitted through their free "Add URL" system, but removes it if their spider also finds the page by following a link from another site (they do this in an attempt to find and penalize "doorway" pages). And third, you'll get traffic from the websites that link to you.

Getting links isn't that hard. When you find a website that has content similar to yours, email the webmaster and ask for a link, pointing out why it would be appropriate. If he has content on his site useful to your visitors, link to him without even offering to trade links. Link to him, then email him and ask for a link back.

The biggest search engine to use link popularity is Google (who came up with the idea), and the two sites that generate the most link "value" on Google are Yahoo and Open Directory. For many people, the true value of the $299 a year cost of a Yahoo listing isn't the clicks from Yahoo, but the boost in their rankings on Google. Open Directory doesn't cost anything, but getting in can be time consuming.

A note about Framed sites

Many "experts" also say that using frames to construct your website can hurt their rankings. My experience is that this is not so, as long as you construct your frames properly. The trick is this: make sure that your "frameset" page has a proper title tag and meta tags. Similarly, your subframe pages should have the same ingredients (perhaps with modified contents), as well as a little bit of javascript that "pops" the user to the proper framed presentation if they surf into the subframe page.

Once you've got your pages configured, simply promote them all (the frameset page and the subframe pages) to the search engines. The MultiSubmitter tool, which you get access to by forking over $10 or more, makes this easy!

Checking your HTML

Almost all websites have HTML errors, even those that appear to display nicely on your browser. Browsers do a pretty good job of being tolerant of errors, but even so, it's a good idea to make sure your HTML is as perfect as possible, as this increases the chance that your website will display the way you want it to on as many browsers as possible (both past, present and future).

What about a "robots.txt" file?

The robots.txt file is a special file you can place on your webserver to restrict access by some or all webcrawling robots to some or all of your site. You only need one if you want to place some areas of your website "off-limits" to robots. If your whole website is open to them, you don't need one.

You can only have a robots.txt file if you own your own domain, because they are always located in the same place (so the robot can find them!). Thus, my robots.txt file is located at http//selfpromotion.com/robots.txt.

One caution: some robots interpret a blank robots.txt file as meaning "don't crawl any pages on this website." So if you don't need a robots.txt file, don't have one (even a blank one!) on your server.

Graduate School: Shopping Cart Traps

This tip is only relevant if you're running a shopping cart or tracking your users individually as they move through your website. It's a very technical issue, but it nails a lot of advanced website operators, so I thought I would mention it. This problem does not apply to 99% of the people reading this page, so if it sounds like it's written in martian, don't worry about it.

Here's the problem in a nutshell. A visitor arrives at your site, http//www.coolstuff.com, and your super-duper advanced web-commerce server redirects them to a url that encodes an individual user context, either a url with a ? in it (eg: http//www.coolstuff.com/?CART=12345678) or a url with the users shopping cart ID encoded into it (eg: http//www.coolstuff.com/12345678/). The classic example of this kind of thing is http//Amazon.com; type "http//www.amazon.com" into your browser and you'll end up at someplace like "http//www.amazon.com/exec/obidos/subst/home/redirect.html/002-3104337-1035221"

These kind of urls give search engines apoplexy, and the most usual response is to not index them at all.

Another common trick to maintain a user context is to use cookies, in particular what I call the "Stupid ASP trick" of redirecting to the same webpage to set a cookie (so named because it's a standard trick used by Microsoft's ASP software). Search engines ignore cookies, so it doesn't work.

Individual user contexts cause huge problems for search engines, and they can cause huge problems for you as well. A whacked-out web spider going through your whole catalog adding all 327,123 of your products to a single shopping cart can really make your day exciting.

The solution is simple: (1) don't create an individual user context until you need to; don't create a shopping cart until the first item is ordered. (2) make sure all actions that create or change an individual user context are hidden behind POST forms -- because search engines don't follow such links.

In some cases, the software you are using doesn't allow you to do this. If so, consider running two versions of your site, the public http//www.coolstuff.com, and the "real" webcommerce version, "http//sales.coolstuff.com". The sites are identical, except that all the "buy" buttons from http//www.coolstuff.com go to http//sales.coolstuff.com, and there are no links back. So people who need a shopping cart shift from the public site to the sales site, but the search engines stay on the public side. Everyone's happy. PS: make sure that http//sales.coolstuff.com only handles requests that come with a referer from http//www.coolstuff.com!