Spider Friendly Content Pages

By Bob McElwain
Posted Saturday, May 24, 2003

It is virtually impossible to build a site in which each page brings good search engine position. The home page, for example, will likely change frequently. Thus spiders will not find it the same when they return, which they do, roughly once each month.

Pages devoted to selling product do not often rank well. The same is true of a page where visitors can subscribe to your newsletter. Or the one you pop up to say thanks when they do subscribe. So how does one go about getting good search engine positions?

Great Content Is The Answer

So what is great content? Any information surfers may need. However, it must also be a topic that enhances your site purpose. That is, there is no room on a site devoted to baseball for a piece describing the inner workings of steam engines.

Assuming you have a clear read on who your visitors are, then it's only a matter of selecting a topic likely to be of interest to at least some of them. Given this, write the page for your visitors, not the search engines. Then do what you can to make the spiders happy.

Happy Spiders?

Not likely. It is impossible to please them all. Some see "Market," "MARKET," and "market" as separate words; others see only one repeated three times. Some see "market" as "marketing;" most require a specific match. "Markets" may be seen as "Market," but in other cases both forms may be required.

Okay, we'll include all cases in our keyword tag: Market, MARKET, market, Markets, MARKETS, markets, Marketing, MARKETING, marketing.

That's got it covered fine, but how do we make this work with a spider that considers more than three repetitions as spam? One that might even consider all of the above as 9 repetitions of one word?

You Can't Get There From Here

Search engines are competing in a multi-billion dollar race. The winner will be the one that can most consistently present the most relevant information available in response to a query.

Be assured that with the stakes this high, the competition is fierce. They are not about to reveal their latest wrinkle to improve their listings. Which leaves us with empirical evidence and educated guesses.

Try to sort this all out for each search engine, and you'll go crazy. Not to mention constant changes which mean one or more of the carefully defined "rules" no longer holds.

Even supposing you had an accurate listing of the rules for each engine. Would you seriously consider creating a separate page for each? Not me. I have much more profitable ways in which to use my time.

Take the longer view. Spiders are getting smarter every day. And they are becoming smarter at a rapidly increasing rate. Some are now reading a page as if with a thesaurus in hand, thus being able to see house and home as having similar meanings.

Grammar checkers exist; I expect to see these and related tools implemented in spider logic. In the not-to-distant future, those keyword-rich doorway pages are going to be discarded.

Meanwhile we need to create some great content pages and try to make the spiders as happy as possible. Here's my approach.

Finding Keywords

Given a topic and a mental draft of what needs to be written, I identify 1 to 3 keyword phases. I work at this, trying to put myself in the shoes of one who will search for this information. If I am building a major page, or one of a set of related topics, I may take the time to visit http//GoTo.Com to find phrases actually entered.

Meta Tags

I build a rough draft of the title and description tags before beginning to write. They must serve two purposes. First the title is the headline of an ad which draws the reader into the ad copy (description). And the description must compel a click to my site. Second, though, to please the spiders, keywords need to be included, and the closer to the beginning of the statements the better. (I try not to think about the fact that some spiders will ignore both tags.)

Since Excite limits a title to 70 characters, I try to hold under this. If I go over, I try to work things out so that truncation does little harm. I try to hold the description under 150 characters, the limit at AltaVista. I use these limits because together, AltaVista and Excite dominate among search engines.

These two tags are so vitally important, that I review them often as I write.

The keyword tag, on the other hand, gets little attention. This tag has been so abused, I simply can't get a handle on what works best. Some meta tag checkers still claim you ought to use all 1000 characters allowed. This seems unwise.

I include only my keyword phrases, all in lower case. But I do add the plural case and "ing" when appropriate.

The Content

When I begin to write, I think only of communicating as effectively as possible with my visitor. I keep the keywords in mind and seek to build in a theme based upon them. After editing a first draft, I will often lay it aside for a day or two before continuing. My visitors are my target here, not the spiders.

The Spider's Turn

If I can build some header tags with keywords, I will. I don't bother with ALT assignments or comments in the source, although this reportedly gives a boost with some search engines.

I work at including keywords as close to the top of the page as possible, in the first 100-200 words. For this is the part of the page in which one expects to find the subject defined, followed by further explanation and expansion. Even now, spiders also expect this.

I also work at rephrasing things to add more repetitions of keywords and to bring them as close to the beginning of paragraphs as possible.

And I make a point of repeating the keywords in the close of the page, a sort of "theme" wrap up, if you will.

One further thing I do is look for words I incidentally used so frequently they may dilute the weight of the keywords. For example, if I have used "buildings" too often, I may replace some instances with "structures" or a specific name for a type.

But throughout, I absolutely refuse to sacrifice readability. To me, my visitor is far more important than any search engine.

Other Guidelines

Keyword density is the percentage of words that the keywords are to the total number of words. It is considered quite differently by different spiders. Some suggest as much as 15% of a page be keywords. To me this is nonsense, for it makes the page unintelligible to a visitor. I have never been able to get above 2% without decreasing readability, even when using three keywords.

Page length expected also differs drastically. Many claim short pages are better. 300-600 words is often suggested. But Excite doesn't care how long a page is. I say what needs saying as briefly as possible and call it good.

Never Look Back

When the page is polished, I submit it to the major search engines. Then I do something you really ought to try.

I never look back. The page is up and that's that. I've got more important things to do than worry about what position it has today. Or where it may be tomorrow. If I've done the job properly, my visitors will enjoy the page. And that's the end of it.