Search Engine Cloaking
By James O'Keefe
Posted Saturday, November 6, 2004
Cloaking
As soon as you mention the idea of page cloaking you can be sure that someone somewhere is going to jump to the conclusion that you intend to cheat the search engines. There is no denying the fact that search engine spammers have used page cloaking in the past, and that the reputation of cloaking has been damaged as a result. The fact remains however that there are still many legitimate reasons for wanting to dabble in various forms of page cloaking, so don't be too quick to judge!
What is Page Cloaking
Page cloaking is the practice of altering contents of a page depending on the software, method and/or IP address being used to access it. Cloaking has been made possible through the use of server side technologies such as ASP and PHP (amongst others), and has been abused so much in the past that it will probably never shake off it's bad reputation.
Cloaking is used in all manor of situations, some of them you won't even be aware of until it is pointed out to you. I'll admit to using it on my own site, and I'm not ashamed of the fact either!
Cloaking Methods
There are several different ways of implementing cloaking; each is based on the environment variables that are passed to the server as part of the actual request.
What browser/program is making the request? (HTTP_USER-AGENT)
What IP address are they trying to connect from? (REMOTE_HOST)
What page (if any) did they come from? (HTTP_REFERER)
Did they recently request a page? Have they already set preferences? (Session data)
While the environment variables available to us can be used independently to make generalizations and educated guesses, they become powerful customization tools when combined.
By identifying the program used to make the request, we get an idea of how the data will be used and the capabilities of the browser. Suddenly we can use specialized page designs without having to use or redirect users to different URLs depending on what they could handle. It can also prove an effective way to reduce needless data transfer costs; by sending only the code the browser or bot will understand.
When we want to target groups of users based on their geographical location or connection speed, we can use IP addresses to make an educated guess. Different ISPs will own different IP blocks; these ISPs may operate in a specific region or offer only broadband services. While gathering and maintaining current IP data can be a chore, it is probably the best way to get information on the location of the user without specifically asking them.
The use of referral or session data is probably the most interesting area of cloaking however, as this is where we can learn more about the person making the request. With a little research, you can match visitor types by the sites they come from and tailor the site to their needs. Referral and session data can tell us a lot about the visitor without actually needing to identify them; the possibilities are limited only by the imagination!
Why use Cloaking?
There are many different reasons why people want to use page cloaking, and while some of these can be devious in nature, others are designed to be helpful or to standardize the user experience. Some reasons for using page cloaking include: -
To serve known search engine bots with content optimized for best results.
To hide the real keywords and text used for rankings from normal visitors.
To alter the identity of the site depending on who accessed it.
To protect the server and it's contents from malicious bots or misuse.
To re-brand shared content depending on the domain name used to access it.
To limit the options available depending on browser features or capabilities.
To iron out browser inconsistencies.
To serve relevant information to the user based on Operating System and/or browser.
To create a more personalised experience for the visitor.
You see page cloaking is perhaps more common than you first think, and when you look at the whole picture it doesn't seem as bad as some people make it out to be. It's not so much whether or not you use it, but rather how you use it.
I use ASP and dynamically edit the page to show the style sheet optimized for the browser. Even though my server changes just one line from one browser to the next, it is technically page cloaking; the style sheet it serves to the visitor using Opera is not the same style sheet it sends to the visitor using Internet Explorer. Ironically, if I chose to use a script on the client side to do the same thing, it would not be considered to be page cloaking!
So why does it matter where the script is executed if they both do the same thing?
It's all because we can look at the source code of the script when it is on the client side; thus we can see the alternatives that are available simply by looking at the way the script works. On the server side however, all the logic is hidden away in some script or server module and we only see the results of the server's decision; not how it came to the conclusion on how it should respond to the request.
The Dark side of Cloaking
The use of cloaking is controversial because of the way it has been abused in the past. Some search engine optimisers use it in their promotional campaigns to serve pages optimised for each of the search engines. While this may seem unfair to other site owners fighting for rankings with the same keywords, some of the arguments for doing so are certainly feasible.
Why increase the visitors download time by including Meta tags they don't normally see?
Why send images to the search engines when all they really look at is the text?
Why should they share keyword lists they spend the time researching?
All the reasons above may seem rather selfish, but just because a Webmaster wants to minimize their data transfer costs and protect the investment of their time and effort does not make them an unethical person. Only when the topic of the alternative pages strays from the page the visitor sees can the practice of optimizing for the search engines in this way be deemed to be unethical. There is a fine line between optimizing a page for the search engines and misleading them altogether!
Page jacking occurs when a site owner takes the source code of a page that performs well in the search engines and cloaks it behind their own site. If this is done properly, a search engine spider will see the optimized page and do one of two things; improve the ranking of the unrelated site, or remove the original site from their indexes. This unethical use of cloaking is unfortunately difficult to spot, and is one of the reasons why the use of cloaking is so controversial.
Is Cloaking all that Bad?
Lets forget for a moment the role and ethics of using cloaking for promotion and concentrate on the human factor. After all the main interest of any site owner should be the thoughts and feelings of their visitors.
Given the choice, I'm sure you would like to make your web site more intelligent. Does this particular visitor use a Mac or a PC? Opera, Netscape or Internet Explorer? Why should you waste the user's time by asking them for information that is readily available by the way their browser identifies itself? Wouldn't it be easier to just establish how best to optimize the page source code for the user's browser and operating system before they get it?
By establishing details from the user agent string and determining IP addresses, it is also possible to make an educated guess at the best default language or localized settings. It would be far more convenient if your visitors didn't need to convert the prices into their own currency or search through a long list of international branches to find their nearest store!
Although the examples used above describe the use of program identification and IP address to customize the user experience, user-friendly cloaking scripts can also make use of things such as referrer and session data. Knowing that a visitor came from a search engine while looking for a particular keyword gives you a great opportunity to optimise the page for better conversions. This can even work for informational sites; if they came to that page looking for information, there is a chance that they might be interested in buying a book on the subject!
Cloaking could also be used to protect yourself, your site and your visitors in a variety of ways. For example, it can be used to: -
Ban IPs of known email harvesters (useful if you run a community site).
Minimise the damage and cost of bots flooding the server.
Filter out questionable content from visitors that enter from a child oriented site.
Limit access by countries that require it by law or who are likely to find the site content objectionable.
Disable site features such as order forms to those surfing from problem IP addresses or countries thought to be involved with fraud.
Such legitimate uses of cloaking ought not to be penalized by the search engines. However the activities of site owners that optimize for the engines have meant that those that dare to implement such features to their site run the risk of being banned by the search engines.
The Future could be Bright!
The use of cloaking is sure to become more commonplace as time goes on. Not everyone will hold back on implementing features that improve the usability of their site just because they are worried they will be banned by the search engines.
It would be sad to see the legitimate use of cloaking thwarted by the minority of site owners who choose to use it purely to achieve higher rankings through deception. Cloaking has the scope to make sites seem far more intelligent than they are, to the point where it becomes invisible to the vast majority of people who visit the site that use it.
Wouldn't it be nice to make a visitor feel more "at home" on your site the first time they visit it? Or to provide alternatives to those with limited access so they don't have to miss out entirely?
Cloaking can make such features a reality. If we are willing to let it!
If, you are determined to possibly reap the big rewards as well, Click Here!
James O’Keefe is the owner of My Millionaire Friend. offering FREE articles, tips, hints, and real-world advice on how to make money with your website. Visit his site or join his FREE newsletter by sending a blank email to mailto:newsletter@mymillionairefriend.com.
Webmasters and Ezine Publishers:
Free content for your newsletter or your website!
You are invited to use this article in your newsletter or website. The only requirement is the inclusion of the following, after each article used:
About the Author
James O’Keefe is the owner of My Millionaire Friend offering FREE articles, tips, hints, and real-world advice on how to make money with your website. Visit his site or join his FREE newsletter, by sending a blank email to mailto:newsletter@mymillionairefriend.com.
