Product Review: Robogen

By Richard Lowe
Posted Wednesday, October 20, 2004

Are you familiar with the Robot Exclusion Standard? This is a way whereby you can inform search engines and other automated spiders (programs which scan your web sites for information) what pages and directories are not to be examined. The standard is precisely defined and a little obscure, and while not overly complicated it can be difficult to understand.

There is help available, believe it or not. That obscure file called "robots.txt" no longer needs to be hand-edited. No longer do you need to type in server path names, remember robot and spider names and enter strange symbols which don't seem to make any sense.

Okay, here's a quick review. The "robots.txt" file simply lists the spiders that you want to prevent from listing all or part of your site. Along with each spider is the list of directories and files that it is not allowed to examine. Wildcards are allowed.

It's not very difficult but it is time consuming if you have lots of robots and files to specify. In addition, it's easy to misspell something and fali to achieve your goal.

Now, someone has created a nice little piece of shareware called RoboGen, and it's not very expensive considering the benefits that you will receive.

I stumbled across this entirely by accident, since it never occurred to me that someone would write a program such as this. I downloaded the 30 day demonstration and quickly discovered how wonderful it is. Robogen has already saved me hours and hours of work.

Okay, here's how it works. You enter your FTP login information and log into your site. You then select robots by name in the left pane, and indicate which directories and/or files are to be excluded by the robot in the right pane.

You can create a new file or edit an existing one. Even better, Robogen will work directly off the "robots.txt" file on your web site. It will log in, read the existing file, allow you to edit to your hearts content, then update the file directly.

I like little specialized tool such as this which truly make a webmasters life easy. The product is inexpensive (there is a limited free version as well), well documented and complete. Highly recommended.

About the Author
Richard Lowe Jr. is the webmaster of Internet Tips And Secrets at (http://www.internet-tips.net) - Visit our website any time to read over 1,000 complete FREE articles about how to improve your internet profits, enjoyment and knowledge.