You Can Participate in Crawling the Web! From Your PC!

By Mike Banks Valentine
Posted Tuesday, February 10, 2004

Oh, for the simplicity of the good old days of 1995! Where all a web site owner had to do was put important keywords about her web site in the metatags of the HTML code of the front page of her site with a clear description and title to rank well in the search engines and make it easy for web searchers to find her.
What made it become so difficult to gain and maintain a bit of internet visibility? It got crowded on the web. Crowds made it more difficult to get noticed - so instead of simply including proper metatags, web site owners had to begin to make themselves stand out to get noticed. Many did that by resorting to trickery, like keyword stuffing and link farms.

Since then, it has become increasingly difficult to get a web site indexed and ranked well at the major search engines. Strategies have evolved that seem to work, then mergers and partnerships change and so do all of the rules.

It never ends and the confusion grows with each passing day. It seemed that paid inclusion and pay-per-click were the only ways to get your site listed anywhere besides Google lately, let alone ranked well.

LookSmart has come up with a brilliant idea to grow their own index that, if it catches on, will offer huge benefits to both LookSmart and to the participants in the program. That seems to be the elusive win-win situation that all but Google have been looking for. It's called distributed web crawling and gives web site owners a hand in growing the Looksmart index as well as the ability to include their own site in the results!

This win-win situation requires participants to download the "Grub" distributed crawler client to their PC and allow it to run when the computer is otherwise idle. The software to run a LookSmart Grub web crawler is available at:

Looksmart Grub

It is based on open source software and is made available to all to distribute and even modify! Since installing "Grubby" on my own PC three days ago, I've personally contributed over 100,000 URL's to the LookSmart database! It's fascinating to watch and the GUI is both entertaining and attractive. It can run invisibly when minimized in the system tray or you can sit and watch as the URL's fly by during a crawl from within two available views on your PC desktop.

I love the concept and plan to continue helping LookSmart grow their database. I've also taken advantage of the option to have "Grubby" crawl both my own sites and those of clients by including a small text file with a digital ID in the root directory of each server I have access to.

There are no drawbacks here, just benefits. I help "Grubby" to crawl the web for LookSmart and Grubby helps me to get my own sites listed there by agreeing to crawl them regularly. No payment involved other than the donated bandwidth and the PC computing power that would be running a silly screen saver otherwise. They even provide a cool screensaver of their own that simulates the crawling process with colorful orbs that represent the sites you are crawling to help weave the web!

When you visit the front page at the URL above, you see the latest statistics on your contribution to crawling the web. They've even introduced a measure of competition into the program by showing top contributors and ranking them. Sunday the stats at the top of the page read, "323 clients running - crawling 24,355,058 URLs in the last 24 hours." It will be interesting and very telling to see if that number grows significantly or very soon because that will be some measure of the success of the program over time.

This process and the software all seem to harken back to friendlier times on the web where community mattered more than profit. This could even threaten the mighty Google if it spreads, since we all love to participate in things that interest us and LookSmart has a really interesting little Grub helping to grow the web from my home office and yours.

About The Author
Mike Banks Valentine is a Search Engine Optimization specialist practicing ethical small business SEO searchengineoptimism.com/SEO_Tutorial/