Spiders 101 What are Spiders and How do they Work?
By Joelene Wickens-Orlando
Posted Tuesday, June 29, 2004
When I first came across this word “spiders”, I had no clue what it was, what it did, or how it worked. (We’re not talking about the hairy 8-legged creatures). I decided to conduct my own research and to talk to some webmasters to get a better grip of exactly what the purpose and function of spiders are. With this knowledge and understanding of spiders, I thought I would share what I have gathered on spiders. Let’s begin the Spiders 101 tutorial.
Spiders are applications that crawl (henceforth the definition) the web, indexing and retrieving content, usually for the benefit of search engines, information resources and news organizations. As spiders travel, they become responsible for a bit of traffic that gets counted in traffic statistics and ad campaign reports. That still sounds a bit confusing after that description, doesn’t it? Let’s break it down a bit to simplify the process.
* Most of the popular, heavy-traffic, search engines on the Internet today are spiders or crawlers that move along the web in search of web pages to index in their databases.
* Feed a spider a single URL, (let’s say we’re trying to find discount beauty salon equipment) and the first thing it will do with that page is harvest all the links on that page and put them in a queue for a future visit.
* This way the movement of the spider is self-perpetuating as it is programmed to visit these web sites, one by one in it’s queue. Then, secondly, a crawler will make note of the title of the page.
* Once that is done, the crawler will ask itself, “How relevant is the content of the page to its title.”
* Notice that the words: “discount, beauty, salon, and equipment” appear in almost every listing, in either the link text or the regular text.
* The words appear over 100 times as a matter of fact, on that page.
* Now the search engine thinks this page is about discount beauty salon equipment.
* It is now a valuable, content-rich resource for those interested in discount beauty salon equipment.
When you break it down, and see how the spider operates in a step-by-step format, it is now easier to understand how spiders program and operate. Another thought to keep in mind is that spiders help your positioning or ranking in the search engines based on the keywords you select. That’s why choosing a correct set of keyword phrases is paramount to your website achieving top ranking in the search engines. (Notice that I didn’t say in the top 10). That is because getting great positioning in the top 10 search engines is next to impossible. Let’s hope that this little 101 on spiders will help you achieve the best, realistic positioning in the search engines as possible.
About the Author
Joelene Orlando is a Web enthusiast and a staff writer/consultant for eMarketing Answers with a broad knowledge of topics covering Internet marketing and communication strategies for both consumer and small business owners.(http://www.emarketinganswers.com) is dedicated to providing free, comprehensive emarketing resources and online promotion tools.
