Submitting to the Spidering Search Engines

By Jill Whalen
Posted Friday, September 17, 2004

I'm often asked how often one should submit or resubmit their site to the spidering search engines. Many people believe they should submit their site on a periodic basis to keep their site in the search engines' "faces." Submitting URLs that are already in a search engine's database, which hasn't had any significant content changes, is not a good idea.

A general rule of thumb for submitting and resubmitting pages is to first check each engine to see if the new page has been indexed. There is no reason to resubmit a page to an engine if the updated page is already in the database.

Submitting to Google

Google is the best engine for spidering through sites. It's only necessary to add your main URL and Google will eventually spider the rest of your site. You can check if Google has indexed your site by typing your URL into their search box. They currently take about one-to-two months to index and reindex pages. Representatives from Google have warned that they prefer to find sites and pages on their own, rather than you submitting them through their form. However, they also state that there is no penalty for submitting to them through their form. Google has no immediate plans to start a pay-for-inclusion program. Google's Add URL form can be found here.

Submitting to Hotbot/Inktomi

At HotBot, you can type (www.yoursite.com) into their search box and see if it shows up. Since HotBot uses the Inktomi database for its search results, you can simply wait for the Inktomi spider to find your site. If you've been listed in the LookSmart directory, the Inktomi spider should easily find you in no time. You can also submit to them through any Inktomi partner site. As with most of the search engines, HotBot/Inktomi does not guarantee that they will list all sites that are submitted to them for free. For guaranteed inclusion in their database, you might want to use the Inktomi Paid Inclusion Program. Basically, Inktomi has partnered with some submission companies that, for a fee, guarantee the Inktomi spider will crawl your submitted pages every 48 hours. This will also keep your URL in their database for a year. You won't get any boost in rankings, but you'll have peace of mind knowing your site is listed.

Submitting to AltaVista

AltaVista used to be one the quickest to index new pages. However, those days are long gone, unless you use their paid inclusion program. In order to keep the amount of spam submittals in check, AV has instituted an interesting submittal procedure. When you submit a URL to AV's free "Add-URL form," you'll see a graphical image of some strange- looking letters. You are instructed to type those letters into a box in order to continue the submission. Once you do that, you can then submit up to five URLs. I think it's a neat idea, to thwart automatic submission programs. Since I hand submit anyway, it's not a problem for me. As long as it means that my submissions get added in a relatively short time period, I'm all for it!

To see if your pages are in AltaVista, type in "URL:www.yoursite.com" into their search box (substituting your own domain name for yoursite.com). Your indexed pages will show up. Compare the title and description AV lists, with the Title tag and Meta description tag on your actual pages. If AV shows your old title and/or description, then they haven't yet indexed your new page. If they show your new stuff, then you're all set! Links to AltaVista's various submission programs can be viewed here.

Submitting to Lycos/FAST

Sites have been getting added fairly quickly and flawlessly as of late to Lycos. You can usually type your URL into their search box to see if it's in their database. Lycos has partnered with FAST, whose spider crawls through pages on a regular basis. Very often FAST will find your inner pages without you needing to submit them at all. Lycos/FAST also recently instituted a new pay-for-inclusion program called, InSite Select which will guarantee your pages regular 48 hour spidering. Lycos' add URL programs can be reached here.

Excite, Webcrawler and GO are Dead

Excite and its sister engine, Webcrawler, along with Disney's Go (formerly Infoseek) are all basically dead. They currently use Overture pay-per-click ad links, and have no database of their own.

Patience is a Virtue

Once you've submitted your URL to the engines, the next thing for you to do is practice the fine art of P-A-T-I-E-N-C-E!!! Unless you've paid for inclusion, your best bet is to submit and then forget about it for awhile. Wait for three months, then do a reality check of all the engines as noted above. I know that three months is probably too long for most of you to wait, but if you don't like to be driven crazy, it's a good strategy. I admit that I do check at least once a month after submission, but for the average Web site owner, waiting three months is a smart idea. After three months, you'll probably be pleasantly surprised to find that your site has been indexed by all of the search engines. And better yet…if you optimized them correctly they should be bringing in tons of highly targeted traffic!

About the Author
Contact Jill Whalen by e-mail at stickysauce@highrankings.com.

Jill Whalen of High Rankings is an internationally recognized search engine optimization consultant and host of the free weekly High Rankings Advisor search engine marketing newsletter.

She specializes in search engine optimization, SEO consultations and seminars. Jill's handbook, "The Nitty-gritty of Writing for the Search Engines" teaches business owners how and where to place relevant keyword phrases on their Web sites so that they make sense to users and gain high rankings in the major search engines.