Introduction
Banning spiders and agents
Search engine spider identification
Further learning resources:
The following is a basic listing of search engine spider names and their "owners". This is by no means complete, as there are many thousands of search engines on the Internet, but it covers the more common beneficial spiders. Look for these in your traffic reports or search for the names through your server logs to discover which pages they have been spidering. You'll find that many of the entries will also have accompanying numbers or letters e.g Googlebot/2.1 or Slurp.so/1.0
NAME OWNER
Googlebot Google.com
TeomaAgent Teoma.com
Zyborg Wisenut.com
Gulliver NorthernLight.com
Architext spider Excite.com
FAST-WebCrawler FAST (AllTheWeb.com)
Slurp Inktomi.com
Ask Jeeves AskJeeves.com
ia_archiver Alexa.com
Scooter AltaVista.com
Mercator AltaVista.com
crawler@fast FAST (AllTheWeb.com)
Crawler Crawler.de
InfoSeek sidewinder InfoSeek.com
Lycos_Spider_(T-Rex) Lycos.com
Fluffy the Spider SearchHippo.com
Ultraseek InfoSeek.com
MantraAgent LookSmart.com
Moget Goo.jp
T-H-U-N-D-E-R-S-T-O-N-E Thunderstone.com
MuscatFerret Euroferret.com
VoilaBot Voila.fr
Sleek Spider Search-info.com
KIT_Fireball FireBall.de
WebCrawler Webcrawler.com
If you have spotted any significant activity from these spiders in your reports or logs, there's a good chance that you'll be listed on that particular search engine. But you'll need to be patient; some Search Engines take up to 6 months to refresh their databases!
Michael Bloch
www.tamingthebeast.netMichael Bloch
Taming the Beast.net
http://www.tamingthebeast.ne...
Tutorials, web content, tools and software.
Web Marketing, eCommerce & Development solutions.
_____________________________________________
Copyright information.... This article is free for reproduction but must be reproduced in its entirety & this copyright statement must be included. Visit http://www.tamingthebeast.ne... to view great articles, tutorials and tools for site owners, web developers and Internet marketers! Subscribe for free to our popular ecommerce/web design ezine!