Introduction
Banning spiders and agents
Search engine spider identification
Further learning resources:

The following is a basic listing of search engine spider names and their "owners". This is by no means complete, as there are many thousands of search engines on the Internet, but it covers the more common beneficial spiders. Look for these in your traffic reports or search for the names through your server logs to discover which pages they have been spidering. You'll find that many of the entries will also have accompanying numbers or letters e.g Googlebot/2.1 or Slurp.so/1.0

If you've been surfing search engine optimization web sites, you've no doubt come across the term spider, robot or bot on many occasions.

As is usually the case, this tutorial has come about as a result of a project that I got myself tangled up in...

If you've been surfing search engine optimization web sites, you've no doubt come across the term spider, robot or bot on many occasions.