Are you using your clipboard to it's fullest potential?

Search engine robots that visit your web site

Contents of this page

Search engine robots and others
Link Checkers, Link monitors and bookmark managers
Validators
FTP clients and download managers
Browsers
Offline browsers and other agents
Other miscellaneous agents
Sites that regularly visit
Awards for this page

Search engines and other sites send robots to read and index your pages. This page reverses that process and indexes the robots. This information has been gleaned by looking at the server logs for www.jafsoft.com. Whenever a page is
read from a web site, the log file records a number of details including the time, the IP address and usually the referrer page and the user agent. You can see this in our analysis of a server log sample.

Unlike many pages that list web robots, this page actually tries to go visit the robots themselves. Where possible links are provided to the robots home pages, and descriptions are given of what they're up to. This page is updated regularly as more information is found (the last update was on 5-Jul-2001).

Well behaved robots will identify themselves, often supplying web or email addresses you can contact. In any case, the pattern of pages being read and the IP addresses being used soon sorts the men from the robots.

Good robots will read robots.txt to see what your site policy is, but there are other ways of spotting robots. In addition to the search engine robots, other "user agents" will visit your site, e.g. to validate links to your site from other people's pages. Often these will just access the HEAD of the file, rather than doing a GET on the whole file.

You can also visit our page describing the engines in some detail.

This page is regularly converted from this text file by the author's own text to HTML converter AscToHTM. The last update was on 5-Jul-2001. This software is available as shareware (cost $40)

Search engine robots and others

The following table lists the search engines that spider the web, the IP addresses that they use, and the robot names they send out to visit your site. Version numbers are usually included in the robot names, but are omitted here except where it implies a visit from a different IP address or (as in inktomi) a different search engine.

Often multiple IP addresses are used, in which case we just give a flavour of the names or numbers. Inktomi is a company that offers search engine technology and is used by a number of sites (e.g. www.snap.com and www.hotbot.com)

Wherever <nn> appears this indicates a number of different digits may be used.

Link Checkers, Link monitors and bookmark managers

Link checkers and bookmark managers are run by people wanting to keep their pages and bookmarks up to date. Being visited by a link checker is good news as it means that someone has linked to you, and cares that you're still alive. Link monitors regularly check your pages for changes, usually because someone has selected your page as "one to watch".

(pause for warm glow :-)

If you have access to the server log, check the referrer page to try and get the URL from which you are linked. Sometimes these URLs are inside password protected parts of sites, so you won't be able to view the page.

If you build up a list of sites that link to you, these are the guys you should tell when you move (moral - never move)

It's also quite common for the Link checker to give no indication of which URL it's coming from. Some link checkers always come from the same IP address, more usually they come from the client's site. It depends on whether the site owner has purchased a copy of the link checking software, or signed up to some centralized link checking service. If you get the client's IP address you can always try visiting that if they blank the referrer URL field, and surfing their site.

Some of these tools appear to imply they're extracting email addresses (e.g. emailSiphon). As such they're probably unwelcome visitors since these addresses are probably being collected for spammers. You can read more about this at www.csc.ncsu.edu/~brabec/antispam.html

A page listing various link checkers (and other tools) can be found at www.softwareqatest.com/qatweb1.html#LINK

Validators

Validators check your web pages for HTML correctness and standards compliance. Since other people are unlikely to send a validator to your site, you don't usually see much of this. Consequently the "list" below is restricted to the on-line validators I've used myself.

However if you choose to validate your own site, then the validation attempts will appear in your logs. The following list is thus limited to the on-line validator I use (and recommend) and a URL submission service that I use.

FTP clients and download managers

If you offer files for download, then you'll start to be visited by various FTP clients. Clients like Go!Zilla and GetRight are smart in that they can resume downloads that have been interrupted. This relies on your web server supporting the necessary protocol, but that's fairly standard these days.

If your download files are over 1Mb in size (or if your server is slow), you'll often see the same IP address make multiple partial downloads of your file (look at the file size). In the case of Clients line Go!Zilla and GetRight if these add up to the right number of bytes, then chances are the download succeeded.

Browsers

Most browsers identify themselves with a string that begins "Mozilla...". I've chosen not to document those (as yet). Here are a few of the rarer browser identifiers that I've seen.

Offline browsers and other agents

Other miscellaneous agents

These agents are ones that we've seen, but been unable to get information for, or which are slightly unusual in origin. If you have any additional information on any of these, feel free to send it to search@jafsoft.com

Sites that regularly visit

Some IP addresses, or sites may regularly visit you, although the user agent may be obscure, or even change.

Here are a few that I've been able to work out

Awards for this page

All awards gratefully received :-)

This page is © 2000-2001 John A Fotheringham. It may not be reproduced without permission,
although you are welcome to save a copy for personal use to your hard disk.

home - search engines - contact us - news - product index - search this site For more information contact search@jafsoft.com.