Convert HTML to text or remove HTML markup with Detagger
	Convert your text files into web pages (like this one was) Are you using your clipboard to it's fullest potential?

---

Search engine robots that visit your web site

Contents of this page

Search engine robots and others
Browsers
Link Checkers, Link monitors and bookmark managers
Validators
FTP clients and download managers
Research projects
Software packages
Offline browsers and other agents
Other miscellaneous agents
Sites that regularly visit
Other useful sites
...And finally, some fakers
Awards for this page

Search engines and other sites send robots to read and index your pages. This page reverses that process and indexes the robots. This information has been gleaned by looking at the server logs for www.jafsoft.com. You can read a detailed description of how we hunt spiders

Whenever a page is read from a web site, the log file records a number of details including the time, the IP address and usually the referrer page and the user agent. You can see this in our analysis of a server log sample.

Unlike many pages that list web robots, this page actually tries to go visit the robots themselves. Where possible links are provided to the robots home pages, and descriptions are given of what they're up to. This page is updated regularly as more information is found (the last update was on 30-Jan-2006).

Well behaved robots will identify themselves, often supplying web or email addresses you can contact. In any case, the pattern of pages being read and the IP addresses being used soon sorts the men from the robots.

Good robots will read robots.txt to see what your site policy is, but there are other ways of spotting robots. In addition to the search engine robots, other "user agents" will visit your site, e.g. to validate links to your site from other people's pages. Often these will just access the HEAD of the file, rather than doing a GET on the whole file.

You can also visit our page describing the engines in some detail.

This page is regularly converted from this text file by the author's own text to HTML converter AscToHTM. The last update was on 30-Jan-2006. This software is available as shareware (cost $30)

Search engine robots and others

The following table lists the search engines that spider the web, the IP addresses that they use, and the robot names they send out to visit your site. Version numbers are usually included in the robot names, but are omitted here except where it implies a visit from a different IP address or (as in inktomi) a different search engine.

Often multiple IP addresses are used, in which case we just give a flavour of the names or numbers. Inktomi is a company that offers search engine technology and is used by a number of sites (e.g. www.snap.com and www.hotbot.com)

Wherever <nn> appears this indicates a number of different digits may be used.

Browsers

Most browsers identify themselves with a string that begins "Mozilla...". I've chosen not to document those (as yet). Here are a few of the rarer browser identifiers that I've seen.

Link Checkers, Link monitors and bookmark managers

Link checkers and bookmark managers are run by people wanting to keep their pages and bookmarks up to date. Being visited by a link checker is good news as it means that someone has linked to you, and cares that you're still alive. Link monitors regularly check your pages for changes, usually because someone has selected your page as "one to watch".

(pause for warm glow :-)

If you have access to the server log, check the referrer page to try and get the URL from which you are linked. Sometimes these URLs are inside password protected parts of sites, so you won't be able to view the page.

If you build up a list of sites that link to you, these are the guys you should tell when you move (moral - never move)

It's also quite common for the Link checker to give no indication of which URL it's coming from. Some link checkers always come from the same IP address, more usually they come from the client's site. It depends on whether the site owner has purchased a copy of the link checking software, or signed up to some centralized link checking service. If you get the client's IP address you can always try visiting that if they blank the referrer URL field, and surfing their site.

Some of these tools appear to imply they're extracting email addresses (e.g. emailSiphon). As such they're probably unwelcome visitors since these addresses are probably being collected for spammers.

A page listing various link checkers (and other tools) can be found at www.softwareqatest.com/qatweb1.html#LINK

Validators

Validators check your web pages for HTML correctness and standards compliance. Since other people are unlikely to send a validator to your site, you don't usually see much of this. Consequently the "list" below is restricted to the on-line validators I've used myself.

However if you choose to validate your own site, then the validation attempts will appear in your logs. The following list is thus limited to the on-line validator I use (and recommend) and a URL submission service that I use.

FTP clients and download managers

If you offer files for download, then you'll start to be visited by various FTP clients. Clients like Go!Zilla and GetRight are smart in that they can resume downloads that have been interrupted. This relies on your web server supporting the necessary protocol, but that's fairly standard these days.

If your download files are over 1Mb in size (or if your server is slow), you'll often see the same IP address make multiple partial downloads of your file (look at the file size). In the case of Clients line Go!Zilla and GetRight if these add up to the right number of bytes, then chances are the download succeeded.

Research projects

These agents come from research projects. Of course that's how Google started...

Software packages

These agents are the default identifiers for various software packages. Software developers uses these packages to add Internet functionality to their own applications. As such it's impossible to say without looking at the pattern of access what these agents are being used for as the same agent name may be used by different developers fo achieve differemt results.

While many of these packages allow you to change the user agent, some do not, and many developers are too lazy to change the agent string.

Offline browsers and other agents

Other miscellaneous agents

These agents are ones that we've seen, but been unable to get information for, or which are slightly unusual in origin. If you have any additional information on any of these, feel free to send it to info@jafsoft.com

Sites that regularly visit

Some IP addresses, or sites may regularly visit you, although the user agent may be obscure, blank, or even change.

Here are a few that I've been able to work out

Other useful sites

Here are links to other sites you might find useful when looking into web robots

...And finally, some fakers

Increasingly security and privacy concerns mean that users and companies are wary about giving away information to sites they visit through the user agent and other fields that appear in server logs.

Some browsers will allow you to select the user agent you present when visiting a site. The Opera browser does this, for example, to allow it's users to pretend to be either IE or Netscape when visiting web sites coded in a way that forgets there are other browsers in use.

Also as firewalls become more common, we will see more and more user agent fields beling blocked by the firewall, that will prevent this information being transmitted to the outside world.

Just to prove that you can never rely on the user agent, here is a selection of user agent strings I've seen in my log files that tell us nothing about the software being used (although some of them speak volumes about the person driving the software). I'm omitting any IP addresses I may have to protect the identities of those concerned :-)

Awards for this page

I've been told this page is referenced in the book Spidering Hacks

All awards gratefully received :-)

This page is © 2000-2005 John A Fotheringham. It may not be reproduced without permission,
although you are welcome to save a copy for personal use to your hard disk.

home - search engines - contact us - news - product index - search this site Affiliated sites: Starmount - suppliers of CD/DVD duplicators For more information contact info@jafsoft.com.