Re: robots.txt

Jeff Woods ( )
Fri, 02 May 1997 09:40:16 -0400

At 11:55 PM 5/1/97 -0500, you wrote:

>I have noticed in my web logs, about every minute or so an entry like this:
>, -, 4/30/97, 0:00:53, W3SVC, TESLA,, 0, 18, 0, 200, 0,
HEAD, /robots.txt, -,

You are using WHATSUP by IPSWITCH to monitor your Web Server. This is the
file that WHATSUP looks for. Whether it finds it or not, if it gets a
mesage back from the web server, the web server is up.

>I know that the robots.txt file is used to restrict robot access to my
site, but can someone tell me what process is accessing this file on
localhost? I suspect something like MS Index Server but I'd like to find
out if that is the case and how do I make it not run every minute of the day?