RE: Web spiders

Jason Levine's List Subscription ( (no email) )
Sat, 28 Mar 98 10:44:15 -0500


Semi-representing webmasters everywhere (and in frustration due to the large
number of robots that don't adhere to it), please make sure that the
robot/spider that you choose adheres to the robot exclusion standard so that
it doesn't index or search any pages that the webmaster has deemed off-limits:

If it doesn't, there are a lot of webmasters (us included) that just tend to
block access for the entire offending network; that's probably NOT in your
client's best interest.


>I have a client that wants perform a daily search over the internet or at
>least over the main search engines of each country with a main office.
>I need that the search show me only the last modifications or aditions,
>something like a news clipping of the company.
>Can someone help me?