[Dshield] What is this?
jsage at finchhaven.com
Sat Jul 13 14:24:46 GMT 2002
On Tue, Jul 02, 2002 at 06:25:52PM +0300, Antti Tolamo wrote:
> I've been getting these alteast two days, from same source.
> Somebody going through my servers Apache document webpages.
> This is only the part of files looked.
> Yes, I do have the Apache's own web pages as a default web page
> on root source: What's a 'Scooter' ? A robot?
You might want to read up on creating a robots.txt file in your web
server's root directory, and on adding a robots META tag to your web
Many of the legitimate spiders actually honor such a thing.
"The Robots Exclusion Protocol:
A Web site administrator can indicate which parts of the site
should not be vistsed by a robot, by providing a specially formatted
file on their site, in http://.../robots.txt.
The Robots META tag:
A Web author can indicate if a page may or may not be indexed,
or analysed for links, through the use of a special HTML META tag."
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
"Obviously, we do not want to leave zombies around."
PGP key http://www.finchhaven.com/pages/gpg_pubkey.html
Fingerprint FE 97 0C 57 08 43 F3 EB 49 A1 0C D0 8E 0C D0 BE C8 38 CC B5
More information about the list