[Dshield] What is this?

John Sage jsage at finchhaven.com
Sat Jul 13 14:24:46 GMT 2002


On Tue, Jul 02, 2002 at 06:25:52PM +0300, Antti Tolamo wrote:
> I've been getting these alteast two days, from same source.
> Somebody going through my servers Apache document  webpages.
> This is only the part of files looked.
> 
> Yes, I do have the Apache's own web pages as a default web page
> on root source: What's a 'Scooter' ? A robot?

You might want to read up on creating a robots.txt file in your web
server's root directory, and on adding a robots META tag to your web
pages.

Many of the legitimate spiders actually honor such a thing.

See:

http://www.robotstxt.org/wc/robots.html

and specifically:

http://www.robotstxt.org/wc/exclusion.html

"The Robots Exclusion Protocol:

A Web site administrator can indicate which parts of the site
should not be vistsed by a robot, by providing a specially formatted
file on their site, in http://.../robots.txt.

The Robots META tag:

A Web author can indicate if a page may or may not be indexed,
or analysed for links, through the use of a special HTML META tag."

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">



- John
-- 
"Obviously, we do not want to leave zombies around."

PGP key      http://www.finchhaven.com/pages/gpg_pubkey.html
Fingerprint  FE 97 0C 57 08 43 F3 EB 49 A1 0C D0 8E 0C D0 BE C8 38 CC B5 




More information about the list mailing list