[Dshield] Test

Ed Truitt ed.truitt at etee2k.net
Mon Feb 20 12:25:39 GMT 2006


Maybe the idea was to read robots.txt to find out what the site owner didn't want indexed - then treat *that* as the highest priority stuff to grab...

-EdTr.
-----Original Message-----
From: Valdis.Kletnieks at vt.edu
Date: Sun, 19 Feb 2006 20:40:04 
To:General DShield Discussion List <list at lists.dshield.org>
Subject: Re: [Dshield] Test

On Sun, 19 Feb 2006 15:44:43 EST, "George A. Theall" said:
> Well, the fact that it checks for a robots.txt file and ignores
> restrictions found there has caused me to ban it from web sites I
> control.

Whatever happened to malware writer's pride in workmanship?  Fetching a
robots.txt and obeying what it says is one thing.  Totally failing to bother
looking for a robots.txt that you intend to ignore anyhow, to optimize your data
raiding, is another.

But going to all the trouble of fetching it to ignore it is just square-wheel.

"Nothing screams poor worksmanship quite like wrinkles in the duct tape".  

_________________________________________
Learn about Intrusion Detection in Depth from the comfort of your own couch:
https://www.sans.org/athome/details.php?id=1341&d=1

_______________________________________________
send all posts to list at lists.dshield.org
To change your subscription options (or unsubscribe), see: http://www.dshield.org/mailman/listinfo/list


Cheers,
-E D Truitt

Sent via my BlackBerry from Cingular Wireless


More information about the list mailing list