ed.truitt at etee2k.net
Mon Feb 20 12:25:39 GMT 2006
Maybe the idea was to read robots.txt to find out what the site owner didn't want indexed - then treat *that* as the highest priority stuff to grab...
From: Valdis.Kletnieks at vt.edu
Date: Sun, 19 Feb 2006 20:40:04
To:General DShield Discussion List <list at lists.dshield.org>
Subject: Re: [Dshield] Test
On Sun, 19 Feb 2006 15:44:43 EST, "George A. Theall" said:
> Well, the fact that it checks for a robots.txt file and ignores
> restrictions found there has caused me to ban it from web sites I
Whatever happened to malware writer's pride in workmanship? Fetching a
robots.txt and obeying what it says is one thing. Totally failing to bother
looking for a robots.txt that you intend to ignore anyhow, to optimize your data
raiding, is another.
But going to all the trouble of fetching it to ignore it is just square-wheel.
"Nothing screams poor worksmanship quite like wrinkles in the duct tape".
Learn about Intrusion Detection in Depth from the comfort of your own couch:
send all posts to list at lists.dshield.org
To change your subscription options (or unsubscribe), see: http://www.dshield.org/mailman/listinfo/list
-E D Truitt
Sent via my BlackBerry from Cingular Wireless
More information about the list