[Dshield] What's the Deal with "oBot?"

George Theall theall at tifaware.com
Thu Jan 13 17:39:31 GMT 2005


On Thu, Jan 13, 2005 at 12:06:45PM -0500, David Cary Hart wrote:

> This European spider doesn't seem to obey "disallow" properly. Is this
> nefarious or simply poor programming?

No, it ignores robots.txt on purpose. 

From what I could determine, obot is belongs to what used to be Cobion,
a German company acquired last year by ISS.  Among other things, they
sold global brand and online copyright protection services, which
explains why they'd prefer to scour entire sites rather than respect
settings in a robots.txt file. 

Before I blocked them from my sites, I noted that it was only interested
in binaries (pictures, zips, etc) and that it apparently had a bug that
led to problems following links. 

A similar 'bot you might *not* have noticed belongs to Cyveillance,
which snoops out illegal material on the web.  The 'bot operates from
addresses in NET-63-148-99-224-1 (63.148.99.224 - 63.148.99.255),
doesn't bother requesting robots.txt, and claims to be "Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 5.0". 

There are plenty of others like these too.

George
-- 
theall at tifaware.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://www.dshield.org/pipermail/list/attachments/20050113/22ca7824/attachment.bin


More information about the list mailing list