[Dshield] Syslog Server Software

Valdis.Kletnieks@vt.edu Valdis.Kletnieks at vt.edu
Thu Mar 16 19:02:10 GMT 2006


On Thu, 16 Mar 2006 09:16:21 EST, "Jon R. Kibler" said:

> Most versions of syslog that I am familiar with do rate limited logging of non-unique messages or messages from a given source... thus, working around the above problem for the most part. 

So make that script say:
for i in `seq 1 10000`; do logger -d -p local7.info "Yo! D00dz! This is $i of 10000"; done;

(So much for rate-limiting of non-unique traffic.. ;)

And this actually happens in production - I'm currently chasing a Linux kernel
bug that causes the audit subsystem to generate some 10 megabytes/second of
broken messages.  I made the mistake of sending it to syslog instead of auditd
during testing - and the kernel printk timer was enabled, so each message had
a subtly different timestamp.

Did you know that the Fedora syslogd likes to fsync() a lot?  It takes a *long*
time to log 10 meg of messages with an fsync() after each one.. (literally - I
finally rebooted after 45 minutes of total oinkage...)

Or be more nefarious - use a C program that generates the packets directly,
spoofing the addresses, and does something like spew out a bunch of faked
sendmail messages that make it *look* like some spammer is doing a dictionary
run.

And log it to auth.notice rather than mail.notice - *that* will keep the
analysts busy for a while. ;)

> Also, someone would have to had compromised a system first to be able to run
> logger -- meaning that the 'interesting' information would have already been
> logged. If the attacker was internal, and a legit user of the system, the fact
> that they generated so much garbage would be an immediate red flag.

One word (acronym, really) - UDP.

Unless you're doing proper ingress filtering (and a lot of sites don't), that
UDP packet on port 514 could have come from *anywhere*.  And even if your syslog
server has an iptables or similar firewall with punch-outs for each individual
machine, you still don't have a really high level of confidence where a packet
came from.

If I 0wn a PC on your net, I can generate the spewage to look like the machine
I'm about to attack, cause your syslog to burp, then run the attack while it's
burping, and then the 'interesting' info is *not* logged.

Remember - the intent here is *not* to be totally invisible, merely to hide
exactly what happened.  If I can then nuke the logs that are local to the machine,
you're suddenly left with no good evidence.  "Something overflowed our logs,
and then we missed something, but we're not sure what it was".  A defense lawyer
will have a field day in court with that. ;)

You have a lot better confidence if you install syslog-ng on everything and use
TCP transport instead of UDP, but that may not be an option.

> I never said the idea was perfect -- any security can be defeated, especially
> with inside knowledge -- but, I believe what I recommend is far better than the
> default.

I'm merely adding caveats and additional info.  Yes, any security *can* be beaten.
And if you're the defender, you're much better off if you know beforehand what
the weak spots are.

Exercise for the reader: Knowing that your syslog is subject to flooding with
plausible but bogus packets, what defense(s) can you think of to mitigate the
problem?  Which ones are cost-effective, and which aren't?

And the flooding issue may not even be malicious - we've had one mailserver
that flooded our syslog server several times at 800 msgs/second when it
got confused and STARTTLS broke, and one Snort install that generated alerts
when it saw a syslog packet - and it alerted to another machine via syslog.
And I already mentioned the kernel bug....

> Finally, I should add that in my experience, I have seen logs (and entire
> systems) wiped, but the central hard copy log has almost always contained
> enough information to figure out what happened. 

Out of curiosity, how many times have you seen the attacker actually realize
you have a central log server, and gone after it to clean up after themselves?

(Personally, I like the "phantom log server" approach, where you syslog to an
IP address that blackholes because there isn't a system there, and then use
a packet sniffer on a read-only tap to suck in the packets.)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 228 bytes
Desc: not available
Url : http://www.dshield.org/pipermail/list/attachments/20060316/502507fb/attachment.bin


More information about the list mailing list