[Dshield] Event submission frequency
jullrich at sans.org
Tue Jun 4 21:22:08 GMT 2002
We try to get to hourly processing. With the addition of some of the
reports for the internet storm center pages, we are now at about 4hrs.
With some tweaks I hope to get to 2hrs shortly.
I suggest to submit logs hourly or maybe 3-4 times a day, but not less
once a day.
While we accept logs up to 30 days old, they are mostly of 'historic'
> If so, are mailed logs more or less burdensome than automated HTTP
> form submissions?
I am working on a simple http interface to add to the e-mail interface.
lot of people are asking for it. However, e-mail is still the prefered
and http submissions will not be added to the database any faster. They
will just be added to the same queue. There is an http submission form,
it is not very suitable for scripted use.
> (the latter seem preferable to me, as they're a lot closer to a SOAP
> or XML-RPC model
yuck... I know XML is "the way to go". I just have yet to find a reason
add all the overhead.
Some of the reasoning behind the current system:
e-mail (or better the SMTP protocol) is a great way to queue and
distribute loads. It includes automatic fail over and is much more
robust 'out of the box' than http. Sure, you can do all this with
http. But not in a 10 line bash script (1st clients).
Initially, I looked at different methods to submit data. There is a
nice IETF draft that allows for the exchange of IDS logs via XML
messages. However, like so many XML standards, it suffers from the
problem that it is way too flexible, making parsing a pain.
The tab delimited format we have right now can be read straight into the
database and all we have to do is validate the lines with a regular
Another decision we made right from the start was to queue instead of
allowing 'instant inserts'. This is mostly based on experience from
e-commerce sites that went down if they got 'attacked' by google.
Basic rule: Try not to allow the end user to trigger database
inserts. Also, back when we started, innodb tables where still
considered experimental and mysql (which we still use) did not have
row locking. I just did a test a couple weeks back and switched
the main database table to innodb on a new idle system. The main
processing took about 3 times longer! You just can't beat MyISAM
tables for raw search speed. But this table type does not allow
us to do user-triggered inserts efficiently, so everything has to
Just for the hardware/system junkies: Our main database is currently
running on a dual PIII 1 GHz system with 2 Gig RAM. We are soon (this
week?) switching to a faster system. The main improvement is a 64bit
PCI bus and better RAID card. In my tests, the new system is about
twice as fast as the old system (other than that: dual PIII 1GHz,
2 Gig RAM, 4 x 73 GByte Seagate 10k drives)
ok. this email got a bit longer than required to answer the question.
Recommendation: Submit every 1-4 hours. Not less than once a day.
For 'instant' submissions: try to keep the content larger than the
jullrich at sans.org Collaborative Intrusion Detection
More information about the list