[Dshield] Database Cleanup

Johannes B. Ullrich euclidian at euclidian.com
Wed Mar 14 22:15:16 GMT 2001


  I started removing some of the older records from the database. These
old reocrds (currentlyeverything before Feb. 1st 2001) will be made
available in a separate, slower search. I mainly do this to avoid problems
as we receive more and more reports (we where at 3+ Million rows in our
main report table). 

  The plan is to cut down the 'live' table to 1 Month worth of data and
maybe one week of 'pings' (ICMP) data. This should keep it below 1 Million
rows. Overall, the database is holding up pretty good lately, considering
it runs on a K6-3 450 MHz machine. I continue to tune it to get the best
possible performance out of it.

  To respond to the data integrity issues raised latelty I stated watching
incoming reports more closely and I will notify users if they appear to
submit bad records. These records will not be deleted however (exepctions:
obvious problems like source ip = target ip, source ip in LAN range (e.g.
10.x, 192.168.x , known port scna services ...).





More information about the list mailing list