[Dshield] db status update

Johannes B. Ullrich jullrich at sans.org
Fri Jun 10 10:19:01 GMT 2005

First of all sorry for not commenting on this earlier. However, I was 
traveling for a week with limited Internet access.

The quick rundown on what happened:

The main parser script choked on one of the logs it parsed, and exited 
without deleting its lock file. These lock files prevent multiple parser 
scripts from running at the same time. As a result, no parser script was 
started during the next scheduled start.

Usually, the parser script runs until there are no more unparsed logs. 
Every 15 minutes, a cron job checks if there are new logs, and starts 
the parser script (unless one is already running).

Now with this stale lock file around, no logs got parsed. Eventually, 
the disk that holds the inbound logs filled up. Usually, I would have 
gotten plenty alerts. However, being g on a plane and without Internet 
access for a couple days didn't give me a chance to respond.

By now, logs are parsed again as usual. There are still some old logs 
that ended up on an overflow system while the disk was full. I need to 
copy them over manually.

In order to prevent this from happening again, I setup a process to 
automatically delete these stale lock files.

Once the locks where removed and parsing started again, it still took a 
day for reports and such to get back to normal. The system was running 
in 'catch up mode'. Whenever the incoming queue exceeds a certain size, 
some of the reports are no longer updated in order to allow for a faster 
import of the data.

Johannes Ullrich                        jullrich at sans.org
Chief Research Officer                     (617) 639 5000
PGP Key: https://secure.dshield.org/PGPKEYS

"We use [isc.sans.org] every day to keep on top of
  security at our bank" Matt, Network Administrator.

More information about the list mailing list