[Dshield] db status update
Johannes B. Ullrich
jullrich at sans.org
Fri Jun 10 10:19:01 GMT 2005
First of all sorry for not commenting on this earlier. However, I was
traveling for a week with limited Internet access.
The quick rundown on what happened:
The main parser script choked on one of the logs it parsed, and exited
without deleting its lock file. These lock files prevent multiple parser
scripts from running at the same time. As a result, no parser script was
started during the next scheduled start.
Usually, the parser script runs until there are no more unparsed logs.
Every 15 minutes, a cron job checks if there are new logs, and starts
the parser script (unless one is already running).
Now with this stale lock file around, no logs got parsed. Eventually,
the disk that holds the inbound logs filled up. Usually, I would have
gotten plenty alerts. However, being g on a plane and without Internet
access for a couple days didn't give me a chance to respond.
By now, logs are parsed again as usual. There are still some old logs
that ended up on an overflow system while the disk was full. I need to
copy them over manually.
In order to prevent this from happening again, I setup a process to
automatically delete these stale lock files.
Once the locks where removed and parsing started again, it still took a
day for reports and such to get back to normal. The system was running
in 'catch up mode'. Whenever the incoming queue exceeds a certain size,
some of the reports are no longer updated in order to allow for a faster
import of the data.
Johannes Ullrich jullrich at sans.org
Chief Research Officer (617) 639 5000
PGP Key: https://secure.dshield.org/PGPKEYS
"We use [isc.sans.org] every day to keep on top of
security at our bank" Matt, Network Administrator.
More information about the list