[Dshield] Trend analysis at DShield
Johannes B. Ullrich
jullrich at sans.org
Tue Jun 15 14:43:00 GMT 2004
On Tue, 2004-06-15 at 07:06, Pete Cap wrote:
> Does anyone remember when the Trends page at DShield used to include
> an explanation of how the trends are calculated?
> It still has a quick blurb in there about what the values mean and it
> still says "For more details, see the end of this page." But since
> the re-org the explanation of the maths involved seems to have vanished.
> Hoping it will be possible to get that back--I want to implement
> something similar here but I don't want to reinvent the wheel...
Well, the reason I removed the blurb was that I am reinventing the
wheel ;-). Trying out different ways to get a better trend number.
The basic idea was (and will likely still be):
(1) calculate the average number of reports per day for
a particular port.
(2) divide this number by the total number of reports
received. This is important, as we try to compare this
number to recent reports, and we may not have all submissions
in yet for the current day.
(3) divide the number of reports received for this port today
by the total number of reports received for today.
So "in math" you got:
C = total number of reports
Cp = total number of reports for a particular port
Cpd = number of reports for a particular port during the
Cd = number of all reports for a particular day
Dp = Average reports per day for a port.
Rp = Recent (last day) reports per day for port
ok. to get a trend, we could just divide these two
T=Rp/Dp... if its >1, we got an increase, if its
<1 we got a decrease.
However, if we take the log of this, we got it <0
for a decrease and >0 for an increase
T=ln(Rp/Dp) is what I used to calculate the trend.
Next, you would like to know if the trend is
significant. So you need to figure out the error.
I used standard counting statistic (Poison) for
that. The std error on our count is the square
root of the count. Then you plug it into standard
Terr= sqrt( 1 / (C*Cp*Cpd*Cd) )
(or something close to this... proof left as an exercise.. but
essentially, the more reports, the smaller the error ;-) )
CTO SANS Internet Storm Center http://isc.sans.org
phone: (617) 837 2807 jullrich at sans.org
contact details: http://johannes.homepc.org/contact.htm
More information about the list