[Dshield] Mangled traffic and its effects on IDS performance
pmelson at gmail.com
Wed Apr 11 14:03:19 GMT 2007
> I'm currently attempting to troubleshoot an IDS issue involving dropped
packets. The sensor is
> currently dropping around 60% of the traffic it's observing. The CPU and
memory are getting pegged and
> it's trying to monitor about 60-70k TCP "conversations" at a time.
What product are we talking about here? Are you certain that the problem is
not the IDS?
> Packet analysis with wireshark shows that about 1/3 of the packets, at any
given time, are dupes,
> transmitted out-of-order, are truncated, etc. Basically, it's pretty
mangled. If you look at
> transmission times, about half the traffic is fine, and about 1/3 is very,
very slow. So, in
> statistical terms, it's like there are two distributions.
A couple things that could cause this: hardware/firmware failure in the
switch, VLAN routing misconfiguration, a failing NIC, a host NIC that is in
promiscuous mode and the host has routing issues, and my personal favorite -
If the problem is not the switch or switch routing config, then just looking
at packet counters by port might give you some idea of the culprit.
> However, a coworker has suggested that 30% mangled traffic is normal for
any enterprise and should not
> choke the IDS--so the solution is simply to add more appliances and
attempt some kind of load-balancing.
Way to set the bar low. I would disagree with your coworker both in his
belief that the traffic is 'normal' and that adding more hardware will fix
the problem, since the throughput is less than 1Mbps.
> Alternately, if someone can give me guidance on recreating this in a lab
environment, then that would be > good as well--I can test our current
solution, another vendor's, and the old standby snort. Any ideas?
This should be easy enough - use tcpdump, wireshark, or whatever to capture
the same traffic the IDS is seeing. Then use tcpreplay** to put that
traffic back on the wire with the IDS and observe. This isn't perfect - and
I suspect won't yield the same results because what wireshark isn't showing
you is bad frames that the IDS is probably still trying to normalize. This
would be a good time to bring the vendor into the loop as well. Use your
support contract to offload some of the product config work on them.
* A patch cable in a chemistry lab once took down an entire campus because
the copper oxidized like an old car battery terminal (something in the air?)
and caused it to flood the switch with bad frames.
More information about the list