[unisog] HTTP Session Reconstruction and Monitoring

Peter Van Epp vanepp at sfu.ca
Sat Dec 11 06:20:12 GMT 2004

On Fri, Dec 10, 2004 at 02:23:20PM -0700, Jacob Roberts wrote:
> We would like to improve our ability to monitor inappropriate web
> surfing activity on our public access workstations.
> We have 3 basic requirements. The system can:
> 1. Handle our large amount of traffic
> 2. Reconstruct HTTP sessions (e.g. an analyst can retrieve a view of the
> visited web site based on captured packet data)
> 3. Configure rules for specific traffic matching.
> Does anyone know of any enterprise level applications that can do these
> things.  We'd prefer an Open-source solution.
> We are currently testing Steel-Cloud by Computer Associates.  It meets
> reqs 2 and 3 but fails to meet our needs for req 1.
> Thanks,
> Jake Roberts
> Brigham Young University
> _______________________________________________
> unisog mailing list
> unisog at lists.sans.org
> http://www.dshield.org/mailman/listinfo/unisog

	While I don't know of an application that will do what you want out of
the box, assuming that you can get the required traffic to disk via tcpdump
(and appropriate filters if required), which may of course not be a trivial
task depending how fast your link is :-), then an open source application 
called tcpflow will reconstruct session data from tcpdump input (the filtering
for 3) would need to be tcpdump filters and you would probably need some perl
or such like to mung the tcpstreams in to appropriate html to ship at a browser
again to get it to display for your analyst. The etherial network sniffing 
package is also reputed to be able to recover tcpstreams as well, although I
expect for what you want tcpflows file output will be more useful.
	Since it looked like item 1 was a problem for the solution that you 
found, then, assuming you aren't Linux adverse, there is a kernel mod 
(ringbuffer) from www.ntop.org that may help. It basically short circuits the 
entire tcp stack and mmaps the input buffer from the adapter in to a modified
copy of libpcap. Assuming that solution can read from a disk file of the 
transaction that may mean problem solved with a tcpdump that can get the data
to disk at wire speed and then feeding it to the CA application.
	On my argus sensor box (which is only using the first 128 bytes of the 
packet though, and you will need all of it presumably) I can keep up with a 
jumbo frame transfer at ~950 megabits per second on a gig link without apparant
packet loss (before this the same machine lost %50 of the traffic even at only
128 bit slice length). Unfortunatly the jumbo frames are the best case option 
and the more typical web mix of smaller packets may still induce packet loss 
from the author's testing (as may trying to go to disk without a good 
multispindle raid controller!) depending on your link speed. 
	There is a fair bit of tinkering required here (and possibly a lot of 
high speed data capture knowhow required :-)) but it may be possible and 
possibility increases as link speed reduces and thus the strain of the capture 
reduces as well.
	I'm also presuming that you have considered the privacy and legal 
implications of doing this in your juristiction. Depending on whether the user
has a reasonable expectation of privacy (which may be negated by signon 
banners or messages on the wall, but you likely need legal advise by a lawyer
familiar with your juristiction's laws) this may be considered wiretapping and 
thus illegal.

Peter Van Epp / Operations and Technical Support 
Simon Fraser University, Burnaby, B.C. Canada

More information about the unisog mailing list