[unisog] New Student Weekend Hel^H^H^H

Clarke Morledge chmorl at wm.edu
Mon Aug 25 21:27:52 GMT 2003


Thanks for the excellent write up.  We got swamped, too.  First, it was
Blaster, but Nachi/Welchia took care of that for us :-)

We were not able to get a netreg-type system in place in time to do what
you are doing, and we now wish we could have done something like that.  
But I'm sure that what we would have planned to put in would have gotten
overrun, just as you found out.

Since most of the variants of the RPC/DCOM worms spread via tftp, we were
able to block tftp on the ingress on all ports at the closet level. That
kept the spread of the worm down, but it did nothing stop new systems
(brought in by students and faculty) from causing havoc.  We have found
that it only takes about a half dozen to a dozen Nachi infected systems to
begin to degrade network performance on certain segments of our network.

We find ourselves knocking off between 20 to 70 infected systems per day
since last Wednesday (students started showing up then), sending teams out
to the dorms and offices to clean and patch.

Our measures to protect us at the Internet border worked great.  We just
got whipped by attacks from the inside.

Clarke Morledge
College of William and Mary
Information Technology - Network Engineering
Jones Hall (Room 18)
Williamsburg VA 23187
chmorl at wm.edu

On Mon, 25 Aug 2003, Phillip G Deneault wrote:

> This weekend was WPI's frosh weekend.  After having a major outbreak last 
> week RIGHT before NSO, we were still recovering and were a little paranoid 
> about this coming weekend.  Since WPI is very early in starting its school 
> year, I figured I'd post some notes and issues we ran into in getting kids 
> in the dorms and registered on the network.  
> We have an automatic registration system which assigns people a temporary 
> IP and allows them to register and get accounts on their own.  It used to 
> be a P2 300 on which sat the website and queried switches.  Between 
> regular registrations, a quick and dirty vulnerability scanner which told 
> students they needed to patch, and the viruses pounding the server...it 
> quickly melted down.  We brought up a Dell 2650(Dual P4 with gigs of 
> memory) and things were MUCH better.
> At first we had everyone in the dorms getting people scanned, patched and
> cleaned.  This worked rather well until a few hundred hosts were online
> and the registration system kept breaking down, then the procedure fell
> apart.  Once the procedures fell apart, some of the techs resorted to
> configuring people manually but forgot to patch and scan.  These students
> quickly got turned off when they got infected.
> By day 2, I was just sitting in the office turning off systems left and
> right, it was just easier.  Having someone sitting at the NOC(network 
> operations center) was also useful for coordinating support teams and 
> working with the Helpdesk in identifying systems with problems.  We are a 
> school of 5000 and we had two people doing this work.  Other schools might 
> want to scale accordingly.
> We also abandoned separate filters on the dorm switches blocking port 135
> in favor of the edge egress filters which would allow us to catch ALL the
> various infections.  The internal ingress filters were doing little to
> prevent the spread of infection anyway since the virus already existed in
> all the buildings.  It was just pushing the limits on the CPU's and making
> the registration system take extra time and fail due to lack of response.
> McAfee's Stinger tool did a much better job of finding and eliminating the 
> virus(as well as many others which we would have needed to chase later).  
> Some people were taking shortcuts by only scanning c:\%windows%\system32 
> and it worked out pretty well.
> Support teams were armed with CDs and USB keys with all required patches, 
> service packs, and anti-virus tools.  Links to patches and tools were also 
> put on the registration server so people who were scanned and determined 
> vulnerable were forced to download and install them before registering.  
> This did _not_ work with Windows 98/ME machines.  They needed to manually 
> register with a network guy(who knew the right URL to go to) and would be 
> patched via Windows Update as soon as possible.  Since most of the 
> Blaster/variants don't infect anything less than Windows 2000, this was an 
> acceptable risk and to my knowledge worked out pretty well.
> Over 2 days... 1 in 10 of those registered on campus are or were infected. 
> Of those infected... 40% have been cleaned up but most of the effort has 
> been in getting new students online rather than cleaning infections.  
> If anyone has any questions or found this useful, let me know.
> Phil
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> Phil Deneault     "We work in the dark, We do what we can,
> deneault at wpi.edu   We give what we have. Our doubt is our passion,
> WPI NetOps         and our passion is our task. The rest is the
> InfoSec            madness of art." - Henry James
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

More information about the unisog mailing list