There are moments that every CIO and IT administrator dreads. One of them, possibly highest on the list of dreaded moments, is when a system administrator comes to you and says “one of our servers has been compromised”. When that happens, you must immediately start thinking about technological, legal and social issues all at once. If possible, you also have to try to keep track of what you are learning as the situation unfolds. And these situations always unfold — you never have all the information right at the outset.
At CIS, we had one of those dreaded moments on the afternoon of January 5th. That’s when we discovered that unauthorized attempts to log in to several of our servers had been successful earlier in the day (at 2:15am). It was the vigilance of an IT administrator at the Claremont Consortium that first drew our attention to a problem: he had noticed a number of unsuccessful login attempts, all coming from one machine on our network.
The Sakai server was one that had been compromised. You will remember that HMC is the “lead college” for Sakai; we provide the Sakai service to all the other Claremont Colleges. And people keep a lot of data on Sakai. So this was potentially a very serious security breach. If data had been accessed then we would have to notify everyone in Claremont and possibly much further afield. Reluctantly, we made the decision to take Sakai off-line for a full investigation. A team of six people from HMC and CUC spent most of January 6th working on this issue, and on creating a new Sakai server for use in case it was needed. We determined that the intrusion had been minimal (a six second login, probably automated and designed to establish the fact that the account had been compromised). By the evening of January 6th we were able to bring Sakai back on line.
The machine (a desktop) that was attempting to reach other systems had been compromised, but that the starting point of this attack was a departmental web server, which had been compromised some time in December. The departmental web server had been subject to a “SQL injection attack”, a way of sending queries to a server that end up allowing a hacker to gain control of the machine. Once the hackers had gained control of the web server, they waited and captured some usernames and passwords that allowed them to move on to another machine.
I am glad to say that we got the attack under control very quickly and that no data was exposed. We learned many lessons in the process.
Changes we’ve since made include:
- reviewed all production servers for signs of compromise
- locked down production servers
- changed all system administrator passwords
- introduced better password management policies for our system administrators
- solicited information from security firms for vulnerability scanning
We will be performing vulnerability scanning on servers managed by CIS and are interested in working with departments to provide the service to them too.
We were fortunate this time.
Please keep in mind that your own security practices contribute to the overall security of the network to which all of our systems are connected. Make sure your system is kept up to date with security patches, you are running anti-virus software and you are running only the services you need to run (eg. don’t run a web server if you don’t need to). You’ll find a (flash based) tutorial on digital self-defense from the Rochester Institute of Technology at http://security.rit.edu/contest/dsdsite.html. If you have any questions or concerns, please make sure to contact the CIS Help Desk at firstname.lastname@example.org or (909) 607-7777.