Friday, January 23, 2009

Automatic Updates: Frying Pan vs Fire

Ah, yet another IT related disaster. They make such great blog posts! This time, though, unlike my last one which was a clear cautionary tale about backups worth of a Grimm Brothers tale, the story of the poor boffins at Sheffield Teaching Hospitals Trust has no such simple answer.

Search around with any kind of security related checklist, column, book, blog, or chewing gum wrapper, and one of the constants will be to make sure that all of your security patches are applied. After all, you can be pretty sure that if the hole is sufficiently well known enough for the bad guys to be ready to exploit it. The simplest, most effective, and efficient way to do this is to simply enable whatever automated mechanism your OS has, whether it's up2date on RedHat, updatesd on Fedora, or Automatic Updates on Windows.

In the case of Sheffield, they opted to disable Automatic Updates, and were promptly rewarded with a hospital wide virus outbreak. While the hospital was at least wise enough to design their workflows such that they were able to maintain an acceptable level of patient care, at a minimum they're throwing money out the window on cleanup efforts, including virus removal, and secondary effects such as having to reschedule non-critical procedures.

At first glance - and in just about any other such tale - the moral would be a simple "Leave Automatic Updates on!" But there's a catch. Why were Automatic Updates disabled, you ask? As a matter of fact, until just a few days prior to the outbreak, they were not only enabled, but had a domain policy ensuring that it remained enabled, verified the patches got installed after an internal testing period, and forced a reboot to make the patches take effect, rather than leaving the machine running with the old vulnerable code still in memory.

In exchange for their diligence, Sheffield ended up with a PC deciding to reboot in the middle of a surgery. Can you just imagine being the poor front line helpdesk schmuck who has to explain to a surgeon why his computer decided to reboot all of a sudden? Trying to tell him or her that it's really for the best with a straight face?

Security patches are a critical part of ensuring security of any computer system. Not applying them entails risk; however, given that applying these patches will by definition change behavior somehow, applying them carries its own risk. For far too many computers (mostly, but by no means limited to, Windows), sprinting along on the patch treadmill is the only line of defense against any other machines on the same network. The OS itself is brittle, with nearly any intrusion easily leveraged into total control of the entire machine. Progress is being made, such as UAC on Windows, or SELinux, but that doesn't help with the millions of legacy machines already out there.

Patching these days is a nasty catch 22. With every patch release, you have to take a guess which will be worse - the fallout from applying the patch, or the fallout from not applying the patch. Admins with strict requirements for both availability and security are stuck walking a narrow path, without even any assurance that the patch even exists.

No comments: